Skip to content

Data Pipeline

Jump to bottom
Emirhan Uçan edited this page Aug 5, 2026 · 12 revisions

Data Pipeline

The on-device detection assets (whitelists, signatures, XOR filters, ML models) are generated offline from public threat-intel and NSRL sources, then bundled into the APK.

Scripts Overview

Script Output Source Purpose
gen_whitelist_packages.py whitelist_packages.csv NSRL RDS SQLite Full-detail Android package whitelist (key,md5 pairs) for on-device CSV lookup
gen_whitelist_apk.py Binary-fuse XOR filter NSRL RDS Whole-APK MD5 hashes for native whitelist filter
gen_ip_lists.py IP XOR filters Public blocklists Known-malicious IP/C2 addresses
build_url_xfilter.py URL XOR filters Public blocklists Malicious/phishing URL patterns
build_xfilters.sh XOR filters Various Domain-level filtering
clam_juice.py Filtered ClamAV DB ClamAV Android-relevant ClamAV signatures
nsrl_sql.py (intermediate SQLite) NSRL RDS NSRL database processing utilities — offline only, not bundled in APK
fdroid-downloader.py Benign APK collection F-Droid Downloads F-Droid APKs for the benign dataset pipeline

Workflow

┌────────────────────┐    ┌────────────────────┐    ┌────────────────────┐
│  Public Sources    │    │  Generation Script │    │  Output Assets     │
├────────────────────┤    ├────────────────────┤    ├────────────────────┤
│ NSRL RDS           │───►│ gen_whitelist_*.py │───►│ whitelist_packages │
│ Public IP/Domain   │───►│ gen_ip_lists.py    │───►│ IP/URL XOR filters │
│   blocklists       │    │ build_url_xfilter.py│───►│ IP/URL XOR filters │
│ ClamAV Database    │───►│ clam_juice.py      │───►│ Filtered .cld/.cdb │
└────────────────────┘    └────────────────────┘    └────────────────────┘
                                                           │
                                                           ▼
                                                  ┌────────────────────┐
                                                  │ Bundled into APK   │
                                                  │ app/src/main/assets│
                                                  └────────────────────┘

ClamAV Filtering

clam_juice.py filters ClamAV signatures to keep only Android-relevant platforms:

python clam_juice.py --directory database_non_filtered --output database_filtered --profile cross-platform

Kept: Andr, Unix, Linux, Email, PUA + all Phishing signatures Excluded: Win, Osx, Java

Running the Pipeline

All scripts run with Python 3.x and the required dependencies:

# Install dependencies
pip install requests

# Generate whitelists
python gen_whitelist_packages.py
python gen_whitelist_apk.py

# Generate IP/URL filters
python gen_ip_lists.py
python build_url_xfilter.py

# Filter ClamAV database
python clam_juice.py --directory database_non_filtered --output database_filtered --profile cross-platform

# Build XOR filters
bash build_xfilters.sh

Dataset Pipeline

  • fdroid-downloader.py — downloads benign APKs from the official F-Droid repository.
  • dataset/benign/F-Droid/ — 2805 F-Droid open-source APKs (benign by nature).
  • dataset/malware/MalwareBazaar/ — 987 malware APKs from MalwareBazaar.

YARA Rule Generation (yarGen)

Generate YARA rules from the dataset malware samples:

cd dev-tools\yarGen

# 1. Download goodware string DBs from online repository
python yarGen.py --update

# 2. Create a benign reference DB from F-Droid APKs
python yarGen.py -g "C:\...\dataset\benign\F-Droid\16-07-2026-14.49" -c -i android_benign_fdroid

# 3. Generate YARA rules from malware APKs, excluding F-Droid goodware strings
python yarGen.py -m "C:\...\dataset\malware\MalwareBazaar\27.06.2026 - 203930_212345\apk" ^
  -g "C:\...\dataset\benign\F-Droid\16-07-2026-14.49" ^
  --excludegood --meaningful-words-only --nofilesize --nosimple ^
  -r "https://github.com/HydraDragonAntivirus" -l "GPLv2" -a "HydraDragonAntivirus" ^
  -e "C:\...\dataset\malware\...\apk_strings" ^
  -o "C:\...\yara-x\machine_learning_apk.yar"

ML Model Pipeline (hydradragonml)

Building the ML model requires three steps in order:

cd hydradragonml

# Step 1 — train the model (requires the vocab.json vocabulary, built over the
# same corpus so training and inference tokenize identically). Writes
# model.mpk plus features.json (per-corpus percentile normalization) and a
# vocab.json copy next to it.
cargo run --release --bin hydradragonml-train -- `
  --benign  ..\dataset\benign `
  --malware ..\dataset\malware `
  --vocab   vocab.json `
  --output  model.mpk

# Step 2 — evaluate against the dataset
cargo run --release --bin hydradragonml-scan -- `
  --dataset ..\dataset\ `
  --model   model.mpk `
  --vocab   vocab.json `
  --features features.json `
  --threshold 0.5

hydradragonml-train writes features.json next to model.mpk and copies vocab.json there too; copy all three files into app/src/main/assets/scan/ before building the Android APK (the on-device tokenizer needs vocab.json and inference needs features.json at load time).

See Also

Clone this wiki locally