Skip to content

Rust API Reference

Jump to bottom
Emirhan Uçan edited this page Aug 4, 2026 · 10 revisions

Rust API Reference

Function-level reference for every native (Rust) crate in HydraDragonAV Mobile. The native engine is compiled into libhydradragonandroid.so (one per ABI under app/src/main/jniLibs/) and called from Java via JNI through com.hydradragon.antivirus.engine.NativeScanner.

The workspace contains five crates:

Crate Path Role
hydradragonandroid hydradragonandroid/ JNI bridge + full scan pipeline + on-device metadata build
hydradragonclamav hydradragonclamav/ Pure-Rust ClamAV-compatible signature engine + YARA-X bridge
hydradragonml hydradragonml/ Burn neural-network APK classifier
hydradragonextractor hydradragonextractor/ Recursive archive/APK extraction + zip-bomb detection
hydradragonxorfilter hydradragonxorfilter/ Binary-Fuse xor filter format shared by writer + on-device reader

Cross-references: YARA-X-Modules for the hydradragon module schema (incl. the former androguard static-APK part); Detection-Engines for how these crates combine; Developer-Guide for build instructions.

hydradragonandroid — JNI bridge + scan pipeline

The crate root is hydradragonandroid/src/. It exposes native methods on the Java class com.hydradragon.antivirus.engine.NativeScanner, loads every scan asset (ClamAV DB, compiled YARA .yrc rulesets, Burn ML model .mpk + vocab.json, NSRL xor-filter whitelist, package whitelist, MinHash benign signatures, URL/IP xor filters) from the APK assets/ via AAssetManager, runs the full per-file scan pipeline, and returns JSON verdicts to Java.

Source files: asset_reader.rs, benign_db.rs, dex_scan.rs, elf.rs, emulate.rs, ip_scan.rs, lib.rs, url_scan.rs.

asset_reader.rs — AAssetManager FFI wrapper

Thin FFI wrapper around Android's AAssetManager C API. Stores a native AAssetManager* in a process-global AtomicPtr so background threads (which do not hold a JNI env) can read bundled asset bytes by relative path.

Signature Behavior
pub fn from_java(env: *mut c_void, asset_manager: *mut c_void) -> *mut c_void Wraps AAssetManager_fromJava. Input: JNI env pointer + Java AssetManager jobject (both raw c_void). Output: native AAssetManager* or null on FFI failure. Call once on the JNI thread, then pass the result to init. (asset_reader.rs:34)
pub fn init(mgr: *mut c_void) Stores mgr into the static AASSET_MANAGER AtomicPtr (Relaxed). No-op return. Must be called before any read. (asset_reader.rs:43)
pub fn read_file_bytes(relative_path: &str) -> Option<Vec<u8>> Opens relative_path under the stored manager in buffer mode (AASSET_MODE_BUFFER = 3), reads its full length, closes the asset. Returns None if the manager is null, the path has an interior NUL, the asset doesn't exist, or AAsset_read returns negative. Truncates to the actually-read count if fewer bytes than AAsset_getLength were read. (asset_reader.rs:47)
pub fn list_assets(asset_dir: &str) -> Option<Vec<String>> Opens asset_dir as an AAssetDir and iterates AAssetDir_getNextFileName, returning all non-empty filenames. None if manager null / dir open fails / path has a NUL. (asset_reader.rs:72)
pub fn read_all_assets(asset_dir: &str) -> HashMap<String, Vec<u8>> Lists asset_dir, then read_file_bytes of every "{asset_dir}/{name}", returning a map keyed by filename. Empty map on failure. Used at init to slurp the whole assets/ tree into memory. (asset_reader.rs:99)

benign_db.rs — MinHash/Jaccard benign-content whitelist

Content-based benign-APK whitelist. At init it parses benign_signatures.bin; at scan time it computes a 64-value MinHash signature from an APK's token set and declares KNOWN_BENIGN if any stored signature for that package has estimated Jaccard similarity ≥ 0.85, so heavy scanning (ClamAV/ML) can be skipped. This is the MinHash usage in the codebase — it is a benign-content skip, not the ML anomaly detector.

Item Signature Behavior
const pub const K: usize = 64 Number of MinHash permutations; must match gen_benign_signatures.py. (benign_db.rs:25)
const pub const THRESHOLD: f32 = 0.85 Jaccard similarity threshold for KNOWN_BENIGN. (benign_db.rs:28)
type pub type Sig = [u64; K] A single 64-value MinHash signature. (benign_db.rs:33)
struct pub struct BenignDb { sigs: HashMap<String, Vec<Sig>> } In-memory DB: package name → list of known-benign signatures. Fields private. (benign_db.rs:36)
method pub fn load(data: &[u8]) -> Option<Self> Parses the binary format: u32 package count, then per package u8 name length + UTF-8 name + u32 sig count + that many 64×u64 sigs (all little-endian). None on short read / UTF-8 / parse error. Duplicate packages merged via or_default().extend. (benign_db.rs:43)
method pub fn is_known_benign(&self, package_name: &str, tokens: &HashSet<u64>) -> bool false if the package isn't in the DB. Otherwise computes compute_sig(tokens) and returns true if jaccard(&query, s) >= THRESHOLD for any stored s. Used in run_scan's skip_heavy construction. (benign_db.rs:73)
method pub fn package_count(&self) -> usize Number of packages in the DB. (benign_db.rs:88)
method pub fn signature_count(&self) -> usize Total signatures across all packages. (benign_db.rs:93)
fn pub fn compute_sig(tokens: &HashSet<u64>) -> Sig MinHash: sig = [u64::MAX; K]; for each token t and permutation i, h = (t ^ i).wrapping_mul(FNV_PRIME) (FNV_PRIME = 0x0000_0100_0000_01b3), keeping the minimum per slot. (benign_db.rs:100)
fn pub fn jaccard(a: &Sig, b: &Sig) -> f32 Estimated Jaccard similarity = `

dex_scan.rs — DEX string pool + static analysis

DEX string + static-analysis extraction using the dex-analysis/dex-core crates (FossRust dex-parser-analyzer). Produces the decoded string pool, flattened static-analysis findings, and a unique API-call histogram that feeds the hydradragon.api_call() YARA-X module function.

Item Signature Behavior
struct pub struct DexScan { pub text: String, pub findings: Vec<DexFinding>, pub api_calls: Vec<String> } text = decoded string pool (strings + method/class/field names), \n-joined, capped at MAX_TEXT = 8 MiB. findings = static-analysis findings (any severity), capped at MAX_FINDINGS = 64. api_calls = unique API call signatures in Lpkg/Cls;->method(params)return format, serialized as "sig\tcount", capped at MAX_API_CALLS = 4096. (dex_scan.rs:12)
struct pub struct DexFinding { severity, kind: String, class_descriptor: String, message: String } Flattened from dex_analysis::Finding; message is "{id}: {message}". (dex_scan.rs:25)
fn pub fn scan(bytes: &[u8]) -> Option<DexScan> Parses the DEX via dex_core::parse_dex, builds the string pool (breaking at MAX_TEXT), runs dex_analysis::analyze_dex with AnalysisConfig::default() taking the first MAX_FINDINGS, then builds the API-call histogram via graphs::build_xrefs + semantics::pretty_method for each callee in xrefs.method_calls, accumulating counts per signature. The whole body is wrapped in catch_unwind(AssertUnwindSafe(...)) so a malformed DEX returns None instead of panicking. (dex_scan.rs:34)
fn pub fn is_severe(sev: Severity) -> bool matches!(sev, Severity::Critical) — only Critical findings count toward a malicious verdict; High and below are too false-positive-prone. (dex_scan.rs:91)

elf.rs — minimal ELF32/64 parser

Minimal, dependency-free ELF32/ELF64 (little-endian only) parser — just enough to drive Unicorn emulation: machine type, entry point, PT_LOAD segments, plus .dynsym/.dynstr linear scans to locate an exported symbol by name (e.g. JNI_OnLoad) and to enumerate imported (undefined) symbols with their GOT patch addresses.

Item Signature Behavior
const pub const EM_ARM: u16 = 40, EM_AARCH64: u16 = 183, EM_386: u16 = 3, EM_X86_64: u16 = 62 ELF e_machine constants. (elf.rs:8-11)
struct #[derive(Clone, Debug)] pub struct Segment { pub vaddr: u64, pub offset: u64, pub filesz: u64, pub memsz: u64 } One PT_LOAD segment. (elf.rs:16)
struct #[derive(Clone, Debug)] pub struct ElfInfo { pub is_64: bool, pub machine: u16, pub entry: u64, pub segments: Vec<Segment> } is_64 kept for API completeness; emulate.rs derives everything from machine. (elf.rs:24)
fn pub fn parse_elf(data: &[u8]) -> Option<ElfInfo> Validates \x7fELF magic, requires ei_data == 1 (little-endian; rejects big-endian as "not an Android target"), reads e_machine at offset 18, and depending on ei_class (1=32, 2=64) reads e_entry, e_phoff, e_phentsize, e_phnum. Iterates up to min(phnum, 4096) program headers, keeping only PT_LOAD (type 1) segments with 0 < memsz < 0x1000_0000. None for anything malformed. (elf.rs:52)
fn pub fn find_dynsym(data: &[u8], want: &str) -> Option<u64> Locates .dynsym via the section-header table (using e_shstrndx), resolves its paired .dynstr via sh_link, then linearly scans symbols (capped at 65536). Returns the virtual address (st_value) of the first symbol whose name equals want and whose st_value != 0 (exported/defined). None if malformed/big-endian/.dynsym absent/name not found. (elf.rs:125)
struct #[derive(Clone, Debug)] pub struct Import { pub name: String, pub got_addr: u64 } An undefined (imported) dynamic symbol and the relocation address (r_offset) that should hold its resolved pointer. (elf.rs:224)
fn pub fn find_imports(data: &[u8]) -> Vec<Import> Walks all SHT_REL (9) / SHT_RELA (4) relocation sections (capped at 1,000,000 entries each), extracts r_offset and r_info, computes sym_idx = r_info >> 32 (64-bit) or r_info >> 8 (32-bit), and for each nonzero sym_idx whose dynsym entry is undefined (st_value == 0) emits Import { name, got_addr: r_offset }. Rel vs Rela addend distinction intentionally ignored. Not handled: Android's packed .android.rela (APS2/LEB128) format. Empty vec for malformed/big-endian ELF or missing .dynsym. (elf.rs:234)

emulate.rs — Unicorn CPU emulation

Pure-CPU Unicorn-Engine emulation of native .so libraries to reveal runtime-decoded strings (rolling-XOR/RC4/custom-cipher decode loops) and traced calls into a curated set of suspicious imported APIs (network/file/exec/anti-debug/fingerprinting). No syscalls, no JNI, no libc are emulated; execution halts on the first real unmodeled call. Bounded by both instruction count and a wall-clock timeout, plus a watchdog that force-stops a hung emu_start via a direct uc_emu_stop FFI call to work around a known ARM64-host JIT hang.

Constants: MAX_INSN = 200_000, TIMEOUT_US = 500_000, STACK_BASE = 0x7000_0000, STUB_BASE = 0x6000_0000, HARD_DEADLINE = 600 ms, MIN_STRING_LEN = 6, MAX_CONCURRENT_EMULATIONS = 2, MAX_HARD_TIMEOUTS = 1.

Item Signature Behavior
fn pub fn probe_emulation() -> bool One-shot startup probe (cached in EMULATION_PROBE: AtomicI8, 0=untested/1=available/-1=unavailable). Spawns a thread running a 4-instruction ARM64 snippet (mov x0,#42; add x0,x0,#1; mov x1,#0; ret) under a 10 ms timeout; caller waits via rx.recv_timeout(300 ms). true iff it succeeded. Subsequent calls return the cached result. (emulate.rs:73)
fn pub fn emulation_available() -> bool Reads EMULATION_PROBE; if 0 (not yet probed) assumes true (the scan path has a per-call HARD_DEADLINE safety net), otherwise returns v > 0. (emulate.rs:121)
fn pub fn host_arch() -> &'static str Compile-time cfg!(target_arch=...) → "ARM64 (AArch64)" / "ARMv7 (32-bit)" / "x86_64" / "x86 (32-bit)" / "unknown". (emulate.rs:137)
fn pub fn unsupported_reason() -> &'static str Constant diagnostic: "ARM64 JIT backend hang detected — Unicorn emulation disabled for this session". Read by nativeEmulationReason. (emulate.rs:155)
struct #[derive(Clone, Debug)] pub struct ApiCall { pub name: String } One observed call into a tracked imported API. (emulate.rs:245)
struct #[derive(Clone, Debug, Default)] pub struct EmulationResult { pub strings: Vec<String>, pub api_calls: Vec<ApiCall> } Result of emulating one buffer. (emulate.rs:254)
fn pub fn emulate(so_bytes: &[u8]) -> EmulationResult Entry point called from lib.rs (wrapped in catch_unwind). Fast-path returns Default if emulation_available() is false, parse_elf returns None, no segments, machine isn't ARM/AArch64/x86/x86_64, or find_dynsym(.., "JNI_OnLoad") (falling back to info.entry) is 0. Then claims one of 2 process-wide slots via a non-blocking CAS on EMULATION_SLOTS (skips immediately if both busy — never waits). Spawns a worker thread running emulate_body, sharing a stop_handle: Arc<AtomicUsize> that the worker stores the raw uc_engine handle into before emu_start. Caller waits via rx.recv_timeout(HARD_DEADLINE = 600 ms). On timeout it calls uc_emu_stop(handle) through C FFI, increments HARD_TIMEOUT_COUNT, and at MAX_HARD_TIMEOUTS = 1 permanently disables emulation (EMULATION_PROBE = -1). Returns the worker's result, or default on timeout. (emulate.rs:278)

Internal emulate_body (emulate.rs:363): selects (arch, mode, sp_reg, pc_reg) and lr_reg (None on x86) from info.machine. Maps every PT_LOAD segment page-aligned RWX (skipping segments > 64 MiB), copies file bytes in, maps a dedicated stack at 0x7000_0000 (size 4 * PAGE), sets SP near the top. Maps a stub page at STUB_BASE = 0x6000_0000; for every tracked import from elf::find_imports, patches that import's GOT slot to a unique stub address and records stub_addr -> name in a shared Rc<RefCell<HashMap>>. Adds a code hook over the stub page that, on hit, records the ApiCall (capped at 256), then "returns" from the fake call by setting PC to LR (ARM/AArch64) or popping the return address off the stack (x86/x86_64); if the return target is 0 it calls emu_stop. Arms stop_handle with uc.get_handle() before uc.emu_start(entry, 0, TIMEOUT_US = 500_000, MAX_INSN = 200_000), clears it after. Then harvests every mapped region's live memory and extracts new printable ASCII strings (length ≥ 6, capped at 64) that do NOT already occur verbatim in the original file bytes (extract_new_ascii_strings, emulate.rs:539).

TRACKED_APIS (emulate.rs:228): socket, connect, send, sendto, recv, recvfrom, gethostbyname, getaddrinfo, inet_addr, inet_pton, open, open64, fopen, creat, unlink, remove, rename, system, popen, execve, execl, execvp, fork, vfork, dlopen, dlsym, ptrace, kill, __system_property_get.

ip_scan.rs — malicious-IP exact-membership lookup

Native malicious-IP exact-membership lookup against per-category Binary-Fuse (xor) blocklists (the non-CIDR entries of the allips set). No subnet/CIDR matching.

Item Signature Behavior
struct pub struct IpScanner { filters: Vec<CatFilter> } Private CatFilter { category: &'static str, filter: XorFilter }. (ip_scan.rs:11)
fn pub fn from_bytes_map(files: &HashMap<String, Vec<u8>>) -> Option<IpScanner> For each (stem, category) in the priority-ordered CATS table (ipmalware→MALWARE_IP, ipphishing→PHISHING_IP, ipbruteforce→BRUTEFORCE_IP, ipddos→DDOS_IP, ipspam→SPAM_IP), loads {stem}.xf via XorFilter::from_bytes. None if zero filters loaded. Order = severity priority. (ip_scan.rs:26)
fn pub fn scan(&self, ip: &str) -> Option<&'static str> Trims input; None if empty. Iterates filters in priority order and returns Some(category) for the first filter.contains(ip) hit, else None. Exact match on the trimmed canonical textual IP. (ip_scan.rs:44)

url_scan.rs — URL/domain threat lookup

Native URL/domain threat lookup — the Rust port of Java's UrlThreatScanner. All membership is Binary-Fuse (xor) filters loaded from .xf assets. Mirrors Java: an http(s) URL with a real path → URL scan against the *_URL xor filters (full scheme-less URL); otherwise → domain scan against the domain xor filters (registrable main domain via the public-suffix list).

Item Signature Behavior
struct pub struct UrlScanner { filters: Vec<CatFilter>, suffixes: HashSet<String> } Private CatFilter { category: &'static str, is_url: bool, filter: XorFilter }. (url_scan.rs:24)
fn pub fn load_from_assets(files: &HashMap<String, Vec<u8>>) -> Option<UrlScanner> For each (stem, category, is_url) in CATS (malwareurl→MALWARE_URL/true, phishingurl→PHISHING_URL/true, phishing→PHISHING/false, malicious→MALICIOUS/false, malicious_mail→MAIL/false, abuse→ABUSE/false, spam→SPAM/false, mining→MINING/false), loads {stem}.xf. None if zero filters loaded. Also parses public_suffixes.txt into suffixes (lowercased, skipping blank and //-comment lines). (url_scan.rs:43)
fn pub fn scan(&self, url: &str) -> Option<&'static str> Lowercases + trims; None if not http:// / https://. Strips the scheme to get norm (host[:port]/path). Computes has_path (a slash that isn't the last char). Strips any :port from the host. If has_path → URL scan: first is_url filter whose filter.contains(norm) matches (http and https to the same host+path normalize to the same key). Else → domain scan: main_domain(host) against the non-URL filters. Some(category) or None. (url_scan.rs:96)
fn (private) fn main_domain<'a>(&self, host: &'a str) -> String PSL registrable-domain derivation: mc.yandex.ru → yandex.ru; a host whose suffix is listed (e.g. com.tk) returns p[i-1..].join("."). Falls back to last-two-labels. (url_scan.rs:73)

lib.rs — JNI bridge and scan pipeline (4196 lines)

The JNI bridge exposing native methods on com.hydradragon.antivirus.engine.NativeScanner, plus the whole per-file scan pipeline: extract → metadata build → YARA-x/ClamAV → ML → result aggregation → JSON verdict. Manages the Engine (RwLock-guarded, OnceLock'd), hot rule learning, VPN rules, and all user-configurable atomic toggles.

Module-level constants and globals

Item Location Notes
const YRC_FILES: &[&str] lib.rs:96 Rulesets loaded at init: clean_rules_filtered_verified.yrc, valhalla-rules_filtered_verified.yrc, machine_learning_apk.yrc, hydradragon.yrc, hips_rules_filtered_verified.yrc.
const MODEL_MPK: &str = "model.mpk" lib.rs:110 Burn ML model file (.mpk recorder), loaded via Model::load.
const WHITELIST_XF: &str = "whitelist.xf" lib.rs:125
const WHITELIST_PACKAGES_DB: &str = "whitelist_packages.csv" lib.rs:134
const BENIGN_SIGNATURES: &str = "benign_signatures.bin" lib.rs:135
static NATIVE_EMULATION_ENABLED: AtomicBool = true lib.rs:184
static MAX_SCAN_SIZE_MB: AtomicU32 = 650 lib.rs:196
static SCAN_RELEVANT_ONLY: AtomicBool = true lib.rs:214
static VPN_RULES_LOADED / VPN_SCAN_ENABLED: AtomicBool lib.rs:238 / 243
static JAVA_VM: OnceLock<jni::JavaVM> lib.rs:172 Lets Rust background threads call back into Java.
static ENGINE: OnceLock<RwLock<Engine>> lib.rs:178
static ASSET_FILES: Mutex<Option<HashMap<String, Vec<u8>>>> lib.rs:257
static INIT_DIR: OnceLock<String> lib.rs:267 Writable path for generated_rules/.
static INIT_STARTED: AtomicBool lib.rs:271
static LAST_PANIC: Mutex<Option<String>> lib.rs:387
static INIT_STATUS: Mutex<String> lib.rs:481
static SCAN_SERIAL: Mutex<()> lib.rs:412 SCAN_SERIAL_MAX_WAIT = 5s (lib.rs:413).

The Engine struct (lib.rs:144)

struct Engine {
    clamav: Option<ClamavEngine>,
    model: Option<Model>,
    whitelist: Option<XorFilter>,
    package_whitelist: HashMap<String, String>,
    benign_db: Option<benign_db::BenignDb>,
    url_scanner: Option<url_scan::UrlScanner>,
    ip_scanner: Option<ip_scan::IpScanner>,
}

JNI-exposed functions (25 total)

All live on Java class com.hydradragon.antivirus.engine.NativeScanner. Each uses env.with_env(...).resolve::<LogErrorAndDefault>() so a pending JNI exception is logged and a default value returned rather than aborting. The Rust function names follow Java_com_hydradragon_antivirus_engine_NativeScanner_<methodName>.

Rust fn (file:line) Java signature Returns to Java
nativeInit (1689) boolean nativeInit(String assetDir, boolean loadAutoRules, Object assetManager, String filesDir) true only if the engine was already loaded; otherwise spawns the ~70 s background init thread, sets INIT_STARTED, returns false. Stores the JVM in JAVA_VM, converts the Java AssetManager via asset_reader::from_java, reads all assets, builds the engine via do_init_from_assets, sets ENGINE if clamav or model loaded. Lowers its own scheduler priority to ANDROID_PRIORITY_BACKGROUND (nice 10) so ~8 s of cold-start CPU doesn't starve the UI thread.
nativeIsReady (1810) boolean nativeIsReady() ENGINE.get().is_some() — true once the async init finished.
nativeSetEmulationEnabled (1821) void nativeSetEmulationEnabled(boolean enabled) Stores into NATIVE_EMULATION_ENABLED.
nativeIsEmulationAvailable (1858) boolean nativeIsEmulationAvailable() emulate::probe_emulation() — cached; Java shows R.string.unicorn_unsupported when false.
nativeEmulationReason (1868) String nativeEmulationReason() emulate::unsupported_reason().
nativeHostArch (1880) String nativeHostArch() emulate::host_arch().
nativeSetMaxScanSizeMb (1894) void nativeSetMaxScanSizeMb(int maxMb) Stores max_mb.max(1) into MAX_SCAN_SIZE_MB.
nativeSetDetectZipBomb (1939) void nativeSetDetectZipBomb(boolean enabled) Calls hydradragonextractor::set_bomb_detection_enabled.
nativeSetScanRelevantOnly (1951) void nativeSetScanRelevantOnly(boolean on) Stores into SCAN_RELEVANT_ONLY.
nativeLearnRule (1996) boolean nativeLearnRule(String yarPath) Hot-loads one .yar into the live ClamAV engine via a brief write lock on ENGINE; true if add_yara_source_file returned Ok(Some(_)).
nativeStatus (2145) String nativeStatus() The INIT_STATUS report string (what loaded/failed during init).
nativeIsHashWhitelisted (2173) boolean nativeIsHashWhitelisted(String md5) whitelist.contains(md5) against the NSRL xor filter.
nativeIsHashWhitelistedForFile (2196) boolean nativeIsHashWhitelistedForFile(String path, String md5) Same, but first reads the file's first 2 bytes and requires PK\x04\x03 (ZIP magic) so non-APKs are never whitelisted.
nativeScanUrl (2234) String nativeScanUrl(String url) The malicious category (e.g. "PHISHING") from url_scanner.scan, or "" if clean/not a URL.
nativeScanIp (2257) String nativeScanIp(String ip) The category (e.g. "MALWARE_IP") from ip_scanner.scan, or "".
nativeScanText (2283) String nativeScanText(String text) Comma-joined matched rule/sig names from scan_text, or "".
nativeScanHips (2303) String nativeScanHips(String hipsJson) JSON `{"malicious":..,"matches":[..],"suggestion":"uninstall"
nativeEnableVpnScan (2421) void nativeEnableVpnScan(boolean enable) On true: load_vpn_rules() then sets VPN_SCAN_ENABLED; on false: clears VPN_SCAN_ENABLED (rules stay loaded).
nativeScanPackets (2459) String nativeScanPackets(String packetsJson) JSON {"malicious":..,"matches":[..]} from scan_packets; {"malicious":false} no-op if VPN scan disabled.
nativeScanApk (2479) String nativeScanApk(String path, String hydradragonJson, String fileMd5, boolean zeroTrust) The full scan verdict JSON from scan_apk → run_scan.

Key internal (non-JNI) functions

Signature Behavior
fn do_init_from_assets(files: &HashMap<String, Vec<u8>>, load_auto_rules: bool) -> Engine (lib.rs:489) Parallel init via std::thread::scope spawning 7 threads: ClamAV (+ all YRC_FILES compiled in parallel + optional generated_rules/*.yar from INIT_DIR), Burn ML model, NSRL whitelist, package whitelist, URL scanner, IP scanner, benign DB. Each wrapped in catch_unwind and producing a status-report fragment; set_status publishes the concatenated report. Returns the populated Engine.
fn load_package_whitelist_from_bytes(bytes: &[u8]) -> HashMap<String, String> (lib.rs:796) Parses whitelist_packages.csv (RFC-4180, one "key",md5 per line) from raw bytes into a HashMap<String, String>. Strips surrounding quotes via unquote_csv_field, lowercases the md5, skips empty/malformed lines. No temp file, no SQLite.
fn load_vpn_rules() -> bool (lib.rs:2321) Idempotent (guarded by VPN_RULES_LOADED); under a write lock on ENGINE, compiles emerging-all.yrc from ASSET_FILES and add_compiled_yara into the live ClamAV engine.
fn scan_hips(hips_json: &str) -> String (lib.rs:2351) Early-validates JSON; builds module_meta = [("hydradragon", hips_json)]; scans the JSON bytes named "hips_behavior"; returns `{"malicious":..,"matches":[..],"suggestion":"uninstall"
fn scan_text(text: &str) -> String (lib.rs:2390) Caps text at 8192 bytes; builds module_meta = [("hydradragon", r#"{"screen_text":"..."}"#)]; scans named "screen_text"; returns comma-joined match names.
fn scan_packets(packets_json: &str) -> String (lib.rs:2440) No-op {"malicious":false} if VPN_SCAN_ENABLED false; validates JSON; builds module_meta = [("hydradragon", r#"{"network":{"packets":...}}"#)]; scans named "vpn_traffic"; returns {"malicious":..,"matches":[..]}.
fn scan_apk(env, path, hydradragon_json, file_md5, zero_trust) -> String (lib.rs:2493) Reads the file, takes a read lock on ENGINE, calls run_scan on on_big_stack (64 MB stack). On a thread panic returns {"error":"scan panicked: ...","malicious":false} using LAST_PANIC.
fn run_scan(engine, bytes, path, hydradragon, file_md5, zero_trust) -> String (lib.rs:2697) The master pipeline (see data flow below). Returns the verdict JSON.
fn generate_yara_rule(file_hash, packages, detections, dex_scans, only_index) -> Option<String> (lib.rs:3582) Builds a yarGen-style YARA rule named auto_{file_hash}, import "hydradragon". Strings: up to 40 DEX string-pool entries (length 8–128, no control chars, deduped), scoped to only_index if given. Condition groups: package-name ORs via hydradragon.package_name(...), N of them (N = strings.len().min(6).max(1)), hydradragon.rootkit_behavior() == 1, plus hydradragon.api_call(/.../) > 0 for launcher-hijack / suspicious-API patterns and hydradragon.dex_severe_finding_count() > 0 when severe findings exist. Also emits a HIPS runtime branch (OR'd) using hydradragon.ui_spam / notification_spam / clickjack / ransomware / strandhogg / removal_resistance / launcher_change / network_connections with a package regex. None if both strings and packages are empty.
fn parse_manifest(data: &[u8]) -> Option<Manifest> (lib.rs:4126) Full binary-AXML walker producing Manifest { package, app_name, permissions, activities, services, receivers, main_activity, min_sdk, max_sdk, target_sdk }. Tracks an activity_stack and in_intent_filter/has_action_main/has_category_launcher to find the enabled MAIN/LAUNCHER activity (the icon-hiding enabled="false" + kept intent-filter trick is handled). Guard-bounded (200k chunks, 256 attrs, 64-deep activity stack, 4096 components).
fn build_hydradragon_json(buffers: &[Buf], urls: &[String]) -> Option<String> (lib.rs:4293) Finds the first parseable manifest, extracts the certificate, emits the hydradragon JSON (see module metadata contract).
fn extract_certificate(buffers: &[Buf]) -> Option<CertInfo> (lib.rs:4357) Finds the `META-INF/*.RSA
fn collect_buffers(data, top_md5, path) -> (Vec<Buf>, Vec<(String,String,Vec<String>)>) (lib.rs:4746) Phase 1 extractor. Parallel work-stealing over a shared Mutex<Vec<WorkItem>> stack with an outstanding counter for termination; available_parallelism().clamp(1,4) workers; each wrapped in catch_unwind (a panicking worker sets capped=true to force-stop the rest). Caps: depth < 16, 4096 buffers, 2 GB total decompressed. Zip-bomb errors (hydradragonextractor::is_bomb_error) become HDR.Bomb.Decompression detections. Each Buf carries its apk_lineage (MD5s of ancestor zips, including its own when it is a zip) and entry_name.
fn merge_dex_findings(hydradragon: Option<&[u8]>, dex_scans: &[Option<DexScan>], manifest_report: Option<&str>) -> Option<Vec<u8>> (lib.rs:3978) Folds the manifest report (AXML + URLs + cert) into the root object, adds DEX findings (any severity) into a dex_findings array and aggregates per-API-signature invocation counts into an api_calls array ("sig\tcount", sorted, capped 4096), merged on top of Java's hydradragon JSON (or an empty object). None if the result is an empty object.
fn max_dangerous_perms(buffers: &[Buf]) -> usize (lib.rs:3829) Counts distinct DANGEROUS_PERMS (lib.rs:3762, ~37 entries) present in any buffer ≤ 4 MiB, checking both UTF-8 and UTF-16LE encodings; returns the max over buffers.
fn collect_apk_hashes(buffers, top_md5) -> Vec<String> (lib.rs:4664) Lowercase MD5 of every zip/APK buffer (reuses top_md5 for the top-level), deduped, capped 64.
fn collect_packages(buffers: &[Buf]) -> Vec<String> (lib.rs:4689) axml_package of each buffer, deduped, capped 64.
fn extract_and_scan_urls(engine: &Engine, decoded: &[u8]) -> Vec<String> (lib.rs:2598) Splits decoded emulation strings on \n, keeps lines starting http(s)://, scans each via url_scanner, emits "URL.{cat}: {url}" (capped 16).
fn extract_decode_base64_urls(data, scanner) -> Vec<String> (lib.rs:2644) Scans for the B64_URL_PREFIXES (lib.rs:2628: standard + UTF-16LE base64 of http:///https://), extends over base64 chars + up to 2 = (≤100 bytes), decodes, scans decoded URLs via scanner. Emits "URL.{cat}: {s}" (capped 16).
helpers md5_hex (4651), sha1_hex/sha1_hash (4575/4585, hand-rolled RFC 3174), json_escape (461), is_obfuscated_xml (1185), is_resource_path (1191), is_relevant_buffer (1365), is_executable_buffer (1225), is_media_file (1278), is_image_buffer (1305), has_embedded_data (1424), skip_by_size (853, ≤12 || > MAX_SCAN_SIZE_MB MB), on_big_stack (1667, 64 MB stack thread), acquire_scan_serial_bounded (418, 5 s budget then proceeds without the lock), install_panic_hook (439), last_panic (5016), build_engine_features (917), android_log/rust_timing_log! (62/75).

The Buf struct (lib.rs:4718): { data: Vec<u8>, apk_lineage: Vec<String>, entry_name: Option<String>, fmt: Option<&'static str> }.

Data flow — the main scan pipeline

Entry: nativeScanApk (lib.rs:2479) → scan_apk (lib.rs:2493) → on_big_stack(|| run_scan(...)) (lib.rs:2697). The SCAN_SERIAL mutex is acquired with a 5 s bounded wait first (so one stuck file can't freeze every other scan).

  • Phase 0 — serial gate. acquire_scan_serial_bounded() (lib.rs:418, called at lib.rs:2709).
  • Phase 1 — streaming extract+scan (collect_buffers, lib.rs:4746, called at lib.rs:2764). Recursively unpacks the file and all nested archives (zip/gz/tar/xz/lzma/7z/rar via hydradragonextractor) into Vec<Buf> with apk_lineage MD5 chains, while ClamAV+YARA scan each buffer inline as extraction proceeds (streaming timing/dets accumulated). Zip-bomb errors become HDR.Bomb.Decompression detections returned alongside. Capped at 4096 buffers / 2 GB / depth 16. Timed as extract_ms.
  • Phase 2 — metadata + fast passes + skip computation (lib.rs:2768). All run before any heavy scan: max_dangerous_perms → perm_count; collect_packages (AXML package names); collect_apk_hashes (reuses Java's MD5 for the top-level); build_hydradragon_json (manifest + URLs + cert); whitelist skip_heavy construction (NSRL hash, MinHash benign, ancestor-lineage hash/package checks); DEX static analysis (dex_scan::scan on each non-skipped dex\n buffer, timed as dex_ms); merge_dex_findings → hydradragon_meta; module_meta built (single ("hydradragon", …)); native emulation (emulate::emulate on each non-skipped \x7fELF buffer, deduped by MD5, capped at MAX_EMULATED_BUFFERS = 8, timed as emulate_ms). Timed as phase2_ms.
  • All-whitelisted short-circuit (lib.rs:2953). If every skip_heavy[i] is true, skip all Phase 3 and return a clean verdict (only bomb_dets can make it malicious).
  • Phase 3 — parallel heavy phases (lib.rs:3006). YARA rescan, emulation signal, ML and URL scanning all depend on read-only buffers/dex_scans/module_meta, so they run concurrently inside one std::thread::scope (wall time = slowest phase, not the sum). Threads: (1) module-meta YARA rescan + emulation signal over flagged buffers, (2) ClamAV/YARA content scan over the precomputed scan_items (see below), (3) ML model over zip buffers. Produces clamav_ms, per-engine yara_total_ms, ml_ms; whole scope wall time logged as scope_ms.
  • Scan-item hoist (lib.rs:3039). The eligible-buffer work list (ScanItem { idx, obj_path, data, is_vid }) is computed once up front so ClamAV and YARA passes run on two separate threads over the same read-only slice, overlapping runtimes (wall ≈ max, not sum). Video buffers are re-encoded to metadata via media_scan::extract_metadata; media gating honors SCAN_MEDIA_ENABLED/SCAN_RELEVANT_ONLY.
  • Detection aggregation. detections = bomb_dets + streaming_dets (minus whitelisted) + yara_dets + ml_lineages + severe DEX findings (DEX/{sev}: {msg}).
  • Phase 4 — generated rule + JSON assembly (lib.rs:~3577). malicious = !detections.is_empty(). Computes scoped_entry_idx (when all detections share the same outer!/entry suffix, scope the generated rule to that entry). generate_yara_rule is built iff malicious || zero_trust (auto-rule generation also gated by the AUTO_RULE_GEN_ENABLED native flag). Builds per-entry entry_md5s maps (capped 1024). Emits the per-stage timing breakdown to logcat (HydraDragon-RustTiming): extract/dex/emulate/phase2/clamav/yara/ml/scope. Final JSON shape:
{"malicious":..,"matches":[..],"detections":[{"name","object_path","hashes"}],
 "permissions":N,"packages":[..],"hashes":[..],"md5":"..",
 "ml":{"malicious":..,"jaccard":..,"anomaly":..,"nearest":..|null},
 "generated_rule":".."|null,"entry_md5s":{..}[,"error":".."]}

Module metadata contract

The YARA-X module metadata is passed as module_meta: &[(&str, &[u8])] to hydradragonclamav::ClamavEngine::scan_bytes_named[_with_breakdown]. That engine forwards these to yara-x's set_module_metadata(name, bytes) before scanning. One metadata key is pushed for APK scans, consumed by the single hydradragon YARA-X module (the former separate androguard module is folded into it — there is no androguard key anymore):

hydradragon (built by merge_dex_findings, lib.rs:3978): starts from Java's live-network/HIPS hydradragon JSON (the hydradragonJson arg to nativeScanApk, parsed if it's a JSON object) or an empty {}, merges in the manifest report (built by build_hydradragon_json, lib.rs:4293) as the flat object, then adds:

{
  "package_name": String|null,
  "app_name": String|null,
  "main_activity": String|null,
  "activities": [String],
  "services": [String],
  "receivers": [String],
  "permissions": [String],
  "new_permissions": [String],
  "urls": [String],
  "min_sdk_version": String|null,
  "max_sdk_version": String|null,
  "target_sdk_version": String|null,
  "certificate": { "subjectDN": String|null, "IssuerDN": String|null, "sha1": String|null },
  "dex_findings": [ { "severity": "Critical"|..., "kind": "...", "class_descriptor": "...", "message": "..." } ],
  "api_calls": [ "Lpkg/Cls;->method(params)return\t<count>", ... ]
}

Source: parse_manifest for everything except urls (swept inline from the buffers) and certificate.* (extract_certificate → CertInfo). build_hydradragon_json returns None (no manifest fields merged) if no buffer parses as a binary AXML manifest. certificate.* is parsed from the PKCS#7 signature block (META-INF/*.RSA|.DSA|.EC); subject/issuer are OpenSSL-style /key=value/... DNs and sha1 is the SHA-1 of the full certificate DER.

dex_findings is added only if non-empty; api_calls is the aggregated invocation-count histogram across all DEX buffers (each entry "sig\tcount", the count being the sum across buffers). merge_dex_findings returns None (no hydradragon entry pushed) if the merged object is empty.

Special-purpose scans build their own single-entry module_meta: scan_hips → [("hydradragon", hips_json)] named "hips_behavior"; scan_text → [("hydradragon", r#"{"screen_text":"..."}"#)] named "screen_text"; scan_packets → [("hydradragon", r#"{"network":{"packets":...}}"#)] named "vpn_traffic".

The full schema of the module (every field the YARA-X side reads) is in YARA-X-Modules.

hydradragonclamav — ClamAV engine

Pure-Rust reimplementation of ClamAV's non-hash signature loading and scanning engine. Crate root: hydradragonclamav/src/. Dependencies that shape the design: yara-x (fork, features pulley + hydradragon-module + dex-module) for YARA scanning; regex for PCRE subsigs and .cdb/.ftm regex fields; memchr for SIMD literal pre-checks; jdb_xorf (Binary-Fuse16 filters) for the atom prefilter; image + rustdct for the perceptual image fuzzy hash (byte-identical to ClamAV's libclamav_rust).

lib.rs re-exports the headline types: Bytecode, BytecodeSet (from bytecode); ContainerSignature, ContainerType, Database, FileTypeMagic, LoadError, LoadReport, NumSpec, UnsupportedRecord (from database); Engine, ScanMatch, ScanOptions, ScanView, SignatureKind (from scanner); YaraEngine (from yara_scan). All other modules are pub mod (reachable as hydradragonclamav::pattern::Pattern etc.) but not re-exported at the crate root.

Source files (21): atomfilter.rs, atomfilter_build.rs, atomfilter_hash.rs, atomscan.rs, bytecode.rs, bytecode_vm.rs, cert.rs, database.rs, filtering.rs, fuzzy.rs, icon.rs, icon_match.rs, lib.rs, logical.rs, main.rs, pattern.rs, pe.rs, phishing.rs, scanner.rs, version_info.rs, yara_scan.rs (+ phishing/tld.rs).

ENGINE_FLEVEL = 240 (in bytecode_vm.rs) is the crate-wide engine functionality level; database and logical both gate signatures against it.

lib.rs (22 lines)

Crate root. Declares all 14 public modules and re-exports the headline types above. No logic.

pe.rs (112 lines) — PE header parser

Minimal PE/PE32+ header parser used by the scanner and the version-info/icon modules.

  • pub struct PeInfo (:2) — entry_point_offset: Option<usize>, sections: Vec<Section>, vinfo: Vec<u32> (sorted file offsets where VS_VERSION_INFO string entries begin), res_rva: u32 (resource data-directory RVA), size_of_headers: u32.
  • pub struct Section (:16) — raw_start: usize, raw_size: usize, virtual_address: u32, virtual_size: u32.
  • pub fn parse_pe(data: &[u8]) -> Option<PeInfo> (:23) — validates MZ + PE\0\0, reads COFF section count + optional-header size, accepts PE32 (0x10b) or PE32+ (0x20b), builds the section table, resolves the entry-point RVA to a file offset via rva_to_offset, reads SizeOfHeaders and dirs[2], calls version_info::version_info_offsets. None on any bounds/magic failure.
  • pub fn rva_to_offset(rva: u32, sections: &[Section]) -> Option<usize> (:93) — maps an RVA to a raw file offset.

version_info.rs (298 lines) — VS_VERSION_INFO offset extraction

Faithful port of ClamAV's findres(0x10,…) + the VS_VERSION_INFO walk in pe.c. Produces the sorted set of file offsets where version-info string entries begin (ClamAV's peinfo->vinfo, each entry vptr - baseptr + 6), so a VI-anchored signature matches only at one of these.

  • pub fn version_info_offsets(data: &[u8], res_rva: u32, sections: &[Section], hdr_size: u32) -> Vec<u32> (:281) — sorted, deduped file offsets. Empty when res_rva == 0 or no parseable RT_VERSION resource.

cert.rs (238 lines) — .crb Authenticode cert trust/block loader

Loader for .crb (Authenticode certificate trust/block rules), ported from cli_loadcrt. Parses every record byte-exactly; the PKCS#7 verification engine is a separate follow-up, so certs are loaded and counted but not yet matched at scan time.

  • pub struct CertEntry (:33) — name: String, blocked: bool (trusted==0), subject: [u8; 20] (SHA-1), serial: Option<[u8; 20]> (None = ignore_serial), pubkey: Vec<u8>, code_sign: bool, time_sign: bool, cert_sign: bool, not_before: Option<i64>, source: SourceLocation.
  • pub struct CertTrustDb (:54) — certs: Vec<CertEntry>.
    • pub fn is_empty(&self) -> bool (:59); pub fn len(&self) -> usize (:63).
    • pub fn add_line(&mut self, line: &str, source: SourceLocation, flevel: u32) -> Result<bool, String> (:70) — parses name;trusted;subject;serial;pubkey;exp;codesign;timesign;certsign;notbefore;comment[;minFL[;maxFL]]. Returns Ok(false) when f-level gating excludes the engine, Ok(true) when added, Err on malformed.

icon.rs (253 lines) — .idb icon fingerprint loader

Loader for .idb icon signatures (Name:Group1:Group2:Metric), ported from cli_loadidb. Parses the 124-nibble (62-byte) fuzzy-image fingerprint of a PE icon, bucketed by icon size (16/24/32 → bucket (size>>3)-2 = 0/1/2), with interned group names. The matcher (icon_match.rs) is separate.

  • pub struct IconMetric (:23) — name: String, group: [u32; 2], [u32; 3] triples for color_avg/x/y, gray_*, bright_*, dark_*, edge_*, noedge_*, then rsum/gsum/bsum/ccount: u32, source.
  • pub struct IconMatcher (:56) — icons: [Vec<IconMetric>; 3] (size buckets), group_names: [Vec<String>; 2].
    • pub fn is_empty(&self) -> bool (:62); pub fn len(&self) -> usize (:67).
    • pub fn add_line(&mut self, line: &str, source: SourceLocation) -> Result<(), String> (:82) — validates 4 tokens, 124-nibble metric, size byte ∈ {16,24,32}, centroid bounds, color/gray avg ≤ 4072, rsum+gsum+bsum ≤ 103, ccount ≤ 100.

icon_match.rs (1003 lines) — PE icon matcher (matchicon)

Faithful port of ClamAV's pe_icons.c (USE_FLOATS path): walk RT_GROUP_ICON/RT_ICON resources, decode the BMP icon, scale to 16/24/32, compute the getmetrics fingerprint, and compare against loaded .idb fingerprints via matchpoint/matchbwpoint. Evaluates the IconGroup1/IconGroup2 TDB constraint.

  • pub fn matchicon(data: &[u8], sections: &[Section], hdr_size: u32, res_rva: u32, matcher: &IconMatcher, grp1: Option<&str>, grp2: Option<&str>) -> bool (:984) — true iff some PE icon matches an .idb fingerprint in the requested groups. Empty matcher or res_rva == 0 → false.

Internal load-bearing functions: find_resources (general findres, cap 64), getmetrics (the full fingerprint: HSV color presence, top-3 non-overlapping color/gray/bright/dark areas, Sobel edge detection via labdiff, separable Gaussian blur, edge/no-edge area sums, bwonly fallback), matchpoint/matchbwpoint (centroid-distance + average-difference similarity), icon_confident (per-candidate confidence test), parseicon (BMP decode for depths 1/4/8/16/24/32, palette, AND-mask → alpha, alpha-blend over white, scale modes 0/1/2), groupiconscan, GroupSet, lab/labdiff/hsv.

fuzzy.rs (216 lines) — image perceptual pHash

Faithful port of ClamAV's fuzzy_hash_calculate_image, using the same image + rustdct crates so the 64-bit hash is byte-identical to fuzzy_img#<hash> subsignatures. Only exact (hamming distance 0) matches supported, same as ClamAV.

  • pub fn calculate_image(buffer: &[u8]) -> Option<[u8; 8]> (:74) — fast-rejects non-image magic and >50 MB buffers, decodes under catch_unwind, drops alpha, applies ITU-R 601-2 luma, resizes to 32×32 Lanczos3, runs 2-D DCT-2, takes the top-left 8×8 block, thresholds against the median of 64 values, packs big-endian into 8 bytes.
  • pub fn parse_fuzzy_img(raw: &str) -> Result<[u8; 8], String> (:163) — parses fuzzy_img#<hex>[#<distance>]. Rejects unknown algorithms, non-16-char hex, non-zero hamming distance.

filtering.rs (292 lines) — shift-or 2-gram filter

Rust port of ClamAV's libclamav/filtering.c ("A fast filter for static patterns"). A bit-parallel shift-or automaton over overlapping little-endian 2-grams with a second end table; can false-positive but never false-negative.

  • pub struct Filter (:36) — b: Box<[u8; 65536]>, end: Box<[u8; 65536]>, loaded: bool. 128 KiB, boxed.
    • pub fn new() -> Self (:53) — filter_init: all bits set.
    • pub fn is_loaded(&self) -> bool (:61).
    • pub fn add_static(&mut self, pattern: &[u8]) -> Option<usize> (:93) — filter_add_static: picks the best MAXSOPATLEN (8)-byte subpattern by ClamAV's signed-int scoring, registers every overlapping 2-gram, marks the end. Returns the registered length or None if len < 2. Caps at MAXPATLEN (255).
    • pub fn search(&self, data: &[u8]) -> i64 (:182) — filter_search (__hot__): approximate match start position, or -1.
    • pub fn search_ext(&self, data: &[u8]) -> Option<usize> (:205) — exact position of the first possible match end.

atomfilter_hash.rs (103 lines) — rolling polynomial hash

Rolling polynomial hash over fixed-width byte windows, used to derive atom keys for the Binary-Fuse16 filters. Incrementally updatable per byte, so sweeping every length-len window is O(n).

  • pub const ATOM_LENGTHS: [usize; 5] = [2, 4, 8, 12, 16] (:20) — canonical atom-key window lengths (2..=16).
  • pub fn bucket_len(atom_len: usize) -> Option<usize> (:25) — largest entry in ATOM_LENGTHS ≤ atom_len, or None if below 2.
  • pub fn hash_window(data: &[u8]) -> u64 (:32) — from-scratch rolling hash (h = h*ROLL_BASE + b, wrapping mod 2⁶⁴).
  • pub fn roll_windows(data: &[u8], len: usize, emit: impl FnMut(usize, u64)) (:51) — sweeps every length-len window, calling emit(start, hash) incrementally in O(1) amortized per byte.
  • pub(crate) const ROLL_BASE: u64 = 0x0000_0100_0000_01b3 (FNV-1a prime).

atomfilter.rs (161 lines) — atom-filter data model

Data model for the Binary-Fuse16 atom/counter/threshold promotion scanner. A slot's counter reaching threshold is the match — no byte-level re-verification of the atom (Avast-style).

  • pub type SlotId = u32 (:21).
  • pub enum SlotTarget (:24) — Extended { sig_index: u32 } (an .ndb/.db signature) or LogicalSubsig { sig_index: u32, subsig_index: u32 } (one subsig of an .ldb signature, feeding LogicalExpr::eval).
  • pub struct SlotDef (:34) — target: SlotTarget, threshold: u32 (almost always 1).
  • pub struct AtomBucket (:46) — bf: Bf16, slots: HashMap<u64, Box<[SlotId]>>.
    • pub fn resolve(&self, key: u64) -> Option<&[SlotId]> (:61) — returns the slots if bf.has(key) AND key is genuinely in slots (absorbs bare Bf16 false positives).
  • pub enum ExtSlot (:71) — Atom(SlotId) (every pattern had a usable atom; extended sigs match on ANY pattern) or AutoMatch (some pattern fully wildcarded → unconditionally matched).
  • pub enum SubsigSlot (:82) — Atom(SlotId), AutoMatch (a Body subsig with no usable atom), External (non-Body subsig: Pcre/ByteCompare/Fuzzy/Unsupported — left for the scanner's exact-evaluation carve-out).
  • pub struct AtomFilterDb (:101) — buckets: [Option<AtomBucket>; 5], buckets_nocase: [Option<AtomBucket>; 5] (case-sensitive and case-folded, never merged), slots: Vec<SlotDef>, ext_slot: Vec<ExtSlot>, log_subsig_slots: Vec<Box<[SubsigSlot]>>.
    • pub fn empty() -> Self (:120).

atomfilter_build.rs (267 lines) — build the AtomFilterDb

Builds an AtomFilterDb from a loaded Database — the Binary-Fuse16 replacement for ClamAV's Aho-Corasick atom prefilter. Same atom-selection logic (Pattern::required_atom/required_atom_nocase, 2..=16 window, case/nocase split). Every usable atom of every signature/subsignature is indexed.

  • pub struct AtomFilterBuilder; (:124) — unit struct.
    • pub fn build(db: &Database) -> AtomFilterDb (:127) — for each extended signature: if all patterns have usable atoms, allocates one SlotId with threshold: 1 targeting SlotTarget::Extended, registers every pattern's atom, stores ExtSlot::Atom; else ExtSlot::AutoMatch. For each logical signature: per atom-indexable Body subsig, allocates a SlotId targeting SlotTarget::LogicalSubsig; non-Body subsigs become SubsigSlot::External; Body subsigs with any no-atom variant become SubsigSlot::AutoMatch. Finalizes per-bucket Bf16::from(&keys) + the exact key → slot-list map.

atomscan.rs (278 lines) — scan-time atom sweep

Scan-time path for the Binary-Fuse16 atom/counter/threshold scheme. One sliding-window sweep per canonical atom length (both case-sensitive and case-folded) resolves every window's rolling hash against its bucket's Bf16 + exact key map, incrementing hit counters. The sweep is the match for Body subsigs/patterns.

  • pub struct SlotCounts (:28) — per-slot hit counts + last-seen window offset.
    • pub fn get(&self, slot: SlotId) -> u32 (:34); pub fn last_offset(&self, slot: SlotId) -> Option<usize> (:40) — highest-offset resolved window (u32::MAX = never hit).
  • pub struct AtomFilterScanner<'a> (:46) — wraps &'a AtomFilterDb.
    • pub fn new(db: &'a AtomFilterDb) -> Self (:51).
    • pub fn scan(&self, data: &[u8]) -> SlotCounts (:58) — sweeps buckets (case-sensitive), then (only if any nocase bucket exists) sweeps buckets_nocase against a lowercased copy. Single fused pass over all active bucket lengths.
  • pub fn ext_matched(ext_slot: ExtSlot, slots: &[SlotDef], counts: &SlotCounts) -> bool (:151) — AutoMatch → true; Atom(id) → counts.get(id) >= threshold.
  • pub fn logical_initial_counts_into(out: &mut [usize], sub_slots: &[SubsigSlot], slots: &[SlotDef], counts: &SlotCounts) -> usize (:165) — fills out for LogicalExpr::eval: Body slot counter, 1 for AutoMatch, 0 for External. Returns elements written.
  • pub fn logical_initial_counts(sub_slots, slots, counts) -> Vec<usize> (:190) — allocating convenience wrapper.
  • pub fn subsig_anchor_offset(trigger: SubsigSlot, counts: &SlotCounts) -> usize (:199) — buffer offset to anchor a ByteCompare subsig's read at.

bytecode.rs (149 lines) — .cbc Stage-1 header/trigger parsing

Stage-1 parsing of .cbc ClamAV bytecode files: the ClamBC… header and the line-2 logical-signature trigger. Does not execute bytecode (that's bytecode_vm).

  • pub struct Bytecode (:26) — name: String, trigger: Option<String> (line 2), min_func_level: Option<u32>, source: String (full .cbc text).
  • pub struct BytecodeLoadReport (:39) — files_seen, loaded, skipped: usize.
  • pub struct BytecodeSet (:47) — bytecodes: Vec<Bytecode>, report: BytecodeLoadReport.
    • pub fn load_from_dir(dir: &Path) -> Self (:54) — scans for *.cbc.
    • pub fn from_bytes_map(files: &HashMap<String, Vec<u8>>) -> Self (:78).
  • pub fn parse_cbc(text: &str) -> Option<Bytecode> (:100) — None if not ClamBC…; parses header + line-2 trigger.

bytecode_vm.rs (2781 lines) — ClamBC VM: decoder + interpreter

The full ClamBC bytecode VM: Phase-1 faithful decoder of the nibble-armored .cbc program format (0x60|nibble), Phase-2a interpreter prep pass (stack-frame layout + operand remapping), Phase-2b interpreter (cli_vm_execute), Phase-3 API table, Phase-4 execution context. Detection signalled via setvirusname → ctx.virname.

Opcode constants (:19-69): pub const OP_BC_*: u16 for all 51 opcodes (ADD, SUB, MUL, UDIV, SDIV, UREM, SREM, SHL, LSHR, ASHR, AND, OR, XOR, TRUNC, SEXT, ZEXT, BRANCH, JMP, RET, RET_VOID, ICMP_EQ/NE/UGT/UGE/ULT/ULE/SGT/SGE/SLE/SLT, SELECT, CALL_DIRECT, CALL_API, COPY, GEP1, GEPZ, GEPN, STORE, LOAD, MEMSET, MEMCPY, MEMMOVE, MEMCMP, ISBIGENDIAN, ABORT, BSWAP16/32/64, PTRDIFF32, PTRTOINT64, INVALID). pub const BC_START_TID: u16 = 69 (:88). pub const ENGINE_FLEVEL: u32 = 240 (:208).

  • pub enum TypeKind (:102) — Function, PackedStruct, Struct, Array, Pointer.
  • pub struct BcType (:111) — kind, num_elements, contained: Vec<u16>, size, align.
  • pub enum Ops (:121) — None, Unary(u32), Binop([u32; 2]), Three([u32; 3]), Cast { source, mask: u64, size: u8 }, Branch { condition, br_true: u16, br_false: u16 }, Jump(u16), Call { funcid: u16, is_api: bool, ops: Vec<u32> }, GepN(Vec<u32>).
  • pub struct BcInst (:137) — opcode: u16, ty: u16, dest: u32, interp_op: u8, ops: Ops.
  • pub struct BcBB (:146) — insts: Vec<BcInst>.
  • pub struct BcFunc (:151) — num_args, num_locals, num_insts, num_values, num_constants, return_type: u16, types: Vec<u16> (bit 0x8000 = pointer local), bbs: Vec<BcBB>, constants: Vec<u64>, num_bytes: u32 (stack-frame size).
  • pub struct Bc (:179) — format_level, kind, min_func, max_func, num_types, num_func, start_tid, types, funcs, uses_apis, apis, globals, global_tys, lsig: Option<String>, hook_name: Option<String>, skipped: bool, num_global_bytes, global_bytes, prepared: bool.
    • pub fn prepare_interpreter(&mut self) -> DResult<()> (:1262) — mirrors cli_bytecode_prepare_interpreter: lays out globals, per-function stack frame, remaps operands from value index to byte offset. Sets prepared = true.
    • pub fn run(&self, ctx: &mut BcCtx) -> Result<i64, VmError> (:2523) — registers ctx globals (match counts → GID 1, kind → GID 2, filesize → GID 5, pedata → GID 4, globalBytes → GID 7), then vm.execute(0). Detection is ctx.virname.
  • pub enum DecodeError (:419) — Malformed(String), Skip(String).
  • pub fn decode_bytecode(text: &str) -> DResult<Option<Bc>> (:1098) — top-level decode driver over Lsig → Types → Apis → Globals → MdOptHeader → FuncHeader → Bb. Validates header magic, format level 6 or 7, type start id 69.
  • pub struct PeSection (:1540) — 9× u32 exposed to get_pe_section.
  • pub struct BcCtx<'a> (:1556) — file: &'a [u8], file_size: u32, off: u32, virname: Option<String>, lsigcnt: [u32; 64], lsigoff: [u32; 64], kind: u16, sections: Vec<PeSection>, nsections: u16, hdr_size: u32, ep: u32, max_ops: u64 (DoS bound, default 50,000,000).
    • pub fn new(file: &'a [u8]) -> Self (:1573).
  • pub enum VmError (:1626) — Bytecode(String), Timeout.

Implemented APIs (107-entry API_TABLE at :1607): read/write/seek/setvirusname/pe_rawaddr/file_find/file_byteat/malloc/get_pe_section/read_number/memstr/hex2ui/atoi/file_find_limit/engine_functionality_level/dconf_level/ilog2; debug/tracing APIs no-op; unimplemented APIs safe-default to 0/null.

database.rs (1335 lines) — signature loading pipeline

The non-hash signature loader and the Database container. Dispatches by file extension (mirroring readdb.c's cli_load), parses each per-line text format, and accounts for 100% of files seen (no silent drops). Two entry points: load_dir (filesystem) and from_bytes_map (AAssetManager). Also hosts the ClamAV→Rust regex sanitizer.

  • pub struct NameSpan (:14) — start: u32, len: u32 (interned name in Database::name_arena).
  • pub fn intern_name(arena: &mut String, name: &str) -> NameSpan (:20).
  • pub struct Database (:27) — extended: Vec<ExtendedSignature>, logical: Vec<LogicalSignature>, container: Vec<ContainerSignature>, file_type_magic: Vec<FileTypeMagic>, phishing: PhishingDb, unsupported: Vec<UnsupportedRecord>, bytecode_programs: Vec<Bc>, name_arena: String.
    • pub fn ext_name(&self, sig: &ExtendedSignature) -> &str (:46).
    • pub fn pattern_mem_stats(&self) -> crate::pattern::MemStats (:52).
    • pub fn load_dir(path: impl AsRef<Path>) -> io::Result<(Self, LoadReport)> (:238) — first pass collects .ign/.ign2 ignore names; second pass loads each file. Shrinks-to-fit the main vecs.
    • pub fn from_bytes_map(files: &HashMap<String, Vec<u8>>) -> (Self, LoadReport) (:257).
  • pub struct ExtendedSignature (:81) — name: NameSpan, target: Option<u32>, offset: OffsetSpec, patterns: Box<[Pattern]>, source.
  • pub struct ContainerSignature (:98) — name, container_type, container_size: NumSpec, has_filename: bool, size_in_container: NumSpec, size_real: NumSpec, encrypted: Option<bool>, file_pos: NumSpec, source. Filename/encrypted-constrained sigs are parsed but skipped at scan time.
  • pub enum ContainerType (:115) — Any, Format(&'static str) (zip/gz/xz/7z/tar), Unsupported.
  • pub enum NumSpec (:126) — Any, Exact(u64), Range(Option<u64>, Option<u64>). pub fn parse(raw: &str) -> Result<Self, String> (:855), pub fn matches(&self, value: u64) -> bool (:877), pub fn is_constrained(&self) -> bool (:887).
  • pub struct FileTypeMagic (:134) — offset: OffsetSpec, patterns: Box<[Pattern]>, clamav_type: Box<str>, source.
  • pub struct SourceLocation (:142) — path: Arc<Path>, line: usize.
  • pub struct UnsupportedRecord (:150) — source, reason: String.
  • pub struct LoadError (:156) — source, message: String.
  • pub struct LoadReport (:162) — many counters: files_seen, lines_seen, extended_loaded, db_loaded, logical_loaded, container_loaded, ftm_loaded, hash_files_skipped, unsupported_files, unsupported_records, bytecodes_loaded, ign_entries, ignored_skipped, tdb_attr_skipped, phishing_files/phishing_loaded, icon_files/icon_loaded, cert_files/cert_loaded, ioc_files, config_files, metadata_files, container_db_files, deprecated_files, unknown_files, by_extension: BTreeMap<String, usize>, errors: Vec<LoadError>.
  • pub struct OffsetSpec (:218) — anchor: OffsetAnchor, max_shift: Option<usize>.
    • pub fn any() -> Self (:275); pub fn parse(raw: &str) -> Self (:282) — * → Any; EOF-N → EofMinus; EP/SE/SL/S<n>/VI → various; decimal → Absolute.
    • pub fn scan_ranges(&self, data_len: usize) -> Vec<(usize, usize)> (:310).
  • pub enum OffsetAnchor (:224) — Any, Absolute(usize), EofMinus(usize), EntryPoint(i64), SectionStart { index, delta }, SectionEntire { index }, LastSectionStart { delta }, VersionInfo, MacroGroup(String), Unsupported(String).
  • pub(crate) fn sanitize_clamav_regex(pattern: &str) -> String (:604) — translates ClamAV regex dialect to Rust regex.

Extension dispatch (classify_extension, :1140): ndb/ndu/sdb → parse_extended_signature (name:target:offset:hex[:minFL[:maxFL]]); db → parse_db_signature (name=hex); ldb/ldu → logical::parse_logical_signature; cdb → parse_container_signature; ftm → parse_ftm; pdb/gdb → phishing.protected.add_line; wdb → phishing.allow.add_line; hdb/hsb/hdu/hsu/mdb/msb/mdu/msu/imp/fp/sfp → skipped (hash engine); idb/crb → counted; .cbc handled by the scanner's finish_engine_init.

pattern.rs (1836 lines) — compiled hex-pattern matcher

ClamAV-compatible hex-pattern representation and matcher. Patterns compile to u16 instructions plus a () special table for alternations/boundaries/gaps. Matching uses a recursive match_rec with SIMD literal anchoring (memchr/memmem) and fuel budgets. This is the engine that verifies atom-filter candidates.

Constants (:10): CLI_MATCH_CHAR = 0x0000, CLI_MATCH_NOCASE = 0x0100, CLI_MATCH_IGNORE = 0x0200, CLI_MATCH_NIBBLE_HIGH = 0x0300, CLI_MATCH_NIBBLE_LOW = 0x0400, CLI_MATCH_SPECIAL = 0x0700.

  • pub struct Modifiers (:90) — nocase, wide, ascii, fullword. pub fn parse(raw: &str) -> Result<Self, String> (:98).
  • pub enum Special (:117) — AltChar { bytes, negative } ((2e|2f|40)), AltStrFixed { strs, len, negative } ((dead|beef)), AltStr { branches, min, negative } ((aa|bbbb)), Boundary ((B)/(L)/(W), zero-width), Gap { min, max } (*/{n-m}).
  • pub const INLINE_CAP: usize = 22 (:236).
  • pub enum Instructions (:239) — Inline { buf: [u8; INLINE_CAP], len: u8 } (zero-alloc short literals), Pure(Box<[u8]>), Complex(Box<[u16]>). pub fn pure(bytes: Vec<u8>) -> Self (:255), pub fn len(&self) -> usize (:266), pub fn get_u16(&self, index) -> u16 (:280).
  • pub struct Pattern (:292) — instructions, specials: Box<[Special]>, best_literal_offset: u16, best_literal_len: u16, fullword.
    • pub fn best_literal(&self) -> Option<(usize, usize)> (:317) — the most-selective fixed-byte run.
    • pub fn from_instructions(inst: Vec<u16>, fullword: bool) -> Self (:326); pub fn from_parsed(inst, specials, fullword) -> Self (:332).
    • pub fn parse_hex(hex: &str) -> Result<Vec<u16>, String> (:450).
    • pub fn is_match(&self, data: &[u8]) -> bool (:455).
    • pub fn max_match_len(&self) -> Option<usize> (:483) — None = unbounded.
    • pub fn find_all(&self, data: &[u8], ranges: &[(usize, usize)], limit: usize) -> Vec<MatchRange> (:573) — main scan. Fast path: anchor on best_anchor() via SIMD, verify with match_rec. Else find_any_fixed_byte/find_all_fallback_bruteforce.
    • pub fn find_all_at(&self, data, ranges, limit, hints: &[u32]) -> Vec<MatchRange> (:908) — prefilter-hinted scan.
    • pub fn match_at(&self, data: &[u8], start: usize) -> Option<usize> (:1035) — bounded by MATCH_FUEL_BUDGET = 2000.
    • pub fn mem_stats(&self) -> MemStats (:1257).
    • pub fn required_atom(&self) -> Option<Vec<u8>> (:1282) — the best literal's bytes (prefilter atom).
    • pub fn required_atom_nocase(&self) -> Option<Vec<u8>> (:1291) — longest CHAR|NOCASE run, lowercased.
  • pub struct MemStats (:1356); pub struct MatchRange (:1392) — start: usize, end: usize.
  • pub fn compile_pattern_variants(raw: &str, modifiers: Modifiers) -> Result<Vec<Pattern>, String> (:1398) — emits the ascii variant and the wide variant (if wide).

logical.rs (1487 lines) — .ldb logical signatures + expression engine

Parses .ldb/.ldu logical signatures (name;TDB;expression;subsigs…) and evaluates the boolean/count expression tree over subsignature match counts. Handles Body/Pcre/ByteCompare/Fuzzy/Unsupported subsignatures, the TDB constraint block, PCRE trigger/regex/flags, byte-compare specs, and the lazy-regex memory strategy.

  • pub struct LazyRegex (:23) — source: Arc<str>, compiled: Arc<OnceLock<Option<Regex>>>. PCRE subsig regex compiled on first trigger (dominant memory saving: ~170 MB → few MB resident).
    • pub fn get(&self) -> Option<&Regex> (:37); pub fn is_match(&self, data: &[u8]) -> bool (:43).
  • pub struct LogicalSignature (:49) — name, target, file_size, container, nos, ep, icongrp1, icongrp2, handlertype, intermediates, tdb_unsupported, expression: LogicalExpr, subsignatures: Vec<Subsignature>, source, bytecode: Option<usize>.
  • pub struct PcreSubsig (:92) — trigger: LogicalExpr, regex: LazyRegex, global: bool.
  • pub enum Subsignature (:99) — Body { offset: Option<Box<OffsetSpec>>, patterns: Box<[Pattern]> }, Pcre(Box<PcreSubsig>), ByteCompare(Box<ByteCompareSpec>), Fuzzy([u8; 8]), Unsupported(Box<str>).
  • pub enum ByteReadType (:119) — HexAscii, DecimalAscii, Auto, BinaryLe, BinaryBe.
  • pub struct ByteCompareSpec (:133) — trigger_subsig: usize, offset_sign: i64, offset_value: usize, read_type, exact: bool, num_bytes: usize, comparisons: Vec<(CompareOp, u64)>.
    • pub fn evaluate(&self, data: &[u8], trigger_offset: usize) -> bool (:996).
  • pub struct LogicalExpr (:164) — flat post-order array of ExprNode (Subsig/And/Or/Compare); root = last node.
    • pub fn eval(&self, counts: &[usize]) -> EvalStats (:468) — ClamAV cli_ac_chklsig.
    • pub fn can_still_match(&self, counts: &[usize], evaluated: &[bool]) -> bool (:486) — over-approximate feasibility.
    • pub fn has_nonmonotone_compare(&self) -> bool (:506).
    • pub fn is_definitely_matched(&self, counts: &[usize], evaluated: &[bool]) -> bool (:526).
  • pub enum CompareOp (:227) — Equal, Greater, GreaterEqual, Less, LessEqual.
  • pub struct EvalStats (:236) — matched: bool, hits: usize, ids: u64.
  • pub fn parse_logical_signature(line: &str, source: SourceLocation) -> Result<(LogicalSignature, Vec<String>), String> (:284) — PCRE-aware ldb_tokenize; ExprParser recursive-descent; parse_tdb; parse_subsignature. Returns Err("unrecognised TDB attribute …") for unknown TDB keys.

scanner.rs (1792 lines) — scan pipeline + Engine

The scan entry point and the engine that ties everything together: file-type detection, the atom-filter sweep, extended/logical signature evaluation, PCRE/byte-compare/fuzzy/image-icon subsigs, bytecode execution, YARA delegation, and phishing.

  • pub struct TimingBreakdown (:10) — clamav_ns: u128, yara_per_engine: Vec<(String, u128)>. pub fn accumulate(&mut self, other: TimingBreakdown) (:17).
  • pub struct Engine (:24) — database: Database, private atomfilter_db, pub yara: Vec<YaraEngine>.
  • pub struct ScanOptions (:39) — scan_archives: bool, max_recursion: usize (default 16), max_child_size: usize (default 650 MiB). impl Default.
  • pub struct ScanMatch (:56) — name: String, kind: SignatureKind, source: SourceLocation, object_path: String, view: ScanView.
  • pub enum SignatureKind (:102) — Extended, Logical, Container, Phishing, Yara.
  • pub enum ScanView (:113) — Raw (only variant).

Engine public API:

  • pub fn prefilter_mem_report(&self) -> String (:191).
  • pub fn from_database_dir(path: impl AsRef<Path>) -> io::Result<(Self, LoadReport)> (:202).
  • pub fn from_bytes_map(files: &HashMap<String, Vec<u8>>) -> (Self, LoadReport) (:216).
  • pub fn load_yara_rules(&mut self, path: impl AsRef<Path>) -> Option<()> (:279).
  • pub fn add_yara_source_file(&mut self, path: impl AsRef<Path>) -> Option<()> (:286).
  • pub fn add_compiled_yara_file(&mut self, path: impl AsRef<Path>) -> Option<()> (:294) — add a .yrc.
  • pub fn add_compiled_yara(&mut self, engine: YaraEngine) (:304).
  • pub fn scan_path(&self, path: impl AsRef<Path>, options: ScanOptions) -> io::Result<Vec<ScanMatch>> (:308).
  • pub fn scan_bytes(&self, data: &[u8], options: ScanOptions) -> Vec<ScanMatch> (:318).
  • pub fn scan_bytes_named(&self, data, object_path, options, module_meta: &[(&str, &[u8])]) -> Vec<ScanMatch> (:322) — the main entry.
  • pub fn scan_bytes_named_with_breakdown(&self, data, object_path, options, module_meta) -> (Vec<ScanMatch>, TimingBreakdown) (:338).
  • pub fn detect_target(&self, data: &[u8]) -> Option<u32> (:471).

Scan pipeline (scan_object, :353): (1) size guard; (2) type detection (.ftm + builtin magic: \x7fELF→6, %PDF→10, GIF/PNG/JPEG→5, dex\n→16, PK\x03\x04→17); (3) archive guard (skip RAR); (4) ClamAV engine gate — CLAMAV_ALLOWED_TARGETS = [3,5,6,7,10,16,17] (HTML/Graphics/ELF/text/PDF/DEX/ZIP-APK); (5) scan_context — one AtomFilterScanner::scan sweep → SlotCounts, then scan_extended + scan_logical; (6) YARA per ruleset (Android targets only, module metadata forwarded); (7) phishing for HTML with a protected DB. No ZIP extraction here — the caller (hydradragonandroid's collect_buffers) walks archives via hydradragonextractor.

Extended scan (scan_extended/scan_one_extended, :521/:537): iterates ext_slot; if atomscan::ext_matched, checks target_matches, rejects unsupported offset anchors, runs pattern.find_all across all patterns.

Logical scan (scan_logical/scan_one_logical, :583/:644): per sig — TDB gating; initial counts from atomscan::logical_initial_counts_into; byte-confirmation of Body subsigs with the real Pattern (without this, atom shadows like mira for mirai would fire spuriously); can_still_match short-circuit; Fuzzy via ctx.image_fuzzy_hash()→fuzzy::calculate_image; Phase-2 Pcre (lazy LazyRegex::get, capped to first PCRE_MAX_SCAN_BYTES = 10_000_000 bytes) + ByteCompare (ByteCompareSpec::evaluate anchored at trigger's last_offset); LogicalExpr::eval; on match: handlertype suppresses, bytecode runs run_bytecode, else emit ScanMatch.

Bytecode (run_bytecode, :925): builds BcCtx::new(ctx.data), copies up to 64 trigger counts into lsigcnt, calls bc.run; detection is bctx.virname.

yara_scan.rs (151 lines) — YARA-x bridge

Bridges to the yara-x fork. Compiles YARA source or loads pre-compiled .yrc rulesets, caches one yara_x::Scanner per engine per thread (amortizing Scanner::new's setup + WASM runtime instantiation), and scans Android-relevant file types only.

  • pub fn is_target_allowed(target: Option<u32>) -> bool (:38) — true iff target in ALLOWED_TARGETS = [3, 5, 6, 7, 10, 16, 17]. Excludes PE/OLE2/Mail/Mach-O/SWF/Java.
  • pub struct YaraEngine (:47) — id: u64, pub name: String, rules: Box<yara_x::Rules> (boxed so its address is stable across Vec::push reallocations, since thread-local Scanners hold &'static Rules).
    • pub fn from_source_file(path: impl AsRef<Path>) -> Option<Self> (:62).
    • pub fn from_source(source: &str, name: String) -> Option<Self> (:69).
    • pub fn from_compiled(bytes: &[u8], name: String) -> Option<Self> (:80) — Rules::deserialize(bytes) (the fast Android asset path).
    • pub fn from_compiled_file(path: impl AsRef<Path>) -> Option<Self> (:86).
    • pub fn scan(&self, data: &[u8], object_path: &str, module_meta: &[(&str, &[u8])]) -> Vec<ScanMatch> (:93) — reuses a thread-local Scanner keyed by self.id; scanner.fast_scan(true); with module metadata builds ScanOptions and calls scan_with_options (feeding per-module JSON reports); each matching rule becomes ScanMatch named YARA-X.<identifier>.

phishing.rs (677 lines) + phishing/tld.rs (78 lines) — phishing detection

tld.rs: verbatim port of ClamAV's iana_tld.h/iana_cctld.h gperf wordlists, sorted for binary search.

  • pub fn is_tld(s: &str) -> bool (tld.rs:8); pub fn is_cctld(s: &str) -> bool (tld.rs:13).

phishing.rs: HTML phishing detection (protected/allow domain DBs loaded from .pdb/.gdb/.wdb). The protected/allow line parsers and scan_html/get_domain logic mirror ClamAV's phishcheck.c.

main.rs (423 lines) — CLI driver

A hydradragonclamav CLI binary: --database/-d, --scan/-s, --no-archives, --max-recursion, --max-child-size (K/M/G suffixes), --list-unsupported, HDC_MEM_STATS=1 memory profiler, HDC_HOLD steady-state sampler. Loads the DB, prints a full coverage report, trims the working set (K32EmptyWorkingSet on Windows), scans files (exit 1 on detection, 2 on error). Not shipped in the APK — host-side testing tool.

hydradragonml — Burn classifier

A pure-Rust neural-network malware classifier (Burn 0.21 on the ndarray backend) that scores an APK 0.0–1.0 from its tokenized strings plus 18 content-derived Android engine features. No ONNX, no external runtime: weights ship as a Burn .mpk recorder file and inference runs fully in-process. Crate root: hydradragonml/src/.

The model is a Burn neural network, not "ONNX/tract" or "MinHash/LSH + Isolation Forest" as some older docs said. FNV-1a token hashing still exists in this crate, but only for the benign-content DB lookup (features::extract_minhash), not for classification.

src/lib.rs

pub mod features;
pub mod model;

pub use features::axml;
pub use features::dex;
pub use features::elf;

pub const DEFAULT_CONFIDENCE_THRESHOLD: f32 = 0.95;
pub const SUSPICIOUS_THRESHOLD: f32 = 0.90;

pub struct Model {
    classifier: model::ApkClassifier<B>,  // B = NdArray<f32>
    tokenizer: features::Tokenizer,
    confidence_threshold: f32,
    device: NdArrayDevice,
}

pub struct ScanResult {
    pub malicious: bool,    // confidence >= confidence_threshold
    pub suspicious: bool,   // !malicious && confidence >= SUSPICIOUS_THRESHOLD
    pub confidence: f32,    // clamped 0.0..1.0
}

impl Model:

  • pub fn load(model_bytes, device) -> Result<Self, ...> (lib.rs:30) — writes model_bytes to a temp .mpk file and calls ApkClassifier::load_weights. Tokenizer uses stateless FNV-1a Hashing Trick — no vocab.json needed. Threshold starts at DEFAULT_CONFIDENCE_THRESHOLD (0.95).
  • pub fn load_from_path(model_path, device) (lib.rs:51) — same but reads the .mpk directly from a file path.
  • pub fn set_threshold(&mut self, t: f32) (lib.rs:67) — self.confidence_threshold = t.clamp(0.0, 1.0).
  • pub fn scan(&self, apk: &[u8]) -> Option<ScanResult> (lib.rs:71) — extracts EngineFeatures from apk via features::EngineFeatures::extract_from_apk and calls scan_with_features.
  • pub fn scan_with_features(&self, apk, engine_feats) -> Option<ScanResult> (lib.rs:83) — tokenizes the APK via tokenizer.tokenize(apk), builds the [18] engine-feature tensor, runs classifier.forward, and maps the sigmoid output scalar to malicious/suspicious/confidence. None if the APK isn't a readable ZIP or yields no tokens.

src/model.rs — the network

ApkClassifier<B> (model.rs:10) — a neural network module built on Burn 0.21:

Layer Shape → Activation
embedding [VOCAB_SIZE=20000, EMBED_DIM=64], then mean-pool over the token dim —
linear1 (64 + 18) → 64 (pooled embedding ⊕ 18 EngineFeatures concat) Relu
linear2 64 → 32 Relu
output 32 → 1 Sigmoid

forward_batch (model.rs:36) is the batched path used by training; forward (model.rs:54) is the single-sample wrapper. Weights are saved/loaded via Burn's CompactRecorder — save_weights(path) (:64) and load_weights(path, device) (:71).

src/features/mod.rs & features/features.rs — tokenizer + engine features + ripzip

Constants (features/features.rs:10-19): VOCAB_SIZE = 20000, EMBED_DIM = 64, MIN_STR_LEN = 5, MAX_TOKENS = 4096, MAX_ENTRY_SCAN = 16 MiB, ENGINE_FEATURE_COUNT = 18.

  • Tokenizer (features.rs:199): stateless FNV-1a Hashing Trick tokenizer. hash_token(key) (:208) computes fnv1a_64(key.to_lowercase()) % (VOCAB_SIZE-1) + 1, guaranteeing every token maps to 1..(VOCAB_SIZE-1) — ID 0 is reserved for padding. No vocab.json is loaded or required. tokenize(apk) (:213) opens the APK using ripzip; sub_tokenize (:234) splits entry names on delimiters (. / ; : - \ _), then harvest_strings (:249) pulls printable ASCII and UTF-16LE runs from AndroidManifest.xml, resources.arsc, *.dex, and META-INF/* entries. None if no content entry or empty token list.
  • EngineFeatures (features.rs:22): 18 floats — 5 DEX (dex_class_count, dex_string_count, dex_api_call_count, dex_finding_high, dex_finding_critical), 6 ELF (elf_count, elf_emulated_strings, elf_network_calls, elf_file_calls, elf_exec_calls, elf_anti_debug), 7 AXML Manifest (manifest_dangerous_permissions, manifest_total_permissions, manifest_activities, manifest_services, manifest_receivers, manifest_min_sdk, manifest_target_sdk). to_vec() (:47) normalizes each value to [0,1]. extract_from_apk(apk) (:73) uses ripzip to extract all features directly from raw APK bytes.
  • extract_minhash(apk) (features.rs:334) + fnv1a (:312) + token(prefix, s) (:321): FNV-1a hashed string tokens (namespaced name:/dex:/manifest:/res:/perm:/api:/url:) used by the benign-content DB (benign_db.rs), not by the classifier.

src/bin/train.rs — hydradragonml-train binary

Trains ApkClassifier (autodiff Autodiff<NdArray<f32>>, Adam, binary cross-entropy) over --benign <dir> / --malware <dir> corpora and writes weights to --output model.mpk:

cargo run --release --bin hydradragonml-train -- \
    --benign ./benign --malware ./malware \
    --output model.mpk [--epochs 6] [--lr 0.001] [--batch-size 8]

No --vocab argument — the Hashing Trick tokenizer is stateless and requires no pre-built vocabulary. Samples are tokenized via Tokenizer and EngineFeatures::extract_from_apk; the corpus is split 80/20 train/valid and the learning rate halves each epoch.

src/main.rs — hydradragonml-scan CLI binary

CLI scanner tool for APK files or directories:

cargo run --release --bin hydradragonml-scan -- \
    [--model model.mpk] [--threshold 0.95] [--json] <target_file_or_dir>

Parses CLI arguments, loads Model::load_from_path, walks directories for .apk/.zip files, runs model.scan(&bytes), and prints human-readable or JSON verdicts.

On-device consumption

hydradragonandroid/src/lib.rs uses hydradragonml::Model. The model ships as a Burn .mpk recorder loaded via asset name MODEL_MPK: &str = "model.mpk". No vocab.json is needed — the Hashing Trick tokenizer is stateless. The model is loaded once at init into Engine.model: Option<Model> via Model::load(model_bytes, NdArrayDevice::default()) and scored via scan_with_features(data, engine_features) on each non-whitelisted APK buffer during run_scan.

hydradragonextractor — archive extraction

Detects and extracts Android-relevant archive formats (ZIP/APK, TAR, GZIP, XZ, 7z, RAR, bzip2, LZMA, and more) either to disk or fully in memory, with parallel ZIP decompression, decompression-bomb detection, per-archive entry caps, and path-traversal protection. Crate root: hydradragonextractor/src/. Dependencies: oxiarc-archive v0.3 (multi-format backend), unrar-ng (from the HydraDragonAntivirus/unrar.rs fork), thiserror v2.

src/lib.rs — public API

Error type:

#[derive(Debug, thiserror::Error)]
pub enum ExtractError {                        // lib.rs:27
    Io(#[from] std::io::Error),
    OperationFailed { reason: String },
    DecompressionBomb { format: &'static str },
}

Public structs:

  • pub struct ExtractResult { pub files: Vec<PathBuf>, pub output_dir: PathBuf } (:22).
  • pub struct ZipEntryInfo { pub name: String, pub size: u64, pub compressed_size: u64 } (:118).
  • pub struct EntryInfo { pub name: String, pub size: u64 } (:159).

Top-level functions:

  • pub fn detect_format(data: &[u8]) -> Option<&'static str> (:45) — returns one of "rar", "zip", "gz", "xz", "tar", "7z", "bz2", "zst", "lz4", "br", "snappy", "cab", "lzh", "iso", "lzma", or None. RAR detected first by magic (Rar!\x1a\x07\x00 v1.5 / Rar!\x1a\x07\x01\x00 v5); else oxiarc_archive::detect::ArchiveFormat::from_magic, with ustar TAR at offset 257 (is_tar :75) and raw LZMA ([0x5d, 0x00], is_lzma :80) fallbacks.
  • pub fn extract_archive(path: &Path, output_dir: &Path) -> Result<ExtractResult> (:92) — reads the file, dispatches RAR to rar::extract_to_dir and everything else to extract_to_dir. Creates output_dir. Returns the extracted file paths.
  • pub fn extract_archive_from_bytes(data: &[u8]) -> Result<Vec<(String, Vec<u8>)>> (:111) — the primary in-memory API used on-device. Returns (in_archive_name, decompressed_bytes) pairs so detections report against the real member. RAR → rar::extract_from_bytes; else extract_to_memory.
  • pub fn zip_list_entries(data: &[u8]) -> Result<Vec<ZipEntryInfo>> (:127) — lists ZIP entries (skipping directory entries), capped at MAX_ARCHIVE_ENTRIES (4096).
  • pub fn zip_extract_entry(data: &[u8], name: &str) -> Result<Vec<u8>> (:145).
  • pub fn list_entries(data: &[u8]) -> Result<Vec<EntryInfo>> (:166) — format-agnostic. Single-stream formats (gz/xz/bz2/zst/lz4/br/snappy/lzma) return a single EntryInfo { name: "decompressed", size: 0 }.
  • pub fn extract_entry(data: &[u8], name: &str) -> Result<Vec<u8>> (:206) — format-agnostic. zstd/lz4/br/snappy return OperationFailed (not implemented).

Decompression-bomb guard (public):

  • pub(crate) const MAX_DECOMPRESSED_SIZE: usize = 200_000_000 (:371); const BOMB_RATIO: usize = 1000 (:372); const MIN_RATIO_CHECK_SIZE: usize = 10_000_000 (:373); static DETECT_BOMBS: AtomicBool = AtomicBool::new(true) (:378).
  • pub fn set_bomb_detection_enabled(enabled: bool) (:380).
  • pub(crate) fn is_decompression_bomb(compressed_len, decompressed_len) -> bool (:384) — triggers if decompressed_len > 200 MB, OR decompressed_len >= 10 MB && decompressed_len / compressed_len > 1000.
  • pub fn is_bomb_error(e: &ExtractError) -> bool (:401) — matches DecompressionBomb.

safe_output_path(output_dir, name) (:723) is the path-traversal guard — only Normal and CurDir components are accepted; any ParentDir/RootDir/Prefix yields None and the entry is skipped.

Parallel ZIP extraction (zip_to_memory, :510)

  • pub(crate) const MAX_ARCHIVE_ENTRIES: usize = 4096 (:508).
  • Splits entry names into chunks: n_threads = available_parallelism().min(4), chunk_size = ceil(names.len() / n_threads). std::thread::scope spawns one thread per chunk; each builds its own ZipReader from the shared data slice and extracts its chunk. A shared AtomicBool bomb_found records any per-entry bomb; results flow through an mpsc::channel. After join: if bomb_found, returns DecompressionBomb { format: "zip" }. ~4× wall-clock speedup on a 1000-entry APK on a 4-core device.

src/rar.rs — RAR support (236 lines, private module)

Uses unrar::Archive. Because unrar requires a file path, the in-memory RAR helpers spill to a temp file under std::env::temp_dir().join(format!("hdrartmp_{:x}", crate::rand_byte())) and clean it up on every exit path.

  • pub fn extract_to_dir(path: &Path, output_dir: &Path) -> Result<Vec<PathBuf>> (:6).
  • pub fn extract_from_bytes(data: &[u8]) -> Result<Vec<(String, Vec<u8>)>> (:46) — writes to archive.rar in a temp dir, loops read_header() (stopping at MAX_ARCHIVE_ENTRIES), header.read() → (data, rest), bomb check, pushes (entry_name, data).
  • pub fn list_entries(data: &[u8]) -> crate::Result<Vec<crate::EntryInfo>> (:118).
  • pub fn extract_entry(data: &[u8], name: &str) -> crate::Result<Vec<u8>> (:172).

Supported formats

Format detect in-memory extract disk extract single-entry bomb check
ZIP / APK "zip" zip_to_memory (parallel) zip_to_dir zip_extract_entry per-entry + aggregate
TAR "tar" tar_to_memory tar_to_dir tar_extract_entry —
GZIP "gz" gzip_to_memory (auto-tar) gzip_to_dir decompress_gzip yes
XZ "xz" xz_to_memory (auto-tar) xz_to_dir decompress_xz yes
bzip2 "bz2" bzip2_to_memory (auto-tar) bzip2_to_dir decompress_bzip2 yes
7z "7z" sz_to_memory sz_to_dir sz_extract_entry per-entry
RAR v1.5 / v5 "rar" rar::extract_from_bytes rar::extract_to_dir rar::extract_entry per-entry
LZMA (raw) "lzma" xz_to_memory xz_to_dir decompress_xz yes
zstd/lz4/br/snappy detected listing-only stub not supported not implemented —
CAB / LZH / ISO detected not supported not supported — —

Recursion limits

The crate itself does not recurse — it extracts one archive level. Recursion is driven by the on-device caller: hydradragonandroid/src/lib.rs guards if item.depth < 16 && fmt.is_some() before calling extract_archive_from_bytes, pushing children with depth: item.depth + 1. So nested-archive recursion is capped at 16 levels on-device. Additionally the on-device worker caps extraction at 4096 buffers total or 2 GB cumulative bytes, independent of the crate's own MAX_ARCHIVE_ENTRIES = 4096 per single archive.

On-device consumption

collect_buffers in hydradragonandroid/src/lib.rs calls extract_archive_from_bytes per work item. On Err(e) if is_bomb_error(&e) it pushes a ("HDR.Bomb.Decompression", obj_path, lineage) detection rather than treating it as an extraction failure. The user toggle is wired through nativeSetDetectZipBomb → set_bomb_detection_enabled. Extracted (name, bytes) pairs become Bufs scanned by ClamAV/YARA/ML.

hydradragonxorfilter — Binary-Fuse filters

The single source of truth for the .xf on-disk container. Both the offline builder (dev-tools/xorfilter_writer) and the on-device native scanner depend on this crate, so a filter written on x86 is queryable byte-for-byte on arm64. Only BinaryFuse16 (Bf16) is used — ~2.16 bytes/key, the project-wide standard for URL/domain/IP blocklists and the MD5 whitelist. Crate root: hydradragonxorfilter/src/. Dependencies: jdb_xorf v0.13 (with bitcode feature — Bf8/Bf16/Bf32), bitcode v0.6, memmap2 v0.9.

src/lib.rs — public API

const TAG_BF16: u8 = 16;                       // lib.rs:25 — first byte of every .xf

pub fn key(s: &str) -> u64                     // lib.rs:35
pub fn key_bytes(b: &[u8]) -> u64              // lib.rs:49

pub struct XorFilter(Bf16);                    // lib.rs:61

impl XorFilter {
    pub fn contains_key(&self, k: u64) -> bool  // lib.rs:65
    pub fn contains(&self, s: &str) -> bool     // lib.rs:70
    pub fn from_bytes(bytes: &[u8]) -> Option<XorFilter>  // lib.rs:76
    pub fn load(path: &Path) -> Option<XorFilter>        // lib.rs:88
}

pub fn build_from_keys(mut keys: Vec<u64>) -> Result<Vec<u8>, String>  // lib.rs:104

Key derivation:

  • key(s: &str) -> u64 (:35) — FNV-1a 64-bit (offset 0xcbf29ce484222325, prime 0x100000001b3) over the ASCII-lowercased bytes. Used for case-insensitive items — hostnames, URLs, hex digests.
  • key_bytes(b: &[u8]) -> u64 (:49) — same FNV-1a over raw bytes with no lowercasing — for case-sensitive keys (binary signature atoms).

On-disk .xf format: [1 byte tag = 16 (TAG_BF16)] ++ bitcode::encode(&Bf16). Byte 0 is always 0x10; the remainder is the bitcode-serialized jdb_xorf::Bf16 body. bitcode is deterministic and platform-independent, so host-encoded bytes decode identically on arm64.

Query:

  • contains_key(k: u64) (:65) — self.0.has(&k).
  • contains(s: &str) (:70) — self.contains_key(key(s)).

Decode:

  • from_bytes(bytes: &[u8]) -> Option<XorFilter> (:76) — split_first() reads the tag; None on a bad tag (!= 16) or malformed body.
  • load(path: &Path) -> Option<XorFilter> (:88) — File::open, unsafe { memmap2::Mmap::map(&file) }, then catch_unwind(|| XorFilter::from_bytes(&mmap)). The returned filter owns its data (decoded onto the native heap); the mapping is dropped on return.

Build:

  • build_from_keys(mut keys: Vec<u64>) -> Result<Vec<u8>, String> (:104) — sort_unstable + dedup (binary-fuse construction requires distinct keys), rejects empty input, bitcode::encode(&Bf16::from(&keys)) inside catch_unwind, prepends TAG_BF16.

Writer/reader byte-compatibility: guaranteed by construction — both xorfilter_writer and hydradragonandroid call key/key_bytes, build_from_keys, and XorFilter::from_bytes/contains from this one crate. No format constants are duplicated.

On-device consumption

  • hydradragonandroid/src/lib.rs:23 use hydradragonxorfilter::XorFilter;. WHITELIST_XF = "whitelist.xf" (:125) is the NSRL MD5 whitelist, loaded once into Engine.whitelist: Option<XorFilter>.
  • url_scan.rs:21 filter: XorFilter per category; load_from_assets (:43) calls XorFilter::from_bytes for each <stem>.xf; scan calls f.filter.contains(norm). Stems: malwareurl, phishingurl, phishing, malicious, malicious_mail, abuse, spam, mining.
  • ip_scan.rs:4 use XorFilter;; stems ipmalware, ipphishing, ipbruteforce, ipddos, ipspam; f.filter.contains(ip).

The .xf files live in app/src/main/assets/scan/ and are memory-mapped at init. See NSRL-Whitelisting for the whitelist layering and Malicious-URL-Scanning for the URL/domain scan path.

Dev-tools (offline build tools)

The dev-tools build the assets the on-device crates consume. See Data-Pipeline for the orchestration scripts; this section covers the Rust dev-tools.

dev-tools/hydradragon_yara_x_compile — offline .yrc compiler

Same yara-x features as the on-device crate. See YARA-X-Modules#the-yar--to-yrc-build-chain for the full build chain and filtering logic. CLI: hydradragon_yara_x_compile [--check] [--filtered] <source_dir> [<output_dir>]. Output defaults to app/src/main/assets/scan.

dev-tools/xorfilter_writer — offline .xf blob builder

Path-depends on hydradragonxorfilter so it uses the exact same key derivation, width, and on-disk format as the device. Three modes: build (xorfilter_writer <in.txt> <out.xf>), check (--check <file.xf> <item>), false-positive-rate (--fp <file.xf> <count> <hex|dom>, fixed seed 0x9E3779B97F4A7C15 → reproducible xorshift64* PRNG). The build_xfilters.sh/build_xfilters.cmd orchestration invokes this writer for each category stem into app/src/main/assets/scan/. Stems must match the CATS tables in url_scan.rs/ip_scan.rs.

dev-tools/yarGen — vendored YARA rule generator

A vendored copy of Florian Roth's yarGen v0.24.0 — creates YARA rules from strings found in malware files while removing strings that also appear in goodware. Python, build-time only; its .yar output is an input to hydradragon_yara_x_compile. See Data-Pipeline#yara-rule-generation-yargen for the project's actual usage. Companion: convert_cuckoo_to_hydradragon.py rewrites import "cuckoo" → import "hydradragon" (whole-token regex) before compilation; truncateruledesc64kb.py truncates rule descriptions to 7 KB.

Offline → on-device invariants

Dev-tool (offline, x86) Output artifact On-device consumer (arm64) Load mechanism
xorfilter_writer <stem>.xf, whitelist.xf hydradragonandroid (url_scan.rs, ip_scan.rs, lib.rs) XorFilter::from_bytes / XorFilter::load (shared hydradragonxorfilter)
hydradragon_yara_x_compile <stem>.yrc hydradragonclamav → hydradragonandroid yara_x::Rules::deserialize (YaraEngine::from_compiled)
yarGen (vendored) *.yar source rules input to hydradragon_yara_x_compile —
hydradragonml-train (cargo run --release --bin hydradragonml-train -- --benign <dir> --malware <dir> --output model.mpk) model.mpk (Burn .mpk recorder) hydradragonml::Model::load Burn NamedMpkFileRecorder / ApkClassifier::load_weights. No vocab.json — Hashing Trick tokenizer is stateless.

Two shared invariants make the offline→on-device flow safe: (1) hydradragonxorfilter is the single source of truth for .xf format and key derivation; (2) the yara-x fork with pulley is the single source of truth for .yrc bytecode — the compiler and the on-device hydradragonclamav/hydradragonandroid must use the same feature set (serialized module is backend-specific).

Clone this wiki locally