FuzzyHash Scanner

FuzzyHash Scanner

Block-level similarity matching against a curated database of known offensive tools. Detects partial code reuse, modified variants, and shared techniques even when whole-file hashes don't match. Powered by ssdeep.

It's a doppelganger module, like Blender — surfaces results on /doppelganger?type=fuzzy and never feeds the Detection Score Explained number directly. Operators use it as an attribution / variant-detection signal.

---

What it tells you

• Code reuse — which known tool's binary your payload borrows from.
• Variants — modified versions of an existing tool you've already cataloged.
• Technique fingerprints — shellcode / loader patterns shared across families.
• Source attribution — when your DB groups by Git origin, the matched repo is the attribution.

---

Database — create_db_from_folder(folder, extensions)

Builds the offensive-tool baseline once, then reads it on every comparison.

# CLI
grumpycat.py doppelganger-db --folder C:\Tools\Offensive --extensions .exe .dll .bin

Or via the page (/doppelganger?type=fuzzy → "Create database").

Steps:

1. File discovery — recursive walk of folder, filtered by extensions.
2. Source detection — find_git_root() walks up from each file looking for .git/config, parses the remote URL, and tags every file under that repo with the repo's HTTPS URL. Manual SSH→HTTPS rewrite for consistency. Files outside any git tree get a generic local tag.
3. Block split — _create_blocks() chunks each file into 4 KiB blocks; computes an ssdeep fuzzy hash per block.
4. Compress + persist — entire DB serialized as JSON with shortened keys, then zlib-compressed to Utils/DoppelgangerDB/FuzzyHash/FuzzyHash.db.

Block data is stored alongside the hashes so the result page can show the actual matched bytes (hex / ASCII) — not just hash equivalence.

---

Comparison — analyze_files([target], threshold)

Triggered by GET /doppelganger?type=fuzzy&hash=<md5> or the doppelganger-analyze <hash> --type fuzzy --threshold N CLI command.

Steps:

1. Block target — same 4 KiB block split as DB creation.
2. Per-block compare — _compare_blocks() walks the DB and runs ssdeep similarity on every (target_block × db_block) pair.
3. Region detection — consecutive matching blocks at adjacent file offsets are merged into a single region. A 32 KiB matching region is more meaningful than 8 scattered 4 KiB matches.
4. Rank — top 3 source files per target, ordered by aggregate region similarity.

threshold (0–100) is the ssdeep similarity cutoff — lower threshold = noisier results, higher = stricter. The CLI defaults to 1 (catch anything); the MCP / page UI defaults to 85 (strong matches only).

---

Why block-level

Whole-file ssdeep is flat — a tool with 100 KiB of unique-to-it code embedded in a 5 MiB host won't match the original tool's hash at all because the host bytes dominate. Block-level matching keeps the small-but-meaningful match visible.

The trade-off is DB size: storing every block's hash + raw bytes is much larger than one hash per file. zlib compression on the entire DB keeps the working file in the tens of MB even with thousands of source files. Decompression is once per query (entire DB lifted into memory, compared, dropped).

---

Data structures

FileMetadata     = {path, md5, size, blocks: [BlockMetadata], date_added, source}
BlockMetadata    = {index, hash, offset_start, offset_end, data (compressed)}
MatchingRegion   = {source_offsets, target_offsets, length, similarity, block_count}

source carries the Git URL when discovered, otherwise local. The page groups results by source so an operator can see "matched 30% of XYZ from github.com/foo/bar" at a glance.

---

Storage

File	Purpose
Utils/DoppelgangerDB/FuzzyHash/FuzzyHash.db	The compressed block-hash database
Config/config.yaml → analysis.doppelganger.db.fuzzy_extensions	Default extensions for create_db_from_folder

Cleanup endpoint clears the DB along with Uploads/Results — call out the operator before doing this if the DB took a while to build.

---

See also

• Blender Scanner — the other doppelganger module (live-host runtime IOCs)
• Detection Score Explained — fuzzy-hash matches don't feed the score directly
• GrumpyCats CLI — doppelganger-analyze and doppelganger-db commands
• ssdeep — fuzzy hashing library used under the hood