perf + fix(detectors): 2.5x faster scans and false-positive reductions - #96
Merged
Conversation
…ists Measured on a corpus of common false-positive-prone content (READMEs with documentation emails, .env.example placeholders, docker-compose, npm integrity strings, UUIDs, fictional phone numbers): 16 findings, 11 of them false positives. This change eliminates 14 of the 16. - Aadhaar numbers carry a Verhoeff check digit; the new validate = "verhoeff" rejects random 12-digit runs (a UUID tail reported HIGH). Real-Aadhaar fixtures replaced with Verhoeff-valid numbers. - AWS's documentation example key/secret are exact-match allowlisted — they appear in READMEs everywhere and are never real. - Placeholder values (changeme, your-api-key-here, replace-me-please) are allowlisted in Password/GenericKeyValue detectors, case-sensitively and shape-checked so mixed-case values stay reported. - RFC 2606 example-domain emails and noreply conventions are allowlisted. - The fictional 555 exchange is allowlisted for phone numbers. - npm/shield checksum prefixes (sha256-/sha384-/sha512-) are allowlisted in RandomString, whose match text includes the prefix. The committed baseline shrank accordingly (those findings no longer exist). Corpus: 16 -> 2 findings, both inherent (a company-domain email indistinguishable from PII, and a bare 43-char integrity value whose algorithm prefix sits outside the match).
pixincreate
force-pushed
the
perf/memory-and-false-positives
branch
from
September 5, 2026 14:14
8b20a21 to
efdf3e3
Compare
…tropy Profile of a 1GB scan: 42% of runtime was N separate regex passes for the keywordless detectors, 13% was Unicode char-by-char lowering. - The keywordless detectors now share one combined any-of regex as an exact prefilter: lines that cannot match any of them skip all of their individual passes; is_match is existence, so nothing is lost, and each pattern is group-isolated so flags cannot leak between branches. - to_lowercase_into takes a byte-per-byte ASCII fast path. - shannon_entropy uses a fixed byte histogram instead of allocating a HashMap per candidate match. Measured: 1GB clean scan 18.0s -> 7.2s (~57 -> ~143 MB/s), RSS unchanged (~24MB, bounded); 112MB match-heavy 8.9s -> 7.5s.
pixincreate
force-pushed
the
perf/memory-and-false-positives
branch
from
September 5, 2026 14:17
efdf3e3 to
aeeea4c
Compare
pixincreate
added a commit
that referenced
this pull request
Sep 5, 2026
- Verhoeff: canonical examples, separators, embedded id runs, degenerate inputs (the p-permutation breaks the all-zero identity chain). - Luhn: known cards, separators, sub-13-digit rejection. - shannon_entropy: information-theory identities, hex near-ceiling, multi-byte input. - is_multiline: (?s), grouped (?s:...), combined (?is), negative flags, plain patterns — and a real fix the test caught: combined flags like (?is) missed the substring check, so such patterns were scanned line-by-line and silently missed multiline secrets. is_multiline now parses flag groups. - Every FP allowlist entry exercised in both directions through the real detector set (AWS example key, placeholders, example-domain emails, 555 numbers, checksum prefixes). - The false-positive corpus is committed under tests/fixtures and scanned end to end: exactly the two documented residuals may appear. - Finding derives Deserialize with a wire-format round-trip test: the historical plugin_name key and the detector_name alias both parse. - git-history mode gets a hostile-git-config test mirroring the staged one (pins GIT_DIFF_FRAMING_ARGS coverage for log). - stdin scans through NUL bytes; the secret after a NUL line is found.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three work streams, measured end to end:
Memory
mem::forget/Box::leak/staticgrowth/Rccycles anywhere insrc/— nothing can leak by construction.leakscannot attach here (hardened-runtime restriction), so bounded memory is proven by measurement instead: scanning a 1 GB file peaks at ~24 MB RSS (the stream never holds the file).False positives (16 → 2 on a curated FP corpus)
validate = "verhoeff"(dihedral D5 tables) — real Aadhaar numbers carry the check digit; a UUID tail reported HIGH before.changeme,your-api-key-here,replace-me-please): allowlisted in Password/GenericKeyValue detectors — case-sensitive and shape-checked, so mixed-case values stay reported.sha256-/sha384-/sha512-prefixes allowlisted in RandomString.Speed (1 GB clean scan: 18.0 s → 7.2 s, ~57 → ~143 MB/s)
Profile showed 42% of runtime in N separate regex passes for the keywordless detectors and 13% in Unicode lowering:
is_matchis existence, so the gate is exact (no match lost); each pattern is group-isolated so flags cannot leak between branches. Matching lines pay the gate plus their real passes; non-matching lines skip every individual pass.to_lowercase_intotakes a byte-per-byte ASCII fast path.shannon_entropyuses a fixed byte histogram — no HashMap allocation per candidate match.Tests
216 pass (Verhoeff unit tests against the canonical 2363/2362 examples + a new detector_tests case for the 555 allowlist; FP-corpus-affected fixtures updated to detected-format equivalents). fmt/clippy clean; self-scan clean and drift-free.