Sift Secrets

Credential and sensitive-data exposure triage for file shares.

When an open share turns up, the question is never "does this repo have a leaked key". It is "what just got exposed, and what do I have to roll before close of business." sift is built for that question: high recall, a fast review queue, and a feedback loop so that anything you spot by eye becomes a rule that finds the other two hundred copies.

Python 3.11+, standard library only. No pip install, no internet, no build step. It runs on a locked-down IR laptop, which is where you need it.

sift survey \\fileserver\openshare     # how big is this thing
sift copy   \\fileserver\openshare C:\IR\case-4471    # take a throttled copy
sift scan   C:\IR\case-4471            # scan it, opens the triage UI

Scanning in place works too. Try it on a share of fabricated credentials first:

sift demo C:\temp\demoshare

sift.cmd is a launcher that works from any directory. To type sift instead of the full path, add C:\Dev\sift to PATH:

setx PATH "%PATH%;C:\Dev\sift"

For a machine with no Python at all, python build_portable.py builds dist/sift-secrets-<version>-portable-win64.zip: the official python.org embeddable runtime plus this source tree, unzip-and-run via the bundled sift.cmd. ~11 MB, no install, no admin rights, and nothing in it is compiled or repacked — see the docstring in build_portable.py for why that beats a frozen .exe on a locked-down IR laptop.

---

Why not just gitleaks or trufflehog

Both are good tools solving a different problem.

They are precision tools built for CI, where a false positive costs a developer their afternoon, so they fire mainly on things shaped like a known vendor API key. trufflehog goes further and prefers secrets it can verify by calling the vendor's API, which is a genuinely excellent signal that regex cannot reproduce.

Share triage inverts the economics. A human is already reading every hit, so a false positive costs three seconds. What costs you is a miss.

Vendor API keys absolutely do leak on shares - a web-root backup, a deployment script, someone's project folder copied to the departmental drive, and there is an .env with a live Stripe key in it. Those are worth catching, and sift catches them. But they are also the part gitleaks and trufflehog already handle well. The gap is everything else, and on a file share it is most of it:

What the CI scanners miss	Why they walk past it
web.config with a SQL connection string	Not a known key format, no vendor to verify against
Map-Drives.ps1 with net use ... /user:	Just a shell command with a word after it
New Hire Setup Guide.docx	Office file, read as binary, skipped entirely
unattend.xml, GPP Groups.xml	Windows deployment artefacts nobody wrote a detector for
confCons.xml, .rdg, WinSCP.ini	Reversible stored passwords, but not a "secret format"
passwords.xlsx	It is a ZIP. Plain-text scanners see binary and move on
.kdbx, .pfx, id_rsa	Opaque bytes - the filename is the finding
A .bak with a connection string inside	Binary, so never read

sift covers those, ships its own vendor-key rules, and imports other tools' rule packs and findings - gitleaks TOML, Kingfisher/Titus YAML, and trufflehog JSON - so you are not choosing between tools.

Closest prior art is Snaffler, which is excellent at the filename-and-classification half of this and is the direct inspiration for the filename rules. What it does not have - and what turns out to be the actual bottleneck once you have 400 hits - is a review loop.

---

The loop

1. Scan the share.
2. Work the queue. Each finding shows its surrounding lines with the match highlighted. Arrow keys widen the context; one click opens the whole file in VS Code at that line, or in Notepad.
3. Spot a miss. You will. Highlight it in the preview and press r.
4. sift proposes patterns and tells you live how many times each would match across everything already read.
5. Save it. The cached rescan takes about a second, and the new hits appear in the queue with your existing triage decisions untouched.

Step 5 is the part that makes the rest worth doing. Findings are keyed on (path, rule, line, value-hash), so a rescan re-inserts the same rows and your status, notes, and owner ride along. Without that you would re-review the same 300 hits on every iteration and give up on the third one.

---

Commands

# size it up first: file count, total bytes, biggest folders, transfer estimates
sift survey \\fileserver\share

# take a rate-limited local copy (resumable; gentle = 5 MB/s by default)
sift copy \\fileserver\share C:\IR\case-4471 --speed gentle

# scan a share and open the triage UI
sift scan \\fileserver\share

# maximum recall: more noise, but a human is reading anyway
sift scan D:\dfs\dept --tier 3

# re-open the UI over the most recent scan
sift ui

# build a share of fabricated credentials, scan it, open the UI
sift demo C:\temp\demoshare

# inherit other tools' vendor-key rules, then use them in the live rescan loop
sift import-rules gitleaks.toml                    # gitleaks TOML
sift import-rules path/to/kingfisher/data/rules    # a directory of YAML

# pull in what the other scanners found, into the same queue
trufflehog filesystem \\fileserver\share --json > th.json
sift import-findings th.json

# hand off to the incident record (redacted unless you say otherwise)
sift export --fmt pdf --status confirmed --out ir-4471.pdf
sift export --fmt csv --out ir-4471.csv
sift export --fmt pdf --no-redact       # plaintext; handle as evidence

sift rules                              # what is loaded
sift selftest                           # detection tests against a synthetic share

# ask vendors whether confirmed findings are still live. NETWORK. Opt-in.
sift validate --status confirmed

Anywhere sift appears you can use python -m sift instead, from the C:\Dev\sift directory.

Options worth knowing

Flag	Effect
--tier 1|2|3	Recall dial. 1 = high signal, 2 = default, 3 = miss nothing
--redact	Mask values in the store and exports. Use if the DB leaves the incident boundary
--no-ui	Populate the store and exit, for scripted runs
--no-browser	Start the UI server but do not open a browser (useful over RDP)
--include/--exclude GLOB	Narrow the walk
--no-archives	Do not open docx/xlsx/zip containers
--no-strings	Do not run a strings pass over binaries
--no-large	Skip files over --max-size instead of reading them in blocks
--jobs N	Worker processes (default: auto)
--max-size MB	Skip files above this (default 25)
--port N	UI port (default 8973)

Where state lives

Findings go to a per-target folder under %LOCALAPPDATA%\sift\, never the working directory - the database holds plaintext credentials, and running the tool from your home directory should not silently drop one there. Each share gets its own store, so two engagements never share a triage queue. sift ui with no arguments reopens the most recent one; --data DIR overrides.

Custom rules are global, at %LOCALAPPDATA%\sift\user-rules.json, so a pattern you write during one engagement helps on the next share you look at.

---

Before you scan: survey and acquire

Both are in the header of the UI, next to the path box, and on the CLI.

Check size is a stat-only walk. Nothing is opened, so it is cheap even over SMB, and it tells you the file count, total bytes, largest folders, the split by extension, how much sift would actually read, and how long a copy would take at each speed. Pointing a scanner at an unknown DFS root and waiting is how an afternoon disappears.

Copy locally pulls the share to a local folder first. Worth doing because:

• scanning a local copy is far faster than thousands of SMB round trips;
• it is repeatable, so adding a rule and rescanning does not re-hammer the file server;
• the original is left alone, which is the difference between "we looked at it" and "we preserved it" if the incident becomes legal.

The transfer is rate limited, defaulting to 5 MB/s. Saturating the link to a production file server at 2pm turns your investigation into a second incident. Open it up when you know the path is idle.

Speed	Rate
gentle (default)	5 MB/s
normal	25 MB/s
fast	100 MB/s
unlimited	whatever the link gives

Transfers resume: a destination file with the same size and mtime is skipped, so an acquisition interrupted at 80% picks up where it stopped. Files that are locked or denied are recorded and skipped rather than aborting the run.

---

Reports

PDF, CSV, MD, and JSON, from the UI header or sift export --fmt.

The PDF is the one to hand to an incident record: a cover page with the target, totals by severity and triage state, the most frequent rules, values reused across files, then the findings grouped by severity. It is generated directly, with no PDF library, so it works on a machine that has never seen pip.

Exports are redacted by default. Values are masked, every page is banner-marked, and the endpoint only disables masking on an explicit redact=0 - a truncated or malformed request cannot leak. Turning redaction off in the UI requires confirming a warning dialog, and the resulting file is banner-marked UNREDACTED - CONTAINS PLAINTEXT CREDENTIALS on every page.

The findings database still holds the real values, because an analyst needs to know which password leaked to know what to roll. The boundary is what leaves the tool.

---

Working the queue

Everything is clickable. The sort control at the top of the results list reorders by severity, file path, rule, triage status, or most recent, with a button to reverse it. The sidebar facets filter by severity, category, rule, and reused value. Triage, context expansion, opening a file, and rule creation are all buttons.

The keyboard shortcuts below are accelerators for a long queue, not the only way to drive it.

Key	Action
j / k	next / previous finding
↑ / ↓	widen context up / down
c / f	confirm / mark false positive
x	toggle selection for bulk triage
o / n	open in VS Code at the line / open the file in Notepad
r	build a rule from the highlighted text
y	copy the value
/	search

The snapshot button renders the highlighted snippet to a PNG for pasting into an incident ticket; copy snippet does the same as markdown.

---

Security

The UI renders live plaintext credentials in a browser window, so:

• binds 127.0.0.1 only, and refuses anything else without --unsafe-bind;
• requires a random per-launch token, handed over in the launch URL and then held in a SameSite=Strict cookie;
• validates the Host header, so a hostile page cannot DNS-rebind to it;
• sends a CSP with no external origins at all - nothing on the page can exfiltrate what it renders;
• refuses to read or open any path that is not already in the findings database, so /api/context is not an arbitrary file read and /api/open is not an arbitrary process launch.

The findings database contains plaintext credentials by design - an IR analyst needs to know which password leaked to know what to roll. Treat %LOCALAPPDATA%\sift\<target>\findings.db as evidence: same handling as the share itself, and delete it when the engagement closes. Use --redact if it will leave the incident boundary.

And the obvious one: run sift only against systems you are authorized to access.

---

How the rules are organised

sift/rules_builtin.py - content rules. sift/rules_filename.py - filename rules. Both are plain Python with raw-string patterns, so they are readable and diffable; user rules live in JSON at .sift/user-rules.json.

Three tiers let you trade precision for recall:

• Tier 1 - unambiguous. Vendor key formats, PEM blocks, GPP cpassword, NTLM dumps, LDAP bind passwords. Shape alone is proof.
• Tier 2 (default) - proximity rules. "A secret-ish word near a value." Where most real share exposure lives.
• Tier 3 - unlabelled high-entropy strings, long hex, IBANs. Noisy, but when you are scoping a breach you would rather read 400 hits than miss one.

A rule can also carry min_digits, min_lowercase, min_uppercase, and min_special, so a single noisy rule can be tightened without touching the rest, and examples - strings it must still match.

Rules that check themselves

examples is the useful half. sift selftest runs every rule against the strings it was written for, through the whole path: match, extract, then the suppression filters. That last step is the one that matters, because the regression that actually happens is not a pattern that stops matching - it is a noise filter, tightened for good reasons somewhere else, quietly eating a real finding on its way out.

It earns its keep immediately. Adding examples to the existing rules turned up a live gap: an underscore is a word character, so the leading \b in the generic-assignment rule refused to match inside DB_PASSWORD, MYSQL_PASSWORD, or REDIS_PASSWORD - three of the most common credential variable names in existence, silently missed. The example looked obviously correct and did not match, which is exactly what it is for.

Rules you write in the UI get this free: the line you selected is saved as the rule's example, so a rule you edit six months later tells you when it has stopped matching the thing that made you write it.

Importing other tools' rule packs

sift import-rules takes a gitleaks .toml, a Kingfisher/Titus-style .yml, or a directory of them. Against Kingfisher's pack that is 1,073 of 1,082 rules imported, bringing their entropy floors, digit and case requirements, and examples with them. YAML is read by sift/yamlmini.py, a reader for the subset these packs use - no dependency, and it raises on anchors and tags rather than pretending to understand them.

Two things are deliberately dropped on the way in:

• validation: blocks, which name a URL per rule. sift only ever contacts hosts hardcoded in validate.py. A rule pack that could name an endpoint would be choosing where your findings get sent, and a rule file is data, not a decision.
• Any rule whose pattern no longer matches its own documented example. The packs are written for Rust and Hyperscan, where [[:alnum:]] is a POSIX class; Python reads it as a set of literal characters, compiles it happily, and matches the wrong thing. Translation is checked against each rule's examples, so a pattern that survived compilation but changed meaning is rejected instead of quietly never firing. Twelve of Kingfisher's rules fail that check and are not imported.

Whether sift's own suppression would drop a given example does not disqualify the rule - those packs ship deliberately fake samples (keyXXXXXXXX, ...EXAMPLE), so the placeholder filter is right about the example and says nothing about the pattern. Being strict about that discarded 121 working rules before the distinction was drawn.

Suppression is doing as much work as detection

Generic secret rules get abandoned because of noise, so the noise filters are tuned as carefully as the patterns. Measured against a real 6,000-file tree, the suppression rules below cut findings from 1,088 to 257 with no loss of recall on the test corpus:

• Dotted code references. password: process.env.DB_PASS is a variable, not a value. This one suppression removes most generic-rule noise in source trees.
• Type annotations. def login(user: str, password: str) is a signature.
• Values leading with punctuation are a regex that clipped an expression mid-way, not a credential.
• Prose requires a digit or symbol. Without it, the rule reads English about credentials - "a credential in the URL is reported and stripped" yields "reported". 99 false positives on one tree taught that.
• Build and cache directories (.wrangler, .next, site-packages, node_modules, …) plus minified bundles and source maps are skipped. Machine- generated text produces only machine-generated false positives.

There is deliberately no https://user:pass@host content rule with a loose tail. The obvious version fired 476 times on one dev tree, because minified JSON has no whitespace and the pattern ran from one URL across quotes and commas until it found an unrelated @ hundreds of characters later.

---

Tests

python tests/run_all.py

Twelve suites: detection against a synthetic share (split into "what the CI scanners already catch", "the gap this tool targets", and "decoys that must stay quiet"), the command line, rule-suggestion ranking, survey and throttled acquisition (including measuring the rate limit against wall clock), the PDF writer (parsed back the way a reader would, to prove redaction reached the page content), the gitleaks/trufflehog importers, the HTTP API including every security guard, static analysis of the UI, and the block reader for oversized files (asserting a secret in the last block still reports its true line number in the whole file).

To get a share to play with:

sift demo C:\temp\demoshare

Every credential in that corpus is fabricated.

---

Verification

Two levels, because they carry very different risk.

Checksums are free and always on. ghp_, npm_, Atlassian ATATT, Bitbucket ATCTT and GitLab's routable glpat- tokens all carry a CRC32 over their own body. Recomputing it answers offline what used to need the internet: is this a real token, or the example someone pasted into a README? Findings are tagged checksum ok or malformed in the queue. A checksum proves form, not life - a well-formed token may have been revoked a year ago.

Live validation is off until you run it. sift validate asks the vendor whether a credential still works. It is a separate command, not a flag on scan, and it makes you type validate at a prompt that first names every endpoint it will contact. The safety rules:

• Endpoints are hardcoded in sift/validate.py. No rule - built-in, user-written, or imported from someone else's pack - can supply a URL. Without that, importing a rule pack would be enough to ship every credential on the share to an address of the pack author's choosing.
• The secret travels in a header, never in a URL or query string, so it stays out of proxy logs and vendor access logs.
• Redirects are not followed. A 302 is an instruction to send the credential somewhere else.
• Only credentials that already pass their checksum are sent, so obvious fakes never leave the machine.
• It refuses to run against a --redact store: if values are masked because the database is leaving the incident boundary, transmitting the plaintext is the exact thing being guarded against.
• Every call is written to validation-log.json next to the findings database
  • provider, endpoint, time, outcome - so the engagement can say precisely what was contacted.

Validation attempts land in the credential owner's audit logs, attributed to your address, at that moment. That is sometimes exactly what you want and sometimes tips off an adversary who is watching. Decide before you run it; that is why it asks.

Currently GitHub, npm, Slack, and Stripe. trufflehog still verifies far more - run it and import-findings to get both.

---

Known limits

• Verification is narrow. Checksums cover five token families; live validation covers four providers. trufflehog verifies hundreds - run it and import-findings; verified hits sort to the top.
• Encrypted containers are filename-only. A .kdbx or .pfx is reported by name; sift does not attempt to open it.
• SQLite databases are read as tables, not bytes. Detected by header rather than extension, opened read-only and immutable so nothing is written beside a file that is evidence. Key/value config tables are recombined into name=value, without which the secret word and its value sit in separate columns and no rule sees the pair.
• .7z, .rar, and nested archives are flagged but not extracted. Only ZIP- based formats and .eml messages are read inside.
• Files past --max-size are read in blocks, not cached. They are scanned (a connection string in a 4 GB .bak is found, at its real line number), but the text cache holds only what fits, so the one-second cached rescan does not cover them - a new rule reaches a large file on the next full scan.
• No share enumeration. Point it at a path you already have. Finding open shares is a different tool's job.
• .doc/.xls/.pdf (pre-2007 and PDF) go through a strings pass rather than a real parser, so recall on those is lower than on the OOXML formats.

---

License

Apache-2.0. See LICENSE.

The detection test corpus (sift/selftest.py, tests/) contains format-correct but fabricated credentials on purpose; secret-scanning alerts for those paths are suppressed via .github/secret_scanning.yml.