Sift Secrets

Credential and sensitive-data exposure triage for file shares.

When an open share turns up, the question is never "does this repo have a leaked key". It is "what just got exposed, and what do I have to roll before close of business." sift is built for that question: high recall, a fast review queue, and a feedback loop so that anything you spot by eye becomes a rule that finds the other two hundred copies.

Python 3.11+, standard library only. No pip install, no internet, no build step. It runs on a locked-down IR laptop, which is where you need it.

sift survey \\fileserver\openshare     # how big is this thing
sift copy   \\fileserver\openshare C:\IR\case-4471    # take a throttled copy
sift scan   C:\IR\case-4471            # scan it, opens the triage UI

Scanning in place works too. Try it on a share of fabricated credentials first:

sift demo C:\temp\demoshare

sift.cmd is a launcher that works from any directory. To type sift instead of the full path, add C:\Dev\sift to PATH:

setx PATH "%PATH%;C:\Dev\sift"

For a machine with no Python at all, python build_portable.py builds dist/sift-secrets-<version>-portable-win64.zip: the official python.org embeddable runtime plus this source tree, unzip-and-run via the bundled sift.cmd. ~11 MB, no install, no admin rights, and nothing in it is compiled or repacked — see the docstring in build_portable.py for why that beats a frozen .exe on a locked-down IR laptop.

---

Why not just gitleaks or trufflehog

Both are good tools solving a different problem.

They are precision tools built for CI, where a false positive costs a developer their afternoon, so they fire mainly on things shaped like a known vendor API key. trufflehog goes further and prefers secrets it can verify by calling the vendor's API, which is a genuinely excellent signal that regex cannot reproduce.

Share triage inverts the economics. A human is already reading every hit, so a false positive costs three seconds. What costs you is a miss - and the things that leak on a file share are almost never vendor API keys:

What actually leaks on a share	Why the CI scanners walk past it
web.config with a SQL connection string	Not a known key format, no vendor to verify against
Map-Drives.ps1 with net use ... /user:	Just a shell command with a word after it
New Hire Setup Guide.docx	Office file, read as binary, skipped entirely
unattend.xml, GPP Groups.xml	Windows deployment artefacts nobody wrote a detector for
confCons.xml, .rdg, WinSCP.ini	Reversible stored passwords, but not a "secret format"
passwords.xlsx	It is a ZIP. Plain-text scanners see binary and move on
.kdbx, .pfx, id_rsa	Opaque bytes - the filename is the finding
A .bak with a connection string inside	Binary, so never read

sift covers those, keeps the vendor-key rules too, and will import gitleaks' rule pack and trufflehog's findings so you are not choosing between tools.

Closest prior art is Snaffler, which is excellent at the filename-and-classification half of this and is the direct inspiration for the filename rules. What it does not have - and what turns out to be the actual bottleneck once you have 400 hits - is a review loop.

---

The loop

1. Scan the share.
2. Work the queue. Each finding shows its surrounding lines with the match highlighted. Arrow keys widen the context; one click opens the whole file in VS Code at that line, or in Notepad.
3. Spot a miss. You will. Highlight it in the preview and press r.
4. sift proposes patterns and tells you live how many times each would match across everything already read.
5. Save it. The cached rescan takes about a second, and the new hits appear in the queue with your existing triage decisions untouched.

Step 5 is the part that makes the rest worth doing. Findings are keyed on (path, rule, line, value-hash), so a rescan re-inserts the same rows and your status, notes, and owner ride along. Without that you would re-review the same 300 hits on every iteration and give up on the third one.

---

Commands

# size it up first: file count, total bytes, biggest folders, transfer estimates
sift survey \\fileserver\share

# take a rate-limited local copy (resumable; gentle = 5 MB/s by default)
sift copy \\fileserver\share C:\IR\case-4471 --speed gentle

# scan a share and open the triage UI
sift scan \\fileserver\share

# maximum recall: more noise, but a human is reading anyway
sift scan D:\dfs\dept --tier 3

# re-open the UI over the most recent scan
sift ui

# build a share of fabricated credentials, scan it, open the UI
sift demo C:\temp\demoshare

# inherit gitleaks' vendor-key rules, then use them in the live rescan loop
sift import-rules gitleaks.toml

# pull in what the other scanners found, into the same queue
trufflehog filesystem \\fileserver\share --json > th.json
sift import-findings th.json

# hand off to the incident record (redacted unless you say otherwise)
sift export --fmt pdf --status confirmed --out ir-4471.pdf
sift export --fmt csv --out ir-4471.csv
sift export --fmt pdf --no-redact       # plaintext; handle as evidence

sift rules                              # what is loaded
sift selftest                           # detection tests against a synthetic share

Anywhere sift appears you can use python -m sift instead, from the C:\Dev\sift directory.

Options worth knowing

Flag	Effect
--tier 1|2|3	Recall dial. 1 = high signal, 2 = default, 3 = miss nothing
--redact	Mask values in the store and exports. Use if the DB leaves the incident boundary
--no-ui	Populate the store and exit, for scripted runs
--no-browser	Start the UI server but do not open a browser (useful over RDP)
--include/--exclude GLOB	Narrow the walk
--no-archives	Do not open docx/xlsx/zip containers
--no-strings	Do not run a strings pass over binaries
--jobs N	Worker processes (default: auto)
--max-size MB	Skip files above this (default 25)
--port N	UI port (default 8973)

Where state lives

Findings go to a per-target folder under %LOCALAPPDATA%\sift\, never the working directory - the database holds plaintext credentials, and running the tool from your home directory should not silently drop one there. Each share gets its own store, so two engagements never share a triage queue. sift ui with no arguments reopens the most recent one; --data DIR overrides.

Custom rules are global, at %LOCALAPPDATA%\sift\user-rules.json, so a pattern you write during one engagement helps on the next share you look at.

---

Before you scan: survey and acquire

Both are in the header of the UI, next to the path box, and on the CLI.

Check size is a stat-only walk. Nothing is opened, so it is cheap even over SMB, and it tells you the file count, total bytes, largest folders, the split by extension, how much sift would actually read, and how long a copy would take at each speed. Pointing a scanner at an unknown DFS root and waiting is how an afternoon disappears.

Copy locally pulls the share to a local folder first. Worth doing because:

• scanning a local copy is far faster than thousands of SMB round trips;
• it is repeatable, so adding a rule and rescanning does not re-hammer the file server;
• the original is left alone, which is the difference between "we looked at it" and "we preserved it" if the incident becomes legal.

The transfer is rate limited, defaulting to 5 MB/s. Saturating the link to a production file server at 2pm turns your investigation into a second incident. Open it up when you know the path is idle.

Speed	Rate
gentle (default)	5 MB/s
normal	25 MB/s
fast	100 MB/s
unlimited	whatever the link gives

Transfers resume: a destination file with the same size and mtime is skipped, so an acquisition interrupted at 80% picks up where it stopped. Files that are locked or denied are recorded and skipped rather than aborting the run.

---

Reports

PDF, CSV, MD, and JSON, from the UI header or sift export --fmt.

The PDF is the one to hand to an incident record: a cover page with the target, totals by severity and triage state, the most frequent rules, values reused across files, then the findings grouped by severity. It is generated directly, with no PDF library, so it works on a machine that has never seen pip.

Exports are redacted by default. Values are masked, every page is banner-marked, and the endpoint only disables masking on an explicit redact=0 - a truncated or malformed request cannot leak. Turning redaction off in the UI requires confirming a warning dialog, and the resulting file is banner-marked UNREDACTED - CONTAINS PLAINTEXT CREDENTIALS on every page.

The findings database still holds the real values, because an analyst needs to know which password leaked to know what to roll. The boundary is what leaves the tool.

---

Working the queue

Everything is clickable. The sort control at the top of the results list reorders by severity, file path, rule, triage status, or most recent, with a button to reverse it. The sidebar facets filter by severity, category, rule, and reused value. Triage, context expansion, opening a file, and rule creation are all buttons.

The keyboard shortcuts below are accelerators for a long queue, not the only way to drive it.

Key	Action
j / k	next / previous finding
↑ / ↓	widen context up / down
c / f	confirm / mark false positive
x	toggle selection for bulk triage
o / n	open in VS Code at the line / open the file in Notepad
r	build a rule from the highlighted text
y	copy the value
/	search

The snapshot button renders the highlighted snippet to a PNG for pasting into an incident ticket; copy snippet does the same as markdown.

---

Security

The UI renders live plaintext credentials in a browser window, so:

• binds 127.0.0.1 only, and refuses anything else without --unsafe-bind;
• requires a random per-launch token, handed over in the launch URL and then held in a SameSite=Strict cookie;
• validates the Host header, so a hostile page cannot DNS-rebind to it;
• sends a CSP with no external origins at all - nothing on the page can exfiltrate what it renders;
• refuses to read or open any path that is not already in the findings database, so /api/context is not an arbitrary file read and /api/open is not an arbitrary process launch.

The findings database contains plaintext credentials by design - an IR analyst needs to know which password leaked to know what to roll. Treat %LOCALAPPDATA%\sift\<target>\findings.db as evidence: same handling as the share itself, and delete it when the engagement closes. Use --redact if it will leave the incident boundary.

And the obvious one: run sift only against systems you are authorized to access.

---

How the rules are organised

sift/rules_builtin.py - content rules. sift/rules_filename.py - filename rules. Both are plain Python with raw-string patterns, so they are readable and diffable; user rules live in JSON at .sift/user-rules.json.

Three tiers let you trade precision for recall:

• Tier 1 - unambiguous. Vendor key formats, PEM blocks, GPP cpassword, NTLM dumps, LDAP bind passwords. Shape alone is proof.
• Tier 2 (default) - proximity rules. "A secret-ish word near a value." Where most real share exposure lives.
• Tier 3 - unlabelled high-entropy strings, long hex, IBANs. Noisy, but when you are scoping a breach you would rather read 400 hits than miss one.

Suppression is doing as much work as detection

Generic secret rules get abandoned because of noise, so the noise filters are tuned as carefully as the patterns. Measured against a real 6,000-file tree, the suppression rules below cut findings from 1,088 to 257 with no loss of recall on the test corpus:

• Dotted code references. password: process.env.DB_PASS is a variable, not a value. This one suppression removes most generic-rule noise in source trees.
• Type annotations. def login(user: str, password: str) is a signature.
• Values leading with punctuation are a regex that clipped an expression mid-way, not a credential.
• Prose requires a digit or symbol. Without it, the rule reads English about credentials - "a credential in the URL is reported and stripped" yields "reported". 99 false positives on one tree taught that.
• Build and cache directories (.wrangler, .next, site-packages, node_modules, …) plus minified bundles and source maps are skipped. Machine- generated text produces only machine-generated false positives.

There is deliberately no https://user:pass@host content rule with a loose tail. The obvious version fired 476 times on one dev tree, because minified JSON has no whitespace and the pattern ran from one URL across quotes and commas until it found an unrelated @ hundreds of characters later.

---

Tests

python tests/run_all.py

Eight suites, 194 checks: detection against a synthetic share (split into "what the CI scanners already catch", "the gap this tool targets", and "decoys that must stay quiet"), the command line, rule-suggestion ranking, survey and throttled acquisition (including measuring the rate limit against wall clock), the PDF writer (parsed back the way a reader would, to prove redaction reached the page content), the gitleaks/trufflehog importers, the HTTP API including every security guard, and static analysis of the UI.

To get a share to play with:

sift demo C:\temp\demoshare

Every credential in that corpus is fabricated.

---

Known limits

• No verification. sift cannot tell you a key is live. That is trufflehog's real advantage - run it and import-findings; verified hits sort to the top.
• Encrypted containers are filename-only. A .kdbx or .pfx is reported by name; sift does not attempt to open it.
• .7z, .rar, and nested archives are flagged but not extracted. Only ZIP- based formats are read inside.
• No share enumeration. Point it at a path you already have. Finding open shares is a different tool's job.
• .doc/.xls/.pdf (pre-2007 and PDF) go through a strings pass rather than a real parser, so recall on those is lower than on the OOXML formats.

---

License

Apache-2.0. See LICENSE.

The detection test corpus (sift/selftest.py, tests/) contains format-correct but fabricated credentials on purpose; secret-scanning alerts for those paths are suppressed via .github/secret_scanning.yml.