Skip to content

v0.48.0 — close v0.47 held-out underfit, cleanly

Choose a tag to compare

View all tags
@byevincent byevincent released this 10 Jun 03:31
· 14 commits to main since this release
v0.48.0
724adfd

ShareSift v0.48.0 — same-day follow-up to v0.47. Closes the held-out underfit by running the discipline experiment properly: lock NEW held-out FIRST, then write rules from OLD held-out failures only, then validate.

TL;DR

Gate v0.47 v0.48
Corpus (training) 18/19 (95%) 18/19 (95%)
Held-out v1 4/11 (36%) 10/11 (91%)
Held-out v2 (new locked) n/a 7/10 (70%)
MSF3 / MSF2 / DiskForge recall 1.000 / 1.000 / 0.923 1.000 / 1.000 / 0.923 (held)
v0.48 rule FP contribution n/a 0 across all three

The generalization signal: ShareSiftKeepBrowserSavedCreds was authored as "generalize Firefox to other Chromium-base browsers." It directly closed 2 held-out v2 probes (Chrome + Edge Login Data) that were locked BEFORE the rule was written — pattern-level generality catching parallel patterns. That's the discipline working as intended.

Full writeup in docs/v0p48_results.md.

Seven new rules (close OLD held-out, sourced #78/#135/#67/#46)

Rule Tier Match Closes
ShareSiftKeepCiscoEnableSecret Red Content #78 (Cisco IOS enable secret/password/type-7)
ShareSiftKeepCiscoSnmpCommunity Red Content #78 (SNMP RW community)
ShareSiftKeepCiscoSnmpCommunityRo Yellow Content #78 (SNMP RO community)
ShareSiftKeepFileZillaSavedSites Black FilePath #135 (sitemanager.xml saved FTP/SFTP)
ShareSiftKeepFileZillaRecentServers Yellow FilePath #135 (recentservers.xml)
ShareSiftKeepDotNetAppSettingsConnString Red Content #67 (.NET appsettings.json conn string)
ShareSiftKeepBrowserSavedCreds Black FilePath #46 (Chrome/Edge/Brave/Opera Login Data)

Both extra_rules.json (engine) and extra_rules.py (pysnaffler compat).

New held-out v2 (locked test set)

benchmarks/snaffler_issues/heldout_v2.jsonl — 10 probes from previously-unread Snaffler PR sources:

  • #198 (CMD set PASSWORD=)
  • #155 (Azure CLI az login --password)
  • #124 (XML <password> with nested tag)
  • #98 (loose "credential" filename keyword)
  • #46 (Chrome + Edge Login Data — Firefox cousins)

Pre-rule baseline: 5/10 (the v0.47 KeepDoubleDashPassphrase already generalized to Azure CLI patterns — free signal). Post-rule: 7/10 (browser-creds meta-rule catches Chrome + Edge).

eval_snaffler_issues.py grows --set {corpus,heldout,heldout_v2,all}.

What's NOT in v0.48 (deliberate discipline)

3 held-out v2 fails come from sources I MINED for held-out v2:

  • heldout-v2-198-cmd-set-pgpassword-quoted — set "PGPASSWORD=val"
  • heldout-v2-98-credential-in-filename — credentials_2024.xlsx
  • heldout-v2-98-credentials-export — CustomerCredentialsExport.csv

Adding rules for these in v0.48 would be tuning toward held-out v2 (discipline violation). They become v0.49 candidates — a future held-out v3 will validate them against patterns I haven't yet read.

This is how a discipline-honest research cycle should grow: each version locks the next test set BEFORE writing the rules that close the previous one.

Existing benchmark impact

Benchmark v0.47 R v0.48 R v0.48 rule FP
MSF3 1.000 1.000 0
MSF2 1.000 1.000 0
DiskForge 0.923 0.923 0

Zero v0.48 rules fired on any of the three (neither TP nor FP). The Cisco IOS / FileZilla / ADO / browser-creds patterns don't appear in those substrates — MSF3 is AD Windows-shaped, MSF2 Linux Metasploitable, DiskForge a forensic disk image. Rules are surgical to corporate-share patterns.

Binary

77.2 MB single-file binary attached (sharesift). Verified:

wget https://github.com/byevincent/ShareSift/releases/latest/download/sharesift
chmod +x sharesift
./sharesift --version  # sharesift 0.48.0

v0.49 candidate list

  1. Close held-out v2 remaining gaps (CMD set "VAR=val" quoted variant, loose "credential" filename keyword)
  2. Lock held-out v3 from yet-unread sources (#112 SCCM, #140 Kerberos, #139 MDE Linux)
  3. After v0.49: three generations of held-out signal = calibrated confidence in "corporate-share benchmark progress"

🤖 Generated with Claude Code

Assets 3
Loading