v0.4.0 — holdout: regeneration becomes a proof

gauntlet holdout — the failure mode's answer, made provable.

After a holdout leaks, the doctrine says: retire it, regenerate from an unused seed. Until now that regeneration was manual and hoped to be fresh. v0.4 makes freshness a property you can check, not a hope you hold.

The new subsystem

gauntlet holdout new    family.json --manifest m.jsonl --private-dir ~/.private/holdout/ --count 3
gauntlet holdout retire family.json --instance task-0001 --manifest m.jsonl --private-dir ~/.private/holdout/
gauntlet holdout status family.json --manifest m.jsonl --private-dir ~/.private/holdout/
gauntlet holdout verify family.json --manifest m.jsonl --private-dir ~/.private/holdout/ --deep

• Family templates (private-side JSON): typed slots — int, int_list, choice, permutation — filled into {{ name }} prompt/fixture templates. Optional answer_cmd derives the verifier's expected value from the slot values; instance dirs carry prompt.txt, fixtures/, and a ready-to-feed task.json.
• Version-stable PRNG: counter-mode SHA-256 with rejection sampling. stdlib random is NOT stable across Python versions — byte-drift would silently rot provenance. A golden-hash test pins one family+seed digest; the CI matrix (3.11 + 3.13) proves it byte-identical.
• Dual freshness gate: same-family instances share boilerplate by design, so text distance would lie — intra-family freshness is exact slot-digest collision; cross-family is ≥2-shingle overlap against the manifest. Skipped counters are recorded honestly; exhaustion exits 1 with the budget named.
• Retirement ledger: append-only, hash-chained JSONL, 0600, private side. retire writes the manifest first (guard reads the manifest — stop the bleed), ledger second; a crash between the two writes is a named divergence, not a silent one. verify --deep re-derives every instance from template + seed and proves content hashes still match.
• Content-free ledger: hashes and ids only — a leaked ledger leaks nothing (redaction contract, ADR-0002).
• Cardinality accounting: small families are enumerable; status says so out loud.

Design pre-registered in docs/HOLDOUT_DESIGN.md + ADR-0012 BEFORE code. A fresh-eyes review pass landed its hardening before release: family validation on every operation, seq-stamped ledger entries, exclusive operation lock, full-digest content hashing, 2^63 slot-range cap.

Gates

106 tests (34 new), CI matrix green (3.11/3.13), ruff + ty clean, demo + selftest pass, published via trusted publishing (OIDC, no token).

v0.3.3 — the zero-setup hook: guard audit

gauntlet guard audit — the zero-setup hook — and a README that answers "is this my problem?" before asking for install.

The new command

uvx --from gauntlet-guard gauntlet guard audit

Walks the machine (default: home; caps: depth 6, 250k entries, 20 s — reported honestly when truncated) and reports agent memory surfaces: memory files, agent state dirs, transcript/log dirs, vector stores.

Content-free by construction: it never opens a file. Evidence is path, kind, size, mtime — nothing else. If a secret hides inside LESSONS.md, the audit output cannot leak it, because the content is never read (pinned mechanically by a test). Discovery is name-based and heuristic: it finds the surfaces to scan, not the leaks — the leak check still needs a manifest (guard scan). Exit code is always 0: an audit is information, not a verdict.

Dogfooding on a real home directory: 98k entries in 2.3 s. Package-manager and plugin caches (which ship bundled AGENTS.md docs) are skipped as noise. Extends the redaction contract (ADR-0002); recorded in ADR-0011.

README rebuilt around the first-time visitor

• a problem-fit checklist with an honest disqualifier ("stateless agent, one-shot evals? you don't need this")
• the 30-second machine audit as the hook, before the demo
• "why not just keep the test set secret?"
• a plain-language glossary
• real captured output for the grade verdict
• examples/mini — a copy-paste end-to-end walkthrough of the whole protocol (~15 s, $0, no LLM): sealed trials → blind pack → KEEP verdict → planted leak caught by guard scan

Gates

69 tests (15 new for audit), CI matrix green (3.11/3.13), ruff + ty clean, demo and selftest pass. Published via trusted publishing (OIDC, no token).

v0.3.2 — did your agent improve, or did it remember the test?

gauntlet demo — the failure mode, made impossible to miss.

• Three acts, ten seconds, $0, deterministic: an agent "improves" by +1.0 by
  writing the task into its own memory; guard catches the EXACT contamination
  (content-free report), the instance retires, a fresh holdout collapses the
  verdict back to provisional. Scripted agents exercise the real public API.
• The demo asserts its own narrative and runs in CI: if any act deviates,
  the release goes red. (ADR-0010)
• README rebuilt around the demo: hero question, before/after diagram,
  "How an agent can accidentally cheat your eval" (5 concrete paths).
• First release published through trusted publishing (OIDC, no token):
  the pipeline proven in v0.3.0's duplicate-rejection test now ships for real.

---

First release published through PyPI trusted publishing (OIDC) — zero human-handled
tokens, pipeline verified live (uvx --from gauntlet-guard gauntlet demo).
The demo, hero README, diagram, and accidental-cheat section were shaped by unsolicited
feedback from a stranger who found this repo on GitHub — thank you. This is what
measurement culture looks like: decisions written down, claims made runnable.

v0.2.0 — the trial protocol

The trial protocol: gauntlet run.

• run init|prep|exec|close|status|verify|blindpack — tasks x arms x repeats
  into opaque single-use slots. Fixtures copied (never linked) with per-file
  sha verification; symlink-bearing fixture trees refused.
• Environment drift: python + installed-package fingerprints at prep and close;
  any drift is enumerated on the close record and marked sealed-drift.
  (Born from a real incident: a mid-experiment pip install that silently
  invalidated every later trial.)
• Seal + verify: outputs manifest (path,size,sha256) recomputable — post-close
  edits, deletions, additions are detected.
• blindpack: judge-facing artifacts contain pseudonyms only; arm assignments
  live exclusively in a chmod-600 unblind.json on the private side.
• run coordinates any agent command (exit + duration captured); it records
  and proves, it does not judge — verdicts are v0.3 grade.
• 28 tests, CI matrix unchanged (3.11/3.13). PyPI: gauntlet-guard v0.2.0
  pending token refresh (distribution name; import and CLI remain gauntlet).

v0.1.0 — the gauntlet ships

First release: the integrity layer that ships today — guard + manifest.

• gauntlet manifest add|list|retire — sealed instance registry (JSONL):
  full-content SHA-256, normalized text, and 8-word shingle fingerprints.
  Manifests are private-side artifacts; instances retire on leak.
• gauntlet guard scan — persistence-leak scanner for agent stores
  (LESSONS/notes/journals/transcripts). Findings: exact / near (≥5% shingle
  overlap) / trace (≥2 shingles). CI-friendly exit codes; --json output.
• Redaction by design: scanner reports contain hash references, never matched
  text — the report is safe to publish (verified by assertion in tests/CI).
• Operational hardening after first real-world scan: file size caps with
  reported skips, binary sniffing, single-pass shingle computation
  (271 files in 3.1 s, zero runtime dependencies).
• gauntlet guard selftest — end-to-end proof: planted paraphrase caught,
  clean file untouched, oversized + binary files skipped-and-reported,
  report verified content-free.
• Python ≥3.11, MIT, 12 tests, CI matrix (3.11/3.13): pytest + ruff + ty + selftest.

Roadmap: run (blind paired arms, isolated workspaces, drift fingerprint),
grade (keep/revert verdict engine, ≥N repeats + spread), holdout (seed
generation + retirement ledger).