Repository navigation
v0.2.0a0
Pre-release
Pre-release
Added
- v0.2 subject layer (
skill_harness.subject, optional[inspect]extra):
HarnessPin.capture()— the subject-harness configuration captured from the live
environment as an admissibility field (refusesversion="auto"), and
build_paired_tasks()— the Full-vs-Null contrast as two Inspect tasks identical
except for the skill under test (inspect_swe.claude_code(skills=…)), with
file_contains/command_succeedsoutcome oracles resolved against the agent's
pinned cwd. Supply-chain review:docs/supply-chain/inspect-audit-2026-07-09.md. skill-harness skill audit <SKILL.md>— fully offline preflight (no API key, no DB,
no cost): structural lint against Anthropic's published authoring spec plus an
evaluability report stating which axes a paid run could mechanically measure and
which claims would return UNMEASURED.--strictexits 1 on warnings for CI.
Fixed
- Installed copies of v0.1 were broken (worked only from a source checkout);
caught by a fresh-venv install test:hedge_indexloaded its wordlist fromtests/…(not shipped in the wheel) —
now package data atsrc/skill_harness/oracles/tier1/fixtures/.- SQL migrations resolved from the repo root — now package data at
src/skill_harness/storage/migrations_sql/; installed copies can bootstrap DBs. tiktokenandstatsmodelswere imported at runtime but declared only in dev
extras — now runtime dependencies.
examples/README.mdstill claimed the extractor had no OpenRouter fallback
(stale sinceb5b9fe6).- Repaired all references left dangling by the 2026-07-08 privacy scrub
(COUNCIL_FINDINGS.md, root CLAUDE.md) across CONTRIBUTING, CODEOWNERS, PR/issue
templates, CHANGELOG, PRD, PLAN, release notes, and the case study. - Hostile-review hardening pass on the unreleased v0.2 subject, aggregation,
and Tier-1 oracle layers: 39 confirmed findings fixed across the
ablation / aggregation / calibration / extractor / CLI / storage paths —
direction-aware ablation wins, BH-FDR gate consumption, scorer-crash
inadmissibility, resume cost attribution, sentinel-verdict exclusion from
statistics, and single-source pricing/tokenizer. The four remaining
test-quality findings are tracked, not silently dropped. - Citation oracle no longer arm-differentially deflates hyperlink references:
lowercase prose is no longer miscounted as a flag emission (flag matching is
now case-sensitive), and an externalhttp(s)markdown-link target is
preserved as a citation rather than stripped away with its link text.
Changed
- README rewritten for its actual audience (Claude Code users with a skills folder):
value proposition first, two-command keyless quickstart, honest pre-alpha status,
fact-checked comparison table (skill-eval-harness, promptfoo, Inspect). PRD.md,PLAN.md,RELEASE-NOTES-v0.1.mdmoved underdocs/.docs/findings/v0.2-reaim-gate.mdamended after the 2026-07-09 competitive sweep:
adds "Harness pin" and "Differentiation vs field" pre-registration fields and a
correction block recording that the original lock predated the sweep.
Security
- Untrusted skill names and model output are escaped with
rich.markup.escape
and asserted NUL/control-free before terminal rendering, so bracketed text
cannot inject Rich markup into CLI output. - CI workflow actions are pinned by commit SHA, and structural enforcement bans
(e.g. rawsqlite3.connect) are wired as tests rather than left as convention.