v0.4.0

This section records the user-visible changes since v0.3.0. Internal test, CI
and docs-only commits are omitted unless they change a published surface.

Read the first entry under Changed before you upgrade. One rename in this range
breaks consumers silently. Nothing raises an error and nothing logs a warning.
A program that matched the old value stops matching and reports nothing.

Changed

• The domain glossary is renamed from CONTEXT.md to GLOSSARY.md (#722).
  Pocock v1.3.1 names the domain glossary GLOSSARY.md; a reader following
  any live pointer now lands on that file. Git recognizes the move as a
  rename. The title line names the glossary; the body is byte-identical.
  docs/agents/domain.md is replaced with the v1.3.1
  setup-matt-pocock-skills template, and AGENTS.md names GLOSSARY.md on
  both live instruction lines. Existing CHANGELOG entries that name the old
  path stay as written.

• BREAKING. report_schema_version goes from 1.4.0 to 2.0.0, because the
  aggregation_method value bh_fdr_fallback is renamed to
  bounded_pooling_refused (#360, #441, in 03ee27d). The constant lives at
  src/skill_harness/aggregation/report.py:44. The new value is written at
  src/skill_harness/aggregation/fit.py:759. There is no alias on the write
  path and none on the read path. The old string survives only in comments at
  fit.py:255 and report.py:17.

  What breaks. Any consumer that compares against the literal
  "bh_fdr_fallback" stops matching. The comparison returns false rather than
  raising, so the consumer reports nothing instead of failing. Code that
  branches on the value takes the wrong branch. Code that filters on it returns
  an empty result.

  What to do. Match "bounded_pooling_refused". If you must read both
  generations, branch on report_schema_version rather than accepting either
  string, because the two values report different outcomes. The fit is refused
  and pooled at the admission bound, rather than falling back to BH-FDR. A
  consumer that cannot migrate yet can stay on 0.3.0
  (pip install skill-harness==0.3.0). That release reports
  report_schema_version 1.4.0 and still writes "bh_fdr_fallback".

  docs/PRD.md:757 states the rule this follows. Removals, renames, and type
  changes take a major bump, because they break diff skill consumers.

• The runtime dependency floors rise to the values pyproject.toml declares
  at this PR's head. An environment that pins any package below its floor
  no longer satisfies this package's requirements. Measured from
  git diff v0.3.0 HEAD -- pyproject.toml:

  • anthropic>=1.15.0,<2 (was >=1.2.0; the floor passed through 1.5.0 in
    #548 and 1.11.0 in #727, which re-asserts the post-patch bound for
    GHSA-q5f5-3gjm-7mfm and GHSA-w828-4qhx-vxx3, and landed at 1.15.0 in
    #749). The <2 cap is new (#732). anthropic 2.0.0 is a breaking major
    that no audit has reviewed, so an environment that already resolved
    2.x no longer satisfies this package.
  • openai>=3.31.0,<4 (was >=2.41,<4; the lower bound rose in #604 and
    again in #732. 3.17.0 is the first release that requires the patched
    httpx2>=2.12.0; the floor takes Dependabot's 3.31.0, inside the
    audited 3.x range. The <4 cap is unchanged).
  • click>=8.5.0 (was >=8.1; #606).
  • scipy>=1.18.1 (was >=1.11; #729 is the last of the floor bumps in
    the EB-MoM numerical stack).
  • statsmodels>=0.15.0 (was >=0.14; #605).
  • pydantic>=2.14.1 (was >=2.6; #750).
  • tiktoken>=0.14.0,<1.0 (was >=0.7,<1.0; #751. The <1.0 cap is
    unchanged).

  rich did not move. Dev-extra floors moved with Dependabot and are not
  runtime requirements.

• The ablation runner accepts a whole prompt assembly as an arm, not only a
  clause of one card (#554, #615, in 9d956e8a). RunConfig carries a
  declared arm set serialized into runs.config_json. An assembly may name a
  foreign skill body by path. Duplicate assemblies are refused before any
  spend. Arm samples are written to the append-only evidence.arm_samples
  table (migration 1200), and the A41 reconciler sums them with samples.usd.

• Every external-check refusal reason is persisted (#555, #626, #629,
  in 6b6b74a8). Migration 1100's CHECK constraint listed ten
  UnmeasuredSubReason literals; external_check_missing was never added, and
  INSERT OR IGNORE silently dropped the rows. Migration 1300 rebuilds
  clause_run_outcomes with all eleven members, adds a nullable
  sought_oracle column, and recreates the append-only triggers. Historical
  runs that lost those rows stay lost; new runs record the reason.

• A renamed published skill no longer loses its value_class (#601, #618,
  in d629227e). The registry key closure-mode-at-boundaries still pointed
  at the pre-rename card name, so value_class_for("closure-mode") returned
  None and the skill took the unclassified path instead of
  trap-discipline. The key is now closure-mode, and a staleness test
  fails when any non-retired key stops naming a live skill.

• A clause whose scoring axis reaches no comment or clarity outcome is
  refused before sampling, with a clause-specific reason (#555, #626, in
  6996e68d). The blanket BLOCKER-1 refusal is replaced by an
  external_check_missing sub-reason that names the axis.

• The CLI's rich Console reads COLUMNS at render time, not at import
  (#564, #624, in ddf1373e). A module-level Console() froze the width for
  the life of the process, so terminal resize and
  CliRunner(env={"COLUMNS": "200"}) were inert.

• The published receipt pages stop printing machine tokens as values
  (#583, in 330eca7b). A boolean renders yes or no; a None whose
  schema field names its own null case uses that word (value_class renders
  unclassified); everything else uses the absent class treatment. The
  site also gains page types, a shell with a current-page marker, a 404,
  schema-sourced measurement-table labels, and --base-url so the generator
  stops baking its own host into the output. The landing page sits behind
  --landing (default off) and its one instruction is a command that runs.

• The ablation results table gains a gate-2 (registered floor) column
  (#546, #558, in c2fa804). A clause that previously rendered a green
  PASSED now renders PASSED (scalar only), HARM (registered floor),
  EQUIVALENT (registered floor), or UNRESOLVED (registered floor). The
  verdict words did not change meaning. The report now shows which decision
  produced them. Anyone who parses this table by column position must re-read
  it.

• Tie-bearing ablation clause decisions route through the Gate-2 discordant
  rule (#368, #493, in 0274dd1). A clause whose decision turned on a tie
  can receive a different verdict than it did under 0.3.0.

• paired-gate2 refuses a run that carries undecided epochs (#438, #491,
  in 1a5027a). A hazard entry is counted per simple command rather than per
  tool-call blob. When an arm carries undecided epochs, the command emits a
  HAZARD_UNDECIDED refusal and exits 1. A run that previously produced a
  score now produces a typed refusal instead.

• The PyPI package description is replaced (#497, in 3ba4f69). It now
  carries the repository's GitHub About text.

Added

• skill-harness run pi-paired CONFIG [--execute], a paired subject lane
  for the Pi runtime (src/skill_harness/cli/main.py:3452). The default is a
  dry run that performs the pre-spend gate only. --execute runs the Full and
  Null epochs and spends money. The subcommand requires a pi binary that this
  package does not install. Without it the launcher raises
  PiExecutableNotFoundError
  (src/skill_harness/subject/pi/launcher.py:164). The runtime version is
  measured from the binary that ran and is never assumed. No sized Pi
  measurement has been run, so no Pi skill effect is admissible yet.

• SERS 1.4.0 and SERS 1.5.0. 1.4.0 splits the subject and extractor model
  roles and adds subject_identity.subject_model, required only under the
  1.4.0 conditional (#487, in c398cca). 1.5.0 adds four optional top-level
  keys from the On-Irreducibility additions (#526, #547, in cfb5cec). Both
  are additive for existing receipts. A consumer holding an older copy of the
  schema rejects a new receipt, which the standard treats as intended
  generational non-comparability.

• SERS 1.6.0. Separates the verdict from its present validity (#643, #654,
  in c879633, finished in #655/#660, in 282a5064). A receipt reader must
  handle three new optional top-level objects that are absent on pre-1.6.0
  receipts: verdict_scope (required on a 1.6.0 KEEP, with model_id,
  task_family and delivery_mechanism), currentness (VALIDATED requires
  basis FULL_VALIDATION; CARRIED_FORWARD requires verdict_scope.tested_at
  and renders a carry-forward disclaimer; STALE names a newer metric version
  or hash mismatch), and drift_policy. Receipts from different
  sers_version values are not comparable.

• SERS 1.7.0. Adds the online-FDR ledger facts and the claim-level ladder
  to the receipt (#644, #686, in bd3467c; ladder fields from #647, #671, in
  3f8d4a90). A receipt reader must handle two changes. First, a 1.7.0 KEEP
  carries a required multiplicity object (LORDdep procedure, target FDR
  0.05, anytime_valid_p, wealth before/after, batch order) and a
  verdict_scope.control_world_result that must equal pass; a 1.7.0 CUT
  must not carry multiplicity at all. Second, KEEP receipts gain
  designer, designer_independent, family_replication, model_replication
  and claim_level (KEEP / REPLICATED / ROBUST), with the ladder thresholds
  enforced on write.

• The claim-level ladder (#647, #671, in 3f8d4a90). A family record
  carries five mechanism fields and registration checks...

v0.3.0

See CHANGELOG.md § 0.3.0 for the full record.

The first release since 2026-08-15, carrying 194 commits from main. The reason to cut it: the package PyPI served as 0.2.3 exits 1 on a cp1252 Windows console when it runs the README's own documented command; the fix has been on main since 2026-08-31 (#321, #322).

Why a minor bump: the SERS receipt schema moved from 1.0.0 to 1.3.0 in this range (a wire-format change for anything that reads receipts), and scripts/release_gate.py defines the 0.3 line as "the assurance phase is closed and its lane has run green" (G7, G8); both pass at this version. No commit in the range carries a BREAKING CHANGE: marker.

Highlights: Windows cp1252 crash fixed (#322); [sitegen] extra so a core install imports cleanly (#425); category-nested screen profile (#380); stale-pin refusal on screen verdicts (#398); aggregation refuses malformed evidence (#231, #232, #375); SERS 1.1.0 to 1.3.0 (#300, #392, #424); Gate-2 matched-evidence bridge (#246, #247, #263, #396, #445); detector v2 (#393); assurance phase closed with ten registered detectors (#174, #353 to #369); release gate G7/G8 (#206); README states the spend rule per command and quotes the verbatim 0.3.0 audit output.

v0.2.3

See CHANGELOG.md § 0.2.3 for the full record.

Highlights: independently observable vacuity flag / falsifying case (#136), Tier-1 axis registry with fail-closed classifier (#117), guarded oracle-verdict mint (#81), task-frontier phase firewall tracer (#90), mechanical ratification preflight gate (#57), OC engine Gate-1/Gate-2/frontier (#54/#55/#56), drift-check CI (#53), pi_c invocation detection (#52), estimand registry (#51), paired-path freeze branch + both-PASS-tie fix.

v0.2.2 — surface lockstep + release gate

Surface-consistency fix release, cut same-day as 0.2.1. No functional changes
to measurement, storage, or verdicts, and no change to the measured record:
still zero measured KEEPs; the paired Full-vs-Null path remains coded but
unfired.

0.2.1 — the first PyPI release — shipped with internally inconsistent
surfaces: the wheel's skill_harness.__version__ said 0.2.0 while
skill-harness --version said 0.2.1; the PyPI project description (frozen at
upload time) shows a v0.2.0 status banner, a pip-from-git install path, a
broken banner image, and relative links that 404. A PyPI description can only
be corrected by a new upload — that is this release.

Fixed

• skill_harness.__version__ now matches the released version (the 0.2.1
  wheel carried 0.2.0), and the package docstring drops the retired v0.1
  "clause-ablation" framing.
• README: current status banner, pip install skill-harness as the install
  path, and absolute image/link URLs so the PyPI-rendered description
  renders instead of 404ing.
• SECURITY.md no longer hardcodes a version in its supported-versions
  statement.
• publish.yml: all five actions pinned to full commit SHAs (matching the
  repo's existing CI convention), workflow-level read-only permissions.

Added

• Release gate (scripts/release_gate.py), run on every CI push/PR and
  as the first job of the publish pipeline. It checks version lockstep
  (pyproject vs __version__), a rolled CHANGELOG entry, the README status
  banner, README PyPI-render safety, workflow SHA pins, and tag/version
  match. The build and upload jobs depend on it, so a tag cut on stale
  surfaces is blocked by default (not impossible — the workflow itself is
  editable, and the gate cannot reach surfaces outside the repo). Run
  against the 0.2.1 tree, the gate fails with 23 findings; this release is
  its first live pass.

v0.2.1 — First PyPI release: offline audit, deterministic audit ordering, launch-surface corrections

First release distributed on PyPI: pip install skill-harness. No change to
measured results — still zero measured KEEPs; the paired Full-vs-Null path
remains coded but unfired. This is a packaging, reliability, and
honesty-of-surfaces release.

Added

• PyPI trusted publishing — OIDC release workflow (publish.yml); installs
  no longer require pip-from-GitHub.
• Per-sample setup= hook on build_paired_tasks (#18) — bash setup runs
  in the sandbox before the agent, same script to both arms; guarded against
  NUL and path-shaped values; serialized into the .eval log so ingest
  provenance covers it.
• FAQ (#13) — conceptual Q&A on UNMEASURED, zero measured KEEPs, KEEP vs
  CUT, and why with-vs-without; folded into the docs map.

Fixed

• skill audit is now truly offline (#21) — lazy tiktoken load; runs on a
  cold cache with no network.
• Deterministic ordering on append-only audit queries (#19, #20) —
  calibration_events queries tie-broke on SQLite's hidden rowid, so the
  "newest calibration" could silently flip across a DB restore/repack. All
  timestamp-final ORDER BYs now carry a unique-PK secondary key; regression
  tests proven RED at pre-fix code; structural ban (pre-commit hook + CI step)
  prevents reintroduction.
• harness_version stamp (#14) — derived from __version__ instead of a
  stale hardcoded literal; source-tree-run reports no longer carry the wrong
  version.

Docs & metadata

• Launch-surface fact corrections aligning README/CHANGELOG/docs with the
  registered record (#21): paired k=8 shakedown vs "never fired", 26/26 Null
  epochs across 6 tasks, real audit output, KEEP / CUT / CAN'T-TELL-YET verdict
  vocabulary, dead reference cleanup.
• Stale "pre-alpha" labels version-anchored across README, SECURITY,
  CONTRIBUTING, classifiers (#12); honest caveats unchanged.
• Discovery metadata: PyPI keywords, [project.urls], category-led
  description (#16); CHANGELOG binary-name correction (#17).
• PRD amendment note: §12 demotions and §18 CLI lock predate shipped v0.2 (#22).

Full changelog: v0.2.0...v0.2.1

v0.2.0

skill-harness v0.2.0

Screen a skill and the harness now gives you a decision — KEEP, CUT, or CAN'T-TELL-YET — instead of a raw pass rate. When the evidence can't tell whether the skill or the model did the work, it says UNMEASURED rather than guess a number.

(a) "Every task is run twice: with the skill and without it." → today's screens run the no-skill arm first and only escalate to the paired run when the model fails without help; (b) "a handful of ~26" → same epoch/verdict correction as this PR; (c) "an interrupted run needs redoing" for llm-judge-calibration → per the pre-registration the screen completed 3/3; what's pending is store backfill (per-trial assembly across logs from the credit outage), not a rerun.

What's new

• The keep/cut decision. A screen result becomes a plain call: keep the skill, cut it (the model already does the task, or the skill didn't help when measured), or not enough evidence yet. The thresholds are set before the run, not adjusted afterward to fit the result.
• An append-only record of every screen. Results are written once and never edited; pass rates are recomputed from the raw runs, not cached. Invalid runs are kept with a reason and left out of the totals.
• The batch-1 screens now trace to that record instead of prose write-ups. All came back CUT — the model already handled the task. (One, llm-judge-calibration, is still pending; an interrupted run needs redoing.)

Where it stands

• No skill has scored KEEP yet. The code that would award one is written and tested, but it only runs when a skill's no-skill baseline comes back below a perfect score — and so far, every skill screened, the model already passes without it.
• Two skills have run end to end, both CUT: append-only-evidence-design and a hardened git-pull-rebase-trap. The model passed every run without the skill, so there was nothing to add.
• The LLM judge stays off until it's calibrated against human ratings. It isn't yet, so anything relying on it returns UNMEASURED.
• Most screens aren't in the record yet — a handful of ~26 are; the rest are being moved over from write-ups.

The plan behind the v0.1 → v0.2 change was written down first, and it's public: docs/findings/v0.2-reaim-gate.md. This is also the tool behind the evidence in MrBinnacle/skills.

v0.2.0a0

Added

• v0.2 subject layer (skill_harness.subject, optional [inspect] extra):
  HarnessPin.capture() — the subject-harness configuration captured from the live
  environment as an admissibility field (refuses version="auto"), and
  build_paired_tasks() — the Full-vs-Null contrast as two Inspect tasks identical
  except for the skill under test (inspect_swe.claude_code(skills=…)), with
  file_contains / command_succeeds outcome oracles resolved against the agent's
  pinned cwd. Supply-chain review: docs/supply-chain/inspect-audit-2026-07-09.md.
• skill-harness skill audit <SKILL.md> — fully offline preflight (no API key, no DB,
  no cost): structural lint against Anthropic's published authoring spec plus an
  evaluability report stating which axes a paid run could mechanically measure and
  which claims would return UNMEASURED. --strict exits 1 on warnings for CI.

Fixed

• Installed copies of v0.1 were broken (worked only from a source checkout);
  caught by a fresh-venv install test:
  • hedge_index loaded its wordlist from tests/… (not shipped in the wheel) —
    now package data at src/skill_harness/oracles/tier1/fixtures/.
  • SQL migrations resolved from the repo root — now package data at
    src/skill_harness/storage/migrations_sql/; installed copies can bootstrap DBs.
  • tiktoken and statsmodels were imported at runtime but declared only in dev
    extras — now runtime dependencies.
• examples/README.md still claimed the extractor had no OpenRouter fallback
  (stale since b5b9fe6).
• Repaired all references left dangling by the 2026-07-08 privacy scrub
  (COUNCIL_FINDINGS.md, root CLAUDE.md) across CONTRIBUTING, CODEOWNERS, PR/issue
  templates, CHANGELOG, PRD, PLAN, release notes, and the case study.
• Hostile-review hardening pass on the unreleased v0.2 subject, aggregation,
  and Tier-1 oracle layers: 39 confirmed findings fixed across the
  ablation / aggregation / calibration / extractor / CLI / storage paths —
  direction-aware ablation wins, BH-FDR gate consumption, scorer-crash
  inadmissibility, resume cost attribution, sentinel-verdict exclusion from
  statistics, and single-source pricing/tokenizer. The four remaining
  test-quality findings are tracked, not silently dropped.
• Citation oracle no longer arm-differentially deflates hyperlink references:
  lowercase prose is no longer miscounted as a flag emission (flag matching is
  now case-sensitive), and an external http(s) markdown-link target is
  preserved as a citation rather than stripped away with its link text.

Changed

• README rewritten for its actual audience (Claude Code users with a skills folder):
  value proposition first, two-command keyless quickstart, honest pre-alpha status,
  fact-checked comparison table (skill-eval-harness, promptfoo, Inspect).
• PRD.md, PLAN.md, RELEASE-NOTES-v0.1.md moved under docs/.
• docs/findings/v0.2-reaim-gate.md amended after the 2026-07-09 competitive sweep:
  adds "Harness pin" and "Differentiation vs field" pre-registration fields and a
  correction block recording that the original lock predated the sweep.

Security

• Untrusted skill names and model output are escaped with rich.markup.escape
  and asserted NUL/control-free before terminal rendering, so bracketed text
  cannot inject Rich markup into CLI output.
• CI workflow actions are pinned by commit SHA, and structural enforcement bans
  (e.g. raw sqlite3.connect) are wired as tests rather than left as convention.