Skip to content

v0.10.0 — standards re-grade gap closure (5 live gaps closed)

Choose a tag to compare

@momomuchu momomuchu released this 05 Jul 11:36

⚠️ Correction — see v0.10.1. An independent adversarial re-grade confirmed all 5 fixes are real (revert→red→substance), but found the verify-forgery fix bypassable via MNM_DONE_DIR (redirect the re-execution at a clean decoy dir) and MNM_SKIP_REVERIFY. The claim below that "a DONE typed on broken code is refused" is therefore overstated for v0.10.0. v0.10.1 removes MNM_DONE_DIR (re-execution now runs in the project cwd, fail-closed on redirect via cwd-relative anchors), makes MNM_SKIP_REVERIFY loud, and corrects the claim to name what is not closed in-band.


v0.10.0 — standards re-grade gap closure

An independent, adversarial standards re-grade (ISO/IEC 25010, ISO/IEC/IEEE 29119/12207/1012, SWEBOK v4 — docs/evals/standards/) named five concrete, independently-reproduced live gaps where the harness's own DONE or its evidence could fail to mean what it says. All five are now closed on substance, each locked by a negative control that goes red without the fix.

Closed

  1. Coverage measured the whole environment, not the project--source=. scopes the coverage gate, so unrelated installed packages can no longer block a fully-tested repo.
  2. Integrity anchor set excluded the enforcement layerfreeze_dod now anchors the hooks, every SKILL.md, and agents/*.md; tampering a hook trips INTEGRITY-COMPROMISED (negative control: tamper a hook → exit 4).
  3. A caught cheat froze the whole repoGAMING-DETECTED is now scoped to the cheating task_id (a cheat on task A no longer blocks task B's honest DONE); INTEGRITY-COMPROMISED stays repo-wide. Enforced identically in verdict_state.py, log_verdict.sh, and the self-contained .githooks/pre-commit, locked to agree.
  4. False STUCK-OSCILLATING — the convergence detector is scoped to one task and counts distinct passing gate names (excluding the verify re-run), killing a double-count that flagged converging runs as thrashing.
  5. Verify-forgery (the #1 gap) — a DONE now re-executes the deterministic gate stack at write time instead of trusting logged rows, so a DONE typed on broken code is refused when the real gates fail.

Honest residual (named, not gamed)

Verify-forgery's dangerous case — a false DONE on broken code — is closed. The residual: an agent whose code genuinely passes the gates can still type a verify row to skip the adversarial panel. Closing that fully is not an in-band property without violating the author≠verifier independence the harness exists to enforce; the real roots are the ed25519-signed manifest, the independent mnm-verify sub-agent, and CI/pre-commit re-running the gates in a clean environment.

Metric honesty (Campbell / Goodhart)

The re-grade's strict figure — 30 MET / 74 PARTIAL / 23 NOT-MET / 2 N/A = 23.3% strict across 129 items — is treated as a directional signal, not truth. The denominator mixes incommensurable standards, and several PARTIALs are one honest step from MET. 80% strict MET is not honestly reachable without N/A-gaming the denominator; this release moves the number by fixing substance, not by redefining the metric.

Verification

  • 546 unit tests (incl. property-based + cross-task regression + a live-re-execution lock), 0 failed
  • 16 self-eval scenarios (incl. verify-forgery-broken-code), 0 failed
  • Both CI workflows green (unit suite + coverage; signed-manifest integrity + independence-guard lints)