Skip to content

v0.6.2 — standards eval + honest corrections

Choose a tag to compare

@momomuchu momomuchu released this 03 Jul 18:52

A standards-anchored evaluation (ISO/IEC 25010:2023, ISO/IEC/IEEE 29119/12207/1012/29148, SWEBOK v4 — docs/evals/standards/) graded the product honestly: not a complete, professionally-tested product — an unusually honest v0 with real strengths and real, named gaps. Its anti-rubber-stamp critic found concrete defects; all were verified by hand and fixed:

  • Hash-chain honesty (the key correction). The verdict.log hash chain is tamper-EVIDENT (catches accidental corruption + any append outside log_verdict.shINTEGRITY-COMPROMISED) but NOT forgery-proof — it is an unkeyed SHA-256 over public scripts, so an adversary who reproduces the formula can forge a valid DONE (shown in ~15 lines). Earlier "DONE can't be forged" wording overstated it and is corrected. Real forgery-proofing needs an out-of-band trust root the agent can't reach — the deferred R13 signed manifest + external CI (a local HMAC key would be theater: a shell agent can read any key the harness can).
  • pre-commit consistency (real bug): its inlined logic had drifted from verdict_state.py and wrongly blocked a legitimate red→green DONE. Now identical latest-iteration rule.
  • All 6 terminal states now reachable: STUCK-OSCILLATING / STUCK-INCONCLUSIVE were defined but never emitted; run_gates now emits both (an all-skipped run is no longer a false green). Proven by new scenarios — self-eval suite 11 → 13 green.
  • Re-freeze in the same repo (mode-444 lock) fixed; R1 sole-writer CI lint added (was claimed, didn't exist); CONTRIBUTING de-staled.

Still open, now clearly named: no unit tests / no coverage number for the harness's own 15 scripts (biggest test-adequacy gap); verifier independence is architectural, not mechanically enforced; DONE needs gate evidence but not a logged verifier-CONFIRMED row; forgery-proofing awaits R13.

Full assessment: docs/evals/standards/COMPLETENESS-VERDICT.md.