Skip to content

v0.7.0 — Blind controlled studies, absorbed reporting standards, de-AI tone pass

Latest

Choose a tag to compare

@asimfish asimfish released this 01 Sep 20:34
· 18 commits to main since this release

Highlights

Evidence: two blind controlled studies published

  • evals/runs/controlled/codex-20260825/ and codex-20260826/ — each 28 held-out cases × 2 conditions × 3 fresh-context seeds = 168 real Codex executions, 84 randomized A/B pairs blind-rated by two independent non-OpenAI model raters, ratings frozen before unblinding, paired bootstrap (10,000 resamples) against preregistered gates.
  • Second run passes the primary quality gate for the first time: composite +0.315, 95% CI [0.110, 0.579] (threshold +0.3); framework ahead on 6/7 dimensions, concision at parity, win rate 33.3% → 45.2%, zero framework responses leaking a local path (45 of 46 critical-error records on the baseline side).
  • Efficiency gates still fail structurally: 7.9x median output-token overhead (p90 26.7x) against a 1.15x/1.3x budget; semantic density per 1,000 tokens 1.9 vs 19.9. The controller keeps effectiveness_claim_eligible: false and the claim status insufficient_evidence. No readability or efficiency gain is claimed. Both READMEs' Evidence status sections now report this instead of only the one-case pilot.

Contract fixes motivated by the first study

  • The final response must contain the report itself — never a saved-file path, pointer, or scratch path — in every host adapter, the micro-contract, SKILL.md, and the core contract.
  • Proportional ceremony: short single-session answers skip checkpoints, draft files, and script audits; the file-backed bookend is reserved for long, multi-session, multi-agent, or durable-artifact work.

Absorbed reporting standards (all mappings in docs/REPORTING-STANDARDS.md)

  • BLUF (AR 25-50), SBAR, Minto pyramid, US plain-language / ISO 24495-1, NN/g scanning research, WCAG 2.2 heading structure, Google SRE postmortems, IBCS SUCCESS / ISO 24896:2026.
  • A two-round scientific-reporting literature survey: ASA p-value discipline, effect sizes with intervals, causal-language ladder, CONSORT/PRISMA accounting, uncertainty-visualization and error-bar semantics, benchmarking-crimes countermeasures.
  • De-AI tone capability distilled from the MIT-licensed shuorenhua skill: a fidelity contract (tone edits never change facts), CN/EN signal-to-action table, misfire protection for technical vocabulary.

New surface area

  • 3 display modules: ablation, benchmarking, natural-tone (5 → 8).
  • 4 template assets: sbar-handoff, executive-onepager, rebuttal-response, release-card (8 → 12), each with an audit-clean finished example and receipts under examples/templates-2026082{6,8}/.
  • 7 deterministic audit warnings: generic-heading, heading-level-skip, long-sentence (CJK-aware), deep-list-nesting, cjk-halfwidth-punctuation, ai-tone-boilerplate, plus the chart-first slide contract; all silent on known-good fixtures and showcase reports.
  • Tightened slide surface contract from showcase feedback: 3+ data points must be charted with the takeaway encoded inside the visual.

Repository

  • READMEs restructured after ARIS open-source conventions (numbered contents, dated What's New with PR links, collapsible release history, acknowledgements, star history).
  • Evidence-first GitHub Pages landing page with the showcase deck PDF.
  • reportctl --version, CITATION.cff, and the skill card are back in sync at 0.7.0.

Verification

  • 309/309 unit tests (Python 3.9), check_test_env preflight clean, benchmark harness smoke, and dist reproducibility all green at 2ade309.
  • Protocol and template content are unchanged since PR #5; this release adds only the version bump, the Evidence status rewrite, and the changelog close.

Full detail: CHANGELOG.md