v0.5.1 — raw-model A/B workflow

Highlights

Run a real raw model vs harness-on A/B comparison. In v0.5.0, ja-output-toggle off only stopped the Stop hook — ~/.codex/AGENTS.md still shipped the quality-gate block, so the "baseline" was contaminated. This release closes that loop.

Added

• ja-output-toggle off --full / on --full: evicts / restores the AGENTS.md managed block to AGENTS.md.bak-toggle. Handles the stale-backup case (installer re-ran and re-inserted the block) without overwriting the original backup. status now also surfaces the block presence and .bak-toggle state.
• ja-output-stats scan-sessions: walks $CODEX_HOME/sessions/YYYY/MM/DD/rollout-*.jsonl, extracts role=assistant Japanese turns, lints via rules.py, and prints ok-rate + Wilson 95% CI. Flags: --since / --until (date-only --until is end-of-day so --since D --until D covers the whole UTC day; untimestamped rows are dropped when a filter is active), --include-archived, --config, --output-jsonl.
• ja-output-stats ab-report --source-path PATH: read an ad-hoc jsonl instead of the live lite / metrics stream. Wires scan-sessions --output-jsonl straight into the A/B comparison.

Comparison flow

ja-output-toggle off --full                         # raw model, restart Codex
# ...use Codex...
ja-output-stats scan-sessions \
  --since 2026-04-24T19:00 --until 2026-04-24T19:30 \
  --output-jsonl raw.jsonl

ja-output-toggle on --full                          # harness back, restart Codex
# ...use Codex same length...

ja-output-stats ab-report \
  --baseline 2026-04-24:2026-04-24 \
  --test     2026-04-24:2026-04-24 \
  --source-path raw.jsonl

Review

6 rounds of codex review --uncommitted (see PR #83). Final verdict: no actionable correctness issues.

Fixed

• off --full now removes a re-inserted managed block even when .bak-toggle already exists (was previously a no-op).

Tests

• 245 passing (30 new cases across test_toggle.py, test_stats_scan_sessions.py, test_stats_ab_report.py)

Full changelog: v0.5.0...v0.5.1