Skip to content

feat: add Step 0 benchmark measurement foundation - #9

Merged
Lee-take merged 3 commits into
mainfrom
codex/ds-agent-step-0
Jul 16, 2026
Merged

feat: add Step 0 benchmark measurement foundation#9
Lee-take merged 3 commits into
mainfrom
codex/ds-agent-step-0

Conversation

@Lee-take

Copy link
Copy Markdown
Owner

What changed

  • C0B adds the versioned benchmark task/run contracts, deterministic A/Q/H/S/F classification, verifier/evidence completion gate, VOCR, guardrail metrics, and focused tests.
  • C0C adds deterministic synthetic T1 fixtures, per-fact provenance, and six fail-closed verifiers.
  • C0D adds the offline five-run T1 runner, independent disk readback, negative tamper coverage, and machine/human Step 0 baseline reports.

Why

Step 0 needs a repeatable, evidence-bound measurement foundation before later DS Agent product work begins. Generating files alone is not treated as completion.

Scope and impact

This PR contains exactly C0B, C0C, and C0D on top of 344081caa7c002dbbb5c6b37cd3f73103628aa2f. It is internal benchmark and measurement infrastructure only. It adds no new user-facing capability, connector, model call, production Office path, UI, installer, tag, or Release.

Commits:

  • e1d6056be38c422822698a8214ab4e819c5f8dcd — benchmark contracts and metrics
  • 9d37c1f168cd2621a5506a17feb3b9bd2057851a — deterministic T1 benchmark verification
  • cfc4662de6b5debd66118d4f58eb6fb905096925 — offline T1 benchmark baseline

Validation

  • Focused benchmark: 30 passed, 0 failed.
  • Full npx pnpm@9.15.9 test: Rust 877 passed, 0 failed, 7 ignored; frontend and Node/UI/build checks passed.
  • npx pnpm@9.15.9 test:release-source: passed, 207 files.
  • cargo fmt --manifest-path apps/desktop/src-tauri/Cargo.toml -- --check: passed.
  • node scripts/secret-scan.mjs: passed, 207 files and 5 self-tests.
  • git diff --check and exact changed-file/source-only review: passed.
  • Fresh explicit-root readback: 5 independent runs, 30/30 verifier PASS, A/Q/H/S/F = 5/0/0/0/0, VOCR 100%, false completion 0, completion completed.

Committed report integrity:

  • JSON SHA-256: bd09049a9cec2a47511aa881b6db175a9b1d3529f1ea108a962a49330a8484da
  • Markdown SHA-256: 555d3aa2ac8ee47d3415a81c1147c996a30ebbbd22606ddbfb1b972bec1bd59a
  • Canonical payload SHA-256: 941f78a6159028fd5ce7af3f6633d7e86bc99802afdf7fa7964f7112249a8674
  • Markdown body SHA-256: 5b91b7e55e86294de3c8226acafd81217b089d576c739181a88351153dd4cc6f

@Lee-take
Lee-take merged commit fe361a5 into main Jul 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant