Skip to content

docs: run-record schema — duplication metrics and per-run budget ceiling#27

Merged
saldestechnology merged 1 commit into
mainfrom
bench/run-record-duplication
Jul 11, 2026
Merged

docs: run-record schema — duplication metrics and per-run budget ceiling#27
saldestechnology merged 1 commit into
mainfrom
bench/run-record-duplication

Conversation

@saldestechnology

Copy link
Copy Markdown
Collaborator

What

Additive, optional extension of the benchmark run-record schema (schema_version stays 1, same $id, per the documented versioning policy):

  • metrics.duplication (optional object): production near-duplicate accounting for the run's diff -- base_pairs, final_pairs, delta, new_pairs, removed_pairs, plus the threshold / min_tokens parameters used. A pair is excluded iff both endpoints' files are under tests/; delta = final - base is signed, so deduplication shows up as a negative value.
  • max_budget_usd (optional top-level number >= 0): the effective per-run budget ceiling -- the per-task override when one is configured, otherwise the study default.

docs/benchmark/run-record.md is updated in lockstep: field-table rows, the example record (now carries a duplication object and max_budget_usd), and a versioning note for the Phase 3 additive extension.

Why

This is the recording end of the ctx-bench Phase 3 duplication endpoint. In the pilot, ctx score's new_duplication floors at zero: it can only count duplication added, never duplication removed, which made the refactor/dedup task (t2) structurally unmeasurable -- an agent that eliminated existing near-duplicates scored identically to one that did nothing. A signed duplication.delta makes dedup work visible and rewardable. Scoping pairs to production code (excluding test-internal pairs) keeps the metric from being dominated by idiomatic test-fixture boilerplate, and max_budget_usd records the budget ceiling each run actually ran under so cost/timeout analysis doesn't have to reconstruct it from study config.

The production-scoping rationale (and a request for first-class support in ctx duplicates / ctx score itself) is tracked in #26. This PR only extends the record schema; existing writers and readers are unaffected.

Verification

Drift guard is green:

cargo test --all-features --test study_scripts
test run_record_schema_and_example_stay_in_sync ... ok
test result: ok. 4 passed; 0 failed

🤖 Generated with Claude Code

@saldestechnology
saldestechnology merged commit b0f35d1 into main Jul 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant