docs: run-record schema — duplication metrics and per-run budget ceiling#27
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Additive, optional extension of the benchmark run-record schema (
schema_versionstays1, same$id, per the documented versioning policy):metrics.duplication(optional object): production near-duplicate accounting for the run's diff --base_pairs,final_pairs,delta,new_pairs,removed_pairs, plus thethreshold/min_tokensparameters used. A pair is excluded iff both endpoints' files are undertests/;delta = final - baseis signed, so deduplication shows up as a negative value.max_budget_usd(optional top-level number >= 0): the effective per-run budget ceiling -- the per-task override when one is configured, otherwise the study default.docs/benchmark/run-record.mdis updated in lockstep: field-table rows, the example record (now carries aduplicationobject andmax_budget_usd), and a versioning note for the Phase 3 additive extension.Why
This is the recording end of the ctx-bench Phase 3 duplication endpoint. In the pilot,
ctx score'snew_duplicationfloors at zero: it can only count duplication added, never duplication removed, which made the refactor/dedup task (t2) structurally unmeasurable -- an agent that eliminated existing near-duplicates scored identically to one that did nothing. A signedduplication.deltamakes dedup work visible and rewardable. Scoping pairs to production code (excluding test-internal pairs) keeps the metric from being dominated by idiomatic test-fixture boilerplate, andmax_budget_usdrecords the budget ceiling each run actually ran under so cost/timeout analysis doesn't have to reconstruct it from study config.The production-scoping rationale (and a request for first-class support in
ctx duplicates/ctx scoreitself) is tracked in #26. This PR only extends the record schema; existing writers and readers are unaffected.Verification
Drift guard is green:
🤖 Generated with Claude Code