diff --git a/README.ja.md b/README.ja.md index 6d04daf..985b068 100644 --- a/README.ja.md +++ b/README.ja.md @@ -337,40 +337,55 @@ path の履歴は `commitlore context ` で読みます。より小さな -**112 runs recorded.** No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone. +**1160 measurements across 1240 rows.** 80 row(s) are superseded by a re-run of the same task, arm and seed, and the analysis counts the survivor. No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone. | Where it comes from | | |---|---| -| Results | `bench/results/t702-m4-final.jsonl` (112 rows) | -| Run id | `20260727T120103Z-aa5eab`, `20260728T025523Z-db4659`, `20260728T025635Z-e3d669`, `20260728T025817Z-d8d0dc` | +| Results | `bench/results/m5-seeds-1-10-rerun.jsonl` (200 rows), `bench/results/m5-seeds-11-20-rerun.jsonl` (200 rows), `bench/results/m5-seeds-21-30.jsonl` (200 rows), `bench/results/m5-seeds-31-40.jsonl` (200 rows), `bench/results/m5-seeds-41-50.jsonl` (200 rows), `bench/results/m5-seeds-51-58.jsonl` (160 rows), `bench/results/m5-seeds-55-58-rerun.jsonl` (80 rows) | +| Run id | `20260802T124657Z-ae3ba0`, `20260802T230855Z-00da79`, `20260803T100356Z-aeb38a`, `20260803T203631Z-77df15`, `20260806T230824Z-60e31e`, `20260807T095937Z-bf2b05`, `20260807T234037Z-6dd0a2` | | Driver | `claude-headless` | -| Model | not recorded | -| Matrix | 8 tasks, seeds 1, 2, 3, 4, 5, 6, 7 | -| Status | final (declared in `bench/report.ts`, pending a manifest field) | +| Model | `sonnet` | +| Matrix | 10 tasks, seeds 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58 | +| Status | final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-1-10-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-11-20-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-21-30.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-31-40.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-41-50.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-51-58.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-55-58-rerun.jsonl`) | **Re-proposal and violation rates, every recorded run:** | Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens | |---|---|---|---|---|---|---|---| -| `commitlore-guard` | 56 | 41 | 0.732 | 0 | 0.000 | 14.8 | 18965 | -| `commitlore-on` | 56 | 35 | 0.625 | 0 | 0.000 | 14.2 | 18091 | +| `commitlore-off` | 620 | 110 | 0.177 | 58 | 0.094 | 19.3 | 38488 | +| `commitlore-on` | 620 | 16 | 0.026 | 7 | 0.011 | 16.2 | 39265 | -**Analysis set — all 112 rows.** Nothing was excluded: no simulated rows, no failed runs, no run that never started. +**Analysis set — 1169 of 1240 rows** (71 excluded: error = 71). A row that failed carries `reproposed: false` because the field is required, not because the agent declined to re-propose; leaving it in the denominator would let the arm that crashed more often look like the arm that behaved better. The excluded runs are counted here, never dropped silently. -**Significance:** not computed — guard exposure is unknown for 112 analysis rows +| Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens | +|---|---|---|---|---|---|---|---| +| `commitlore-off` | 584 | 110 | 0.188 | 58 | 0.099 | 20.4 | 40842 | +| `commitlore-on` | 585 | 16 | 0.027 | 7 | 0.012 | 17.1 | 41568 | + +**Significance:** + +| Quantity | Value | +|---|---| +| Arms | `commitlore-on` (treatment) vs `commitlore-off` (baseline) | +| Re-proposed / did not | `commitlore-on` 16/569, `commitlore-off` 110/474 | +| Fisher exact, two-tailed | p = 1.52e-20 | +| Rate difference, treatment minus baseline | -16.1pp, 95% CI [-19.6pp, -12.7pp] | +| Odds ratio | 0.1212 | +| Paired (task, seed) cells | 579 | +| Rows excluded from the analysis set | 71 | **How the runs ended** — failures are reported, not filtered: | Condition | completed | timeout | over-turns | over-tokens | error | |---|---|---|---|---|---| -| `commitlore-guard` | 56 | 0 | 0 | 0 | 0 | -| `commitlore-on` | 55 | 0 | 1 | 0 | 0 | +| `commitlore-off` | 414 | 3 | 157 | 10 | 36 | +| `commitlore-on` | 459 | 2 | 109 | 15 | 35 | **Read these numbers with their limits:** -- No model is recorded — neither on the rows nor in a manifest. A re-proposal rate whose model is unknown is not a comparable number, and these figures must not be quoted against another model's. - Every rate here is conditional on the model that produced it. Re-proposal is a behaviour, and behaviours differ between models, so these figures are not evidence about any other model. -- 112 runs in the analysis set: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md). +- 585 and 584 runs per arm: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md). +- Fisher exact treats the runs as independent while the design is paired by (task, seed). It is the pre-registered result, but it is not a valid paired-data test. See the correction in [`docs/VERDICT-M4.md`](docs/VERDICT-M4.md). diff --git a/README.ko.md b/README.ko.md index dc153a5..b9e8c11 100644 --- a/README.ko.md +++ b/README.ko.md @@ -332,40 +332,55 @@ CommitLore-Version: 2.0.0 -**112 runs recorded.** No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone. +**1160 measurements across 1240 rows.** 80 row(s) are superseded by a re-run of the same task, arm and seed, and the analysis counts the survivor. No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone. | Where it comes from | | |---|---| -| Results | `bench/results/t702-m4-final.jsonl` (112 rows) | -| Run id | `20260727T120103Z-aa5eab`, `20260728T025523Z-db4659`, `20260728T025635Z-e3d669`, `20260728T025817Z-d8d0dc` | +| Results | `bench/results/m5-seeds-1-10-rerun.jsonl` (200 rows), `bench/results/m5-seeds-11-20-rerun.jsonl` (200 rows), `bench/results/m5-seeds-21-30.jsonl` (200 rows), `bench/results/m5-seeds-31-40.jsonl` (200 rows), `bench/results/m5-seeds-41-50.jsonl` (200 rows), `bench/results/m5-seeds-51-58.jsonl` (160 rows), `bench/results/m5-seeds-55-58-rerun.jsonl` (80 rows) | +| Run id | `20260802T124657Z-ae3ba0`, `20260802T230855Z-00da79`, `20260803T100356Z-aeb38a`, `20260803T203631Z-77df15`, `20260806T230824Z-60e31e`, `20260807T095937Z-bf2b05`, `20260807T234037Z-6dd0a2` | | Driver | `claude-headless` | -| Model | not recorded | -| Matrix | 8 tasks, seeds 1, 2, 3, 4, 5, 6, 7 | -| Status | final (declared in `bench/report.ts`, pending a manifest field) | +| Model | `sonnet` | +| Matrix | 10 tasks, seeds 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58 | +| Status | final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-1-10-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-11-20-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-21-30.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-31-40.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-41-50.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-51-58.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-55-58-rerun.jsonl`) | **Re-proposal and violation rates, every recorded run:** | Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens | |---|---|---|---|---|---|---|---| -| `commitlore-guard` | 56 | 41 | 0.732 | 0 | 0.000 | 14.8 | 18965 | -| `commitlore-on` | 56 | 35 | 0.625 | 0 | 0.000 | 14.2 | 18091 | +| `commitlore-off` | 620 | 110 | 0.177 | 58 | 0.094 | 19.3 | 38488 | +| `commitlore-on` | 620 | 16 | 0.026 | 7 | 0.011 | 16.2 | 39265 | -**Analysis set — all 112 rows.** Nothing was excluded: no simulated rows, no failed runs, no run that never started. +**Analysis set — 1169 of 1240 rows** (71 excluded: error = 71). A row that failed carries `reproposed: false` because the field is required, not because the agent declined to re-propose; leaving it in the denominator would let the arm that crashed more often look like the arm that behaved better. The excluded runs are counted here, never dropped silently. -**Significance:** not computed — guard exposure is unknown for 112 analysis rows +| Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens | +|---|---|---|---|---|---|---|---| +| `commitlore-off` | 584 | 110 | 0.188 | 58 | 0.099 | 20.4 | 40842 | +| `commitlore-on` | 585 | 16 | 0.027 | 7 | 0.012 | 17.1 | 41568 | + +**Significance:** + +| Quantity | Value | +|---|---| +| Arms | `commitlore-on` (treatment) vs `commitlore-off` (baseline) | +| Re-proposed / did not | `commitlore-on` 16/569, `commitlore-off` 110/474 | +| Fisher exact, two-tailed | p = 1.52e-20 | +| Rate difference, treatment minus baseline | -16.1pp, 95% CI [-19.6pp, -12.7pp] | +| Odds ratio | 0.1212 | +| Paired (task, seed) cells | 579 | +| Rows excluded from the analysis set | 71 | **How the runs ended** — failures are reported, not filtered: | Condition | completed | timeout | over-turns | over-tokens | error | |---|---|---|---|---|---| -| `commitlore-guard` | 56 | 0 | 0 | 0 | 0 | -| `commitlore-on` | 55 | 0 | 1 | 0 | 0 | +| `commitlore-off` | 414 | 3 | 157 | 10 | 36 | +| `commitlore-on` | 459 | 2 | 109 | 15 | 35 | **Read these numbers with their limits:** -- No model is recorded — neither on the rows nor in a manifest. A re-proposal rate whose model is unknown is not a comparable number, and these figures must not be quoted against another model's. - Every rate here is conditional on the model that produced it. Re-proposal is a behaviour, and behaviours differ between models, so these figures are not evidence about any other model. -- 112 runs in the analysis set: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md). +- 585 and 584 runs per arm: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md). +- Fisher exact treats the runs as independent while the design is paired by (task, seed). It is the pre-registered result, but it is not a valid paired-data test. See the correction in [`docs/VERDICT-M4.md`](docs/VERDICT-M4.md). diff --git a/README.md b/README.md index 5dda84b..95eb59a 100644 --- a/README.md +++ b/README.md @@ -391,40 +391,55 @@ What is measured — retrieval, exposure, latency and scaling, hook overhead — -**112 runs recorded.** No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone. +**1160 measurements across 1240 rows.** 80 row(s) are superseded by a re-run of the same task, arm and seed, and the analysis counts the survivor. No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone. | Where it comes from | | |---|---| -| Results | `bench/results/t702-m4-final.jsonl` (112 rows) | -| Run id | `20260727T120103Z-aa5eab`, `20260728T025523Z-db4659`, `20260728T025635Z-e3d669`, `20260728T025817Z-d8d0dc` | +| Results | `bench/results/m5-seeds-1-10-rerun.jsonl` (200 rows), `bench/results/m5-seeds-11-20-rerun.jsonl` (200 rows), `bench/results/m5-seeds-21-30.jsonl` (200 rows), `bench/results/m5-seeds-31-40.jsonl` (200 rows), `bench/results/m5-seeds-41-50.jsonl` (200 rows), `bench/results/m5-seeds-51-58.jsonl` (160 rows), `bench/results/m5-seeds-55-58-rerun.jsonl` (80 rows) | +| Run id | `20260802T124657Z-ae3ba0`, `20260802T230855Z-00da79`, `20260803T100356Z-aeb38a`, `20260803T203631Z-77df15`, `20260806T230824Z-60e31e`, `20260807T095937Z-bf2b05`, `20260807T234037Z-6dd0a2` | | Driver | `claude-headless` | -| Model | not recorded | -| Matrix | 8 tasks, seeds 1, 2, 3, 4, 5, 6, 7 | -| Status | final (declared in `bench/report.ts`, pending a manifest field) | +| Model | `sonnet` | +| Matrix | 10 tasks, seeds 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58 | +| Status | final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-1-10-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-11-20-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-21-30.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-31-40.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-41-50.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-51-58.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-55-58-rerun.jsonl`) | **Re-proposal and violation rates, every recorded run:** | Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens | |---|---|---|---|---|---|---|---| -| `commitlore-guard` | 56 | 41 | 0.732 | 0 | 0.000 | 14.8 | 18965 | -| `commitlore-on` | 56 | 35 | 0.625 | 0 | 0.000 | 14.2 | 18091 | +| `commitlore-off` | 620 | 110 | 0.177 | 58 | 0.094 | 19.3 | 38488 | +| `commitlore-on` | 620 | 16 | 0.026 | 7 | 0.011 | 16.2 | 39265 | -**Analysis set — all 112 rows.** Nothing was excluded: no simulated rows, no failed runs, no run that never started. +**Analysis set — 1169 of 1240 rows** (71 excluded: error = 71). A row that failed carries `reproposed: false` because the field is required, not because the agent declined to re-propose; leaving it in the denominator would let the arm that crashed more often look like the arm that behaved better. The excluded runs are counted here, never dropped silently. -**Significance:** not computed — guard exposure is unknown for 112 analysis rows +| Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens | +|---|---|---|---|---|---|---|---| +| `commitlore-off` | 584 | 110 | 0.188 | 58 | 0.099 | 20.4 | 40842 | +| `commitlore-on` | 585 | 16 | 0.027 | 7 | 0.012 | 17.1 | 41568 | + +**Significance:** + +| Quantity | Value | +|---|---| +| Arms | `commitlore-on` (treatment) vs `commitlore-off` (baseline) | +| Re-proposed / did not | `commitlore-on` 16/569, `commitlore-off` 110/474 | +| Fisher exact, two-tailed | p = 1.52e-20 | +| Rate difference, treatment minus baseline | -16.1pp, 95% CI [-19.6pp, -12.7pp] | +| Odds ratio | 0.1212 | +| Paired (task, seed) cells | 579 | +| Rows excluded from the analysis set | 71 | **How the runs ended** — failures are reported, not filtered: | Condition | completed | timeout | over-turns | over-tokens | error | |---|---|---|---|---|---| -| `commitlore-guard` | 56 | 0 | 0 | 0 | 0 | -| `commitlore-on` | 55 | 0 | 1 | 0 | 0 | +| `commitlore-off` | 414 | 3 | 157 | 10 | 36 | +| `commitlore-on` | 459 | 2 | 109 | 15 | 35 | **Read these numbers with their limits:** -- No model is recorded — neither on the rows nor in a manifest. A re-proposal rate whose model is unknown is not a comparable number, and these figures must not be quoted against another model's. - Every rate here is conditional on the model that produced it. Re-proposal is a behaviour, and behaviours differ between models, so these figures are not evidence about any other model. -- 112 runs in the analysis set: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md). +- 585 and 584 runs per arm: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md). +- Fisher exact treats the runs as independent while the design is paired by (task, seed). It is the pre-registered result, but it is not a valid paired-data test. See the correction in [`docs/VERDICT-M4.md`](docs/VERDICT-M4.md). diff --git a/README.zh-CN.md b/README.zh-CN.md index 780ea7c..8ed24b2 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -326,40 +326,55 @@ CommitLore-Version: 2.0.0 -**112 runs recorded.** No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone. +**1160 measurements across 1240 rows.** 80 row(s) are superseded by a re-run of the same task, arm and seed, and the analysis counts the survivor. No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone. | Where it comes from | | |---|---| -| Results | `bench/results/t702-m4-final.jsonl` (112 rows) | -| Run id | `20260727T120103Z-aa5eab`, `20260728T025523Z-db4659`, `20260728T025635Z-e3d669`, `20260728T025817Z-d8d0dc` | +| Results | `bench/results/m5-seeds-1-10-rerun.jsonl` (200 rows), `bench/results/m5-seeds-11-20-rerun.jsonl` (200 rows), `bench/results/m5-seeds-21-30.jsonl` (200 rows), `bench/results/m5-seeds-31-40.jsonl` (200 rows), `bench/results/m5-seeds-41-50.jsonl` (200 rows), `bench/results/m5-seeds-51-58.jsonl` (160 rows), `bench/results/m5-seeds-55-58-rerun.jsonl` (80 rows) | +| Run id | `20260802T124657Z-ae3ba0`, `20260802T230855Z-00da79`, `20260803T100356Z-aeb38a`, `20260803T203631Z-77df15`, `20260806T230824Z-60e31e`, `20260807T095937Z-bf2b05`, `20260807T234037Z-6dd0a2` | | Driver | `claude-headless` | -| Model | not recorded | -| Matrix | 8 tasks, seeds 1, 2, 3, 4, 5, 6, 7 | -| Status | final (declared in `bench/report.ts`, pending a manifest field) | +| Model | `sonnet` | +| Matrix | 10 tasks, seeds 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58 | +| Status | final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-1-10-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-11-20-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-21-30.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-31-40.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-41-50.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-51-58.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-55-58-rerun.jsonl`) | **Re-proposal and violation rates, every recorded run:** | Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens | |---|---|---|---|---|---|---|---| -| `commitlore-guard` | 56 | 41 | 0.732 | 0 | 0.000 | 14.8 | 18965 | -| `commitlore-on` | 56 | 35 | 0.625 | 0 | 0.000 | 14.2 | 18091 | +| `commitlore-off` | 620 | 110 | 0.177 | 58 | 0.094 | 19.3 | 38488 | +| `commitlore-on` | 620 | 16 | 0.026 | 7 | 0.011 | 16.2 | 39265 | -**Analysis set — all 112 rows.** Nothing was excluded: no simulated rows, no failed runs, no run that never started. +**Analysis set — 1169 of 1240 rows** (71 excluded: error = 71). A row that failed carries `reproposed: false` because the field is required, not because the agent declined to re-propose; leaving it in the denominator would let the arm that crashed more often look like the arm that behaved better. The excluded runs are counted here, never dropped silently. -**Significance:** not computed — guard exposure is unknown for 112 analysis rows +| Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens | +|---|---|---|---|---|---|---|---| +| `commitlore-off` | 584 | 110 | 0.188 | 58 | 0.099 | 20.4 | 40842 | +| `commitlore-on` | 585 | 16 | 0.027 | 7 | 0.012 | 17.1 | 41568 | + +**Significance:** + +| Quantity | Value | +|---|---| +| Arms | `commitlore-on` (treatment) vs `commitlore-off` (baseline) | +| Re-proposed / did not | `commitlore-on` 16/569, `commitlore-off` 110/474 | +| Fisher exact, two-tailed | p = 1.52e-20 | +| Rate difference, treatment minus baseline | -16.1pp, 95% CI [-19.6pp, -12.7pp] | +| Odds ratio | 0.1212 | +| Paired (task, seed) cells | 579 | +| Rows excluded from the analysis set | 71 | **How the runs ended** — failures are reported, not filtered: | Condition | completed | timeout | over-turns | over-tokens | error | |---|---|---|---|---|---| -| `commitlore-guard` | 56 | 0 | 0 | 0 | 0 | -| `commitlore-on` | 55 | 0 | 1 | 0 | 0 | +| `commitlore-off` | 414 | 3 | 157 | 10 | 36 | +| `commitlore-on` | 459 | 2 | 109 | 15 | 35 | **Read these numbers with their limits:** -- No model is recorded — neither on the rows nor in a manifest. A re-proposal rate whose model is unknown is not a comparable number, and these figures must not be quoted against another model's. - Every rate here is conditional on the model that produced it. Re-proposal is a behaviour, and behaviours differ between models, so these figures are not evidence about any other model. -- 112 runs in the analysis set: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md). +- 585 and 584 runs per arm: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md). +- Fisher exact treats the runs as independent while the design is paired by (task, seed). It is the pre-registered result, but it is not a valid paired-data test. See the correction in [`docs/VERDICT-M4.md`](docs/VERDICT-M4.md). diff --git a/bench/report.ts b/bench/report.ts index 5114107..3445ab9 100644 --- a/bench/report.ts +++ b/bench/report.ts @@ -77,23 +77,50 @@ export interface DeclaredSource { */ export const README_SOURCES: readonly DeclaredSource[] = [ { - file: "bench/results/t702-m4-final.jsonl", + file: "bench/results/m5-seeds-1-10-rerun.jsonl", status: "final", status_note: - "The registered M4 measurement (PREREGISTRATION.md §16), the qualification-gated matrix whose primary " + - "comparison is `commitlore-guard` against `commitlore-on`. Run from an isolated checkout against frozen " + - "code `081d858c1`, under the environment controls of §5-b; every row carries a uniform `harness_commit` " + - "and `dist_digest`. bench/VERDICT-M4.md is historical; docs/VERDICT-M4.md is the correction. M1 " + - "(bench/VERDICT-M1.md), M1-b " + - "(bench/VERDICT-M1b.md) and M2 (bench/VERDICT-M2.md) are not revised and are not pooled here — a " + - "different task set and a third arm make them a different matrix, the same reason the ablation log below " + - "stays out of this list. M3 is void (§15). No manifest exists for this file: `runner.ts` does not write " + - "`model` onto a row, and none was written for this run either, so the model behind these 112 rows is " + - "unrecorded rather than declared (bench/README.md, \"Open after T-702\", item 1). Each file's own " + - "manifest is the authority on its status; this declaration is the fallback.", + "The registered M5 measurement (bench/PREREGISTRATION-M5.md), whose primary comparison is `commitlore-on` against `commitlore-off` on the re-proposal task set. Seven shards, 1,160 rows, one harness commit `788a9db3` and one dist digest `f54cda4795cc` across every one. Three shards are re-runs (deviations 3 and 4): 400 rows were lost to a temp reaper and re-produced, and seeds 55-58 were re-run whole after 70 of their surviving rows carried no measurement. 80 original cells are superseded by a rule in bench/m5-analysis.ts -- an original is dropped whenever a re-run shard holds the same task, condition and seed -- rather than by a trimmed file nobody can review. One row carrying `stopped_by: error` is excluded by the registered rule. The arms truncate unequally, 28.5% control against 21.2% treatment, which deviations 1 and 2 require the verdict to report because truncation suppresses re-proposal and so shrinks rather than manufactures the gap. Every record in the run rendered `[claim]`: no arm passed --trusted-author, so this measures the weaker of the two trust tiers and not the `[directive]` path 0.7.1 made reachable (#415). bench/VERDICT-M5.md is the verdict. M1 (bench/VERDICT-M1.md), M1-b, M2 and M4 (docs/VERDICT-M4.md) are not pooled here -- different task sets and, for M4, a third arm, the same reason the ablation log stays out of this list. Each file's own manifest is the authority on its status; this declaration is the fallback.", + }, + { + file: "bench/results/m5-seeds-11-20-rerun.jsonl", + status: "final", + status_note: + "Part of the registered M5 set declared above; see the first entry for the full status.", + }, + { + file: "bench/results/m5-seeds-21-30.jsonl", + status: "final", + status_note: + "Part of the registered M5 set declared above; see the first entry for the full status.", + }, + { + file: "bench/results/m5-seeds-31-40.jsonl", + status: "final", + status_note: + "Part of the registered M5 set declared above; see the first entry for the full status.", + }, + { + file: "bench/results/m5-seeds-41-50.jsonl", + status: "final", + status_note: + "Part of the registered M5 set declared above; see the first entry for the full status.", + }, + { + file: "bench/results/m5-seeds-51-58.jsonl", + status: "final", + status_note: + "Part of the registered M5 set declared above; see the first entry for the full status.", + }, + { + file: "bench/results/m5-seeds-55-58-rerun.jsonl", + status: "final", + status_note: + "Part of the registered M5 set declared above; see the first entry for the full status.", }, ]; + /* * The ablation log is deliberately NOT listed above. * @@ -346,21 +373,40 @@ export interface Progress { readonly recorded: number; /** Total planned runs across the manifests that declare a plan, or null. */ readonly planned: number | null; + /** + * Rows left after a re-run of the same cell supersedes the original. + * + * A study that re-runs a shard has more rows on disk than measurements, and + * a line reporting the file count invites the reader to take the larger + * number for the study's size. M5 has 1,240 rows across seven shards and + * 1,160 cells, because 400 rows were re-produced after a temp reaper and one + * seed range was re-run whole. The supersession rule lives in + * `bench/m5-analysis.ts`; this counts the same way so the published line and + * the registered analysis cannot disagree. + */ + readonly distinct: number; } +/** One measurement: a task, an arm and a seed. Re-runs repeat it. */ +const cellKey = (row: Record): string => + `${String(row['task'])}\u0000${String(row['cond'])}\u0000${String(row['seed'])}`; + export const progressOf = (sources: Sources): Progress => { const recorded = sources.rows.length; - if (recorded === 0) return { state: "no-measurement", recorded, planned: null }; + const distinct = new Set( + sources.rows.map((row) => cellKey(row as unknown as Record)), + ).size; + if (recorded === 0) return { state: "no-measurement", recorded, planned: null, distinct }; const planned = sources.present.reduce((total, source) => { if (total === null || source.plannedRuns === null) return null; return total + source.plannedRuns; }, 0); - if (planned === null) return { state: "unknown-plan", recorded, planned: null }; - if (recorded < planned) return { state: "in-progress", recorded, planned }; - if (recorded > planned) return { state: "over-planned", recorded, planned }; - return { state: "measured", recorded, planned }; + if (planned === null) return { state: "unknown-plan", recorded, planned: null, distinct }; + if (recorded < planned) return { state: "in-progress", recorded, planned, distinct }; + if (recorded > planned) return { state: "over-planned", recorded, planned, distinct }; + return { state: "measured", recorded, planned, distinct }; }; export interface ModelReport { @@ -545,7 +591,12 @@ const statusLine = (progress: Progress, sources: Sources): string => { } if (progress.state === "unknown-plan") { return ( - `**${progress.recorded} runs recorded.** No manifest declares how many runs the matrix was meant to produce, ` + + (progress.distinct === progress.recorded + ? `**${progress.recorded} runs recorded.** ` + : `**${progress.distinct} measurements across ${progress.recorded} rows.** ` + + `${progress.recorded - progress.distinct} row(s) are superseded by a re-run of the same task, ` + + "arm and seed, and the analysis counts the survivor. ") + + "No manifest declares how many runs the matrix was meant to produce, " + "so completeness cannot be checked from the logs alone." ); } diff --git a/test/bench-report.test.ts b/test/bench-report.test.ts index e465380..2869aaa 100644 --- a/test/bench-report.test.ts +++ b/test/bench-report.test.ts @@ -32,8 +32,7 @@ import { buildReport, declarationFor, readSources, - renderSection, -} from '../bench/report.ts'; + renderSection, progressOf } from '../bench/report.ts'; const REPO_ROOT = path.resolve(import.meta.dirname, '..'); const FIXTURES = 'test/fixtures/bench'; @@ -281,7 +280,15 @@ describe('a partial matrix is labelled as one', () => { it('says how many of the planned runs are recorded', () => { const markdown = report(PARTIAL); expect(markdown).toContain('**Measurement in progress — 12 of 24 planned runs recorded.**'); - expect(json(PARTIAL).progress).toEqual({ state: 'in-progress', recorded: 12, planned: 24 }); + // `distinct` counts task-arm-seed cells rather than rows, so a study that + // re-ran a shard does not publish its file count as its size. Equal here + // because nothing in this fixture is a re-run. + expect(json(PARTIAL).progress).toEqual({ + state: 'in-progress', + recorded: 12, + planned: 24, + distinct: 12, + }); }); it('calls a finished matrix complete', () => { @@ -512,3 +519,28 @@ describe('the CLI refuses what it cannot answer honestly', () => { expect(runReport().status).toBe(2); }); }); + +describe('a re-run shard does not inflate the published size', () => { + it('counts cells rather than rows, and names the difference', () => { + // M5 has 1,240 rows across seven shards and 1,160 measurements, because + // 400 rows were re-produced after a temp reaper and one seed range was + // re-run whole. Reporting the row count invites a reader to take the + // larger number for the study's size, and the registered analysis in + // bench/m5-analysis.ts already drops the superseded original. + const base = { task: 't1', cond: 'commitlore-on', seed: 1 }; + const rows = [ + { ...base, run_id: 'first' }, + { ...base, run_id: 'rerun' }, + { task: 't1', cond: 'commitlore-off', seed: 1, run_id: 'first' }, + ] as unknown as Parameters[0]['rows']; + + const progress = progressOf({ + rows, + present: [], + missing: [], + } as unknown as Parameters[0]); + + expect(progress.recorded).toBe(3); + expect(progress.distinct).toBe(2); + }); +});