Publish the study the README's claim actually rests on

README.ja.md

@@ -337,40 +337,55 @@ path の履歴は `commitlore context <path>` で読みます。より小さな
 <!-- Generated by `node bench/report.ts --section` from the result logs named below. Do not edit by hand:
      CI regenerates this block and fails if a single byte differs (scripts/check-readme-numbers.mjs). -->
 
-**112 runs recorded.** No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone.
+**1160 measurements across 1240 rows.** 80 row(s) are superseded by a re-run of the same task, arm and seed, and the analysis counts the survivor. No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone.
 
 | Where it comes from | |
 |---|---|
-| Results | `bench/results/t702-m4-final.jsonl` (112 rows) |
-| Run id | `20260727T120103Z-aa5eab`, `20260728T025523Z-db4659`, `20260728T025635Z-e3d669`, `20260728T025817Z-d8d0dc` |
+| Results | `bench/results/m5-seeds-1-10-rerun.jsonl` (200 rows), `bench/results/m5-seeds-11-20-rerun.jsonl` (200 rows), `bench/results/m5-seeds-21-30.jsonl` (200 rows), `bench/results/m5-seeds-31-40.jsonl` (200 rows), `bench/results/m5-seeds-41-50.jsonl` (200 rows), `bench/results/m5-seeds-51-58.jsonl` (160 rows), `bench/results/m5-seeds-55-58-rerun.jsonl` (80 rows) |
+| Run id | `20260802T124657Z-ae3ba0`, `20260802T230855Z-00da79`, `20260803T100356Z-aeb38a`, `20260803T203631Z-77df15`, `20260806T230824Z-60e31e`, `20260807T095937Z-bf2b05`, `20260807T234037Z-6dd0a2` |
 | Driver | `claude-headless` |
-| Model | not recorded |
-| Matrix | 8 tasks, seeds 1, 2, 3, 4, 5, 6, 7 |
-| Status | final (declared in `bench/report.ts`, pending a manifest field) |
+| Model | `sonnet` |
+| Matrix | 10 tasks, seeds 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58 |
+| Status | final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-1-10-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-11-20-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-21-30.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-31-40.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-41-50.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-51-58.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-55-58-rerun.jsonl`) |
 
 **Re-proposal and violation rates, every recorded run:**
 
 | Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens |
 |---|---|---|---|---|---|---|---|
-| `commitlore-guard` | 56 | 41 | 0.732 | 0 | 0.000 | 14.8 | 18965 |
-| `commitlore-on` | 56 | 35 | 0.625 | 0 | 0.000 | 14.2 | 18091 |
+| `commitlore-off` | 620 | 110 | 0.177 | 58 | 0.094 | 19.3 | 38488 |
+| `commitlore-on` | 620 | 16 | 0.026 | 7 | 0.011 | 16.2 | 39265 |
 
-**Analysis set — all 112 rows.** Nothing was excluded: no simulated rows, no failed runs, no run that never started.
+**Analysis set — 1169 of 1240 rows** (71 excluded: error = 71). A row that failed carries `reproposed: false` because the field is required, not because the agent declined to re-propose; leaving it in the denominator would let the arm that crashed more often look like the arm that behaved better. The excluded runs are counted here, never dropped silently.
 
-**Significance:** not computed — guard exposure is unknown for 112 analysis rows
+| Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens |
+|---|---|---|---|---|---|---|---|
+| `commitlore-off` | 584 | 110 | 0.188 | 58 | 0.099 | 20.4 | 40842 |
+| `commitlore-on` | 585 | 16 | 0.027 | 7 | 0.012 | 17.1 | 41568 |
+
+**Significance:**
+
+| Quantity | Value |
+|---|---|
+| Arms | `commitlore-on` (treatment) vs `commitlore-off` (baseline) |
+| Re-proposed / did not | `commitlore-on` 16/569, `commitlore-off` 110/474 |
+| Fisher exact, two-tailed | p = 1.52e-20 |
+| Rate difference, treatment minus baseline | -16.1pp, 95% CI [-19.6pp, -12.7pp] |
+| Odds ratio | 0.1212 |
+| Paired (task, seed) cells | 579 |
+| Rows excluded from the analysis set | 71 |
 
 **How the runs ended** — failures are reported, not filtered:
 
 | Condition | completed | timeout | over-turns | over-tokens | error |
 |---|---|---|---|---|---|
-| `commitlore-guard` | 56 | 0 | 0 | 0 | 0 |
-| `commitlore-on` | 55 | 0 | 1 | 0 | 0 |
+| `commitlore-off` | 414 | 3 | 157 | 10 | 36 |
+| `commitlore-on` | 459 | 2 | 109 | 15 | 35 |
 
 **Read these numbers with their limits:**
 
-- No model is recorded — neither on the rows nor in a manifest. A re-proposal rate whose model is unknown is not a comparable number, and these figures must not be quoted against another model's.
 - Every rate here is conditional on the model that produced it. Re-proposal is a behaviour, and behaviours differ between models, so these figures are not evidence about any other model.
-- 112 runs in the analysis set: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md).
+- 585 and 584 runs per arm: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md).
+- Fisher exact treats the runs as independent while the design is paired by (task, seed). It is the pre-registered result, but it is not a valid paired-data test. See the correction in [`docs/VERDICT-M4.md`](docs/VERDICT-M4.md).
 <!-- BENCH:END -->
 
 </details>

README.ko.md

@@ -332,40 +332,55 @@ CommitLore-Version: 2.0.0
 <!-- Generated by `node bench/report.ts --section` from the result logs named below. Do not edit by hand:
      CI regenerates this block and fails if a single byte differs (scripts/check-readme-numbers.mjs). -->
 
-**112 runs recorded.** No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone.
+**1160 measurements across 1240 rows.** 80 row(s) are superseded by a re-run of the same task, arm and seed, and the analysis counts the survivor. No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone.
 
 | Where it comes from | |
 |---|---|
-| Results | `bench/results/t702-m4-final.jsonl` (112 rows) |
-| Run id | `20260727T120103Z-aa5eab`, `20260728T025523Z-db4659`, `20260728T025635Z-e3d669`, `20260728T025817Z-d8d0dc` |
+| Results | `bench/results/m5-seeds-1-10-rerun.jsonl` (200 rows), `bench/results/m5-seeds-11-20-rerun.jsonl` (200 rows), `bench/results/m5-seeds-21-30.jsonl` (200 rows), `bench/results/m5-seeds-31-40.jsonl` (200 rows), `bench/results/m5-seeds-41-50.jsonl` (200 rows), `bench/results/m5-seeds-51-58.jsonl` (160 rows), `bench/results/m5-seeds-55-58-rerun.jsonl` (80 rows) |
+| Run id | `20260802T124657Z-ae3ba0`, `20260802T230855Z-00da79`, `20260803T100356Z-aeb38a`, `20260803T203631Z-77df15`, `20260806T230824Z-60e31e`, `20260807T095937Z-bf2b05`, `20260807T234037Z-6dd0a2` |
 | Driver | `claude-headless` |
-| Model | not recorded |
-| Matrix | 8 tasks, seeds 1, 2, 3, 4, 5, 6, 7 |
-| Status | final (declared in `bench/report.ts`, pending a manifest field) |
+| Model | `sonnet` |
+| Matrix | 10 tasks, seeds 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58 |
+| Status | final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-1-10-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-11-20-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-21-30.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-31-40.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-41-50.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-51-58.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-55-58-rerun.jsonl`) |
 
 **Re-proposal and violation rates, every recorded run:**
 
 | Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens |
 |---|---|---|---|---|---|---|---|
-| `commitlore-guard` | 56 | 41 | 0.732 | 0 | 0.000 | 14.8 | 18965 |
-| `commitlore-on` | 56 | 35 | 0.625 | 0 | 0.000 | 14.2 | 18091 |
+| `commitlore-off` | 620 | 110 | 0.177 | 58 | 0.094 | 19.3 | 38488 |
+| `commitlore-on` | 620 | 16 | 0.026 | 7 | 0.011 | 16.2 | 39265 |
 
-**Analysis set — all 112 rows.** Nothing was excluded: no simulated rows, no failed runs, no run that never started.
+**Analysis set — 1169 of 1240 rows** (71 excluded: error = 71). A row that failed carries `reproposed: false` because the field is required, not because the agent declined to re-propose; leaving it in the denominator would let the arm that crashed more often look like the arm that behaved better. The excluded runs are counted here, never dropped silently.
 
-**Significance:** not computed — guard exposure is unknown for 112 analysis rows
+| Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens |
+|---|---|---|---|---|---|---|---|
+| `commitlore-off` | 584 | 110 | 0.188 | 58 | 0.099 | 20.4 | 40842 |
+| `commitlore-on` | 585 | 16 | 0.027 | 7 | 0.012 | 17.1 | 41568 |
+
+**Significance:**
+
+| Quantity | Value |
+|---|---|
+| Arms | `commitlore-on` (treatment) vs `commitlore-off` (baseline) |
+| Re-proposed / did not | `commitlore-on` 16/569, `commitlore-off` 110/474 |
+| Fisher exact, two-tailed | p = 1.52e-20 |
+| Rate difference, treatment minus baseline | -16.1pp, 95% CI [-19.6pp, -12.7pp] |
+| Odds ratio | 0.1212 |
+| Paired (task, seed) cells | 579 |
+| Rows excluded from the analysis set | 71 |
 
 **How the runs ended** — failures are reported, not filtered:
 
 | Condition | completed | timeout | over-turns | over-tokens | error |
 |---|---|---|---|---|---|
-| `commitlore-guard` | 56 | 0 | 0 | 0 | 0 |
-| `commitlore-on` | 55 | 0 | 1 | 0 | 0 |
+| `commitlore-off` | 414 | 3 | 157 | 10 | 36 |
+| `commitlore-on` | 459 | 2 | 109 | 15 | 35 |
 
 **Read these numbers with their limits:**
 
-- No model is recorded — neither on the rows nor in a manifest. A re-proposal rate whose model is unknown is not a comparable number, and these figures must not be quoted against another model's.
 - Every rate here is conditional on the model that produced it. Re-proposal is a behaviour, and behaviours differ between models, so these figures are not evidence about any other model.
-- 112 runs in the analysis set: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md).
+- 585 and 584 runs per arm: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md).
+- Fisher exact treats the runs as independent while the design is paired by (task, seed). It is the pre-registered result, but it is not a valid paired-data test. See the correction in [`docs/VERDICT-M4.md`](docs/VERDICT-M4.md).
 <!-- BENCH:END -->
 
 </details>

README.md

@@ -391,40 +391,55 @@ What is measured — retrieval, exposure, latency and scaling, hook overhead —
 <!-- Generated by `node bench/report.ts --section` from the result logs named below. Do not edit by hand:
      CI regenerates this block and fails if a single byte differs (scripts/check-readme-numbers.mjs). -->
 
-**112 runs recorded.** No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone.
+**1160 measurements across 1240 rows.** 80 row(s) are superseded by a re-run of the same task, arm and seed, and the analysis counts the survivor. No manifest declares how many runs the matrix was meant to produce, so completeness cannot be checked from the logs alone.
 
 | Where it comes from | |
 |---|---|
-| Results | `bench/results/t702-m4-final.jsonl` (112 rows) |
-| Run id | `20260727T120103Z-aa5eab`, `20260728T025523Z-db4659`, `20260728T025635Z-e3d669`, `20260728T025817Z-d8d0dc` |
+| Results | `bench/results/m5-seeds-1-10-rerun.jsonl` (200 rows), `bench/results/m5-seeds-11-20-rerun.jsonl` (200 rows), `bench/results/m5-seeds-21-30.jsonl` (200 rows), `bench/results/m5-seeds-31-40.jsonl` (200 rows), `bench/results/m5-seeds-41-50.jsonl` (200 rows), `bench/results/m5-seeds-51-58.jsonl` (160 rows), `bench/results/m5-seeds-55-58-rerun.jsonl` (80 rows) |
+| Run id | `20260802T124657Z-ae3ba0`, `20260802T230855Z-00da79`, `20260803T100356Z-aeb38a`, `20260803T203631Z-77df15`, `20260806T230824Z-60e31e`, `20260807T095937Z-bf2b05`, `20260807T234037Z-6dd0a2` |
 | Driver | `claude-headless` |
-| Model | not recorded |
-| Matrix | 8 tasks, seeds 1, 2, 3, 4, 5, 6, 7 |
-| Status | final (declared in `bench/report.ts`, pending a manifest field) |
+| Model | `sonnet` |
+| Matrix | 10 tasks, seeds 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58 |
+| Status | final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-1-10-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-11-20-rerun.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-21-30.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-31-40.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-41-50.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-51-58.jsonl`), final (declared in `bench/report.ts`, pending a manifest field) (`bench/results/m5-seeds-55-58-rerun.jsonl`) |
 
 **Re-proposal and violation rates, every recorded run:**
 
 | Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens |
 |---|---|---|---|---|---|---|---|
-| `commitlore-guard` | 56 | 41 | 0.732 | 0 | 0.000 | 14.8 | 18965 |
-| `commitlore-on` | 56 | 35 | 0.625 | 0 | 0.000 | 14.2 | 18091 |
+| `commitlore-off` | 620 | 110 | 0.177 | 58 | 0.094 | 19.3 | 38488 |
+| `commitlore-on` | 620 | 16 | 0.026 | 7 | 0.011 | 16.2 | 39265 |
 
-**Analysis set — all 112 rows.** Nothing was excluded: no simulated rows, no failed runs, no run that never started.
+**Analysis set — 1169 of 1240 rows** (71 excluded: error = 71). A row that failed carries `reproposed: false` because the field is required, not because the agent declined to re-propose; leaving it in the denominator would let the arm that crashed more often look like the arm that behaved better. The excluded runs are counted here, never dropped silently.
 
-**Significance:** not computed — guard exposure is unknown for 112 analysis rows
+| Condition | n | Re-proposed | Re-proposal rate | Runs with violations | Violation rate | Mean turns | Mean tokens |
+|---|---|---|---|---|---|---|---|
+| `commitlore-off` | 584 | 110 | 0.188 | 58 | 0.099 | 20.4 | 40842 |
+| `commitlore-on` | 585 | 16 | 0.027 | 7 | 0.012 | 17.1 | 41568 |
+
+**Significance:**
+
+| Quantity | Value |
+|---|---|
+| Arms | `commitlore-on` (treatment) vs `commitlore-off` (baseline) |
+| Re-proposed / did not | `commitlore-on` 16/569, `commitlore-off` 110/474 |
+| Fisher exact, two-tailed | p = 1.52e-20 |
+| Rate difference, treatment minus baseline | -16.1pp, 95% CI [-19.6pp, -12.7pp] |
+| Odds ratio | 0.1212 |
+| Paired (task, seed) cells | 579 |
+| Rows excluded from the analysis set | 71 |
 
 **How the runs ended** — failures are reported, not filtered:
 
 | Condition | completed | timeout | over-turns | over-tokens | error |
 |---|---|---|---|---|---|
-| `commitlore-guard` | 56 | 0 | 0 | 0 | 0 |
-| `commitlore-on` | 55 | 0 | 1 | 0 | 0 |
+| `commitlore-off` | 414 | 3 | 157 | 10 | 36 |
+| `commitlore-on` | 459 | 2 | 109 | 15 | 35 |
 
 **Read these numbers with their limits:**
 
-- No model is recorded — neither on the rows nor in a manifest. A re-proposal rate whose model is unknown is not a comparable number, and these figures must not be quoted against another model's.
 - Every rate here is conditional on the model that produced it. Re-proposal is a behaviour, and behaviours differ between models, so these figures are not evidence about any other model.
-- 112 runs in the analysis set: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md).
+- 585 and 584 runs per arm: this matrix is only powered to detect a large effect, so a non-significant result from it is a statement about the sample size, not about CommitLore. The exact power table is in [`bench/README.md`](bench/README.md).
+- Fisher exact treats the runs as independent while the design is paired by (task, seed). It is the pre-registered result, but it is not a valid paired-data test. See the correction in [`docs/VERDICT-M4.md`](docs/VERDICT-M4.md).
 <!-- BENCH:END -->
 
 </details>