docs(bench): phase-8 pre-registration + ABC audit + operator manual (#117)

bench/ABC_CHECKLIST.md

@@ -0,0 +1,57 @@
+# ABC checklist — benchmark-rigor self-audit
+
+A self-audit against the **Agentic Benchmark Checklist** (ABC) from *Establishing Best Practices for
+Building Rigorous Agentic Benchmarks* (Zhu et al., NeurIPS 2025 D&B; arXiv **2507.02825**). The ABC
+groups failure modes into **task validity** (can the task be solved as specified?), **outcome
+validity** (does the grader actually measure success — no shortcuts, no false pass/fail?), and
+**reporting**. The paper found *all 10* benchmarks it audited had reporting gaps; this file is our
+attempt not to be one of them. Item ids follow the paper's grouping.
+
+Legend: ✅ satisfied · ⚠️ partial / by-design limitation · n/a not applicable to this benchmark.
+
+## Task validity
+
+| Item | How this benchmark addresses it |
+|---|---|
+| **T.1** tool/version specificity | Model `model_id`s pinned in `config.toml`, copied into `run.json`; `surf --version` recorded. Python env pinned by `uv.lock`. ✅ |
+| **T.2/T.3** API availability / interruption handling | Per-request 120 s timeout + bounded retries; a cell that still fails is logged with `error` and **excluded** from rates (not counted as a failure). ✅ |
+| **T.4** clean state between tasks | Each grade runs in a **fresh temp workspace** (`grade_code`) / each multi-turn trial in a **fresh sandbox** (`tools_runtime.scenario_sandbox`). No cross-task leakage. ✅ |
+| **T.5** isolate agent from ground truth | The drifted dependency is withheld from the prompt (`hidden_paths`); graders compute ground truth separately by probing the real dependency. The agent never sees the grader. ✅ |
+| **T.6** freeze environment at release | Scenario set frozen + git-tagged at pre-registration (see `PREREGISTRATION.md` freeze gate). ✅ |
+| **T.7/T.8** verify ground-truth annotations | Ground truth is **derived from the real code**, not hand-annotated: code graders import and probe the hidden dependency; QA rubrics encode the T1 behaviour the committed code implements. ✅ |
+| **T.9** oracle/reference solution | Every scenario ships `.author/solution_correct.*` and `solution_stale.*`; `tools/validate_scenario.py` runs the live graders on both and proves they discriminate, offline, before any spend. ✅ |
+| **T.10** inspect pilot outliers | Staged rollout (`mock` → per-provider smoke → small-N pilot → full) with the oracle as a tripwire; the v1 pilot's outlier (a leaked stale value) was caught this way (#113). ✅ |
+
+## Outcome validity
+
+| Item | How this benchmark addresses it |
+|---|---|
+| **O.a** semantic-equivalence / redundant-word robustness | QA grading parses a **fixed `VERDICT:` field format** (regex per field) rather than free-text matching; "last match wins" tolerates the model restating the format. Code grading runs real tests, so wording is irrelevant. ✅ |
+| **O.b.2/O.b.3** no trivial success (list-all / empty answer) | Code: an empty or non-compiling answer fails the hidden test. QA: a missing/garbled `VERDICT` parses as neither correct nor misled (not a pass). ✅ |
+| **O.d.1** manually verify test cases | Each grader's correct/misled tests are validated by `validate_scenario.py` against the reference solutions; the **misled** test must pass on the stale solution and fail on the correct one (and vice-versa). ✅ |
+| **O.f.2** eliminate non-determinism in tests | Graders are deterministic (no clocks/network/randomness in the checked logic; fixed probe inputs). ✅ |
+| **O.h.1** specify output format | `task.md` states the exact contract: a `FILE: <path>` block (code) or a `VERDICT: …` line (QA). ✅ |
+| **O.h.2** tasks resistant to guessing | The reason multi-turn is the centerpiece: in single-shot a hidden value is only knowable from the doc, so "follow the stale doc" is partly rational. Multi-turn lets the agent *verify*, removing the tautology. QA verdicts use **two fields** to cut a 50/50 guess. ⚠️ (single-shot cascade is guess-resistant only because the value is genuinely unknowable without the doc — stated as a limitation) |
+| **O.i.1** metric correlates with the construct | `verified_then_correct` validates the verification metric (reading the truth should rescue the answer); the misled metric directly encodes "asserted the stale claim". ✅ |
+
+## Reporting
+
+| Item | How this benchmark addresses it |
+|---|---|
+| **R.1/R.2** open data + eval code | The entire harness, scenarios, graders, and reference solutions are in this repo. `raw.jsonl` is preserved per run. ✅ |
+| **R.3/R.4** contamination | Scenarios are bespoke fixtures, not scraped; the drift values live only in `hub_stale.md`. (Frontier models may have seen similar idioms — noted as residual risk.) ⚠️ |
+| **R.5/R.6** state the capability assessed | `PREREGISTRATION.md` §9 states exactly what is and isn't assessed. ✅ |
+| **R.7** mitigation for unavoidable flaws | The neutral system prompt + neutral `task.md` rules (#113 lesson) mitigate doc-trust bias; the oracle mitigates fixture defects. ✅ |
+| **R.8/R.9** quantify limitation impact | Per-scenario and per-tier breakdowns expose any single-fixture or single-difficulty artifact. ✅ |
+| **R.10** statistical significance | Wilson intervals, bootstrap CIs + p-values, **Holm–Bonferroni** across the confirmatory family. ✅ |
+| **R.11** interpretation guidance | `README.md` §4 explains what each metric means; `PREREGISTRATION.md` separates confirmatory from exploratory. ✅ |
+| **R.12/R.13** baselines/controls | **C0** (no-doc baseline) and **Cw** (warning-only control) isolate, respectively, the value of accurate prose and whether recovery is Surface's *fix* vs generic suspicion. ✅ |
+
+## Known limitations (disclosed, per R.7)
+
+- **Curated fixtures, not real repositories** (external validity); a real-OSS case study is deferred
+  (#109).
+- **Read-only agent loop** — no edit/run/test "thrash"; deferred by design.
+- **Languages:** Python + TypeScript only.
+- **`author.py` seals only the first anchor**, so all current scenarios are single-anchor (no genuine
+  T3 multi-claim); a tooling follow-up would lift this.

bench/PREREGISTRATION.md

@@ -0,0 +1,95 @@
+# Pre-registration — Surface agent-impact benchmark (v2 / multi-turn)
+
+**Status:** to be **frozen and git-tagged before the headline run.** This file states the
+hypotheses, design, metrics, and analysis *in advance* so the result can't be accused of HARKing
+(hypothesizing after results are known) or of cherry-picking an analysis. Nothing below is decided
+by looking at outcomes. Operating instructions are in [`README.md`](README.md).
+
+> **Freeze gate.** Tag this only once: (a) the scenario set is final (all `cascade-*` PRs merged),
+> (b) `python tools/author.py --all` and `python tools/validate_scenario.py --all` are green across
+> the set, and (c) a full `--models mock` run + `python -m surface_bench.oracle` is clean. Record
+> the tag and the `surf --version` here when frozen: `__TBD__`.
+
+## 1. Question
+
+Does keeping documentation accurate (what Surface enforces) change an agent's task performance, and
+*through what mechanism*? Surface's value to an agent = the performance gap between working from
+fresh vs rotted docs, plus whether surfacing the drift recovers it.
+
+## 2. Conditions (the only thing that varies)
+
+Same code + same task + same model in every cell; only the documentation block changes:
+**C0** code only · **C1** code + stale doc · **C2** code + fresh doc · **C3** code + stale doc + real
+`surf check --format json` report · **Cw** code + stale doc + a generic "may be outdated" warning
+(no corrected code). Run in **multi mode** (the agent has read-only `read_file`/`grep`/`list_dir`
+tools and may choose to verify) as the primary design; **single mode** is run as a secondary,
+cheaper comparison.
+
+## 3. Hypotheses (confirmatory)
+
+- **H1 — accuracy beats rot:** success(C2) > success(C1).
+- **H2 — rot is worse than nothing:** misled(C1) > misled(C0).
+- **H3 — surfacing drift recovers it:** success(C3) ≈ success(C2) (and ≫ C1).
+- **H4 — a confident stale doc suppresses verification:** verification_rate(C1) < verification_rate(C0).
+- **H5 — the harm flows through skipped verification (mediation):** within C1, success among rows
+  that verified > success among rows that did not; verifiers are correct, non-verifiers misled.
+- **H6 — recovery is the corrected code, not mere suspicion:** success(C3) > success(Cw).
+
+Direction is pre-specified for every hypothesis. Anything not listed here (e.g. per-provider
+contrasts, tier gradients, token-cost deltas) is **exploratory** and will be reported as such.
+
+## 4. Metrics (pre-defined)
+
+- **success** — `ok`: the agent produced the current (T1) answer. Code scenarios: hidden tests that
+  probe the real dependency. QA scenarios: a `VERDICT:` line matched against a fixed rubric. **No LLM
+  judge.**
+- **misled** — `misled`: the agent asserted the stale (T0) claim.
+- **verification_rate** (multi only) — the agent called `read_file`/`grep` on a path matching the
+  scenario's `hidden_paths` glob *before* `final_answer`. Computed from the logged `verified_hidden`
+  per row. `verified_then_correct` (success among verifiers) is its validity check.
+- **output tokens** and **cost** — secondary (token-tax + spend), exploratory.
+
+## 5. Models, scale, sampling (pre-specified)
+
+- **Models:** `haiku`, `sonnet`, `opus` (Anthropic) + `gpt` (OpenAI) + `gemini` (Google). Exact
+  `model_id`s and prices are pinned in `config.toml` at freeze time and copied into `run.json`.
+- **Trials (tiered):** N = 10 per cell for the **cascade** family (the headline); N = 5 for the
+  **comprehension** family (success-ceilinged — kept only for the token story). Comprehension may be
+  omitted from multi mode entirely (it doesn't test verification); that choice is recorded in
+  `run.json`, not chosen by results.
+- **temperature = 1.0** (stochasticity is part of what we measure); **max_turns = 8**;
+  `max_tokens = 1024`.
+
+## 6. Analysis plan (pre-specified)
+
+- Rates reported with 95% **Wilson** intervals.
+- Each hypothesis tested as an unpaired difference of proportions with a 95% **bootstrap** CI and a
+  bootstrap two-sided p-value (10,000 resamples, fixed seed).
+- **Holm–Bonferroni** correction applied across the family of confirmatory success-delta tests
+  (every model × pre-registered pair). A hypothesis is **confirmed** if its CI excludes 0 **and** it
+  survives Holm; reported as suggestive if CI-significant but not Holm-significant.
+- H5 (mediation) reported as the within-C1 success split (verified vs not) per model.
+- Per-scenario and per-tier breakdowns reported for transparency (exploratory).
+
+## 7. Exclusions / data handling (pre-specified)
+
+- A cell that errors (API failure) is logged with `error` and excluded from rates; the count of
+  excluded cells is reported. No silent retries beyond the client's built-in retry.
+- The oracle (§ README) must be clean on the real run: any scenario failing C2-fresh ≥ 90% or (for
+  cascade) C1-misled > 0 is treated as a **fixture defect** and is flagged; whether to drop it is
+  decided by the pre-stated oracle rule, not by whether it helps the result.
+
+## 8. Stopping rule
+
+The matrix runs to completion (fixed N above). No data-dependent stopping, no N top-ups chosen by
+looking at significance. If a stage smoke (mock → per-provider → small-N pilot) reveals a *harness*
+bug, it is fixed and the affected stage re-run; the confirmatory full run begins only after this file
+is tagged.
+
+## 9. What this does and does not assess
+
+Assesses: whether documentation accuracy changes single-shot and multi-turn task outcomes on curated
+cascade fixtures across five models, and whether the effect is mediated by verification. Does **not**
+assess: real-repository generalization (curated fixtures by design — see deferred #109), an
+edit/run/test agent loop (tools are read-only by design), or non-English / languages beyond Python
+and TypeScript.