docs: agent guides, extending & report-schema references, and coder-eval.com/docs migration

[uipreliga]

Summary

Fills the biggest documentation gaps, fixes defects found auditing the docs against the code, and migrates all doc links to the new home at coder-eval.com/docs.

New docs

• docs/agents/ — moved the Codex guide to agents/CODEX.md and added:
  • agents/CLAUDE_CODE.md — the default agent: config surface, direct vs. Bedrock auth, sandbox isolation, early stop, telemetry.
  • agents/ANTIGRAVITY.md — the Google Antigravity (Gemini) agent (previously undocumented): [antigravity] extra, GEMINI_API_KEY auth, thinking_level, native skills, and the caveat that it ignores permission_mode/tool allowlists.
• docs/EXTENDING.md — the plugin SPI: custom agents (the Agent ABC checklist + coder_eval.plugins entry point), custom criteria (@register_criterion), and register_pricing.
• docs/REPORT_SCHEMA.md — field-level reference for run.json / task.json / variant.json / experiment.json / suite.json, token buckets, and FinalStatus.

USER_GUIDE

• Documented the aggregate command and report --format; completed the run flag table (--type/--repeats/--resume/--sample/--sample-per-stratum/--include-skipped) and added plan --experiment, evaluate --run-dir.
• Added a Suite Thresholds & Classification Metrics section; added agent-auth env vars (CODEX_*, GEMINI_API_KEY, ANTIGRAVITY_MODEL); clarified UIPATH_PLUGIN_MARKETPLACE_DIR vs PLUGIN_TOOLS_DIR.

TASK_DEFINITION_GUIDE fixes

• Added the missing commands_efficiency and classification_match sections and completed the scoring-types summary (all 14 types).
• Fixed llm_judge max_tokens default (1000 → 2000), agent_judge default tools (read-only Bash/Read/Glob/Grep, not Write/Edit), and the llm_judge verdict wording (forced submit_verdict tool call).

Link migration → coder-eval.com/docs

• llms.txt: absolute doc URLs rewritten to coder-eval.com/docs/* in the Starlight slug scheme (lowercase-kebab, no trailing slash). Relative .md cross-links left untouched — the site's sync-docs rewrites them at build time.
• README.md: Docs badge + comparison/full-docs links repointed; Ruff badge removed.
• mkdocs.yml site_url and pyproject.toml Documentation URL → coder-eval.com/docs.

Other

• Fixed phantom experiments/example.yaml references in the agent guides.
• Regrouped mkdocs nav (Agents / Running & scaling / Extending / Reference); aligned README/index/llms.txt doc indexes.
• Removed docs/IDEAS.md (internal backlog) and its mkdocs exclude.

Verification

• mkdocs build --strict passes with zero warnings.
• No uipath.github.io references remain in source; all model IDs used in docs resolve in the pricing table.
• Docs-only change — not part of make verify / the Python gate.

🤖 Generated with Claude Code

[github-actions]

Claude finished @uipreliga's task in 1m 31s —— View job

---

Reviewing PR #48: Documentation Updates

Todo List

• Read code review guidelines (.github/code_review.md)
• Read project conventions (CLAUDE.md)
• Get full PR diff (origin/main...HEAD)
• Review new documentation files
  • docs/agents/CLAUDE_CODE.md
  • docs/agents/ANTIGRAVITY.md
  • docs/EXTENDING.md
  • docs/REPORT_SCHEMA.md
• Review modified documentation
  • docs/TASK_DEFINITION_GUIDE.md
  • docs/USER_GUIDE.md
  • docs/agents/CODEX.md (renamed)
  • docs/index.md
  • docs/llms.txt
• Review project file changes
  • README.md
  • mkdocs.yml
  • pyproject.toml
• Check for removed files (IDEAS.md)
• Verify cross-file consistency
• Check "what's missing" analysis
• Post final review

[bai-uipath]

Fix what you agree with, otherwise lgtm — the docs land well; my notes are all about the nav/IA, none blocking.

Nav organization

• Top-level sprawl — 8 groups, two of them single-item. Extending and Reference each wrap exactly one page, so the group label just restates the page title (Extending → "Extending Coder Eval", Reference → "Report Schema") and costs an expand to reach one doc. Fold the long tail under one Advanced group and the top level drops to 6 with both single-item groups gone:

- Home
- Tutorials
- Guides:      User Guide, Task Definition Guide
- Agents:      Claude Code, Codex, Antigravity
- Advanced:    A/B Experiments, Bring Your Own Dataset, Docker Isolation,
               CI Gate & GitHub Action, Extending Coder Eval, Report Schema
- How It Compares

• Report Schema is the one page that might earn its own group. It targets JSON consumers (dashboards, CI parsers), not people running evals — a distinct audience. But with one page today, a Reference group isn't justified yet; merge it into Advanced now and split it back out the moment there's a second reference doc.
• Running & scaling is a grab-bag. "Scaling" doesn't naturally cover Docker isolation or CI gating — the group is really "everything that isn't a tutorial or an agent," which is what Advanced says more honestly.
• Ordering is curated, not alphabetical — keep it that way. Home → Tutorials → Guides → Agents is the right journey-first order; don't sort it A–Z. Minor: inside Running & scaling the order (A/B, BYOD, Docker, CI) is neither alphabetical nor a journey — reorder or leave it, doesn't matter much once it's Advanced.
• Capitalization is inconsistent across group labels. Title Case (Home, Tutorials, Guides, Agents, Extending, Reference) mixed with sentence case (Running & scaling, How it compares). Pick one — I'd go Title Case for section headers (Running & Scaling, How It Compares).

Follow-up, not this PR

• Four parallel doc indexes now need hand-syncing — the mkdocs nav, the README docs table, docs/index.md, and docs/llms.txt. Adding a doc means editing four places and they drift. Out of scope here, but whatever grouping you settle on above, apply it to at least nav + index.md so the two grouped surfaces agree, and consider generating the flat indexes from one source later.

Minor: the docs.yml push-trigger disable is the only behavioral change in an otherwise docs PR — it's well-commented and reversible, fine to ship.

[uipreliga]

Review: coder_eval — pr:48 (25 files) axis:1,2,3,4,5,6,7,8

Scope: pr:48 (25 files) axis:1,2,3,4,5,6,7,8 · branch docs/agent-guides-extending-report-schema · 57d3ad9 · 2026-07-24T00:46Z · workflow variant

Change class: trivial — documentation, MkDocs site config, and package metadata only; no product code path is touched (the only non-docs edits are a commented-out docs auto-publish trigger, nav/site_url, and a Documentation URL)

The engine is healthy (code quality, typing, security, architecture and error handling all 9.7+, zero high-severity findings, no production-code defects), but risk is concentrated in the documentation adopters actually execute: an unresolvable CI action pin, a run command that silently evaluates zero tasks and exits 0, and two harness contracts (stratified-sample seeding, effective allowed_tools baseline) whose omission lets identical agent output produce a different task score — no code needs fixing, but about a day of doc and release-tag corrections is required before these pages are safe to copy-paste.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	9.7 / 10	0	0	0	3	docs/EXTENDING.md prints calculate_cost with keyword names that do not exist on the real function
2. Type Safety	9.9 / 10	0	0	0	1	EXTENDING.md config-class snippet omits the # type: ignore[assignment] every in-repo instance of that discriminator override carries
3. Test Health	9.4 / 10	0	0	1	1	Disabling the docs.yml push: trigger leaves mkdocs gh-deploy --strict reachable only via workflow_dispatch — no automatic mkdocs/link check remains in CI
4. Security	9.8 / 10	0	0	0	2	docs/CI_GATE.md Security section omits workflow-hardening guidance (least-privilege permissions: and SHA-pinning the recommended third-party action)
5. Architecture & Design	9.8 / 10	0	0	0	2	EXTENDING.md lists discard_pending_turn() as an optional override, contradicting the Agent base-class contract
6. Error Handling & Resilience	9.8 / 10	0	0	0	2	docs/CI_GATE.md overstates the score gate's NaN handling — non-finite rows are skipped, not failed closed
7. API Surface & Maintainability	8 / 10	0	1	2	0	New CI reference tells every adopter to pin UiPath/coder_eval@v0, a ref that does not exist on the repo
8. Evaluation Harness Quality	6.4 / 10	0	2	3	1	Agent guides publish coder-eval run experiments/<file>.yaml (experiment YAML passed positionally as TASK_FILE) — the documented command does not work

Overall Score: 9.1 / 10 · Weakest Axis: Evaluation Harness Quality at 6.4 / 10
Totals: 🔴 0 · 🟠 3 · 🟡 6 · 🔵 12 across 8 axes.

Blockers

1. [Axis 7] New CI reference tells every adopter to pin UiPath/coder_eval@v0, a ref that does not exist on the repo (docs/CI_GATE.md:22) — docs/CI_GATE.md:22 is the canonical, newly-published reference for the composite Action and its copy-paste snippet reads:

- uses: UiPath/coder_eval@v0    # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly

(repeated verbatim at docs/CI_GATE.md:67). But no v0 ref exists on the repository at PR HEAD. Verified two ways: gh api repos/UiPath/coder_eval/tags --paginate -q '.[].name' returns only v0.8.9 v0.8.8 v0.8.7 v0.8.6 v0.8.5 v0.8.4 v0.8.3 v0.8.2 v0.0.1, and gh api repos/UiPath/coder_eval/git/matching-refs/tags/v0 returns only those same v0.8.x tags — no bare refs/tags/v0. The branch list (gh api repos/UiPath/coder_eval/branches) likewise contains no v0. GitHub Actions resolves @v0 as a tag-or-branch ref, so every user who copies this snippet gets a hard Unable to resolve action UiPath/coder_eval@v0 at job start.

The moving-major-tag machinery exists but has never run against a release: .github/workflows/release.yml:217-227 (Move major action tag (vN -> this release) → MAJOR="v${VERSION%%.*}"; git tag -f "$MAJOR" "v${VERSION}"; git push -f origin "$MAJOR") — the tag will only appear on the next cut release.

Fix: either (a) cut/push the v0 tag before merging this doc (a one-line git tag -f v0 v0.8.9 && git push -f origin v0), or (b) change the documented pin to a ref that resolves today, e.g. - uses: UiPath/coder_eval@v0.8.9 # pin exactly; @v0 tracks the latest 0.x once published. Note README.md:105/:142 and docs/tutorials/02-ci-pipeline.md:172 carry the same @v0 (pre-existing), so fix (a) resolves all four sites at once.
2. [Axis 8] Agent guides publish coder-eval run experiments/<file>.yaml (experiment YAML passed positionally as TASK_FILE) — the documented command does not work (docs/agents/ANTIGRAVITY.md:63) — ANTIGRAVITY.md:60-63 says "Or override the agent type for every task in an experiment:" followed by coder-eval run experiments/model-comparison.yaml --type antigravity; docs/agents/CODEX.md:75-78 carries the identical form (coder-eval run experiments/model-comparison.yaml --type codex) and that line was edited by this PR. The positional argument of run is task_files (src/coder_eval/cli/run_command.py:126-129: "Path(s) to task YAML file(s). Defaults to all tasks/ recursively."); experiments are supplied via --experiment, -e (run_command.py:229-234). I executed the documented form: uv run coder-eval run experiments/model-comparison.yaml --run-dir tmp/_probe_run → "Skipping task file experiments/model-comparison.yaml — ValueError: Invalid task definition: 2 validation errors for TaskDefinition (task_id, success_criteria)", then "Running 0 task(s)", "Results: 0/0 succeeded", process exit code 0. A user (or a CI job) that copies this line gets a green run that evaluated nothing. Fix both files to the form already used in docs/AB_EXPERIMENTS.md:37 — coder-eval run -e experiments/model-comparison.yaml --type antigravity (same for codex).
3. [Axis 8] New --sample-per-stratum row (docs/USER_GUIDE.md:46) omits that stratified sampling re-draws every run; dataset.sample_seed — the knob that pins it — is documented nowhere in docs/ (docs/USER_GUIDE.md:46) — The new flag rows read: line 45 | --sample N | For dataset-backed tasks, run a fixed-seed random N-row sample (reproducible; cheap smoke test). ... | and line 46 | --sample-per-stratum N | For dataset-backed tasks, keep up to N rows per stratum (stratify_field). Overridden by --sample. |. The juxtaposition implies both are reproducible, but src/coder_eval/orchestration/task_loader.py:399-403 seeds only the flat sample (_SMOKE_SAMPLE_SEED = 0, line 27, used at line 405) while the stratified path uses stratum_seed = ds.sample_seed — None by default, so random.Random(None) re-draws a different row set on every run. A suite gated on pass_rate / classification suite_thresholds therefore scores a different row set each run for identical agent behaviour, and dataset.sample_seed (the knob that pins it) appears in no file under docs/ (grep -rn sample_seed docs/ → no hits). Add the caveat to line 46 and document dataset.sample_seed — e.g. "stratified sampling re-draws each run unless dataset.sample_seed is set; set it to pin a reproducible sample".

Non-blocking, but please consider before merge

1. [Axis 3] Disabling the docs.yml push: trigger leaves mkdocs gh-deploy --strict reachable only via workflow_dispatch — no automatic mkdocs/link check remains in CI (.github/workflows/docs.yml:18) — The PR replaces the trigger with # push: / # branches: [main] (lines 18-19), leaving only workflow_dispatch: (line 24). The job's final step is run: mkdocs gh-deploy --force --strict (line 55) — that --strict build was the ONLY mechanical check in the repo that fails on a broken internal link, a nav: entry pointing at a missing file, or an orphaned page. I confirmed nothing replaces it: grep -n -i "mkdocs|docs/" .github/workflows/pr-checks.yml returns zero hits; tests/lint/runner.py's ALL_RULES are all AST rules over .py files; the only doc-facing check is tests/lint/doc_env_parity.py (CE027), whose own docstring scopes it to framework-prefixed env-var assignments ("only assignments are checked, not bare prose mentions"). This is the highest-risk moment to drop it: the same PR adds 5 new pages, restructures nav: into 4 new groups, renames docs/CODEX_AGENT_GUIDE.md → docs/agents/CODEX.md, and deletes docs/IDEAS.md. Add a pull_request: job (or a step in pr-checks.yml) that runs mkdocs build --strict (build only, no deploy — it needs no secrets and no Pages), so link/nav breakage fails CI even while gh-pages publishing stays disabled.
2. [Axis 7] USER_GUIDE CLI reference mis-enumerates coder-eval report --format (omits junit) and omits --junit-xml from the run flag table (docs/USER_GUIDE.md:102) — docs/USER_GUIDE.md bills itself as "The full command, configuration, and output reference" (line 10), and this PR expanded both its run and report flag tables — but the JUnit surface is missing from both, while the sibling doc added in the same PR documents it.

(a) The newly-added row at docs/USER_GUIDE.md:102 reads:

| `--format, -f` | `md` (default) or `html`. `html` re-renders each `task.json` under the run dir to a `task.html` beside it (or to `-o` when exactly one task is found). |

The code accepts three values — src/coder_eval/cli/report_command.py:53: if fmt not in ("md", "html", "junit"): — and report_command.py:26-32 documents 'junit' (JUnit XML from run.json). A reader taking this table as exhaustive would conclude -f junit does not exist, yet docs/CI_GATE.md:105 instructs exactly coder-eval report runs/latest -f junit.

(b) The run flag table (docs/USER_GUIDE.md:32-53), which this PR grew by six rows, has no --junit-xml entry, though the flag exists (src/coder_eval/cli/run_command.py:176-180: "--junit-xml", help="Write a JUnit XML report of task results to this path (for CI test-report ingestion).") and docs/CI_GATE.md:102 instructs coder-eval run tasks/*.yaml --junit-xml coder-eval-junit.xml.

Fix: change line 102 to `md` (default), `html`, or `junit` (noting junit defaults to <run-dir>/junit.xml per report_command.py:66), and add a --junit-xml row to the run table linking to CI_GATE.md.
3. [Axis 7] docs/EXTENDING.md intro claims all three seams work "without editing the base package", but the criterion seam requires editing models/criteria.py and coder_eval/criteria/ (docs/EXTENDING.md:10) — docs/EXTENDING.md:10-14 opens with:

Coder Eval is extensible along three seams, all designed so a third party can add
capability **without editing the base package**:

1. **Agents** — register a new `agent.type` via the plugin SPI.
2. **Criteria** — add a new success-criterion type via a decorator + auto-discovery.
3. **Pricing** — contribute USD rates for models your plugin runs.

Seams 1 and 3 are genuinely out-of-tree (plugins.py::load_plugins → entry-point register(AgentRegistry); pricing.register_pricing). Seam 2 is not, and the same page says so 124 lines later — docs/EXTENDING.md:134: "In models/criteria.py, subclass BaseSuccessCriterion … then add it to the SuccessCriterion union", and docs/EXTENDING.md:146: "Union membership is required — a run validates that every union member's type has a registered checker, and rejects unknown type tags in YAML." Both files named there live inside the base package (src/coder_eval/models/criteria.py), and BaseSuccessCriterion carries extra="forbid", so a criterion registered purely out-of-tree via @register_criterion can never be referenced from task YAML.

This is the one page a plugin author reads before planning work, so the over-claim costs a wasted design round. Fix: scope the intro claim to the seams that hold, e.g. "Agents and pricing register fully out-of-tree via the plugin SPI; a new criterion additionally needs its model added to the SuccessCriterion union in the base package (see §2) — auto-discovery covers the checker, not the YAML schema."
4. [Axis 8] "Bring Your Own Dataset" pointers resolve to BYOD.md ("Bring Your Own Docker") — dataset fan-out has no documented page (docs/USER_GUIDE.md:185) — New line 185 reads Any criterion on a **dataset-backed** task (see [Bring Your Own Dataset](BYOD.md)), and new line 45 ends See [BYOD](BYOD.md). — but docs/BYOD.md:8 is titled # Bring Your Own Docker (BYOD) and its front-matter description is "Use a custom Docker image with Coder Eval"; grep -n dataset docs/BYOD.md returns nothing. The other new pointer, docs/TASK_DEFINITION_GUIDE.md:817 (on a [dataset-backed task](#task-yaml-structure)), targets the "## Task YAML Structure" section (TASK_DEFINITION_GUIDE.md:39-56), whose top-level key list has no dataset: entry. Net effect: the dataset: key, ${row.<field>} substitution, stratify_field, sample_per_stratum and sample_seed — the fan-out feature the whole new "Suite Thresholds & Classification Metrics" section and the new classification_match criterion depend on — are documented nowhere in docs/. Either add a real dataset section (to TASK_DEFINITION_GUIDE.md's Task YAML Structure) and repoint these links at it, or stop labelling BYOD.md "Bring Your Own Dataset" (mkdocs.yml:91, docs/index.md:81, README.md:184 use that label for the same Docker page).
5. [Axis 8] docs/agents/CLAUDE_CODE.md config table misstates field contracts (nullability of ignore_patterns; model/allowed_tools defaults overridden by experiments/default.yaml) (docs/agents/CLAUDE_CODE.md:99) — Line 99 says | allowed_tools | list[str] \| null | Tool allowlist. Unset ⇒ all tools allowed. | and line 97 says | model | str \| null | Specific model id. Omit to use the backend/route default. | (line 78 repeats it: model: claude-sonnet-4-5-20250929 # optional; omit to use the route default). Both are wrong for a real run: layer 1 of the merge is experiments/default.yaml (shipped in the wheel as coder_eval/resources/default_experiment.yaml per pyproject.toml:133), which sets model: claude-sonnet-4-6 and allowed_tools: ["Bash", "Read", "Write", "Edit", "Glob", "Grep", "Skill"]. Verified with a task whose agent block is only type: claude-code: uv run coder-eval plan prints Default agent config: {... 'model': 'claude-sonnet-4-6', 'allowed_tools': ['Bash', 'Read', 'Write', 'Edit', 'Glob', 'Grep', 'Skill'] ...} and Variant 'default': claude-code (claude-sonnet-4-6). So omitting allowed_tools silently withholds Task/WebFetch/TodoWrite/etc. State the resolved baseline ("unset ⇒ the baseline allowlist from experiments/default.yaml; set allowed_tools: null explicitly for all tools") rather than the bare field default.
6. [Axis 8] REPORT_SCHEMA's token-bucket invariant isn't implementable as written: the per-message bucket fields are never named, and input_tokens denotes different quantities on TranscriptMessage vs TokenUsage (docs/REPORT_SCHEMA.md:134) — Lines 134-138 state "Summing the four token buckets across messages (assistant + the synthetic reconciliation entry) equals token_usage exactly", but the TurnRecord entry (line 129) describes messages only as list[TranscriptMessage] and never names its bucket fields. A consumer implementing from this doc hits a name collision: AssistantMessage/ReconciliationMessage carry input_tokens / output_tokens / cache_creation_tokens / cache_read_tokens (src/coder_eval/models/telemetry.py:220-241, 291-297) and the residual is computed against usage.uncached_input_tokens (streaming/collector.py:111-114), whereas the doc's own TokenUsage section (line 208-210) defines input_tokens as the computed sum of all three input buckets — so sum(messages[].input_tokens) == token_usage.uncached_input_tokens, NOT token_usage.input_tokens. Also, docs/agents/CLAUDE_CODE.md:168-171 says "each turn carries one synthetic role="reconciliation" entry", but collector.py:115-116 returns the messages unchanged when every residual is 0 (and skips it entirely when token_usage is None, collector.py:175-176). Spell out the four per-message field names and the input_tokens ↔ uncached_input_tokens mapping, and soften the "each turn carries one" claim to "when the residual is non-zero".

Nits

1. [Axis 1] docs/EXTENDING.md prints calculate_cost with keyword names that do not exist on the real function (docs/EXTENDING.md:225) — Line 225 documents the API as calculate_cost(model, uncached_input, output, cache_creation=0, cache_read=0). The real signature in src/coder_eval/pricing.py:156-162 is calculate_cost(model: str, uncached_input_tokens: int, output_tokens: int, cache_creation_tokens: int = 0, cache_read_tokens: int = 0), so a caller copying cache_creation=0 / cache_read=0 gets a TypeError. Print the real parameter names.
2. [Axis 1] mkdocs.yml nav keeps the old coder_eval branding the rest of this branch normalized to "Coder Eval" (mkdocs.yml:77) — Line 77 reads - 02 · Running coder_eval in CI: tutorials/02-ci-pipeline.md, while the same PR renamed that title everywhere else — the page's own H1 (docs/tutorials/02-ci-pipeline.md:8 # Tutorial 02 — Running Coder Eval in CI), docs/tutorials/README.md:17, README.md:51/:95, docs/comparison.md:132 and docs/llms.txt:41 all now say "Running Coder Eval in CI". mkdocs.yml is the only surviving coder_eval spelling in a user-visible title; update it to match.
3. [Axis 1] Moved docs/agents/CODEX.md was not brought to parity with its two new sibling agent guides (docs/agents/CODEX.md:192) — CODEX.md now sits in the docs/agents/ group alongside two new guides, but grep -n '](\.\|](\.\./\|](USER_GUIDE\|](TASK_DEF' docs/agents/CODEX.md returns zero hits — it contains no intra-repo links at all, and its References section (lines 190-193) lists only three external OpenAI URLs. Both siblings end with a cross-linking block (docs/agents/CLAUDE_CODE.md:189-194 links Task Definition Guide, User Guide, Codex, Antigravity and EXTENDING.md; docs/agents/ANTIGRAVITY.md:198-203 does the same). Add the equivalent ../TASK_DEFINITION_GUIDE.md / ../USER_GUIDE.md / CLAUDE_CODE.md / ANTIGRAVITY.md / ../EXTENDING.md links so the three agent guides read as one set.
4. [Axis 2] EXTENDING.md config-class snippet omits the # type: ignore[assignment] every in-repo instance of that discriminator override carries (docs/EXTENDING.md:76) — The copy-pasteable snippet at docs/EXTENDING.md:75-77 is class MyAgentConfig(BaseAgentConfig): / type: Literal["my-agent"] = "my-agent". BaseAgentConfig.type is declared type: str | None = Field(default=None, ...) (src/coder_eval/models/agent_config.py:130), so narrowing it to a Literal in a subclass is an incompatible mutable-attribute override — every in-tree instance suppresses it explicitly: ClaudeCodeAgentConfig (agent_config.py:187), AntigravityAgentConfig (agent_config.py:270), and the BYOA worked example this same page links to, tests/fixtures/byoa_demo_plugin/byoa_demo.py:33 (type: Literal["byoa-demo"] = "byoa-demo" # type: ignore[assignment]). A plugin author copying the doc into a type-checked package gets a reportIncompatibleVariableOverride with no hint. Add the trailing # type: ignore[assignment] to the snippet, matching the fixture.
5. [Axis 3] agent_judge failure-mode bullet left on the old JSON-parse wording while the parallel llm_judge bullet was updated (docs/TASK_DEFINITION_GUIDE.md:790) — The PR rewrote the llm_judge failure modes to - The judge never emits the forced ``submit_verdict`` tool call (no verdict returned) (line 713), but the agent_judge section 77 lines later still reads - Non-JSON final message from the judge (parse failure) (line 790). agent_judge uses the same forced-tool mechanism: src/coder_eval/criteria/agent_judge.py:271 does config.allowed_tools = list({*(config.allowed_tools or []), SUBMIT_VERDICT_MCP_TOOL_NAME}) and its system prompt (agent_judge.py:88-89) says "Do NOT emit JSON in text — use the submit_verdict tool. If you do not call submit_verdict the grade is zero." Update line 790 to mirror line 713.
6. [Axis 4] docs/CI_GATE.md Security section omits workflow-hardening guidance (least-privilege permissions: and SHA-pinning the recommended third-party action) (docs/CI_GATE.md:91) — docs/CI_GATE.md is the new authoritative CI reference and its ### Security section (lines 91-96) covers only two risks: "Evaluated tasks execute agent-generated code. Do not run this action under pull_request_target with secrets exposed to untrusted fork PRs. For untrusted tasks, use the Docker driver — the tempdir driver is not a security boundary." It never tells the caller to restrict the job's GITHUB_TOKEN. Neither documented snippet (- uses: UiPath/coder_eval@v0 at line 22 and line 67) sits inside a workflow that declares permissions:, and the hand-rolled workflow the page links to (docs/tutorials/02-ci-pipeline.md lines 49-141) also has no permissions: block — so a copied setup inherits the repo default, which on repos created before Feb-2023 defaults (or is org-configured) to read/write for contents, packages, issues, etc. That token is present in the runner env of the very job that, by this page's own admission, "execute[s] agent-generated code" in a sandbox that is "not a security boundary". Contrast the repo's own workflows, which set an explicit block in every single case: .github/workflows/pr-checks.yml:18-19 (permissions: / contents: read) — and that is the workflow that dogfoods this action (uses: ./ at pr-checks.yml:856) — plus codeql.yml:13-14, conventional-commits.yml:12-14, publish-testpypi.yml:31-32, release.yml:44-46, docker-publish.yml:30-32, and claude-pr-review.yml:20 (permissions: {}). Fix: add one line to the ### Security section recommending permissions: contents: read (narrowed further only where a step needs it) on any workflow calling the action, and add the block to the snippets at CI_GATE.md:21-28 / :66-73 and to the tutorial workflow at docs/tutorials/02-ci-pipeline.md:76-83. CVSS:3.1/AV:L/AC:H/PR:L/UI:R/S:U/C:L/I:L/A:N
7. [Axis 4] docs.yml is the only workflow with contents: write and the only one using mutable action tags plus an unpinned PyPI range (.github/workflows/docs.yml:37) — This PR edits docs.yml (comments out the push: trigger, lines 12-24) but leaves a pinning gap that is unique to this file among all eight workflows. docs.yml grants permissions: / contents: write (lines 26-27) — the only workflow in the repo with write access to repo contents — yet checks out and builds using mutable tags: - uses: actions/checkout@v4 (line 37) and - uses: actions/setup-python@v5 (line 38), plus an unpinned dependency range run: pip install "mkdocs-material[imaging]>=9.5,<10" (line 47). Every other workflow SHA-pins, without exception: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 and actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 appear in pr-checks.yml:44,47, codeql.yml:28, release.yml:81,90, publish-testpypi.yml:47,50, docker-publish.yml:54, conventional-commits.yml:41, and claude-pr-review.yml:51; even the published action.yml:80 pins astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a # v4.2.0. A repointed upstream tag or a malicious mkdocs-material release therefore executes in a job holding a contents: write token (the tj-actions/changed-files March-2025 tag-repointing incident is the exact shape). Exposure is currently reduced because this PR leaves only workflow_dispatch: (line 24), so a maintainer must trigger it — but the comment at line 17 says "Re-enable by uncommenting the push: trigger below", which restores automatic execution on every docs/** merge to main. Fix: SHA-pin lines 37-38 to the same commits the other workflows use, and pin mkdocs-material to an exact version (or a lockfile) — ideally before the push: trigger is uncommented. CVSS:3.1/AV:N/AC:H/PR:L/UI:R/S:U/C:L/I:L/A:N
8. [Axis 5] EXTENDING.md lists discard_pending_turn() as an optional override, contradicting the Agent base-class contract (docs/EXTENDING.md:91) — docs/EXTENDING.md:91-92 reads: "Optional overrides (sensible defaults exist): kill(), kill_sync() (called from a non-asyncio watchdog thread — must not await), discard_pending_turn()." kill()/kill_sync() are genuinely meant to be overridden (src/coder_eval/agent.py:255, :282 are no-op defaults), but discard_pending_turn() is not: src/coder_eval/agent.py:264-281 owns the cross-retry rollback of _iteration_was_incremented and self._iteration that the orchestrator relies on after every failed communicate(), and CLAUDE.md's "Adding a New Agent" step 5 states "discard_pending_turn() and get_state() are concrete on the base and need no override." Presenting it as an optional override invites a third-party agent to reimplement that bookkeeping and silently break iteration-count rollback. Move discard_pending_turn() out of the "optional overrides" list and state that it (and get_state()) are concrete on the base and must not be overridden.
9. [Axis 5] Doc index duplicated across four surfaces with no guard; mkdocs nav label and llms.txt URL scheme have already drifted (mkdocs.yml:77) — The same doc index is now maintained in four independent places — README.md:180-189, docs/index.md:73-90, docs/llms.txt:22-46, and the mkdocs.yml nav: block (lines 74-99) — with nothing in tests/ or CI checking them against each other, and two drifts already exist at PR HEAD. (1) mkdocs.yml:77 reads - 02 · Running coder_eval in CI: tutorials/02-ci-pipeline.md while every other surface uses the brand spelling introduced by commit 5dea1c7: docs/tutorials/02-ci-pipeline.md:8 (# Tutorial 02 — Running Coder Eval in CI), docs/tutorials/README.md:17, README.md:51, docs/llms.txt:41 — so the rendered sidebar is the only place showing the old repo-name spelling. (2) The canonical URL scheme is inconsistent between the two surfaces this PR touched: mkdocs.yml:6 sets site_url: https://coder-eval.com/docs/ with no use_directory_urls/slugify override, so an mkdocs build emits …/docs/USER_GUIDE/, whereas docs/llms.txt:26 advertises https://coder-eval.com/docs/user-guide for that same page — one of the two must be wrong whichever site actually serves the content. Fix the mkdocs.yml:77 label, and pick one canonical URL scheme (either set site_url to whatever the externally-synced coder-eval.com site actually serves, or regenerate llms.txt from the file paths); longer term, generate docs/llms.txt (and ideally the README/index tables) from mkdocs.yml nav: so there is a single source of truth.
10. [Axis 6] docs/CI_GATE.md overstates the score gate's NaN handling — non-finite rows are skipped, not failed closed (docs/CI_GATE.md:89) — Lines 88-89 state: "Errored tasks (null score) are left to coder-eval's exit code; a malformed/NaN score fails closed." The action's gate does the opposite for a mixed run — action.yml:188 filters rows with if isinstance(s, (int, float)) and not isinstance(s, bool) and math.isfinite(s):, so a NaN/inf weighted_score row is excluded from rows; worst = min(rows) (line 192) then only ranges over the finite rows and ok = worst is not None and worst[0] >= floor (line 193) passes. It only "fails closed" when every row is unusable (empty rows → the "no scored tasks in run.json" error at line 199). Reword to match: "a malformed/NaN score is excluded from the comparison so it cannot mask a below-floor task; the gate fails only if no scored task remains."
11. [Axis 6] EXTENDING.md's criterion error contract omits the JudgeInfrastructureError fail-loud escape hatch (docs/EXTENDING.md:178) — Lines 177-178 tell plugin authors: "Do not override check() — it's final and wraps _check_impl with error handling (an exception becomes a score-0.0 result with the error captured)." That is true for every exception except one: src/coder_eval/criteria/base.py:75-78 re-raises JudgeInfrastructureError ("Judge infra failure is NOT an agent failure — do not score it 0.0. Propagates to Orchestrator.run()'s broad except → FinalStatus.ERROR"). As written, a third-party criterion whose backing service is down has no documented way to fail the run loud and will silently score the agent 0.0 — an evaluation-correctness trap. Add one line: infrastructure failures (network/service unavailable, not an agent fault) should raise JudgeInfrastructureError (exported from coder_eval.errors), which propagates and marks the task FinalStatus.ERROR instead of scoring 0.0.
12. [Axis 8] TASK_DEFINITION_GUIDE's table of contents and shared-field table not updated for the two criterion sections this PR adds (docs/TASK_DEFINITION_GUIDE.md:20) — The PR adds ### commands_efficiency (line 619) and ### classification_match (line 795), but the criterion list in the Table of Contents (lines 20-32) still enumerates only 12 entries — jumping from command_executed (line 29) straight to uipath_eval (line 30) and from agent_judge (line 32) to skill_triggered (line 33) — while line 383 states "The framework supports 14 criterion types." Add the two missing TOC rows. While there, the shared-field table at lines 389-394 lists description / weight / pass_threshold / stop_when but not suite_thresholds, even though the new sections (lines 817, 842) and USER_GUIDE.md's new Suite Thresholds section tell authors to set it on a criterion.

What's Missing

Tests:

• 🟠 No parity/golden test guards the brand-new published contract in docs/REPORT_SCHEMA.md (+251) against the Pydantic models it describes — nothing asserts that the documented keys of run.json / variant.json / task.json / suite.json still match models/results.py, models/telemetry.py and the dict keys reports_experiment.py actually writes. The token-bucket-invariant defect (per-message bucket names never spelled out; messages[].input_tokens ↔ token_usage.uncached_input_tokens) is the first instance of exactly the drift class this page now invites, and there is no mechanism to catch the second. Add a test that walks the documented field tables and asserts each name exists on the corresponding model (or generate the tables from the models). (trigger: docs/REPORT_SCHEMA.md) (restates: Axis 8: REPORT_SCHEMA's token-bucket invariant isn't implementable as written)
• 🟠 Nothing validates that copy-paste commands published in docs actually parse against the real Typer CLI. coder-eval run experiments/model-comparison.yaml --type <kind> (docs/agents/ANTIGRAVITY.md:63, docs/agents/CODEX.md:78) exits 0 having evaluated 0 tasks, and no test or lint rule would have flagged it. Cheapest mechanical guard: a CE0nn lint rule that flags any fenced coder-eval run line whose positional argument is a path under experiments/ (must use -e), plus a smoke test that extracts fenced coder-eval … blocks from docs/ and asserts each parses via the Typer app without executing. (trigger: docs/agents/ANTIGRAVITY.md) (restates: Axis 8: Agent guides publish coder-eval run experiments/<file>.yaml — the documented command does not work)
• 🟡 After commenting out docs.yml's push: trigger the repo has zero automatic mkdocs validation — mkdocs appears in no other workflow, pr-checks.yml has no docs step, and every tests/lint/rules/* rule is an AST rule over .py. This PR is the worst moment to lose it: 5 new pages, a 4-group nav restructure, one rename (CODEX_AGENT_GUIDE.md → agents/CODEX.md) and one deletion (IDEAS.md). Add a build-only mkdocs build --strict job on pull_request (needs no secrets and no Pages) so broken internal links and nav entries pointing at missing files fail CI while gh-pages publishing stays off. (trigger: .github/workflows/docs.yml) (restates: Axis 3: Disabling the docs.yml push trigger leaves mkdocs --strict reachable only via workflow_dispatch)
• 🟡 The USER_GUIDE CLI flag tables are hand-maintained with no test asserting they enumerate every Typer option and every accepted enum value — --junit-xml (run_command.py:176-180) is absent and --format is documented as md|html while report_command.py:55 accepts md|html|junit. A test that diffs the option names declared in cli/*.py against the table rows in docs/USER_GUIDE.md (and asserts documented value enumerations are supersets of the code's membership checks) would have caught both halves at authoring time. (trigger: docs/USER_GUIDE.md) (restates: Axis 7: USER_GUIDE CLI reference mis-enumerates coder-eval report --format and omits --junit-xml)

Parallel paths:

• 🟡 CLAUDE.md's "Extension Points" / "Adding a New Agent" / "Registering Model Pricing" sections are the in-repo parallel copy of the new docs/EXTENDING.md, and neither was updated to reference the other. They already disagree: CLAUDE.md states "discard_pending_turn() and get_state() are concrete on the base and need no override", while EXTENDING.md:91 lists discard_pending_turn() as an optional override. Pick one as authoritative and have the other link to it, so the plugin-SPI contract has a single source of truth. (trigger: docs/EXTENDING.md) (restates: Axis 5: EXTENDING.md lists discard_pending_turn() as an optional override)
• 🟡 The doc index is now maintained in four hand-written surfaces (README.md:177-191, docs/index.md:73-90, docs/llms.txt:22-46, mkdocs.yml nav:74-99) and this PR added the new pages to all four independently — with no guard. Residual drift at PR HEAD: mkdocs.yml:77 is the only surface still carrying the pre-5dea1c7 coder_eval spelling, and README.md's table is the only one lacking a "How it compares" row that index.md and llms.txt both carry. Generate llms.txt (and ideally the two tables) from mkdocs nav:, or add a parity test. (trigger: mkdocs.yml) (restates: Axis 5: Doc index duplicated across four surfaces with no guard)
• 🟡 The unresolvable UiPath/coder_eval@v0 action pin is not confined to the new page — the same ref appears at README.md:105 and :142 and docs/tutorials/02-ci-pipeline.md:172, so a doc-only fix in CI_GATE.md would leave three copies broken. Fix once at the source (git tag -f v0 v0.8.9 && git push -f origin v0, which release.yml:217-227 will maintain from the next cut) rather than editing four files. (trigger: docs/CI_GATE.md) (restates: Axis 7: New CI reference tells every adopter to pin UiPath/coder_eval@v0, a ref that does not exist)
• 🔵 The CODEX_AGENT_GUIDE.md → docs/agents/CODEX.md rename left two stale mentions of the old path in files outside this PR's scope: tests/lint/doc_env_parity.py:20 (rule docstring example) and tests/test_custom_lint.py:677 ("[Codex Agent Guide](docs/CODEX_AGENT_GUIDE.md)" fixture string). Neither is a live link so nothing breaks today, but both are now references to a path that no longer exists and should be renamed with the file. (No surviving reference to the deleted docs/IDEAS.md exists anywhere in the repo — that deletion is clean.) (trigger: docs/agents/CODEX.md)

Downstream consumers:

• 🟡 This PR repoints every externally-published docs URL to https://coder-eval.com/docs (README.md badges + :24/:25/:205, pyproject.toml:56 Documentation, and all 18 URLs in docs/llms.txt) in the same commit range that disables the only workflow which publishes the site — yet nothing in the repo describes what syncs coder-eval.com or verifies the new URL shape. mkdocs.yml has no use_directory_urls/slugify override, so a build serves …/docs/USER_GUIDE/ while llms.txt advertises …/docs/user-guide; every one of the 12 non-tutorial doc URLs is affected. State the sync mechanism and reconcile the slug scheme (or add a link-check of llms.txt against the live site). (trigger: docs/llms.txt) (restates: Axis 5: Doc index duplicated across four surfaces with no guard)
• 🔵 pyproject.toml's Documentation = "https://coder-eval.com/docs" is packaging metadata that ships to PyPI on the next release, and the PyPI long_description is README.md — whose in-body doc links this PR also moved off-repo. No release-time check asserts those targets resolve, so a broken or not-yet-synced docs host becomes a published-package defect that can only be fixed by cutting a new version. Add a cheap link check to release.yml (or the pre-release checklist) covering the project URLs and the README's absolute links. (trigger: pyproject.toml)

Display & mapping dicts:

• 🔵 The new Agents: nav group (mkdocs.yml:82-85) and the matching agent-guide rows in README.md/docs/index.md/docs/llms.txt cover 3 of the 4 registered AgentKind values — none (NoOpAgent, src/coder_eval/agents/noop_agent.py, AgentKind.NONE) has no entry and no pointer, even though it is a documented user-facing agent.type with its own section at docs/TASK_DEFINITION_GUIDE.md:142. Add a one-line "No-op / system tasks" pointer to the Agents group so the guide set matches the registry. (trigger: mkdocs.yml)
• 🔵 docs/TASK_DEFINITION_GUIDE.md's hand-maintained index tables were not extended for the material this PR adds: the criterion TOC (lines 20-32) still lists 12 of the 14 types (missing commands_efficiency at :619 and classification_match at :795, while line 383 asserts "14 criterion types"), and the shared-field table (lines 389-394) omits suite_thresholds even though the new sections tell authors to set it. Same class as an unextended rendering dict — a new variant added without updating the map that enumerates them. (trigger: docs/TASK_DEFINITION_GUIDE.md) (restates: Axis 8: TASK_DEFINITION_GUIDE's table of contents and shared-field table not updated)

Daily/nightly:

• 🟡 The PR touches no Python, so the nightly/cron run path, the DockerRunner entrypoint and the Bedrock judge route are behaviourally unaffected — but the blast radius is not zero and is unstated. (1) docs/REPORT_SCHEMA.md now publishes the run-record/task.json/run.json shape that the out-of-repo coder-eval-uipath / evalboard consumer reads; the PR names no owner, no versioning policy, and no rule that a change to reports*.py or models/results.py must update this page in lockstep. (2) docs/CI_GATE.md hands every downstream CI adopter the pin UiPath/coder_eval@v0, which does not resolve — a gate copied from this page fails at job start. State both explicitly, and add REPORT_SCHEMA.md to the checklist for any report-schema change. (trigger: docs/REPORT_SCHEMA.md) (restates: Axis 7: New CI reference tells every adopter to pin UiPath/coder_eval@v0, a ref that does not exist)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] Implement CE026 — the workflow-YAML rule already claimed (but never built) in /Users/religa/src/coder_eval/.claude/harness-candidates.md:16. New non-AST checker /Users/religa/src/coder_eval/tests/lint/workflow_hardening.py, wired as a dedicated test class in tests/test_custom_lint.py exactly the way CE027 (tests/lint/doc_env_parity.py) is wired (the runner in tests/lint/runner.py is AST-over-.py only, so this rule must not be a BaseRule). Three checks over .github/workflows/*.yml, action.yml, and every ```yaml fence in docs/**/*.md + `README.md`: (1) every `uses:` step must pin a 40-hex commit SHA (allowlist first-party `uses: ./` and `UiPath/coder_eval@…`); (2) every workflow file must declare a top-level `permissions:` block, and any documented workflow snippet that invokes the action must show one; (3) any `pip install "pkg>=X,<Y"` in a job whose `permissions:` grants a `write` scope must pin an exact version or a lockfile. Prevents: Directly catches `.github/workflows/docs.yml:37-38` (`actions/checkout@v4`, `actions/setup-python@v5` — the only floating tags in the repo, in the only job holding `contents: write` at docs.yml:26-27) and `docs.yml:47` (`mkdocs-material[imaging]>=9.5,<10` unpinned in that same job), plus the missing `permissions: contents: read` in the published snippets at `docs/CI_GATE.md:21-28` / `:66-73` and `docs/tutorials/02-ci-pipeline.md:76-83`.
• [bandit-codeql] Add a GitHub-Actions static analyzer to the quality-gate job in .github/workflows/pr-checks.yml:31 — either zizmor (uvx zizmor .github/workflows/ action.yml) or CodeQL's actions language pack in .github/workflows/codeql.yml. Both flag unpinned third-party actions, over-broad permissions:, and pull_request_target + untrusted-checkout patterns out of the box, so this is the belt to CE026's braces for the workflow files themselves (CE026 still carries the doc-fence and pip-pin checks, which neither tool sees). Prevents: Same two Axis-4 findings — .github/workflows/docs.yml:26-27/37-38 (contents: write + mutable tags, the tj-actions/changed-files tag-repointing shape) and the absent least-privilege guidance in docs/CI_GATE.md:91.
• [ce-lint] New CE028 — CLI ↔ docs/USER_GUIDE.md flag parity. Checker /Users/religa/src/coder_eval/tests/lint/doc_cli_parity.py, wired as a test class in tests/test_custom_lint.py (CE027 pattern; it imports live objects the same way CE027 imports Settings). Introspect the Typer app (coder_eval.cli:app → click command tree) and assert: (a) every declared option long-name of run/plan/report appears verbatim in the corresponding flag table in docs/USER_GUIDE.md; (b) for any option whose accepted values are a closed set, every value appears in that option's doc row. To make (b) mechanical rather than grep-guessy, also convert report_command.py's ad-hoc membership check if fmt not in ("md", "html", "junit") (src/coder_eval/cli/report_command.py:55) into a StrEnum/Literal typer choice — the same closed-set-instead-of-string-tuple move CE018 already enforces for FinalStatus — so the rule can read the choices off the enum. Prevents: Catches both halves of the medium docs/USER_GUIDE.md:102 finding: the --format, -f row enumerating only md/html while the code accepts junit, and the missing --junit-xml row (the sole omission out of run_command.py's 20 declared options). Would also fail on any future flag added to run/report without a doc row.
• [ce-lint] New CE029 — Pydantic schema ↔ documentation parity, one module /Users/religa/src/coder_eval/tests/lint/doc_schema_parity.py with four sub-checks (again wired as a tests/test_custom_lint.py class, CE027 style, since all inputs are Markdown + live models): (1) every top-level field of TaskDefinition appears as a key in the ## Task YAML Structure block of docs/TASK_DEFINITION_GUIDE.md:39-56; (2) every member of the SuccessCriterion union (src/coder_eval/models/criteria.py:972) has both a ### <type> section and a Table-of-Contents entry, and the prose count ("The framework supports N criterion types", TASK_DEFINITION_GUIDE.md:383) equals len(union); (3) every field of BaseSuccessCriterion appears in the shared-field table at TASK_DEFINITION_GUIDE.md:389-394; (4) every field name mentioned in docs/REPORT_SCHEMA.md resolves to a real field on the model it is documented under, and every field of the models that doc claims to describe (TurnRecord, TokenUsage, AssistantMessage/ReconciliationMessage) is named at least once. Prevents: Sub-check (1) catches the undocumented dataset: key behind the mislabeled 'Bring Your Own Dataset' pointers (docs/USER_GUIDE.md:45, :185, docs/TASK_DEFINITION_GUIDE.md:817) — dataset, stratify_field, sample_per_stratum, sample_seed appear nowhere in docs/ today. (2) and (3) catch the missing commands_efficiency / classification_match TOC rows and the missing suite_thresholds shared-field row (docs/TASK_DEFINITION_GUIDE.md:20). (4) catches docs/REPORT_SCHEMA.md:134 stating the four-bucket invariant while never naming input_tokens/output_tokens/cache_creation_tokens/cache_read_tokens or the input_tokens ↔ uncached_input_tokens mapping.
• [ce-lint] New CE030 — documented-API signature parity. Extend doc_schema_parity.py (or a sibling module) with a check that scans docs/**/*.md + README.md for call snippets of an explicit whitelist of exported callables (calculate_cost, register_pricing, create_agent, AgentRegistry.register) in both inline-code spans and fenced blocks, parses each call with ast.parse, and validates every keyword name and the arity against inspect.signature of the real function. Whitelisted, not repo-wide, to keep false positives at zero. Prevents: Catches docs/EXTENDING.md:225 — calculate_cost(model, uncached_input, output, cache_creation=0, cache_read=0) versus the real calculate_cost(model, uncached_input_tokens, output_tokens, cache_creation_tokens=0, cache_read_tokens=0) (src/coder_eval/pricing.py:156-162). Note the defect sits in prose backticks, not a python fence, so the doc-snippet typecheck below cannot reach it — hence the separate rule.
• [pyright] Type-check the documentation's Python snippets: a small extractor (tests/lint/doc_snippets.py) that writes every ```python fence in docs/**/*.md into `tmp/doc_snippets/_.py`, plus a `make docs-typecheck` target running `pyright` over that directory with the repo's existing standard-mode config, added to `make verify`. Snippets that are intentionally partial opt out via a `` marker above the fence. Prevents: Catches `docs/EXTENDING.md:76` — the copy-pasteable `class MyAgentConfig(BaseAgentConfig): type: Literal["my-agent"] = "my-agent"` snippet missing the `# type: ignore[assignment]` that every in-tree instance carries (`agent_config.py:187`, `:270`, `tests/fixtures/byoa_demo_plugin/byoa_demo.py:33`); pyright reports `reportIncompatibleVariableOverride` on exactly that shape. Also guards future drift in the plugin/criterion snippets at EXTENDING.md:43-51, :137-144, :156-173, :199-211.
• [ce-lint] New CE031 — doc-index single-source parity. Checker asserting that the four hand-maintained indexes list the same set of pages and the same labels: mkdocs.yml nav: (lines 74-99), README.md:180-189, docs/index.md:73-90, and docs/llms.txt:22-46; and that each nav: label matches the target page's H1/front-matter title after normalization. Add a second assertion that every URL in docs/llms.txt maps to a real docs file under the site_url scheme configured in mkdocs.yml:6. Longer term the rule becomes moot by generating docs/llms.txt (and the README/index tables) from nav: — the rule is the cheap interim guard. Prevents: Catches mkdocs.yml:77 (02 · Running coder_eval in CI vs. the page's own H1 Tutorial 02 — Running Coder Eval in CI and all three other surfaces, post-5dea1c7) and the site_url …/docs/USER_GUIDE/ vs. docs/llms.txt:26 …/docs/user-guide scheme drift. A near-miss variant of the label check (link text sharing most tokens with the target H1 but not matching it) also flags [Bring Your Own Dataset](BYOD.md) pointing at a page titled Bring Your Own Docker (BYOD) — docs/USER_GUIDE.md:45/:185, docs/index.md:81, README.md:184, mkdocs.yml:91.
• [ce-lint] New CE032 — agent-config doc tables must state the resolved baseline, not the bare Pydantic default. For each field documented in a table under docs/agents/*.md, if the packaged layer-1 baseline (experiments/default.yaml, shipped as coder_eval/resources/default_experiment.yaml per pyproject.toml:133) sets that key, require the doc cell to name experiments/default.yaml or quote the baseline value. Cheap to implement: parse the YAML, parse the Markdown table, string-contains check. Prevents: Catches docs/agents/CLAUDE_CODE.md:97 (model — 'omit to use the backend/route default', but layer 1 sets claude-sonnet-4-6), :99 (allowed_tools — 'Unset ⇒ all tools allowed', but layer 1 sets a 7-tool allowlist genuinely enforced at claude_code_agent.py:1077), and the same claim repeated at :78.

Harness improvements (not statically reachable):

• Add a mkdocs build --strict job (build only, no deploy, no secrets, no Pages) to .github/workflows/pr-checks.yml triggered on pull_request, plus a make docs-check target so it runs locally. Keep .github/workflows/docs.yml's publish path disabled as this PR intends — the point is to separate validation from publishing, which the repo currently conflates in one mkdocs gh-deploy --force --strict step (docs.yml:53). Why not static: Needs a real MkDocs build: the link graph, nav:→file resolution, and theme/plugin rendering are only known once the site is assembled; no grep or AST pass resolves relative links across 40+ pages. Note --strict empirically does NOT fail on orphaned pages, so the CE031 index-parity rule still earns its keep alongside it. Prevents: The medium finding at .github/workflows/docs.yml:18 — disabling the push: trigger left the only mechanical link/nav check reachable solely via workflow_dispatch, exactly when the PR adds 5 pages, restructures nav: into 4 groups, renames docs/CODEX_AGENT_GUIDE.md → docs/agents/CODEX.md and deletes docs/IDEAS.md; the preceding Docs run on main (id 30047136850) failed on precisely this rot class. Also subsumes the deferred 'dead-relative-link checker' candidate in .claude/harness-candidates.md.
• Make coder-eval run fail loudly instead of silently green: exit non-zero when zero tasks were resolved, and when any task file was skipped for a validation error (today orchestration/experiment.py:638 prints ⚠ 1 task file(s) skipped, then Running 0 task(s) / Results: 0/0 succeeded, exit code 0 — measured). Add a regression test asserting the exit code for (a) an empty resolution and (b) a partially-skipped batch, with an explicit opt-out flag for callers who genuinely want best-effort behavior. Why not static: It is a runtime exit-code contract, not a source pattern — the offending invocation parses cleanly and only misbehaves after YAML loading and TaskDefinition validation at run time. Prevents: The high finding on docs/agents/ANTIGRAVITY.md:63 / docs/agents/CODEX.md:78 (coder-eval run experiments/model-comparison.yaml --type …, missing -e): the doc fix repairs two lines, but this change removes the whole hazard class of a CI gate going green having evaluated nothing.
• Docs-command smoke test: extract every fenced coder-eval … invocation from docs/**/*.md + README.md, run each through coder-eval plan (dry, no agent) in a pytest, and assert it exits 0 and resolves ≥1 task. Commands needing credentials or a live agent opt out via a <!-- no-smoke --> marker. Complements the deferred 'docs-vs-harness smoke test' candidate already listed in .claude/harness-candidates.md. Why not static: Requires the loader to actually resolve the YAML path into task definitions — coder-eval run experiments/foo.yaml --type codex is syntactically valid CLI and only fails semantically once validation rejects the experiment file. Prevents: The same ANTIGRAVITY.md:63 / CODEX.md:78 finding, plus the general class of documented commands that parse but evaluate nothing; would also have surfaced the -f junit / --junit-xml surfaces documented only in docs/CI_GATE.md:109/:112 and absent from docs/USER_GUIDE.md.
• Close the release-process gap behind the @v0 pin: (a) add a post-release verification step to .github/workflows/release.yml after the Move major action tag (vN -> this release) step (lines 217-227) asserting git ls-remote origin refs/tags/v$MAJOR resolves; (b) add one CI job (or a nightly) that consumes the published ref uses: UiPath/coder_eval@v0 instead of uses: ./ — today the only dogfood job (pr-checks.yml:856) uses the local checkout, so the ref users are told to copy is never exercised; (c) as an immediate unblock, push the tag once: git tag -f v0 v0.8.9 && git push -f origin v0. Why not static: Ref existence lives on the remote — a lint rule can check that the documented major matches the project version, but only a network call (or an actual Actions resolution) can prove refs/tags/v0 exists. Prevents: The high finding at docs/CI_GATE.md:22 (repeated at :67, README.md:105/:142, docs/tutorials/02-ci-pipeline.md:172): UiPath/coder_eval@v0 does not resolve on the repo, so every adopter copying the canonical snippet gets Unable to resolve action UiPath/coder_eval@v0 at job start.
• Reproducibility guard for stratified sampling: persist the effective seed and the drawn row ids into run.json for every dataset-backed run, and emit a warning when a task carries suite_thresholds while the stratified path runs unseeded (orchestration/task_loader.py:403, stratum_seed = ds.sample_seed defaulting to None). A gated run then becomes replayable from its own artifact rather than only from a hand-set dataset.sample_seed. Why not static: The seed is drawn at run time; a static rule can at most flag the suite_thresholds-without-sample_seed YAML combination, and even that is a legitimate configuration (the nightly activation suite deliberately re-draws each run). Prevents: The high finding at docs/USER_GUIDE.md:46 — --sample-per-stratum documented beside the reproducible --sample with no hint that it re-draws each run, and dataset.sample_seed documented nowhere in docs/; a suite gated on pass_rate or classification suite_thresholds scores a different row set each run for identical agent behaviour.
• Golden/property test for the token-bucket invariant across both agents and both modes: assert Σ(assistant + reconciliation buckets) == token_usage on recorded fixtures, with explicit cases for (a) a zero-residual turn where EventCollector emits no reconciliation entry (streaming/collector.py:115-116, :175-176), and (b) simulation mode, where UserMessage carries its own input_tokens/output_tokens (models/telemetry.py:181-182) that the collector deliberately excludes — so a consumer summing all of TurnRecord.messages over-counts. Why not static: Needs a live or recorded event stream reduced through EventCollector; the invariant is arithmetic over runtime-produced messages, not any source-level shape. Prevents: The medium finding at docs/REPORT_SCHEMA.md:134 and its companion overstatements in docs/agents/CLAUDE_CODE.md:168-173 ('each turn carries one synthetic reconciliation entry'; invariant stated over all of TurnRecord.messages) — a test that fails on both the zero-residual and simulation cases forces the doc wording to track the code.

Top 5 Priority Actions

1. Restore the reproducibility caveat the CLI help already carries at docs/USER_GUIDE.md:46 — src/coder_eval/orchestration/task_loader.py:403 passes stratum_seed = ds.sample_seed (None by default), so a suite gated on suite_thresholds scores a different row subset every run for identical agent behaviour; also document dataset.sample_seed and stratify_field, which appear nowhere under docs/.
2. Correct the allowed_tools/model contract at docs/agents/CLAUDE_CODE.md:99 (and :78/:97) — 'Unset ⇒ all tools allowed' is false once experiments/default.yaml (shipped as coder_eval/resources/default_experiment.yaml) injects allowed_tools [Bash, Read, Write, Edit, Glob, Grep, Skill] and model claude-sonnet-4-6, so omitting the key silently withholds Task/WebFetch/TodoWrite and depresses scores; state the resolved baseline plus the explicit 'allowed_tools: null' escape hatch.
3. Add the JudgeInfrastructureError escape hatch to the criterion error contract at docs/EXTENDING.md:177-178 — src/coder_eval/criteria/base.py:75-78 re-raises it so infrastructure outages become FinalStatus.ERROR instead of a score-0.0 verdict, and as written a third-party criterion whose backing service is down silently grades the agent 0.0.
4. Fix the experiment invocation at docs/agents/ANTIGRAVITY.md:63 and docs/agents/CODEX.md:78 to 'coder-eval run -e experiments/model-comparison.yaml --type ...' (the form already correct at docs/AB_EXPERIMENTS.md:37) — the positional arg is task_files (src/coder_eval/cli/run_command.py:126-129), and the documented command reproduces as 'Running 0 task(s) / Results: 0/0 succeeded' with exit code 0, an unconditionally green CI gate that evaluated nothing.
5. Cut and push the missing moving major tag (git tag -f v0 v0.8.9 && git push -f origin v0) — docs/CI_GATE.md:22 and :67, README.md:105/:142 and docs/tutorials/02-ci-pipeline.md:172 all pin UiPath/coder_eval@v0, which resolves to nothing today because release.yml:217-227's major-tag step landed after the last release, so every adopter hits 'Unable to resolve action'; one tag fixes all five sites with zero doc edits.

---

Stats: 0 🔴 · 3 🟠 · 6 🟡 · 12 🔵 across 8 axes reviewed.