fix(eval): harden LLM judge against contradictory / journey-based verdicts

[leggetter]

Summary

The Docs agent eval (CI slice) flaked on main (run 28672821140). Scenario 02-basics-typescript.md passed every heuristic (9/9) and its generated script ran and published a live event (evt_…), but the LLM judge marked the Execution criterion pass=false — while its own evidence said "Overall execution criterion passes based on live API evidence." It penalized self-corrected mid-session stumbles (a run from the wrong working directory → ERR_MODULE_NOT_FOUND, and an SDK argument shape that was fixed) rather than judging the final deliverable. Because the overall verdict is the AND of all criteria, that one contradictory false failed the whole run.

This hardens the judge in two layers so this class of false negative doesn't recur.

Changes

• Prompt (llm-judge.ts) — generalizable fix:
  • Judge the final state of the deliverable and its last run; a criterion passes if the final artifact satisfies it, even when earlier attempts failed and were corrected. Explicitly do not fail execution/correctness criteria for self-corrected intermediate problems (wrong cwd, ERR_MODULE_NOT_FOUND, later-fixed args) when a later run clearly succeeded.
  • Strengthen the pass↔evidence contract so a "but I must flag…" aside can't flip a met criterion to false.
  • Reinforced the same in the secrets-available harness rules and user tail.
• Code safety net — criterionSelfContradicts() / findContradictoryCriteria() conservatively detect a criterion whose pass boolean contradicts an explicit conclusion in its own evidence (never fires on mixed/ambiguous evidence). The judge loop re-asks the model once (within the existing 3-attempt budget) to reconcile — it never silently flips a verdict, and the model's booleans stay authoritative if unresolved.
• Tests — 5 new cases in llm-judge-parse.test.ts, including the exact observed false negative, the inverse, aligned non-contradictions, and mixed-evidence (no false trigger).
• .gitignore — ignore .worktrees/ (matches the website/core convention).

Test plan

• npm test in docs/agent-evaluation passes (llm-judge-parse.test: OK).
• No linter errors on edited files.
• Next scheduled/push run of the CI slice (scenarios 01, 02) is green.

Note: npm run typecheck has pre-existing, project-wide failures (missing @types/node: console/process/fetch/node:*) that also affect untouched files like score-eval.ts. CI runs the eval via tsx at runtime, not tsc, so it is not a gate; left out of scope.

Co-Authored-By: Claude noreply@anthropic.com

Made with Cursor

[Copilot]

Pull request overview

Hardens the docs agent-evaluation “LLM judge” to avoid false-negative CI failures caused by criteria where pass contradicts the criterion’s own evidence, and to ensure verdicts are based on the final deliverable/last successful run rather than intermediate missteps.

Changes:

• Strengthens the judge prompt to explicitly score the final artifact + last run and enforce evidence↔boolean alignment.
• Adds a conservative contradiction detector (criterionSelfContradicts / findContradictoryCriteria) that triggers a single targeted re-ask (within the existing retry budget) instead of silently flipping booleans.
• Adds unit tests covering the observed failure mode, inverse contradictions, aligned cases, and mixed-evidence cases; updates .gitignore to ignore .worktrees/.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.

File	Description
docs/agent-evaluation/src/llm-judge.ts	Adds contradiction detection + retry note wiring, and tightens judge instructions to focus on final deliverable/last run.
docs/agent-evaluation/src/llm-judge-parse.test.ts	Adds unit tests for the new contradiction detection behavior.
.gitignore	Ignores .worktrees/ directory.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.