test(evals): tighten codegen evaluation pipeline

[denolfe]

Overview

Hardens the eval pipeline used to test the Payload skill's codegen quality. Adds deterministic AST-level structural assertions, inlines the full skill (SKILL.md plus every reference/*.md) into the eval prompt, fixes dashboard / cache drift, and tightens the LLM scorer rubric.

Full agent evals coming soon.

Key Changes

• AST assertions for codegen cases

  • New test/evals/assertions/ module: a TS-compiler-API parser plus a small DSL (collectionExists, fieldExists, fieldOption, fieldHook, collectionHook, collectionAccess, blockField).
  • CodegenEvalCase gains an optional assertions field. Failures short-circuit the case to fail before the LLM scorer runs, so structural mismatches no longer get smoothed over by a "mostly correct" rating.
  • All collections and fields cases backfilled with assertions matching the canonical patterns in FIELDS.md, HOOKS.md, and ACCESS-CONTROL.md.

• Full skill context in the eval prompt

  • The eval LLM has no tool access, so the markdown links inside SKILL.md (e.g. [FIELDS.md](reference/FIELDS.md)) were inert text. Reference docs were effectively invisible to the model under test.
  • New skillContent.ts reads SKILL.md plus every reference/*.md and concatenates them under labeled headers. runner/systemPrompts.ts consumes this; cache.getSkillHash hashes the same content so cache keys invalidate when any reference doc changes.
  • System prompt grows from roughly 5K to 42K tokens; OpenAI prompt caching keeps the per-run cost bounded.

• LLM scorer rubric tightening

  • When the expected outcome names a specific construct or API shape, using a different one caps correctness at 0.4. "Minor" applies only when construct, location, and API surface are correct and only an option value differs.

• Skill content adjustments

  • Quick-Reference row for computed fields now states "field-level hooks.afterRead returning the value".
  • Hook Example expanded with explicit collection-vs-field framing and an inline virtual-field example.
  • Removed unsupported "use as const for field options" guidance.
  • Added a rule to annotate extracted named constants with the matching Payload type or use satisfies.

Design Decisions

• AST assertions over scorer-only judgment. The scorer (gpt-4o-mini) rated wrong-shape answers as "mostly correct" even after rubric tightening. Structural checks belong in deterministic code, not in another LLM. The DSL covers structural concerns; the scorer continues to handle semantic correctness on cases where assertions are absent or pass.
• Eager-inline of reference docs over a tool-using runner. Tool use would mirror real Claude Code behavior more faithfully but adds non-determinism and runner complexity. Eager inline is simple, deterministic, and prompt caching is sufficient for now.
• Prune-on-write rather than purge-on-startup. Cache file accumulation comes from changing inputs (fixture content, skill hash) producing new keys. Pruning the previous logical-case entry on each write keeps the dashboard surface clean without a destructive global cleanup step.
• starterContent stored on each result. Reading the live fixture at render time tied diff correctness to the current fixture file. Persisting the starter alongside the answer makes diffs immutable to later edits.

Overall Flow

sequenceDiagram
  participant V as Vitest case
  participant R as Runner LLM
  participant T as tsc
  participant A as AST assertions
  participant S as Scorer LLM
  participant C as Cache
  V->>R: instruction + starter + full skill (SKILL.md + reference/*.md)
  R-->>V: modified config
  V->>T: typecheck against fixtures tsconfig
  T-->>V: errors / clean
  V->>A: evaluate structural assertions
  A-->>V: errors / clean
  alt tsc fail or assertion fail
    V->>C: write fail result, prune siblings
    V-->>V: ✗ FAIL [TSC | ASSERT]
  else clean
    V->>S: score correctness + completeness
    S-->>V: score + reasoning
    V->>C: write scored result, prune siblings
  end

Loading

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1214639058034041

[github-actions]

📦 esbuild Bundle Analysis for payload

This analysis was generated by esbuild-bundle-analyzer. 🤖
This PR introduced no changes to the esbuild bundle! 🙌