test(evals): variant filter, structured transcripts, skill nudge

[denolfe]

Overview

Improves the evals dashboard and the Claude Code agent runner so we can see, at a glance, whether a run actually exercised the Payload skill, what the agent did, and how much that run cost.

Key Changes

• Dashboard filter cleanup

  • Removed the legacy all / users / admins / maintainers audience filter row from the results table (audience UI remains in the Compare view).
  • Added a Variant filter strip driven by the shared getVariant(result) classifier, with buttons derived from the variants present in the loaded entries (Agent Baseline, Agent Skill, Baseline, Skill).

• Structured Claude Code transcripts

  • Runner now invokes claude --print --output-format stream-json --verbose and parses the NDJSON event stream.
  • Persists a typed TranscriptEvent[] on CodegenRunnerResult and EvalResult covering text, thinking, tool_use, and tool_result blocks.
  • Per-event content truncated to ~4k chars; total events capped at 200 (first 100 + last 100 + marker) to bound cache size.
  • Token usage is extracted from the stream-json result event and populated on the runner result (no more zeros for agent runs).
  • Stderr is captured separately and surfaces in agentLog only when non-empty, preserving timeout and spawn-error diagnostics.

• Transcript UI in the expanded row

  • New TranscriptView renders the event timeline with collapsible blocks for tool_use, tool_result, and thinking.
  • Tool result errors render in red; tool calls show as → name, results as ← result.
  • The Transcript section opens by default, as do its inner blocks.
  • Falls back to the existing plaintext <pre> when only agentLog is available (legacy cache entries).

• Skill invocation visibility

  • A badge at the top of each transcript indicates whether the agent invoked the Skill tool (green when invoked, red when not), using v4 design tokens so it renders in both themes.
  • Agent runner appends a system prompt directing the agent to invoke the payload skill, gated on skillInstall === 'embedded' so the agent-baseline lane stays untouched.
  • That directive is part of the codegen cache key, so future tweaks to it auto-invalidate cached agent-skill results.

Design Decisions

Plaintext first, structured second. The first pass at transcripts kept the existing agentLog plaintext and rendered it in a <pre>. That only showed the final assistant message because --print strips tool calls. We then upgraded to stream-json with a typed event union, but kept agentLog on the type as an optional fallback so older cache entries still render and stderr has somewhere to go.

Shared variant classifier. The dashboard's per-row Variant value uses the getVariant(result) helper from test/evals/variant.ts rather than re-inferring from systemPromptKey/modelId in the table. The Variant filter buttons derive their option set from entries, so new variants appear without further changes.

System-prompt nudge over skill description change. The payload skill description was permissive ("Use when working with Payload CMS projects"), so the agent often skipped it. Rather than rewrite the skill description (which would affect production users), the runner appends a --append-system-prompt directive on the agent-skill lane only. This preserves a clean A/B with agent-baseline.

Cache invariance. The cache key already factored skillInstall and skillHash. The new directive is included as well so any future copy change forces a fresh run.

Audience stripped from list view only. EvalEntry.audience and audience.ts remain because CompareTable still uses them. Scope was kept to the dashboard list as requested.

Overall Flow

sequenceDiagram
    participant Test as vitest eval case
    participant Runner as runCodegenEval
    participant Claude as claude CLI
    participant Cache as cache.ts
    participant UI as EvalDashboard

    Test->>Runner: run(instruction, fixture, opts)
    Runner->>Cache: codegenKey({ ..., agentSystemPrompt })
    Cache-->>Runner: cached EvalResult?
    alt cache hit
        Runner-->>Test: cached result
    else cache miss (agent-skill)
        Runner->>Claude: --output-format stream-json --append-system-prompt
        Claude-->>Runner: NDJSON (system, assistant, user, result)
        Runner->>Runner: parse to TranscriptEvent[] and usage
        Runner->>Cache: persist EvalResult with transcript, usage, agentLog?
        Runner-->>Test: result
    end
    UI->>UI: read cache, flatten to EvalEntry[]
    UI->>UI: filter by Variant button
    UI->>UI: ExpandedRow renders TranscriptView and SkillInvocationBadge

Loading

[github-actions]

📦 esbuild Bundle Analysis for payload

This analysis was generated by esbuild-bundle-analyzer. 🤖
This PR introduced no changes to the esbuild bundle! 🙌