feat: default case config (shared evaluators across cases)

[christso]

Problem

When writing EVAL.yaml with many cases, common evaluators (latency, safety, format) must be repeated on every case.

Proposal

Add a defaults block that applies to all cases unless overridden:

name: customer-support-eval
version: "1.0"

defaults:
  evaluators:
    - name: latency_check
      type: latency
      max_ms: 5000
    - name: safety
      type: llm_judge
      prompt: evaluators/no-ai-disclosure.md

cases:
  - id: refund-request
    input: "I want a refund"
    evaluators:
      - name: helpful
        type: llm_judge
        prompt: evaluators/helpful.md
    # Also gets the two default evaluators

  - id: greeting
    input: "Hello"
    # Only gets default evaluators

Merge Behavior

• Default evaluators are appended to case-specific evaluators
• To skip defaults for a case: skip_defaults: true

Why Not $ref or Templates?

YAML already has anchors (&anchor / *anchor) for reuse. Adding $ref or a custom template system introduces new concepts without adding capability. Keep the config surface minimal.

Design Principles Alignment

• ✅ Non-Breaking Extension — defaults is optional; existing configs unchanged
• ✅ Lightweight Core — minimal parser change, no new concepts
• ✅ AI-First — less repetition = fewer tokens for AI to reason about

Acceptance Criteria

• defaults.evaluators parsed and appended to all cases
• skip_defaults: true per case to opt out
• Backward compatible
• Unit tests
• Documentation

[christso]

Agent-Ready Implementation Plan (Final)

Design change: Use execution.evaluators at the root level — NOT a defaults block. This follows the existing pattern where root execution.target and execution.trials already serve as file-level defaults.

Branch

git checkout -b feat/225-default-case-config

Reference files to read first

1. packages/core/src/evaluation/yaml-parser.ts — Primary file to modify. Contains loadEvalSuite(), loadEvalCases(), case iteration logic
2. packages/core/src/evaluation/types.ts — EvalSuite, EvalCase, EvaluatorConfig types
3. packages/core/src/evaluation/loaders/config-loader.ts — Config loading pattern
4. packages/core/test/evaluation/yaml-parser.test.ts — Existing parser tests
5. examples/features/composite/evals/dataset.yaml — Current per-case execution.evaluators pattern
6. examples/features/trials/evals/dataset.yaml — Root execution.trials pattern (precedent for file-level defaults)

YAML schema

name: customer-support-eval
version: "1.0"

execution:
  target: default
  evaluators:                    # NEW — file-level default evaluators
    - name: latency_check
      type: latency
      max_ms: 5000
    - name: safety
      type: llm_judge
      prompt: evaluators/no-ai-disclosure.md

cases:
  - id: refund-request
    input: "I want a refund"
    execution:
      evaluators:                # Per-case evaluators
        - name: helpful
          type: llm_judge
          prompt: evaluators/helpful.md
      # Also gets latency_check + safety from root

  - id: greeting
    input: "Hello"
    # Only gets root-level evaluators (latency_check + safety)

  - id: special-case
    input: "Urgent"
    execution:
      skip_defaults: true        # Opt out of root-level evaluators
      evaluators:
        - name: urgency
          type: llm_judge
          prompt: evaluators/urgency.md

Merge behavior

• Root execution.evaluators are appended to per-case evaluators (case-specific run first)
• skip_defaults: true on a case's execution block opts out
• If root has no evaluators → no-op (backward compatible)

Step-by-step implementation

Step 1: Update YAML type definitions

In the types file where the raw YAML schema lives:

• Allow evaluators on the root execution block (currently only per-case)
• Add skip_defaults?: boolean to the per-case execution schema

Step 2: Update packages/core/src/evaluation/yaml-parser.ts

In the function that processes cases:

1. Extract root evaluators from suite.execution.evaluators
2. For each case, after building the case's own evaluator list:
   • If case.execution.skip_defaults !== true AND root evaluators exist:
   • Append root evaluators to the case's list

   const rootEvaluators = suite.execution?.evaluators ?? [];
   for (const evalCase of cases) {
     if (!evalCase.execution?.skip_defaults && rootEvaluators.length > 0) {
       evalCase.evaluators = [...(evalCase.evaluators ?? []), ...rootEvaluators];
     }
   }

Step 3: Add Zod validation (if applicable)

Add evaluators to the root execution schema and skip_defaults to per-case execution schema.

Step 4: Add unit tests

In packages/core/test/evaluation/yaml-parser.test.ts:

• Test: root evaluators appended to all cases
• Test: skip_defaults: true opts out
• Test: case with own evaluators + root → both present (case-specific first)
• Test: no root evaluators → existing behavior unchanged
• Test: case with no evaluators → gets only root defaults

Step 5: Add example

Create examples/features/default-evaluators/ with evals/dataset.yaml and README.md.

Verification checklist

bun run typecheck
bun run lint
bun run test
bun run build

Coordination note

This touches the YAML parser alongside #226 (external dataset loading). The two are independent: #226 handles HOW cases are loaded, #225 handles WHAT evaluators they get. File-loaded cases should also receive root evaluators.