feat: external dataset loading (CSV, JSONL, glob patterns)

[christso]

Problem

AgentV only supports inline case definitions in EVAL.yaml. Large eval suites (100+ cases) and datasets from external sources need file-based loading.

Proposal

Support file:// references for loading cases from external files:

name: large-eval-suite
version: "1.0"

cases:
  - id: inline-test
    input: "Hello"

  - file://cases/accuracy-tests.yaml
  - file://cases/regression.jsonl

Supported Formats

• YAML — native, same schema as inline cases
• JSONL — one case per line, standard for large datasets

Glob Patterns

cases:
  - file://cases/**/*.yaml

Why Not CSV, Google Sheets, Combinatorial Expansion?

Per design principle #1 (Lightweight Core), these are niche features:

• CSV/Google Sheets → convert to YAML/JSONL with an external script
• Combinatorial expansion → generate cases with a script, output JSONL

The core should load files, not transform data.

Design Principles Alignment

• ✅ Lightweight Core — just file loading, no data transformation
• ✅ Non-Breaking Extension — file:// is optional, inline cases unchanged
• ✅ Industry Standard — file:// references are universal

Acceptance Criteria

• file:// references resolve YAML and JSONL files
• Glob patterns discover and load multiple files
• Clear error messages for missing/malformed files
• Unit tests
• Documentation

[christso]

Agent-Ready Implementation Plan

Branch

git checkout -b feat/226-external-dataset-loading

Reference files to read first

1. packages/core/src/evaluation/yaml-parser.ts — Primary file to modify. Contains loadEvalSuite(), loadEvalCases(), case iteration, detectFormat()
2. packages/core/src/evaluation/types.ts — EvalCase type definitions
3. packages/core/src/evaluation/loaders/config-loader.ts — Config loading patterns
4. packages/core/test/evaluation/yaml-parser.test.ts — Parser tests
5. packages/core/package.json — Check if fast-glob or micromatch already available (micromatch IS a dep)

Step-by-step implementation

Step 1: Create file loader utility

Create packages/core/src/evaluation/loaders/case-file-loader.ts:

import { parse as parseYaml } from 'yaml';
import { readFile } from 'fs/promises';
import { resolve, dirname } from 'path';

export interface LoadedCase {
  // Same shape as inline case from YAML
  [key: string]: unknown;
}

/**
 * Load cases from a file:// reference.
 * Supports .yaml, .yml, .jsonl formats.
 * Supports glob patterns via micromatch.
 */
export async function loadCasesFromFile(
  fileRef: string,
  basePath: string
): Promise<LoadedCase[]> {
  // Strip 'file://' prefix
  const filePath = fileRef.replace(/^file:\/\//, '');

  // Check if it's a glob pattern
  if (filePath.includes('*') || filePath.includes('?')) {
    return loadCasesFromGlob(filePath, basePath);
  }

  const resolvedPath = resolve(basePath, filePath);
  return loadSingleFile(resolvedPath);
}

async function loadSingleFile(filePath: string): Promise<LoadedCase[]> {
  const content = await readFile(filePath, 'utf-8');

  if (filePath.endsWith('.jsonl')) {
    return content.split('\n')
      .filter(line => line.trim())
      .map(line => JSON.parse(line));
  }

  if (filePath.endsWith('.yaml') || filePath.endsWith('.yml')) {
    const parsed = parseYaml(content);
    // YAML file should contain an array of cases or a single case
    return Array.isArray(parsed) ? parsed : [parsed];
  }

  throw new Error(`Unsupported file format: ${filePath}. Use .yaml, .yml, or .jsonl`);
}

async function loadCasesFromGlob(
  pattern: string,
  basePath: string
): Promise<LoadedCase[]> {
  const { default: micromatch } = await import('micromatch');
  const { readdir } = await import('fs/promises');
  // Use fast-glob or manual recursive glob
  const fg = await import('fast-glob');
  const files = await fg.default(pattern, { cwd: basePath, absolute: true });

  const allCases: LoadedCase[] = [];
  for (const file of files.sort()) {
    const cases = await loadSingleFile(file);
    allCases.push(...cases);
  }
  return allCases;
}

Note: Check if fast-glob is already a dependency. If not, use the glob module from Node.js or Bun.glob(). apps/cli/package.json already has fast-glob as a dependency — check if core also has it or can use it.

Step 2: Update packages/core/src/evaluation/yaml-parser.ts

In the function that processes the cases array from YAML:

1. Detect file:// entries: When iterating cases, check if an entry is a string starting with file://
2. Load external cases: Call loadCasesFromFile() and splice results into the cases array
3. Mix inline + external: Support both inline case objects and file:// strings in the same array

// In the cases processing loop:
const resolvedCases: EvalCase[] = [];
for (const entry of rawCases) {
  if (typeof entry === 'string' && entry.startsWith('file://')) {
    const loaded = await loadCasesFromFile(entry, evalFileDir);
    resolvedCases.push(...loaded.map(processCaseEntry));
  } else {
    resolvedCases.push(processCaseEntry(entry));
  }
}

Step 3: Error handling

• Missing file: Error: External case file not found: cases/missing.yaml (referenced from evals/dataset.yaml)
• Malformed JSONL: Error: Invalid JSON on line 3 of cases/data.jsonl: <parse error>
• Empty file: Warning, skip silently
• Glob matches nothing: Warning, not an error

Step 4: Add unit tests

In packages/core/test/evaluation/yaml-parser.test.ts or new file packages/core/test/evaluation/loaders/case-file-loader.test.ts:

• Test: load cases from .yaml file
• Test: load cases from .jsonl file
• Test: glob pattern resolves multiple files
• Test: mixed inline + file:// cases
• Test: missing file throws clear error
• Test: malformed JSONL throws clear error
• Test: file:// path resolved relative to eval file directory

Step 5: Add example

Create examples/features/external-datasets/:

evals/
  dataset.yaml          # Main eval with file:// refs
  cases/
    accuracy.yaml       # External YAML cases
    regression.jsonl    # External JSONL cases
README.md

dataset.yaml:

name: external-dataset-demo
version: "1.0"

cases:
  - id: inline-case
    input: "Hello"
    expected_outcome: "A greeting response"

  - file://cases/accuracy.yaml
  - file://cases/regression.jsonl

cases/accuracy.yaml:

- id: accuracy-1
  input: "What is 2+2?"
  expected_outcome: "4"
- id: accuracy-2
  input: "What is the capital of France?"
  expected_outcome: "Paris"

cases/regression.jsonl:

{"id": "reg-1", "input": "edge case input", "expected_outcome": "expected output"}
{"id": "reg-2", "input": "another edge case", "expected_outcome": "expected output 2"}

Verification checklist

bun run typecheck
bun run lint
bun run test
bun run build

Coordination note

This touches the YAML parser alongside #225 (default case config). The two features are independent: #226 handles HOW cases are loaded (inline vs file), #225 handles WHAT happens to loaded cases (default evaluators). Ensure file-loaded cases also receive defaults from #225.