feat(cli): incremental eval runs — resume, append, and aggregate

[christso]

Summary

Implements three related capabilities for incremental eval runs (#1071):

1. agentv eval aggregate <runDir> subcommand

A pure function: reads index.jsonl, deduplicates by (test_id, target) keeping the last entry, recomputes benchmark.json + timing.json, prints summary.

agentv eval aggregate .agentv/results/runs/2026-04-13T12-00-00/

2. --resume flag on agentv eval run

Resume an interrupted run: skip completed tests, append new results, aggregate at end.

agentv eval foo.yaml --output ./my-run/ --resume

3. --rerun-failed flag on agentv eval run

Rerun failed/errored tests while keeping passing results. Implies --resume.

agentv eval foo.yaml --output ./my-run/ --rerun-failed

Changes

• artifact-writer.ts: Add deduplicateByTestIdTarget(), aggregateRunDir(), writePerTestArtifacts()
• jsonl-writer.ts: Support append mode (flags: 'a')
• output-writer.ts: Pass append option through
• commands/aggregate.ts: New subcommand
• commands/run.ts: Add --resume and --rerun-failed flags
• run-eval.ts: Resume/rerun skip logic, append writer, aggregate after run
• aggregate.test.ts: 10 new tests for dedup, aggregate, and per-test artifact writing

Testing

• 472 existing tests pass (0 regressions)
• 10 new tests covering deduplication, aggregation, and per-test artifact writing
• Build and lint clean

Closes #1071

[cloudflare-workers-and-pages]

Deploying agentv with    Cloudflare Pages

Latest commit:	2202938
Status:	⚡️  Build in progress...

View logs

[christso]

E2E UAT Results

Red (main)

• agentv eval run --help on main: No --resume or --rerun-failed flags exist
• agentv eval --help on main: No aggregate subcommand exists
• No way to resume a partially-completed eval run

Green (branch)

1. Full baseline run — 3 tests, all pass (100%, 96%, 82%)

RESULT: PASS  (3/3 scored >= 80%, mean: 93%)
Artifact workspace written to: /tmp/uat-1071-e2e/baseline

2. Partial run + resume — ran 1/3 tests, then --resume completed remaining 2:

# Partial: ran only greeting
# Resume:
Resume: found 1 existing result(s), skipping 1 completed.
0/2   🔄 with-custom-eval | azure
0/2   🔄 skip-defaults | azure
RESULT: PASS  (3/3 scored >= 80%, mean: 97%)

3. Artifact parity — resumed run produces identical file structure to baseline:

# Both have identical structure:
benchmark.json, timing.json, index.jsonl, transcript.jsonl
+ 3 per-test dirs (grading.json, timing.json, input.md, response.md)

# Resumed benchmark.json has all 3 tests in tests_run ✅
# Resumed index.jsonl has exactly 3 entries (no duplicates) ✅

4. Compare compatibility — agentv compare works correctly on resumed output:

Comparing: baseline/index.jsonl → resumed/index.jsonl
  greeting              1.00       1.00     +0.00  = tie
  with-custom-eval      0.96       0.95     -0.01  = tie
  skip-defaults         0.82       0.95     +0.13  ✓ win
Summary: 1 win, 0 losses, 2 ties | Mean Δ: +0.040 | Status: improved

5. --rerun-failed — correctly reruns only quality_failure/execution_error tests:

# Modified greeting to quality_failure (score 0.3):
Rerun-failed: found 3 existing result(s), skipping 2 completed.
0/1   🔄 greeting | azure    # only reruns the failure
RESULT: PASS  (3/3 scored >= 80%, mean: 97%)

6. agentv eval aggregate — deduplicates and recomputes:

# 4 entries in JSONL (3 original + 1 rerun) → 3 after dedup
Aggregated 3 test result(s) across 1 target(s)
# greeting score = 1.0 (last-entry-wins), not 0.3 ✅
# pass_rate mean = 1 ✅

7. Edge case: all tests complete — exits cleanly:

Resume: found 3 existing result(s), skipping 3 completed.
Nothing to resume — all 3 test(s) already completed.

Verdict: ✅ PASS — all scenarios verified with live Azure OpenAI (gpt-5.4-mini)