tracking: AgentV feature parity roadmap

[christso]

Purpose

Parent issue to prioritize and orchestrate work across AgentV feature parity issues. Each issue below is designed for an agent to pick up and execute independently.

---

Priority 1: SDK Providers (AgentV differentiator)

Structured tool call extraction from coding agents is what sets AgentV apart. These are the highest-value issues.

• feat(core): add claude provider using @anthropic-ai/claude-agent-sdk #213 — Claude Agent SDK provider (replaces CLI subprocess)
• feat(core): add OpenAI Codex SDK provider replacing CLI subprocess #236 — OpenAI Codex SDK provider (replaces CLI subprocess, supersedes feat: add execution metrics support to Codex provider #99)

Dependencies: None. Can be worked in parallel.

Priority 2: Core Primitives

Essential infrastructure that cannot be implemented as plugins.

• feat: default case config (shared evaluators across cases) #225 — Default case config (shared evaluators across cases)
• feat: external dataset loading (CSV, JSONL, glob patterns) #226 — External dataset loading (file:// for YAML/JSONL + globs)
• feat: JUnit XML and JSON output formats #232 — JUnit XML and JSON output formats
• feat: LLM response caching across eval runs #234 — LLM response caching across eval runs
• feat: assert-set evaluator with threshold logic (N% must pass) #235 — Assert-set threshold mode on composite evaluator
• feat: evaluate multiple targets #179 — Matrix evaluation (multiple targets per eval)

Dependencies: #225 and #226 affect the YAML parser and should be coordinated. Others are independent.

Priority 3: Documentation & Examples

These use existing code_judge primitives. No core changes needed.

• docs: deterministic evaluator examples (contains, regex, JSON validation) #223 — Deterministic evaluator examples (contains, regex, JSON validation)
• docs: tool-call F1 scoring metric example #229 — Tool-call F1 scoring metric example
• docs: NLP metrics and semantic similarity evaluator examples (ROUGE, BLEU, similarity) #227 — NLP metrics examples (ROUGE, BLEU, semantic similarity)
• docs: trace-based evaluation examples for agent internals #231 — Trace-based evaluation examples

Dependencies: None. All independent. Good for parallel agent execution.

Priority 4: Showcases & Existing Issues

• Showcase: strengthen export-screening CI gating (multi-sample runs + stability-aware thresholds) #118 — Export-screening CI gating showcase
• Showcase: evaluator conformance harness (compatibility + consistency fixtures, CI gate) #119 — Evaluator conformance harness showcase
• feat: agent skill to convert chat conversations into eval cases #182 — Agent skill to convert chat conversations into eval cases
• feat: add execution metrics support to Pi Coding Agent provider #98 — Pi Coding Agent execution metrics
• Feat: TOON support #139 — TOON support

---

Execution Notes

• P1 issues follow the established provider pattern from PR feat(core): add copilot-sdk provider #211 (Copilot SDK). Agents should read that PR as reference.
• P2 issues each touch different parts of core. They can run in parallel except feat: default case config (shared evaluators across cases) #225/feat: external dataset loading (CSV, JSONL, glob patterns) #226 (both touch config loader).
• P3 issues are docs-only. Each creates example code judges + EVAL.yaml + README in examples/.
• All work follows the branch workflow in CLAUDE.md: git checkout -b <type>/<issue-number>-<desc> → PR → squash merge.

[christso]

Orchestration Guide (Final)

Blocker: #238 and #239 must land first

#238 — Rename cases to tests in EVAL schema and codebase
#239 — Rename eval-id CLI option to test-id

All feature issues touch types, YAML schema, CLI output, or examples affected by this rename. Agents: do not start feature work until #238 and #239 are merged to main.

Issue status

#	Title	Priority	Status	Notes
#213	Claude Agent SDK provider	P1	Ready	@anthropic-ai/claude-agent-sdk confirmed
#236	OpenAI Codex SDK provider	P1	Ready	@openai/codex-sdk confirmed
#225	Default case config	P2	Ready	Uses execution.evaluators at root (not defaults)
#226	External dataset loading	P2	Ready	file:// + YAML/JSONL + globs
#232	JUnit XML and JSON output	P2	Ready
#235	Assert-set threshold mode	P2	Ready	Lenient (borderline = pass) with warning
#179	Matrix evaluation	P2	Ready	execution.targets array, multi --target
#223	Deterministic evaluator examples	P3	Ready	Use context.criteria not expectedOutcome
#229	Tool-call F1 scoring example	P3	Ready
#227	NLP metrics examples	P3	Ready
#231	Trace-based evaluation examples	P3	Ready
#118	Export-screening CI gating	P4	Ready	Uses trials from #214 (landed)
#119	Evaluator conformance harness	P4	Ready
#182	Chat-to-eval skill	P4	Ready
#98	Pi execution metrics	P4	Ready
#234	LLM response caching	P2	Deferred	Benefits minimal for variability testing
#139	TOON support	P4	Closed	No adoption signal

Parallelization strategy

Fully independent — run in parallel:

• P1: feat(core): add claude provider using @anthropic-ai/claude-agent-sdk #213 and feat(core): add OpenAI Codex SDK provider replacing CLI subprocess #236 (both follow copilot-sdk pattern from PR feat(core): add copilot-sdk provider #211)
• P2: feat: JUnit XML and JSON output formats #232, feat: assert-set evaluator with threshold logic (N% must pass) #235 (different parts of core)
• P3: docs: deterministic evaluator examples (contains, regex, JSON validation) #223, docs: tool-call F1 scoring metric example #229, docs: NLP metrics and semantic similarity evaluator examples (ROUGE, BLEU, similarity) #227, docs: trace-based evaluation examples for agent internals #231 (all docs/examples, zero core changes)
• P4: feat: add execution metrics support to Pi Coding Agent provider #98, feat: agent skill to convert chat conversations into eval cases #182 (different code)

Must coordinate (touch same files):

• feat: default case config (shared evaluators across cases) #225 + feat: external dataset loading (CSV, JSONL, glob patterns) #226 — both modify yaml-parser.ts. Run feat: default case config (shared evaluators across cases) #225 first (simpler), then feat: external dataset loading (CSV, JSONL, glob patterns) #226.

Complex — run after simpler P2:

• feat: evaluate multiple targets #179 — touches runner, YAML parser, CLI args, output. Start after feat: default case config (shared evaluators across cases) #225/feat: external dataset loading (CSV, JSONL, glob patterns) #226.

Recommended single-agent order

1. feat: add execution metrics support to Pi Coding Agent provider #98 (Pi metrics — smallest scope)
2. feat: assert-set evaluator with threshold logic (N% must pass) #235 (threshold mode — small composite.ts change)
3. feat: default case config (shared evaluators across cases) #225 (default evaluators — YAML parser)
4. feat: external dataset loading (CSV, JSONL, glob patterns) #226 (external dataset loading — YAML parser)
5. feat(core): add claude provider using @anthropic-ai/claude-agent-sdk #213 (Claude SDK provider)
6. feat(core): add OpenAI Codex SDK provider replacing CLI subprocess #236 (Codex SDK provider)
7. feat: JUnit XML and JSON output formats #232 (JUnit/JSON output)
8. feat: evaluate multiple targets #179 (matrix evaluation — most complex)
9. docs: deterministic evaluator examples (contains, regex, JSON validation) #223, docs: tool-call F1 scoring metric example #229, docs: NLP metrics and semantic similarity evaluator examples (ROUGE, BLEU, similarity) #227, docs: trace-based evaluation examples for agent internals #231 (docs/examples — parallel)
10. Showcase: strengthen export-screening CI gating (multi-sample runs + stability-aware thresholds) #118, Showcase: evaluator conformance harness (compatibility + consistency fixtures, CI gate) #119, feat: agent skill to convert chat conversations into eval cases #182 (showcases/skills)

Codebase conventions (from CLAUDE.md)

• Runtime: Bun (never npm)
• Test locally: bun agentv run ... from repo root
• Branch: feat/<issue>-<desc> or docs/<issue>-<desc>
• Commit: feat(scope): / fix(scope): / docs(scope):
• PR workflow: branch → squash merge to main
• Verify: bun run typecheck && bun run lint && bun run test && bun run build

[christso]

Roadmap Complete ✅

All 16 issues have been implemented, reviewed, and merged:

P1: SDK Providers

• feat(core): add claude provider using @anthropic-ai/claude-agent-sdk #213 → PR feat(providers): replace claude-code CLI with claude Agent SDK provider (#213) #246 ✅ Claude Agent SDK
• feat(core): add OpenAI Codex SDK provider replacing CLI subprocess #236 → PR feat(codex): replace CLI subprocess with @openai/codex-sdk provider (#236) #247 ✅ Codex SDK

P2: Core Primitives

• feat: default case config (shared evaluators across cases) #225 → PR feat(core): default evaluators merged across tests (#225) #244 ✅ Default evaluators
• feat: external dataset loading (CSV, JSONL, glob patterns) #226 → PR feat(core): add external dataset loading with file:// references (#226) #245 ✅ External dataset loading
• feat: JUnit XML and JSON output formats #232 → PR feat(cli): add JSON and JUnit XML output formats (#232) #248 ✅ JUnit/JSON output
• feat: LLM response caching across eval runs #234 → PR feat(core): add LLM response caching with --no-cache flag (#234) #258 ✅ LLM response caching
• feat: assert-set evaluator with threshold logic (N% must pass) #235 → PR feat(core): add threshold aggregation mode to composite evaluator (#235) #243 ✅ Threshold aggregation
• feat: evaluate multiple targets #179 → PR feat(eval): add matrix evaluation — evaluate multiple targets (#179) #257 ✅ Matrix evaluation
• feat: add execution metrics support to Pi Coding Agent provider #98 → PR feat(pi-coding-agent): add execution metrics support (#98) #242 ✅ Pi execution metrics

P3: Documentation & Examples

• docs: deterministic evaluator examples (contains, regex, JSON validation) #223 → PR docs(examples): add deterministic evaluator examples (#223) #249 ✅ Deterministic evaluator examples
• docs: tool-call F1 scoring metric example #229 → PR docs(examples): add tool-call F1 scoring example (#229) #250 ✅ Tool-call F1 scoring
• docs: NLP metrics and semantic similarity evaluator examples (ROUGE, BLEU, similarity) #227 → PR docs(examples): add NLP metrics evaluator examples (#227) #251 ✅ NLP metrics examples
• docs: trace-based evaluation examples for agent internals #231 → PR docs(examples): add trace-based evaluation examples (#231) #252 ✅ Trace-based evaluation

P4: Showcases & Skills

• Showcase: strengthen export-screening CI gating (multi-sample runs + stability-aware thresholds) #118 → PR feat(showcase): strengthen export-screening CI gating (#118) #256 ✅ Export-screening CI gating
• Showcase: evaluator conformance harness (compatibility + consistency fixtures, CI gate) #119 → PR feat(showcase): add evaluator conformance harness (#119) #254 ✅ Evaluator conformance harness
• feat: agent skill to convert chat conversations into eval cases #182 → PR feat(skill): add chat-to-eval conversation converter (#182) #255 ✅ Chat-to-eval skill

Deferred/Closed

• Feat: TOON support #139 TOON support — Closed (no adoption signal)