Skip to content

feat(cli): add *.eval.ts auto-discovery#1120

Merged
christso merged 2 commits intomainfrom
feat/1116-eval-ts-discovery
Apr 16, 2026
Merged

feat(cli): add *.eval.ts auto-discovery#1120
christso merged 2 commits intomainfrom
feat/1116-eval-ts-discovery

Conversation

@christso
Copy link
Copy Markdown
Collaborator

Summary

Adds auto-discovery and execution of *.eval.ts files alongside YAML evals. Each TS file exports an EvalConfig and runs through evaluate() with the same output/artifact pipeline.

Closes #1116
Depends on #1119 (#1115)

Changes

New: TS Eval Loader (packages/core/src/evaluation/loaders/ts-eval-loader.ts)

  • Loads *.eval.ts via dynamic import() (native in Bun)
  • Tries exports in priority order: default, config, evalConfig
  • Clear error when no valid EvalConfig export found

Discovery (apps/cli/src/commands/eval/shared.ts)

  • Extension regex and auto-glob include .ts
  • Default patterns: **/evals/**/*.eval.ts added

Format Detection (packages/core/src/evaluation/loaders/jsonl-parser.ts)

  • detectFormat() returns 'typescript' for .ts/.mts

CLI Integration (apps/cli/src/commands/eval/run-eval.ts)

  • TS files separated from YAML files and processed via evaluate()
  • CLI options forwarded: --workers, --threshold, --filter, --cache, --verbose, --max-retries, --agent-timeout
  • Results flow through same artifact writer and reporter
  • Tag filtering: TS files skipped when --tag/--exclude-tag active (TS files don't carry YAML tags)
  • Boolean options correctly forwarded (uses !== undefined checks)

Tests

  • 5 unit tests for loadTsEvalFile (default export, named exports, error case)
  • Format detection tests for .ts extension
  • resolveEvalPaths tests for .eval.ts inclusion
  • All 475 tests pass

E2E Verification

Green (.eval.ts file discovered and executed):

$ bun apps/cli/src/cli.ts eval run greeting.eval.ts --dry-run
Running TS eval: greeting.eval.ts
RESULT: PASS  (1/1 scored >= 80%, mean: 100%)

Test Results

475/475 pass, 0 failures

@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages bot commented Apr 16, 2026

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 7864abe
Status: ✅  Deploy successful!
Preview URL: https://e0b657c9.agentv.pages.dev
Branch Preview URL: https://feat-1116-eval-ts-discovery.agentv.pages.dev

View logs

Base automatically changed from feat/1115-programmatic-api to main April 16, 2026 03:40
christso and others added 2 commits April 16, 2026 03:40
Add TypeScript eval file support to `agentv run`. TS eval files export
an EvalConfig (default, `config`, or `evalConfig` named export) and are
discovered alongside YAML files via the same glob/path resolution.

Changes:
- shared.ts: Include .ts in file extension regex and directory auto-glob
- config-loader.ts: Add **/evals/**/*.eval.ts to DEFAULT_EVAL_PATTERNS
- jsonl-parser.ts: Add typescript format detection in detectFormat()
- ts-eval-loader.ts: New loader that imports TS modules and extracts EvalConfig
- run-eval.ts: Integrate TS files through evaluate() with CLI overrides,
  feeding results through the same artifact/reporting pipeline
- run.ts: Update CLI description to mention .ts files
- index.ts: Export loadTsEvalFile and TsEvalResult from @agentv/core

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@christso christso force-pushed the feat/1116-eval-ts-discovery branch from 4c051a1 to 7864abe Compare April 16, 2026 03:42
@christso christso merged commit 840acbc into main Apr 16, 2026
4 checks passed
@christso christso deleted the feat/1116-eval-ts-discovery branch April 16, 2026 03:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add *.eval.ts auto-discovery so TS-first eval authors don't need a hand-wired runner

1 participant