-
Notifications
You must be signed in to change notification settings - Fork 0
feat(pipeline): add --llm-scores flag to pipeline bench #790
Copy link
Copy link
Closed
Description
Objective
Add a --llm-scores <file> option to agentv pipeline bench so LLM scores can be provided via a file path instead of stdin only.
Problem
pipeline bench currently only accepts LLM scores via stdin. This causes issues on Windows PowerShell where the < operator is reserved:
ParserError: The '<' operator is reserved for future use.
The workaround (Get-Content file | agentv pipeline bench <dir>) is non-obvious and undocumented in the skill instructions.
Proposed Change
Add an optional --llm-scores flag to pipeline bench:
# Existing stdin approach (still works)
cat llm_scores.json | agentv pipeline bench <dir>
# New file flag approach
agentv pipeline bench <dir> --llm-scores llm_scores.jsonWhen --llm-scores is provided, read from that file instead of stdin. When omitted, fall back to stdin (backward compatible).
Implementation
In apps/cli/src/commands/pipeline/bench.ts:
- Add optional
--llm-scoresoption (type:string) - If provided, read from the file path; otherwise call
readStdin()as today
Acceptance Signals
-
agentv pipeline bench <dir> --llm-scores scores.jsonworks - Stdin still works when
--llm-scoresis omitted - Works on both Windows and Unix
Non-Goals
- Changing the scores JSON format
- Auto-discovery of scores files in the export directory
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Status
Done