Skip to content

feat(pipeline): add --llm-scores flag to pipeline bench #790

@christso

Description

@christso

Objective

Add a --llm-scores <file> option to agentv pipeline bench so LLM scores can be provided via a file path instead of stdin only.

Problem

pipeline bench currently only accepts LLM scores via stdin. This causes issues on Windows PowerShell where the < operator is reserved:

ParserError: The '<' operator is reserved for future use.

The workaround (Get-Content file | agentv pipeline bench <dir>) is non-obvious and undocumented in the skill instructions.

Proposed Change

Add an optional --llm-scores flag to pipeline bench:

# Existing stdin approach (still works)
cat llm_scores.json | agentv pipeline bench <dir>

# New file flag approach
agentv pipeline bench <dir> --llm-scores llm_scores.json

When --llm-scores is provided, read from that file instead of stdin. When omitted, fall back to stdin (backward compatible).

Implementation

In apps/cli/src/commands/pipeline/bench.ts:

  • Add optional --llm-scores option (type: string)
  • If provided, read from the file path; otherwise call readStdin() as today

Acceptance Signals

  • agentv pipeline bench <dir> --llm-scores scores.json works
  • Stdin still works when --llm-scores is omitted
  • Works on both Windows and Unix

Non-Goals

  • Changing the scores JSON format
  • Auto-discovery of scores files in the export directory

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions