Skip to content

fix: suppress noisy 'file not found' warning for inline llm-grader prompts #901

@christso

Description

@christso

Problem

When an llm-grader assertion uses an inline prompt: string (not a file path), the resolver tries to resolve it as a file path first, fails, then falls back to using it as an inline prompt. This produces noisy warnings in CI logs:

Warning: Inline prompt used for evaluator 'tone_check' in 'greeting' (file not found: Evaluate whether the response has a professional and helpful tone.
Score 1.0 if the response is polite, clear, and professional.
...
)
  Tried: /home/runner/work/agentv/agentv/examples/features/default-evaluators/evals/Evaluate whether...
  Tried: /home/runner/work/agentv/agentv/examples/features/default-evaluators/Evaluate whether...
  Tried: /home/runner/work/agentv/agentv/examples/features/Evaluate whether...

The eval works correctly — the warning is cosmetic but clutters CI output, especially when running many evals.

Affected example

examples/features/default-evaluators/evals/dataset.eval.yaml — uses inline prompt: strings for llm-grader assertions.

Suggested fix

Detect inline prompts early (e.g., contains newlines or is longer than a reasonable file path) and skip the file resolution attempt. Only warn when the string looks like it could be a file path but wasn't found.

Metadata

Metadata

Assignees

Labels

in-progressClaimed by an agent — do not duplicate work

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions