-
Notifications
You must be signed in to change notification settings - Fork 0
fix: suppress noisy 'file not found' warning for inline llm-grader prompts #901
Copy link
Copy link
Closed
Labels
in-progressClaimed by an agent — do not duplicate workClaimed by an agent — do not duplicate work
Description
Problem
When an llm-grader assertion uses an inline prompt: string (not a file path), the resolver tries to resolve it as a file path first, fails, then falls back to using it as an inline prompt. This produces noisy warnings in CI logs:
Warning: Inline prompt used for evaluator 'tone_check' in 'greeting' (file not found: Evaluate whether the response has a professional and helpful tone.
Score 1.0 if the response is polite, clear, and professional.
...
)
Tried: /home/runner/work/agentv/agentv/examples/features/default-evaluators/evals/Evaluate whether...
Tried: /home/runner/work/agentv/agentv/examples/features/default-evaluators/Evaluate whether...
Tried: /home/runner/work/agentv/agentv/examples/features/Evaluate whether...
The eval works correctly — the warning is cosmetic but clutters CI output, especially when running many evals.
Affected example
examples/features/default-evaluators/evals/dataset.eval.yaml — uses inline prompt: strings for llm-grader assertions.
Suggested fix
Detect inline prompts early (e.g., contains newlines or is longer than a reasonable file path) and skip the file resolution attempt. Only warn when the string looks like it could be a file path but wasn't found.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
in-progressClaimed by an agent — do not duplicate workClaimed by an agent — do not duplicate work