-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Description
Summary
The interactive TUI flow for running evals should list available models for the selected provider/target instead of requiring users to already know valid model strings.
Problem
When running evals interactively, users may want to switch the model for targets like Codex or Copilot, but AgentV currently expects them to know the correct model name ahead of time.
This causes a few issues:
- trial and error with invalid model names
- provider-specific confusion about whether a string is a model name or a reasoning preset
- inconsistent UX compared with other interactive selection flows in the TUI
Proposed behavior
When the interactive TUI asks for a model override, AgentV should:
- detect the selected provider/target type
- query or enumerate the available models for that provider when possible
- show a selectable list in the TUI
- still allow manual entry as a fallback
Nice-to-have
- indicate deprecated models
- indicate recommended/default models
- show reasoning-related settings separately from model selection when the provider supports them
- cache provider model lists for a short period to keep the TUI responsive
Why this matters
This would make the interactive eval workflow much more discoverable and reduce provider-specific guesswork, especially for Codex, Copilot, and similar agent providers where supported models evolve over time.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels