Skip to content

Interactive eval TUI should list available models by provider #520

@christso

Description

@christso

Summary

The interactive TUI flow for running evals should list available models for the selected provider/target instead of requiring users to already know valid model strings.

Problem

When running evals interactively, users may want to switch the model for targets like Codex or Copilot, but AgentV currently expects them to know the correct model name ahead of time.

This causes a few issues:

  • trial and error with invalid model names
  • provider-specific confusion about whether a string is a model name or a reasoning preset
  • inconsistent UX compared with other interactive selection flows in the TUI

Proposed behavior

When the interactive TUI asks for a model override, AgentV should:

  • detect the selected provider/target type
  • query or enumerate the available models for that provider when possible
  • show a selectable list in the TUI
  • still allow manual entry as a fallback

Nice-to-have

  • indicate deprecated models
  • indicate recommended/default models
  • show reasoning-related settings separately from model selection when the provider supports them
  • cache provider model lists for a short period to keep the TUI responsive

Why this matters

This would make the interactive eval workflow much more discoverable and reduce provider-specific guesswork, especially for Codex, Copilot, and similar agent providers where supported models evolve over time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions