fix(setup): configure and validate the backend model selector - #44
Merged
Conversation
`argus --setup --backend claude` persisted only backend, auth_mode and validated_version, so model resolution fell through to Argus's shared default `gpt-5.5` — an OpenAI-catalog id the Claude CLI rejects. Readiness checked that the CLI existed, ran and was authenticated, but never that the model id was one that CLI could serve, so `argus --doctor` reported "all blocking checks passed" on a machine where every call failed. The front door surfaces any non-zero backend exit as "[not dispatched] Manager could not classify this message", so the model name never reached the operator. Only the Pi path ever persisted a model; every other backend persisted none. - setup adopts a backend-appropriate model when the operator chose none, seeding it before readiness runs (mirroring the existing Pi branch) and persisting it with the validated profile - readiness rejects a model id that positively matches a foreign vendor catalog, hard-failing an id nobody chose and warning on one the operator set by hand (a private gateway may really serve it) - the doctor now prints readiness warnings instead of discarding them - docs/agent-install.md gains a model-selector step; an installing agent following it previously had no step that configured a model at all Backends whose CLI fronts several catalogs (copilot, qoder, pi, opencode) and dsh (whose selector goes through the ARGUS_DSH_* overlay) are exempt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Configures backend-specific model defaults during setup and validates model/backend catalog compatibility during readiness checks.
Changes:
- Persists adopted models and validates catalog compatibility.
- Displays readiness warnings through doctor diagnostics.
- Documents model-selector verification and adds regression tests.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
argus_skill/core/backend_readiness.py |
Adds model adoption, validation, and persistence. |
argus_skill/tools/setup.py |
Seeds adopted models before readiness checks. |
argus_skill/webapi/diagnostics.py |
Exposes readiness warnings. |
docs/agent-install.md |
Documents model verification. |
tests/core/test_backend_readiness.py |
Tests model validation and persistence. |
tests/tools/test_setup_readiness.py |
Updates setup test doubles. |
Suppressed comments (1)
argus_skill/core/backend_readiness.py:404
- A persisted value is no longer evidence that a human chose it:
persist_validated_profile()now writes setup's adopted model. After setting up Claude, then setting up Codex,claude-opus-5is read here aschosen, the foreign-catalog mismatch is downgraded to a warning, and setup succeeds while retaining that model because persisted writes merge. Codex calls then fail. Track whether setup owns the persisted model (and replace or hard-fail it on a backend switch) separately from genuinely operator-set values.
chosen = _explicit_model_selection(
role_env, env=env_map, persisted=persisted_map
)
if chosen:
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+256
to
261
| _MODEL_ROLES: tuple[tuple[str, str], ...] = ( | ||
| ("manager", "ARGUS_SKILL_MANAGER_MODEL"), | ||
| ("planner", "ARGUS_SKILL_PLAN_MODEL"), | ||
| ("engineer", "ARGUS_SKILL_ENGINEER_MODEL"), | ||
| ("reviewer", "ARGUS_SKILL_REVIEWER_MODEL"), | ||
| ) |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
argus --setup --non-interactive --backend claudecompletes,argus --doctorprintsall blocking checks passed, and then every single message comes back as:The model name never reaches the operator. It is only visible in
~/.argus-skill/cost-control.jsonl:{"provider":"claude","model":"gpt-5.5","run_label":"manager-frontdoor-classify", "error":"There's an issue with the selected model (gpt-5.5). It may not exist or you may not have access to it."}Why it happens
Three independent gaps line up:
Setup never persists a model for agent-CLI backends.
persist_validated_profilewritesARGUS_SKILL_RUNNER_BACKEND,ARGUS_SKILL_BACKEND_AUTH_MODE,ARGUS_SKILL_BACKEND_VALIDATED_VERSION— and nothing else. Only the Pi path (_persist_pi_profile) ever wrote a model. Resolution therefore falls through to the shared defaultgpt-5.5(core/knobs.py,capability_vault._DEFAULT_TEXT_MODEL), an OpenAI-catalog id. Perdocs/backend-providers.md, Argus sends it toclaudeverbatim.Readiness never validates the selector. It checks binary, version and auth. The only model validation was
_check_pi_model_routing, gated onprofile.backend == "pi". Reproduced on a working install:docs/agent-install.mdhas no step that configures a model. An agent following it end to end hits every checkpoint the doc defines and reports success truthfully.The generic error text comes from
life/router.py, which collapses any non-zero backend exit intofailure_sink("classifier backend failed");webapi/manager_dispatch.pythen fails closed.The change
persist_validated_profile(report, *, model="")— persists the model adopted for the backend. Newdefault_model_for_backend()returns an id only when the backend has a verified default and the operator chose nothing, so re-running setup never retunes a hand-configured machine.ARGUS_SKILL_MODEL— then persist it._check_backend_model_catalog()rejects a model id that positively matches a foreign vendor catalog. Severity splits on who chose it, matching the Pi precedent: an id nobody chose is a hard problem (Argus's own default, deterministic failure); an id the operator set by hand is a warning (a private gateway may really serve it). An id matching no known catalog is never second-guessed.docs/agent-install.mdgains a "Confirm the model selector" step usingargus --config-help(verified to exist and to print each knob's resolved value and source).Exempt by design:
copilotandqoder(their CLIs front several catalogs — Copilot resells Anthropic ids),pi/opencode(provider-agnostic; Pi keeps its real catalog probe), anddsh(its selector goes through theARGUS_DSH_*overlay, so judging the bare id would misfire).Only
claude → claude-opus-5is seeded, because that is the one id verified against a real CLI here. Other backends keep current behaviour and now get an actionable failure instead of a silent one; adding a default later is a one-line table entry.Tests
tests/core/test_backend_readiness.py— thegpt-5.5-on-claude regression, an accepted Anthropic id, operator-chosen foreign id warns instead of failing, unknown ids stay silent, multi-catalog backends exempt, and the adopted-model persistence.Two pre-existing test doubles stubbed
persist_validated_profilepositionally and were widened to accept the new keyword.test_grok_readiness_accepts_api_key_without_spending_a_turnconfigured no model at all, so under a hermetic home it resolved to the OpenAI default — exactly what this change now rejects; it sets agrok-id, which is what a real Grok install carries.ruffalso caught that the new interactive-setup line used a backslash escape inside an f-string, which is a syntax error on the project's 3.11 floor; it now uses literal glyphs like the surrounding code.Note for the maintainer
tests/core/test_release.py::test_release_manifest_matches_current_shipped_sourcefails, becausecompute_source_digestcoversargus_skill/**/*.pyandgenerate_manifestrefuses to update withoutbuild_releaserebuilding the shipped frontends. I did not commit a frontend rebuild to keep this diff reviewable — please regenerate the manifest as part of the release flow.Pre-existing failures unrelated to this change, confirmed by running them on a clean
main:tests/tools/test_ppt_master.py(5) andtests/webapi/test_pairing.py(5).🤖 Generated with Claude Code