Skip to content

A bare backend name was read as a model name, and the CLI's failures … - #56

Merged
Shashankss1205 merged 1 commit into
mainfrom
fix/backend-name-as-model-spec
Aug 3, 2026
Merged

A bare backend name was read as a model name, and the CLI's failures …#56
Shashankss1205 merged 1 commit into
mainfrom
fix/backend-name-as-model-spec

Conversation

@Shashankss1205

Copy link
Copy Markdown
Collaborator

…said nothing

Two defects that hid each other. Both found by installing 0.1.2 from PyPI into a clean venv and driving it against a real claude v2.1.220.

split_spec only consulted BACKENDS when the spec contained a slash. A bare backend name fell through to "assume the default backend, keep the whole string as the model", so:

split_spec("claude-cli") -> ("claude-cli", "claude-cli")
split_spec("mock")       -> ("claude-cli", "mock")

--model claude-cli therefore shelled out to claude -p --model claude-cli and was refused by the CLI on every call (3/3), while grapharc models --check went on reporting the backend usable and grapharc models claude-cli printed model: claude-cli without complaint — two commands whose job is to say whether a spec will work, both saying yes about one that never did.

--model mock was the worse half: it named the paid subscription backend and spawned the real binary, so the double models --check describes as "scripted test double; never reaches a provider" reached for one. It happened to fail before billing only because mock is not a model name.

This is the same "silently folded into a model name … fails much later with a confusing error" failure split_spec already refuses for a mistyped backend with a slash; it just could not see the case without one.

A bare backend name now resolves to that backend. claude-cli takes its own default model and mock takes the scripted double (which ignores the model segment entirely). openrouter, openai and ollama front catalogues rather than a model, so a bare name there is refused with a spelling that works rather than a guess about what to bill someone for. Slash forms and bare model names are untouched.

The gateway read the wrong stream. claude -p fails with a non-zero exit, an empty stderr, and its whole explanation in the JSON envelope on stdout:

{"is_error": true, "result": "There's an issue with the selected model
 (claude-cli). It may not exist or you may not have access to it."}

_invoke_cli reported proc.stderr, so the error was claude -p exited 1: — a sentence that stops at the colon. That is how the bug above presented itself: as no message at all. The one string that would have diagnosed it in seconds was captured, held in proc.stdout, and discarded. Note the code already parses this shape a few lines below, but only on the returncode == 0 path, and the CLI sets is_error and exits non-zero, so it took the branch that ignores stdout.

stdout is read first now, falling back to stderr when it is not the documented envelope. The recovered text also feeds _cli_failure, which classifies transient-vs-deterministic and was previously classifying from "".

Tests: every new test was confirmed to fail against the old code and pass against the new — including the mock one, which asserts no subprocess is created rather than just checking the returned type, since the type was what was wrong and a refactor could fix the type and still shell out. BARE_BACKEND_MODEL["claude-cli"] is a second copy of the model class's default so the registry need not import a backend to split a string, and a test pins the two together the way CI already pins __version__ to the packaged version.

Verified: grapharc demo stage1 --model claude-cli now completes (8 nodes, target_met, exit 0) where it previously died with an empty error; --model mock resolves to the double with no subprocess; --model openrouter exits 2 with an example. Full suite green on 3.12 and 3.13; ruff clean.

…said nothing

Two defects that hid each other. Both found by installing 0.1.2 from PyPI into a
clean venv and driving it against a real `claude` v2.1.220.

**`split_spec` only consulted BACKENDS when the spec contained a slash.** A bare
backend name fell through to "assume the default backend, keep the whole string
as the model", so:

    split_spec("claude-cli") -> ("claude-cli", "claude-cli")
    split_spec("mock")       -> ("claude-cli", "mock")

`--model claude-cli` therefore shelled out to `claude -p --model claude-cli` and
was refused by the CLI on every call (3/3), while `grapharc models --check` went
on reporting the backend `usable` and `grapharc models claude-cli` printed
`model: claude-cli` without complaint — two commands whose job is to say whether
a spec will work, both saying yes about one that never did.

`--model mock` was the worse half: it named the *paid* subscription backend and
spawned the real binary, so the double `models --check` describes as "scripted
test double; never reaches a provider" reached for one. It happened to fail
before billing only because `mock` is not a model name.

This is the same "silently folded into a model name … fails much later with a
confusing error" failure `split_spec` already refuses for a mistyped backend
*with* a slash; it just could not see the case without one.

A bare backend name now resolves to that backend. `claude-cli` takes its own
default model and `mock` takes the scripted double (which ignores the model
segment entirely). `openrouter`, `openai` and `ollama` front catalogues rather
than a model, so a bare name there is refused with a spelling that works rather
than a guess about what to bill someone for. Slash forms and bare *model* names
are untouched.

**The gateway read the wrong stream.** `claude -p` fails with a non-zero exit, an
empty stderr, and its whole explanation in the JSON envelope on stdout:

    {"is_error": true, "result": "There's an issue with the selected model
     (claude-cli). It may not exist or you may not have access to it."}

`_invoke_cli` reported `proc.stderr`, so the error was `claude -p exited 1: ` —
a sentence that stops at the colon. That is how the bug above presented itself:
as no message at all. The one string that would have diagnosed it in seconds was
captured, held in `proc.stdout`, and discarded. Note the code already parses this
shape a few lines below, but only on the `returncode == 0` path, and the CLI sets
`is_error` *and* exits non-zero, so it took the branch that ignores stdout.

stdout is read first now, falling back to stderr when it is not the documented
envelope. The recovered text also feeds `_cli_failure`, which classifies
transient-vs-deterministic and was previously classifying from `""`.

Tests: every new test was confirmed to fail against the old code and pass against
the new — including the mock one, which asserts no subprocess is *created* rather
than just checking the returned type, since the type was what was wrong and a
refactor could fix the type and still shell out. `BARE_BACKEND_MODEL["claude-cli"]`
is a second copy of the model class's default so the registry need not import a
backend to split a string, and a test pins the two together the way CI already
pins `__version__` to the packaged version.

Verified: `grapharc demo stage1 --model claude-cli` now completes (8 nodes,
target_met, exit 0) where it previously died with an empty error; `--model mock`
resolves to the double with no subprocess; `--model openrouter` exits 2 with an
example. Full suite green on 3.12 and 3.13; ruff clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Shashankss1205
Shashankss1205 merged commit da8825f into main Aug 3, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants