crossmodel is v0.1.0: it works end to end, but it is early and the API may move. This issue is the open roadmap — comments, disagreement, and PRs are all welcome.
Where it stands
| Piece |
State |
Provider registry (cli + OpenAI-compatible http) |
works |
crossmodel CLI |
works; exit 2 on failure so an error is never mistaken for an answer |
delegate subagent |
works |
#route hook + routing.json policy |
works |
Benchmark: code, review, format, classify |
works |
Verified against Codex (gpt-5.6-*) and Claude Code headless (opus/sonnet/haiku).
The gemini, ollama, openrouter and openai-compatible entries are written but
not yet exercised by anyone — see below.
1. Confirm the untested providers
gemini, ollama, openrouter and the generic openai-compatible transport are
implemented and unverified. Each needs one real call plus a benchmark run.
If you use any of them, running node bench/battery.mjs --models <your-alias> --suites format
and pasting the result is the single most useful contribution right now. Argument shapes
drift between CLI versions and I would rather fix a wrong args() than ship a
plausible-looking one.
OpenRouter is the highest-leverage one — one key, hundreds of models, so it turns
this from "a few providers" into "most of them".
2. An MCP mode, to drop the subagent middleman
Today a delegated call costs quota twice: the delegate subagent is itself a Claude
model that builds the prompt, shells out, and filters the reply.
Exposing the registry as an MCP server would make crossmodel(model, prompt) a plain
tool call — one round-trip, no middleman. Strictly better for small structured outputs.
The subagent would stay useful for large outputs, where isolating the raw response from
the main context is the actual win.
Trade-off: a server to maintain versus zero infrastructure today.
3. A benchmark suite for work inside a real codebase
The honest gap. Every task shipped here is self-contained: full spec in the prompt,
no repository, no architecture decision, tests already written.
That is a real class of work, and it is the class that offloads well — but it is not
what most people mean by "can this model code". A suite that measures implement this
change inside an existing project would be far more useful and is much harder to score
deterministically. Ideas welcome; this is a design problem before it is a coding one.
4. More task sets, and better ones
- More
review samples. Twelve is enough to find a floor, not to rank.
- Non-JavaScript targets — Python and Go at minimum. The
code suite currently assumes
node --test; the runner needs a per-language verifier hook.
- Sharper "clean" samples in
review. Ours were not clean enough (see below), which
makes the false-positive metric unreliable today. Do not trust that column yet.
5. Quota introspection
There is no way to ask "how much of my window is left" before firing a batch. If a
provider exposes it, surfacing it in --list would prevent a long run from dying
halfway through.
Two things worth knowing before you trust any number here
A benchmark score measures the pair model + verifier, not the model. The code
suite runs near-ceiling for most models precisely because node --test runs underneath.
Remove the verifier and the number tells you nothing. This is why the delegate agent
refuses work that nothing downstream can check.
Our own answer keys were wrong four times, and the models caught all four. One
flagged a genuine month-end overflow bug in a sample we had labelled "clean" — it was
right and we were wrong. When every model disagrees with your key, suspect the key.
Same discipline applies to any result posted in this thread, including mine.
Contributing
No process yet beyond: open an issue, or send a PR. Adding a provider is a data change —
see crossmodel.config.example.json — and does not require touching the transport code.
crossmodelis v0.1.0: it works end to end, but it is early and the API may move. This issue is the open roadmap — comments, disagreement, and PRs are all welcome.Where it stands
cli+ OpenAI-compatiblehttp)crossmodelCLI2on failure so an error is never mistaken for an answerdelegatesubagent#routehook +routing.jsonpolicycode,review,format,classifyVerified against Codex (
gpt-5.6-*) and Claude Code headless (opus/sonnet/haiku).The
gemini,ollama,openrouterandopenai-compatibleentries are written butnot yet exercised by anyone — see below.
1. Confirm the untested providers
gemini,ollama,openrouterand the genericopenai-compatibletransport areimplemented and unverified. Each needs one real call plus a benchmark run.
If you use any of them, running
node bench/battery.mjs --models <your-alias> --suites formatand pasting the result is the single most useful contribution right now. Argument shapes
drift between CLI versions and I would rather fix a wrong
args()than ship aplausible-looking one.
OpenRouter is the highest-leverage one — one key, hundreds of models, so it turns
this from "a few providers" into "most of them".
2. An MCP mode, to drop the subagent middleman
Today a delegated call costs quota twice: the
delegatesubagent is itself a Claudemodel that builds the prompt, shells out, and filters the reply.
Exposing the registry as an MCP server would make
crossmodel(model, prompt)a plaintool call — one round-trip, no middleman. Strictly better for small structured outputs.
The subagent would stay useful for large outputs, where isolating the raw response from
the main context is the actual win.
Trade-off: a server to maintain versus zero infrastructure today.
3. A benchmark suite for work inside a real codebase
The honest gap. Every task shipped here is self-contained: full spec in the prompt,
no repository, no architecture decision, tests already written.
That is a real class of work, and it is the class that offloads well — but it is not
what most people mean by "can this model code". A suite that measures implement this
change inside an existing project would be far more useful and is much harder to score
deterministically. Ideas welcome; this is a design problem before it is a coding one.
4. More task sets, and better ones
reviewsamples. Twelve is enough to find a floor, not to rank.codesuite currently assumesnode --test; the runner needs a per-language verifier hook.review. Ours were not clean enough (see below), whichmakes the false-positive metric unreliable today. Do not trust that column yet.
5. Quota introspection
There is no way to ask "how much of my window is left" before firing a batch. If a
provider exposes it, surfacing it in
--listwould prevent a long run from dyinghalfway through.
Two things worth knowing before you trust any number here
A benchmark score measures the pair model + verifier, not the model. The
codesuite runs near-ceiling for most models precisely because
node --testruns underneath.Remove the verifier and the number tells you nothing. This is why the
delegateagentrefuses work that nothing downstream can check.
Our own answer keys were wrong four times, and the models caught all four. One
flagged a genuine month-end overflow bug in a sample we had labelled "clean" — it was
right and we were wrong. When every model disagrees with your key, suspect the key.
Same discipline applies to any result posted in this thread, including mine.
Contributing
No process yet beyond: open an issue, or send a PR. Adding a provider is a data change —
see
crossmodel.config.example.json— and does not require touching the transport code.