Skip to content

fix: preserve configured model order#412

Closed
bussyjd wants to merge 2 commits intomainfrom
fix/model-config-order
Closed

fix: preserve configured model order#412
bussyjd wants to merge 2 commits intomainfrom
fix/model-config-order

Conversation

@bussyjd
Copy link
Copy Markdown
Collaborator

@bussyjd bussyjd commented May 4, 2026

Summary

This PR removes the hidden model-quality ranking heuristic from Hermes/OpenClaw model selection.

Instead of guessing the best primary model from provider names, parameter-count tags, or hardcoded model-family defaults, model.Rank now treats the configured LiteLLM model_list order as the source of truth. The only remaining adjustment is that known embedding-only model names containing embed are kept as fallbacks behind chat-capable models so they do not become the default chat model by accident.

What we have today

Today the primary model is selected by a hidden heuristic:

flowchart TD
  A["Available model names"] --> B["model.Rank"]
  B --> C{"Cloud-looking name?"}
  C -->|"claude / gpt / o-series"| D["Sort by cloudPrecedence"]
  C -->|"everything else"| E["Local model path"]
  E --> F{"Has parseable size token?"}
  F -->|"llama3.2:1b / qwen3.5:9b"| G["Bigger parameter count wins"]
  F -->|"no size token"| H{"Known family prefix?"}
  H -->|"llama3.3 / qwen3 / etc"| I["Use untaggedFamilyDefaults"]
  H -->|"unknown"| J["rank = 0"]
  D --> K["Primary + fallbacks"]
  G --> K
  I --> K
  J --> K
Loading

That means the stack can override the order the user configured because the code thinks, for example, that a cloud-looking name or a larger-size local tag is better. That is too much hidden product policy for a generic model router.

Proposed behavior in this PR

flowchart TD
  A["Configured LiteLLM model_list"] --> B["Preserve order"]
  B --> C["First chat-capable model is primary"]
  B --> D["Remaining models stay fallbacks in configured order"]
  E["Embedding-only entries"] --> F["Move behind chat-capable models"]
Loading

This makes model selection predictable:

Pattern Behavior
obol model setup --provider openai --model gpt-4.1 gpt-4.1 becomes primary.
obol model setup --provider anthropic --model claude-sonnet... The configured Claude model becomes primary.
obol model setup custom --model qwen36-fast on a fresh stack qwen36-fast becomes primary.
Multiple configured models The first configured chat-capable model wins; the rest are fallbacks in order.
Embedding model before chat model Embedding model is moved behind chat models.

UX question

@OisinKyne, what UX would you prefer here?

The implementation in this PR assumes configured order should be the contract, but there are two product choices worth making explicitly:

  1. Should we add a first-class command like obol model prefer <model> so users can reorder the primary without remove/re-add?
  2. Should obol model setup always place explicitly selected models before wildcard entries such as anthropic/* or openai/*, so the model the user selected is visibly the primary?

My recommendation is yes to both, but I kept this PR scoped to removing the hidden ranking heuristic and documenting the new selection contract.

Validation

Passed:

go test ./internal/model ./internal/hermes ./internal/openclaw -run 'TestRank' -count=1
git diff --check

Known unrelated note: go test ./... on current main still hits the existing Hermes extras expectation mismatch. That is handled in the separate flow/RBAC QA branch, not here, to keep this PR focused.

@bussyjd bussyjd requested a review from OisinKyne May 4, 2026 10:27
@OisinKyne OisinKyne force-pushed the fix/model-config-order branch from ae303a7 to a7568f6 Compare May 4, 2026 13:39
@OisinKyne OisinKyne marked this pull request as ready for review May 4, 2026 17:01
@bussyjd bussyjd mentioned this pull request May 4, 2026
6 tasks
@OisinKyne
Copy link
Copy Markdown
Contributor

favouring #417

@OisinKyne OisinKyne closed this May 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants