Closed
Conversation
ae303a7 to
a7568f6
Compare
Contributor
|
favouring #417 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR removes the hidden model-quality ranking heuristic from Hermes/OpenClaw model selection.
Instead of guessing the best primary model from provider names, parameter-count tags, or hardcoded model-family defaults,
model.Ranknow treats the configured LiteLLMmodel_listorder as the source of truth. The only remaining adjustment is that known embedding-only model names containingembedare kept as fallbacks behind chat-capable models so they do not become the default chat model by accident.What we have today
Today the primary model is selected by a hidden heuristic:
flowchart TD A["Available model names"] --> B["model.Rank"] B --> C{"Cloud-looking name?"} C -->|"claude / gpt / o-series"| D["Sort by cloudPrecedence"] C -->|"everything else"| E["Local model path"] E --> F{"Has parseable size token?"} F -->|"llama3.2:1b / qwen3.5:9b"| G["Bigger parameter count wins"] F -->|"no size token"| H{"Known family prefix?"} H -->|"llama3.3 / qwen3 / etc"| I["Use untaggedFamilyDefaults"] H -->|"unknown"| J["rank = 0"] D --> K["Primary + fallbacks"] G --> K I --> K J --> KThat means the stack can override the order the user configured because the code thinks, for example, that a cloud-looking name or a larger-size local tag is better. That is too much hidden product policy for a generic model router.
Proposed behavior in this PR
This makes model selection predictable:
obol model setup --provider openai --model gpt-4.1gpt-4.1becomes primary.obol model setup --provider anthropic --model claude-sonnet...obol model setup custom --model qwen36-faston a fresh stackqwen36-fastbecomes primary.UX question
@OisinKyne, what UX would you prefer here?
The implementation in this PR assumes configured order should be the contract, but there are two product choices worth making explicitly:
obol model prefer <model>so users can reorder the primary without remove/re-add?obol model setupalways place explicitly selected models before wildcard entries such asanthropic/*oropenai/*, so the model the user selected is visibly the primary?My recommendation is yes to both, but I kept this PR scoped to removing the hidden ranking heuristic and documenting the new selection contract.
Validation
Passed:
Known unrelated note:
go test ./...on currentmainstill hits the existing Hermes extras expectation mismatch. That is handled in the separate flow/RBAC QA branch, not here, to keep this PR focused.