feat: add Token Overdrive local inference provider (llama.cpp DMR router) - #498
Closed
jecruz wants to merge 1 commit into
Closed
feat: add Token Overdrive local inference provider (llama.cpp DMR router)#498jecruz wants to merge 1 commit into
jecruz wants to merge 1 commit into
Conversation
Token Overdrive is a new local inference provider backed by a custom llama.cpp fork (yama.cpp) with a Dynamic Model Router that hot-swaps GGUF models without server restart. Metal-accelerated on Apple Silicon. Changes: - local-inference.js: add token-overdrive case to getLocalProviderBaseUrl, getLocalProviderHealthCheck, getLocalProviderContainerReachabilityCheck, and validateLocalProvider (port 8101, OpenAI-compatible /v1/models health) - local-inference.js: add getYamaModelOptions, getDefaultYamaModel, getYamaProbeCommand, validateYamaModel — queries live /v1/models endpoint to enumerate router-loaded models; falls back to DEFAULT_YAMA_MODEL - inference-config.js: add token-overdrive provider case with label "Token Overdrive (llama.cpp DMR)"; default model qwen35-35b-a3b (60 tps) - token-overdrive-models.json: local model catalog with tuned defaults (qwen35-35b-a3b at 60 tps, qwen35-27b at 17 tps, both via Docker Model Runner) Provider ID: "token-overdrive" Port: 8101 Health: GET /v1/models (OpenAI-compatible) Probe: POST /v1/chat/completions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
📝 WalkthroughWalkthroughThe pull request introduces support for a new local inference provider called "Token Overdrive" (llama.cpp DMR). This involves adding provider configuration, model discovery utilities, validation helpers, and a model catalog file that defines available Qwen models with their performance characteristics and recommendations. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (4)
bin/lib/inference-config.js (1)
71-75: Consider a lookup map if more local providers are added.The nested ternary works for two special cases, but could become unwieldy if more local providers with distinct defaults are introduced. A simple object lookup would improve readability:
♻️ Optional refactor for maintainability
function getOpenClawPrimaryModel(provider, model) { - const localDefault = - provider === "ollama-local" ? DEFAULT_OLLAMA_MODEL : - provider === "token-overdrive" ? DEFAULT_YAMA_MODEL : - DEFAULT_CLOUD_MODEL; + const LOCAL_DEFAULTS = { + "ollama-local": DEFAULT_OLLAMA_MODEL, + "token-overdrive": DEFAULT_YAMA_MODEL, + }; + const localDefault = LOCAL_DEFAULTS[provider] || DEFAULT_CLOUD_MODEL; const resolvedModel = model || localDefault; return resolvedModel ? `${MANAGED_PROVIDER_ID}/${resolvedModel}` : null; }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@bin/lib/inference-config.js` around lines 71 - 75, Replace the nested ternary that computes localDefault with a lookup map keyed by provider to improve readability and maintainability: create a map (e.g., providerDefaultMap) that maps "ollama-local" to DEFAULT_OLLAMA_MODEL and "token-overdrive" to DEFAULT_YAMA_MODEL, then set localDefault = providerDefaultMap[provider] || DEFAULT_CLOUD_MODEL and keep resolvedModel = model || localDefault; update references to localDefault, provider, DEFAULT_OLLAMA_MODEL, DEFAULT_YAMA_MODEL, DEFAULT_CLOUD_MODEL, and resolvedModel accordingly.bin/lib/token-overdrive-models.json (1)
7-14: Clarify the discrepancy betweenidandcanonical_idnaming.The
idusesqwen35-35b-a3bwhilecanonical_idusesqwen3.5-35b-a3b(note35vs3.5). This inconsistency could cause confusion when mapping models. Ifidis used for API calls andcanonical_idis for display/documentation purposes, this is fine—but consider documenting the convention.Also, the
providerfield is"docker-model-runner"here, while the PR describes the provider ID as"token-overdrive". Verify whether this field represents the underlying engine vs. the user-facing provider selection.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@bin/lib/token-overdrive-models.json` around lines 7 - 14, The id/canonical_id mismatch should be resolved: pick the canonical naming convention (e.g., use dot notation like "qwen3.5-35b-a3b" or remove the dot to "qwen35-35b-a3b") and make both "id" and "canonical_id" identical, updating the "id" or "canonical_id" field in the JSON to match the chosen convention; additionally verify the "provider" field and set it to the intended provider ID ("token-overdrive") if this entry represents the user-facing provider, or keep "docker-model-runner" only if it intentionally denotes the underlying engine, and add a brief inline comment or documentation entry describing the convention for id vs canonical_id and provider semantics.bin/lib/local-inference.js (2)
70-74: Consider consistent terminology in error messages.The error messages reference "yama" and "yama router" while the user-facing provider is "token-overdrive" with label "Token Overdrive (llama.cpp DMR)". This could confuse users who selected "token-overdrive" but see "yama" in errors.
♻️ Proposed terminology alignment
case "token-overdrive": return { ok: false, - message: `Local yama (llama.cpp DMR router) was selected, but nothing is responding on http://localhost:${YAMA_PORT}. Start the router with: python3 scripts/run_dmr_router_workflow.py --model <model-id>`, + message: `Token Overdrive (llama.cpp DMR router) was selected, but nothing is responding on http://localhost:${YAMA_PORT}. Start the router with: python3 scripts/run_dmr_router_workflow.py --model <model-id>`, };case "token-overdrive": return { ok: false, message: - `Local yama router is responding on localhost, but containers cannot reach http://host.openshell.internal:${YAMA_PORT}. Ensure llama-server binds to 0.0.0.0:${YAMA_PORT} (add --host 0.0.0.0 to the router preset).`, + `Token Overdrive router is responding on localhost, but containers cannot reach http://host.openshell.internal:${YAMA_PORT}. Ensure llama-server binds to 0.0.0.0:${YAMA_PORT} (add --host 0.0.0.0 to the router preset).`, };Also applies to: 103-108
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@bin/lib/local-inference.js` around lines 70 - 74, The error messages in the case "token-overdrive" branch reference "yama"/“yama router”, which is inconsistent with the provider label "Token Overdrive (llama.cpp DMR)"; update the user-facing strings in the case "token-overdrive" return objects (the message that mentions http://localhost:${YAMA_PORT} and the similar message later in the file) to use consistent terminology such as "Token Overdrive (llama.cpp DMR)" or "Token Overdrive router" (and keep the YAMA_PORT variable) so users who selected "token-overdrive" see matching names.
141-158: Consider surfacing HTTP/connection errors distinctly from model errors.When
outputis falsy (line 143), it could mean the router is down, the model isn't loaded, or a timeout occurred. The current message assumes the model needs loading, but a connection failure would give the same symptom. This is a minor UX consideration—the existing behavior is acceptable.Additionally, for consistency with the terminology point above:
♻️ Optional: align "yama model" with "Token Overdrive"
if (!output) { return { ok: false, message: - `yama model '${model}' did not answer the probe. The DMR router may need to load it first. ` + + `Token Overdrive model '${model}' did not answer the probe. The DMR router may need to load it first. ` + `Run: python3 scripts/run_dmr_router_workflow.py --model ${model}`, }; } try { const parsed = JSON.parse(output); if (parsed && typeof parsed.error === "string" && parsed.error.trim()) { - return { ok: false, message: `yama model '${model}' probe failed: ${parsed.error.trim()}` }; + return { ok: false, message: `Token Overdrive model '${model}' probe failed: ${parsed.error.trim()}` }; } } catch {}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@bin/lib/local-inference.js` around lines 141 - 158, The validateYamaModel function treats any falsy runCapture output as "model not loaded"; update validateYamaModel to distinguish connection/HTTP/timeout failures from model probe failures by checking runCapture error/exit status (or capturing stderr) when calling runCapture(getYamaProbeCommand(model)), and produce distinct messages: one indicating router/connection issues (include the underlying error or status) and another indicating the model probe returned no/empty response or returned parsed.error; also keep the existing parsed JSON handling for parsed.error (parsed.error) and adjust wording to consistently use the preferred "Token Overdrive" terminology if desired.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@bin/lib/inference-config.js`:
- Around line 71-75: Replace the nested ternary that computes localDefault with
a lookup map keyed by provider to improve readability and maintainability:
create a map (e.g., providerDefaultMap) that maps "ollama-local" to
DEFAULT_OLLAMA_MODEL and "token-overdrive" to DEFAULT_YAMA_MODEL, then set
localDefault = providerDefaultMap[provider] || DEFAULT_CLOUD_MODEL and keep
resolvedModel = model || localDefault; update references to localDefault,
provider, DEFAULT_OLLAMA_MODEL, DEFAULT_YAMA_MODEL, DEFAULT_CLOUD_MODEL, and
resolvedModel accordingly.
In `@bin/lib/local-inference.js`:
- Around line 70-74: The error messages in the case "token-overdrive" branch
reference "yama"/“yama router”, which is inconsistent with the provider label
"Token Overdrive (llama.cpp DMR)"; update the user-facing strings in the case
"token-overdrive" return objects (the message that mentions
http://localhost:${YAMA_PORT} and the similar message later in the file) to use
consistent terminology such as "Token Overdrive (llama.cpp DMR)" or "Token
Overdrive router" (and keep the YAMA_PORT variable) so users who selected
"token-overdrive" see matching names.
- Around line 141-158: The validateYamaModel function treats any falsy
runCapture output as "model not loaded"; update validateYamaModel to distinguish
connection/HTTP/timeout failures from model probe failures by checking
runCapture error/exit status (or capturing stderr) when calling
runCapture(getYamaProbeCommand(model)), and produce distinct messages: one
indicating router/connection issues (include the underlying error or status) and
another indicating the model probe returned no/empty response or returned
parsed.error; also keep the existing parsed JSON handling for parsed.error
(parsed.error) and adjust wording to consistently use the preferred "Token
Overdrive" terminology if desired.
In `@bin/lib/token-overdrive-models.json`:
- Around line 7-14: The id/canonical_id mismatch should be resolved: pick the
canonical naming convention (e.g., use dot notation like "qwen3.5-35b-a3b" or
remove the dot to "qwen35-35b-a3b") and make both "id" and "canonical_id"
identical, updating the "id" or "canonical_id" field in the JSON to match the
chosen convention; additionally verify the "provider" field and set it to the
intended provider ID ("token-overdrive") if this entry represents the
user-facing provider, or keep "docker-model-runner" only if it intentionally
denotes the underlying engine, and add a brief inline comment or documentation
entry describing the convention for id vs canonical_id and provider semantics.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 085fdec0-b214-4184-b51f-84368cb75921
📒 Files selected for processing (3)
bin/lib/inference-config.jsbin/lib/local-inference.jsbin/lib/token-overdrive-models.json
Author
|
Closing — not ready for upstream review yet. Will reopen when complete. |
6 tasks
43 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Token Overdrive as a fourth local inference provider alongside
nim-local,ollama-local, andvllm-local.Token Overdrive is backed by a custom llama.cpp fork with a Dynamic Model Router that enables hot-swapping GGUF models without server restart. It runs Metal-accelerated on Apple Silicon and exposes a standard OpenAI-compatible API on port 8101.
token-overdrive8101/v1/models,/v1/chat/completions)qwen35-35b-a3b(60 tps on Apple Silicon)GET /v1/models/v1/modelsendpointChanges
bin/lib/local-inference.jstoken-overdrivecase in all provider switch blocks;getYamaModelOptions,getDefaultYamaModel,getYamaProbeCommand,validateYamaModelbin/lib/inference-config.jstoken-overdriveprovider config with label "Token Overdrive (llama.cpp DMR)"; correct default model ingetOpenClawPrimaryModelbin/lib/token-overdrive-models.jsonTest plan
getLocalProviderHealthCheck("token-overdrive")returns correct curl command for port 8101validateLocalProvider("token-overdrive", ...)returns descriptive error when router not runninggetYamaModelOptionsreturns["qwen35-35b-a3b"]fallback when router unreachablegetProviderSelectionConfig("token-overdrive")returns correctproviderLabelgetOpenClawPrimaryModel("token-overdrive", null)resolves toinference/qwen35-35b-a3bnpm testpasses🤖 Generated with Claude Code
Summary by CodeRabbit
Release Notes