Local model: lean prompt on the active-model path too (not just agent)#186
Merged
Merged
Conversation
#184 gave the on-device agent a lean prompt, but only via sendViaLocalAgent. When gemma-4 is the ACTIVE model, turns go through sendMessage → sendLocal with the full ~100-tool system prompt (~8k tokens), so on-device queries still hit the prompt-too-long guard (8241 tokens > 4096) and failed. Extract the lean-prompt build into a shared `leanOnDevicePrompt` helper and use it for BOTH on-device paths (active-model sendMessage and fast-tier sendViaLocalAgent) so they can't diverge again. On-device (.local and .appleOnDevice) now always gets persona/location/memory only — no tool dump, no playbook/shortcuts/OpenClaw. sendLocal still receives includeTools and appends its own reduced ~12-tool block, so the model keeps usable tools. Gates: build-for-testing, full suite 1603/0, Release — all green. Needs on-device confirmation the token count now drops well under 4096 and a local-model question returns an answer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #184. On-device confirmation of #184 showed the local model still got an 8241-token prompt and hit the new guard (
Prompt is too long … 8241 tokens; limit 4096).Why #184 missed it
#184 gave the on-device model a lean prompt only via
sendViaLocalAgent(the fast-tier agent route). But when gemma-4 is the user's active model, turns go throughsendMessage→sendLocalwith the full ~100-tool system prompt (~8k tokens). That path still built the fat prompt. The good news: #184's guard did its job — a clean "switch to a cloud model" error instead of the earlier Jetsam/OOM crash.Fix
Extract the lean-prompt build into a shared
LLMService.leanOnDevicePrompt(...)and use it for both on-device routes so they can't diverge again:sendMessagenow branches onisOnDevice(.local/.appleOnDevice) → lean prompt (persona/location/memory only; no ~100-tool dump, no playbook/shortcuts/OpenClaw).sendViaLocalAgentrefactored to call the same helper.sendLocalstill receivesincludeToolsand appends its own reduced ~12-tool block, so the model keeps usable tools — only the prompt drops the dump.Gates
build-for-testing ✅ · full suite 1603 / 0 ✅ · Release ✅
Confirm the log's
tokenIDs.shapenow drops well under 4096 for a normal question and the local model returns an answer. The token-count reduction can't be measured headlessly (no MLX in the sim); the guard from #184 remains the backstop if any path still overshoots.🤖 Generated with Claude Code