Fix gemma-4 local agent crash: load it as text, not vision#184
Merged
Conversation
The talk button and Field Assist crashed the app when the on-device agent model was active. Root cause: `LocalLLMService.visionModelIds` listed `mlx-community/gemma-4-e2b-it-4bit`, forcing it through `VLMModelFactory`. The vision factory resolves `model_type: "gemma4"` to mlx-swift-lm's `MLXVLM.Gemma4`, whose forward pass fatally traps in an uncatchable MLX assertion (`mlx_array_dim` inside the backbone). The June 2 candidate fix (a custom text-registry port + 2D token shaping) never took effect because the model never loaded through the text factory it registered into. Fix: stop routing the agent model through the vision factory. mlx-swift-lm registers this exact model in `LLMModelFactory` (the text factory), so it loads as text and uses the library's supported, tested Gemma-4 text model. Also removes the bespoke ~760-line `Gemma4Text.swift` port and its `LLMTypeRegistry` override: it was an unvalidated workaround that shadowed the library's Gemma-4 for *every* text load, and was only ever bypassed. Using the library's own registration is simpler and the supported path. Regression test (`LocalModelRoutingTests`): the agent model must not be in `visionModelIds`. Gates: build-for-testing, full suite 1601/0, Release build — all green. NEEDS ON-DEVICE VERIFICATION: MLX/Metal doesn't run in the simulator, so this environment can confirm the routing change compiles and the model no longer loads via the vision factory, but NOT that the text path generates cleanly on-device. Please rebuild, re-enable the local agent (default off), and confirm talk + Field Assist respond without crashing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With the vision→text routing fixed, the gemma-4 agent got *past* the vision crash into generation — and then the app was Jetsam-killed. On-device log: `tokenIDs.shape=[1, 8241]` for "what's the weather", and a JetsamEvent naming OpenGlasses as the largest process. Cause: `sendViaLocalAgent` built the full cloud-scale system prompt (~100 native-tool descriptions, ~8k tokens) and fed it to a 2B model on a phone; prefill + KV cache at that length exhausts memory. - `sendViaLocalAgent`: build a LEAN prompt for the on-device path (`includeTools: false`) — `sendLocal` already appends its own reduced ~12-tool block, so the model keeps usable tools without the ~8k-token dump. Cloud-agent path unchanged (still gets the full prompt). - `LocalLLMService.generate`: reject a prompt over `maxPromptTokens` (4096) with a catchable `LocalLLMError.promptTooLong` *before* handing it to MLX. MLX allocation failure is an uncatchable trap that Jetsam-kills the process, so the oversized input must be refused up front — a throw lets the caller fall back to cloud instead of the app dying. Tests: guard sits below the observed OOM point (8241) and above a normal lean prompt; error message names the cloud fallback. Gates: build-for-testing, full suite 1603/0, Release — all green. Still needs on-device confirmation that the lean-prompt local turn now completes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
straff2002
force-pushed
the
fix/gemma4-agent-model-routing
branch
from
July 5, 2026 04:55
d4997b9 to
aff3ec4
Compare
RBKunnela
pushed a commit
to RBKunnela/OpenGlasses
that referenced
this pull request
Jul 9, 2026
straff2002#184 gave the on-device agent a lean prompt, but only via sendViaLocalAgent. When gemma-4 is the ACTIVE model, turns go through sendMessage → sendLocal with the full ~100-tool system prompt (~8k tokens), so on-device queries still hit the prompt-too-long guard (8241 tokens > 4096) and failed. Extract the lean-prompt build into a shared `leanOnDevicePrompt` helper and use it for BOTH on-device paths (active-model sendMessage and fast-tier sendViaLocalAgent) so they can't diverge again. On-device (.local and .appleOnDevice) now always gets persona/location/memory only — no tool dump, no playbook/shortcuts/OpenClaw. sendLocal still receives includeTools and appends its own reduced ~12-tool block, so the model keeps usable tools. Gates: build-for-testing, full suite 1603/0, Release — all green. Needs on-device confirmation the token count now drops well under 4096 and a local-model question returns an answer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The crash
The talk button and Field Assist crashed the app whenever the on-device Gemma-4 agent model was active. I pulled the crash logs off the device — all three were the same: an uncatchable MLX assertion (
_mlx_error→mlx_array_dim→MLXArray.subscript) insideGemma4TextBackbone.callAsFunction, i.e. mlx-swift-lm'sMLXVLM.Gemma4forward pass. ASIGTRAPnotry/catchcan stop (the MLX C++ error handler traps the process).Root cause
LocalLLMService.visionModelIdslistedmlx-community/gemma-4-e2b-it-4bit, so the agent model — a text model — was loaded throughVLMModelFactory. The vision factory resolvesmodel_type: "gemma4"to the crashingMLXVLM.Gemma4. The text factory (LLMModelFactory) resolves the same id to mlx-swift-lm's Gemma-4 text model, which doesn't have that crashing subscript — and the library explicitly registers this exact model as text (gemma4_e2b_it_4bitinLLMModelFactory).A June 2 "candidate fix" had registered a custom port into the text registry (
LLMTypeRegistry) — but that registry is only read by the text factory, which this model never reached. So the fix sat on a path the model never took, and thelocalAgentEnabledtoggle was defaulted off to hide the crash.The fix
gemma-4-e2b-it-4bitfromvisionModelIds→ it loads as text viaLLMModelFactoryand uses the library's supported, tested Gemma-4 text model.Gemma4Text.swiftport + itsLLMTypeRegistryoverride. It was unvalidated (never actually ran), and it shadowed the library's Gemma-4 for every"gemma4"text load — a latent landmine. Using mlx-swift-lm's own registration is the supported path and −775 lines.LocalModelRoutingTests): the agent model must never be invisionModelIds.Net diff: +41 / −775.
Gates
build-for-testing ✅ · full suite 1601 / 0 failures ✅ · Release ✅
MLX/Metal doesn't run in the simulator, so this environment confirms the routing change compiles and the model no longer resolves through the vision factory — but not that the text path generates cleanly on real hardware. Please:
If the text path also misbehaves on-device, the fallback is to switch
defaultAgentModelIdto a known-good on-device model (e.g. a Qwen build) rather than resurrect the port — happy to do that as a follow-up.🤖 Generated with Claude Code