mlx-vlm text lane: small multimodal models (gemma-4-e2b, Ministral-3-2512) return incoherent text / hang

[raullenchai]

Summary

Serving certain small multimodal checkpoints for plain text through the mlx-vlm (VLM) lane produces broken output. Found while trying to add Gemma/Mistral legs to the L1 CI smoke (PR #1366) — their only small models are multimodal.

Evidence — real macos-14 (M1) hosted runner, rapid-mlx 0.11.4, mlx-vlm 0.6.3

[vision] installed → auto-routed to the mlx-vlm lane:

• gemma-4-e2b-4bit: serve READY, coherence gate [P0] Add an always-on output-coherence gate (real serve path, golden answers + garbage detector) #1247 → 0/6 golden (all wrong / incoherent).
• Ministral-3-3B-Instruct-2512-4bit: serve READY, first chat completion hangs — coherence request times out at 90s, server becomes unreachable.

--no-mllm (force text-only mlx-lm lane):

• Ministral-3-2512: 5/6 golden (one flaky miss) — usable.
• gemma-4-e2b: still 0/6 on macos-14 (M1) / 0.11.4 — but 6/6 on local M3 / rapid-mlx 0.11.0. So gemma-4-e2b text-lane coherence diverges by engine-version and/or hardware.

Impact

• A user who installs [vision] and chats text-only with these small VLMs gets garbage (gemma) or a hang (Ministral): the vision lane's text-generation path is unsafe for these checkpoints.
• Blocks expanding the L1 CI smoke (PR ci: L1 small-model GPU smoke on free macos-14 — coherence + tool-call (Qwen+Llama) #1366) past Qwen+Llama, since Gemma/Mistral only ship small models as multimodal.

Repro

pip install -e ".[vision]"
rapid-mlx serve gemma-4-e2b-4bit  --port 8123 --no-thinking   # ask "capital of Japan?" -> garbage
rapid-mlx serve ministral-3b-4bit --port 8123 --no-thinking   # first completion hangs

Asks

1. Fix/triage the mlx-vlm text-generation path for these small VLMs (or make a text-only request fall back to the LM backbone instead of hanging/garbling).
2. Root-cause the gemma-4-e2b text-lane 0/6 on 0.11.4/M1 vs 6/6 on 0.11.0/M3 — regression vs an M1 Metal-kernel issue.

Workaround

Serve with --no-mllm to force the text-only mlx-lm lane (fixes Ministral; gemma-4-e2b still needs ask #2).

Filed from the release-flow L1 gate work (PR #1366).

[raullenchai]

Revalidation update (2026-08-05)

We re-ran the original #1247 coherence reproduction against current main (aa6964e6) on two physical Macs, using the default VLM route (rapid-mlx serve gemma-4-e2b-4bit --no-thinking) and the same MLX line as the original report.

Gemma 4 e2b results:

• M3 Pro 18 GB, mlx-vlm 0.6.3: 36/36 golden cases passed.
• M3 Pro 18 GB, mlx-vlm 0.6.10 (the version a fresh Desktop sidecar build currently resolves): 6/6 passed.
• Original v0.11.4 source on that same M3 Pro with mlx 0.31.2, mlx-lm 0.31.3, mlx-vlm 0.6.3: 30/30 passed.
• M2 Pro 32 GB, current main, mlx-vlm 0.6.3: 30/30 passed.
• M2 Pro 32 GB, current main, mlx-vlm 0.6.10: 18/18 passed.

So Gemma is not a generally broken alias. The original macos-14/M1 0/6 result remains valuable evidence of an M1 runner/environment-specific discrepancy, but it does not reproduce on M2 Pro or M3 Pro and was not caused solely by the v0.11.4 source revision. A current M1 rerun is still warranted before declaring that hardware-specific leg resolved.

Ministral result remains a current, release-relevant failure:

• With mlx-vlm 0.6.3, the default VLM route fails startup because PixtralProcessor requires PyTorch/Torchvision.
• With mlx-vlm 0.6.10, the server reaches ready, but the first text completion times out. The MLLM scheduler loops on Model.__call__() missing 1 required positional argument: mask; coherence is 0/6.
• --no-mllm is coherent (18/18), but the public alias and Desktop default route do not add that workaround.

Action: remove the public ministral-3b-4bit alias at the rapid-mlx catalog level so users cannot select a known-hanging default route. Keep the Gemma alias. A PR will reference this issue.

[raullenchai]

The Rapid-MLX-side mitigation for Ministral has landed in #1498 (squash commit 50d278bc).

Current behavior on main:

• ministral-3b-4bit is no longer advertised by rapid-mlx models or Rapid Desktop.
• The retired short alias exits before download/model startup with an actionable stderr message.
• The explicit HF path remains available for text-only upstream investigation with --no-mllm; size metadata is retained for download/disk preflight.
• Codex review converged to 0 findings; pr_validate returned MERGE-SAFE; targeted 827/827 and full pytest 15,586/15,586 passed; all GitHub required checks passed.

I am leaving this issue open only for the historical M1 / macOS 14 Gemma e2b discrepancy. Current reruns remain green on M2 Pro and M3 Pro, so Gemma support was not removed. A current M1/macOS 14 reproduction is still needed before changing that support decision.

[raullenchai]

Re-measured on the sidecar rapid-mac-v0.12.5 will actually ship

This issue was filed against mlx-vlm 0.6.3 on a macos-14 M1 hosted
runner. The 0.12.5 app bundles mlx-vlm 0.6.10 (build-sidecar.sh installs
--no-deps --upgrade inside >=0.6.3,!=0.6.4,<0.7, so it floats), against
transformers 5.12.1. Re-ran the golden set through that exact bundled
binary, M3 Ultra:

alias	lane	result
gemma-4-e2b-4bit	mllm (auto)	6/6 — was 0/6 on 0.6.3 / M1
ministral-3b-4bit	mllm (auto)	still hangs, root cause below
ministral-3b-4bit	--no-mllm	6/6

Root cause of the Ministral hang

It is not slowness or a timeout. The MLLM lane calls the model with the wrong
signature and swallows the result:

ERROR:rapid_mlx.mllm_scheduler:Error in MLLM process loop:
    Model.__call__() missing 1 required positional argument: 'mask'

mllm_scheduler's process loop catches this, logs it, and goes round again —
565 identical lines in one 90 s client request. /v1/status stays at
steps_executed: 0, total_requests_processed: 0. The HTTP request never
receives a response or an error, so from the caller it is an unbounded hang.
That is the mllm_scheduler client-error-swallow shape again: an exception
inside the loop should fail the in-flight request, not spin.

So ask #1 splits in two:

1. Ministral (and any checkpoint whose mlx-vlm class signature drifts) —
   the concrete fix is the mask argument, but the general fix is that a
   repeated exception in the MLLM process loop must surface to the waiting
   request instead of looping. Right now any future signature drift produces
   a silent hang rather than a 500.
2. gemma-4-e2b — I cannot close this leg. This issue itself records
   6/6 on M3 / 0.11.0 versus 0/6 on M1 / 0.11.4, so a green run on M3 does not
   refute an M1-specific failure. Needs an M1 to settle; my result is only
   evidence that 0.6.10 is not worse on M3.

What this means for #1496's denylist

• ministral-3b-4bit — justified, verified still broken on the shipping
  sidecar.
• gemma-4-e2b-* — unverified on this hardware. Possibly over-broad now,
  but hiding it is the safe side of the trade while the M1 leg is unknown, and
  fix(rapid-mac): hide known-broken multimodal-only aliases from the model picker #1496's evidence-bounded scope is the right instinct. Worth re-testing on an
  M1 before un-hiding.

Related, found while measuring this

The bundled sidecar ships mlx-vlm 0.6.10, which declares
transformers>=5.14.0, on transformers 5.12.1 — --no-deps silences the
conflict, and nothing validates it. Meanwhile pip install 'rapid-mlx[vision]'
resolves to 0.6.3, because with dependencies honoured the engine's own
transformers<5.13 cap forces a backtrack. So the desktop app and a pip
install run different vision stacks, and the mask mismatch above is exactly
the runtime API drift that arrangement invites. Filing separately.

Measured with the rapid-mac-v0.12.5 release build (#1499).

[raullenchai]

Status update on the two legs:

Ministral hang leg — fixed by #1810 (430b9a6c, merged 2026-08-10). The root cause was that _step_no_queue only caught (ValueError, RuntimeError) around the forward pass, so the TypeError (Model.__call__() missing 1 required positional argument: 'mask') escaped to _process_loop's outer handler, which logged + retried the same poisoned batch every step — the 565-identical-lines silent hang. #1810 added _fail_all_inflight, so any exception escaping a step now fails every in-flight request with a sanitized 500-class error and tears down the poisoned batch generator instead of spinning. Its test test_process_loop_failure_unblocks_every_inflight_request pins exactly this mask TypeError through the async loop. Combined with the #1498 alias retirement, the Ministral path no longer hangs the worker.

gemma-4-e2b coherence leg — still open, unverifiable here. As noted above, this reproduces as 0/6 golden only on M1 / 0.11.4; it is 6/6 on M3 across mlx-vlm 0.6.3/0.6.10. A green M3 run doesn't refute the M1-specific failure, so this leg needs an M1 to settle. Keeping the issue open for it.

(For the record: I drafted an inner-_step_no_queue variant of the hang fix in #1815 but closed it — #1810 already covers the hang, and surfaces a clearer error than the inner path would.)