⚠️ Known Issue: llama.cpp regression breaks vision & audio analysis

[ayushh0110]

TL;DR: Newer versions of llama-server (build b10244+) have a regression in the multimodal projector (mmproj) that causes both screenshot analysis and audio transcription to fail — outputting garbage <unused49> tokens instead of real text. If you're experiencing this, pin llama-server to build b9193 until a permanent fix is in place.

---

What's happening

Recent releases of llama.cpp have introduced a regression in how the multimodal projector (mmproj-BF16.gguf) handles Gemma 4 inputs. This affects all multimodal features in ScreenMind:

• ❌ Screenshot analysis (vision) — produces garbage output
• ❌ Voice memo transcription (audio) — produces garbage output
• ❌ Meeting transcription (audio) — produces garbage output
• ✅ Text-only chat — works fine
• ✅ Screenshot capture — works fine (capture is unaffected, only AI analysis breaks)

Symptoms

• Transcriptions show as "Voice memo (transcription unavailable)"
• Screenshot analysis returns nonsensical or empty results
• Server logs show the model outputting <unused49> tokens repeatedly
• Input token count is abnormally low (e.g., 87 tokens for an audio file that should be 1000+)

Root cause

The --mmproj-auto flag downloads a multimodal projector that is incompatible with certain llama.cpp builds when paired with the Gemma 4 E2B Q4_K_M GGUF from Unsloth. The audio/vision encoder fails to properly tokenize media inputs, causing the model to see only the text prompt and fill the context window with garbage tokens.

This is tracked upstream:

• ggml-org/llama.cpp#23688 — Audio regression
• ggml-org/llama.cpp#24146 — <unused49> token output

---

Recommended fix

Pin llama-server to build b9193 (commit 1a68ec937), which is the last known stable version with working multimodal support for ScreenMind.

If you already updated

1. Stop ScreenMind
2. Delete the llama/ directory in your ScreenMind install
3. Download build b9193 manually from llama.cpp releases
4. Extract to llama/ in your ScreenMind directory
5. Restart ScreenMind

If you're doing a fresh install

setup_llama.py currently pulls the latest release, which is affected. I'm working on pinning the setup script to a known-good version. In the meantime, install manually using the link above.

---

What I'm doing about it

• Investigating the exact mmproj incompatibility across llama.cpp versions
• Working on pinning setup_llama.py to a tested, stable build
• Adding a version compatibility check to warn users if their llama-server has this regression
• Monitoring upstream fixes in llama.cpp

I'll update this issue once a permanent fix is ready. If you have any additional info or workarounds, please share below.

---

Affected versions: llama-server builds after b9193 (confirmed broken on b10244)
Unaffected versions: llama-server b9193 and earlier
ScreenMind versions: All current versions (v0.1.2 and below)