On Device LLM

On-Device LLM (Gemma)

Zerm's third model is a small local language model that powers the "agentic" layer — making Read Aloud sound human and cleaning up dictation. It runs entirely on-device via llama.cpp.

Code lives in Zerm/LocalLLM/.

Default model

Gemma 4 E2B Instruct, Q4_K_M GGUF (~3.1 GB). E2B is the smallest Gemma 4 (effective ~2B params, Per-Layer Embeddings for on-device use). Downloaded from the public unsloth/gemma-4-E2B-it-GGUF repo. Users can swap in other GGUF models — the bridge uses each model's built-in chat template (with a Gemma fallback), so other instruct models work without code changes.

Components

File	Role
LocalLLMModelManager.swift	Singleton downloader/loader (mirrors KokoroModelManager)
LlamaEngine.swift	Swift actor that serializes inference off the main thread
LlamaBridge.h / LlamaBridge.mm	Objective-C++ wrapper around llama.cpp

The ggml isolation problem (important)

whisper.cpp and llama.cpp both vendor ggml, at different versions. Both ship a Clang module that exports the ggml headers. Importing both into Swift makes the compiler see two conflicting definitions of ggml_op/ggml_type → a hard build error.

Solution: confine #import <llama/llama.h> to a single Objective-C++ translation unit (LlamaBridge.mm). Swift only ever sees the Foundation-only LlamaBridge interface — it never imports the llama module, so ggml never collides with whisper's.

flowchart LR
    SW[Swift code] -->|Foundation only| BR[LlamaBridge.h]
    BR --- MM[LlamaBridge.mm<br/>#import llama.h]
    MM --> LL[llama.framework<br/>+ its ggml]
    LW[LibWhisper.swift] -->|import whisper| WF[whisper.framework<br/>+ its ggml]
    style MM fill:#1f2937,color:#fff
    note[Each ggml lives in a separate TU — never co-imported]

Loading

Inference details (llama.cpp build b9699)

• Load: llama_model_load_from_file → llama_init_from_model; n_gpu_layers = 999 (Metal).
• Prompt: llama_model_chat_template + llama_chat_apply_template (Gemma fallback), folding system + user into one user turn.
• Decode loop: llama_batch_get_one → llama_decode → llama_sampler_sample(-1); stop on llama_vocab_is_eog.
• Sampler chain: top-k 40, top-p 0.95, temp 0.3, dist — low temperature keeps rewrites faithful.
• KV reset between requests: llama_memory_clear(llama_get_memory(ctx), true).

Two runtime gotchas (fixed)

• Metal exit crash: ggml's Metal residency-set collection is freed by a C++ static destructor at process exit, racing its own background init → ggml_abort. Fixed by setenv("GGML_METAL_NO_RESIDENCY", "1", 1) before llama_backend_init() (disables only that memory optimization; GPU inference unaffected).
• <end_of_turn> spoken aloud: small/quantized models sometimes emit the turn delimiter as literal text instead of the special EOG token. The bridge stops generation at <end_of_turn>/<start_of_turn> and strips stray control tokens.

Where it's used

• Read Aloud → Smart Reading: TTSNaturalizer rewrites cleaned text into natural prose. See Smart Reading.
• AI Enhancement: AIProvider.localLLM routes dictation enhancement through the same model — no API key, recommended by default. See AI Enhancement.

Both consumers share one LocalLLMModelManager / one loaded model.

Packaging

llama.xcframework (b9699, at $(HOME)/Zerm-Dependencies/llama/build-apple/) is a dynamic framework with a module map; it's linked and embedded + signed (mirroring whisper.xcframework). No bridging header needed for the framework — LlamaBridge.mm includes it directly.

See: Architecture · Build & Release