Repository navigation
v0.2.0 - LLMs on the Apple Neural Engine
Decoder LLMs now run on the ANE end to end.
- Prefill + resident KV-cache decode for Llama/Qwen models, auto-segmented past the ~2GB single-program ceiling.
- Speculative decoding: exact, 2.28x on Qwen3-8B (verify(K) ~ verify(1) on the latency-bound ANE).
- Mixture-of-Experts from GGUF: Qwen3-MoE and Qwen2-MoE run the full model (router + all experts) on the ANE.
- Hybrid models: Qwen3.5 / Qwen3-Next (gated DeltaNet + gated attention). The real Qwen3.5-27B decodes coherently on a pure ANE from its GGUF at int8 (int8/fp16 compute is lower-fidelity than llama.cpp's fp32-accumulated Q4; see the LLMs guide).
- ONNX import and on-ANE finetuning of imported models.
- Config-driven model adapters: a new architecture is an adapter, not new branches in the runtime.