Repository navigation
v0.4.0 - pretrained models on the engine: LLMs, CLIP, ViT, and Whisper
Pretrained models across four modalities now load from Hugging Face weights and run on the Apple Neural Engine, and LLM decode is faster by default. 46 commits since v0.3.0.
New model loaders
- Decoder LLMs: GPT-2 (#210), Gemma (GeGLU + scaled embeddings + (1+w) RMSNorm, #205), and Mistral (GQA + sliding window, #206) adapters, alongside the existing Llama/Qwen/MoE/hybrid path. Unsupported architectures are rejected rather than silently mis-loaded (#214).
- CLIP zero-shot image/text classification, both towers on the ANE (#225).
- ViT image classifiers (#212) and CrossEncoder rerankers, including RoBERTa/XLM-R (#208) and DistilBERT (#223).
- ResNet 18/34/50/101 from torchvision (#209).
- Whisper speech-to-text (whisper-base.en), audio encoder and text decoder both on the ANE (#237).
Faster LLM decode
- GPT-2 runs through the unified LLM runner with resident KV-cache decode, ~140 tok/s on gpt2-medium (#224, #229).
- The lm_head runs on the ANE by default, tiled along vocab, replacing the host fp32 matmul. Greedy stays bit-exact via a tie-margin host fallback, and an execute-path fallback keeps generate from breaking on a misconfigured target (#240, #243, #245). Pass ane_lm_head=False for the host matmul.
Demos and fixes
- RAG on the engine: chat with a folder of docs using embeddings, a reranker, and an LLM all on the ANE (examples/rag_chat.py, #231-236).
- The BERT encoder pads and masks a batch so varied-length inputs share one program instead of one per length (#233).
Applied math
Docs
- The manual now covers the full model-loader set, tightened to plain ASCII.
Install: pip install -U aneforge (pretrained loaders: pip install "aneforge[models]").