Repository navigation
Releases
v0.5.0 - macOS 27.2 fixes, signal processing, and more linear algebra
Fixes for macOS 27.2, where v0.4.0 fails to compile native attention and returns wrong results from pivoted LU, plus a round of signal-processing, linear-algebra and special-function additions. 63 commits since v0.4.0.
macOS 27.2
If you are on macOS 27.2, upgrade: v0.4.0 is affected by both of these.
- Native bridge programs compile again. The framework now rejects a bridge bundle with two or more weight files (
verifyBundleAtPath: invalid model), which brokeaf.sdpa, causal decoder blocks and fused ONNX attention. Such a bundle is retried with its weights merged into one file (#309). - A slice offset on the last axis no longer loses its offset. Two such slices at different offsets feeding matmuls in one fused program returned the wrong column or failed to compile on an M5 Pro, which made
linalg.lu_pivoted,det,slogdet,solveandinvreturn wrong or non-finite results. The compiler now lowers a last-axis-offsetslice_by_size(and socropand static gather) off the last axis;ANEFORGE_ROUTE_WIDTH_SLICES=0restores the old lowering (#311, #314).bench/slice_offset_probe.pyshows whether a machine has the underlying behaviour (#313).
Verified on an M5 Pro under macOS 27.2.1: the full on-device suite and the corpus are green. Other chips on 27.2 have not been checked. See the new "macOS 27.2" section of the FAQ.
Signal processing
dsp.hilbert(analytic signal, #265),dsp.detrend(#278), and thesawtooth,squareandchirpgenerators (#264).- Windows:
flattop,blackmanharris,nuttall,cosine,gaussian(#268). fft.irfft(#261).
Linear algebra and special functions
linalg.kron(#262),linalg.polar(#259),linalg.matrix_rankandlinalg.cond(#260).special.digammaandspecial.beta(#250),special.bessel_j1andspecial.bessel_i1(#251).
Models and LLMs
- MPNet encoders (all-mpnet-base-v2) in the sentence-transformers drop-in (#255); RoBERTa-family encoders apply the
pad_id + 1position offset (#257). load_resnetaccepts a Hugging Face repo id (#256).- Qwen2/2.5 attention biases are loaded; the dense adapter dropped them (#253).
- Deep small-dimension models no longer hit the ANE op ceiling in decode (#254).
- MoE top-k selects exactly k experts on exact fp16 ties (#298).
ONNX
MeanVarianceNormalization,Hardmaxand the hyperbolic ops (#263).onnx.coverage_reportlists a model's unsupported ops before you try to compile it (#269).
Compiler
- The compile cache is reused instead of forcing recompilation (#284).
- A
compress=request that falls back to a wider encoding now warns (#249).
Benchmarks
- Rooflines for M1 (MacBook Air), M2 Max, M4 Pro, M4 Max and M6, and a re-run of the M1 MacBook Pro.
- ANE power from the SMC rail where powermetrics reports none (#294), with follow-up fixes for chips whose rail or IOReport layout differs (#305, #307, #308).
- A run whose idle baseline was not idle is flagged, and its perf/W is withheld from the table (#306).
- Result files are strict JSON: no bare
NaN(#307, #312).
Examples
- Conway's Game of Life (#272), the Mandelbrot set (#273) and the Lorenz attractor (#274), each stepped on the ANE.
- The whisper.cpp backend exports a usable encoder bundle (#275, #276).
Install: pip install -U aneforge (pretrained loaders: pip install "aneforge[models]").
artifact-v1: TACO energy-roofline paper artifact
Archival snapshot of the code, benchmark drivers, and committed result files behind A predictive, per-engine energy roofline of the Apple Neural Engine (ACM Transactions on Architecture and Code Optimization, accepted 2026).
The measurement drivers are in bench/ (device_saturation_sweep.py, device_bandwidth_roofline.py, roofline_suite.py, roofline_analysis.py, decode_measurement.py, and related scripts), and their outputs are in bench/results/.
This is not a package release, and nothing is published to PyPI. For the library, use the latest v* release. Cite the software as Bryngelson, Software: Practice and Experience (2026), doi:10.1002/spe.70124.
v0.4.0 - pretrained models on the engine: LLMs, CLIP, ViT, and Whisper
Pretrained models across four modalities now load from Hugging Face weights and run on the Apple Neural Engine, and LLM decode is faster by default. 46 commits since v0.3.0.
New model loaders
- Decoder LLMs: GPT-2 (#210), Gemma (GeGLU + scaled embeddings + (1+w) RMSNorm, #205), and Mistral (GQA + sliding window, #206) adapters, alongside the existing Llama/Qwen/MoE/hybrid path. Unsupported architectures are rejected rather than silently mis-loaded (#214).
- CLIP zero-shot image/text classification, both towers on the ANE (#225).
- ViT image classifiers (#212) and CrossEncoder rerankers, including RoBERTa/XLM-R (#208) and DistilBERT (#223).
- ResNet 18/34/50/101 from torchvision (#209).
- Whisper speech-to-text (whisper-base.en), audio encoder and text decoder both on the ANE (#237).
Faster LLM decode
- GPT-2 runs through the unified LLM runner with resident KV-cache decode, ~140 tok/s on gpt2-medium (#224, #229).
- The lm_head runs on the ANE by default, tiled along vocab, replacing the host fp32 matmul. Greedy stays bit-exact via a tie-margin host fallback, and an execute-path fallback keeps generate from breaking on a misconfigured target (#240, #243, #245). Pass ane_lm_head=False for the host matmul.
Demos and fixes
- RAG on the engine: chat with a folder of docs using embeddings, a reranker, and an LLM all on the ANE (examples/rag_chat.py, #231-236).
- The BERT encoder pads and masks a batch so varied-length inputs share one program instead of one per length (#233).
Applied math
Docs
- The manual now covers the full model-loader set, tightened to plain ASCII.
Install: pip install -U aneforge (pretrained loaders: pip install "aneforge[models]").
v0.3.0 — linear algebra, ONNX coverage, and hardened compiles
Linear algebra & scientific computing
linalg.solve(on-ANE pivoted LU) with vector and matrix right-hand sideslinalg.expm(scaling-and-squaring)det/slogdet, rank-kpinv,solve_triangular, matrix norms (fro/1/inf),matrix_powereinsumellipsis support and diagonal extractiondspKaiser/Bartlett/Tukey windows and Welch PSD
ONNX frontend
Large coverage expansion: LSTM/GRU/RNN (unrolled over the static sequence length),
Attention (opset 23) onto the native fused-attention layer, control flow (If / Loop),
Slice with step > 1, GlobalLpPool, Mod, Pad/Cast/Shrink/OneHot, tanh-approximate Gelu,
and roughly thirty more ops.
Compile correctness & robustness
- A differential fuzzer plus workarounds for the Espresso bugs it found
- The matmul fp16 saturation cliff (~fp16_max/2) characterized
opt=2fails closed when a variant's accuracy cannot be computedopt=1keeps the int8 variant when declared input activation ranges prove it safe
LLM inference
- Batched prefill (~45x lower TTFT)
- The 512-token attention context ceiling lifted (M*dh <= 65536)
- Llama-3.1
rope_scaling, multi-token EOS stop set
Tooling
- MLPerf-lite harness (ResNet-50 + LLM decode, real LoadGen)
- Cross-chip roofline suite and numeric-cliff sweeps
- Tiled decode vocab head so tok/s measures on every ANE family
Also
Python 3.14 in CI, AGENTS.md for coding-agent contributors, numeric-cliffs docs.
Full changelog: v0.2.0...v0.3.0
v0.2.0 - LLMs on the Apple Neural Engine
Decoder LLMs now run on the ANE end to end.
- Prefill + resident KV-cache decode for Llama/Qwen models, auto-segmented past the ~2GB single-program ceiling.
- Speculative decoding: exact, 2.28x on Qwen3-8B (verify(K) ~ verify(1) on the latency-bound ANE).
- Mixture-of-Experts from GGUF: Qwen3-MoE and Qwen2-MoE run the full model (router + all experts) on the ANE.
- Hybrid models: Qwen3.5 / Qwen3-Next (gated DeltaNet + gated attention). The real Qwen3.5-27B decodes coherently on a pure ANE from its GGUF at int8 (int8/fp16 compute is lower-fidelity than llama.cpp's fp32-accumulated Q4; see the LLMs guide).
- ONNX import and on-ANE finetuning of imported models.
- Config-driven model adapters: a new architecture is an adapter, not new branches in the runtime.
v0.1.4
Internal refactor + tooling release. Public API unchanged; all changes behavior-preserving (full on-device suite, 563 passing).
- Unified
graph_rewriteengine:canonicalize(lossless, on atopt>=1) + accuracy-gated numeric folding; single-pass tuner. - Repo restyled to tinygrad-compact 2-space; design rationale moved to
docs/developer/; in-code prose thinned. - CI gates: pyright (0), ruff C4/A, 2-space pylint lock, pre-commit hooks.
- Dedup: shared netplist bridge helpers, bench dead-code removal, test-helper consolidation.
ANEForge 0.1.3
Query-tiling for the attention helpers.
- af.sdpa and af.mha tile the query at large sequence, materializing [tile, Skv] score blocks instead of the full score matrix (~3x at S=1500). Exact (#17).
- cross_attention tiles the query when both query and context are long (S >= 768 and T >= 512, ~2.4x past the materialization wall). Small-T cross-attention (e.g. SD text conditioning) stays single-shot and byte-identical (#19).
No API changes.
v0.1.2
What's Changed
- Add arXiv paper (badge + citation); point docs at Read the Docs by @sbryngelson in #7
- perf(rms_norm): use fused reduce_l2_norm path (~48x faster) by @sbryngelson in #8
- Add Whisper encoder ANE benchmark by @sbryngelson in #9
- Report Whisper encoder fidelity on the trained checkpoint by @sbryngelson in #10
- Add Whisper encoder WER check and the whisper.cpp Metal anchor by @sbryngelson in #11
- Correct Whisper encoder latency framing, add CoreML comparison by @sbryngelson in #12
- Correct Whisper encoder benchmark to trained-weight numbers by @sbryngelson in #13
- Make channel_layer_norm a first-class op by @sbryngelson in #14
- Query-tile the Whisper encoder attention (kernel fission, ~2x) by @sbryngelson in #15
- release: bump to 0.1.2 by @sbryngelson in #16
Full Changelog: v0.1.1...v0.1.2
ANEForge 0.1.1
A maintenance release: bug fixes and a smoother first run. No API changes.
Fixes
- OpenMP no longer aborts on import.
import aneforgenow setsKMP_DUPLICATE_LIB_OKbefore NumPy loads, so the NumPy/Torch + ANE-runtime duplicate-OpenMP clash no longer aborts the process (#5). - OpenMP fork segfault in the test/bench harness fixed.
- Cross-compile guard: sub-MIL-floor targets are rejected cleanly instead of raising (#2).
- Runnable examples fixed: the
topkbatch path and the point-cloud demo no longer reference a path left dangling by the public-release scrub (#6).
Hygiene & docs
- Dropped the
KMP_DUPLICATE_LIB_OK=TRUEprefix from documented run commands and the stale_bridges"Run:" hints. - ASCII-only source; colourised example output; dropped ASCII-art dividers (#1, #3).
- Version derived from the git tag (hatch-vcs); added PyPI / Read the Docs / Zenodo badges; install leads with
pip install aneforge.
Install: pip install --upgrade aneforge
ANEForge 0.1.0
Initial public release.
ANEForge compiles tensor graphs and trains and runs them directly on the Apple Neural Engine from Python, with no CoreML.
pip install aneforge(Apple Silicon, macOS 14+); the e5rt dispatch shim compiles on first use.- Graph -> compile -> run; on-engine training (forward, backward, Adam); native layers via bridges; cross-chip compile for M1-M5.
- Docs: https://aneforge.readthedocs.io