Skip to content

Releases

v0.5.0 - macOS 27.2 fixes, signal processing, and more linear algebra

Choose a tag to compare

@sbryngelson sbryngelson released this 21 Nov 15:48
e06669a

Fixes for macOS 27.2, where v0.4.0 fails to compile native attention and returns wrong results from pivoted LU, plus a round of signal-processing, linear-algebra and special-function additions. 63 commits since v0.4.0.

macOS 27.2

If you are on macOS 27.2, upgrade: v0.4.0 is affected by both of these.

  • Native bridge programs compile again. The framework now rejects a bridge bundle with two or more weight files (verifyBundleAtPath: invalid model), which broke af.sdpa, causal decoder blocks and fused ONNX attention. Such a bundle is retried with its weights merged into one file (#309).
  • A slice offset on the last axis no longer loses its offset. Two such slices at different offsets feeding matmuls in one fused program returned the wrong column or failed to compile on an M5 Pro, which made linalg.lu_pivoted, det, slogdet, solve and inv return wrong or non-finite results. The compiler now lowers a last-axis-offset slice_by_size (and so crop and static gather) off the last axis; ANEFORGE_ROUTE_WIDTH_SLICES=0 restores the old lowering (#311, #314). bench/slice_offset_probe.py shows whether a machine has the underlying behaviour (#313).

Verified on an M5 Pro under macOS 27.2.1: the full on-device suite and the corpus are green. Other chips on 27.2 have not been checked. See the new "macOS 27.2" section of the FAQ.

Signal processing

  • dsp.hilbert (analytic signal, #265), dsp.detrend (#278), and the sawtooth, square and chirp generators (#264).
  • Windows: flattop, blackmanharris, nuttall, cosine, gaussian (#268).
  • fft.irfft (#261).

Linear algebra and special functions

  • linalg.kron (#262), linalg.polar (#259), linalg.matrix_rank and linalg.cond (#260).
  • special.digamma and special.beta (#250), special.bessel_j1 and special.bessel_i1 (#251).

Models and LLMs

  • MPNet encoders (all-mpnet-base-v2) in the sentence-transformers drop-in (#255); RoBERTa-family encoders apply the pad_id + 1 position offset (#257).
  • load_resnet accepts a Hugging Face repo id (#256).
  • Qwen2/2.5 attention biases are loaded; the dense adapter dropped them (#253).
  • Deep small-dimension models no longer hit the ANE op ceiling in decode (#254).
  • MoE top-k selects exactly k experts on exact fp16 ties (#298).

ONNX

  • MeanVarianceNormalization, Hardmax and the hyperbolic ops (#263).
  • onnx.coverage_report lists a model's unsupported ops before you try to compile it (#269).

Compiler

  • The compile cache is reused instead of forcing recompilation (#284).
  • A compress= request that falls back to a wider encoding now warns (#249).

Benchmarks

  • Rooflines for M1 (MacBook Air), M2 Max, M4 Pro, M4 Max and M6, and a re-run of the M1 MacBook Pro.
  • ANE power from the SMC rail where powermetrics reports none (#294), with follow-up fixes for chips whose rail or IOReport layout differs (#305, #307, #308).
  • A run whose idle baseline was not idle is flagged, and its perf/W is withheld from the table (#306).
  • Result files are strict JSON: no bare NaN (#307, #312).

Examples

  • Conway's Game of Life (#272), the Mandelbrot set (#273) and the Lorenz attractor (#274), each stepped on the ANE.
  • The whisper.cpp backend exports a usable encoder bundle (#275, #276).

Install: pip install -U aneforge (pretrained loaders: pip install "aneforge[models]").

artifact-v1: TACO energy-roofline paper artifact

Choose a tag to compare

@sbryngelson sbryngelson released this 14 Nov 01:18
1a3f156

Archival snapshot of the code, benchmark drivers, and committed result files behind A predictive, per-engine energy roofline of the Apple Neural Engine (ACM Transactions on Architecture and Code Optimization, accepted 2026).

The measurement drivers are in bench/ (device_saturation_sweep.py, device_bandwidth_roofline.py, roofline_suite.py, roofline_analysis.py, decode_measurement.py, and related scripts), and their outputs are in bench/results/.

This is not a package release, and nothing is published to PyPI. For the library, use the latest v* release. Cite the software as Bryngelson, Software: Practice and Experience (2026), doi:10.1002/spe.70124.

v0.4.0 - pretrained models on the engine: LLMs, CLIP, ViT, and Whisper

Choose a tag to compare

@sbryngelson sbryngelson released this 22 Aug 14:18
d7e17d4

Pretrained models across four modalities now load from Hugging Face weights and run on the Apple Neural Engine, and LLM decode is faster by default. 46 commits since v0.3.0.

New model loaders

  • Decoder LLMs: GPT-2 (#210), Gemma (GeGLU + scaled embeddings + (1+w) RMSNorm, #205), and Mistral (GQA + sliding window, #206) adapters, alongside the existing Llama/Qwen/MoE/hybrid path. Unsupported architectures are rejected rather than silently mis-loaded (#214).
  • CLIP zero-shot image/text classification, both towers on the ANE (#225).
  • ViT image classifiers (#212) and CrossEncoder rerankers, including RoBERTa/XLM-R (#208) and DistilBERT (#223).
  • ResNet 18/34/50/101 from torchvision (#209).
  • Whisper speech-to-text (whisper-base.en), audio encoder and text decoder both on the ANE (#237).

Faster LLM decode

  • GPT-2 runs through the unified LLM runner with resident KV-cache decode, ~140 tok/s on gpt2-medium (#224, #229).
  • The lm_head runs on the ANE by default, tiled along vocab, replacing the host fp32 matmul. Greedy stays bit-exact via a tie-margin host fallback, and an execute-path fallback keeps generate from breaking on a misconfigured target (#240, #243, #245). Pass ane_lm_head=False for the host matmul.

Demos and fixes

  • RAG on the engine: chat with a folder of docs using embeddings, a reranker, and an LLM all on the ANE (examples/rag_chat.py, #231-236).
  • The BERT encoder pads and masks a batch so varied-length inputs share one program instead of one per length (#233).

Applied math

  • linalg.inv and QR-based least squares (#197); erf computed as its own polynomial (#196).

Docs

  • The manual now covers the full model-loader set, tightened to plain ASCII.

Install: pip install -U aneforge (pretrained loaders: pip install "aneforge[models]").

v0.3.0 — linear algebra, ONNX coverage, and hardened compiles

Choose a tag to compare

@sbryngelson sbryngelson released this 05 Aug 15:45
6410847

Linear algebra & scientific computing

  • linalg.solve (on-ANE pivoted LU) with vector and matrix right-hand sides
  • linalg.expm (scaling-and-squaring)
  • det/slogdet, rank-k pinv, solve_triangular, matrix norms (fro/1/inf), matrix_power
  • einsum ellipsis support and diagonal extraction
  • dsp Kaiser/Bartlett/Tukey windows and Welch PSD

ONNX frontend

Large coverage expansion: LSTM/GRU/RNN (unrolled over the static sequence length),
Attention (opset 23) onto the native fused-attention layer, control flow (If / Loop),
Slice with step > 1, GlobalLpPool, Mod, Pad/Cast/Shrink/OneHot, tanh-approximate Gelu,
and roughly thirty more ops.

Compile correctness & robustness

  • A differential fuzzer plus workarounds for the Espresso bugs it found
  • The matmul fp16 saturation cliff (~fp16_max/2) characterized
  • opt=2 fails closed when a variant's accuracy cannot be computed
  • opt=1 keeps the int8 variant when declared input activation ranges prove it safe

LLM inference

  • Batched prefill (~45x lower TTFT)
  • The 512-token attention context ceiling lifted (M*dh <= 65536)
  • Llama-3.1 rope_scaling, multi-token EOS stop set

Tooling

  • MLPerf-lite harness (ResNet-50 + LLM decode, real LoadGen)
  • Cross-chip roofline suite and numeric-cliff sweeps
  • Tiled decode vocab head so tok/s measures on every ANE family

Also

Python 3.14 in CI, AGENTS.md for coding-agent contributors, numeric-cliffs docs.

Full changelog: v0.2.0...v0.3.0

v0.2.0 - LLMs on the Apple Neural Engine

Choose a tag to compare

@sbryngelson sbryngelson released this 28 Jun 16:21
c0440cc

Decoder LLMs now run on the ANE end to end.

  • Prefill + resident KV-cache decode for Llama/Qwen models, auto-segmented past the ~2GB single-program ceiling.
  • Speculative decoding: exact, 2.28x on Qwen3-8B (verify(K) ~ verify(1) on the latency-bound ANE).
  • Mixture-of-Experts from GGUF: Qwen3-MoE and Qwen2-MoE run the full model (router + all experts) on the ANE.
  • Hybrid models: Qwen3.5 / Qwen3-Next (gated DeltaNet + gated attention). The real Qwen3.5-27B decodes coherently on a pure ANE from its GGUF at int8 (int8/fp16 compute is lower-fidelity than llama.cpp's fp32-accumulated Q4; see the LLMs guide).
  • ONNX import and on-ANE finetuning of imported models.
  • Config-driven model adapters: a new architecture is an adapter, not new branches in the runtime.

Docs: https://aneforge.readthedocs.io

v0.1.4

Choose a tag to compare

@sbryngelson sbryngelson released this 25 Jun 20:05
1b1bf90

Internal refactor + tooling release. Public API unchanged; all changes behavior-preserving (full on-device suite, 563 passing).

  • Unified graph_rewrite engine: canonicalize (lossless, on at opt>=1) + accuracy-gated numeric folding; single-pass tuner.
  • Repo restyled to tinygrad-compact 2-space; design rationale moved to docs/developer/; in-code prose thinned.
  • CI gates: pyright (0), ruff C4/A, 2-space pylint lock, pre-commit hooks.
  • Dedup: shared netplist bridge helpers, bench dead-code removal, test-helper consolidation.

ANEForge 0.1.3

Choose a tag to compare

@sbryngelson sbryngelson released this 22 Jun 16:35
a549bc7

Query-tiling for the attention helpers.

  • af.sdpa and af.mha tile the query at large sequence, materializing [tile, Skv] score blocks instead of the full score matrix (~3x at S=1500). Exact (#17).
  • cross_attention tiles the query when both query and context are long (S >= 768 and T >= 512, ~2.4x past the materialization wall). Small-T cross-attention (e.g. SD text conditioning) stays single-shot and byte-identical (#19).

No API changes.

v0.1.2

Choose a tag to compare

@sbryngelson sbryngelson released this 22 Jun 07:59
849bb58

What's Changed

  • Add arXiv paper (badge + citation); point docs at Read the Docs by @sbryngelson in #7
  • perf(rms_norm): use fused reduce_l2_norm path (~48x faster) by @sbryngelson in #8
  • Add Whisper encoder ANE benchmark by @sbryngelson in #9
  • Report Whisper encoder fidelity on the trained checkpoint by @sbryngelson in #10
  • Add Whisper encoder WER check and the whisper.cpp Metal anchor by @sbryngelson in #11
  • Correct Whisper encoder latency framing, add CoreML comparison by @sbryngelson in #12
  • Correct Whisper encoder benchmark to trained-weight numbers by @sbryngelson in #13
  • Make channel_layer_norm a first-class op by @sbryngelson in #14
  • Query-tile the Whisper encoder attention (kernel fission, ~2x) by @sbryngelson in #15
  • release: bump to 0.1.2 by @sbryngelson in #16

Full Changelog: v0.1.1...v0.1.2

ANEForge 0.1.1

Choose a tag to compare

@sbryngelson sbryngelson released this 14 Jun 21:50

A maintenance release: bug fixes and a smoother first run. No API changes.

Fixes

  • OpenMP no longer aborts on import. import aneforge now sets KMP_DUPLICATE_LIB_OK before NumPy loads, so the NumPy/Torch + ANE-runtime duplicate-OpenMP clash no longer aborts the process (#5).
  • OpenMP fork segfault in the test/bench harness fixed.
  • Cross-compile guard: sub-MIL-floor targets are rejected cleanly instead of raising (#2).
  • Runnable examples fixed: the topk batch path and the point-cloud demo no longer reference a path left dangling by the public-release scrub (#6).

Hygiene & docs

  • Dropped the KMP_DUPLICATE_LIB_OK=TRUE prefix from documented run commands and the stale _bridges "Run:" hints.
  • ASCII-only source; colourised example output; dropped ASCII-art dividers (#1, #3).
  • Version derived from the git tag (hatch-vcs); added PyPI / Read the Docs / Zenodo badges; install leads with pip install aneforge.

Install: pip install --upgrade aneforge

ANEForge 0.1.0

Choose a tag to compare

@sbryngelson sbryngelson released this 12 Jun 21:15

Initial public release.

ANEForge compiles tensor graphs and trains and runs them directly on the Apple Neural Engine from Python, with no CoreML.

  • pip install aneforge (Apple Silicon, macOS 14+); the e5rt dispatch shim compiles on first use.
  • Graph -> compile -> run; on-engine training (forward, backward, Adam); native layers via bridges; cross-chip compile for M1-M5.
  • Docs: https://aneforge.readthedocs.io