Skip to content

v1.0.1 — ANE default restored

Choose a tag to compare

@john-rocky john-rocky released this 22 Apr 01:23
· 148 commits to main since this release
5771589

Hotfix for v1.0.0. The default compute units for Qwen3.5 had been temporarily switched to GPU during profile/benchmark debugging and weren't restored before the ship commit. v1.0.0 ran on GPU by default, consuming ~3 GB of extra Metal heap beyond the model weights.

Fix

Default restored to .cpuAndNeuralEngine for both prefill and decode. Behaves as the v1.0.0 release notes originally described.

Memory note (applies to both ANE and GPU paths)

A 0.8B fp16 model inherently needs ~1.4 GB of RAM just for weights — this is unified memory on Apple Silicon, shared between CPU mmap and the ANE plan cache. Total app memory on ANE mode sits around 1.6-2 GB (weights + ~200 MB ANE runtime + ~300 MB app baseline).

"0 GB Metal heap" means the GPU is not allocating additional buffer memory. On GPU mode, Metal allocates another ~3 GB on top of the 1.6 GB baseline, bringing total to ~4.6 GB.

path Metal heap total app memory
ANE (default) 0 GB ~1.6 GB
GPU (bit-exact) ~3 GB ~4.6 GB
CPU 0 GB ~1.6 GB

Included

  • #116 fix(qwen3.5): restore ANE default compute units