Skip to content

inferd 0.4.4

Choose a tag to compare

@johntdavies johntdavies released this 16 Nov 13:46
· 4 commits to main since this release

The Neural Engine prompt split is on by default again on M4 and M4 Pro (26 units).

0.4.2 turned the split off because, with several agents and the drafter, some replies stopped after 2–3 tokens. 0.4.3 fixed the cause: a decoder's draft token tree was written after a split step had already run. 0.4.4 restores the 0.4.0–0.4.1 default. ANE_UNITS still overrides it (ANE_UNITS=0 turns the split off).

Mac mini M4 Pro (20-core GPU, 64 GB), Qwen3.8-27B 4-bit + DFlash2, agent fleets on 8k contexts, defaults only, 0.4.4 and 0.4.3 interleaved, 2 trials each, 0 errors:

Agents 0.4.3 (split off) turns/min 0.4.4 (split on) turns/min Slowest first token (p95), 0.4.3 → 0.4.4
8 5.29 7.15 (+35%) 75.7 → 52.2 s
16 5.38 7.34 (+36%) 154.0 → 108.1 s
  • Replies cut short (under 16 tokens): 0 of 96 turns on 0.4.4 (shortest reply 73 tokens).
  • The release gate passed on the M4 Pro with this tarball.
  • Nothing else changes: on Macs with GPU neural accelerators (M5, M6) and on every other Mac without them, 0.4.4 runs exactly as 0.4.3.

Also in this release's tree: external results from a Mac Studio M5 Max 128 GB (#4, thanks @aquariumyossy), docs/RESULTS.md (ah).

macOS 27 only; there is no macOS 26 file yet (see the README).