Skip to content

inferd 0.2.0

Choose a tag to compare

View all tags
@johntdavies johntdavies released this 03 Nov 23:02
v0.2.0
181c315

inferd 0.2.0: a native inference engine for Apple Silicon (C++, Objective-C++ and Metal), built for many AI agents
at once on one Mac. This is the first public-ready release. It is experimental software.

What's in it

  • Serving: continuous batching, chunked prefill interleaved with decoding, admission by KV budget with
    preemption, admission lookahead with aging, and a prefill budget beside the decoders.
  • Speed paths: W4A8 (int8 activations on the GPU's integer tensor path), automatic int8 prefill (PF8), DFlash2
    speculative decoding for every agent (tree verification for one agent), and Flash-Next MTP plus copy drafts.
  • Memory: 8-bit paged KV cache, a prefix trie with recurrent-state snapshots for hybrid models, and an SSD tier
    with checksums (SPILL_DIR).
  • Agent API: POST/GET/DELETE /v1/contexts (saved and pinned contexts), sessions, cache: drop, per-agent
    effort, max_context and priority, GET /v1/cache, and cached_tokens in every reply.
  • OpenAI-compatible front end: chat completions with streaming, tool calls, stop strings, strict JSON Schema
    (response_format) and images (VISION=1).
  • Operation: run.sh start | stop | status, a startup warm-up, loopback by default, an optional API key
    (required off loopback), request, queue and body limits, and validated checkpoints and requests.
  • Models: Qwen3.8-27B (4-bit, 8-bit, bf16) with the incoai DFlash2 drafter, and Qwen3.8-Flash-Next (oQ4e). Smoke
    ports: Qwen3.6-35B-A3B, Qwen3.5-9B, Muse Glimmer 30B and Gemma-4-31B.

Measured

Same weights, drafter, machine and harness for every engine. Full tables are in
docs/RESULTS.md.

  • Mac Studio M5 Ultra:
    • 32 agents on 16k contexts: 615–620 tok/s in total (620 on this release's tarball). Splash 1.5 levels off at about 369.
    • 32 agents on 64k contexts: the last agent starts in 3.8 s, against 48.9 s.
    • 8 Qwen3.8-27B agents released together: all finish in 125.6 s, against 177.6 s on TensorFold 1.2.2 (177.9 s on 1.2.1).
    • Agents coming back after unrelated work: 32 forked agents relaunch in 1.03 s on the Studio; on the M5 Pro relaunch 2.1 s and resume 0.25 s, against 195 s and 15.4 s on TensorFold 1.2.2 with its disk cache.
    • 16 agents arriving at random: all finish in 78.7 s, against 116.7 s.
  • Mac minis:
    • M5 Pro, 16 agents at 16k: 14.3 turns per minute, against 12.0 on Splash 1.5.
    • M6 (32 GB), measured 4 November: Qwen3.8-27B agent fleets on 8k contexts, 16 agents: 9.7 turns per minute against
      8.8 on Splash 1.5 (14.3 with the experimental Neural Engine split). With 8–16 agents on 16k contexts Splash 1.5 is
      faster (86–89 vs 64–66 tok/s): the 32 GB KV pool holds about four 16k contexts. TensorFold 1.2.2 cannot load the
      27B on 32 GB. (This line replaced "25% ahead of Splash 1.5 at 16 agents", which the 4 November runs contradict.)
  • Where others lead:
    • Single-stream decode, and one agent on short contexts: Splash, TensorFold and mlx-serve are faster.
    • Flash-Next with 1–5 agents starting together: TensorFold finishes sooner; with Flash-Next agents arriving at random the two are level on total time and TensorFold's slowest first tokens come sooner.
  • Quality:
    • Logits match mlx_lm (top-1).
    • HLE Lite: 27B 4-bit 32.4%, 8-bit 34.1%, Flash-Next 41.6%.
    • Strict JSON: 100/100 replies valid on the first attempt.

Requirements

  • System: Apple Silicon on macOS 27. The prebuilt binaries need macOS 27.
  • GPUs: M5-class and newer GPUs are the main target. M3/M4 run on their own kernel path and are lightly tested.
    M1/M2 are untested.
  • Software: Python 3.12 for the front end (engine/requirements.txt). Building from source needs Xcode 27's
    Metal toolchain.

Prebuilt binaries

inferd-0.2.0-macos27-arm64.tar.gz contains build/ (the inferd server, tools and core.metallib) for Apple
Silicon on macOS 27.

  1. Check it with shasum -a 256 -c inferd-0.2.0-macos27-arm64.tar.gz.sha256.
  2. Unpack it inside engine/ of a checkout of this tag.
  3. Run ./run.sh without building.

These binaries were tested on a Mac mini M4 Pro (logits, speculative equals plain, and the examples suite).

Acknowledgements

inferd stands on the shoulders of these projects:

  • Apple's MLX: inferd vendors its quantized-matmul, GEMM and attention kernels
    (MIT).
  • mlx-lm: the reference every logit is checked against.
  • mlx-vlm (Prince Canuma): the vision path.
  • Qwen's open weights.
  • DFlash (z-lab) and the
    incoai DFlash2 drafter.
  • TensorFold: the fastest single stream on a Mac, and the mlx.fast competition.
  • Splash: it showed what a Mac can do for agents.
  • oMLX and mlx-serve.
  • MiaAI-Lab: their benchmark recipes.
  • vLLM: its serving design.
  • Hugging Face.

Licence: Apache-2.0 (see NOTICE).

Tested on this tarball (4 November): Mac Studio M5 Ultra, Mac mini M5 Pro, M6 and M4 Pro; results in docs/RESULTS.md (n).

Downloads

macOS File Status
27 inferd-0.2.0-macos27-arm64.tar.gz (+ .sha256) tested on Mac Studio M5 Ultra, Mac mini M6 / M5 Pro / M4 Pro
26 inferd-0.2.0-macos26-arm64.tar.gz (+ .sha256) beta, added 4 November: built from the same tag for a macOS 26 target. Run tools/release_gate.sh once. Do not use it on macOS 27 (wrong logits at 256+ row prefill chunks there)
Assets 6
Loading