inferd 0.3.0

inferd 0.3.0: a native inference engine for Apple Silicon (C++, Objective-C++ and Metal), built for many AI agents
at once on one Mac. It is experimental software.

Getting started in four steps

1. Download inferd-0.3.0-macos27-arm64.tar.gz and its .sha256 below, check it (shasum -a 256 -c …) and unpack it.
2. ./install.sh --model 27b: a Python 3.12 venv for the front end, plus Qwen3.8-27B 4-bit and its DFlash2 drafter in ~/models.
3. ./run.sh start --model 27b: the server on port 8300, sized from your Mac's memory (--dry-run shows the command lines).
4. Call it with any OpenAI client at http://localhost:8300/v1.

The tarballs are self-contained (engine binaries, front end, release gate, licences; no sources).

• macOS 27: inferd-0.3.0-macos27-arm64.tar.gz, tested on Mac Studio, Mac minis and MacBook Pros.
• macOS 26 (beta): inferd-0.3.0-macos26-arm64.tar.gz, added 5 Nov. Built natively on macOS 26.6.2 with Xcode 26.5
  (in a macOS 26 virtual machine, from the same commit 8ba6afa; host tests pass). Not yet run on a macOS 26 Mac's GPU:
  run the release gate once (PY=~/.venvs/inferd/bin/python tools/release_gate.sh ~/models/Qwen3.8-27B-4bit ~/models/Qwen3.8-27B-DFlash2) and tell us how it went. Building from source (git checkout v0.3.0 && cd engine && ./build.sh) remains an option. Never run a build made for macOS 26 on macOS 27 (run.sh refuses it).

What's new since 0.2.0

• Getting started: install.sh, the self-contained tarball (tools/package.sh), and run.sh start --model 27b|27b-8bit|flash-next|<dir> with named options. The positional form, stop and status work as before;
  status now reports the log directory start used.
• QA8, on by default on GPUs with neural accelerators: int8 prefill read straight from the packed 4-bit weights,
  with no weight copy (it replaces PF8 there). This frees memory for KV. Adapted from oMLX's oq_a8_nax.metal
  (Apache-2.0; see NOTICE).
• Memory: 8-bit drafter context rings (default) and opt-in 4-bit KV (KV4=1, Hadamard-rotated).
• Sampling: one contract on every path, with exact speculative sampling up to top_k 64. Requests that omit
  top_k/top_p take the model's generation_config defaults (Qwen3.8: 20 / 0.95), as vLLM does. "spec_sampling": "truncate" is opt-in.
• Qwen3.6-35B-A3B: the quantization group size is now read per checkpoint. 0.2.0 misloaded Qwen's official
  group-128 build and computed wrong output, so its 35B figures are withdrawn. The DFlash2 drafter now works on MoE
  targets, and an 8-row MoE verify path is added.
• Release gate (tools/release_gate.sh): chunks 1/24/256/1536 on 4 × 2,048 tokens with cos ≥ 0.995 and KL ≤ 0.03
  enforced, speculative divergences judged by their reference logit gap (tie_check.py), and the sampling contract
  checked on the GPU.
• Fixes: findings R01–R08 from the 4 November code and security review, and two external test reports (an M4 Max
  on macOS 26 and 27, an M4 Pro). Notably ocr_agents now fails on a bad field. Details are in the
  docs/CHANGES-* files.

Measured

Same weights, drafter, machine and harness for every engine. Full tables are in
docs/RESULTS.md (r)–(t).

• Mac Studio M5 Ultra, Qwen3.8-27B, 32 agents on 16k contexts: 711–727 tok/s, against 613–621 for 0.2.0 in the
  same session. The slowest agent's first token is 1.6–1.7 s, against 2.2–2.3 s. With 1 and 8 agents the two
  releases are level. Splash 1.5 levels off at about 369.
• Mac mini M6, 32 GB: prefill on a 16k prompt went from 62 s to 39–40 s (QA8). With opt-in 4-bit KV, 8 agents on
  16k contexts reach 92.6 tok/s against 66.6 with 8-bit KV.
• Mac minis, Qwen3.6-35B-A3B with the drafter: on the M5 Pro, one agent decodes at 151–153 tok/s (0.2.0 on the
  same checkpoint: 102) and the fleet reaches 59–62 turns per minute (64–71 with 32 running). Splash 1.5 is still
  faster on this model: 209–219 tok/s and 76–81 turns per minute.
• M3/M4 GPUs (external M4 Pro, 48 GB): with the drafter, 16 agents on 4k follow-ups reach 69.0 tok/s, against 35.3
  on an earlier build and 63.2 without the drafter. The drafter now pays at 16 agents there too.

Where others lead

• Single-stream decode, and one agent on short contexts: Splash, TensorFold and mlx-serve are faster.
• On a 32 GB Mac mini with the 27B at 16k and 8–16 agents, Splash 1.5 is faster with 8-bit KV.
• On Qwen3.6-35B-A3B, Splash 1.5 is faster (see above).

Checked for this release on a MacBook Pro M5 Max (macOS 27.0.1) from a fresh unpack of the tarball: install.sh and
the full release gate. The engine binaries are identical to the build measured above on the Mac Studio and the minis.