inferd 0.4.1

inferd 0.4.1: first-run fixes on top of 0.4.0. The engine is unchanged: same kernels and same performance.

Downloads

• macOS 27: inferd-0.4.1-macos27-arm64.tar.gz
• macOS 26: inferd-0.4.1-macos26-arm64.tar.gz, built natively on macOS 26.6.2 with Xcode 26.5. 0.4.0's macOS 26 file
  (same engine and toolchain) passed the full release gate on two testers' Macs on macOS 26.6: a 128 GB Mac and a MacBook
  Pro M4 Max, 36 GB. Never run a macOS 26 build on macOS 27 (run.sh refuses it).

Then ./install.sh --model 27b and ./run.sh start --model 27b (see the README's Getting started).

What's fixed (from external testers' reports)

• Building from source with only the Command Line Tools selected: build.sh now finds Xcode by itself
  (/Applications/Xcode*.app with a Metal compiler), says so, and tells you how to make it permanent
  (sudo xcode-select -s …). Without Xcode it says exactly what to install.
• install.sh on a venv made by uv, which has no pip: it now uses uv pip, else ensurepip, else a clear message.
• The SSD tier persists across restarts by design, so agents can come back from it. The new ./run.sh clean [port]
  removes a port's tier; it refuses while that server is running.
• GET /version now reports the release; it said 0.3.0-dev.
• examples/run_all.sh defaults its machine name to the Mac's name.
• examples/agents-at-once writes its results under $TMPDIR, not into the checkout.
• Docs:
  • the macOS 26 status is corrected, now with two Macs that passed the gate;
  • on macOS 26 the gate may show a Python crash report, because test_image_decode aborts a helper process on purpose;
    it is not a failure;
  • the Neural Engine split is on by default for M4 / M4 Pro only, and off on Max chips;
  • the coding-smoke score ranges are stated per thinking effort.

New: check it yourself

examples/agents-at-once runs N agents at once on their own contexts and checks every agent's answer. It reports tok/s,
and with any watt meter, tokens per joule. On a Mac Studio with Flash-Next:

agents	answers right	total tok/s	wall power	tokens per joule
64	128 / 128	1,326	298 W	4.4
100	200 / 200	1,457	300 W	4.9
128	256 / 256	1,487	293 W	5.1

In every run, all the agents were generating at the same moment.

Performance numbers are unchanged from 0.4.0. See the README and docs/RESULTS.md.

inferd 0.4.0

inferd 0.4.0: a native inference engine for Apple Silicon (C++, Objective-C++ and Metal), built for many AI agents
at once on one Mac. It is experimental software.

64 agents at once on one Mac Studio: 849 tok/s on Qwen3.8-27B and 1,289–1,323 tok/s on Qwen3.8-Flash-Next, each
agent on its own 16k context. On a Mac mini, an agent fleet on the 27B completes 13–19 turns a minute (M6 32 GB, M5 Pro
64 GB), and the 35B MoE model 81–93.

Getting started in four steps

1. Download the file for your macOS below and its .sha256, check it (shasum -a 256 -c …) and unpack it.
2. ./install.sh --model 27b: a Python 3.12 venv for the front end, plus Qwen3.8-27B 4-bit and its DFlash2 drafter. If
   the model is already in ~/models, the Hugging Face cache or LM Studio, it is used where it is (no download).
3. ./run.sh start --model 27b: the server on port 8300, sized from your Mac's memory.
4. Call it with any OpenAI (or vLLM) client at http://localhost:8300/v1.

• macOS 27: inferd-0.4.0-macos27-arm64.tar.gz.
• macOS 26 (beta): inferd-0.4.0-macos26-arm64.tar.gz, built natively on macOS 26 with Xcode 26.5 from the same
  commit. It has not yet run on a macOS 26 Mac's GPU: run the release gate once
  (PY=~/.venvs/inferd/bin/python tools/release_gate.sh ~/models/Qwen3.8-27B-4bit ~/models/Qwen3.8-27B-DFlash2). Never
  run a macOS 26 build on macOS 27 (run.sh refuses it). Building from source (git checkout v0.4.0 && cd engine && ./build.sh) remains an option.

What's new since 0.3.0

• vLLM-compatible API.
  • New endpoints: /v1/completions, /tokenize, /detokenize, /version, /ping and a Prometheus /metrics.
  • New request fields:
    • logprobs, from the verify rows so speculation stays on;
    • presence, frequency and repetition penalties, min_p, logit_bias and min_tokens;
    • n;
    • guided JSON, choice and regex, and structured_outputs;
    • every tool_choice mode.
  • Unsupported endpoints (embeddings, rerank, audio, LoRA) answer a clean 501.
  • Of 78 probe items, 0 are broken (19 before). Details: docs/API-COMPAT.md.
• Qwen3.6-35B-A3B:
  • int8 experts on the QA8 path, the router as a matmul from 64 rows, and drafter attention on simdgroup matrices: the
    M5 Pro fleet goes from 60–71 to 81–93 turns/min;
  • decode snapshots turn off when a chat template drops earlier reasoning: on the Studio, 16k x 32 agents goes from 269
    to 1,079 tok/s.
• M4 / M4 Pro: the Neural Engine FFN split is on by default (26 units). Prefill is about 1.5x faster (8k prompt 64.2
  → 42.8 s) and decode is level. ANE_UNITS=0 turns it off.
• Models you already have: install.sh and run.sh start --model look in ~/models, the Hugging Face cache and LM
  Studio, then use the first complete copy. Known equivalent builds are accepted, with a note, when their config matches;
  for example Qwen's own MLX 4-bit, or z-lab's byte-identical drafter. --exact takes only the exact repo ids.
• Sizing for many agents (run.sh start):
  • the Mac Studio gets 192 snapshots: the 27B fleet with 64 in flight goes from 59.6 to 95.3 turns/min;
  • Flash-Next gets 64 running and 128 snapshots: 64 agents at 16k go from 209–287 to 1,289–1,323 tok/s.
• Release: a native macOS 26 build. package.sh checks that every file the release gate needs is in the tarball.
• Fixes: the M4 Max report findings, run.sh's CI tests, and more. See docs/CHANGES-*.

Measured (5 November, full tables in docs/RESULTS.md (t)–(y))

Mac mini

• M6, 32 GB:
  • 27B agent fleet on 8k, 13.3–14.0 turns/min;
  • 9B, 16 agents on 16k, 164 tok/s (8-bit 117);
  • agents launched from a saved context in 2.3 s, resumed in 0.4 s.
• M5 Pro, 64 GB:
  • 27B fleets 18.2–19.0 (16k) and 14.4–15.7 (32k) turns/min;
  • the 35B fleet 81–93;
  • 35B, 32 agents, 610 tok/s at 4k and 467 at 16k;
  • agents resumed in 0.14–0.25 s.
• M4 Pro, 64 GB: 27B, 8k prompt, first token 42.8 s (0.3.0: 64.2 s).

Mac Studio M5 Ultra

• 27B:
  • 64 agents on 16k, 849 tok/s (32 agents: 722);
  • the agent fleet with 64 in flight, 95.3 turns/min;
  • 32 agents on 64k, the last agent's first token in 3.3–3.4 s.
• Flash-Next, 64 agents, 1,289–1,323 tok/s.
• 35B, 32 agents on 16k, 1,079 tok/s.
• One agent saves a 33.5k-token context and 31 start from it: median first token 0.72 s.

Compared with other engines, same weights, machine and harness, run back to back:

• Studio 27B fleet: 80–95 vs Splash 1.5's 48–50 turns/min.
• M5 Pro:
  • 27B fleets ahead of Splash 1.5 by 41–60%;
  • 35B fleet 81–93 vs 78–79;
  • 9B fleets 7.5x (4-bit) and 15.6x (8-bit) TensorFold 1.2.2 at 32 agents.
• M6:
  • 27B fleet 13.3 vs Splash's 8.8 turns/min;
  • TensorFold 1.2.2 cannot load the 27B on 32 GB.

Where others lead

• One agent on its own: Splash, TensorFold and mlx-serve decode a single stream faster.
• Splash is faster with 4–8 agents taking turns on the minis, and for the 35B on the 32 GB M6 at 16k.
• TensorFold finishes Flash-Next sooner with 1–5 agents starting together.

Checked for this release on a MacBook Pro M5 Max (macOS 27), from a fresh unpack of the macOS 27 tarball: install.sh
and the full release gate. The performance numbers were measured on main builds whose engine code matches this release
for those paths; the A/Bs are in RESULTS.md.

inferd 0.3.0

inferd 0.3.0: a native inference engine for Apple Silicon (C++, Objective-C++ and Metal), built for many AI agents
at once on one Mac. It is experimental software.

Getting started in four steps

1. Download inferd-0.3.0-macos27-arm64.tar.gz and its .sha256 below, check it (shasum -a 256 -c …) and unpack it.
2. ./install.sh --model 27b: a Python 3.12 venv for the front end, plus Qwen3.8-27B 4-bit and its DFlash2 drafter in ~/models.
3. ./run.sh start --model 27b: the server on port 8300, sized from your Mac's memory (--dry-run shows the command lines).
4. Call it with any OpenAI client at http://localhost:8300/v1.

The tarballs are self-contained (engine binaries, front end, release gate, licences; no sources).

• macOS 27: inferd-0.3.0-macos27-arm64.tar.gz, tested on Mac Studio, Mac minis and MacBook Pros.
• macOS 26 (beta): inferd-0.3.0-macos26-arm64.tar.gz, added 5 Nov. Built natively on macOS 26.6.2 with Xcode 26.5
  (in a macOS 26 virtual machine, from the same commit 8ba6afa; host tests pass). Not yet run on a macOS 26 Mac's GPU:
  run the release gate once (PY=~/.venvs/inferd/bin/python tools/release_gate.sh ~/models/Qwen3.8-27B-4bit ~/models/Qwen3.8-27B-DFlash2) and tell us how it went. Building from source (git checkout v0.3.0 && cd engine && ./build.sh) remains an option. Never run a build made for macOS 26 on macOS 27 (run.sh refuses it).

What's new since 0.2.0

• Getting started: install.sh, the self-contained tarball (tools/package.sh), and run.sh start --model 27b|27b-8bit|flash-next|<dir> with named options. The positional form, stop and status work as before;
  status now reports the log directory start used.
• QA8, on by default on GPUs with neural accelerators: int8 prefill read straight from the packed 4-bit weights,
  with no weight copy (it replaces PF8 there). This frees memory for KV. Adapted from oMLX's oq_a8_nax.metal
  (Apache-2.0; see NOTICE).
• Memory: 8-bit drafter context rings (default) and opt-in 4-bit KV (KV4=1, Hadamard-rotated).
• Sampling: one contract on every path, with exact speculative sampling up to top_k 64. Requests that omit
  top_k/top_p take the model's generation_config defaults (Qwen3.8: 20 / 0.95), as vLLM does. "spec_sampling": "truncate" is opt-in.
• Qwen3.6-35B-A3B: the quantization group size is now read per checkpoint. 0.2.0 misloaded Qwen's official
  group-128 build and computed wrong output, so its 35B figures are withdrawn. The DFlash2 drafter now works on MoE
  targets, and an 8-row MoE verify path is added.
• Release gate (tools/release_gate.sh): chunks 1/24/256/1536 on 4 × 2,048 tokens with cos ≥ 0.995 and KL ≤ 0.03
  enforced, speculative divergences judged by their reference logit gap (tie_check.py), and the sampling contract
  checked on the GPU.
• Fixes: findings R01–R08 from the 4 November code and security review, and two external test reports (an M4 Max
  on macOS 26 and 27, an M4 Pro). Notably ocr_agents now fails on a bad field. Details are in the
  docs/CHANGES-* files.

Measured

Same weights, drafter, machine and harness for every engine. Full tables are in
docs/RESULTS.md (r)–(t).

• Mac Studio M5 Ultra, Qwen3.8-27B, 32 agents on 16k contexts: 711–727 tok/s, against 613–621 for 0.2.0 in the
  same session. The slowest agent's first token is 1.6–1.7 s, against 2.2–2.3 s. With 1 and 8 agents the two
  releases are level. Splash 1.5 levels off at about 369.
• Mac mini M6, 32 GB: prefill on a 16k prompt went from 62 s to 39–40 s (QA8). With opt-in 4-bit KV, 8 agents on
  16k contexts reach 92.6 tok/s against 66.6 with 8-bit KV.
• Mac minis, Qwen3.6-35B-A3B with the drafter: on the M5 Pro, one agent decodes at 151–153 tok/s (0.2.0 on the
  same checkpoint: 102) and the fleet reaches 59–62 turns per minute (64–71 with 32 running). Splash 1.5 is still
  faster on this model: 209–219 tok/s and 76–81 turns per minute.
• M3/M4 GPUs (external M4 Pro, 48 GB): with the drafter, 16 agents on 4k follow-ups reach 69.0 tok/s, against 35.3
  on an earlier build and 63.2 without the drafter. The drafter now pays at 16 agents there too.

Where others lead

• Single-stream decode, and one agent on short contexts: Splash, TensorFold and mlx-serve are faster.
• On a 32 GB Mac mini with the 27B at 16k and 8–16 agents, Splash 1.5 is faster with 8-bit KV.
• On Qwen3.6-35B-A3B, Splash 1.5 is faster (see above).

Checked for this release on a MacBook Pro M5 Max (macOS 27.0.1) from a fresh unpack of the tarball: install.sh and
the full release gate. The engine binaries are identical to the build measured above on the Mac Studio and the minis.

inferd 0.2.0

inferd 0.2.0: a native inference engine for Apple Silicon (C++, Objective-C++ and Metal), built for many AI agents
at once on one Mac. This is the first public-ready release. It is experimental software.

What's in it

• Serving: continuous batching, chunked prefill interleaved with decoding, admission by KV budget with
  preemption, admission lookahead with aging, and a prefill budget beside the decoders.
• Speed paths: W4A8 (int8 activations on the GPU's integer tensor path), automatic int8 prefill (PF8), DFlash2
  speculative decoding for every agent (tree verification for one agent), and Flash-Next MTP plus copy drafts.
• Memory: 8-bit paged KV cache, a prefix trie with recurrent-state snapshots for hybrid models, and an SSD tier
  with checksums (SPILL_DIR).
• Agent API: POST/GET/DELETE /v1/contexts (saved and pinned contexts), sessions, cache: drop, per-agent
  effort, max_context and priority, GET /v1/cache, and cached_tokens in every reply.
• OpenAI-compatible front end: chat completions with streaming, tool calls, stop strings, strict JSON Schema
  (response_format) and images (VISION=1).
• Operation: run.sh start | stop | status, a startup warm-up, loopback by default, an optional API key
  (required off loopback), request, queue and body limits, and validated checkpoints and requests.
• Models: Qwen3.8-27B (4-bit, 8-bit, bf16) with the incoai DFlash2 drafter, and Qwen3.8-Flash-Next (oQ4e). Smoke
  ports: Qwen3.6-35B-A3B, Qwen3.5-9B, Muse Glimmer 30B and Gemma-4-31B.

Measured

Same weights, drafter, machine and harness for every engine. Full tables are in
docs/RESULTS.md.

• Mac Studio M5 Ultra:
  • 32 agents on 16k contexts: 615–620 tok/s in total (620 on this release's tarball). Splash 1.5 levels off at about 369.
  • 32 agents on 64k contexts: the last agent starts in 3.8 s, against 48.9 s.
  • 8 Qwen3.8-27B agents released together: all finish in 125.6 s, against 177.6 s on TensorFold 1.2.2 (177.9 s on 1.2.1).
  • Agents coming back after unrelated work: 32 forked agents relaunch in 1.03 s on the Studio; on the M5 Pro relaunch 2.1 s and resume 0.25 s, against 195 s and 15.4 s on TensorFold 1.2.2 with its disk cache.
  • 16 agents arriving at random: all finish in 78.7 s, against 116.7 s.
• Mac minis:
  • M5 Pro, 16 agents at 16k: 14.3 turns per minute, against 12.0 on Splash 1.5.
  • M6 (32 GB), measured 4 November: Qwen3.8-27B agent fleets on 8k contexts, 16 agents: 9.7 turns per minute against
    8.8 on Splash 1.5 (14.3 with the experimental Neural Engine split). With 8–16 agents on 16k contexts Splash 1.5 is
    faster (86–89 vs 64–66 tok/s): the 32 GB KV pool holds about four 16k contexts. TensorFold 1.2.2 cannot load the
    27B on 32 GB. (This line replaced "25% ahead of Splash 1.5 at 16 agents", which the 4 November runs contradict.)
• Where others lead:
  • Single-stream decode, and one agent on short contexts: Splash, TensorFold and mlx-serve are faster.
  • Flash-Next with 1–5 agents starting together: TensorFold finishes sooner; with Flash-Next agents arriving at random the two are level on total time and TensorFold's slowest first tokens come sooner.
• Quality:
  • Logits match mlx_lm (top-1).
  • HLE Lite: 27B 4-bit 32.4%, 8-bit 34.1%, Flash-Next 41.6%.
  • Strict JSON: 100/100 replies valid on the first attempt.

Requirements

• System: Apple Silicon on macOS 27. The prebuilt binaries need macOS 27.
• GPUs: M5-class and newer GPUs are the main target. M3/M4 run on their own kernel path and are lightly tested.
  M1/M2 are untested.
• Software: Python 3.12 for the front end (engine/requirements.txt). Building from source needs Xcode 27's
  Metal toolchain.

Prebuilt binaries

inferd-0.2.0-macos27-arm64.tar.gz contains build/ (the inferd server, tools and core.metallib) for Apple
Silicon on macOS 27.

1. Check it with shasum -a 256 -c inferd-0.2.0-macos27-arm64.tar.gz.sha256.
2. Unpack it inside engine/ of a checkout of this tag.
3. Run ./run.sh without building.

These binaries were tested on a Mac mini M4 Pro (logits, speculative equals plain, and the examples suite).

Acknowledgements

inferd stands on the shoulders of these projects:

• Apple's MLX: inferd vendors its quantized-matmul, GEMM and attention kernels
  (MIT).
• mlx-lm: the reference every logit is checked against.
• mlx-vlm (Prince Canuma): the vision path.
• Qwen's open weights.
• DFlash (z-lab) and the
  incoai DFlash2 drafter.
• TensorFold: the fastest single stream on a Mac, and the mlx.fast competition.
• Splash: it showed what a Mac can do for agents.
• oMLX and mlx-serve.
• MiaAI-Lab: their benchmark recipes.
• vLLM: its serving design.
• Hugging Face.

Licence: Apache-2.0 (see NOTICE).

Tested on this tarball (4 November): Mac Studio M5 Ultra, Mac mini M5 Pro, M6 and M4 Pro; results in docs/RESULTS.md (n).

Downloads

macOS	File	Status
27	inferd-0.2.0-macos27-arm64.tar.gz (+ .sha256)	tested on Mac Studio M5 Ultra, Mac mini M6 / M5 Pro / M4 Pro
26	inferd-0.2.0-macos26-arm64.tar.gz (+ .sha256)	beta, added 4 November: built from the same tag for a macOS 26 target. Run tools/release_gate.sh once. Do not use it on macOS 27 (wrong logits at 256+ row prefill chunks there)