Skip to content

inferd 0.4.0

Choose a tag to compare

View all tags
@johntdavies johntdavies released this 05 Nov 11:21
· 17 commits to main since this release
v0.4.0
eb32329

inferd 0.4.0: a native inference engine for Apple Silicon (C++, Objective-C++ and Metal), built for many AI agents
at once on one Mac. It is experimental software.

64 agents at once on one Mac Studio: 849 tok/s on Qwen3.8-27B and 1,289–1,323 tok/s on Qwen3.8-Flash-Next, each
agent on its own 16k context. On a Mac mini, an agent fleet on the 27B completes 13–19 turns a minute (M6 32 GB, M5 Pro
64 GB), and the 35B MoE model 81–93.

Getting started in four steps

  1. Download the file for your macOS below and its .sha256, check it (shasum -a 256 -c …) and unpack it.
  2. ./install.sh --model 27b: a Python 3.12 venv for the front end, plus Qwen3.8-27B 4-bit and its DFlash2 drafter. If
    the model is already in ~/models, the Hugging Face cache or LM Studio, it is used where it is (no download).
  3. ./run.sh start --model 27b: the server on port 8300, sized from your Mac's memory.
  4. Call it with any OpenAI (or vLLM) client at http://localhost:8300/v1.
  • macOS 27: inferd-0.4.0-macos27-arm64.tar.gz.
  • macOS 26 (beta): inferd-0.4.0-macos26-arm64.tar.gz, built natively on macOS 26 with Xcode 26.5 from the same
    commit. It has not yet run on a macOS 26 Mac's GPU: run the release gate once
    (PY=~/.venvs/inferd/bin/python tools/release_gate.sh ~/models/Qwen3.8-27B-4bit ~/models/Qwen3.8-27B-DFlash2). Never
    run a macOS 26 build on macOS 27 (run.sh refuses it). Building from source (git checkout v0.4.0 && cd engine && ./build.sh) remains an option.

What's new since 0.3.0

  • vLLM-compatible API.
    • New endpoints: /v1/completions, /tokenize, /detokenize, /version, /ping and a Prometheus /metrics.
    • New request fields:
      • logprobs, from the verify rows so speculation stays on;
      • presence, frequency and repetition penalties, min_p, logit_bias and min_tokens;
      • n;
      • guided JSON, choice and regex, and structured_outputs;
      • every tool_choice mode.
    • Unsupported endpoints (embeddings, rerank, audio, LoRA) answer a clean 501.
    • Of 78 probe items, 0 are broken (19 before). Details: docs/API-COMPAT.md.
  • Qwen3.6-35B-A3B:
    • int8 experts on the QA8 path, the router as a matmul from 64 rows, and drafter attention on simdgroup matrices: the
      M5 Pro fleet goes from 60–71 to 81–93 turns/min;
    • decode snapshots turn off when a chat template drops earlier reasoning: on the Studio, 16k x 32 agents goes from 269
      to 1,079 tok/s.
  • M4 / M4 Pro: the Neural Engine FFN split is on by default (26 units). Prefill is about 1.5x faster (8k prompt 64.2
    → 42.8 s) and decode is level. ANE_UNITS=0 turns it off.
  • Models you already have: install.sh and run.sh start --model look in ~/models, the Hugging Face cache and LM
    Studio, then use the first complete copy. Known equivalent builds are accepted, with a note, when their config matches;
    for example Qwen's own MLX 4-bit, or z-lab's byte-identical drafter. --exact takes only the exact repo ids.
  • Sizing for many agents (run.sh start):
    • the Mac Studio gets 192 snapshots: the 27B fleet with 64 in flight goes from 59.6 to 95.3 turns/min;
    • Flash-Next gets 64 running and 128 snapshots: 64 agents at 16k go from 209–287 to 1,289–1,323 tok/s.
  • Release: a native macOS 26 build. package.sh checks that every file the release gate needs is in the tarball.
  • Fixes: the M4 Max report findings, run.sh's CI tests, and more. See docs/CHANGES-*.

Measured (5 November, full tables in docs/RESULTS.md (t)–(y))

Mac mini

  • M6, 32 GB:
    • 27B agent fleet on 8k, 13.3–14.0 turns/min;
    • 9B, 16 agents on 16k, 164 tok/s (8-bit 117);
    • agents launched from a saved context in 2.3 s, resumed in 0.4 s.
  • M5 Pro, 64 GB:
    • 27B fleets 18.2–19.0 (16k) and 14.4–15.7 (32k) turns/min;
    • the 35B fleet 81–93;
    • 35B, 32 agents, 610 tok/s at 4k and 467 at 16k;
    • agents resumed in 0.14–0.25 s.
  • M4 Pro, 64 GB: 27B, 8k prompt, first token 42.8 s (0.3.0: 64.2 s).

Mac Studio M5 Ultra

  • 27B:
    • 64 agents on 16k, 849 tok/s (32 agents: 722);
    • the agent fleet with 64 in flight, 95.3 turns/min;
    • 32 agents on 64k, the last agent's first token in 3.3–3.4 s.
  • Flash-Next, 64 agents, 1,289–1,323 tok/s.
  • 35B, 32 agents on 16k, 1,079 tok/s.
  • One agent saves a 33.5k-token context and 31 start from it: median first token 0.72 s.

Compared with other engines, same weights, machine and harness, run back to back:

  • Studio 27B fleet: 80–95 vs Splash 1.5's 48–50 turns/min.
  • M5 Pro:
    • 27B fleets ahead of Splash 1.5 by 41–60%;
    • 35B fleet 81–93 vs 78–79;
    • 9B fleets 7.5x (4-bit) and 15.6x (8-bit) TensorFold 1.2.2 at 32 agents.
  • M6:
    • 27B fleet 13.3 vs Splash's 8.8 turns/min;
    • TensorFold 1.2.2 cannot load the 27B on 32 GB.

Where others lead

  • One agent on its own: Splash, TensorFold and mlx-serve decode a single stream faster.
  • Splash is faster with 4–8 agents taking turns on the minis, and for the 35B on the 32 GB M6 at 16k.
  • TensorFold finishes Flash-Next sooner with 1–5 agents starting together.

Checked for this release on a MacBook Pro M5 Max (macOS 27), from a fresh unpack of the macOS 27 tarball: install.sh
and the full release gate. The performance numbers were measured on main builds whose engine code matches this release
for those paths; the A/Bs are in RESULTS.md.

Assets 6
Loading