Ishizuki v0.6.1 — faster drafted edits, and a kernel precision fix

Faster drafted code edits, and a fix for mixed-precision verify kernels

Faster drafting on short drafts

• Served code edits are about 16% faster with drafting on, which is the default. A code-edit turn on Ternary Bonsai 2 27B went from 23.1–23.5 to 26.7–27.2 tok/s on an M1 Max. The output is identical token for token, and the gain held under greedy, the app's sampler and plain temperature.
• A short draft no longer pays for a full one. Lookup drafting often proposes fewer than seven tokens. Rows past the end of a short block used to go through slow branches, so a 5–7 row verify of the output head took 7.9–8.7 ms. It now takes 3.4 ms, the cost of a full block of 8.
• The verify kernel now starts at the row count where it's faster, for each matrix. The kernel costs about the same at any row count, so it pays sooner on big matrices: from 2 rows on the output head, 3 on the MLP, 4 on the 6144-wide projections, and 6 on attention k and v. Before, every 5–8 row verify used it, including on matrices where MLX's own kernel is faster.
• No kernel compile in the middle of a reply. Each weight shape's kernel now compiles once when the model loads, instead of the first time drafting reaches a new row count.

DFlash drafting and plain decoding run at the same speed as in 0.6.0 (within 1%).

Fixes

• Verify kernels no longer mix up precisions. MLX caches a custom kernel by name, and the name didn't include the input's precision. So when an fp16 model and its fp32 DFlash drafter shared a matrix shape, one of them ran the kernel compiled for the other. With both in the same step that could crash the GPU with a page fault, and on its own it could give wrong numbers. Each kernel's name now includes its input precision, and the same fix covers the EXL3 rotation and DeepSeek's Sinkhorn mixer.
• bf16 activations work in the verify kernel.

For developers

• FewRowRaceProbe times MLX's matmul, QMVWide and the verify kernel at 2–8 rows on any shapes (ISHIZUKI_RACE=1, ISHIZUKI_RACE_SHAPES=5120x17408,..., ISHIZUKI_RACE_BITS, ISHIZUKI_RACE_F32). Time it in release (swift test -c release -Xswiftc -enable-testing): debug adds about 0.15 ms of host time to every call and makes a small kernel look slow.

These kernel changes came out of the MLX Fast Bonsai 2 contest, where the same kernel took first place for a while at 1.655×.

Full Changelog: v0.6.0...v0.6.1

Ishizuki v0.6.0 — DeepSeek-V4.1, DFlash drafting, OrcaSAQ2

DeepSeek-V4.1 from its release shards, DFlash drafting, and OrcaSAQ2

DFlash 2 drafting doubles code and math

• Qwen3.8 packs now draft in blocks with z-lab's DFlash 2 drafter. The target model verifies the whole block through a new matrix-fragment kernel, so checking eight rows costs about 2.6 plain steps instead of more than five on MLX's own path.
• Greedy and sampled output both work. Sampled drafting keeps the same distribution as plain sampling.

Measured on an idle M1 Max, greedy, 256 tokens:

Pack	Plain	DFlash (math / code)
Ternary Bonsai 2 27B, 2-bit	22.5 tok/s	45.0 / 45.0 tok/s
Qwen3.8-27B EXL3 2.0 bpw	9.2 tok/s	28.7 / 35.4 tok/s
OrcaSAQ2 27B	8.3 tok/s	28.9 / 33.3 tok/s
Qwen3.8-27B IQ2_XS GGUF	10.6 tok/s	23.9 / 24.7 tok/s

On the same machine and the same Bonsai pack, mlx-lm decodes at 20.9 tok/s and llama.cpp runs the GGUF at 12.8 tok/s. Prose gains little from drafting, because the drafter guesses prose far less often. BENCHMARKS.md has the full side-by-side table.

DeepSeek-V4.1-Flash runs from its release shards

• The 48 shards load as they ship, with no conversion step. The fp8 matrices run as MLX mxfp8 and the fp4 experts as mxfp4, byte for byte. The experts stream from disk into as many slots as the machine holds, and the two 100 GB n-gram tables are read a row at a time.
• Every part of the architecture was written against DeepSeek's reference code and checked against it, on a small fixture and on real layers.
• Streamed reads no longer fill the file cache. macOS only skips the cache when a read is aligned to its 16 KB pages, which safetensors offsets almost never are. Reads now go through page-aligned scratch memory, so a long prompt no longer squeezes other apps into compressed memory. Cold decode improved from 0.50 to 0.66 tok/s.
• Off the internal SSD on an M1 Max: 14.2 tok/s prefill and 1.29 tok/s decode, warm.

Load more of what's out there

• OrcaSAQ2 packs load. They are EXL3 with an int8 embedding that has one scale per row, and they answer correctly.

Streamed experts, faster

• Expert slots are now locked in memory. Before, macOS compressed idle slots, and a later hit waited on decompression. On the 125B-A6B the step after a 4k prefill went from 4.55 s to 1.71 s.
• Prefill reads each expert once per chunk, not once per token. 24.6 → 37.5 tok/s on 1k tokens, and 65.8 tok/s on 4k tokens in 2k-token chunks.
• Slot count defaults to Automatic, sized to what fits on the machine.
• Politeness now defaults to Normal. Adaptive, which backs off for heat, cut prefill chunks to a quarter of their size; it's still one setting away.

The app

• A new look: parchment and paper cards, with warm charcoal after dark, moss as the accent colour and New York as the typeface. All text is one size larger.
• The agent can be steered mid-turn, and can stop to ask you a question, with its options shown as buttons above the composer.
• A conversation keeps the model it started with. Opening a conversation preloads it into the cache so its next turn starts sooner.
• Apple Intelligence is hidden when its model files are missing.

For developers

• EngineBenchProbe benches any pack plain, with MTP and with DFlash (ISHIZUKI_BENCH_PACK; set ISHIZUKI_BENCH_SHOW to print what the pack said).
• DeepSeekReleaseProbe and DeepSeekStepProbe time DeepSeek-V4.1 off a drive. They run only when ISHIZUKI_DEEPSEEK is set.
• DFlashProbe compares a drafted run against a saved plain run.

Full Changelog: v0.4.0...v0.6.0

Ishizuki v0.4.0 — EXL3, JANG, and drafting on by default

New pack formats, drafting on by default, and Qwen3.8 Flash Next at full size

Load more of what's out there

• EXL3 packs load natively. ExLlamaV3's trellis-coded weights are decoded inside the matvec and never expanded. All three codebooks work, at whole rates 1–8 and half rates 1.5/2.5/3.5. turboderp's Huihui-Qwen3.8-27B-abliterated 4.00 bpw pack runs at 9 tok/s plain and 14 with MTP drafting on an M1 Max.
• JANG repacks load. Their Hadamard rotation and zero-centred norms are read from the pack's config, so they no longer produce fluent nonsense. dealignai's Bonsai 2 27B CRACK (ternary) is in the curated list, and runs a greedy chat at 20.5 tok/s.
• The GSQ-RCO GGUF in the curated list now brings its vision projector, so it can see images.

Drafting is on by default

• A served chat turn now drafts by prompt lookup inside the pipelined loop, so a step with nothing to draft costs exactly what a plain step does. Sampled output keeps the same distribution as plain sampling.
• Greedy requests on a pack with an MTP head use the head. A rejected draft no longer costs a replay: the kept tokens ride into the next block.
• GGUF files verify a drafted block through matrix fragments, so eight rows cost 0.9–1.8× one row across all eleven formats instead of 3–4×.

Measured on the Qwen3.8-27B IQ2_XS GGUF:

Workload	Plain	Drafted
Code edit, temp 0.7	11.1 tok/s	13.7 tok/s
Prose, temp 0.7	11.1 tok/s	11.4 tok/s
Greedy prose (MTP)	11.2 tok/s	13.2 tok/s
Greedy code edit (MTP)	—	15.0 tok/s

Greedy drafted output matches plain decoding token for token.

Qwen3.8 Flash Next (qwen4_exp)

• +15% decode on Whittle 35B-A3B (25.6 → 29.4 tok/s, same tokens). The hyper-connection mixers were writing a float32 copy of their weights on every call, which was half the decode step. They are now converted once at load.
• Upstream checkpoints convert, including the full 125B-A6B. The n-gram table is found where upstream nests it, and can be stored at 8 bits (58 GB instead of 102 GB for the 125B). The MTP head's experts stay resident when the trunk's experts stream.
• The expert cache now evicts the least recently used expert, and reads a token's misses in parallel. On recorded Whittle 35B-A3B routes, LRU catches 8–13 points more of the expert reads than the old fewest-uses policy with a third to half of the experts held in memory. Holding half of them catches about 95%.

For developers

• Qwen4ExpStepProbe splits a decode step into its components on real weights.
• RouteProbe and Scripts/simulate_expert_cache.py record expert routes and replay them against caches of different sizes and policies.
• ConvertProbe runs a conversion headlessly, for checkpoints too large to hand to the app.

Full Changelog: v0.3.0...v0.4.0

Ishizuki v0.3.0 — Major speedups

Speedups over Llama.cpp

Currently, we're seeing speedups on the same models, tested with
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

Objectively faster by ~2x

Llama.cpp:

pp512 |        131.36 ± 0.18
tg128 |         11.65 ± 2.73

Ishizuki:

pp512 | 272 T/s
tg128 | 28 T/s

Try it with Qwen Whittle and Qwen3.8-Next!

Includes support for next-gen Qwen architectures including engrams on disk and MoE expert streaming

Full Changelog: v0.2.1...v0.3.0

ishizuki v0.2.1

Native macOS inference engine for Ternary Bonsai 2 27B on MLX Swift.

Install

Installer — ishizuki-v0.2.1.pkg, signed and notarized, installs to ~/.local, no admin
password.

Loose binary — ishizuki-v0.2.1-macos-arm64.tar.gz: the executable and the metallib it
loads, together.

Already on v0.2.0? ishizuki update takes it from here.

Since v0.2.0

• ishizuki chat is now ishizuki demo. Same verb, named for what it is for — the one
  command that shows the engine off. just demo runs it from a checkout.

• The reply renders as markdown while it streams. A line is classified from its first few
  characters, so a bullet or a quote gutter is painted before the rest of the line arrives, and
  inline spans — bold, italic, code, links — are held back only until their closing delimiter
  lands. Headings and horizontal rules get their own weight.

  A fence opens a gutter box titled with its language, and each line inside is tokenized as it
  completes: keywords, strings, numbers and comments, with per-language comment markers and
  keyword sets for swift, python, rust, go, javascript, shell and json. A terminal with no
  colour passes the text through untouched rather than eating the markers.

• --reasoning none no longer throws. It asked the chat template for an effort it does not
  support — the template takes xhigh, medium and low — and the request failed before the first
  turn. It now means what it reads as, thinking off, in the flag and in /think alike.

ishizuki v0.2.0

Native macOS inference engine for Ternary Bonsai 2 27B on MLX Swift.

Install

Installer — ishizuki-v0.2.0.pkg, signed and notarized, installs to ~/.local, no admin
password.

Loose binary — ishizuki-v0.2.0-macos-arm64.tar.gz: the executable and the metallib it
loads, together.

Already on v0.1.7? ishizuki update takes it from here.

Since v0.1.7

• ishizuki chat — the engine in one verb. Loads a pack and talks, with the thinking block
  in a muted inset box, prefill and decode rates under every reply, and the session cache kept
  warm so each turn only prefills what changed. /think changes reasoning effort mid-thread;
  /new, /system and /stats do what they say. The header grows a procedural bonsai over its
  stone, sized to the terminal.

• ishizuki update — self-update in place. Fetches the release tarball for this machine's
  architecture and swaps the running executable and the metallib beside it, wherever ishizuki was
  installed from. Nothing is written until the download's sha256 matches the digest GitHub
  publishes and the new executable satisfies a Developer ID requirement naming the team that
  signed the binary it replaces. Each file is renamed aside and the new one renamed in, so a
  running server keeps the inode it already opened and a half-finished swap rolls back.

• Memory is planned from demand, not from the machine. A cold server used to claim a pool of
  up to 8 GB and six prefix caches sized for full 262K context — about 41 GB on a 64 GB Mac,
  before a single request arrived. It now starts at one 8K slot and a 0.5 GB pool and doubles a
  tier whenever the work runs into it, stopping at what the wired ceiling holds. Weights are
  measured from the pack instead of guessed at 9 GB, and the dashboard carries the live tier,
  the peak, and bars for memory and GPU utilization. The tradeoff is a slower warm-up.

• The older ternary packs load and run. Ternary-Bonsai 8B, 4B and 1.7B ship as plain Qwen3
  MLX packs — standard config, scale-only 2-bit affine weights, no Hadamard rotation. Point
  --repo at one and it runs, honouring the pack's own YaRN. A real Hadamard pack missing a
  module record still fails loud.

• Agents launched from ishizuki launch can paint. Foundation spawns children into their own
  process group, so pi, claude and hermes each stopped on SIGTTIN/SIGTTOU at the first raw-mode
  tty access and never drew. The foreground process group is handed over for the life of the
  tool and taken back afterwards.

ishizuki v0.1.7

Native macOS inference engine for Ternary Bonsai 2 27B on MLX Swift.

Install

Installer — ishizuki-v0.1.7.pkg, signed and notarized, installs to ~/.local, no admin
password.

Loose binary — ishizuki-v0.1.7-macos-arm64.tar.gz: the executable and the metallib it
loads, together.

Since v0.1.6

MLX failures no longer take the server down. array::shape(int) is bounds-checked through
std::vector::at, so an out-of-range dimension anywhere inside MLX surfaces as
out_of_range("vector") during eval — which MLX Swift turns into fatalError, killing the
process for one request. Generation now runs inside withError, so that becomes a Swift throw;
the error box is composed into the existing cancellation hook, so the loop stops at the next
step instead of running on over a poisoned graph. A failure after the response is committed
closes the stream rather than writing a second status line, and an MLX fault reports 500 instead
of blaming the caller with a 400.

This contains the reported Fatal error: vector crash — a run that hits it now fails that
one request and keeps serving. It does not identify the out-of-range access itself, which
remains unreproduced: a 64871-token thinking request with the reporting flags, a quantized-KV
stress at 4096 context, and unit tests asserting the quantized cache's window/store invariants
across reallocations all completed clean.

ishizuki v0.1.6

Native macOS inference engine for Ternary Bonsai 2 27B on MLX Swift — 2-bit
Hadamard-rotated weights, vision, tool calling, OpenAI + Anthropic APIs. No Python.

Requires Apple Silicon and macOS 15+.

Install

Installer — ishizuki-v0.1.6.pkg, signed and notarized, installs to ~/.local, no admin
password. Uninstall with ~/.local/libexec/ishizuki/uninstall.sh.

Loose binary — ishizuki-v0.1.6-macos-arm64.tar.gz: the executable and the metallib it
loads. Put the two anywhere, together.

Since v0.1.5

• Stopping a request stops the work. Aborting from the client left the prefill running to
  completion — the worst place to be uninterruptible, since prefill is where the time goes and
  the tokens were already unwanted. Nothing in the server noticed a client going away: it
  stopped reading a connection the moment it had parsed the request, so a FIN or RST never
  registered.

  Each connection now carries a cancellation flag fed by a receive that stays pending for the
  life of the request, the connection state handler, and send errors on writes. Generation polls
  it before prefill, on every prefill chunk and on every decode step, so a stop lands within one
  chunk instead of one whole prompt. Requests are serialized, so one cancelled while queued is
  dropped before any work rather than run in full when its turn comes.

• An interrupted prefill stays reusable. A slot's token list is now reconciled against the
  cache offset instead of trusted, so a stopped prefill leaves a valid prefix rather than a slot
  whose cache holds less than it claims. Stop-and-resend resumes from where it got to instead of
  starting over. Cancelled requests are counted apart from failed ones on the dashboard.

  A client that half-closes while still waiting on its response now reads as a cancel. No
  mainstream HTTP client does that, and llama.cpp and vLLM read it the same way.

ishizuki v0.1.5

Native macOS inference engine for Ternary Bonsai 2 27B on MLX Swift — 2-bit
Hadamard-rotated weights, vision, tool calling, OpenAI + Anthropic APIs. No Python.

Requires Apple Silicon and macOS 15+.

Install

Installer — ishizuki-v0.1.5.pkg, signed and notarized, installs to ~/.local, no admin
password. Uninstall with ~/.local/libexec/ishizuki/uninstall.sh.

Loose binary — ishizuki-v0.1.5-macos-arm64.tar.gz: the executable and the metallib it
loads. Put the two anywhere, together.

Since v0.1.4

• Constrained decoding for response_format / json_schema. The schema is expanded into
  the set of documents it admits, held as a byte trie; each decode step samples only from the
  tokens that keep the output inside it. Structured replies are valid by construction rather
  than by validate-and-retry.

  {"verb":"descend","n":2,"slot":0}
  

  The schema's language must be finite — enums, bounded integer ranges, booleans,
  additionalProperties: false. Unbounded schemas are refused with a 400 rather than accepted
  and ignored, as is GBNF grammar, so a client that probes for a constraint mechanism falls
  through to the one that works instead of sticking with a silent no-op.

• chat_template_kwargs.enable_thinking is honored. llama.cpp and oMLX clients disable
  reasoning through that key rather than a top-level field, so a harness asking for no thinking
  was silently getting it. reasoning_effort: "none" works too. A request carrying a schema
  turns thinking off on its own, since a constrained document has no room for a reasoning block.

Known issue

A std::out_of_range inside MLX's graph eval (Fatal error: vector) has been reported during a
long generation with thinking enabled. It is not reproduced here yet and is not fixed.

ishizuki v0.1.4

Native macOS inference engine for Ternary Bonsai 2 27B on MLX Swift — 2-bit
Hadamard-rotated weights, vision, tool calling, OpenAI + Anthropic APIs. No Python.

Requires Apple Silicon and macOS 15+.

Install

Installer — ishizuki-v0.1.4.pkg, signed and notarized, installs to ~/.local, no admin
password. Uninstall with ~/.local/libexec/ishizuki/uninstall.sh.

Loose binary — ishizuki-v0.1.4-macos-arm64.tar.gz: the executable and the metallib it
loads. Put the two anywhere, together.

tar -xzf ishizuki-v0.1.4-macos-arm64.tar.gz
./ishizuki-v0.1.4-macos-arm64/ishizuki serve

Since v0.1.3

• Prefix caches survive the idle timer. --idle-timeout was dropping every cached prefix
  after 120s, so a pause between turns cost a full re-prefill — tens of thousands of tokens
  discarded to reclaim a buffer pool the residency timer had already freed on its own. Idle now
  releases only that pool; prefix caches are dropped when the model is unloaded
  (--evict-timeout).
• The budget carves out a full context first. Weights and one full context of KV are
  reserved before the buffer pool gets anything, and prefix slots scale with what is left.
  serve prints the reserve and warns when the wired ceiling cannot hold weights plus one full
  context.

Defaults on a 64 GB machine, 3.5-bit KV:

budget     ceiling 54.0 GB, context reserve 4.0 GB, buffer cache 8.0 GB, 6 prefix slots

fp16 KV reserves 16.5 GB and drops to 2 slots; --context-scale 2 reserves 8.0 GB.

Known issue

A std::out_of_range inside MLX's graph eval (Fatal error: vector) has been reported during a
long generation with thinking enabled. It is not reproduced here yet and is not fixed.