bloomery 0.3.10

bloomery 0.3.10 opens the hard-quantized GGUFs people actually download. MiniMax-M3.1-Flash now serves from Unsloth's
2-bit GGUF as well as from its NVFP4 folder; GLM-5.3-Flash, Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash open their
low-bit sets with no re-encode; Nanbeige4.3-33B runs whole on one 24 GB card. All numbers: RTX A6000 + RTX 3090, or the RTX 3090 alone (the
3090 at its 250 W cap), bloomery-serve at the serving defaults (--place unset, every lever unset,
--ctx-size 4300), tools/ref/depth-serve.sh, 2 rounds, a fresh server each round. pp is the prompt rate at P
ids; decode is tg tok/s over 128 generated tokens after that prompt.

RTX A6000 + RTX 3090	pp512	pp4096	decode @ 512	decode @ 4096
MiniMax-M3.1-Flash UD-Q2_K_XL (GGUF)	160.16	165.92	28.63	29.41
MiniMax-M3.1-Flash NVFP4 folder	129.08	131.56	21.12	19.53

RTX 3090 alone (250 W cap)	pp512	pp4096	decode @ 512	decode @ 4096
MiniMax-M3.1-Flash UD-Q2_K_XL (GGUF), a	89.16	96.75	21.36	21.01
Nanbeige4.3-33B Q4_K_M	745.46	693.76	24.22	23.23

tok/s, means of 2 rounds (every row's sd at most 5.6 %, but the 3090-alone M3.1-Flash pp512 at 16.9 %: its first
round read the file cold, 34,323 major faults); the depth is the prompt length the decode follows. On the 3090 alone
the M3.1-Flash GGUF plans a with no flag: most routed experts run on the host.

Recorded run at these defaults: MiniMax-M3.1-Flash UD-Q2_K_XL.

Highlights from our contributors

• Qwen3.8 on a DGX Spark (@juliankang4, 21 PRs):
  • FP4-Local folders start in 78.0 s instead of 225.3 s (#787).
  • NVFP4 prompts are 14.1 % faster on tensor cores (#812).
  • Code decode with the MTP draft is 4.2 % faster (#808).
  • Smaller prefill and TTFT gains come from #817, #822, #831 and #847.
  • His Spark comparison of the two checkpoints (#565): FP4-Local decodes at 30.18 tok/s, NVFP4 at 22.56. The gap is
    the NVFP4 checkpoint's BF16 dense matrices, about 2.9 GB more read per token. Spark work now centres on FP4-Local.
• Threads on any CPU (@juliankang4, #824, #826): the default pool takes every allowed physical core before an SMT
  sibling, and a mask that splits the L3 groups unevenly still pins each allowed CPU once.
• Flint and the common tier legs (@Burntgogi, #781, #782, #783, #799): Flint's head-128 ring attention
  and hosted runtime, ordinary IQ4_XS shared sites, mixed-form expert scratch and the six-slot F32 tier handoff.

The DGX Spark numbers are the pull requests' own, measured on the contributor's machine.

Install

Linux x86-64 with AVX2 (glibc 2.34+), an NVIDIA GPU of compute capability 8.6+ and driver R580 or newer (CUDA 13).
On Windows, run inside WSL2. Docker and Homebrew as in the README.

curl -fsSL https://raw.githubusercontent.com/midagedev/bloomery/main/tools/release/install.sh | sh
bloomery-serve -m PATH/MiniMax-M3.1-Flash-UD-Q2_K_XL-00001-of-00004.gguf --port 8080

Apple Silicon: bloomery-0.3.10-macos-arm64-metal.tar.gz, signed and notarized. Qwen3.6-35B-A3B runs end to end
(Q4_K_M GGUF, or an MLX 4-bit folder through prepare). New on Metal:

• Qwen3-30B-A3B (Q4_K_M) runs, held to the reference sets by its e2e gate (#856).
• Qwen3.6 UD-Q4_K_XL runs (Q8_0, Q5_K and f32 sites, a shared expert of another type than its stacks) (#872).
• A file whose shapes the kernels take opens with no registered instance: the open renders the kernels' geometry and
  compiles them, and says no gate checked that shape (#874).

New

• MiniMax-M3.1-Flash from GGUF (#854).
  • Unsloth's UD-Q2_K_XL (143.9 GB) serves on two cards at the default placement (bp), the routed experts split
    between the cards and the host.
  • UD-IQ2_XXS and UD-IQ1_M plan the same way; this release's pass served UD-Q2_K_XL only.
  • A file without attention.sparse_layers attends sparsely on exactly the layers that carry their indexer tensors,
    the rule of the producer's open llama.cpp PR (ggml-org/llama.cpp#31192).
  • A shared expert with K-quant gate/up over a Q8_0 down (layer 59 in these files) runs on the card.
• Flint 40B-A2.5B (@Burntgogi, #781, #782, #783, #799): unsloth's UD-Q4_K_XL serves whole on one RTX A6000 at the
  serving defaults: chat, streaming, tool calls and --parallel. Head-128 ring attention and the hosted runtime are
  Burntgogi's. Its greedy text has no CPU or llama.cpp referee yet: the card path's gates and the serving checks hold it.
  Recorded run: Flint UD-Q4_K_XL.
• Nanbeige4.3-33B (#838): Q4_K_M loads whole on one 24 GB card. bloomery-serve starts it with no flags. A dense
  model's prompt ubatch is sized to the card, and -ub wins.
• Weight types publishers ship:
  • DeepSeek-V4.1-Flash: ISTA-DASLab's GSQ-RCO 2.00 bpw set opens with no re-encode (#818). IQ1_S, IQ1_M and IQ2_S
    run on the host and the card; IQ2_XS becomes a routed card type.
  • GLM-5.3-Flash: the mixer sites (attention, ssm, indexer, NextN) read K-quant, BF16 and F32 weights (#841). This
    opens Unsloth's low-bit GLM files.
  • GLM-5.3-Flash (#853): the prompt batch projects each mixer site through the GEMM of its weight type.
  • Qwen3.8 (#853): K-quant token embedding, a Q5_K head, and the dense sites by name. This opens the dense
    side of Unsloth's UD-IQ1_S, IQ1_M, Q2_K_XL and IQ3_XXS sets.
  • Nemotron-3.5-Lightning (#857): Q5_0 routed stacks run on the card, so UD-Q4_K_XL holds all 2,944
    experts on the card where the room allows.
• Saved prompt cache and shared prefix on the MiniMax-M3 and Muse Spark seats (#810, #813).
• Metal (#805–#850): build-time model instances, picked at open (#836). generate_metal --plan prints every open
  refusal from the header (#806). Qwen3-30B-A3B decode is a second instance (#849).

Changed

• Toolchain: cuda-oxide is NVIDIA/cuda-rust 9df9e7ce directly, on stable Rust 1.99.1, with no fork (#823). The PTX
  of 20 entries moved and is re-pinned on both card classes (#839, #845).

Fixed

• Nemotron-3.5-Lightning's two-card placement counted the next-token block as a layer (53 vs 52) (#843).
• The feature-combination check refused a valid multi-package call; the reference-set check demanded sets that only a
  GB10 machine holds (#835).
• MiniMax-M3.1-Flash GGUF under --place a: a request after the first failed once host experts streamed into the
  card's ring, whose pass ran the 32-value GEMM only. The ring pass now runs the grouped GEMM the file's i-quant stacks
  read (#876). On the RTX A6000, a short request then a 4K-token one: prompt 126.5 tok/s with the stream, 97.2 with
  BLOOMERY_XSTREAM=off; decode 25.6 and 25.2.

Off on this release

• Flint has no CPU or llama.cpp referee for its greedy text yet; the card path's gates and the serving checks hold it.
• Older models stay supported, but this release was verified on each family's newest model only: DeepSeek-V4-Flash's
  gates did not run (V4.1-Flash's did).

Checked before the tag

• The release gate batch: fixture tier 231 gates green; real tier 96, every red resolved (two stale gemma gates fixed,
  two Flint gates re-run with the model named, one rerun under load, one DeepSeek-V4 gate deferred as above).
• The fix after it (#876): its owning gates at the fixture tier, and the pass below on the rebuilt binary.
• The user-facing pass, this tarball's bloomery-serve at the serving defaults, --ctx-size 16384, a short and a
  4K-token prompt:

model, placement	decode, short	prompt, 4K	decode, 4K
MiniMax-M3.1-Flash UD-Q2_K_XL, bp	22.46	163.2	26.66
MiniMax-M3.1-Flash UD-Q2_K_XL, a	21.72	125.1	26.25
MiniMax-M3.1-Flash NVFP4 folder, bp	15.88	118.8	18.88
Flint UD-Q4_K_XL, a	85.68	801.8	75.99
Nanbeige4.3-33B Q4_K_M, a	29.58	1085.4	25.31

Thanks

To @Burntgogi for Flint and the common tier legs, and to @juliankang4 for the DGX Spark work and the GB10 scans. No issue was filed by a person since 0.3.9.

Written with Claude Code (Claude Opus 5.5); reviewed by the maintainer.