bloomery 0.3.11

bloomery 0.3.11 serves Qwen3.8-27B from NVIDIA's NVFP4 checkpoint folder on one 24 GB card, and lets an agent start
the server by model name with machine-readable output and named exit codes. The card upload now reads ahead of the
copy, so a DeepSeek-V4.1-Flash cold start uploads its weights 30.6 % faster. On a Mac, Metal decodes LFM3-1.7B about
1.5× faster than 0.3.10. All CUDA numbers: the RTX 3090 alone (at its 250 W cap), or RTX A6000 + RTX 3090,
bloomery-serve from this release's first sm86 tarball (f1c391e; the re-tag below changed only how a headless
qwen35 file starts) at the serving defaults (--place unset, every lever unset,
--ctx-size unset), tools/ref/depth-serve.sh, one round (round 2 did not run, below), a fresh server a row. pp is the prompt rate at P
ids; decode is tg tok/s over 128 generated tokens after that prompt.

RTX 3090 alone (250 W cap)	ctx	pp512	pp4096	decode @ 512	decode @ 4096
Qwen3.6-35B-A3B (qwen35moe)	262,144	2476.43	3024.23	157.70	152.53
Qwen3.8-Flash-Next	262,144	575.75	(1378.01)	77.74	(60.97)
GLM-5.3-Flash	16,384	161.73	195.41	28.29	24.40

RTX A6000 + RTX 3090, bp	ctx	pp512	pp4096	decode @ 512	decode @ 4096
MiniMax-M3.1-Flash NVFP4 folder	261,120	109.08	112.16	14.63	14.26
DeepSeek-V4.1-Flash	1,048,576	(104.40)	(308.97)	(41.20)	(35.86)

tok/s, one row each, at the context the unset rule picked (the ctx column). A value in parentheses is out of a mean:
the V4.1-Flash rows read the file cold (27,147 and 56,701 major faults), and the Qwen3.8-Flash-Next 4096 row ran while
the previous server was still exiting. These rows are not comparable with 0.3.10's, which ran at --ctx-size 4300:
MiniMax-M3.1-Flash at bp decodes 14.6 tok/s at its unset context of 261,120 against 21.1 at 4300 in 0.3.10, because
the larger cache leaves one routed expert on the cards (below).

Qwen3.8-27B, RTX 3090 alone, no flags: bloomery-serve -m <NVFP4 folder> loads in 5.2 s at a context of 36,864,
and -m Qwen3.8-27B-Q4_K_M.gguf in 6.2 s at 103,424. Both pass the release checks (a short prompt, a 5,850-token
prompt, and the short one replayed with the same text). They have no timed row in this release.

Highlights from our contributors

• @juliankang4 (37 PRs since 0.3.10):
  • Serving: /v1/responses keeps its responses (store, previous_response_id, background) in a bounded store
    (#968); structured output takes json_object, arrays, anyOf, $ref, integers and numbers, and samples inside
    the mask (#973); n choices on /v1/chat/completions and /v1/completions (#959); logit_bias read on plain
    steps and MTP passes alike (#966); a sampled request drafts beside other busy slots (#964); d opens a live view
    of the slots, placement and requests (#939); one settings page for every seat's flags (#949).
  • --hf reads a repo's architecture before the fetch and picks the seats that check the arguments (#958), and takes
    a revision, <repo>@<rev>[:<quant>] (#963). A seat's flags and levers are checked before the model is fetched
    (#912).
  • Qwen3.8: a q8_0 KV cache, the default on a checkpoint folder (#956); a selecting layer keeps its raw indexer keys
    in a ring of eleven rows, 768 MiB less card memory per slot at a 262,144 context with identical output (#969); the
    hostless card sum fused with the shared add (#974); parts loads that keep the drive reading (#913, #919);
    -ub/--ubatch-size (#881); the verify path's E4M3 and MXFP8 staging (#877, #884, #920).
  • An unset context takes the largest step the model's room holds (#906).
  • DGX Spark: a GB10 installs a linux-aarch64-cuda-sm121a archive or the arm64 image (#878, #882, #883, #892). @juliankang4
    built this release's GB10 archive on a Spark.

Install

Linux x86-64 with AVX2 (glibc 2.34+), an NVIDIA GPU of compute capability 8.6+ and driver R580 or newer (CUDA 13).
On Windows, run inside WSL2. Docker and Homebrew as in the README. An agent asks what fits with bloomery-serve --recommend --json
(docs/agents.md).

curl -fsSL https://raw.githubusercontent.com/midagedev/bloomery/main/tools/release/install.sh | sh
~/.local/bin/bloomery-serve --hf unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M

RTX 50 series: this is the first release with bloomery-0.3.11-linux-x86_64-cuda-sm120a.tar.gz. It carries
sm_120a PTX (native FP4 and MXFP8 kernels); the driver compiles it on the card's first start (#931). Our cards are
sm_86, so this archive is checked by its build and its gates' scans, not by a run on an RTX 50 card.

Apple Silicon: bloomery-0.3.11-macos-arm64-metal.tar.gz, signed with a Developer ID and notarized. Metal opens
MLX folders as downloaded (#880).

New

• Qwen3.8-27B from its NVFP4 checkpoint folder (#960, #941): bloomery-serve -m <folder> serves ModelOpt's dense
  NVFP4 folder with no conversion, on one 24 GB card.
• Agents start the server (#961): --model NAME picks a catalogue model, --recommend and --dry-run print the
  pick, --json and NDJSON events report the start, and every refusal ends with a named exit code
  (docs/agents.md).
• Nemotron-3.5 NVFP4 checkpoint folders (#921, #928, #933, #940): ModelOpt's Lightning-30B-A3B folder opens
  placed. Its gates ran; this release's sitting did not serve it.
• Metal: LFM3-1.7B from its MLX folder, verified against mlx-vlm, text only (#943, #932); dense Qwen3.8-27B MLX,
  verified on its 4-layer cut against mlx-lm within 0.01 (#918); Qwen3-Coder-30B-A3B MLX with XML tool calls,
  verified on the 30B-A3B fixture split only (#886, #887).
• An x86-64 sm_120a archive for RTX 50 cards (#931), above.

Changed

• Faster model load (#971): the card upload reads the next segments ahead of the copy, and block formats split
  into their planes straight into the upload slot. DeepSeek-V4.1-Flash, plan (a), RTX A6000, cold page cache, 3
  rounds: upload 13.42 → 9.31 s (−30.6 % ± 1.4), whole load 48.90 → 43.07 s (−11.9 % ± 8.7); a warm restart's upload
  10.88 → 8.09 s (−25.7 % ± 6.8).
• Metal is faster in the common owners (#972). LFM3-1.7B MLX 4-bit, M1 Pro 32 GB, interleaved sittings: decode
  92.6 → ~140 tok/s; pp512 1059 → 1225 and pp4096 957 → 1138 tok/s. Model open 0.46 → 0.17 s.
• Code quality: placement.rs split into seven modules (#896); Google's Rust style guide as R30–R42 (#910).

Fixed

• The qwen3 seat holds its stage card before it reads the room, so its plan is the CLI's split (#952).
• A unified card's refusal names the host room, and a host-room refusal names the reading that gave it (#944, #885).
• An NVFP4 matrix read whole bounds its read by what the read holds (#923).
• The qwen4exp MTP load refuses a self-contained draft by its own matrices before reading its layer (#962).
• Metal: a stream with a residency set drains its queue before the set is torn down (#916).
• Every seat binary answers --records-schema again (it refused the flag after #961).
• A dense qwen35 file or checkpoint folder with no head (Qwen3.8-27B) started with no --model was refused as a
  Clef backbone; the generative seat now serves it and says on stderr what a decision model needs (--head, or the
  --hf repo that names its head). A qwen35 file no seat serves is still refused by name.

Off on this release

• MiniMax-M3.1-Flash NVFP4 at bp with --ctx-size unset: the context takes 261,120 positions and leaves one routed
  expert on the cards (residency off, no room), so decode is 14.6 tok/s.
• A greedy short request replayed after a long one answered with a few words changed on DeepSeek-V4.1-Flash at bp
  ("gentle" became "light") and on GLM-5.3-Flash at a. Not yet diagnosed; the other models replayed the same text.
• No timed rows this release for DeepSeek-V4.1-Flash on the RTX 3090 alone, GLM-5.3-Flash at bp, and
  Nanbeige4.3-33B: the sitting's runner started those cells while the previous server still held the port or the
  card, or could not read the seat's plan record. The servers themselves loaded; 0.3.10's rows stand for them.
• Metal: GGUF on Metal runs unverified; LFM3 is text only.
• Older models stay supported, but this release was verified on each family's newest model only.

Checked before the tag

• 0.3.11 was first tagged at f1c391e and re-tagged the same morning at 580de63. The difference: a dense qwen35
  file or folder with no head now starts with no --model (Fixed, above). Before the re-tag, this archive's binary
  passed the release checks with no flags on the Qwen3.8-27B folder, the Qwen3.8-27B Q4_K_M GGUF and Clef's backbone
  GGUF. Five serve gates the change touches ran green on the fixture tier; the rest of the checks below ran on
  f1c391e.
• The release gate batch: fixture tier 260 gates green. Real tier 96 items; its reds were stale gates or runs that
  needed a variable: a schema, a scanner, a stage-graph structure clause, a fixture budget, a spill pin, the q8_0
  cache clause of gate-gpu-qwen3-serve and a refusal clause of the MiMo serve gate (each re-derived and fixed), two
  Flint gates re-run with the complete Flint file named, and gate-gpu-p2, which lost its ssh connection after a
  pass. All green on re-run; the last five ran just after the tag on its tree plus the gate fixes, which move no
  shipped binary.
• The release sitting on this tarball's binary (sha256 3bb2f660…): cells 1 and 8 green, the rest red on the runner
  issues named above; the rows it gave are in the tables.
• The Qwen3.8-27B NVFP4 folder on the RTX 3090 alone with this binary: the release checks above, all PASS.
• The macOS archive: signed binaries, build-mac.sh --check, and the e2e ids on the qwen35moe MLX split.

Thanks

To @juliankang4 for the serving API work, Qwen3.8's memory and load work, the DGX Spark install, and the MLX-first
direction on the Mac. No issue was filed by a person other than the maintainer since 0.3.10.

Written with Claude Code (Claude Opus 5.5); reviewed by the maintainer.