Skip to content

bloomery 0.3.1

Choose a tag to compare

View all tags
@midagedev midagedev released this 12 Oct 10:13
· 328 commits to main since this release
v0.3.1
f9ae952

bloomery 0.3.1 takes images on the GLM-5.3-Flash seat, moves GLM-5.3-Flash UD-IQ4_XS's IQ3_S experts to the card,
serves two requests in one pass by default on MiMo-V2.6-Flash and on Qwen3.8 under --place bp, and gives a paged
plan a larger default context.

Install

Linux x86-64 with AVX2 (glibc 2.34+), an NVIDIA GPU of compute capability 8.6+ and driver R580 or newer (CUDA 13).
On Windows, run inside WSL2.

curl -fsSL https://raw.githubusercontent.com/midagedev/bloomery/main/tools/release/install.sh | sh
bloomery-serve --hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL --port 8080

Docker and Homebrew as in the README.

Added

  • Image input on the GLM-5.3-Flash seat. Pass --mmproj with the GLM-5V projector, or start with --hf and the
    seat takes the repo's own projector. The same --hf rule now holds on the V4.1 and Qwen3.6 seats: with no
    --mmproj, the projector beside the model in the repo is fetched with it, and loads when the text model keeps its
    card experts and context with the encoder's reserve (else one line names the cost and the seat serves text).
  • GLM-5.3-Flash card experts for every quant type the card leg runs. UD-IQ4_XS's IQ3_S gate/up stacks now sit on
    the card like its other types. A prompt batch also runs GLM's K-quant dense blocks and shared experts through the
    common gated block. Checked on the release candidate: UD-IQ4_XS with no flags picks bp on an RTX A6000 48 GB,
    reads a 5,548-token prompt at 172 tok/s and decodes at 35.1 tok/s.
  • Two requests in one pass by default:
    • MiMo-V2.6-Flash serves two requests at once, in one pass of the body.
    • Qwen3.8 does the same under --place bp, which used to take the slots in turn.
    • When the default's second slot does not fit, MiMo, GLM-5.3-Flash and Qwen3 serve one slot and print a line saying
      so. They no longer refuse the load. A --parallel you set is never lowered.
  • A larger default context on a paged plan. It grows to what the card fits (28,881 positions on a 12 GB card).
    The per-slot checkpoint reserve grows with it and shrinks the NVMe arena: about 0.9 GB at 28,881 on a 12 GB card,
    about 2.8 % more NVMe reads [derived]. The ctx line does not print this yet; 0.3.2 counts and prints it. Set
    --ctx-size to keep the arena.
  • GGUF files with q4_0 or q4_1 tensors open. Those tensors are sized by ggml's table.
  • --version and /props name the build's commit (#50).

Changed

  • Qwen3-30B-A3B on a card it does not fit whole reads the prompt in wide units, not one row at a time.
    This holds also when every routed expert stays on the host (a Q8_0 or IQ4_XS file, whose experts the card does not
    run). Checked on the release candidate with the Q4_K_M file and no expert on the card: the wide and the
    per-row prompt runs give the same 32 tokens. Speed not re-measured here (0.3.0 read a 5,619-token prompt at 103 tok/s on an RTX 3060 12 GB).
  • Load-time trades are printed. Every capacity trade a load makes is now on stderr: the host room a plan was made
    at, a slot that fell back, and the context it took.

Fixed

  • A load on a small-RAM host is no longer refused at the defaults (#46). In 0.3.0 two paths led there, on any MoE
    model: a plan that pages experts from NVMe filled the RAM it read exactly, and a plan with no paging grew its
    context to the edge of that reading. RAM the server took after the reading then refused the load: 132 MB short on
    an RTX 3060 12 GB with 27.5 GB free (WSL2), and 162 MB short on an RTX 5090 with 64 GB. A plan now stays 512 MiB
    (a chosen margin) under the RAM it reads, and a paged load shrinks its NVMe arena when it finds less RAM than
    planned. Checked on the release candidate: Qwen3.8-Flash-Next UD-Q4_K_XL with no flags loads on the RTX 3060
    12 GB with 32 GB of RAM (one slot of 28,881 positions, 75 GB of experts paged from NVMe).
  • An expert stream's drop now waits for every copy staged behind it (#49).
  • Two tests no longer assume the host's thread layout (#52, reported by @juliankang4).
  • The key-check test compares /v1/models bodies with created pinned inside the list too. Two binds in different
    seconds no longer fail it.

Checked before the tag

The release candidate's bloomery-serve (commit 5ae948b; the tag f9ae952 adds only the GLM-5.3 image prompt fix,
and the image row below ran on the tag's own build), serving defaults (every lever unset, no --parallel,
--ctx-size 16384), greedy, thinking off, one load per row, RTX A6000 (+ RTX 3090 for bp), one lease each, no other
process on the box. Prompt sizes are the server's prompt tokens.

Model, place Long prompt: tokens, prefill tok/s, first token Decode after it Short prompt decode
Qwen3.8 UD-Q4_K_XL, bp 5,673, 1,707, 3.35 s 84.3 tok/s 94.0 tok/s
Qwen3.6-35B-A3B UD-Q4_K_XL, unset (picks a) 5,673, 7,083, 0.81 s 115.0 tok/s 117.5 tok/s
GLM-5.3-Flash UD-IQ4_XS, unset (picks bp) 5,548, 172, 32.3 s 35.1 tok/s 35.0 tok/s
DeepSeek-V4.1-Flash Q3_K_M, unset (picks a) 5,642, 403, 14.0 s 41.9 tok/s 42.7 tok/s
MiMo-V2.6-Flash MXFP4, unset (picks a) 5,620, 104, 53.9 s 34.8 tok/s 36.1 tok/s

GLM-5.3-Flash UD-IQ4_XS with --mmproj (the BF16 projector), --place a, on the tag's build (f9ae952): a text request,
a 448×448 image through /v1/chat/completions (280 prompt tokens) and through /v1/messages, a larger image
(7,928 prompt tokens), then text again: every request 200, and the server kept serving after the images.

Off cells

  • Qwen3.8-Flash-Next on a paged plan (experts paged from NVMe, as on a 12 GB card) serves one slot. A set
    --parallel of 2 or more fails at load; 0.3.2 fixes it. The MTP draft is off there, and a set BLOOMERY_DRAFT=mtp
    is refused by name.
  • GLM-5.3-Flash with --hf under bp (a card that does not hold the whole model) does not load the repo's
    projector when its card reserve would cost card experts: 3.1 GB, or 267 experts of UD-IQ4_XS on an A6000. The
    server prints that trade and answers an image request with 400; --mmproj FILE loads the projector.
  • Known issue: Qwen3.6-35B-A3B on a card that does not hold the whole file fails at load when the file's shared
    expert is not a K-quant type, as in unsloth's UD-Q4_K_XL on a 12 GB card: blk.0.ffn_gate_shexp.weight is not a resident K-quant word plane. 0.3.0 has the same defect. A card that holds the whole file is not affected. 0.3.2
    fixes it.
  • Known issue: some Qwen3-30B-A3B and Qwen3.8 files are refused at load. unsloth's Qwen3-30B-A3B UD-Q6_K_XL
    does not fit a 24 GB card whole; the plan then puts some of its Q6_K experts on the card, and the load refuses
    them (layer 3: … a Q6_K gate·up, which the card kernel table names no entry for here). Qwen3.8 Q8_0 is refused at load (the card leg runs a gate and an up of one type, [Q4_K, Q5_K, IQ3_XXS, IQ4_XS]). Both were checked on 0.3.1 with no flags. The fix, planned for 0.3.2, keeps such experts on the
    host.
  • GLM-5.3-Flash has no thinking switch. Its chat template opens a reasoning block on every reply, so
    enable_thinking: false changes nothing, and a short max_tokens can end inside that block, with the text in
    reasoning_content.

Thanks

  • @juliankang4: #43, #44, #49, #50 and #58 (DGX Spark: the aarch64 hw tests, the link probe and its move onto the
    engine's own builders, the expert stream drop, and the commit in --version), and the reports in #46, #52, #13 and #17 (#46's RTX 5090 report found the small-RAM refusal).
  • @avlp12: #55, #56 and #57 (the sampdraft test's hold, lcpp-warm's stub cleanup, and the module walk's one-line
    attributes).

Release notes and the release pass were written with AI help (Claude).

Contributors

juliankang4 and avlp12
Assets 4
Loading