Skip to content

bloomery 0.3.3

Choose a tag to compare

View all tags
@midagedev midagedev released this 15 Oct 14:47
· 374 commits to main since this release
v0.3.3
5626a09

bloomery 0.3.3 adds two models: Nemotron 3 Nano 4B (bloomery-serve --model nemotron) and MiniCPM-V 4.7 images (#147). It also lays the groundwork for the Qwen3.6 in-file MTP draft and fixes two Qwen3.8 items from 0.3.2's known limits. The release pass on this candidate is the RC tape under "Checked before the tag".

Install

Linux x86-64 with AVX2 (glibc 2.34+), an NVIDIA GPU of compute capability 8.6+ and driver R580 or newer (CUDA 13).
On Windows, run inside WSL2.

curl -fsSL https://raw.githubusercontent.com/midagedev/bloomery/main/tools/release/install.sh | sh
bloomery-serve --hf unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M --port 8080

Docker and Homebrew as in the README.

New models

Model File tested Serves
Nemotron 3 Nano 4B NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf Text, one slot, the whole model on one card. A prompt runs one decode step per token (see known limits)
MiniCPM-V 4.7 35B-A3B, images MiniCPM-V-4_7-35B-A3B-Q4_K_M.gguf with --mmproj Text and images end to end. A tile is capped at 70 tokens
  • Nemotron 3 Nano 4B (#259) is built over #179 (header reader), #185 (pixtral pre-tokenizer), #191 (host Mamba-2
    step), #195 (fixture), #209 (Mamba-2 decode step on the card, one fused kernel per layer), #235 (the common pieces a
    Nemotron-H body attaches to: group-5 flash entry, Q5_0 embed rows, the recurrent store) and #236 (the mainline
    reference set).
    • The body is written on the common owners: site gemvs, the recurrent store, the shared per-head K/V append and the
      common head. It owns only the layer kinds and the residual order.
    • The Mamba-2 state cannot step back. A request keeps what the slot holds only when its prompt extends the whole
      held sequence; any other prompt resets the slot and is read again.

Changed

  • Context is one owner for every seat (#157, #158, #178, #208).
    • A load serves the context it computes from the user's slots and --ctx.
    • A --ctx past the model's trained limit stops the load by name.
    • The old per-seat context searches are gone.
  • DeepSeek V4.1:
    • The default context keeps the image encoder, and the log says when that costs a rung (#203).
    • The limit is again the trained context within i32 positions (#220).
    • The DSpark draft runs on the common draft owner, and its held card bytes are counted from the owner's and the
      layers' own buffers (#126).
  • /props reports the qwen3 seat's placement (#230).
  • Plans count the rope table they stage. The Qwen loads (#232) and MiMo (#217) hold their scratch to the plan.
  • MiMo-V2.6-Flash: the window store is a ring, with a k192 flash and neox append for ring rows (#129).
  • CPU tier (by @juliankang4):
    • Q4_K's one-column kernel, tile and row lanes get AVX-512 VNNI bodies. They are picked at run time and are
      bit-identical to the AVX2 ones (#86).
    • The Q4_K and Q5_K kernels read a block's d and dmin with one f16 conversion (#111).
  • Unified-memory machines and DGX Spark (by @juliankang4):
    • A machine's host and unified pool live in one node list (#94).
    • A model is planned over several unified nodes, on one to four Sparks (#95, #166).
    • A plan the NVMe tier's floor refuses is planned again under a card budget (#100).
    • On an integrated device the NVMe arena is one write-combined pinned allocation (#120).
    • The tier page's counters cross a link (#96, #172).
  • Qwen3.6 in-file MTP draft, groundwork (#30):
    • A qwen35moe file's next-token layer is described as an in-file draft (#165).
    • One draft step runs on the card, gated (#226).
    • The draft is not on the serve path in 0.3.3.

Fixed

  • Two Qwen3.8 items from 0.3.2's known limits:
    • On a small card whose draft auto-yields, the paged log line now names the ask before the yield (asked_draft,
      #121).
    • gate-gpu-qwen4exp-mtp (pg) plans card experts on a machine that counts their route scratch, and reads the
      drafted plan's room from the draft's plan (#255).
  • A paged plan serves the slot count it was asked (#188). With no arena, the paged draft keeps the draft and its
    reason as asked (#138, by @juliankang4).
  • A one-card plan asked for a draft is refused by name (#142).
  • The Qwen3.6 head-256 out gate writes rows of its own, so the queries are kept (#250). Model output is unchanged.
  • Qwen3.6 unjoined files (Q8_0 attention and delta projections): the e2e gate is green (#250, #257). Its forced
    attention taps read the queries rather than the gated rows. Its layer-0 band follows the activation rule of the delta
    projections. Model output is unchanged.
  • gguf refuses malformed files by name (by @juliankang4):
    • a tensor whose bytes or data range pass u64 (#149);
    • a top-k above the expert count (#153).
  • bloomery-serve's draft pick names every place it looked when the line resolves to off (#114).
  • A GLM or Qwen3.8 --plan run no longer probes the GPU lane; a load still probes before it reads the room (#223,
    closes #204).
  • On an integrated device, a small pinned arena no longer starts off a 4 KiB boundary (#120, closes #109).

Tools and contributors' checks (developer-facing)

  • just pre-review runs the Shingler reviewer locally before a push (#199) and refuses a base behind origin/main
    (#231). Shingler ranks correctness first, then duplication, and re-reviews only new commits (#161, #253). It can run
    on Codex (#107, #244, by @juliankang4).
  • Box queue and runners:
    • queued jobs no longer resync each other; INT and TERM cancel only queued jobs; an attach rides out a daemon blip
      (#181, #196, #197, #198, #205, #225);
    • a one-card gate asked of another card is refused by name (#211, #222);
    • a red gate is rechecked on origin/main (#215);
    • the queue's self-test counts every follow before it waits on them (#266).
  • just canon-check BASE proves a kernel move bit-identical (#245). gate-ptx-spill reads per-arch tables, with cc
    12.1 tables for sm_121 (#170, by @juliankang4).
  • tools/gate-paths-merge.py is a merge driver for tools/gate-paths.tsv (#182). recipes.py check --fill adds the
    gates a row lacks (#183).
  • The v3 release check reads each instruction's encoding, so AVX-512 on ymm and xmm is caught (#150, by
    @juliankang4).
  • Native AMD: contributor-owned HIP operator checks under lab/ (#248, by @Burntgogi). This is not a shipped backend.
  • The dead feature-off stub main is gone from the 112 bins that declare required-features (#99, by @avlp12).
  • Groundwork for later models, with no serve path yet, all by @juliankang4:
    • gemma-embedding2's header reader (#238) and the gemma4 tokenizer (#241);
    • EmbeddingGemma 2's and EmbeddingGemma (v1)'s mainline oracle sets and their families (#260, #267);
    • geglu_tanh, the GELU-gated unit over f32 rows on the card (#237);
    • the head-512 rope append and prefill flash of EmbeddingGemma 2's global layers (#233).

Off cells and known limits

  • Nemotron 3 Nano 4B reads a prompt at its decode rate: the body has no prompt batch, so a prompt runs one decode
    step per token.
  • Nemotron 3 Nano 4B serves one slot on one card: --parallel above 1, a placement of more than one card or with a
    tier card, and a draft lever are each refused by name. It runs no prompt cache, and a snapshot or resume is refused
    by name.
  • Qwen3.6 real-tier e2e gate, split residency clause: it can fail on one run and pass on the next at the same
    commit (on an RTX 3090, logits distance from the off run up to 4.879e-1 against its 0.26 band, then 1.6–1.8e-1 on a
    rerun). The experts it streams follow the lane rate measured at runtime, so the clause's distance moves from run to
    run. This is a developer gate, not a change in what a load serves.
  • The Qwen3.6 in-file MTP draft is not served in 0.3.3 (groundwork only, above).
  • EmbeddingGemma and EmbeddingGemma 2 have no embeddings endpoint in 0.3.3 (#190): their groundwork is in this
    release, and the endpoint comes in the next one.
  • On Qwen3.8 loads that page experts through the NVMe tier, gate-gpu-qwen38-serve's real tier was 70/72 on 0.3.2's
    baseline (f382b474), two preexisting paged failures. It was not re-run on its own for this release.
  • Carried over from 0.3.2:
    • Qwen3.6-35B-A3B has no DSpark draft; BLOOMERY_DRAFT=dspark on it is refused by name.
    • DSpark (V4.1) serves one slot: --parallel above 1 with it is refused by name.
    • An image request under DSpark (V4.1) is refused by name.
    • V4.1 DSpark on one card under --place a: the draft shares the target's card with no reserve, so a load past its
      free bytes stops with a CUDA allocation error.

Checked before the tag

  • Release head: 5626a096.
  • The release-candidate pass, its rows and their conditions: https://tape.midagedev.com/r/tz7hzsysnd9u85cpf8mf.
  • Nemotron 3 Nano 4B on the release head: https://tape.midagedev.com/r/tz7hzsysnd9u85cpf8mf.

Thanks

Contributors

juliankang4, Burntgogi, and avlp12
Assets 4
Loading