Skip to content

bloomery 0.2.5

Choose a tag to compare

@midagedev midagedev released this 06 Oct 14:15
· 1132 commits to main since this release

bloomery 0.2.5: one fix. Under WSL2, Qwen3.8, GLM-5.3 and DeepSeek-V4.1 no longer stop a few tokens into a request.

Install

Linux x86-64 with AVX2 (glibc 2.34+), an NVIDIA GPU of compute capability 8.6+ and driver R580 or newer (CUDA 13).
On Windows, run inside WSL2.

curl -fsSL https://raw.githubusercontent.com/midagedev/bloomery/main/tools/release/install.sh | sh
bloomery-serve --hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL --port 8080

Docker and Homebrew as in the README.

Fixed

  • A request no longer hangs under WSL2 (#1). Adaptive residency's expert
    swap waited on the copy stream for a word the host raised only between steps, and a driver that runs the copy and
    the request on one hardware queue (WSL2 does) put the request's readback behind that wait. The staging window now
    stays open whenever the host is not computing a layer, so the word is raised before the readback needs it. This
    covers Qwen3.8, GLM-5.3 and DeepSeek-V4.1. The 0.2.4 workaround, BLOOMERY_RESIDENCY=off, is no longer needed.

Also in this release: the penalty window (repeat_penalty, presence_penalty, frequency_penalty) now sees the
prompt's last repeat_last_n ids, as llama-server's does, not only the generated ones.

Thanks to @avlp12 for the report and the logs.

Full changelog: v0.2.4...v0.2.5