Skip to content

Releases

v0.2.1

Choose a tag to compare

@github-actions github-actions released this 10 Oct 21:49
e72ee25

In this release

  • A machine with 16 GB of memory and an 8 GB NVIDIA GPU now gets Qwen3-30B-A3B. Below 30 GiB of memory, the installer serves it with its experts read from the drive into a 4 GiB cache (--expert-cache 4096) instead of holding the whole 18.6 GB file in memory, where it used to install SmolLM2-360M-Instruct. With 32 GB or more nothing changes.
  • Measured on an RTX 3070 (Ryzen 7 3700X, model on an NVMe drive, context 16K, median of 3): 5.1 GiB of memory against 17.5; decoding at 16.9 tokens a second after a short prompt (44.8 holding the file) and 9.0 after an 8,000-token prompt (29.5); that prompt read at 148 tokens a second (300). The README's "Speed with less memory" has the commands.
  • The engine's measured history is now data: bench/history.json, every step from gate A5 on with its machine, command, run count and statistic, held to the README by tests/bench-history.test.mjs. The README also gains "Speed beside AirLLM".

Known limitation: the 16 GB plan was measured and checked end to end on a 64 GB machine described to the planner as 15.9 GiB. It has not been run on a machine that has only 16 GB, and the plan does not measure the drive, so a SATA SSD or hard drive will be slower than the figures above.

What's Changed

  • docs: gate A12 passed, against release v0.2.0 by @garr3tth in #53
  • A machine with 16 GB of memory gets Qwen3-30B-A3B, its experts read from the drive by @garr3tth in #54
  • The engine's measured history as data, held to the README, and Anchor beside AirLLM by @garr3tth in #55
  • chore: version 0.2.1 by @garr3tth in #56

Full Changelog: v0.2.0...v0.2.1

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 06 Oct 22:19
e780272

What's Changed

  • ci: run the test suite on Windows again, and stop one failure cancelling the rest by @garr3tth in #1
  • A0: say what runs today, and print nothing nobody measured by @garr3tth in #2
  • One home: record the canonical copy of each shared module, and remove the gateway's modules from here by @garr3tth in #3
  • A1: pin SmolLM2-135M, record its token ids and logits, and fix the tolerance A2 must meet by @garr3tth in #4
  • A2: a Llama forward pass in float32 on the CPU that matches the reference by @garr3tth in #5
  • docs: the gate table says A0 to A2 passed by @garr3tth in #6
  • A3: serve the model over an OpenAI-compatible /v1, with usage counted from tokens by @garr3tth in #7
  • docs: the gate table says A3 passed by @garr3tth in #8
  • A6: prefix reuse over the key-value cache, and a JSON mode that masks logits by @garr3tth in #9
  • docs: the gate table says A6 passed, and what it leaves out by @garr3tth in #10
  • A step that fails is an error the client sees, not a reply cut short by @garr3tth in #12
  • A4: the forward pass in float32 on an NVIDIA GPU, matching the reference on one GPU, checked by hand by @garr3tth in #11
  • Sampling sorts the vocabulary by radix: a sampled token about 9x cheaper to choose, the same draws by @garr3tth in #18
  • GPU speed: CUDA graphs, fused kernels and attention split by position, with the same answers by @garr3tth in #13
  • GPU attention scores each position in a lane of its own: decoding about 10% faster at a short prompt by @garr3tth in #14
  • GPU prompts run their tokens together: about 25x faster to the first token on a long prompt by @garr3tth in #15
  • GPU prompt attention tiled like a matrix product: the kernel 1.6 to 1.8x faster on long prompts by @garr3tth in #16
  • GPU decoding: RMSNorm reads x and its weights in one trip, about 5% less GPU time a token by @garr3tth in #17
  • Speculative decoding that changes no reply: exact multi-token steps on the GPU, and generate --speculate by @garr3tth in #19
  • serve --speculate: the server drafts and checks tokens together, and no response changes by @garr3tth in #20
  • JSON mode keeps the mask it finds for each state of the object: the same masks, half the time by @garr3tth in #21
  • Checking drafted tokens costs half what it did on the GPU: every logit unchanged, speculation up to 3.4x by @garr3tth in #22
  • serve --parallel: requests at once, each response the one it gets alone, 3.7x the tokens a second for eight clients by @garr3tth in #23
  • Sampled requests at once: each thread samples with its own sampler, 16x the tokens a second for eight clients by @garr3tth in #24
  • A long prompt runs in pieces between other requests' rounds, bit for bit as whole: stalls from a 2,000-token prompt 81 ms to 36 by @garr3tth in #25
  • docs: the gate table says A4 passed, and A4 checked again after the speed work by @garr3tth in #26
  • A failed test shows what its servers printed on stderr, so a 500 names its CUDA error by @garr3tth in #27
  • A step a test fails leaves nothing running on the GPU, which ends the GPU suite's intermittent 500s by @garr3tth in #28
  • A server that never starts, or an engine run that never ends, fails its test instead of hanging the suite by @garr3tth in #30
  • When a before hook fails, each failed test carries the servers' stderr once, not once per test so far by @garr3tth in #31
  • After a GPU error CUDA cannot recover from, the server answers what is open and exits with status 75, rather than answering 500 forever by @garr3tth in #33
  • feat(reference): gate A5's reference, OLMoE-1B-7B pinned and recorded with every layer's routing by @garr3tth in #29
  • feat(engine): gate A5 step 2, OLMoE on the CPU, matching its reference pinned and unpinned by @garr3tth in #32
  • feat(engine): gate A5 step 3, OLMoE's experts read from the drive with direct I/O and prefetch, the logits unchanged by @garr3tth in #34
  • feat(engine): gate A5 on the GPU, OLMoE with its experts copied into a cache on the device as layers want them, matching its reference by @garr3tth in #35
  • docs: the gate table says A5 passed, and A5 checked again at the merge by @garr3tth in #36
  • perf(engine): a mixture of experts runs several tokens together as rows on the GPU, a prompt 17x faster, with a token at a time's logits bit for bit by @garr3tth in #37
  • run, generate and logits say why a step did not run, instead of blaming --ctx for a GPU error by @garr3tth in #38
  • feat(engine): text embeddings at /v1/embeddings, matched to a pinned reference (A14) by @garr3tth in #43
  • perf(engine): embeddings 1.2 to 1.4 times faster, and A14's row says Passed by @garr3tth in #44
  • docs: the gate ladder plans A7 to A13, parity with Strata by @garr3tth in #39
  • feat(engine): quantized weights from a GGUF file on the CPU, matched to a reference recorded on them (A7, first half) by @garr3tth in #40
  • feat(engine): quantized weights on the GPU, each kernel instantiated for the dtype of each matrix it reads (A7, second half) by @garr3tth in #41
  • feat(engine): the CPU runs the experts the GPU lacks, beside it, and the logits stay the GPU's bits (A8 on OLMoE) by @garr3tth in #42
  • feat(engine): Qwen3-30B-A3B on the CPU and the GPU, from BF16, Q8_0 and Q4_K_M, matched to a reference recorded a layer at a time (A9) by @garr3tth in #45
  • feat(engine): a 32K prompt on an 8 GB GPU, the cache it has no room for in host memory, the logits unchanged bit for bit (A10) by @garr3tth in #46
  • docs: gates A7 to A10 passed, and the decisions of 2026-10-06 in the parity plan by @garr3tth in #47
  • feat(engine): --precision fast, a prompt on the tensor cores, 2.9 times as fast and faster than llama.cpp, measured against the reference by @garr3tth in #48
  • Gate A11: Anthropic's Messages and OpenAI's Responses, with tool calls and reasoning by @garr3tth in #49
  • Gate A12, part 1: a chat page, built into the engine and served at / by @garr3tth in #50
  • Gate A12, part 2: one command that sizes the machine, fetches a fitting model with checks, and serves it by @garr3tth in #51
  • ci(release): CUDA engines for Windows and Linux, the installers filled in, and every release installed on fresh runners by @garr3tth in #52

New Contributors

Full Changelog: v0.1.0...v0.2.0

v0.1.0

Choose a tag to compare

@github-actions github-actions released this 16 Sep 16:13