Repository navigation
Releases
Releases · convexityos/vitna-anchor
v0.2.1
In this release
- A machine with 16 GB of memory and an 8 GB NVIDIA GPU now gets Qwen3-30B-A3B. Below 30 GiB of memory, the installer serves it with its experts read from the drive into a 4 GiB cache (
--expert-cache 4096) instead of holding the whole 18.6 GB file in memory, where it used to install SmolLM2-360M-Instruct. With 32 GB or more nothing changes. - Measured on an RTX 3070 (Ryzen 7 3700X, model on an NVMe drive, context 16K, median of 3): 5.1 GiB of memory against 17.5; decoding at 16.9 tokens a second after a short prompt (44.8 holding the file) and 9.0 after an 8,000-token prompt (29.5); that prompt read at 148 tokens a second (300). The README's "Speed with less memory" has the commands.
- The engine's measured history is now data:
bench/history.json, every step from gate A5 on with its machine, command, run count and statistic, held to the README bytests/bench-history.test.mjs. The README also gains "Speed beside AirLLM".
Known limitation: the 16 GB plan was measured and checked end to end on a 64 GB machine described to the planner as 15.9 GiB. It has not been run on a machine that has only 16 GB, and the plan does not measure the drive, so a SATA SSD or hard drive will be slower than the figures above.
What's Changed
- docs: gate A12 passed, against release v0.2.0 by @garr3tth in #53
- A machine with 16 GB of memory gets Qwen3-30B-A3B, its experts read from the drive by @garr3tth in #54
- The engine's measured history as data, held to the README, and Anchor beside AirLLM by @garr3tth in #55
- chore: version 0.2.1 by @garr3tth in #56
Full Changelog: v0.2.0...v0.2.1
v0.2.0
What's Changed
- ci: run the test suite on Windows again, and stop one failure cancelling the rest by @garr3tth in #1
- A0: say what runs today, and print nothing nobody measured by @garr3tth in #2
- One home: record the canonical copy of each shared module, and remove the gateway's modules from here by @garr3tth in #3
- A1: pin SmolLM2-135M, record its token ids and logits, and fix the tolerance A2 must meet by @garr3tth in #4
- A2: a Llama forward pass in float32 on the CPU that matches the reference by @garr3tth in #5
- docs: the gate table says A0 to A2 passed by @garr3tth in #6
- A3: serve the model over an OpenAI-compatible /v1, with usage counted from tokens by @garr3tth in #7
- docs: the gate table says A3 passed by @garr3tth in #8
- A6: prefix reuse over the key-value cache, and a JSON mode that masks logits by @garr3tth in #9
- docs: the gate table says A6 passed, and what it leaves out by @garr3tth in #10
- A step that fails is an error the client sees, not a reply cut short by @garr3tth in #12
- A4: the forward pass in float32 on an NVIDIA GPU, matching the reference on one GPU, checked by hand by @garr3tth in #11
- Sampling sorts the vocabulary by radix: a sampled token about 9x cheaper to choose, the same draws by @garr3tth in #18
- GPU speed: CUDA graphs, fused kernels and attention split by position, with the same answers by @garr3tth in #13
- GPU attention scores each position in a lane of its own: decoding about 10% faster at a short prompt by @garr3tth in #14
- GPU prompts run their tokens together: about 25x faster to the first token on a long prompt by @garr3tth in #15
- GPU prompt attention tiled like a matrix product: the kernel 1.6 to 1.8x faster on long prompts by @garr3tth in #16
- GPU decoding: RMSNorm reads x and its weights in one trip, about 5% less GPU time a token by @garr3tth in #17
- Speculative decoding that changes no reply: exact multi-token steps on the GPU, and generate --speculate by @garr3tth in #19
- serve --speculate: the server drafts and checks tokens together, and no response changes by @garr3tth in #20
- JSON mode keeps the mask it finds for each state of the object: the same masks, half the time by @garr3tth in #21
- Checking drafted tokens costs half what it did on the GPU: every logit unchanged, speculation up to 3.4x by @garr3tth in #22
- serve --parallel: requests at once, each response the one it gets alone, 3.7x the tokens a second for eight clients by @garr3tth in #23
- Sampled requests at once: each thread samples with its own sampler, 16x the tokens a second for eight clients by @garr3tth in #24
- A long prompt runs in pieces between other requests' rounds, bit for bit as whole: stalls from a 2,000-token prompt 81 ms to 36 by @garr3tth in #25
- docs: the gate table says A4 passed, and A4 checked again after the speed work by @garr3tth in #26
- A failed test shows what its servers printed on stderr, so a 500 names its CUDA error by @garr3tth in #27
- A step a test fails leaves nothing running on the GPU, which ends the GPU suite's intermittent 500s by @garr3tth in #28
- A server that never starts, or an engine run that never ends, fails its test instead of hanging the suite by @garr3tth in #30
- When a before hook fails, each failed test carries the servers' stderr once, not once per test so far by @garr3tth in #31
- After a GPU error CUDA cannot recover from, the server answers what is open and exits with status 75, rather than answering 500 forever by @garr3tth in #33
- feat(reference): gate A5's reference, OLMoE-1B-7B pinned and recorded with every layer's routing by @garr3tth in #29
- feat(engine): gate A5 step 2, OLMoE on the CPU, matching its reference pinned and unpinned by @garr3tth in #32
- feat(engine): gate A5 step 3, OLMoE's experts read from the drive with direct I/O and prefetch, the logits unchanged by @garr3tth in #34
- feat(engine): gate A5 on the GPU, OLMoE with its experts copied into a cache on the device as layers want them, matching its reference by @garr3tth in #35
- docs: the gate table says A5 passed, and A5 checked again at the merge by @garr3tth in #36
- perf(engine): a mixture of experts runs several tokens together as rows on the GPU, a prompt 17x faster, with a token at a time's logits bit for bit by @garr3tth in #37
- run, generate and logits say why a step did not run, instead of blaming --ctx for a GPU error by @garr3tth in #38
- feat(engine): text embeddings at /v1/embeddings, matched to a pinned reference (A14) by @garr3tth in #43
- perf(engine): embeddings 1.2 to 1.4 times faster, and A14's row says Passed by @garr3tth in #44
- docs: the gate ladder plans A7 to A13, parity with Strata by @garr3tth in #39
- feat(engine): quantized weights from a GGUF file on the CPU, matched to a reference recorded on them (A7, first half) by @garr3tth in #40
- feat(engine): quantized weights on the GPU, each kernel instantiated for the dtype of each matrix it reads (A7, second half) by @garr3tth in #41
- feat(engine): the CPU runs the experts the GPU lacks, beside it, and the logits stay the GPU's bits (A8 on OLMoE) by @garr3tth in #42
- feat(engine): Qwen3-30B-A3B on the CPU and the GPU, from BF16, Q8_0 and Q4_K_M, matched to a reference recorded a layer at a time (A9) by @garr3tth in #45
- feat(engine): a 32K prompt on an 8 GB GPU, the cache it has no room for in host memory, the logits unchanged bit for bit (A10) by @garr3tth in #46
- docs: gates A7 to A10 passed, and the decisions of 2026-10-06 in the parity plan by @garr3tth in #47
- feat(engine): --precision fast, a prompt on the tensor cores, 2.9 times as fast and faster than llama.cpp, measured against the reference by @garr3tth in #48
- Gate A11: Anthropic's Messages and OpenAI's Responses, with tool calls and reasoning by @garr3tth in #49
- Gate A12, part 1: a chat page, built into the engine and served at / by @garr3tth in #50
- Gate A12, part 2: one command that sizes the machine, fetches a fitting model with checks, and serves it by @garr3tth in #51
- ci(release): CUDA engines for Windows and Linux, the installers filled in, and every release installed on fresh runners by @garr3tth in #52
New Contributors
Full Changelog: v0.1.0...v0.2.0
v0.1.0
Full Changelog: https://github.com/convexityos/vitna-anchor/commits/v0.1.0