Repository navigation
The server can hold a swappable set of containers — contributed by
@hendrikras in #74 — and the trunk kernel is a property of a model
rather than of the process, by @mfethe1 in #75, which closes #68.
Both were re-measured on this commit rather than taken on trust. With a
Kimi-Linear container: 110 passed, 0 failed, 6 skipped, the serve suite at
417 checks (from 319), and make asan clean with no sanitizer report,
100 passed against 0.8.1's 96. Line for line, against 0.8.1 both runs
differ by the same four added checks — #75's two and the two version
greps below — and the serve suite's own count. Nothing else moved.
The registry was additionally driven over a real socket against real
weights — two containers behind one port, and a second pair of two
different architectures — for the swap ordering, the refusals, and the
per-container facts a swap re-derives.
No ABI move: src/waste.h changes only its version macros.
Added
-
A model registry behind a stable route name, so the container a
route name resolves to can change without reconfiguring clients.
--models PATH[=ID](repeatable) registers what may be switched to;
POST /v1/models/loadswitches;GET /v1/modelslists the registry
with aloadedflag per entry.loadedmeans residency, the same
set the load response reports: with--keep-previousa model the server
has swapped away from but still holds an openwaste_ctxfor answers
loaded: true, and switching back to it is a slot move rather than a
reopen of a multi-gigabyte container.A generation naming a model the registry does not know is a 404, and
one it knows but has not loaded is a 409 pointing at the load
endpoint — not a surprise multi-gigabyte swap in the middle of a
conversation. That strictness follows the operator's opt-in (--models,
or--auto-register) and not the number of ids the registry happens to
hold, which with--auto-registeris something any client can change.
A server started without either flag is unchanged: absent, the
container's own id, or any other string all serve, including the fixed
name most OpenAI clients are configured with and the"model": "waste"
example inserve --help.A swap opens the new container before closing the old one, which is
the opposite of what a tight-RAM machine would prefer and is the point:
a failed open leaves the server serving the model it was serving. It
happens under the current engine's lock, so a generation in flight
finishes first, and a request that queued behind it finds the slot moved
and answers 409 rather than generating on a closed container. Everything
a container decides — reply format, markers, stop tokens, the thinking
default, the shapeGET /v1/modelsreports — moves in a single
assignment, so no request can see one model'swaste_ctxwith another
model's format.--auto-registerletsPOST /v1/models/loadregister a container by
pathname it has not seen before. Off by default, for the reason
--allow-local-imagesis: a client that can reach the port could
otherwise make the server open an arbitrary file on it.--usagecannot
be combined with either flag — a learned hotlist is specific to one
container. Request log lines name the model that served or was refused,
and--no-log-requestssilences them. See docs/SERVE.md, "Swapping
models". -
A swap has to prove it fits, and a load that would not is a 507. A
swap opens the new container before closing the old one — the order that
makes a failed open leave the server serving what it was serving — so
two contexts are resident at once. Documenting that and leaving the
operator to size--budgetfor it would have been worse than "the sum
of the two": with--budget 0each context sizes itself to as much as
3/4 ofwaste_usable_ram(), so a swap would run at ~1.5x what the
process may use — a paging run, not a slow one.So
--modelsrequires an explicit--budget, and the server refuses
to start unless2 x budgetfits, naming the largest budget that does;
the same lines are printed under the startup banner and by--plan,
which is where a budget gets chosen. At runtime the resident set is
counted at every load — each engine's budget, or the floor and expert
cachewaste_memory_usedreports for one that chose its own — and a
load that would not fit is refused with 507
(insufficient_memory) before the container is opened: nothing is
closed, nothing is half-loaded, the previous model keeps serving, and
the message says what it needed next to what was already held. 507
rather than 503 because asking again cannot help.That is also what bounds
--keep-previous, whose resident set grows
with every model ever switched to and which no startup check can price
in advance: the cap is derived from the budgets rather than from a
separate--max-residentcount that could disagree with them, and a
slot move to a model that is already resident is never refused because
it allocates nothing. Evicting a resident model to make room was the
alternative and is not done — changing what is resident behind a
client's back is the failure--keep-previousexists to prevent.A load that arrives with no
--budgetat all — the single-container
default, and the path an--auto-registered container takes — is priced
from the container's own plan at its floor, the least it can run
with, rather than atrecommended_bytes: the engine sizes its expert
cache down on a tighter machine instead of refusing to open, so pricing
at the recommended figure would have refused a load it would have taken
fine.serve/goes to 179 server tests plus a newtests/serve/test_main.py
of 14, for the arithmetic on its own. -
tests/run.shchecks that the version is spelled consistently, and
that the changelog was written in the same commit.src/waste.h
spells the version twice — three numeric macros and
WASTE_VERSION_STRING, whichsrc/version.cis what every banner
prints — and nothing compared them. Bumping the macros for this release
and leaving the string behind madewaste --version, the server's
banner and the header of the suite's own output all say 0.8.1 on a 0.8.2
build, with every check still green, because nothing in the suite read a
version at all. Two greps now: the string against the macros, and a
## <version>section inCHANGELOG.mdagainst the same figure, which
is the rule CLAUDE.md states and the one this release had to go back and
apply by hand.
Fixed
-
A Qwen load no longer moves a model already open onto i8mm (#68,
#75).trunk_kernandsdot4_sgwere file-statics that
matvec_t_inner,matvec_t_batchandprequant_okread for every
4-bit trunk tensor in the process, and 0.8.1 made a container load
write them: the right default for Qwen (LEARNED §83: 29% for a
difference the text cannot see) but applied process-wide. So a host that
opened a Kimi or GLM container and then a Qwen one moved the first
container's trunk onto i8mm as well, mid-generation, and the same prompt
through the same container gave different logits depending on what had
been opened earlier in the same process. i8mm is explicitly not the
exact arithmetic — that is whatWASTE_TRUNK_KERNEL=0is for — so this
was a numerical change and nothing reported it.Measured here before the fix, on two synthetic fixtures in one process:
253 of 256 logits of the already-open non-Qwen model moved, max abs
4.95e-05. The two controls that give that number meaning: a second
non-Qwen container as the thing loaded second moves 0 of 256, and so
does a Qwen container withWASTE_TRUNK_KERNELpinned. After the fix,
0 of 256 in both load orders, while a Qwen container's own logits stay
bit-identical toWASTE_TRUNK_KERNEL=2— the isolation is not
bought by denying Qwen its kernel — and a real Kimi-Linear container is
byte-identical to 0.8.1.tests/test_kernel_isolation.cholds the
check the issue asked for, and SKIPs loudly rather than passing on a
CPU without i8mm or with the kernel pinned.waste_model_set_sdot4()still sets the process default a fresh load
inherits; the newwaste_model_set_kernel(m, mode, sg)retunes one
model.tests/sweep.candtests/kernel_kl.cuse it, which they have
to: both flip the kernel inside one process, and after the change the
process-wide setter no longer reaches a model already loaded, so
kernel_klwould have compared a kernel against itself and reported
KL mean 0.00e+00— a passing-looking number from a tool that had
stopped measuring.tests/run.shnow asserts that KL is nonzero for
exactly that reason.This is also what #70 was waiting for: the i8mm trunk kernel as
DeepSeek-V4.1's default, measured in 0.8.1 at 3.87–3.97 → 4.52–4.56
tok/s and held back only because a per-model default could not be kept
to its own model. It can be now.