server: preemption notices, asynchronous parks and exact concurrency together - #197
Draft
danielhanchen wants to merge 204 commits into
Draft
server: preemption notices, asynchronous parks and exact concurrency together#197danielhanchen wants to merge 204 commits into
danielhanchen wants to merge 204 commits into
Conversation
A slot parked by the preemption path produces nothing until its cells come back, and to a client that is indistinguishable from a hung server: the stream goes silent, read timeouts fire, and a chat that was merely waiting for room is torn down as broken. Push a small out-of-band result to the task's response queue when a streaming slot is parked and when it is restored. The HTTP layer writes it as an SSE comment, ": preempted" and ": resumed", which is legal SSE that every existing client ignores, so the body of the response is unchanged by preemption. While parked the ping runs every 2 s as ": preempt-keepalive" regardless of --sse-ping, so proxies and client read timeouts survive a wait that is long by design. Notices that arrive before the first real result (a slot parked while it was still processing its prompt) are sent in front of it. Non-streaming requests see nothing. Harness test: forced parks every 8 tokens on /completion and /v1/chat/completions carry the comments in park/resume order and generate the same tokens as the unparked run; a non-streaming request is untouched; two streams that overflow the pool together both finish and the parked one says so.
…atch The number of tokens in a batch selects the matmul implementation, the flash attention kernel, and inside several of them how the K loop or the KV cache is divided between threads and blocks. All of those change the order in which the partial products of one destination element are summed, so a request decoding next to three others produces different bits than the same request decoding alone, even at temperature 0. GGML_CUDA_BATCH_INVARIANT=1 computes every destination column, and attends every query row, with the configuration a batch of one would use. =2 does the same but only where the batch-of-one configuration actually differs, which leaves the quantized projections batched because MMVQ already uses the same nwarps for one to four columns. Flash attention additionally pins the vector kernel, pins the split over the KV cache to one block per tile, and scans the mask for the sequence's own extent, so neither the query count nor the length of a shared KV cache selects the algorithm. Measured on a B200 with Qwen3.5-4B-UD-Q4_K_XL: with the KV cache state held equal, the 1831 node decode graph goes from 1190 nodes whose sequence-0 row differs between a one token and a four token ubatch to 0, and 256 greedy decode steps that diverged at step 125 become identical.
…MAX_COLS Splitting a prompt-sized batch costs far more than splitting a decode-sized one: a 273 token prefill becomes 273 single column matmuls and 273 single row attention launches per layer, which took prompt processing from 2731 to 225 tok/s on a B200 while four-chat decode only lost 7 percent. GGML_CUDA_BATCH_INVARIANT_MAX_COLS caps the width the split applies to, 0 keeps the previous unbounded behaviour. At 8 it covers every decode batch the server can form and leaves prefill alone, which restores prompt processing to 2696 tok/s and four-chat wall throughput to 127.2 against 136.4 unpatched. The bound gives up invariance for the prompt phase, so it is opt-in rather than the default.
…ENCY The recurrent half of a hybrid model is not invariant to the shape of the ubatch. With the attention half made exact, a prompt processed in ubatches it shares with other sequences' prompt tokens still leaves a different gated delta net state than the same prompt processed alone: the first node to show it is the layer 1 recurrent state, 2.2e5 of 5.2e5 elements, max 7.3e-4, and over a 512 token generation it flips a token at step 79. split_equal grows an optional cap on the number of sequence sets per ubatch. The hybrid memory passes 1 when the mode is on and some sequence contributes more than one token to the batch, which is the prompt phase. A plain decode step, one token per sequence, is already exact under the gather and stays batched, so the cost falls on prompt processing only.
ggml_backend_event_synchronize() is the only way to find out whether the work recorded before an event has finished, and it answers by waiting for it. A caller that issued an asynchronous copy so that it could get on with something else has no way to ask "is it done yet" without giving that up again. ggml_backend_event_query() is that question. It is optional, and it is the last field of ggml_backend_device_i so that a backend which does not implement it needs no change: a missing entry is NULL and the generic implementation falls back to a blocking synchronize and returns true, which is correct, just no better than what a caller could do already. CUDA implements it with cudaEventQuery, treating cudaErrorNotReady as the answer "not yet" rather than as a failure, and clearing it so it is not reported against the next call. The other sixteen device interfaces get an explicit NULL. Trailing initializers could have been left off, since these are positional aggregate initializers and the new member would be value-initialized, but -Wmissing-field-initializers is part of -Wextra and becomes an error under LLAMA_FATAL_WARNINGS.
Two changes to how a sequence's state is copied out of and back into the cache, the first of which the second one needs. Coalescing. The save side works out which cells belong to the sequence, merges them into ranges and emits one write per range per tensor. The restore side does not: it emits one read per cell, thousands of them, even when the cells it was given are a handful of long runs. Merging fragments that are adjacent in both the tensor and the buffer fixes both sides at once, and covers the transposed V layout where the same runs are emitted once per embedding row. Sequences sharing a unified cache take their cells in turn, so what is left after merging is a regular comb rather than one block; a comb is what a strided copy describes, so runs of one length at a constant stride become a single 2d transfer. Measured on a 4B at -c 8192 with four chats, a 1989-cell sequence goes from 1989 transfers per tensor to about 160, and a sequence that has the cache to itself to one. Asynchronous transfers. llama_state_seq_copy is a transfer that can be issued and left running: it owns the host buffer, a backend per device holding part of the cache so the copies get a stream of their own rather than queueing behind the graphs, and an event per device to say when its half is done. The buffer is pinned where the backend offers pinned memory, which is what makes the copies overlap at all, and grow-only, because page-locking a hundred MiB costs about as long as the copy it is for and a caller parking the same sequence repeatedly asks for a slightly different size each time. The restore side of the asynchronous path deliberately does not use the whole-tensor staging the synchronous one does. Staging reads a tensor, patches the sequence's bytes into the host copy and writes the tensor back, which keeps the neighbours only while nothing else is touching the cache. These copies exist so that decoding can carry on beside them, so the write-back would undo whatever the sequences sharing the tensor wrote to their own cells in the meantime. Writing only this sequence's runs cannot, and coalescing is what makes that affordable. llama_state_seq_copy_init() returns NULL when no backend can copy asynchronously, so a caller keeps the synchronous calls on those.
preempt_save() and preempt_restore() run inside update_slots(), so while one sequence is copied out of or back into the KV pool every other slot stops. On a 4B at -c 8192 with four chats that is a 250 ms freeze at a park and 149 ms at a restore, seen by chats that had nothing to do with either, against a p99 inter-token gap of 19 ms when nothing is being parked. The park was cheap for the slot it saved; it was the other three that paid for it. A park now has two halves. preempt_save() issues the copy and leaves the slot PREEMPTING: the cells are still its own, because the copy is still reading them, and nobody may take them. update_slots() polls the event each iteration and only then releases the cells and marks the slot PREEMPTED. A restore is the mirror, RESTORING: the cells are allocated and owned by the sequence, so nobody else can take them, but they do not hold its state until the copy lands, which is why the slot is not scheduled and its drafter not rearmed until it does. An asynchronous park does not hand its cells back before update_slots() carries on, so it has to fire earlier than a synchronous one, or the slots that keep decoding have nowhere to put their tokens and end up waiting for the copy after all. preempt_n_margin() keeps eight decode steps of every running slot clear ahead of the pool filling, which is about the tenth of a second a copy of one sequence takes. The same figure gates a resume, so that a slot is not put back into a pool it would immediately have to be taken out of again. When that lookahead is not enough the loop waits for the outstanding park rather than let the KV-full path end every request, which is no worse than the synchronous path and is the last thing tried before giving up. Everything that reads a slot's state had to learn the two new ones. is_processing() is deliberately left as "not idle", because it is what keeps NEXT_RESPONSE posted and the loop polling; narrowing it would deadlock a server whose only slots are mid-copy. preempt_kv_used() deliberately still counts them, since a slot on its way out has not released its cells and one on its way back in has already been given them. release() waits for an outstanding copy before freeing the buffer and handing the cells on, which is the path a cancelled request and every error path take, and where a transfer would otherwise outlive the memory on both ends. --preempt-async (LLAMA_ARG_PREEMPT_ASYNC) is on by default and falls back to the synchronous path on a backend that cannot copy asynchronously, saying so once at load. --no-preempt-async keeps the old behaviour, so both can be compared on one binary. The pinned buffers are held for as long as the task that parked owns the slot rather than freed between two of its parks, so --preempt-ram now bounds the host memory actually held; it still reads zero once the slots are released. Measured on the same four chats, survivors now see 38 to 43 ms at a park and 19 ms at a restore under LLAMA_SERVER_PREEMPT_EVERY=64, against 127 to 158 ms and 112 to 115 ms before, and four-chat throughput goes from 179-184 to 243-267 tok/s. What is left is issue cost: about 11500 transfers at 4 us each, because four chats interleaving in one pool leave a sequence in roughly 160 runs per tensor. A sequence that has the pool to itself is 66 transfers and 0.26 ms. Tests: the asynchronous path is byte-identical to an uninterrupted run and to the synchronous path, two slots that overflow the pool together finish with the tokens they produce alone, cancelling while a copy is in flight leaves no slot stuck and no parked memory held, and --no-preempt-async really does switch it off.
A mixture-of-experts decode groups the ubatch's tokens by the expert they routed to, so the column count of an expert's matmul, the rows the per-expert copy gathers and the width the activations are quantized at all depend on what the other tokens in the ubatch picked. The quantized path makes it visible: at one token per ubatch MUL_MAT_ID runs mul_mat_vec_q at ncols_dst 1, four warps dividing the K loop and a shared memory reduction across them, and at more than one token it runs the dedicated MoE kernel, one warp per token with a warp only reduction. Whether those two agree bit for bit depends on the quantization type and on K. Measured on a B200 they agree for Q4_K and Q5_K up to K 2048 and disagree for Q6_K and Q8_0 from K 512, which is why on Qwen3.6-35B-A3B-UD-Q4_K_XL the Q4_K gate and up projections of every layer matched and the three Q6_K down projections, layers 34, 38 and 39, did not. Under GGML_CUDA_BATCH_INVARIANT the node is now computed one token at a time. Each call then sees the shapes a batch of one has, whatever the neighbours routed to, which covers the ids variants of MMVQ, MMQ and MMF and the sorted per expert fallback with one change. GGML_CUDA_BATCH_INVARIANT_MAX_COLS bounds it the way it bounds the MUL_MAT column split. The CUDA graph fallback check is evaluated against the single-token path as well, since that is what a split node runs. On the 35B the sequence-0 slice of one decode step goes from 195 of 3727 nodes differing between a one token and a four token ubatch to 0, and a standalone MUL_MAT_ID probe over Q4_K, Q5_K, Q6_K, Q8_0, Q4_0, Q3_K, Q2_K, MXFP4, F16 and BF16 at K 512, 2048 and 4096 goes to 0 at every token count up to 8.
ggml_cuda_check_fusion_memory_ranges accepts the top-k MoE subgraph through an explicit ggml_nrows(node) == 1 exception, which skips the aliasing test on the grounds that each row is read entirely before it is written. With more than one token the generic overlap test runs instead and refuses. So a request decoding alone computes its routing weights with the fused warp local top-k kernel and the same request decoding next to three others computes them with the softmax, argsort, get_rows, sum, clamp and divide chain. Two algorithms for one set of routing weights is exactly the batch dependence this knob removes, and the same reason mul_mat plus GLU fusion is already off here. A node probe cannot see this, which is worth recording: registering an eval callback disables fusion, so both sides of the comparison take the unfused chain and agree. It only appears when the two are compared without one. On Qwen3.6-35B-A3B, with every node of one decode step already byte-identical, sequence 0's logits differed in 248319 of 248320 entries from the first decode step, by up to 2.6e-1, and the greedy stream flipped a token at step 47. The dense 4B is unaffected either way, and disabling every CUDA fusion removes it, which is what named the fused op. With the routing left unfused under the knob, 512 decode steps of sequence 0 alone against sequence 0 next to three neighbours are byte-identical on the 35B, and every cell of the server matrix reads identical.
…y batch Exact mode still defaults the column policy to unbounded, so every measurement that recovered the prefill cost had to set GGML_CUDA_BATCH_INVARIANT_MAX_COLS by hand, and the value everything was measured at, 8, does not cover a speculative verify ubatch by construction: with --parallel 4 and --spec-type draft-mtp --spec-draft-n-max 2 a verify ubatch holds one accepted token plus two drafts per slot, up to 12 columns, and above the bound neither the MUL_MAT column split nor the MUL_MAT_ID per token split fires. Default the bound to 16 in exact mode instead, which covers four slots at up to three tokens each. An explicitly set GGML_CUDA_BATCH_INVARIANT_MAX_COLS still wins, in either mode, so a deployment with more slots or a wider draft can raise it. Measured on one B200, Qwen3.6-35B-A3B-UD-Q4_K_XL and Qwen3.5-4B-UD-Q4_K_XL, --parallel 4 --kv-unified -c 8192 --flash-attn on -ngl 99 --seed 0, greedy sampling, 512 predicted tokens, P0 solo twice then P0 concurrent with P1 to P3, three rounds per cell. Every P0 completion was byte identical to its solo reference and every solo repeat matched: 35B, draft-mtp n-max 2, MAX_COLS=16 identical x3 35B, draft-mtp n-max 2, MAX_COLS=16, PREEMPT_EVERY=64 identical x3, 98/98 parks 35B, draft-mtp n-max 4, MAX_COLS=8 (verify up to 20) identical x3 35B, draft-mtp n-max 4, MAX_COLS=32 identical x3 4B, draft-mtp n-max 2, MAX_COLS=16, PREEMPT_EVERY=64 identical x3, 98/98 parks 35B, draft-mtp n-max 2, new default, no env var identical x3 35B, spec off, new default, no env var identical x3 The 35B MTP cell at 16 reproduces the acceptance counters of the same cell at 8 exactly, 4091 of 6103 draft tokens, so raising the bound does not perturb the generation. Cost, three interleaved MAX_COLS=8 against MAX_COLS=16 pairs on the 35B with speculation on, medians: solo decode 25.42 against 25.37 tok/s, four chat aggregate decode 54.55 against 54.23 tok/s, four chat wall 19.98 against 20.15 s. Four interleaved pairs with speculation off, where a decode ubatch is four columns wide and both bounds must behave identically, medians 31.39 against 31.40 tok/s solo and 111.41 against 111.41 tok/s aggregate. All within the run to run spread. test-backend-ops test -b CUDA0 -o MUL_MAT_ID,MUL_MAT under LLAMA_EXACT_CONCURRENCY=1 GGML_CUDA_BATCH_INVARIANT=2 MAX_COLS=16: 2136/2136 passed, 2/2 backends.
…de-preemption-integration
…e-preemption-integration The async branch splits both park sites in two, which moves the three places the notify branch emits its notices from. Resolved by intent rather than by side: * Forced park (LLAMA_SERVER_PREEMPT_EVERY), the one textual conflict. The notify side had preempt_save() inside the if condition; the async side lifted it into the body so it can tell a PREEMPTED slot (counted here) from a PREEMPTING one (counted by update_preempt_copies() when the copy lands). Kept the async body and put send_preempt_notice(slot, true) inside it. * KV-full victim park. Auto-merge left the notice after the PREEMPTING early-out, so an asynchronous park would have announced nothing. Moved it to directly after preempt_save() succeeds, which is the point the slot is announced from in both paths. The notice fires when the slot enters PREEMPTING, not when its cells are released. preempt_save() has already called preempt_detach() and left the slot out of every decode, so that is the moment the client's stream goes silent; the cells come back one copy later, and on the KV-full path an unbounded time later. Announcing at the release would leave exactly the unexplained silence the notices exist to remove. * Restore. Auto-merge left the notice on the synchronous tail, past the RESTORING early-out, so an asynchronous restore announced nothing. Kept it there for the synchronous path, where the issue and the landing are one moment, and added one in update_preempt_copies() after preempt_restore_poll() succeeds, which is where the slot is put back into the state it was parked from and can be scheduled again. Announcing at the issue would claim a resume while the slot was still RESTORING and producing nothing. Together these keep the HTTP layer's parked flag, and so the 2 s ": preempt-keepalive", set across all of PREEMPTING, PREEMPTED and RESTORING: it is raised by the park notice at the issue of the copy out and cleared by the resume notice at the landing of the copy back in. Every park emits exactly one ": preempted" and every restore exactly one ": resumed"; the synchronous and asynchronous sites are mutually exclusive, since preempt_save() leaves a slot PREEMPTED only without a transfer and preempt_restore() leaves it RESTORING only with one.
…n-integration No textual conflict: #194 touches the CUDA backend, the batch splitter and the KV cache allocator, and the only file it shares with the two merged branches is ggml/src/ggml-cuda/ggml-cuda.cu, where #192 adds ggml_backend_event_query at the backend interface and #194 works on the op dispatch, far apart in the file. Merged at 65860ea, which is the current head. It carries the eighth commit defaulting GGML_CUDA_BATCH_INVARIANT_MAX_COLS to 16 in exact mode, so the bound that covers a --spec-draft-n-max 2 verify ubatch of four slots is the default and the live runs need no environment override. The interaction that needed checking is the one the two branches do not name to each other: #194's page allocator against the cells #192's park releases and its restore reallocates. Traced rather than assumed, and it composes without a change: * An asynchronous restore reaches the page allocator. server_slot:: preempt_restore() -> llama_state_seq_copy_set() -> llama_context:: state_seq_copy_set() -> state_seq_read_data() -> llama_memory::state_read() -> llama_kv_cache::state_read_meta() with dest_seq_id set -> find_slot(), whose exact_pages branch is the page allocator. The metadata half of the read is synchronous host-side work, so the pages are taken while the restore is being issued, which is exactly what RESTORING is defined to mean: cells allocated and owned before the copy that fills them has landed. * A park releases whole pages. mem.seq_rm(id, -1, -1) in preempt_save_poll() removes every cell of the sequence, and a page belongs to one sequence, so no page is left half empty for find_slot's occupancy reconstruction to trip on. * The PREEMPTING and RESTORING windows are safe against the allocator. It rebuilds page ownership from live cells on every call, and both states hold live cells by construction, so a park still copying out and a restore not yet landed both read as occupied and neither can have its pages handed to somebody else. That is the same reason preempt_kv_used() counts both. * set_input_pages() cannot see a restoring sequence. It emits one page row per query token, keyed by that token's own seq_id, and a slot in either in-flight state is kept out of the batch by preempt_is_out(), so no other sequence can attend into pages whose contents have not arrived yet. * state_read_meta()'s whole-cache branch asserts exact mode is off. Preemption never takes it; it is per-sequence restore throughout. Left standing, and watched in the live runs rather than fixed here: exact mode rounds a sequence up to 256-cell pages while preempt_n_need() and preempt_kv_used() still count tokens, so the admission arithmetic gating a resume is optimistic by up to 255 cells per sequence. A restore that does not fit fails cleanly, as find_slot returns no slot, and the slot stays parked with n_preempt_fail incremented and retries. This is #194's own unvalidated page-aware admission accounting, now reachable through the resume path too.
…he cache A review of #194 pointed at the new GGML_ASSERT in llama_kv_cache::seq_cp, and it is right. Reproduced on this branch with the 4B on one B200: LLAMA_EXACT_CONCURRENCY=1, POST /completion {"n": 2} -> llama-kv-cache.cpp:458: GGML_ASSERT(!exact_pages || seq_id_src == seq_id_dst) failed, through server_context_impl::decode -> common_memory::seq_cp -> process aborted, the next request gets connection refused With the mode off the same request is served normally, so this is reachable by any client of an exact-mode server and takes every other request on the machine with it. Two changes: The server refuses the request. n_cmpl > 1 works by copying the parent's cells to a second sequence id, and exact mode gives a page to one sequence, so there is nowhere for that copy to land. Rejecting it where the task is built turns it into a 400 with a reason. The check reads LLAMA_EXACT_CONCURRENCY from the environment, the same way the KV cache, the batch splitter and the CUDA backend each do, because the answer is needed before a context exists and the mode has no other representation. The cache stops aborting. seq_cp, seq_add and seq_div log an error and return instead of asserting, so a caller this branch does not know about degrades to a refused operation rather than killing the server. The guards also move below the shared-cells early return, which the asserts sat above: a draft cache forwards these calls and copies nothing of its own, and it should not be judged by a rule about cells it does not own. After: the n=2 request returns 400 on both /completion and /v1/completions, the server stays up, and a following ordinary request returns 200.
A review of #194 predicted that exact mode and #184's preemption planner cannot both be right about how full the pool is, and on this branch they are not. Reproduced with the 4B on one B200, LLAMA_EXACT_CONCURRENCY=1, four chats with 1000-token prompts generating 2048 tokens each at --parallel 4 --kv-unified -c 8192 --spec-type draft-mtp, no forced-park knob, three rounds: 0 of 4 completions, every round, every chat ending in "Context size has been exceeded", with 0 parks and 0 restores. Nothing was ever parked. preempt_kv_used(), preempt_n_need() and preempt_kv_reserve() count tokens, and exact mode's allocator hands out 256-cell pages, one page to one (sequence, position / 256) pair. Four sequences can therefore be holding up to 1020 cells that no other sequence can be given, and the planner, seeing room in tokens that find_slot cannot find in pages, never reaches the threshold that would park anybody. The retry ladder then halves n_batch to 1 and ends every request, which is the pre-#184 behaviour that preemption exists to remove. Ask the memory how it allocates instead of assuming. llama_memory_i gains a non-pure alloc_granularity() defaulting to 1, so no module that allocates a cell per token needs a change; llama_kv_cache returns its page size under exact mode and 1 otherwise, the hybrid memory forwards to its attention half, and llama_memory_alloc_granularity() exposes it. The server reads it once at load and rounds: preempt_kv_used() charges every slot's tail page in full preempt_n_need() rounds what a resume must be given, since a restore takes fresh pages preempt_kv_reserve() reserves the cells the next step ADDS rather than its tokens, because on a rounded used figure a step is free until it crosses a page boundary and costs a whole page when it does, and that crossing is the only moment the pool can run out preempt_n_margin() rounds the asynchronous lookahead up to a page, since any of the steps it covers can cost one With a granularity of 1 every one of these is the arithmetic it was, so nothing changes with the mode off. After, same configuration and three rounds: 4 of 4 completions each round, 2 parks and 2 restores each round, no context errors, and P0 byte-identical to its solo run in all three. The same run with exact mode off is also 4 of 4 with 2 parks, unchanged from the parent.
ggml-cuda.cu is compiled for ROCm and for MUSA through the vendor headers, which rename every cuda* name it uses. The non-blocking event query added cudaEventQuery and cudaErrorNotReady, and neither header maps them, so both builds stop at an undeclared identifier while the adjacent cudaEventSynchronize has been mapped all along. hipEventQuery and musaEventQuery have the same signature and the same convention: success when everything recorded before the event has finished, hipErrorNotReady or musaErrorNotReady while it has not, which is exactly what the query reads them as.
…ously state_seq_copy_init() took any device advertising async and events, but ggml_backend_event_query() is optional: a device that does not implement it gets the generic fallback, which answers "is it done" by waiting for it. Metal, Vulkan and SYCL all advertise both capabilities and all leave event_query null, so they were handed a transfer object, told the caller the copies were asynchronous, and then blocked it for the whole copy on its first poll. That is the stall the transfer exists to remove, made worse by the caller no longer expecting it. ggml_backend_dev_supports_event_query() is the question the fallback hides, and state_seq_copy_init() now asks it. A device without a query is left out, so those backends get NULL and keep the synchronous llama_state_seq_*_data_ext calls they always used, which is the documented behaviour and is what the server already falls back to. The reason is logged once.
…s down destroy() resets llama_init and nulls ctx_tgt and ctx_dft, but the slots are declared after llama_init and are still alive at that point, and one of them can be holding a park or a resume that is still reading or writing KV tensors of the context being freed. release() already makes that wait for a single slot, on the path a cancelled request takes; nothing made it for all of them. The sleeping-state path is where it shows: /sleep calls destroy() and the server carries on running, so a copy issued an iteration earlier is left pointing at freed tensors and load_model() then clears the slots, running the transfer destructor's own wait against the same memory. Shutdown has the same hole with less time to notice it. destroy() now waits for every slot's outstanding copy and lets go of the transfers before anything is freed, which also means the next context does not inherit a backend and a host buffer belonging to the previous one.
preempt_n_margin() keeps eight decode steps of every running slot clear ahead of the pool filling, and the resume gate uses the same figure so that a slot is not put back into a pool it would immediately have to leave. It was not doing that for the slot being resumed. The candidate is still PREEMPTED while it is being considered, so the loop that counts running slots skips it, and the runway it needs appears only after it has been let in, at which point the pool is short by exactly that much and somebody gets parked. At -c 256 with a 1 + n_spec step and the eight-step runway, totals from 233 to 240 cells admit a restore that then cannot take its first step, and under load the same slot was seen restored and parked again five times over. preempt_n_margin() takes the number of slots that are about to be running as well as those that already are, and the resume gate passes one for the candidate. Everything else keeps the count it had. preempt_kv_reserve() already reserves a restoring slot's next step; this is the eight-step runway behind it.
… for llama_state_seq_copy_buf_is_pinned() returned can_pin, which is worked out from the buffer type the backend offers and is fixed for the life of the transfer. The header promises the buffer is page-locked. Those are different questions: the CUDA host buffer type is handed out whether or not pinning is available, and under GGML_CUDA_NO_PINNED its allocation falls back to an ordinary CPU buffer, so the server logged "pinned host memory" while every park ran through pageable memory. It was also true before any buffer existed and after buf_free(). buf_resize() already records which it got, by comparing the buffer that came back against the type that was asked for, so is_pinned() now returns that. llama_state_seq_copy_buf_can_pin() is the capability question, for a caller that wants to know before allocating anything. The load banner asked the capability question at a point where no buffer exists and printed the answer as though one did. It now says what the backend offers, in those words, and the first park reports what the buffer it allocated actually turned out to be.
Both issue functions validated only that a buffer existed. The caller's size was handed straight to the io object, which then validated every fragment against that number rather than against the allocation, so a save issued with a size larger than the buffer wrote past the end of it and a restore read whatever was next on the heap and sent it to the device. Unlike the legacy API the library owns this buffer, so it can simply check: a size of zero, or one beyond llama_state_seq_copy_buf_size(), is refused with a log line. The flags word had the same problem from the other end. LLAMA_STATE_SEQ_FLAGS_ON_DEVICE asks for the tensor data to stay in device buffers, and both functions built the host serializers regardless, while llama_state_seq_get_size_ext() with that flag reports a state without the tensor bytes in it. A caller pairing the documented size call with these ones sized a buffer for the metadata and then tried to fill it with the whole sequence. The flag is refused here and the restriction is written down in llama.h; the synchronous calls still serve it. tests/test-state-seq-copy.cpp covers both refusals in both directions, checks that a refused call posts nothing, that the same call at the buffer's own size still round-trips the sequence byte-for-byte, and that a transfer reports itself as pinned only while it holds memory that is. It skips itself where no backend can copy asynchronously.
The cudaErrorNotReady branch of the CUDA event query called cudaGetLastError() on the belief that the result had to be cleared. It does not: cudaEventQuery() returns cudaErrorNotReady as its return value without recording it in the thread's last-error state, so the only thing that call can collect is an error somebody else planted and has not looked at yet. Checked on a B200 with CUDA 13.1. An unrelated cudaSetDevice(99) leaves 101 pending; cudaEventQuery() on an outstanding event returns 600 and cudaPeekAtLastError() still reads 101 afterwards, so the cudaGetLastError() returned 101 and left the state clean. A real launch failure would have been thrown away the same way, and its owner would never have seen it.
ggml_backend_device_i gained event_query, so a device interface built against the previous header is one member shorter than the one ggml now reads. Every in-tree initializer was updated, but a backend loaded from a shared library is not: ggml_backend_reg_load_backend() accepts it on api_version alone, and a prebuilt .so still reporting 2 would have been let in and its iface.event_query read past the end of the object. Rejecting it is what the version is for.
The mode was gated on unified && offload && !v_trans && n_swa == 0 and never on where the attention layers actually run. Only the CUDA FLASH_ATTN_EXT reads src[5]; the CPU, Metal, Vulkan, SYCL, OpenCL and CANN kernels ignore it. So with -ngl 0, a partial -ngl, or a non-CUDA GPU the pool was still paged and the page table was still attached, but attention traversed physical cell order: the output stayed correct and neighbour and relocation independence were silently lost while the mode reported itself as on. Check the placement where the cache is built instead. Every KV layer must be offloaded and its device must belong to the CUDA family backend, which is also built as ROCm and MUSA and carries the same paged kernel; anything else fails the load naming the layer and the backend it landed on. As a second line, the FLASH_ATTN_EXT of every backend that would ignore the page table now refuses an op with src[5] set, so a scheduler decision made after the load cannot route it somewhere that walks the pool in physical order. The CPU is deliberately left accepting it and says so in place: it is the reference test-backend-ops compares the paged CUDA kernel against, and that test builds a mask which selects exactly the listed cells. The four remaining preconditions were one bare GGML_ASSERT each, so a quantized KV cache, an SWA model, a transposed V cache or a -c that is not a multiple of 256 aborted at model load without naming which one failed. Each now logs what it needs and which flag sets it, and the load returns an error the way every other KV cache failure does. The context size check also moved after the shared-source override, so it tests the size the cache is actually built at. -ngl 10 with LLAMA_EXACT_CONCURRENCY=1: llama_kv_cache: LLAMA_EXACT_CONCURRENCY is set but layer 0 keeps its KV cache on CPU, which has no paged attention: every layer must be offloaded to the CUDA backend (pass -ngl to offload all layers and do not pass --no-kv-offload)
get_can_shift() still returned true under exact_pages, so --context-shift and --cache-reuse N passed every startup capability gate and then aborted the whole process on the first seq_add with a nonzero shift. Both are opt-in flags, so the default was safe, but llama-server accepted them silently and died on the first request that needed them. The server already disables both at load for a cache that cannot shift, with a warning, so returning false there reuses that path: srv load_model: ctx_shift is not supported by this context, it will be disabled The remaining aborts are reachable the same way, from one request parameter or one API call, and abort() is not an acceptable answer to either in a network server. seq_cp between two different sequences, a nonzero seq_add and a seq_div now log which transformation was refused and on which sequence and return without touching the cells, and the whole-context branch of state_read_meta() logs and returns false the way the two failure paths next to it already do, so llama_state_load_file() reports a recoverable error through an API that is designed for one instead of killing the process. The page invariant is protected exactly as before: none of these paths can now run and leave a cell outside the page its position belongs to.
Two things were wrong with the one-sequence prompt ubatch rule.
It only existed in the hybrid memory. A dense transformer with a unified cache
takes split_simple, which packs every sequence's prompt tokens into one ubatch,
so its prefill matmuls ran at a width the solo run never sees and, with the
column policy bounded, produced K and V the solo run never produces. Exact mode
was therefore not exact by construction on dense models, which is most of what it
will be pointed at. A pure recurrent model still called the three-argument
split_equal for the same reason the hybrid one no longer does. Both now take the
rule: llama_kv_cache::init_batch keeps split_simple for a plain decode step and
switches to the sequence-set split when a prompt is present, and
llama_memory_recurrent::init_batch passes the flag through.
The rule itself then serialized more than it had to. has_multi_token_seq()
scanned the whole original batch and ignored used[], so it stayed true after the
prompt had been consumed, and the n_seqs_max cap it fed capped every sequence set
including one-token decode sets. One prompt chunk plus three decodes therefore
became four single-sequence ubatches and the three chats decoded one at a time
for the whole prefill, which contradicts the comment saying a plain decode step
stays batched. The predicate now skips used[] tokens, and the cap became an
isolate_multi_token_seqs flag: a sequence set with more than one token left to
place takes a ubatch of its own, sets with one token left keep grouping. One
prompt next to three decodes now costs one extra ubatch, not three.
The KV cache constructor also read getenv("LLAMA_EXACT_CONCURRENCY") directly
while llama_exact_concurrency() and ggml_cuda_exact_concurrency() each cache the
first value they see, so a process that created one context with the knob unset
and then set it got a paged cache on top of a dispatcher still in default mode.
It now reads the same cached value as the other two.
The fixed default of 16 silently turns exact mode off above 16 columns: MUL_MAT, MUL_MAT_ID and the per-row attention split all fall back to the neighbour dependent batched path. --parallel 6 --spec-draft-n-max 2 gives 18 columns and --parallel 8 gives 24; both are ordinary server configurations and neither said anything. The source comment documented the cliff, nothing at runtime did. The bound only ever had to cover the widest ubatch a decode step can build, since a prompt ubatch holds one sequence and gets its exactness from that. So let the caller report that width. ggml_backend_cuda_set_exact_decode_width(), reachable directly or through ggml_backend_reg_get_proc_address(), takes one column per slot times one plus the draft length, and exact mode defaults the bound to it. common computes it from n_parallel and the speculative type and reports it before the warmup, which is the first graph any of these tools computes, and refuses at startup an explicitly set GGML_CUDA_BATCH_INVARIANT_MAX_COLS that is smaller: GGML_CUDA_BATCH_INVARIANT_MAX_COLS is 8 but LLAMA_EXACT_CONCURRENCY needs at least 12 to cover a decode step of 4 slots, above which a matmul is left batched and its rows depend on the other rows in the ubatch. Raise it to 12, set it to 0 for no bound, or unset it to let it default to 12. --parallel 4 with speculation off now defaults to 4 rather than 16 and with --spec-type draft-mtp --spec-draft-n-max 2 to 12 rather than 16, which is the same guarantee over a narrower range of shapes: a decode ubatch of that model cannot be wider than that, and everything above it is a prompt. When nothing reported a width, the default stays 16 and the dispatcher warns once per process the first time a MUL_MAT or MUL_MAT_ID above the bound is left unsplit, naming both numbers. It deliberately stays quiet once a width is known, because then the only batches above the bound are prompt ubatches and warning on those would fire on every prefill for a case that is working as intended.
Page tables are wired into llm_graph_input_attn_kv only. llm_graph_input_attn_k has no self_pages member and its build_attn calls build_attn_mha without a pages argument, as do the DeepSeek sparse and sliding window variants. A model on one of those layouts still got its cells placed in pages by the allocator and then attended in physical order after a park and a restore, so the mode reported itself as on and lost the one invariant it exists for. That is the same silent failure the CUDA placement gate was added to stop, so answer it the same way: log which layout it is and fail the context. Rejecting is the smaller correct change of the two. Wiring self_pages into llm_graph_input_attn_k is four lines and looks tempting, but it fixes one of the four V-less input classes and DeepSeek 3.2 uses two of them: its sparse layers build their mask from a top-k selection and would stay unpaged, leaving the model half paged, which is worse than refused. None of these architectures was measured here, and the paged kernel also requires 256-dimensional K and V heads, which none of them was checked against. Reaches the user through the path llama_init_from_model already has for a context that cannot be built: llm_graph_reject_exact_concurrency: LLAMA_EXACT_CONCURRENCY is set, but this model uses the V-less KV (attn_k) attention layout, which carries no page table and would attend in physical cell order llama_init_from_model: failed to initialize the context: exact concurrency: unsupported attention layout
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…ansfer takes the fences with it The wait for copies in flight before a context shift is applied ran just before the target decode, but pre_decode() had already asked the draft context for its draft, a decode that applies that cache's pending shift in place while a park or restore may still be copying draft cells. The wait now follows update_preemption() and precedes pre_decode(), so both caches shift after the copies have landed. When the pinning probe found pageable memory the server gave its transfers up, but the fences a transfer installs stayed with the context and were recorded after every decode for nobody. The context counts its live transfers now and the last one to go frees the fences.
…aft; the last transfer takes the fences; the resume order read on every load
… resident cycling past the rotation delay
… resident cycling past the rotation delay
… resident cycling past the rotation delay
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…ntext; the recurrent flag assigned on every load
…he shift wait runs before the decode as well synchronize() covers the graph backends, not the copy backend a transfer owns, so a context freed while a transfer was still copying could free the KV buffers under it, and freeing the transfer afterwards touched the dead context. The context keeps the set of its live transfers now: at teardown each is waited for and disowned, and its own free then touches nothing of the context. The wait for copies in flight before a shift is applied runs before pre_decode(), for the draft, and again before the decode: a shift that --cache-reuse asks for is found inside pre_decode(), after the first wait, and the decode applies it in place like any other.
…ntext; the recurrent flag assigned on every load
…ardown; the shift wait before the decode as well; the resume order on the context; the recurrent flag assigned per load
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…at/server-side-preemption-async
…at/server-side-preemption-notify
…ted by the cells it holds, trimmed before a resume
…nto feat/server-side-preemption-integration
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Comment-only pass over the PR's diff: collapse the long explanations to one or two lines each and drop the ones the code already says.
A slot just given a task has not yet passed the prompt checks the STARTED block runs, and the planner could park it first. The park notice opens the stream, so a prompt the checks reject came back as HTTP 200 with an in-stream error where the non-stream 4xx belongs. The checks are one helper now, slot_prompt_rejected(), run by the STARTED block as before and asked by the planner before a started slot can be chosen: a request about to be errored is never given a notice ahead of its error.
…feat/server-side-preemption-integration
…ng hunks keep this branch's text
Member
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 474569ca31
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+949
to
+952
| if (exact_pages && balloc.has_shared_tokens()) { | ||
| LLAMA_LOG_ERROR("%s: exact concurrency does not support tokens shared by several sequence ids; " | ||
| "give every token exactly one sequence id\n", __func__); | ||
| break; |
There was a problem hiding this comment.
When exact mode receives an otherwise well-formed batch containing a token shared by multiple sequences, this break returns a LLAMA_MEMORY_STATUS_FAILED_PREPARE context; llama_context::decode() maps that status to 1, which the public API documents as a transient KV-capacity failure rather than -1 for an invalid batch. Direct API callers may therefore evict cache entries, reduce the batch size, or retry indefinitely even though this batch can never succeed in exact mode. The equivalent hybrid guard has the same status-mapping problem, so reject this during batch validation or propagate a distinct invalid-input status.
Useful? React with 👍 / 👎.
Cuts the comments this branch adds by about 70 percent, keeping the invariants, the citations and the traps and dropping the restatements, banners and measured asides. Comments only; no code changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #184. This branch merges #190 (park notices on the stream), #192 (park and restore copies off the decode loop) and #194 (opt-in exact concurrency) so they can be tested together, plus the fixes that only showed once they shared a binary. It is rebuilt from the three branches' heads as they move, and every fix found here has been carried back to the PR it belongs to, so it now arrives through the merges rather than sitting on this branch alone.
Rebuilt over
75944db80from the current heads:1c5aa4904feat/server-side-preemption-notify(#190,5a791e03c)51eca0837feat/server-side-preemption-async(#192,888603d03)70d49bfacfeat/exact-concurrency(#194,da8556d31)ef257d7b8feat/exact-concurrency(#194,c6c3cb671): speculative verify batches stay grouped, sliced column splitabccc6e95feat/exact-concurrency(#194,379ca5d42): one launch per expert projection under the knob instead of one per tokena7db6262afeat/server-side-preemption(#184,50b617ae4): parked slots come back head of line by park time3f817d880feat/server-side-preemption(#184,8057a74af) andfeat/exact-concurrency(#194,f4e45646d): resume-order knob logged at load; equal-width grouping under exact modee8ab02c94feat/server-side-preemption(#184,2a7e277ab): a parked sequence that cannot fit the pool alone is failed instead of parked for ever; margin waived when nothing is residentdb9558696feat/exact-concurrency(#194,adeed654b): head widths other than 256, tokens with several sequence ids, and hybrid cross-sequence edits refused by name under exact mode70b874c33feat/server-side-preemption-async(#192,01dfecc5e, carrying #18484311fd3dand6dbc4e7b6): when the KV-full retry ladder runs out, resident slots are rewound to the cache boundary and the smallest parked instead of every slot getting the context error; a batch holding a draft is never narrowed through the groupa7e661c14feat/exact-concurrency(#194,c89d87db5): the same, with the page-rounded margin1130aed0b47ef8de54and #1946573242a1(#18486845c15e): a batch holding a draft is narrowed the old way when there is no budget to park intob89302b25feat/server-side-preemption-notify(#190,5864dae95): a park made as a last resort sends the stream comment the planner sends, so Studio shows the pausef8ec5665afeat/server-side-preemption-async(#192,cd1cd4e6d): the lookahead margin counts a slot being restored5e0f99aedfeat/exact-concurrency(#194,0c9fc0ed8): every context reports the widest decode step it can build, so the column bound covers direct library users3fc833954301f480c0, #1920303888e8, #19408e10dabb(#184662ec2029): the planner and the last resort step aside for a context without memory7c8a6d51576baf1a68, #192cac5ee6c0, #19467063e476(#184a9b712eee): unlimited budget enables the last resort; per-slot draft bound for restores and reservations; shared prompt charged once per family; reused slot charged its prefix; context shift before the planner measures800e166ad7fb42b582and #19446e7fa742: parked prompts of a multi-prompt stream kept apart; ngram-cache verify width; DFlash refused by name; an explicit bound below the decode width refused at context creation; width replayed to backendseb428268c53c59a109, #1928d74efbb2, #194b7ace528f(#18464a5064ef): a resident cycling through context shifts takes turns with a parked head9391aa71e356978a26, #19242e536970, #194dbd82ca5e(#1846744b3d9f): a non-causal context with a cache refused at creation under exact mode; one page boundary per waiting prompt slot in the reserve cap; a stream parked before its first token starts with the notice; the rotation park announced; a reused slot trimmed to its shared prefix before it is sized or parked; a round with a context shift waits for every copy in flight; the decode width of every context follows the token figure645d406dd3306d4b50, #192ad3856ae9, #194d4e3fc8a6(#184a1c34dad2), on top of6af0d4e99: a reused slot's need counts from its shared prefix and its trim is safe for memories without partial removal; a waiting child is charged its own cache; equal-count grouping stays on for recurrent and hybrid memories; a width the explicit column bound cannot cover is refused; a restore landing inside the context-shift wait sends the resume notice276dd25296fb0b91c5:LLAMA_SERVER_PREEMPT_POLICY, a test knob to compare victim choices on one workload (off unless set)ecdb0f5317efef20fa, #1922ba0dcf5f, #194bf00ac38a(#184a7a04c2a9): the rotation parks the resident that lets the head in (page-rounded here); soft-capped attention refused at load; width reports serialised and monotonic in the backend; the token figure never lowered089fedd923d179fd0b, #19202a3e11d4, #19498fe86dfc(#18464a3f6e7d): a parked slot, or one whose copy is in flight, survives an aborted round; a rotation counts the head's bytes as leaving; the exact setup runs before any context exists; one lock for the token figure, the sequence count and the widthd72e2a40da06419d00, #192e432b15c2, #19449c66c5bd(#18455f04bbd6): the leader measured by what a reused slot keeps; no rotation while a park is copying and an asynchronous rotation park re-examined when it lands; the rotation budget under asynchronous copies and buffers returned over budget; the cache-reuse shift waits for copies in flight; an isolated ubatch takes only sets that finish in it; page-aware margin at every step testb4a5f18d82b4a69122, #1942f2258dc0: a whole-context restore under exact mode is refused before it can clear the cache; the width is published once construction succeeds; the page table is refused on every backend that ignores it; DSpark refused with DFlash; a transfer that fails part way posts no copies; pageable host memory parks synchronously; no transfer for state that is not on a device; the runway rounding explained48f7d7c0b1d528856f, #192faa3dd3f7, #19423c9dc2ef(#184270fdd6f3): test that a parent and child alone in the pool get the context error and the server lives0611fd99ba4b62f2be, #194918a8bf4f: a park buffer that comes back pageable parks synchronously from then on; the graphs after a restore wait for its copies on the device; the exact-mode cache asks the device whether it runs the paged attention for the layer rather than trusting the backend's namef737e6d49294d2a912: idle parked RAM is given back when another slot needs to park, so a budget that holds one sequence is not spent for good by the first restore; the copies of a park or restore wait for the compute stream on the device instead of draining the hostbf0f731273800ddee9through #190a158200deand #192dffa102e3: a pure recurrent cache is served without preemption; a rotation holds both states at once, so the RAM cap is asked for the resident's state in full and a budget that holds one sequence but not two does not rotate, said once per parka140e80c881eec0bb3through #190ce02834c4and #192a5749d11b: the copies wait for a fence the context records after every decode, so restores issued in one pass run independently on the device; a started slot's reservation counts from the prefix it keeps (page-rounded here, the step starting from that prefix)e36a4b6ddaf560907bthrough #1905ecdba815and #192a56d49e1f: the context shift and the planner run inside the guarded part of the step, so a failure there ends the affected slots rather than the loopb0fae413a00b27d2df, #1904918294ff, #1928cb58dfdfplus 4b91744 here: what a started slot keeps is decided by the batch builder's rule (no caching, aLoRA cutoff); a parked stream keeps a shorter ping it asked for; staging counted by what the buffer charges; fences after the layout check; transfers only where a park can happen; a decode that fits goes ahead beside a park in flight instead of waiting for it; a slot restoring into the prompt phase counts in the page-boundary cap0d39b88baad89538d2, #190801e29ad1, #192114e230ab: the resume order read on every load; the shift wait runs before the draft is asked for, so the draft cache shifts after the copies have landed; the last transfer takes the fences with it7fdb58bb3301caa31d, #190929809ffa, #192b1a31b42d: the resume order belongs to the context and the recurrent flag is assigned on every load; the shift wait runs before the decode as well as before the draft; a context freed with live transfers drains and disowns them1bf8519163c99fafdf, #190a33c56397, #19258c6e3329: the pool is measured by the cells each slot physically holds (page-rounded here), and when nothing fits every started slot is trimmed to the prefix its request keeps before idle slots are cleared, so a resume is not attempted against cells the batch builder has not yet released474569ca390a5094d3and #194b4b0f9bd7: a started slot's prompt is validated before it can be parked, so a request the STARTED block would reject is never given a notice ahead of its non-stream error; shorter comments from #194, the conflicting hunks keeping this branch's textf8fdf063aThe four fixes below now come in with those merges. Two conflicts had to be resolved by intent rather than by text: at the KV-full park site the notice and the new host-memory report both belong, in the order the forced-park site already uses; and #194's synchronous planner meets this branch's asynchronous one, where the resolution keeps the asynchronous margin and its
RESTORINGreservation and takes #194's compile-time-checked rounding helpers, so the two planners share one set of arithmetic and the synchronous branch of the margin rounds as well.Summary
Three merges and four fixes. Merging #190 into #192 needed intent rather than text: git placed two of the three notice sites after the asynchronous early-outs, where an asynchronous park or restore would have announced nothing. The rule applied is that a park is announced when the slot stops taking part in a decode, on entering the copy-out state, and a restore when it starts again, when the copy back lands; the keepalive therefore spans all three in-flight states. #194's page allocator and #192's park compose without change: a park removes whole pages, both in-flight states hold live cells, and a restore allocates through the page branch of the slot finder.
Fixes found here
llama_memory_i::alloc_granularity()now reports the page size and the server rounds its four planner figures by it. After: 4 of 4, two parks per round, the parked chat's bytes unchanged.nabove one reached a cache assert under exact mode and killed the server. It is refused with a 400 at task build, and the cache logs and returns instead of asserting.--sse-ping.Results
Qwen3.5-4B, two MTP drafts,
-c 8192, four slots, seed 0, temperature 0, forced parks every 64 tokens, all three features on, re-run on the rebuilt branch: 4 of 4 in three rounds, 28 parks and 28 restores per round, seven park and seven resume notices on every stream, and the observed chat byte-identical to its solo run in every round, at the same sha2563922db5817b706aa...the branch gave before the rebuild. Survivor stall at a park 52 to 67 ms and at a restore 39 to 85 ms with exact mode on, against 76 to 78 and 57 to 64 with it off, on a shared GPU.Qwen3.6-35B-A3B through Unsloth Studio on an unshared GPU: exact mode loads with Studio's own launch args; a chat's bytes are unchanged by three different neighbours across three rounds and by seven forced parks per stream, while the same cell with the mode off diverges at byte 488; four API chats and eight GUI chats in Chromium and WebKit all completed with zero errors and every one of 21 natural parks lasting up to 96 s resumed with the paused label shown.
Cost
Re-measured on this tree, interleaved: three on/off pairs, four chats, prompts of 937 tokens as the server counts them, 2048 tokens each with
ignore_eos,-c 8192, four slots, unforced, medians of three.70d49bfac70d49bfacef257d7b8ef257d7b8All six runs finished 4 of 4 with parks and restores and no context errors. The 37 to 42 percent this paragraph used to quote was measured on #194's reviewed head and is gone with speculation off: #194's own fixes stopped one prompt serialising every concurrent decode for the whole prefill and dropped the default column bound from a fixed 16 to the width a decode step can reach. What is left there is the CUDA column policy rather than the page bookkeeping, which #194 measures at 0.79 for the kernel half alone against 0.92 for the whole mode.
The speculative cost at
70d49bfacwas the batch splitter, not the column policy: under exact mode it isolated every sequence set with more than one token left as a prompt, and a slot's three-token verify batch is such a set, so each decode step ran the whole graph once per slot. #194'sc6c3cb671isolates by width (llama_set_exact_decode_tokens, one plus the draft length) and slices the column split into batch-of-one-equivalent launches, which is the 0.45 to 0.74 in the table, measured on the #194 tree and carried here byef257d7b8. What remains is the sliced column policy on a twelve-wide verify batch plus paged attention. One B200 shared with another tenant throughout, so the ratios are the result and the absolute figures are not.Tests
Server harness 28 of 28 (22 preemption, 6 notify), the fragmented-restore and sequence-copy ctests on CUDA with the 4B, and
test-backend-ops5234 of 5234 in exact mode.