v1.23: the pull requests' revision of 5 November

A review of #65283 and then a fresh one of both found stalls on the real scheduler, where a request that waits for what it holds back stops the engine: a request whose KV load is done, behind a request that fails its allocation, and a running prefill scheduled nothing. The serving profile loads a call's turn from the CPU tier, so the engine could reach the first. Both pull requests are revised, each with regression tests that fail before the fix, and make-room's patches carry them.

What changed since v1.22

• The budget cap bounds every request whose first tokens are due by the steps it computes nothing: at most T in a row, which its progress resets. A load in flight keeps its T steps from its start. A request a higher priority's hold keeps out does not count those steps, and a running request with one token left is no longer held.
• The KV preemption's gate waits only on a higher-priority prefill that gets a chunk in the step, so a KV load in flight or a prefill scheduled nothing keeps no request out, and room is made for it again if it needs more.
• A search for stalls of this kind (the record) runs 400,000 random workloads on the pull requests, and 200,000 on the module with the serving profile's switches, each with the options off and on. None stalls because of the options, where the same search finds 241 and 153 on the code before. A state that vLLM's main cannot leave either remains, with nothing running.
• The pull requests are rebased on vLLM's main at 0b3a608dd, where #54625's fields landed beside make-room's, and make-room's six patches, the module's pin and compat-upstream carry the new heads.
• make-room's trace against the module the evaluation ran: with the budget cap off, the same but for 98 of the 320 configurations with a KV connector, where the gate's change shows; with it on, different in 1,080 of 1,264. Against v1.22's module, the revision changes 426 of 2,036 configurations, 329 of them with a KV connector.

v1.22: the pull requests rebased on vLLM's main

#65282 and #65283 are rebased on vLLM's main after two changes landed there: #65319's queue of requests waiting to start an async KV load, and #65300's weak reference from a KV cache block to its pool. Neither is in v0.32.0 or v0.33.0, so the backport's code and the module stay as v1.21's.

What changed since v1.21

• The pull requests dereference a block's pool in the count of the blocks preempting frees, and take the requests waiting to start an async KV load into make-room's target scan, its in-flight gate, the budget cap's scan of lower priorities and its waiting hold, each with a test that fails without it.
• The backport's commits are remade with the same trees on the fork's v0.32.0- and v0.33.0- branches: their Backport paragraphs say how the releases differ, and make-room's six patches change in their headers alone.
• compat-upstream names the rebased heads' new lines as adapted under those commits.

v1.21: the pull requests' revision of 3 November

A fresh review of #65282 and #65283 found cases where the KV preemption could make room a request would not get, and where the budget cap could hold prefill back without end or for a request it could not help. Both pull requests are revised, each with regression tests that fail before the fix, and make-room's patches carry the revision.

What changed since v1.20

• The KV preemption bounds the waiting request's prefix-cache hit as the waiting loop does (prompt scoring's limit and the aux-output connector's), and in the step it makes room, a lower-priority request that holds KV blocks but no reservation for the rest, such as a streaming session's next chunk, waits instead of taking the room.
• The budget cap holds for a waiting request that the waiting loop does not admit for at most T steps, never holds a request with one token left, lets a held request's KV load start, holds a lower-priority request that already holds KV blocks only while a request of its priority or a lower one runs, counts a step in flight as one token, and keeps its encoder exemption to a prefill midway through a cached input while a request it is held for needs encoder input.
• The variant counts the admissions the budget cap holds at both of its hold sites, adds an announced turn's priority 0 to the hold beside the new holders, and counts a step in flight as one token in its decode hold. The fork's v0.32.0- and v0.33.0- branches carry the revision, and make-room's patches are written from them.
• The module built on 0.32.0 schedules as v1.20's wherever the budget cap is off (make test-trace against the module Phastform's engine built on 2026-10-27: none of 772 configurations differ) and differs with it on (1,106 of 1,264); the serving profile pins its new hash.
• The report, version 1.21, gives the revised rules in §3.1, §3.2, Algorithms 1 and 2 and §3.4, and §4 discloses that the evaluation predates them.
• The probes the pull requests cite, rerun on the revision in one session with stock vLLM as the control (benchmarks/results/2026-11-04-r3-probes): the KV case's first token after 284 ms against 4,976 ms on stock, and the prefill case's 16th token after 560 ms against 1,374 ms, as with the earlier code.
• The packages are versioned 1.21.0, and are not published to a package index.
• Upgrading from v1.20. A deploy drains and restarts the engine, whose patches changed; it builds the new module at its start, and serving-check expects the new hash.

v1.20: no hold on a prefill whose encoder outputs are cached

A review of #65283 found that its budget cap could stall a multimodal model. A lower-priority prefill midway through an image, whose encoder output filled the encoder cache, was held back while the higher-priority request it was held for waited for that space, which only the held prefill's progress frees. #65283 now holds back no prefill whose encoder outputs are cached, and make-room's patches carry it. The voice agent's model takes no encoder inputs, so the engine schedules as v1.19's.

What changed since v1.19

• The budget cap holds back no prefill whose encoder outputs are cached: #65283's commit 8c753658a, with a regression test. The fork's v0.32.0-priority-prefill-hold and v0.32.0-flatshare-make-room branches and their v0.33.0 counterparts carry it, and make-room's patches are written from them.
• The module built on 0.32.0 differs from v1.19's by that clause alone, and make test-trace against the module Phastform's engine built on 2026-10-27 finds every step the same; the serving profile pins its new hash.
• The report, version 1.20, gives the bound in §3.2 and Algorithm 2.
• The packages are versioned 1.20.0, and are not published to a package index.
• Upgrading from v1.19. A deploy drains and restarts the engine, whose patches changed; it builds the new module at its start, and serving-check expects the new hash.

v1.19: the change's pull requests as #65282 and #65283

In Flatshare v1.19 the change is proposed to vLLM as #65282 and #65283, which succeed #63350 and #63351: the same change, rebased onto vLLM's main as one commit each. make-room's patches now come from the new pull requests and build v1.18's code. No measured number changed.

What changed since v1.18

• The pull requests. #65282 carries the KV preemption and #65283, stacked on it, the budget cap; #63350 and #63351 are closed, each with a pointer to its successor. Every link and every statement of their state names the new ones, open with no maintainer review as of 2026-11-03, while the records, tables and figures keep the first numbers, as the glossary says. ARCHITECTURE's §2 is "The change", without the numbers.
• make-room's patches come from the fork's new backport branches, v0.32.0-priority-kv-preemption, v0.32.0-priority-prefill-hold and v0.32.0-flatshare-make-room, and their v0.33.0 counterparts, with each pull request's part one commit; they keep their names, 63350.patch and 63351.patch. The module they build on 0.32.0 and on 0.33.0 has v1.18's code, its syntax tree equal but for docstrings, and make test-trace against the module Phastform's engine built on 2026-10-27 finds every step the same. Its comments, docstrings and header differ, so the serving profile pins its new hash.
• make compat-upstream checks #65282 and #65283 against vLLM's main, each with its patch (PATCH), and its check of the backport against the pull requests' heads names the new backport commits.
• The report, version 1.19, names the new pull requests in its note, §4 and references [9] and [10], and corrects §2.4, which said a voice reply needs a few dozen steps before it can be spoken: its first sentence takes a step for each of its tokens, 15 at the median and 39 at most (§6.3).
• The packages are versioned 1.19.0, and are not published to a package index.
• Upgrading from v1.18. A deploy drains and restarts the engine, whose patches changed; it builds the new module at its start, and serving-check expects the new hash. The engine schedules as v1.18's.

v1.18: the documents made current and consistent

Flatshare v1.18's documents, as GitHub renders them, follow one form and say where v1.17 stands: the README, the report, the method, operations and decision documents, and the packages' and the records' READMEs. No measured number changed, and the code that serves is v1.17's.

What changed since v1.17

• The README gives each further result one bullet with its section of the report, sets the demo figure after the opening, and puts its Results table's units in the header, so that no cell wraps. Its limits give v1.17's decode hold (beside 24 sessions, 0.8–5.1% of the replies over the SLO in two rounds), and its code blocks fit the page.
• The report, version 1.18, corrects Appendix A, which said BENCHMARKS lists every measurement: it lists the campaigns' measurements, and the records' READMEs the probes and checks around them. Its tables' empty cells are em dashes, as every other document's are.
• BENCHMARKS and the method documents give each question one layout: provenance, how it ran, figure, caption, table, findings. The tables BENCHMARKS shares with the engine points' and the rounds' records set each median over its range, as those records now print them too. PRIOR-ART says the configuration to deploy ends 0.20 s over the SLO, where it said within 0.2 s, and that vLLM 0.33.0 includes #61976 and #54627.
• Every table and code block fits the page at github.com's width, as Chrome measures GitHub's rendering of each file: BENCHMARKS' costs to the agents by mechanism and the follow-up rounds' are two tables each, the soak's reads down, and the FP8 crash's runs are two tables, which summarize.py prints and a test holds to the runs' files; long commands are cut into continuation lines and their comments wrapped, the commands themselves unchanged. The tools' own summaries stay as their tools write them.
• The records' READMEs share one skeleton, and what they say in the present tense is dated or brought up to date: vllm-project/agentic-api#478 merged on 2026-10-26, vLLM 0.33.0 is out and make-room ported to it, and vLLM #64165 is open as of 2026-11-02.
• The packages', operations' and decision records' documents share one skeleton and name make-room's two vLLM releases and the profile's decode hold. flatshare-sim's two tables that ran off the page are two each, and ADR-0006 and ADR-0009 gain dated notes under Later.
• A test holds every relative link of the repository's Markdown, its images and anchors too, to its target as GitHub resolves it.
• The packages are versioned 1.18.0, and are not published to a package index.
• Upgrading from v1.17 needs no change on a host: the code that serves and the serving profile are v1.17's.

v1.17: the decode hold in the serving profile

Flatshare v1.17's serving profile holds the agents' decodes out of a voice reply's first 48 tokens, the budget cap's window, for the voice tail. In two follow-up rounds beside 24 agent sessions the hold put 16 of 720 replies over the SLO against 55 without it, fewer in five of the six rounds and caller counts, for 3.8% of the agents' token rate at the median: past the 3% a rule fixed before the rounds allowed, so that v1.15 left it off. v1.17 takes it for the voice tail.

What changed since v1.16

• The profile. decode_hold_tokens goes from 0 to 48 in rtx-5060-ti-serve.toml, and with it serving.json, the make-room-serve variant, the README's quick start and PRODUCTION's row; serving-check holds the engine to it.
• Its records. The quick start on Phastform with the hold: make-room answered the voice turn in 173–179 ms against 1,543–1,556 ms on stock vLLM, where without the hold it took 272–277 ms (the record). On vLLM 0.33.0, run again: 167–207 ms against 1,778–1,788 ms, where without the hold it took 264–267 ms, and the serving variant's own engine 170–175 ms against 208–211 (the port's record). Each record's test now holds the switches its engine logged to the commands it gives.
• The scheduler's cost a step comes from the baseline measured for this configuration, at a tolerance of 4%: the step the hold keeps clear costs 63 µs less, and a step with only lower-priority work running 48–182 µs more, against 33–151 µs without the hold (REPORT §4, Figure 7). On peacock the check passes in all 17 states, and its negative controls fail by 17–88%.
• The docs. REPORT §6.10 and §7, BENCHMARKS, GLOSSARY, ARCHITECTURE and the engine points' and follow-up rounds' records say the profile holds the decodes, and the follow-up rounds' verdicts stand as measured. Beside 12 agents the five additions take a call's first token from 354 to 88 ms at p95 and its whole reply from 1,226 to 722 ms, for 4% of the agents' token rate (Table 9).
• The packages are versioned 1.17.0, and are not published to a package index.
• Upgrading from v1.16. A release renders the engine's new flag, --additional-config.make_room.decode_hold_tokens 48, and serving-check expects it. An operator who prefers the agents' rate sets decode_hold_tokens = 0 in the profile and renders it again (make profiles UPDATE=1).

v1.16: make-room on vLLM 0.33.0

Flatshare v1.16 ports make-room to vLLM 0.33.0, released the same day: the change builds and passes its suites there, and the quick start's probe and the serving variant's engine ran on the card. The serving profile stays on 0.32.0, which the rounds measured.

What changed since v1.15

• make-room on vLLM 0.33.0. make-room/v0.33.0 holds the three patches: the same 26 commits cherry-picked onto v0.33.0, on the fork's v0.33.0-* branches. Two needed resolving, against 0.33.0's local prefill cadence and its cap of the external hit for auxiliary outputs, and by patch ID every other commit's change to the scheduler is its source's. build.py takes the installed vLLM's release from its metadata, and a source checkout's from the one release whose patches apply to it (--release names one).
• Its checks. The build's tests (make test-vllm VLLM_REF=v0.33.0), the behavior suite, the serve module and the trace against the pull requests' diff on 0.33.0's real scheduler (make test-make-room-ci VLLM_REF=v0.33.0, 570 and 1,081 passed in requirements-cpu-v0.33.0.txt's environment), and make compat-upstream, which now holds a ported release to its own patches. On Phastform the quick start's probe gave the voice turn's first token in 264–267 ms with make-room against 1,779–1,792 ms on stock 0.33.0, and the serving variant's engine ran there, serving-check finding only the release and the module changed (the record).
• The packages are versioned 1.16.0, and are not published to a package index.
• Upgrading from v1.15. No configuration changes: the serving profile still runs vLLM 0.32.0. An operator who moves to 0.33.0 installs it and builds make-room as before, and build.py takes 0.33.0's patches.

v1.15: the serving engine optimized

Flatshare v1.15 optimizes the serving engine: what it adds to v1.14's serving profile, each piece chosen on the box from its scheduler's traces and measured against the configuration before it. Beside 12 agents a call's first token goes from 175 to 64 ms at the median and from 354 to 92 ms at p95, and its whole 48-token reply's p95 from 1,226 to 998 ms. In five rounds of the whole pipeline the four additions cut the replies over the SLO beside 24 sessions from 18.6%, 18.5% and 18.0% to 5.4%, 6.4% and 5.8% with two, four and six callers, fewer in every round, and beside 16 sessions from 7.5% to 3.4%, for 1.6–4.9% of the agents' token rate. Two more rounds turned down a fifth, the agents' decodes held, and speculative decoding, by rules fixed before they ran.

What changed since v1.14

• The calls' prefixes kept on the GPU. A finished call turn's blocks stay referenced until the conversation's next turn takes them (make_room.retain_tokens, retain_s), counted in distinct blocks, as LiveKit's preemptive generation retains a cancelled request beside its replacement.
• The next batch just in time. The serve module schedules the next batch when the step in flight is due to end, by the card's step-time model, so a call arriving during an agent's long step joins the next batch.
• Each turn announced. The voice worker sends a realtime request for one token when the user stops speaking, and the budget cap holds the agents' prefill back for 400 ms from it.
• A call never waits behind an agent's reservations. A call's turn could wait 1.4–4.5 s while agents' prefix loads held the pool's free blocks in reservation: the load admission counted every in-flight reservation, and the waiting loop stopped at the first completed load it could not admit. With make_room.take_reservations a request counts only the reservations of its own priority or higher, and make-room preempts for the rest.
• What was measured and not adopted. The agents' decodes held through the budget cap's window (make_room.decode_hold_tokens 48): in the engine points it took the 48th token's p95 from 998–1,016 to 722–735 ms, and in two follow-up rounds it put 16 of 720 replies over the SLO against 55, but for 3.8% of the agents' token rate at the median, past the 3% its rule allowed. Speculative decoding on top, EAGLE-3 with two drafts and the calls' short loads recomputed (make_room.recompute_tokens), put 12 over against the hold's 16, fewer in three of the six rounds and caller counts where its rule asked for five (the follow-up rounds). Both stay switches, off. Nor were synchronous scheduling, Triton with full CUDA graphs, FP8 KV and two API servers (the engine points).
• The records. The engine points, 82 runs on both nodes with each addition against the point before it; the rounds, five of the four additions against v1.14's configuration and two follow-up rounds; the quick start again, 272–277 ms against 1,544–1,547 ms on stock vLLM, and 173–179 ms while the profile held the agents' decodes; the scheduler's cost a step in the profile's configuration, measured again, a baseline whose 7% tolerance its negative controls exceed by 17–84%.
• The packages are versioned 1.15.0, and are not published to a package index.
• Upgrading from v1.14.
  • The serving profile's engine adds make-room's retain_tokens 3584, retain_s 30, hint_hold_ms 400 and take_reservations true, with decode_hold_tokens 0, and runs the serve module's scheduler, flatshare_make_room_serve.AsyncScheduler, with flatshare_jit at a 30 ms margin and the card's step-time model: a release renders them, and serving-check holds the engine to every one.
  • In serve mode the voice worker announces each turn (the profile's voice turn_hints, FLATSHARE_TURN_HINTS over it). The engine's log line counts the turns announced: a count that stays at 0 while calls run means the hints do not reach the engine.

v1.14: the code reviewed, and the FP8 crash found

Flatshare v1.14 reviews the code's architecture plane by plane, with every defect the review confirmed fixed and held by a test, and finds the cause of the FP8 KV cache crash: stock vLLM's FlashInfer plan race under asynchronous scheduling, which the CPU tier's transfers open on this card and which the open vLLM #64165 closes. The serving profile keeps its BF16 cache. E1's simulated cells at higher pressure change, and the soak's record now says what a harness restart added to it; no measured result changes.

What changed since v1.13

• The system at a glance. ARCHITECTURE.md opens with a figure of the change and the three planes around it, serving, measurement and operations, each part named with its home in the repository.
• The FP8 crash, found. The FP8 record reproduces the 10-27 crash on stock vLLM 0.32.0 and 0.33.0rc3 with none of the change's code, and finds it in FlashInfer's plan(), which refills one page-locked buffer per wrapper while its last copy to the GPU may still be queued (vLLM #61239): it needs the CPU tier, asynchronous scheduling and FlashInfer, and the wait of vLLM #64165 closes it. On this card the CPU tier's DMA holds the plan's copy back on the one host-to-GPU copy engine: without the CPU tier no plan in 10,000 found its copy pending. A GPU core dump names the faulting kernel, FlashInfer's PersistentVariableLengthMergeStatesKernel; XQA decode does not avoid the race, and vLLM main stalls too. benchmarks/fp8 is the harness. The serving profile keeps its BF16 cache.
• The gateway's drain. When drain_timeout_s passed, the process exited under the relays still running, and their records said completed, the only success; they now say drained. A stop also left a request that a covering node holds back unanswered for the whole drain, logged cancelled; it now gets 503 draining with Retry-After: 1 as the drain begins, as at a handover. Two tests run the binary.
• Releases. A dropped session no longer stops a release halfway: the tool's output drops the terminal when a write fails and keeps every line in releases/release.log, and a switch records itself in releases/switching.json, so that the next switch finishes one a killed run left, restarting every active unit. The voice drain trusts only a status the worker wrote after it began and removes its drain file whenever it ends early. A release refuses while an E2 point's drop-ins are linked into cyan's units, as the soak does.
• The simulator. A prefetch paid for every block the CPU tier held, also those the pool had no room for, which it never placed; it now pays for what fits when it begins. E1's prefetching arms change at 16 sessions and more (the oracle at 24 sessions: +4.9% / −3.9% at p50 / p90, from −2.4% / −16.9%), and its conclusions hold: under 8% of the median resume, 28–29% of the tail at low pressure, nothing from session history.
• The harness. A soak's window starts on the millisecond its first scrape is stamped with, so the counters' rise before it never falls in its first hour; neither committed soak changes, the day's having started on a millisecond and voice alone's having no agents. A run of arriving calls lasts its whole time when its last call ends first: voice alone's soak started its callers again 87 s before its end, a second draw that placed one call where the first had placed none, and without it the day's replies over the SLO number 77, not 74, 1.5% either way (the soak's record says so). Each soak's summary.json is held to what its record gives, and the record of E2 names the two points whose summaries leave their failed turns out (the report recomputes every point's V2V from its turns). A record that names an account's home directory fails the tests, since the records are copied from shared hosts.
• What the docs say. A round's sample of the agents is a draw of where each session enters the same traces, since AIPerf's sampler gives every round the same ones, and REPORT §9 states the limitation; serve mode's endpointing, 0.5 s at least and 3 s at most, is a pin, not LiveKit's default for its end-of-turn model; the kill drills' streams ended 6–11, 10–14 and 53–57 ms after the kill command, as their records give them, where the docs began each range at a median; #45558 preempts for KV as #40087 does (PRIOR-ART, REPORT §8 and #63350's description); the restore stall's upper quartile is 0.94, as the simulator's fit computes it; and the bench suite takes four to five minutes. The report is at version 1.14.
• The packages are versioned 1.14.0, and are not published to a package index.
• Upgrading from v1.13.
  • No configuration changes. A request cut at the drain's timeout is counted drained in the request log and the outcome series, where it was completed.
  • A release writes releases/release.log and, while it switches, releases/switching.json, and it refuses while an E2 point's drop-ins are linked on cyan: flatshare-bench e2 reset removes them, and --force releases anyway.