Idea
Make request admission, memory accounting, eviction, and prefix-cache ownership aware of non-KV state in current hybrid multimodal models.
Why It Matters
Current models can combine several state classes that do not share ordinary paged-KV semantics:
- variable media-token expansion plus vision, audio, or video encoder/projector residency;
- full-attention and sliding-window KV with different retention bounds;
- convolutional, DeltaNet, Mamba, or other recurrent state;
- native or external speculative-draft state;
- prefix-cache entries whose safe reuse depends on modality inputs and state-transition boundaries.
Weight-plus-KV estimates can therefore over-admit work, evict the wrong state, or incorrectly alias prefixes even when each subsystem works in isolation.
Current Architecture Candidates
- Step-3.7-Flash: multimodal MoE, mixed full/SWA attention, and three NEXTN layers.
- MiMo-V2.5: multimodal MoE, 128-token SWA/global attention, and one-million-token context.
- Nemotron 3 Nano Omni: multimodal Mamba/attention MoE.
- LFM2.5: convolution/attention MoE with explicit convolution state.
- Qwen3.6 and Nemotron 3 Ultra: recurrent-or-linear/full-attention hybrids with draft-state concerns.
“Architecture candidate” means only that model structure matches this experiment's conceptual target. It does not assert current vLLM model support, quantization compatibility, kernel availability, graph capture, speculative-decoding integration, or acceptable performance. Each must be demonstrated by exact revision, backend-selection evidence, correctness tests, and benchmark results.
Possible vLLM Work
- Define a state inventory API that reports persistent and per-request bytes, lifetime, mutability, eviction rules, and whether state is prefix-shareable.
- Include processor/encoder residency, media expansion, per-layer KV retention, recurrent/conv state, and drafter state in admission estimates.
- Make prefix-cache keys and reuse validation include modality identity and state-transition compatibility.
- Extend scheduler traces so estimated state can be reconciled with allocator and cache telemetry.
- Keep architecture adapters explicit; do not treat recurrent or convolutional state as paged KV.
Acceptance Criteria
- A test matrix covers at least one mixed full/SWA model, one recurrent-or-convolutional hybrid, one multimodal model, and one speculative-draft model.
- Admission estimates do not undercount state allocations observed by the instrumented runtime for the tested request shapes.
- Prefix reuse is rejected when modality inputs or state-transition boundaries differ, and accepted for compatible identical prefixes.
- Eviction never drops required recurrent, convolutional, sparse-index, or draft state while retaining dependent cache entries.
- Scheduler throughput and p95 latency differ by no more than 2% from the existing path when all requests use ordinary dense text-only KV and memory pressure is absent.
Related Issues
Sources
Duplicate Check
Searched the fork backlog for multimodal admission, hybrid-state accounting, prefix-cache ownership, recurrent state, and media-token expansion. Existing issues cover individual memory, KV, prefix, and state-update mechanisms, but none owns their cross-request accounting and cache-sharing contract.
Idea
Make request admission, memory accounting, eviction, and prefix-cache ownership aware of non-KV state in current hybrid multimodal models.
Why It Matters
Current models can combine several state classes that do not share ordinary paged-KV semantics:
Weight-plus-KV estimates can therefore over-admit work, evict the wrong state, or incorrectly alias prefixes even when each subsystem works in isolation.
Current Architecture Candidates
“Architecture candidate” means only that model structure matches this experiment's conceptual target. It does not assert current vLLM model support, quantization compatibility, kernel availability, graph capture, speculative-decoding integration, or acceptable performance. Each must be demonstrated by exact revision, backend-selection evidence, correctness tests, and benchmark results.
Possible vLLM Work
Acceptance Criteria
Related Issues
Sources
Duplicate Check
Searched the fork backlog for multimodal admission, hybrid-state accounting, prefix-cache ownership, recurrent state, and media-token expansion. Existing issues cover individual memory, KV, prefix, and state-update mechanisms, but none owns their cross-request accounting and cache-sharing contract.