ENG-EXPERT-STREAM has been READY since 2026-07-10 with a committed spec and no owner. This issue tracks implementing it, driven by a concrete target.
The target
Qwen/Qwen3.8-2.4T-A95B is registered and binds to Qwen3_5MoeForCausalLM, the same GDN-hybrid MoE backbone as Qwen3.6-35B-A3B at larger scale, with every knob config-driven. The model-matrix row says "2.4T MoE, DOES NOT FIT this hardware". Capacity is the only blocker.
At UD-Q1_0 the checkpoint is 370 GiB against 128 GB of unified memory on dgx.casa (GB10). Expert streaming is the only mechanism that closes that gap on this hardware:
| Row |
Effect on a unified-memory host |
ENG-WEIGHT-OFFLOAD |
None. Host and device share one pool, so moving a weight to the host frees no device bytes |
ENG-HYBRID-PLACEMENT |
None, for the same reason. Its own spec says so |
ENG-EXPERT-STREAM |
Works. NVMe holds the weights and only touched experts become resident |
What the spec's cost estimate no longer has to pay
The spec designs a safetensors to Marlin bank conversion: a versioned bank file, a manifest, a repack step, and a second on-disk copy of the expert bytes. On the GGUF path none of that is needed. The mmap'd file already is the bank: block quantized, per-expert rows contiguous, offsets computable from vt::RowSizeBytes.
The per-expert slicer also already exists and is the production decode path, which #824 corrected in the spec: OwnGgufQuantBlocks(..., row_offset, ...) at qwen3_5_gguf_weights.cpp:57, and GemmRowSlice called per expert at deepseek_v4.cpp:1004,2487 and laguna.cpp:1225.
So the remaining work is the residency policy and the decode dispatch, not a bank builder and not a slicer.
Honest expectation, recorded before the work
At UD-Q1_0, top-10 of 512 experts, each token pulls roughly 7.6 GB of expert weight. dgx NVMe measures 5.0 to 5.3 GB/s, so the naive floor is about 0.7 tok/s. A resident hot-expert cache of about 110 GB covers roughly 28% of experts by count, and real routing skew should raise the hit rate well above that, which puts the plausible band at 2 to 5 tok/s at concurrency 1.
The spec's own verdict stands: viable at c1 to c4, and NOT served at high concurrency, because at conc 32 a single step touches most experts and the per-step I/O approaches the whole bank.
This issue does not promise interactive speed on this hardware. It promises a model that could not run at all becoming a model that runs.
Work
The spec's W0 to W7 leaves, reshaped by the GGUF finding above. First increment is W1, the CPU-side cache policy: fixed-capacity slots, logical-expert to slot mapping, hotness-decayed LFU with LRU tiebreak, and eviction protection for in-flight or currently selected entries. No device allocation, unit tested.
Spec: .agents/specs/expert-streaming.md · related: #490, #797, #907
ENG-EXPERT-STREAMhas beenREADYsince 2026-07-10 with a committed spec and no owner. This issue tracks implementing it, driven by a concrete target.The target
Qwen/Qwen3.8-2.4T-A95Bis registered and binds toQwen3_5MoeForCausalLM, the same GDN-hybrid MoE backbone as Qwen3.6-35B-A3B at larger scale, with every knob config-driven. The model-matrix row says "2.4T MoE, DOES NOT FIT this hardware". Capacity is the only blocker.At UD-Q1_0 the checkpoint is 370 GiB against 128 GB of unified memory on dgx.casa (GB10). Expert streaming is the only mechanism that closes that gap on this hardware:
ENG-WEIGHT-OFFLOADENG-HYBRID-PLACEMENTENG-EXPERT-STREAMWhat the spec's cost estimate no longer has to pay
The spec designs a safetensors to Marlin bank conversion: a versioned bank file, a manifest, a repack step, and a second on-disk copy of the expert bytes. On the GGUF path none of that is needed. The mmap'd file already is the bank: block quantized, per-expert rows contiguous, offsets computable from
vt::RowSizeBytes.The per-expert slicer also already exists and is the production decode path, which #824 corrected in the spec:
OwnGgufQuantBlocks(..., row_offset, ...)atqwen3_5_gguf_weights.cpp:57, andGemmRowSlicecalled per expert atdeepseek_v4.cpp:1004,2487andlaguna.cpp:1225.So the remaining work is the residency policy and the decode dispatch, not a bank builder and not a slicer.
Honest expectation, recorded before the work
At UD-Q1_0, top-10 of 512 experts, each token pulls roughly 7.6 GB of expert weight. dgx NVMe measures 5.0 to 5.3 GB/s, so the naive floor is about 0.7 tok/s. A resident hot-expert cache of about 110 GB covers roughly 28% of experts by count, and real routing skew should raise the hit rate well above that, which puts the plausible band at 2 to 5 tok/s at concurrency 1.
The spec's own verdict stands: viable at c1 to c4, and NOT served at high concurrency, because at conc 32 a single step touches most experts and the per-step I/O approaches the whole bank.
This issue does not promise interactive speed on this hardware. It promises a model that could not run at all becoming a model that runs.
Work
The spec's W0 to W7 leaves, reshaped by the GGUF finding above. First increment is W1, the CPU-side cache policy: fixed-capacity slots, logical-expert to slot mapping, hotness-decayed LFU with LRU tiebreak, and eviction protection for in-flight or currently selected entries. No device allocation, unit tested.
Spec:
.agents/specs/expert-streaming.md· related: #490, #797, #907