Optimizing Qwen3.8 Flash-Next and 27B/DFlash2 on SM120: native speculation, XQA, RecoverSSM, and HiCache/NIXL

[jpezzulli]

I built and qualified an optimized SGLang runtime for the two Qwen3.8 models I wanted to serve on one 96 GB RTX PRO 6000. I have published the complete SGLang-derived source, the two launch recipes, resolved backend tables, upstream provenance, measured results, and the validation suite.

Pennyroyal is one source tree, not separate Flash-Next and 27B forks. Its qualified TP=1 launch profiles are:

Qwen3.8 Flash-Next NVFP4 with native NEXTN MTP

Qwen3.8-27B FP8 with DFlash2

The optimization work is architectural rather than a pile of flags.

For Flash-Next, the selected execution path uses FlashInfer GDN decode and prefill, FlashInfer CUTLASS MoE for both target and native MTP, Triton QSA sparse prefill, and the FlashInfer QSA wrapper resolving to XQA for sparse decode on exact SM120. Native NEXTN MTP shares QSA indexes. RecoverSSM removes the separate intermediate speculative SSM pool and captures target, draft, extend, verify, prefill, and accepted-state recovery graphs. Correct three-axis mRoPE preserves the fused multimodal rotary path. The resulting automatic FP8 target/native-MTP KV capacity is 824,384 tokens at a served 524,288-token factor-2 YaRN context.

For 27B, DFlash2 runs with an independent FP8 draft, gamma 8, a 2,048-token draft window, fused draft KV materialization, and TRTLLM-MHA/XQA target decode plus fixed-width XQA verification. The current launcher exposes 1,118,784 FP8 target and draft KV tokens while retaining the qualified Mamba allocation.

The QSA decode path is not TRTLLM-Gen. Pennyroyal enters FlashInfer's QSA wrapper, which resolves to XQA on exact SM120. TRTLLM-Gen is SM100-targeted. Forced selection fails its architecture check, and directly loading the relevant cubin on SM120 returns CUDA_ERROR_NO_BINARY_FOR_GPU (209). No SM100-specific 35% claim is attributed to this runtime.

HiCache/NIXL is a core part of both launch profiles. The runtime writes pages through a host tier with kernel I/O, page-first layout, and write-through behavior, then persists them through NIXL POSIX FILE storage on NVMe using io_uring and O_DIRECT. The namespaces fingerprint the checkpoint, executable source, model configuration, topology, datatypes, speculative method, attention backends, Mamba state, PyTorch identity, and SM architecture.

More importantly, Pennyroyal persists the complete model representation:

Flash-Next: packed target/native-MTP KV plus GDN, PLE, and compressed QSA state

27B/DFlash2: target KV plus Mamba/GDN and DFlash2 side state

This is why restart restoration remains coherent rather than merely recovering an old KV tensor. Flash-Next restored 489,856 of 489,879 input tokens after restart, recomputed 23, and returned all three needles exactly. Its 62,040.60 tok/s result is effective restored-prefix throughput, not cold prefill. The 27B profile also passed identical-namespace restart, namespace rollback, isolation, and concurrent restoration tests.

The established, separate performance campaigns at a 450 W GPU cap measured:

Flash-Next NVFP4

64K cold prefill: 10,103.70 tok/s

Approximately 490K cold prefill: 7,872.15 tok/s, 3/3 needles exact

C1 decode: 171.09 tok/s

C4 decode: 427.54 tok/s aggregate

Reasoning: 97.49/100

Tools: 30/30 exact and semantically correct

Complete vision validation passed

27B FP8/DFlash2

64K prefill: 6,163.07 tok/s

Approximately 489K prefill: 1,618.31 tok/s, 3/3 needles exact

C1 decode: 108.75 tok/s

C4 decode: 390.23 tok/s aggregate

xhigh reasoning: 98.26

medium reasoning: 95.807

These are different model configurations and campaigns, not a single A/B table.

Pennyroyal v2.1 is a bounded correctness sync rather than a new benchmark campaign:

8de07a0 adopts merged PR #36806, narrowing the QSA wrapper route to exact SM120 and excluding SM121/GB10.

23e51dd adapts merged PR #35821, preventing empty Mamba radix ghost nodes and bounding accepted-state tracking across Pennyroyal's eager, fused CUDA, and KDA paths.

1ba0b2a repairs stale QSA test fixtures only.

The executable source is 1ba0b2a, installed as 0.5.19.dev488+g1ba0b2a1b. The pennyroyal-v2.1.0 tag prefix is intentional because a bare v2.1.0 tag interferes with SGLang package-version resolution.

Seventy-two focused QSA, Mamba, and CUDA tests passed. Both profiles booted, captured CUDA graphs, retained their expected KV capacities, and restored through NIXL. No CUDA errors or retractions were observed. One unrelated pre-existing repository-wide registration failure remains for test_qsa_pool_host_unit.py. I did not publish the incidental v2.1 smoke throughput because it used different workloads from the canonical campaigns.

An independent user also validated Flash-Next FP8, FP8 KV, and native MTP at TP=2/EP=2. Removing the optional overlap plan-stream setting resolved the plan-stream failure. The durable external findings were working TP=2 FP8 MTP, decode scaling through C6, and a 3,182,848-token KV pool. Those results remain separate third-party evidence.

Optimized runtime and source provenance: https://github.com/jpezzulli/sglang-rtxpro6000

Pennyroyal v2.1 Release: https://github.com/jpezzulli/sglang-rtxpro6000/releases/tag/pennyroyal-v2.1.0

Public validation suite and result catalog: https://github.com/jpezzulli/pennyroyal-validation

This is still an early, system-specific Flash-Next integration. I built it for my own workstation and published it because the source-level optimization, hybrid persistence, and validation work may be useful to other SGLang operators.