Running MiniMax H3 (FL2VA) on 4x L40S with SGLang: dynamic FP8, ~2x speedup vs 2-GPU BF16, 14GB/card

[ft54482]

Running MiniMax H3 (FL2VA) on 4x L40S with SGLang: FP8 dynamic quantization, ~2x speedup vs 2-GPU BF16

A working recipe for running MiniMax H3 video generation with 4 GPUs in parallel on a single request (Ulysses sequence parallelism), with dynamic FP8 quantization that cuts VRAM to ~1/4 and speeds up generation ~2x compared to 2-GPU BF16 — at the same output quality as the full-precision model.

Environment

• GPUs: 4x L40S 46 GB (sm_89)
• SGLang: sglang-diffusion multimodal_gen runtime, 2026-08-04 build (Python 3.10, torch 2.x)
• Model: MiniMax H3, FL2VA partition (full-precision weights, ~62 GB transformer)
• Checkpoint layout: standard diffusers-style (FL2VA/transformer/model-*.safetensors etc.)

Launch command

SGLANG_USE_RUNAI_MODEL_STREAMER=0 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
sglang serve \
  --model-path /path/to/MiniMax-H3 \
  --model-variant fl2va \
  --pipeline MiniMaxH3Pipeline \
  --host 127.0.0.1 --port 8062 \
  --performance-mode manual \
  --quantization fp8 \
  --dit-cpu-offload true \
  --dit-layerwise-offload true \
  --layerwise-offload-components dit text_encoder vae \
  --dit-offload-prefetch-size 0 \
  --dit-layerwise-resident-layers 0 \
  --warmup-mode off \
  --num-gpus 4 --ulysses-degree 4 \
  --batching-mode dynamic --batching-max-size 4 --batching-delay-ms 1000

Key points:

• --num-gpus 4 --ulysses-degree 4: sequence parallelism — one request is sharded across all 4 GPUs.
• --quantization fp8: dynamic FP8 quantization (per-row weight scales computed at load). This is what enables the VRAM savings.
• SGLANG_USE_RUNAI_MODEL_STREAMER=0: required — see gotcha Add SRT json decode example #2 below.

Results (768p 16:9, 5s, 20 steps)

Config	Steps/s	VRAM/card	Output
2 GPU BF16 full-precision	~12–16 s/it	—	baseline quality
4 GPU FP8 dynamic	~5.8–6.1 s/it	9.5 GB idle / 14.3 GB peak per card	identical quality (frame std 54–71 vs baseline 44–71)

• ~2x speedup for a single 5s video (120 s denoise wall time), using all 4 cards on one request — something ComfyUI cannot do.
• Only ~30% of the 45 GB L40S VRAM is used during generation, leaving room for higher resolution / longer videos / concurrent requests (dynamic batching works, see below).

Gotchas we hit (and fixes)

1. Pruned FP8 checkpoints (e.g. rzgar/minimax_h3_fl2va_fp8_e4m3fn on ModelScope) are mathematically broken — verify before using

The pruned conversion replaces the 2688-dim AdaLN time-embedding with an 8-dim curve table (adaln_t_table [1025, 8]). We verified the conversion against the original full-precision weights (W_pruned @ table[t] vs W_full @ silu(time_embedder(t))):

• direction cosine only 0.07–0.63 (sometimes negative), magnitude 10–60x off
• even searching the whole table for the best-matching row only reaches cos ≈ 0.64, and all timesteps match only ~56 rows around the table's middle

The generation "succeeds" but the video is a flat/gray still (denoised latent collapses to ~0). Do a math check like the one above before adopting any pruned/curve-format H3 checkpoint.

2. Run:ai Model Streamer casts fp8 tensors → disable it

With SGLANG_USE_RUNAI_MODEL_STREAMER unset, the streamer loads safetensors with fp8 weights pre-cast to bf16. That silently defeats:

• fp8-dequantization at load (gotcha Add install with pip #3), and
• the loader's dtype checks.

Set SGLANG_USE_RUNAI_MODEL_STREAMER=0.

3. ComfyUI-style fp8 checkpoints need explicit dequantization by weight_scale

ComfyUI-style fp8 checkpoints store weights as fp8_value = weight / weight_scale (per-tensor scalar weight_scale, e.g. 0.0107) plus input_scale / comfy_quant keys. SGLang's _maybe_dequantize_fp8 looks up weight_scale from the mapped state dict — but hf_to_custom_state_dict filters keys by valid_target_names=set(model.state_dict().keys()), and a dynamically-quantized model has no weight_scale parameters → the scale is never found → weights load ~90x too large. We added a module-level scale registry populated from the checkpoint files (see patch below).

4. Loading a 62 GB full-precision model with --quantization fp8 OOMs on 4x45 GB

The post-load quantization (process_weights_after_loading) moved the whole model to GPU (model.to(weight_postprocess_device)), and bf16 + fp8 copies together exceed 4x45 GB. Fix: run the postprocessing per module with a GPU round-trip (peak = one layer's size):

# in maybe_load_fsdp_model, replace the "3. postprocessing" block:
for _, module in model.named_modules():
    quant_method = getattr(module, "quant_method", None)
    if quant_method is not None and hasattr(quant_method, "process_weights_after_loading"):
        if weight_postprocess_device is not None:
            module.to(weight_postprocess_device)
        quant_method.process_weights_after_loading(module)
        if weight_postprocess_device is not None:
            module.to("cpu")

5. Persistent buffers (e.g. pruned adaln_t_table) are never loaded

load_model_from_full_model_state_dict matches checkpoint keys against named_parameters() only, and named_buffers() skips None-valued buffers (if buf is not None in torch's Module). A pruned checkpoint's adaln_t_table therefore stays None and the model silently falls back to the sinusoidal time embedding with mismatched input dims (2688 vs 8). Fix: scan _buffers directly and materialize None buffers from the checkpoint files.

6. K<16 linear layers under dynamic fp8

With dynamic FP8, small-K layers (e.g. pruned AdaLN with K=8) fail the cutlass 16-alignment check and fall into triton_scaled_mm, which asserts weight.shape[0] == K — the weight must be [K, N]. Big layers (K=3072 etc.) take the cutlass path and are fine. If you must run tiny-K layers in fp8, transpose the weight before the triton call.

Dynamic batching (concurrency)

With --batching-mode dynamic --batching-max-size 4 --batching-delay-ms 1000, requests arriving within the aggregation window are packed into one denoise batch: 2 concurrent 768p/20-step requests ran at 21.5 GB/card peak (vs 14.3 GB for one), i.e. ~7 GB/card per extra request. We're measuring exact wall-clock throughput numbers now.

Full patch summary (site-packages edits, 2026-08)

• sglang/multimodal_gen/runtime/loader/fsdp_load.py
  1. CPU checkpoint loading (avoid GPU peak during TP/FSDP split)
  2. Persistent-buffer materialization from safetensors (gotcha Typo: rename image_url to image_file #5)
  3. _FP8_SCALE_REGISTRY + fallback lookup in _maybe_dequantize_fp8 (gotcha Add install with pip #3)
  4. Per-module postprocessing with device round-trip (gotcha Json Decode && Mutl-Turns #4)
• sglang/multimodal_gen/runtime/models/dits/minimax_h3.py — pruned-format support: adaln_t_table interpolation in TimeEmbedder.forward, adaptive adaln_input_dim, skip outer SiLU when the curve table is active (the curve already bakes silu(time_embedder(t)), matching ComfyUI's apply_silu = not use_adaln_curves)

Happy to turn any of these into proper PRs — especially #3 (scale registry) and #4 (per-module postprocessing) which look like general fixes for large-model dynamic-quant loading.

[ft54482]

Update (concurrency): Dynamic batching does NOT merge conditioned requests (FL2VA/Ref2VA with keyframe/reference images) — _can_dynamic_batch in scheduler.py rejects any request with image_path set, so video requests with image conditions run sequentially (encoding/decoding stages can still overlap; 2 concurrent 768p requests peaked at ~21.5 GB/card vs 14.3 GB for one — VRAM is not the limiter, the scheduler merge logic is). Pure text-to-video (t2va, no image conditions) requests with identical prompt + params DO merge into one denoise batch.