Running MiniMax H3 Ref2VA on 2x RTX 5090 under a 116 GB cgroup — streaming weight load + resident-layer GPU routing (4 patches, SGLang 0.5.17)

[ft54482]

I got MiniMax H3 Ref2VA (33B DiT + Qwen3VL, full BF16, ~123 GB) to load and run on 2× RTX 5090 under a host cgroup memory.max=116 GB that could not be raised. Load + inference both verified (1344×768 / 4.5 s video out, peak GPU 18.7 GB, peak CPU 114.7 GB).

Why it OOMs by default: SGLang's hf_to_custom_state_dict materializes the entire converted DiT weight dict on CPU at once (~61.7 GB), on top of the ~58 GB resident Qwen3VL text encoder. That sum exceeds 116 GB and the load dies at shard 13/13 every time. TP=2 does not help because both ranks share the same cgroup.

The fix is 4 source patches to sglang/multimodal_gen/ (SGLang 0.5.17):

1. Patch A (utils.py): new stream_hf_to_custom_state_dict generator that yields one converted tensor at a time — the 61.7 GB dict is never built.
2. Patch B (fsdp_load.py): load loop consumes the generator + del full_tensor each iteration; FP8 dequant gets a shim (unsupported during streaming, BF16 never hits it).
3. Patch C (transformer_loader.py + fsdp_load.py): thread resident_layers / resident_layer_attr from server_args down to the loader (the layerwise manager is only configured after loading, so model.layerwise_offload_managers is empty at load time).
4. Patch D (fsdp_load.py): resident-layer params (regex ^blocks\.(\d+), layer_idx < resident_layers) skip cpu_offload and land directly on GPU.

Gotchas worth flagging:

• --pin-cpu-memory false is mandatory — pin_memory allocates ~53 GB of unreleasable shmem that alone exceeds the ceiling.
• model_cls.layer_names is an instance attr (set in __init__, not at load-planning time); fall back to model.layer_names, then hardcode "blocks".
• Install ninja-build (JIT kernels) and ffmpeg (ffprobe post-processing) system-wide.
• NVFP4 would halve the footprint but isn't usable for H3 end-to-end in SGLang yet, so BF16 is the only working precision under a tight cgroup.

Full patches, the sglang serve command, the inference call shape, and the memory-curve numbers are in the gist:

https://gist.github.com/ft54482/b5489349620c7f0fa63e634e3356d79b

Sharing in case anyone else is fighting the same cgroup-OOM during SGLang DiT loading. Happy to turn the streaming loader into a PR if the maintainers are interested — it's a pure memory win with no behavior change for non-cgroup-limited hosts.

[jerkstorecaller]

Hi OP, I have two Sparks (essentially a 5060 with 128GB VRAM) connected using a 200Gbps Connect-X7 network cable. This works great for LLMs, a patched VLLM distributed across two sparks accelerates tok/sec generation.

I've been running H3 on one. The model fits entirely in VRAM so this isn't the same challenge as two 5090s. It's the slow gen times due to weak compute. I saw that sglang seems to support running H3 across two cards, so I was wondering if this type of parallelism accelerates compute if two GPUs are available, when the model already full fits in the RAM of one "GPU". Do you know if it's worth spending the week-end learning to use sglang?

[KushagraKanaujia]

Hey, building Throttle: semantic cache for self-hosted inference. Cuts repeat-query costs 40-60%, zero code changes. Works with vLLM, Ollama, SGLang, LMDeploy.

Open-source, free beta. Looking for users to test:

https://github.com/KushagraKanaujia/throttle