Support LingBot-Video (MoE-30B-A3B)

[NancyFyong]

LingBot-Video (MoE-30B-A3B) Integration

Summary

Add the MoE variant of LingBot-Video — MoE-30B-A3B, 30.1B total parameters with ~2.9B activated per token — on top of the Dense-1.3B integration merged in #1539. Supported:

• ✅ Text-to-video inference
• ✅ Image-to-video inference (condition on a first frame via input_image)
• ✅ Text-to-image inference (same pipeline and DiT with num_frames=1)
• ✅ Video-to-video (input_video + denoising_strength)
• ✅ Low VRAM inference (FP8 weights + disk offload) for all three tasks
• ✅ LoRA training and full-parameter training (T2V / TI2V)

MoE-30B-A3B is a drop-in second checkpoint for LingBotVideoPipeline: one checkpoint serves all three tasks, exactly like Dense-1.3B. The pipeline, VAE, text encoder and processor are unchanged — this PR adds the sparse-MoE path inside LingBotVideoDiT, one registry entry, the VRAM map entries the sparse stack needs, and the examples.

Model Components

Component	File	Class	Params
DiT	diffsynth/models/lingbot_video_dit.py	LingBotVideoDiT (MoE FFN path)	30.1B total / ~2.9B active
Text Encoder	diffsynth/models/krea2_text_encoder.py (reused)	Krea2TextEncoder (Qwen3-VL-4B-Instruct)	4.44B
VAE	diffsynth/models/qwen_image_vae.py (reused)	QwenImageVAE (8x spatial / 4x temporal)	0.13B
Pipeline	diffsynth/pipelines/lingbot_video.py (reused)	LingBotVideoPipeline	-

The MoE FFN is built from LingBotVideoRouter, LingBotVideoGroupedExperts and LingBotVideoSparseMoeBlock. Routing follows the checkpoint config: 48 blocks, 128 routed experts with top-8 selection, group-limited routing (4 groups, top-2 groups), sigmoid gate with a selection-only e_score_correction_bias, norm_topk_prob, routed_scaling_factor=2.5, and 1 shared expert. Expert matmuls use torch._grouped_mm on CUDA with a per-expert loop fallback elsewhere.

Registered in diffsynth/configs/model_configs.py under lingbot_video_series:

{
    # Example: ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors")
    "model_hash": "65b83aa625cd362ff5ff3409fb367a6f",
    "model_name": "lingbot_video_dit",
    "model_class": "diffsynth.models.lingbot_video_dit.LingBotVideoDiT",
    "state_dict_converter": "diffsynth.utils.state_dict_converters.lingbot_video_dit.LingBotVideoDiTStateDictConverter",
    "extra_kwargs": {'depth': 48, 'axes_lens': (4096, 512, 512), 'num_experts': 128, 'moe_intermediate_size': 768, 'n_group': 4, 'topk_group': 2, 'n_shared_experts': 1, 'routed_scaling_factor': 2.5},
}

Examples

All paths are relative to examples/lingbot_video/:

Task	Inference	Low VRAM inference	LoRA training	Full training
T2V (+ V2V)	model_inference/lingbot-video-moe-30b-a3b_t2v.py	model_inference_low_vram/lingbot-video-moe-30b-a3b_t2v.py	model_training/lora/lingbot-video-moe-30b-a3b_t2v.sh	model_training/full/lingbot-video-moe-30b-a3b_t2v.sh
TI2V	model_inference/lingbot-video-moe-30b-a3b_ti2v.py	model_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v.py	model_training/lora/lingbot-video-moe-30b-a3b_ti2v.sh	model_training/full/lingbot-video-moe-30b-a3b_ti2v.sh
T2I	model_inference/lingbot-video-moe-30b-a3b_t2i.py	model_inference_low_vram/lingbot-video-moe-30b-a3b_t2i.py	-	-

Post-training validation scripts live in model_training/validate_lora/ and model_training/validate_full/, and the training scripts launch through model_training/full/accelerate_config_moe.yaml.

Prompts and the TI2V condition frame come from the published example dataset directories that Dense already uses (lingbot_video/lingbot-video-dense-1.3b_{t2v,ti2v,t2i}/), so no new dataset upload is needed. Docs: docs/en/Model_Details/LingBot-Video.md, docs/zh/Model_Details/LingBot-Video.md; both README example tables gain three MoE rows.

Results

Text-to-video — lingbot-video-moe-30b-a3b_t2v.py

Video-to-video — same script, V2V section (denoising_strength=0.7, seed 1)

Image-to-video — lingbot-video-moe-30b-a3b_ti2v.py

Text-to-image — lingbot-video-moe-30b-a3b_t2i.py

Low VRAM inference — model_inference_low_vram/, same prompts and seeds

Training — outputs of validate_lora/ and validate_full/ after running the training scripts

Dependencies

No new packages. torch._grouped_mm is used opportunistically behind a fallback, so no minimum-version bump is needed.

Notes

• VRAM. 480x832x81 in bf16 peaks at ~80 GiB; the low-VRAM examples run the same workloads at ~42 GiB with essentially unchanged step time (16.6 -> 16.9 s/it). Those examples set vram_limit to the whole card, so lowering it trades speed for memory: with vram_limit=12.0 the single-frame T2I run peaks at 12.2 GiB and the 81-frame T2V run at 15.5 GiB (38 s/it). The DiT alone is ~57 GB on disk.
• Low-VRAM module map. LingBotVideoBlock, LingBotVideoRouter and LingBotVideoGroupedExperts are registered as AutoWrappedNonRecurseModule, so each module manages only its own bare parameters and the 128 experts per layer stay streamable instead of being deep-copied as one 1.25 GB block. That wrapper leaves parameter casting to the model, so the router weight, the correction bias and the three expert tensors are cast to the token device and dtype at use, the same way scale_shift_table already is. e_score_correction_bias is an nn.Parameter(requires_grad=False) rather than a buffer so that skip_model_initialization restores it too. Dense also goes through this map; its bf16 and low-VRAM outputs were checked after the change.
• Captions. LingBot-Video is trained on structured JSON captions, and MoE-30B-A3B is markedly less tolerant of plain prose than Dense-1.3B — a free-form sentence degrades into texture noise. The examples load the structured captions shipped with the example dataset; see the prompt-rewriting section in the docs.
• Training. The LoRA and full-parameter scripts launch through examples/lingbot_video/model_training/full/accelerate_config_moe.yaml (DeepSpeed ZeRO-2, optimizer CPU offload, bf16), following the wanvideo/Wan2.1-T2V-14B precedent where the LoRA script reuses the full-training config. LoRA runs at 28.3 s/it with 88.5 GiB per GPU on 4 GPUs. The full-parameter scripts additionally enable --use_gradient_checkpointing_offload, without which the 30B backward pass does not fit. --lora_target_modules "to_q,to_k,to_v,to_out" only reaches the attention projections: the routed experts and the router are stored as bare parameter tensors for grouped matmul, so LoRA leaves them frozen and full-parameter training is needed to update the expert stack — noted in the docs.
• Refiner. The release also publishes a refiner/ DiT for a second-stage refinement pass. It is not wired into LingBotVideoPipeline and is intentionally out of scope here; the examples and docs use the base transformer/ only.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.