_{Every frame is 4 network evaluations. Music: "Goliath" by Scott Buckley, CC BY 4.0.}

This repo contains training, inference and evaluation code for PDMD on MiniMax-H3-33B joint video–audio generation. Train a four-step student from the base model, or run the released four- and two-step checkpoints on a single 24GB or 80GB GPU. More videos, with sound, are on our project page.

📦 Checkpoints · 🎬 Sampling · 🏋️ Training · 📊 Evaluation · 📚 BibTeX

PDMD: Projected Distribution Matching Distillation for Video Diffusion Models
Zimo Wang, Junkun Yuan, Angtian Wang, Haotian Yang, Canyu Zhang, Siyuan Yuan, Xingchang Huang, Bo Liu, Yizhi Wang, Yiding Yang, Chongyang Ma, Gordon Guocheng Qian
UC San Diego, ByteDance

Distribution Matching Distillation (DMD) reduces video diffusion models to a few network evaluations (NFE), but its samples can degrade during training, drifting toward oversaturation and artifacts. We trace this instability to critic errors that accumulate in successive student updates, and introduce Projected Distribution Matching Distillation (PDMD), which projects out the component of the DMD update parallel to the student–critic endpoint residual, an unbiased estimate of the critic's endpoint error. PDMD is a one-line change to DMD, with no extra loss, network, data, model pass, or training stage. With Wan2.1, PDMD reaches a VBench total score of 83.73 at 4 NFE; on MiniMax-H3 joint video–audio generation, it reaches a VideoGen-Eval visual total of 83.17 and the best score on all six audio metrics among the compared 4-NFE models.

This repository contains:

• 🪐 PDMD LoRA checkpoints at 4 NFE and 2 NFE
• ⚡️ Inference scripts that run them on a single 24GB GPU (run_low_vram.py) or a single 80GB GPU (run_bf16.py)
• 🏋️ Multi-node training of student and critic LoRAs, with automatic model and training-data downloads
• 📊 Video evaluation: VBench quality and Qwen semantic scoring on 387 VideoGen-Eval prompts
• 🔊 Audio evaluation: PQ, CE, CU, IS, IB and DeSync

Note. The main script, run_low_vram.py, is optimized for GPUs with little memory, e.g. an NVIDIA A10 (24GB of GPU memory, with 128GB of host RAM). The script quantizes the transformer and the Qwen3-VL text encoder to int8, and offloads their weights to host memory: the transformer is streamed to the GPU one block at a time and the text encoder one layer at a time, and the VAE is moved to the GPU only to decode. On 80GB GPUs, run_bf16.py runs all components in bf16 instead (see Sampling).

First, download and set up the repo:

git clone https://github.com/ZeamoxWang/pdmd.git
cd pdmd

The instructions below set up the Diffusers-based inference workers. For training and the DiffSynth-based benchmark sampler, use the separate environment in Training setup.

Install the inference dependencies (the versions we tested):

pip install torch==2.14.0 torchvision==0.29.0 torchaudio==2.11.0
pip install -r inference/requirements.txt

The inference script reuses the job parsing and audio/video muxing of MiniMax-H3-Turbo:

git clone https://github.com/ModelTC/Minimax-H3-Turbo.git
git -C Minimax-H3-Turbo checkout 02e26d591f7a04d5d1a074c9566d5dd4f22f6225

We train only LoRA adapters, keeping the base transformer frozen. Download the LoRA for your desired NFE and use it with MiniMax-H3; you do not need to download a separate full PDMD transformer.

The full merged 4-NFE transformer (~66 GB) is also available if needed.

All checkpoints are for the base transformer/ of MiniMax-H3 (the FL2VA/T2VA partition), which also provides the text encoder, VAEs and schedulers. Download the base model and the checkpoint(s) you want to run; the checkpoints are independent of each other:

hf download MiniMaxAI/MiniMax-H3 --exclude "transformer_ref/*" --exclude "FL2VA/*" --exclude "Ref2VA/*"
hf download pdmd2026/pdmd_4NFE_lora --local-dir ckpt/pdmd_4NFE_lora
hf download pdmd2026/pdmd_2NFE_lora --local-dir ckpt/pdmd_2NFE_lora

The LoRAs cover the attention projections and both feed-forward layers of every transformer and token-refiner block, and are fused into the base transformer once (on CPU, ~15 min):

python tools/checkpoints/fuse_lora.py --lora ckpt/pdmd_4NFE_lora/lora_model_0.safetensors --output ckpt/pdmd_4NFE_fused
python tools/checkpoints/fuse_lora.py --lora ckpt/pdmd_2NFE_lora/lora_model_0.safetensors --output ckpt/pdmd_2NFE_fused

Generate a video with inference/run_low_vram.py, passing the PDMD transformer and the number of steps. For example, with the 4-NFE checkpoint:

python inference/run_low_vram.py \
  --transformer-path ckpt/pdmd_4NFE_fused --inference-steps 4 \
  --jobs-json inference/examples/giant_cat_harbor_768p_4nfe.json \
  --turbo-repo Minimax-H3-Turbo --output-dir outputs

Use --transformer-path ckpt/pdmd_2NFE_fused --inference-steps 2 --jobs-json inference/examples/giant_cat_harbor_768p_2nfe.json for 2 NFE. Videos are written as outputs/<job>_<index>_<N>nfe_seed<seed>.mp4.

Jobs. A job file lists prompts with their length, resolution and aspect ratio, plus an optional inference_steps (see inference/examples/ for examples). The example jobs generate a 14.4-second, 1344×768 clip of a building-sized cat over a harbor, with sound, at seed 42. Sampling uses time shift 12 for video and no classifier-free guidance (MiniMax-H3 is guidance-distilled). For 2-NFE generation, we recommend audio time shift 6; use 3 for paper metrics. Credit to CALMDUST (@core_tan) on X for the suggestion.

Many videos. run_low_vram.py takes ~25–30 min to load the model. To pay that once, pass several files to --jobs-json, or omit it and the script keeps running as a worker that picks up job files dropped into --queue-dir.

Hardware. run_low_vram.py is built for GPUs with 24GB of memory and needs 128GB of host RAM: the transformer and the Qwen3-VL text encoder are quantized to int8 and streamed from CPU to GPU, and the VAE is moved to the GPU only to decode. On an NVIDIA A10 at 1344×768 and 345 frames:

80GB GPUs. On an 80GB GPU such as an A100 80GB, use inference/run_bf16.py, which takes the same arguments:

python inference/run_bf16.py \
  --transformer-path ckpt/pdmd_4NFE_fused --inference-steps 4 \
  --jobs-json inference/examples/giant_cat_harbor_768p_4nfe.json \
  --turbo-repo Minimax-H3-Turbo --output-dir outputs

run_bf16.py runs all components in bf16, without quantization, so its videos are not identical to those of run_low_vram.py. It moves each component to the GPU as a whole when the component runs, except the transformer, which is still streamed one block at a time because its bf16 weights plus the activations of a long 768p clip do not fit in 80GB (--transformer-offload none keeps the whole transformer on the GPU, for shorter or lower-resolution clips). It also needs 128GB of host RAM. On an A100 80GB at 1344×768 and 345 frames, a 4-NFE video takes ~10 min (~7.7 min denoising), with a peak GPU memory of 62.5 GiB.

Use a separate Python 3.11 environment from the inference workers above. Training and benchmark sampling use the pinned DiffSynth-Studio MiniMax-H3 implementation. Run commands from this repository's root:

python3.11 -m venv .venv-training
source .venv-training/bin/activate
pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0
pip install -r training/requirements.txt
pip install -e .

Install a CUDA build of PyTorch on the training machines. flash-attn is optional. For audio scoring, also run pip install -r evaluation/audio/requirements.txt.

Run on each of 2 nodes with 8 GPUs:

OUTPUT=runs/pdmd_4nfe NNODES=2 NODE_RANK=<0|1> MASTER_ADDR=<node 0 address> bash training/launch.sh

Data downloads itself. The base model MiniMaxAI/MiniMax-H3 and the training data pdmd2026/rcm-vidprom-h3-qwenvl-cache are fetched at pinned revisions on first start. The data is the 248,221 rCM VidProM prompts with their precomputed H3 text embeddings (~290 GB). No videos are needed: the student learns from its own rollouts.

Set MODEL_ROOT and CACHE_ROOT to use local model and embedding copies. To prefetch them:

python tools/download_assets.py --model-root /data/MiniMax-H3 --cache-root /data/pdmd-cache

FSDP and async checkpointing. Multi-GPU training uses FSDP to shard the frozen base within each node; student and critic LoRAs remain replicated, with gradients synchronized across all workers. Checkpoint serialization and disk writes run in a background thread after a CPU snapshot is captured, allowing training to continue during disk I/O.

Outputs. The student LoRA is saved every 500 iterations to runs/pdmd_4nfe/milestones/iter_NNNNNN/student_lora.safetensors; the reported reproduction is iter_002500. Re-running the command resumes from the latest state, saved every 100 iterations.

Time. 2,500 iterations take about 20 hours on 2×8 H100 (29.5 s per iteration, 45.6 GiB peak memory per GPU). One node also works and reaches the same batch by gradient accumulation.

python - <<'PYCONFIG'
import json
from pathlib import Path

config = json.loads(Path("training/configs/pdmd_4nfe_544p.json").read_text())
config["method"] = "dmd"
Path("training/configs/dmd_4nfe_544p.json").write_text(json.dumps(config, indent=2) + "\n")
PYCONFIG

CONFIG=training/configs/dmd_4nfe_544p.json OUTPUT=runs/dmd_4nfe \
NNODES=2 NODE_RANK=0 MASTER_ADDR=NODE0_IP bash training/launch.sh

python tools/download_assets.py --model-root /data/MiniMax-H3 --eval  # text encoder and VAEs, once
MODEL_ROOT=/data/MiniMax-H3 LORA=runs/pdmd_4nfe/milestones/iter_002500/student_lora.safetensors \
PROMPTS=evaluation/data/vgeneval/prompts_vgeneval_h3pe_v2.jsonl OUT=renders/pdmd_4nfe_2500 \
bash inference/sample_vgeneval.sh

This renders the 387 VideoGen-Eval clips with the paper's settings: 4 steps at shift 12/3, seed 42, 124 frames, and each prompt's 544p size from evaluation/data/vgeneval/vgeneval_544p_sizes.json. It runs one process per GPU, about 16 s per clip. LORA also accepts the downloaded LoRA file from pdmd2026/pdmd_4NFE_lora; for your own prompts, see python inference/sample_diffsynth.py --help.

The training reproduction uses 2 nodes of 8 H100s and finishes 2,500 updates in about 20 hours (29.5 s per update, approximately 10.4k tokens/s, 45.6 GiB peak memory per GPU). The reported VideoGen-Eval results use 387 prompts at 544p with seed 42. Higher is better, except DeSync. The paper checkpoints above and this training reproduction are separate models.

The 2,500-iteration reproduction improves Total and Semantic over the paper model, with lower Dynamic. The reported scores use the video and audio workflows in Evaluation.

See the video guide for scorer setup and the audio guide for the existing paper audio workflow. The commands below score the training reproduction from the repository root. For the training audio scorer, install evaluation/audio/requirements.txt and the av-benchmark checkout described below before running it.

# Video: Total = (4 × Quality + Semantic) / 5, from 7 VBench dimensions and 9 judged by Qwen3.8-27B
bash evaluation/video/score.sh renders/pdmd_4nfe_2500/videos /path/to/Qwen3.8-27B scores/pdmd_4nfe_2500 8

# Audio: PQ, CE, CU, IS, IB and DeSync
python evaluation/audio/score_directory.py --videos "$PWD/renders/pdmd_4nfe_2500/videos" --out "$PWD/scores/pdmd_4nfe_2500/audio" \
  --av-benchmark "$PWD/av-benchmark" --expect 387

The audio scorer needs a checkout of av-benchmark with its two weight files, once:

git clone https://github.com/hkchengrex/av-benchmark.git
cd av-benchmark
git checkout f351b9a6fc6abde746d5f8e1d4c47c883319cb41
pip install -e .
mkdir -p weights
wget -P weights https://huggingface.co/lukewys/laion_clap/resolve/main/music_speech_audioset_epoch_15_esc_89.98.pt
wget -P weights https://github.com/hkchengrex/MMAudio/releases/download/v0.1/synchformer_state_dict.pth
cd ..

PQ, CE and CU come from Audiobox Aesthetics; IS, IB and DeSync come from av-benchmark, run as in the paper.

The 2-NFE LoRA still shows some painterly, oil-painting-like texture, oversaturation and degraded audio, but as of the end of September 2026 it is, to our knowledge, the only publicly released 2-NFE distilled LoRA for MiniMax-H3. In some scenes its samples are on par with those of the 4-NFE checkpoint, though overall a gap to 4 NFE remains. Below are three prompts, each sampled at 2 NFE and at 4 NFE (with sound); more videos are on the project page.

2 NFE

4 NFE

2 NFE

4 NFE

2 NFE

4 NFE

@misc{wang2026pdmdprojecteddistributionmatching,
      title={PDMD: Projected Distribution Matching Distillation for Video Diffusion Models},
      author={Zimo Wang and Junkun Yuan and Angtian Wang and Haotian Yang and Canyu Zhang and Siyuan Yuan and Xingchang Huang and Bo Liu and Yizhi Wang and Yiding Yang and Chongyang Ma and Gordon Guocheng Qian},
      year={2026},
      eprint={2609.35768},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.35768},
}

We thank the authors of MiniMax-H3 and Diffusers, DiffSynth-Studio and rCM.