video-diffusion-avatars (VDA)

Video Diffusion Avatars: Taming Video Diffusion Models into Controllable 3D Avatars · Paper (arXiv) · Project page · Weights on Hugging Face

Expression- and camera-controllable portrait animation on WAN2.2-Animate-14B. Give it a reference image, a driving video and a camera trajectory; it generates the reference person performing the driver's expressions, seen from the requested viewpoints.

Both generate at 288×512, 576×1024 (default) or 720×1280.

The teaser examples, generated with the full model; inset: the reference image.

1. Quick start: requirements, install, weights, examples
2. Your own data: reference image, driving video, camera, generate
3. Long videos: the distilled model
4. Runtime and memory
5. Reference: models, documentation, files, licences, citation

More detail: examples · reference images · driving videos · cameras · commands, paths, Python

<ANGLE_BRACKETS> mark values you choose.

• Linux, NVIDIA driver for CUDA 12.8, conda, ffmpeg, gcc 9-14.
• GPU: 48 GB for the full model (all resolutions) and the fast model at 288p; 80 GB for the fast model at 576p / 720p (4).
• Disk: ~65 GB: weights ~44 GB (ours 6.5 GB + the public WAN files 37 GB), the conda env ~13 GB, preprocessing models and caches ~7 GB.
• A free FLAME account. The installer downloads FLAME 2020 with it (its licence does not allow us to ship the model): it asks for the login in the terminal, or reads FLAME_USERNAME / FLAME_PASSWORD; FLAME_ZIP_DIR=<DIR> uses a FLAME2020.zip you already downloaded. Without a terminal and without these variables it stops with an explanation.

git clone https://github.com/SimonGiebenhain/video-diffusion-avatars.git video-diffusion-avatars
cd video-diffusion-avatars
bash envs/setup_env.sh          # one conda env `vda` (VDA_ENV_NAME=<ENV> to rename it); ~30 min, needs a GPU
conda activate vda

All commands below run in the vda env, from this folder. Each step is a Python script in this folder (download.py, prepare_reference.py, prepare_driving.py, render_conditioning.py, generate.py) that calls the code in vda/; every script has --help.

python download.py weights      # our weights (~6.5 GB) from Hugging Face into weights/, md5-checked
python download.py base         # the public WAN2.2-Animate-14B files (~37 GB) into weights/base/

Our weights, the examples and the example driving videos are on Hugging Face. download.py base fetches from ModelScope; add --source huggingface to use the Hugging Face mirror. To keep the weights elsewhere, export VDA_WEIGHTS=<DIR> before both commands (and in your shell profile).

The 25 examples from the project page come with their conditioning precomputed (reference image, proxy render, skeleton and expression codes), so they run without any preprocessing:

python download.py examples     # ~220 MB into examples/: the 25 examples + the example driving videos
python generate.py --cond examples/teaser_swimmer --out outputs/teaser_swimmer.mp4

This is the swimmer from the teaser, driven by a NeRSemble performance (disgust → happy) while the camera goes over her head. Every example runs the same way; docs/examples.md lists all 25 with their references, drivers and cameras. The distilled_* examples are for the fast model (3).

Three preparation steps, each cached in cache/ under a name you choose: the reference photo (once per person), the driving video (once per video), the conditioning for a camera path; then generate.

bash scripts/run_example.sh <PHOTO.jpg> <NAME> runs all of it for a new photo with the example driving video examples/drivers/disgust_happy.mp4 (--driver <DRIVER.mp4> for another one) and the over_the_top_zoom camera, and writes outputs/<NAME>.mp4.

python prepare_reference.py --image <PHOTO.jpg> --name <NAME>    # ~3 min; --min_conf 0.25 for paintings, cartoons, statues

Any near-frontal portrait with some margin around the head. It is cropped to 720×1280 with the face at a fixed size and position, and its FLAME head shape is fitted; both are cached in cache/refs/<NAME>/. What makes a good reference, the crop convention and the shipped references: docs/reference_images.md.

Any video of one face, roughly frontal. The driving videos of the project page come with the examples, in examples/drivers/: disgust_happy, tongue_out, shout_laugh, jaw, surprise_fear and cheeks_nose (NeRSemble performances, 3.4 s each).

python prepare_driving.py --video examples/drivers/tongue_out.mp4 --name tongue_out    # or --video <DRIVER.mp4> --name <DRV>

This tracks the expression, jaw, gaze and blinks in the video and transfers them onto a canonical person, whose head and upper body move naturally (~17 min for these 81-frame videos on an RTX A6000). The result is cached in cache/driving/<DRV>/. To transfer the driver's own head and body motion as well, see docs/driving.md (--motion driver, with SAM 3D Body).

python render_conditioning.py --ref <NAME> --driving tongue_out --camera over_the_top_zoom --out_dir cache/cond/<RUN>

Renders the conditioning (proxy render, skeleton, expression codes) for the reference <NAME> (2.1) and the driving video tongue_out (2.2) into the chosen camera path; <RUN> is any name for the result. The full model uses the first 77 frames: for a driving video longer than that, add --max_frames 77, otherwise the camera path is stretched over the whole video and the clip shows only its beginning.

Recommended cameras: over_the_top_zoom (over the head and back, with zoom), over_the_top_zoom_55 (gentler, for drivers who move a lot), ladder_zoom_m (above → profile → below). All 17 presets with a figure, and how to write your own path: docs/cameras.md. The training cameras cover ±65° azimuth and ±17° elevation; quality degrades gradually beyond that.

python generate.py --cond cache/cond/<RUN> --out <OUT.mp4>

The full model generates the first 77 frames (3.2 s) of the conditioning; a shorter one gives the largest 4k+1 frames (e.g. 69 of 71). For longer videos use the fast model (3).

The full model makes one 77-frame clip (3.2 s) per call. For longer videos use the fast model, a distilled 4-step student that streams the video in 16-frame blocks on a KV cache, any length at ~3.5 fps (288×512, A100, including decoding). It starts with a short pass of its full-quality teacher for the first 77 frames, which keeps the identity stable.

python generate.py --model fast --cond examples/distilled_redbangs --out outputs/distilled_redbangs.mp4 --deflicker

41 s, camera swinging between the sides. --deflicker also writes outputs/distilled_redbangs_dfs.mp4: the student can shift the colour slightly from one 16-frame block to the next, and the deflicker pass measures these jumps on a coarse colour grid and fades them out over a few frames (a post-process; recommended).

The distilled_* examples use the long driving sequence nersemble8 that comes with the weights: eight NeRSemble performances joined by short transitions, 41 s in total (docs/driving.md). With your own reference:

python render_conditioning.py --ref <NAME> --driving nersemble8 --camera sweep_41s --out_dir cache/cond/<RUN>
python generate.py --model fast --cond cache/cond/<RUN> --out <OUT.mp4> --deflicker

A long driving video of your own works the same way (--driving <DRV> from 2.2); the camera path is cut to its length. The fast model generates at 288×512 by default (--width 576 --height 1024 needs 80 GB). It can drift in colour on dark backgrounds after ~20 s; another --seed usually helps.

Measured with this code unless noted. Streamed = DiT blocks (of 40) kept in host memory and copied to the GPU for each forward pass (--offload_blocks, chosen automatically, see below).

The fast model always starts with a one-off "teacher" pass for the first 77 frames; the longer the video, the smaller its share.

GPU memory. Streaming blocks lowers peak VRAM by ~0.8 GB per block and gives exactly the same video, but every denoising step waits for the copies, so it is slower. generate.py picks the smallest setting that fits:

On GPUs with more memory nothing is streamed and generation runs at full speed. Override with --offload_blocks <N>: lower N if your GPU has memory to spare (faster), raise it if you run out of memory (torch.OutOfMemoryError). When blocks are streamed the log says so ([vda] N/40 DiT blocks streamed from host memory; resident X GiB); the offload_blocks entry in <OUT>.json always records what a run used.

Preprocessing (any CUDA GPU ≥ 12 GB): reference 2-3 min; driving video ~17 min for 81 frames (RTX A6000, mostly the per-frame face tracking); conditioning ~1 min per 1000 frames.

Full. WAN2.2-Animate-14B (frozen) + vda_full.safetensors. 20 Euler steps with joint identity + expression guidance (scale 1.5, 2 passes per step). 77 frames per call.

Fast. WAN2.2-Animate-14B + vda_fast_base.safetensors + vda_fast_student.safetensors. A causal 4-step student (distilled with Self Forcing) generates 16-frame blocks on a KV cache. It attends to the reference and to a window of 20 latent frames (77 video frames, the trained clip length), of which the first 12 stay fixed as an attention sink; the first 77 frames come from its bidirectional teacher. The student was distilled from a separately fine-tuned adapter of the same architecture (vda_fast_base.safetensors, also its teacher), hence its own base file.

Conditioning. Per frame, in the target camera: the proxy render (pose_proxy.mp4, a FLAME head with the reference's shape and the driver's expression on a body, shaded by view direction), the 2D skeleton (pose_skel.mp4) and the driver's X-NeMo expression codes (codes.npy). Inside the model the codes pass a frozen expression disentangler (docs/python.md) and a translator into WAN-Animate's motion-code space; the proxy render enters through its own patch embedding next to the skeleton.

The code and the model weights of this repository are released under CC BY-NC-SA 4.0 (see LICENSE). Third-party components keep their own licences:

@article{giebenhain2026vda,
  title   = {Video Diffusion Avatars: Taming Video Diffusion Models into Controllable 3D Avatars},
  author  = {Giebenhain, Simon and Kirschstein, Tobias and Schoneveld, Liam and
             Davoli, Davide and Chen, Zhe and Nie{\ss}ner, Matthias},
  journal = {arXiv preprint arXiv:2610.23772},
  eprint  = {2610.23772},
  archivePrefix = {arXiv},
  url     = {https://arxiv.org/abs/2610.23772},
  year    = {2026}
}