[Bug] WanVideoToVideoPipeline fails due to missing handling of control arguments required by underlying VACE transformer

[Mathias5]

Describe the bug

Problem:
WanVideoToVideoPipeline.call does not expose control_hidden_states or control_hidden_states_scale in its signature.
However, the underlying TransformerWanVACE.forward() requires them and assumes they’re non-None.

This leads to a crash every time the pipeline is used:

pipe(video=my_frames, prompt="studio lighting")
# -> AttributeError: 'NoneType' object has no attribute 'new_ones'

and you can’t fix it by passing the tensors, because the pipeline doesn’t accept those kwargs:

pipe(video=my_frames, prompt="...", control_hidden_states=...)
# -> TypeError: __call__() got an unexpected keyword argument 'control_hidden_states'

Suggested fix:
Expose control_hidden_states and control_hidden_states_scale in WanVideoToVideoPipeline.call, or set neutral defaults internally when not provided.

Reproduction

from diffusers import WanVideoToVideoPipeline, AutoencoderKLWan
from PIL import Image
import torch

# 1. Load the model and VAE
vae = AutoencoderKLWan.from_pretrained(
    "Wan-AI/Wan-VACE-14B",
    subfolder="vae",
    torch_dtype=torch.float32,
)
pipe = WanVideoToVideoPipeline.from_pretrained(
    "Wan-AI/Wan-VACE-14B",
    vae=vae,
    torch_dtype=torch.bfloat16,
).to("cuda")

# 2. Create a dummy video (8 identical frames)
frames = [Image.new("RGB", (512, 288), "gray") for _ in range(8)]

# 3. Run baseline inference
with torch.autocast("cuda", dtype=torch.bfloat16):
    result = pipe(video=frames, prompt="studio lighting")

Expected behavior:
The call should run inference normally, producing an output video.

Actual behavior:
It crashes with:
AttributeError: 'NoneType' object has no attribute 'new_ones'

Logs

Traceback (most recent call last):
  File "repro_vace_crash.py", line 21, in <module>
    result = pipe(video=frames, prompt="studio lighting")
  File ".../diffusers/pipelines/wan/pipeline_wan_video2video.py", line 663, in __call__
    noise_pred = self.transformer(
  File ".../torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File ".../diffusers/models/transformers/transformer_wan_vace.py", line 295, in forward
    control_hidden_states_scale = control_hidden_states.new_ones(len(self.config.vace_layers))
AttributeError: 'NoneType' object has no attribute 'new_ones'

System Info

• 🤗 Diffusers version: 0.35.2
• Platform: Linux-6.8.0-1036-aws-x86_64-with-glibc2.35
• Running on Google Colab?: No
• Python version: 3.12.10
• PyTorch version (GPU?): 2.8.0+cu128 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 0.36.0
• Transformers version: 4.57.1
• Accelerate version: 1.11.0
• PEFT version: 0.17.1
• Bitsandbytes version: 0.48.1
• Safetensors version: 0.6.2
• xFormers version: 0.0.32.post2
• Accelerator: NVIDIA H100 80GB HBM3, 81559 MiB
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

Who can help?

No response

[Mathias5]

I’m currently preparing a PR to address this issue.
The fix adds optional control_hidden_states and control_hidden_states_scale arguments to WanVideoToVideoPipeline.__call__(), with neutral zero/one fallbacks to prevent the crash when VACE control inputs are not provided.

I’ll open the PR shortly and link it here once submitted.

[Mathias5]

Hi! 👋
I’ve opened a PR that addresses this issue: [#12576]

The PR makes WanVideoToVideoPipeline handle control_hidden_states and control_hidden_states_scale safely by:

• Adding them as optional kwargs to __call__,
• Auto-injecting neutral tensors when not provided,
• Skipping them gracefully when the transformer doesn’t support control inputs.

This prevents the 'NoneType' object has no attribute 'new_ones' crash while keeping full backward compatibility.

PR includes updated docstrings and unit tests for both CFG and non-CFG latent paths.

Thanks! 🙌

[DN6]

Hi @Mathias5 why can't we use the WanVACEPipeline directly here?

[Mathias5]

Hi @Mathias5 why can't we use the WanVACEPipeline directly here?

WanVACEPipeline is focused on text-to-video generation - it doesn’t take an input video as conditioning. The “control” inputs it supports (control_hidden_states, etc.) are compact feature maps derived from pre-encoded signals (like motion or layout), not raw video frames. So it can’t actually perform video-to-video transformation.

WanVideoToVideoPipeline, on the other hand, implements the full video-to-video flow:

• It VAE-encodes real input frames,
• Adds noise proportional to strength,
• Denoises using the Wan transformer,
• And finally decodes back to a video.

That makes it the right pipeline for tasks like relighting, stylization, or restoration - where you want to transform an existing clip rather than generate one from scratch.

Currently the V2V pipeline uses the same underlying VACE transformer, which expects control_hidden_states tensors. Without proper handling, inference crashes even in standard cases (no external control).

[DN6]

VACE supports video input

diffusers/src/diffusers/pipelines/wan/pipeline_wan_vace.py

Line 695 in dcfb18a

video: Optional[List[PipelineImageInput]] = None,

[Mathias5]

You’re right my mistake, I initially misunderstood that part. WanVACEPipeline already supports direct video input.

Given that, it’s not clear whether WanVideoToVideoPipeline still serves a distinct purpose - it currently just wraps VACE but doesn’t properly handle the control arguments which leads to inference failures. Should we still use or maintain it going forward?