unet/unet_3d_blocks/.py - dimension size mismatche

[HoseinRanjbar]

Describe the bug

I encountered a RuntimeError while running the pipeline in the ControlNext-SVD-v2 project. When I set the resolution of the video except for certain numbers, I got the following error. The error is related to tensor size mismatches during the forward pass in the unet_3d_blocks module. Below is the full traceback of the error:

Reproduction

Clone the ControlNext-SVD-v2 repository.
Setup the required environment by installing dependencies as specified in the project.
Run the script run_controlnext.py.
The error appears during the pipeline execution.

Logs

Traceback (most recent call last):
  File "/ControlNext/ControlNext-SVD-v2/run_controlnext.py", line 274, in <module>
    video_frames = pipeline(
  File "/controlNext/.env/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/ControlNext/ControlNext-SVD-v2/pipeline/pipeline_stable_video_diffusion_controlnext.py", line 588, in __call__
    noise_pred = self.unet(
  File "/controlNext/.env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/controlNext/.env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/controlNext/.env/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/ControlNext/ControlNext-SVD-v2/models/unet_spatio_temporal_condition_controlnext.py", line 491, in forward
    sample = upsample_block(
  File "/controlNext/.env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/controlNext/.env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/controlNext/.env/lib/python3.12/site-packages/diffusers/models/unets/unet_3d_blocks.py", line 1494, in forward
    hidden_states = torch.cat([hidden_states, res_hidden_states], dim=1)

RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 26 but got size 25 for tensor number 1 in the list.

System Info

Ubuntu 22

Who can help?

No response

[hlky]

When I set the resolution of the video except for certain numbers, I got the following error.

Can you share more details of the parameters you're using that give the error?

[HoseinRanjbar]

When I set the resolution of the video except for certain numbers, I got the following error.

Can you share more details of the parameters you're using that give the error?

for example 1024x576 and 768x512 are okay, but 1024x800 is not.

[hlky]

This suggests resolution must be a multiple of 64.

cc @DN6 @sayakpaul is this a known limitation of SVD? check_inputs says multiple of 8, so it may be an issue with the ControlNeXt-SVD models or pipeline

diffusers/src/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py

Lines 311 to 323 in d041dd5

	def check_inputs(self, image, height, width):
	if (
	not isinstance(image, torch.Tensor)
	and not isinstance(image, PIL.Image.Image)
	and not isinstance(image, list)
	):
	raise ValueError(
	"`image` has to be of type `torch.Tensor` or `PIL.Image.Image` or `List[PIL.Image.Image]` but is"
	f" {type(image)}"
	)
	if height % 8 != 0 or width % 8 != 0:
	raise ValueError(f"`height` and `width` have to be divisible by 8 but are {height} and {width}.")

[sayakpaul]

Yeah that could be.

[hlky]

#6646 related

[DN6]

Yeah, I believe SVD currently only supports multiples of 64. We would need to either update check_inputs since that check is a wrong (but it could be a breaking change) or take over #6646 to finish adding support for multiples of 8.

[hlky]

#6646 is good to go, conflicts were just from when type hints changed FloatTensor->Tensor, tested with the reproducer from the PR. Downstream will need to adopt the changes from src/diffusers/models/unets/unet_spatio_temporal_condition.py that pass in the upsample size.

[HoseinRanjbar]

the error is solved when I set the resolution to a multiple of 64.

[hlky]

Thanks for confirming @HoseinRanjbar. We now support multiple of 8 in SVD, ControlNeXt-SVD can adopt the changes in here for unet_spatio_temporal_condition_controlnext.py. It will require installing Diffusers from source until we release 0.32.

[hlky]

Closed by #6646 on our end. Reopen if there are further issues after the changes are adopted downstream.