RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

[davidlapetina]

Hi,

With the latest version (14.0.3) when doing video generation I got the following error on my Linux server: "RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half"

Any idea what could cause the issue? Thanks.

Here is a full stack.
Generating a 14 frame video from outputs/generated/ghost.png. Device:cuda seed:941192
Loaded weights from /home/david/.cache/huggingface/hub/models--imaginairy--stable-video-diffusion/snapshots/f9dce2757a0713da6262f35438050357c2be7ee6/svd.fp16.safetensors with 0 missing and 0 unexpected keys
EulerEDMSampler: 0%| | 0/26 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/david/.local/lib/python3.10/site-packages/imaginairy/cli/clickshell_mod.py", line 30, in invoke_
command.main(
File "/home/david/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/david/.local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/david/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/david/.local/lib/python3.10/site-packages/imaginairy/cli/videogen.py", line 80, in videogen_cmd
generate_video(
File "/home/david/.local/lib/python3.10/site-packages/imaginairy/api/video_sample.py", line 241, in generate_video
samples_z = model.sampler(denoiser, randn, cond=c, uc=uc)
File "/home/david/.local/lib/python3.10/site-packages/imaginairy/modules/sgm/diffusionmodules/sampling.py", line 126, in call
x = self.sampler_step(
File "/home/david/.local/lib/python3.10/site-packages/imaginairy/modules/sgm/diffusionmodules/sampling.py", line 105, in sampler_step
denoised = self.denoise(x, denoiser, sigma_hat, cond, uc)
File "/home/david/.local/lib/python3.10/site-packages/imaginairy/modules/sgm/diffusionmodules/sampling.py", line 61, in denoise
denoised = denoiser(*self.guider.prepare_inputs(x, sigma, cond, uc))
File "/home/david/.local/lib/python3.10/site-packages/imaginairy/api/video_sample.py", line 235, in denoiser
return model.denoiser(
File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/david/.local/lib/python3.10/site-packages/imaginairy/modules/sgm/diffusionmodules/denoiser.py", line 42, in forward
network(input_tensor * c_in, c_noise, cond, **additional_model_inputs)
File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/david/.local/lib/python3.10/site-packages/imaginairy/modules/sgm/diffusionmodules/wrappers.py", line 30, in forward
return self.diffusion_model(
File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/david/.local/lib/python3.10/site-packages/imaginairy/modules/sgm/diffusionmodules/video_model.py", line 476, in forward
emb = self.time_embed(t_emb)
File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
input = module(input)
File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

[brycedrennan]

Fixed in just released version 14.0.4

[davidlapetina]

Thank you so much!

[davidlapetina]

Hi,

The issue is confirmed fixed on Linux with GPU (CUDA) but when using my Mac with M1 proc (hence using the CPU)
"Apple Silicon MPS (M1, M2, etc) is not currently supported for video generation. Switching to cpu generation."

I still have the same issue "RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half"
return forward_call(*args, **kwargs)
File "/opt/homebrew/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

I tried on my Mac because I just have 8Gb of Memory in my GPU and it seems it is not enough for video generation.

[brycedrennan]

Thanks for the update, I should probably just disable cpu generation as well then.

The issue is that some parameters are in float32 and some in float16. CUDA has an option to paper over issues when numbers are in different formats. The fix was turning that option back on. For CPU and MPS (eventually) it will be necessary to hunt down why some parameters are in a different type and convert them.

I dont think I'm going to have time in the near future to do that.