pipeline does not work with conversion to fp16 in the to cuda call

[raulc0399]

Hi,

i was using and doing some fine-tuning with the Pixart model.
and i found a problem that i could pin it down to:

this code works:

pipe2 = PixArtAlphaPipeline.from_pretrained("PixArt-alpha/PixArt-XL-2-512x512", torch_dtype=torch.float16)
pipe2.to("cuda")
image = pipe2(prompt, num_inference_steps=20).images[0]    

while this does not, it gives out just noise images :

pipe1 = PixArtAlphaPipeline.from_pretrained("PixArt-alpha/PixArt-XL-2-512x512")
pipe1.to("cuda", dtype=torch.float16)
image = pipe1(prompt, num_inference_steps=20).images[0]    

i have looked through the code, and i cannot find any issues so far.
has anyone encountered a similar problem or has tips where to look further?

i am on WSL Ubuntu, diffusers==0.27.2, torch==2.2.2

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[a-r-r-o-w]

Hi, sorry for the late response. Is this still an issue, and if so, could you re-open the issue due to long term of inactivity? AFAIU, changing dtype should not affect inference in a way where fp32 gives good results but fp16 doesn't. Could you share the stack trace resulting in the error as well?

[yiyixuxu]

I think it is related #8604

[raulc0399]

will check it again.
there was no error, just noisy image.

the issue only appears when dtype=torch.float16 is specified in the to call

[sayakpaul]

Is this only relevant for PixArt or does it happen for different pipelines too?

[raulc0399]

so i checked it again, and the problem is still there.
is gives out the following message.

diffusers/diffusers_env/lib/python3.10/site-packages/diffusers/image_processor.py:111: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")

interesting that specifying the dtype in from_pretrained it works, but in to it doesnt

[asomoza]

Hi, as @yiyixuxu pointed out, this is related to that issue and a couple of more issues that are the same. This happens to all the pipelines that use the T5 and is a problem with that model. If you change the precision with the to() it generates different embeddings which makes the model output noise.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.