Stable Cascade Diffusers not working with FP16

[FurkanGozukara]

You know that there is a demo released here : https://huggingface.co/spaces/multimodalart/stable-cascade

I am using it locally. Works perfectly fine

But I have to change float type to float16 from bfloat16 on Kaggle

That is where error starts

\venv\lib\site-packages\diffusers\utils\pil_utils.py:43: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")

I tested on my local computer and fp16 fails - bf16 works

I tested float32 on Kaggle and works but ultra slow

the error causing file : kashif@d67eba0#commitcomment-138598496

[FurkanGozukara]

@DN6 @patrickvonplaten @sayakpaul @kashif

using this version of diffusers : https://github.com/kashif/diffusers/tree/diffusers-yield-callback

[FurkanGozukara]

@multimodalart

People asking me to use with like 1080 TI which has 12 GB VRAM but FP16 support lacking right now

[FurkanGozukara]

I tested (https://github.com/kashif/diffusers/tree/wuerstchen-v3) and FP16 still not working

inference steps fixed

R:\temp1\StableCascade\venv\lib\site-packages\diffusers\utils\pil_utils.py:43: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")

[Disty0]

Prior model itself is borked again just like the SDXL VAE. StabilityAI releases doesn't surprise me with borked FP16 anymore.

[FurkanGozukara]

can you elaborate what you mean by borked FP16 @Disty0

[Disty0]

can you elaborate what you mean by borked FP16 @Disty0

The model is trained using BF16 and saved as BF16.
Training on BF16 can cause the model to go outside of FP16's range and cause invalid numbers / NaNs / black images when used with FP16.
And this is the case with Stable Cascade Prior and SDXL Base VAE.

[FurkanGozukara]

can you elaborate what you mean by borked FP16 @Disty0

The model is trained using BF16 and saved as BF16. Training on BF16 can cause the model to go outside of FP16's range and cause invalid numbers / NaNs / black images when used with FP16. And this is the case with Stable Cascade Prior and SDXL Base VAE.

i see . for SDXL vae people casted it to FP16 and now working. but i get it now

[kashif]

yes @FurkanGozukara i will double-check with the authors on the best course of action here

[FurkanGozukara]

yes @FurkanGozukara i will double-check with the authors on the best course of action here

we made a dirty fix and made it work but proper fix is mandatory for best performance