Black image on SDXL 1.0 + RuntimeWarning: invalid value encountered in cast images = (images * 255).round().astype("uint8")

[belfortf]

Describe the bug

I'm trying to run SDXL in a container environment (Debian). I tried both diffusers[torch]==0.18.0 and diffusers[torch]==0.19.0 with Python 3.10 on A10g and A100 but I get a black image back and RuntimeWarning: invalid value encountered in cast
images = (images * 255).round().astype("uint8") when running 0.18.0.

Basically, when building the container's image I'm doing:

 pipe = diffusers.DiffusionPipeline.from_pretrained(
        model_id,
        use_auth_token=hugging_face_token,
        torch_dtype=torch.float16,
        use_safetensors=True, 
        variant="fp16",
        cache_dir=cache_path
    )
pipe.save_pretrained(cache_path, safe_serialization=True)

After that, when running the function I'm calling:

self.pipe = diffusers.DiffusionPipeline.from_pretrained(
            cache_path).to("cuda")

With 0.19.0 I'm getting Out of memory errors. .enable_model_cpu_offload() didn't work for me. Anyway, with both versions in the end I get a black image.

Reproduction

pipe = diffusers.DiffusionPipeline.from_pretrained(
        model_id,
        use_auth_token=hugging_face_token,
        torch_dtype=torch.float16,
        use_safetensors=True, 
        variant="fp16",
        cache_dir=cache_path
    )
pipe.save_pretrained(cache_path, safe_serialization=True)
    
pipe = diffusers.DiffusionPipeline.from_pretrained(
            cache_path).to("cuda")

Logs

The config attributes {'force_upcast': True} were passed to AutoencoderKL, but are not expected and will be ignored. Please verify your config.json configuration file.
 98%|█████████▊| 49/50 [00:10<00:00,  5.28it/s]100%|██████████| 50/50 [00:10<00:00,  5.27it/s]100%|██████████| 50/50 [00:10<00:00,  4.90it/s]
/usr/local/lib/python3.10/site-packages/diffusers/image_processor.py:65: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")

System Info

0.18.0 and 0.19.0. Linux Debian (don't have access to the container now).

Who can help?

@patrickvonplaten

[patrickvonplaten]

Hmm, interesting that you got this warning:

The config attributes {'force_upcast': True} were passed to AutoencoderKL, but are not expected and will be ignored.

This means that you are probably not using diffusers 0.19.0 but an older version because diffusers has force_upcast added here:

diffusers/src/diffusers/models/autoencoder_kl.py

Line 90 in 9611730

force_upcast: float = True,

[patrickvonplaten]

Are you sure that you have installed the correct diffusers version?

[levi]

I am seeing this too with the same configuration as OP. Will debug further.

[levi]

OK, I am only seeing this when generating validation images for wandb using the dreambooth_sdxl_lora training script. Using master version of diffusers and the training script. Here are my params:

"pretrained_model_name_or_path": "stabilityai/stable-diffusion-xl-base-1.0",
"instance_data_dir": "/local/images",
"output_dir": "/lora/path",
"mixed_precision": "fp16",
"instance_prompt": "a man",
"resolution": 1024,
"train_batch_size": 1,
"gradient_accumulation_steps": 4,
"learning_rate": 0.001,
"report_to": "wandb",
"lr_scheduler": "constant",
"lr_warmup_steps": 0,
"max_train_steps": 500,
"validation_prompt": f"An illustration of a man",
"validation_epochs": 25,
"seed": 0,

All the images in wandb turn out black. I am running this in a container, so there is no pre-cached model I'm using. I looked into the pipeline that the training script uses for validation images and there's nothing out of the ordinary. Here's the output:

Loaded scheduler as EulerDiscreteScheduler from `scheduler` subfolder of stabilityai/stable-diffusion-xl-base-1.0.
Loaded tokenizer as CLIPTokenizer from `tokenizer` subfolder of stabilityai/stable-diffusion-xl-base-1.0.
Loaded tokenizer_2 as CLIPTokenizer from `tokenizer_2` subfolder of stabilityai/stable-diffusion-xl-base-1.0.]
Loading pipeline components...: 100%|██████████| 7/7 [00:00<00:00, 36.25it/s]
torch_dtype: torch.float16, device: cuda
{'algorithm_type', 'lambda_min_clipped', 'dynamic_thresholding_ratio', 'solver_order', 'thresholding', 'variance_type', 'solver_type', 'lower_order_final'} was not found in config. Values will be initialized to default values.
/usr/local/lib/python3.9/site-packages/diffusers/image_processor.py:65: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")

[levi]

Update, it's only the validation inference that's causing this issue. The final LoRA model works fine.

[levi]

The solution was to switch out the VAE to (https://huggingface.co/madebyollin/sdxl-vae-fp16-fix) based on this note: https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_sdxl.md#specifying-a-better-vae

[belfortf]

Hi, thanks for all the answers. Revisiting my post with 0.19.1:

I'm specifying "diffusers[torch]==0.19.1" and "torch==2.0.1+cu117". The pipe in the image gets constructed like this:

 pipe = diffusers.DiffusionPipeline.from_pretrained(
        model_id,
        use_auth_token=hugging_face_token,
        torch_dtype=torch.float16,
        use_safetensors=True, 
        variant="fp16",
        cache_dir=cache_path
    )
    pipe.save_pretrained(cache_path, safe_serialization=True)

When calling the pipe:

self.pipe = diffusers.DiffusionPipeline.from_pretrained(
            cache_path
 )

If I run the inference function like this, I get:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 144.00 MiB (GPU 0; 22.05 GiB total capacity; 19.61 GiB already allocated; 95.12 MiB free; 20.54 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

By adding self.pipe.enable_model_cpu_offload() to my code as per documentation, I can run inference but then I get back a black image and this error / warning:

/usr/local/lib/python3.10/site-packages/diffusers/image_processor.py:65: RuntimeWarning: invalid value encountered in cast images = (images * 255).round().astype("uint8")

Printing the pipe to console I get:

StableDiffusionXLPipeline {
"_class_name": "StableDiffusionXLPipeline",
"_diffusers_version": "0.19.1",
"_name_or_path": "stabilityai/stable-diffusion-xl-base-1.0",
"force_zeros_for_empty_prompt": true,
"scheduler": [
"diffusers",
"EulerDiscreteScheduler"
],
"text_encoder": [
"transformers",
"CLIPTextModel"
],
"text_encoder_2": [
"transformers",
"CLIPTextModelWithProjection"
],
"tokenizer": [
"transformers",
"CLIPTokenizer"
],
"tokenizer_2": [
"transformers",
"CLIPTokenizer"
],
"unet": [
"diffusers",
"UNet2DConditionModel"
],
"vae": [
"diffusers",
"AutoencoderKL"
]
}

I'm a bit lost here regarding what to try next.

[belfortf]

Updating the vae as @levi suggested works well ! However, only with pipe.enable_model_cpu_offload(). I don't get it why I would be getting

OutOfMemoryError: CUDA out of memory. Tried to allocate 144.00 MiB (GPU 0; 22.05 GiB total capacity; 19.61 GiB already
allocated; 69.12 MiB free; 20.54 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting
max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

as this is an A10g gpu and I actually get the same error on an A100 gpu. Is some memory management not working properly here on torch's side ?

Overall, I also don't understand why I had to swap out the VAE to make it work.

[patrickvonplaten]

Can you send a fully reproducible code snippet? Ideally in Python mark down like this:

class GoodFormat:
        example = True

good_format = {}

instead of this:

class GoodFormat:
        example = True

good_format = {}

=> So it would be nice to be able to just copy-paste your code example into a Python intepreter. In the code above I don't see enable_model_cpu_offload() being used anywhere.

[belfortf]

Hi @patrickvonplaten , I quickly converted my script to run it on a google colab. On a V100 the result with cuda enabled is:

OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 15.77 GiB total capacity; 14.18 GiB already allocated; 10.12 MiB free; 14.50 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

!pip install diffusers[torch]==0.19.1
!pip install accelerate ftfy torchvision transformers triton safetensors invisible_watermark
!pip install torch==2.0.1
!pip install xformers

cache_path = "/vol/cache"

def download_models():
    import diffusers
    import torch

    model_id = 'stabilityai/stable-diffusion-xl-base-1.0'

    vae = diffusers.AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
    #hugging_face_token = os.environ["HUGGINGFACE_TOKEN"]
    pipe = diffusers.DiffusionPipeline.from_pretrained(
        model_id,
        vae=vae,
        #use_auth_token=hugging_face_token,
        torch_dtype=torch.float16,
        use_safetensors=True,
        variant="fp16",
        cache_dir=cache_path
    )
    print(pipe)
    pipe.save_pretrained(cache_path, safe_serialization=True)

class StableDiffusion:
    def __enter__(self):
        import diffusers
        import torch
        
        torch.backends.cuda.matmul.allow_tf32 = True

        self.pipe = diffusers.DiffusionPipeline.from_pretrained(
            cache_path
        ).to("cuda")

        return self
        # self.pipe.enable_model_cpu_offload()
        # self.pipe.unet = torch.compile(self.pipe.unet, mode="reduce-overhead", fullgraph=True)
        
        # self.pipe.enable_xformers_memory_efficient_attention()

    def __exit__(self, exc_type, exc_value, traceback):
        pass 

    def run_inference(
        self, 
        prompt: str, 
        num_inference_steps: int = 50, 
        negative_prompt: str = "", 
        seed: int = None, 
        ) -> list[bytes]:
        import torch
        import random
        from PIL import Image as PilImage
        import datetime
        import base64

        if seed is None:
            seed = random.randint(1, 10000000)
        
        generator = torch.Generator(device="cuda").manual_seed(seed)
        
        with torch.inference_mode():
            with torch.autocast("cuda"):
                images = self.pipe(
                    prompt,
                    num_inference_steps=num_inference_steps,
                    guidance_scale=7.5,
                    width=768,  
                    height=768,
                    generator=generator,
                    negative_prompt=negative_prompt
                ).images
                
                
download_models()
with StableDiffusion() as sd:
    prompt = "A sunset over the ocean"
    images = sd.run_inference(prompt, num_inference_steps=50, seed=123)
    

[belfortf]

The following code works on a V100, however it takes 40 seconds to generate an image:

!pip install diffusers[torch]==0.19.1
!pip install accelerate ftfy torchvision transformers triton safetensors invisible_watermark
!pip install torch==2.0.1
!pip install xformers

cache_path = "/vol/cache"

def download_models():
    import diffusers
    import torch

    model_id = 'stabilityai/stable-diffusion-xl-base-1.0'

    vae = diffusers.AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
    #hugging_face_token = os.environ["HUGGINGFACE_TOKEN"]
    pipe = diffusers.DiffusionPipeline.from_pretrained(
        model_id,
        vae=vae,
        #use_auth_token=hugging_face_token,
        torch_dtype=torch.float16,
        use_safetensors=True,
        variant="fp16",
        cache_dir=cache_path
    )
    print(pipe)
    pipe.save_pretrained(cache_path, safe_serialization=True)

class StableDiffusion:
    def __enter__(self):
        import diffusers
        import torch
        
        torch.backends.cuda.matmul.allow_tf32 = True

        self.pipe = diffusers.DiffusionPipeline.from_pretrained(
            cache_path
        )
        self.pipe.enable_model_cpu_offload()
        return self
     
        # self.pipe.unet = torch.compile(self.pipe.unet, mode="reduce-overhead", fullgraph=True)
        
        # self.pipe.enable_xformers_memory_efficient_attention()

    def __exit__(self, exc_type, exc_value, traceback):
        pass 

    def run_inference(
        self, 
        prompt: str, 
        num_inference_steps: int = 50, 
        negative_prompt: str = "", 
        seed: int = None, 
        ) -> list[bytes]:
        import torch
        import random
        from PIL import Image as PilImage
        import datetime
        import base64

        if seed is None:
            seed = random.randint(1, 10000000)
        
        generator = torch.Generator(device="cuda").manual_seed(seed)
        
        images = self.pipe(
                    prompt,
                    num_inference_steps=num_inference_steps,
                    guidance_scale=7.5,
                    width=768,  
                    height=768,
                    generator=generator,
                    negative_prompt=negative_prompt
                ).images
                
        return images[0]
download_models()
with StableDiffusion() as sd:
    prompt = "A sunset over the ocean"
    images = sd.run_inference(prompt, num_inference_steps=50, seed=123)
    

[belfortf]

Running the same script without the VAE @levi mentioned works just fine on Google Colab (so no black image) but not when using my container

[patrickvonplaten]

I see could you try to remove:

        with torch.inference_mode():
            with torch.autocast("cuda"):

autocast is strongly discouraged when using pure fp16 (it leads to slow downs). Maybe also see: #2568

Can you try removing this part?

[patrickvonplaten]

Ah ok the last code snippet seems correct. Note that you can speed it up by using less inference steps and potentially a better scheduler such as: https://huggingface.co/docs/diffusers/api/schedulers/unipc