ValueError: Attempting to unscale FP16 gradients.

[qinchangchang]

Describe the bug

Traceback (most recent call last):
File "train_text_to_image_lora.py", line 950, in
main()
File "train_text_to_image_lora.py", line 777, in main
accelerator.clip_grad_norm_(params_to_clip, args.max_grad_norm)
File "/home/qc/miniconda3/envs/lora-diffussion/lib/python3.8/site-packages/accelerate/accelerator.py", line 2040, in clip_grad_norm_
self.unscale_gradients()
File "/home/qc/miniconda3/envs/lora-diffussion/lib/python3.8/site-packages/accelerate/accelerator.py", line 2003, in unscale_gradients
self.scaler.unscale_(opt)
File "/home/qc/miniconda3/envs/lora-diffussion/lib/python3.8/site-packages/torch/cuda/amp/grad_scaler.py", line 307, in unscale_
optimizer_state["found_inf_per_device"] = self.unscale_grads(
File "/home/qc/miniconda3/envs/lora-diffussion/lib/python3.8/site-packages/torch/cuda/amp/grad_scaler.py", line 229, in unscale_grads
raise ValueError("Attempting to unscale FP16 gradients.")
ValueError: Attempting to unscale FP16 gradients.

Reproduction

export MODEL_NAME="CompVis/stable-diffusion-v1-4"
export DATASET_NAME="lambdalabs/pokemon-blip-captions"
accelerate launch --mixed_precision="fp16" train_text_to_image_lora.py
--pretrained_model_name_or_path=$MODEL_NAME
--dataset_name=$DATASET_NAME --caption_column="text"
--resolution=512 --random_flip
--train_batch_size=1
--num_train_epochs=100 --checkpointing_steps=5000
--learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0
--seed=42
--output_dir="sd-pokemon-model-lora"
--validation_prompt="cute dragon creature" --report_to="wandb"

Logs

No response

System Info

• diffusers version: 0.25.0.dev0
• Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.10
• Python version: 3.8.0
• PyTorch version (GPU?): 2.1.2+cu121 (True)
• Huggingface_hub version: 0.20.1
• Transformers version: 4.36.2
• Accelerate version: 0.25.0
• xFormers version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

No response

[sayakpaul]

We recently fixed it with #6388. Could you please pull in the changes and make sure to use the following:

accelerate launch train_text_to_image_lora.py
--mixed_precision="fp16"
--pretrained_model_name_or_path=$MODEL_NAME
--dataset_name=$DATASET_NAME --caption_column="text"
--resolution=512 --random_flip
--train_batch_size=1
--num_train_epochs=100 --checkpointing_steps=5000
--learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0
--seed=42
--output_dir="sd-pokemon-model-lora"
--validation_prompt="cute dragon creature" --report_to="wandb"

[qinchangchang]

Thank you! I solved the above problem based on your suggestions. Modified train_text_to_image_lora.py according to 6388(1afce49).

[sayakpaul]

Cool, feel free to submit a PR and close this PR :)

[kopyl]

@sayakpaul i still see this issue. Traceback: https://pastebin.com/cTs3TB1m

Launching code:

!accelerate launch --mixed_precision="fp16" train_text_to_image_lora.py \
  --pretrained_model_name_or_path=lambdalabs/miniSD-diffusers \
  --dataset_name=kopyl/3M_icons_monochrome_only_no_captioning \
  --resolution=256 \
  --train_batch_size=2 \
  --max_train_steps=10000000 \
  --checkpointing_steps=100 \
  --learning_rate=1e-4 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --output_dir="/workspace/train-checkpoints/lora" \
  --noise_offset=0.05 \
  --cache_dir="/workspace/dataset-cache" \
  --report_to="wandb" \
  --validation_prompt "cat icon" \
  --seed=1

Putting --mixed_precision="fp16" after the script name does not help either – same error.

@qinchangchang what i did – i only made the change which was in this commit: 1afce49 . What else should be done for it to work?

[kopyl]

@qinchangchang Also how did you fix this issue on validation?

RuntimeError: Input type (float) and bias type (c10::Half) should be the same

when calling pipeline()...

[kopyl]

@qinchangchang i see... with torch.cuda.amp.autocast():

[kopyl]

For anyone wondering how to do the same: https://github.com/kopyl/diffusers/pull/18/commits

I needed to fix it myself in the code since I have a bit of a custom trainer

[sayakpaul]

I didn't have do anything other than what's already there in the training script.

See here: https://colab.research.google.com/gist/sayakpaul/ce4eb75b1e5751f4d389a086ca2880a2/scratchpad.ipynb

[kopyl]

@sayakpaul but the training script had to be fixed in the first place

[FOLSc]

Even when running the latest version of the ”train_text_to_image_lora.py“ script from GitHub, it still throws an error of Attempting to unscale FP16 gradients.