Problem with "Gradient checkpointing"

[ivanced09]

People, I have the following problem when I use the mentioned option both in training with 1.5 and with XL

Additionally, I have been seeing the following two warnings for a couple of updates, I don't see that they affect me but I would be interested to know if there are any settings that I should modify

WARNING because max_grad_norm is set, clip_grad_norm is enabled. consider set to 0 / train_util.py:3807
max_grad_normが設定されているためclip_grad_normが有効になります。0に設定して無効にしたほうがいいかもしれません
WARNING constant_with_warmup will be good / スケジューラはconstant_with_warmupが良いかもしれません train_util.py:3811

My training setup is as follows:

accelerate launch --gpu_ids="0" --num_cpu_threads_per_process=2 "E:\kohya_ss/sd-scripts/sdxl_train_network.py" --bucket_no_upscale --bucket_reso_steps=64 --cache_latents --cache_latents_to_disk --caption_extension=".txt" --flip_aug --gradient_checkpointing --learning_rate="0.0001" --lr_scheduler="constant" --lr_scheduler_num_cycles="1" --max_data_loader_n_workers="0" --max_grad_norm="1" --resolution="1024,1024" --max_train_steps="1920" --mixed_precision="bf16" --network_alpha="8" --network_dim=16 --network_module=networks.lora --no_half_vae --optimizer_args scale_parameter=False relative_step=False warmup_init=False --optimizer_type="Adafactor" --output_dir="D:\Lora work\Metaverso\Elizabeth9\model" --output_name="cara delevingne woman-XL01A" --pretrained_model_name_or_path="E:/webui_forge_cu121_torch21/webui/models/Stable-diffusion/sd_xl_base_1.0.safetensors" --reg_data_dir="D:\Lora work\Metaverso\Elizabeth9\reg" --save_every_n_epochs="1" --save_model_as=safetensors --save_precision="bf16" --save_state --scale_weight_norms="1" --text_encoder_lr=0.0001 --train_batch_size="1" --train_data_dir="D:\Lora work\Metaverso\Elizabeth9\img" --unet_lr=0.0001 --xformers

GPU: RTX 3060 12gb

[bmaltais]

I will let others answer this one. You may also want to ask the question directly on kohya_ss sd-scripts repo: https://github.com/kohya-ss/sd-scripts/tree/main

[ivanced09]

I will let others answer this one. You may also want to ask the question directly on kohya_ss sd-scripts repo: https://github.com/kohya-ss/sd-scripts/tree/main

Thanks for the clarification, my apologies for the confusion.