Describe the bug
When I train controlnet_xl use fill50k dataset ,
File "/app/SDXL_trainer/diffusers/examples/controlnet/train_controlnet_sdxl.py", line 742, in collate_fn prompt_ids = torch.stack([torch.tensor(example["prompt_embeds"]) for example in examples]) File "/app/SDXL_trainer/diffusers/examples/controlnet/train_controlnet_sdxl.py", line 742, in <listcomp> prompt_ids = torch.stack([torch.tensor(example["prompt_embeds"]) for example in examples]) KeyError: 'prompt_embeds
I don’t found "prompt_embeds" process in train code .
And When I use accelerate train the model ,
TypeError: Provided functionwhich is applied to all elements of table returns adictof types [<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>]. When usingbatched=True, make sure provided functionreturns adictof types like(<class 'list'>, <class 'numpy.ndarray'>).
Reproduction
python train_controlnet_sdxl.py \ --pretrained_model_name_or_path=$MODEL_DIR \ --output_dir=$OUTPUT_DIR \ --dataset_name=fusing/fill50k \ --mixed_precision="fp16" \ --resolution=1024 \ --learning_rate=1e-5 \ --max_train_steps=15000 \ --validation_image "./conditioning_image_2.png" \ --validation_prompt "" \ --validation_steps=100 \ --train_batch_size=1 \ --gradient_accumulation_steps=4 \ --report_to="wandb" \ --seed=42 \ --push_to_hub
Logs
No response
System Info
A100
diffusers 2.0
python 3.10
Who can help?
No response
Describe the bug
When I train controlnet_xl use fill50k dataset ,
File "/app/SDXL_trainer/diffusers/examples/controlnet/train_controlnet_sdxl.py", line 742, in collate_fn prompt_ids = torch.stack([torch.tensor(example["prompt_embeds"]) for example in examples]) File "/app/SDXL_trainer/diffusers/examples/controlnet/train_controlnet_sdxl.py", line 742, in <listcomp> prompt_ids = torch.stack([torch.tensor(example["prompt_embeds"]) for example in examples]) KeyError: 'prompt_embedsI don’t found "prompt_embeds" process in train code .
And When I use accelerate train the model ,
TypeError: Providedfunctionwhich is applied to all elements of table returns adictof types [<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>]. When usingbatched=True, make sure providedfunctionreturns adictof types like(<class 'list'>, <class 'numpy.ndarray'>).Reproduction
python train_controlnet_sdxl.py \ --pretrained_model_name_or_path=$MODEL_DIR \ --output_dir=$OUTPUT_DIR \ --dataset_name=fusing/fill50k \ --mixed_precision="fp16" \ --resolution=1024 \ --learning_rate=1e-5 \ --max_train_steps=15000 \ --validation_image "./conditioning_image_2.png" \ --validation_prompt "" \ --validation_steps=100 \ --train_batch_size=1 \ --gradient_accumulation_steps=4 \ --report_to="wandb" \ --seed=42 \ --push_to_hubLogs
No response
System Info
A100
diffusers 2.0
python 3.10
Who can help?
No response