AttributeError: 'str' object has no attribute 'convert' when training controlnet

[XiangjiBU]

Describe the bug

when I training controlnet (by example/controlnet ), AttributeError: 'str' object has no attribute 'convert' appears

Reproduction

I use the same bash with example/controlnet/README.md

accelerate launch train_controlnet.py \
 --pretrained_model_name_or_path=$MODEL_DIR \
 --output_dir=$OUTPUT_DIR \
 --dataset_name=fusing/fill50k \
 --resolution=512 \
 --learning_rate=1e-5 \
 --validation_image "./conditioning_image_1.png" "./conditioning_image_2.png" \
 --validation_prompt "red circle with blue background" "cyan circle with brown floral background" \
 --train_batch_size=1 \
 --gradient_accumulation_steps=4 

I download fill50K dataset and put it in example/controlnet. The folder like this:

example/controlnet:
  -- train_controlnet.py
  -- conditioning_image_1.png
  -- conditioning_image_2.png
  -- fusing/fill50k
        -- train.jsonl
        -- conditioning_images
                -- 0.png
                -- 1.png
                    ......
          -- images
                -- 0.png
                -- 1.png
                    ......

Logs

Downloading data files: 100%|████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 11066.77it/s]
Extracting data files: 100%|██████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1844.46it/s]
Generating train split: 50000 examples [00:00, 1874549.27 examples/s]
02/05/2024 20:05:51 - INFO - __main__ - ***** Running training *****
02/05/2024 20:05:51 - INFO - __main__ -   Num examples = 50000
02/05/2024 20:05:51 - INFO - __main__ -   Num batches each epoch = 50000
02/05/2024 20:05:51 - INFO - __main__ -   Num Epochs = 1
02/05/2024 20:05:51 - INFO - __main__ -   Instantaneous batch size per device = 1
02/05/2024 20:05:51 - INFO - __main__ -   Total train batch size (w. parallel, distributed & accumulation) = 4
02/05/2024 20:05:51 - INFO - __main__ -   Gradient Accumulation steps = 4
02/05/2024 20:05:51 - INFO - __main__ -   Total optimization steps = 12500
Steps:   0%|                                                                                    | 0/12500 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/mnt/sdb/数据2/diffusers/examples/controlnet/train_controlnet.py", line 1142, in <module>
    main(args)
  File "/mnt/sdb/数据2/diffusers/examples/controlnet/train_controlnet.py", line 1007, in main
    for step, batch in enumerate(train_dataloader):
  File "/home/ps/anaconda3/envs/diffusers/lib/python3.10/site-packages/accelerate/data_loader.py", line 384, in __iter__
    current_batch = next(dataloader_iter)
  File "/home/ps/anaconda3/envs/diffusers/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "/home/ps/anaconda3/envs/diffusers/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 677, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/ps/anaconda3/envs/diffusers/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = self.dataset.__getitems__(possibly_batched_index)
  File "/home/ps/anaconda3/envs/diffusers/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2799, in __getitems__
    batch = self.__getitem__(keys)
  File "/home/ps/anaconda3/envs/diffusers/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2795, in __getitem__
    return self._getitem(key)
  File "/home/ps/anaconda3/envs/diffusers/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2780, in _getitem
    formatted_output = format_table(
  File "/home/ps/anaconda3/envs/diffusers/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 629, in format_table
    return formatter(pa_table, query_type=query_type)
  File "/home/ps/anaconda3/envs/diffusers/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 400, in __call__
    return self.format_batch(pa_table)
  File "/home/ps/anaconda3/envs/diffusers/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 515, in format_batch
    return self.transform(batch)
  File "/mnt/sdb/diffusers/examples/controlnet/train_controlnet.py", line 681, in preprocess_train
    images = [image.convert("RGB") for image in examples[image_column]]
  File "/mnt/sdb/diffusers/examples/controlnet/train_controlnet.py", line 681, in <listcomp>
    images = [image.convert("RGB") for image in examples[image_column]]
AttributeError: 'str' object has no attribute 'convert'
Steps:   0%|                                                                                    | 0/12500 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/ps/anaconda3/envs/diffusers/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/ps/anaconda3/envs/diffusers/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/home/ps/anaconda3/envs/diffusers/lib/python3.10/site-packages/accelerate/commands/launch.py", line 986, in launch_command
    simple_launcher(args)
  File "/home/ps/anaconda3/envs/diffusers/lib/python3.10/site-packages/accelerate/commands/launch.py", line 628, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/ps/anaconda3/envs/diffusers/bin/python3.10', 'train_controlnet.py', '--pretrained_model_name_or_path=/home/ps/train_sd_code/diffusers/runwayml/stable-diffusion-v1-5', '--output_dir=./output_test1', '--dataset_name=fusing/fill50k', '--resolution=512', '--learning_rate=1e-5', '--validation_image', './conditioning_image_1.png', './conditioning_image_2.png', '--validation_prompt', 'red circle with blue background', 'cyan circle with brown floral background', '--train_batch_size=1', '--gradient_accumulation_steps=4']' returned non-zero exit status 1.

System Info

I tried diffusers-0.25.0.dev0 and diffusers-0.26.0.dev0, but both have the same problem

Who can help?

@sayakpaul @yiyixuxu @DN6 @patrickvonplaten

[sayakpaul]

Did you ensure the validation images were downloaded beforehand as instructed here?

[XiangjiBU]

Did you ensure the validation images were downloaded beforehand as instructed here?

validation images already in the folder，please my folder structure

[XiangjiBU]

I print the image in this line
images = [image.convert("RGB") for image in examples[image_column]]
it seems image is names，not PIL format

[sayakpaul]

I am able to train with the following command seamlessly:

export MODEL_DIR="runwayml/stable-diffusion-v1-5"
export OUTPUT_DIR="controlnet-sd"

accelerate launch train_controlnet.py \
 --pretrained_model_name_or_path=$MODEL_DIR \
 --output_dir=$OUTPUT_DIR \
 --dataset_name=fusing/fill50k \
 --resolution=512 \
 --learning_rate=1e-5 \
 --validation_image "./conditioning_image_1.png" "./conditioning_image_2.png" \
 --validation_prompt "red circle with blue background" "cyan circle with brown floral background" \
 --train_batch_size=1 \
 --gradient_accumulation_steps=4

[XiangjiBU]

I am able to train with the following command seamlessly:

export MODEL_DIR="runwayml/stable-diffusion-v1-5"
export OUTPUT_DIR="controlnet-sd"

accelerate launch train_controlnet.py \
 --pretrained_model_name_or_path=$MODEL_DIR \
 --output_dir=$OUTPUT_DIR \
 --dataset_name=fusing/fill50k \
 --resolution=512 \
 --learning_rate=1e-5 \
 --validation_image "./conditioning_image_1.png" "./conditioning_image_2.png" \
 --validation_prompt "red circle with blue background" "cyan circle with brown floral background" \
 --train_batch_size=1 \
 --gradient_accumulation_steps=4

can you show me the location and folder structure of your fill50k dataset？

[sayakpaul]

Nothing special. Since we're passing a Hub identifier of the dataset, the datasets library will load it.

diffusers/examples/controlnet/train_controlnet.py

Line 595 in 8bf046b

dataset = load_dataset(

[XiangjiBU]

I tried to train with my own data，I made my dataset same with fill50k，but met the same error

[sayakpaul]

We need to have a reproducible snippet here for helping you here.

[XiangjiBU]

Nothing special. Since we're passing a Hub identifier of the dataset, the datasets library will load it.

diffusers/examples/controlnet/train_controlnet.py

Line 595 in 8bf046b

dataset = load_dataset(

I still don't know how to locate dataset.
As my purpose is training my own dataset, not just train with fill50k, can you give a guidance about the dataset formet?

[sayakpaul]

The dataset format is described on the README. If that doesn't work, I welcome you to refer to the script and adjust the dataloader with other torch utilities.

[XiangjiBU]

The dataset format is described on the README. If that doesn't work, I welcome you to refer to the script and adjust the dataloader with other torch utilities.

I am very sure your README.md not work for custom dataset, because I figure it out! It works with Fill50K because there is a fill50k.py. class Fill50k(datasets.GeneratorBasedBuilder) is define in this python file in order to preprocess data. when I want to use my own dataset, I need define a preprocess class and use it in load_dataset()
I strongly recommend you to upgrade your example and README.md, because no one want to use Fill50k train a controlnet

[sayakpaul]

Yeah, feel free to submit a PR.

[lijiang0514]

I tried to train with my own data，I made my dataset same with fill50k，but met the same error

Hello, I have the same problem, may I ask if you have solved it? Can you share the solution? Thank you