The train_controlnet.py script prevents valid local datasets from being used #2772
Description
Describe the bug
The train_controlnet.py script has a --train_data_dir flag for training with local datasets. However, using it seems impossible. This is because the code works by assuming the data is in the imagefolder format, which AFAICT tell from all the examples, expects a single image, but the controlnet data requires two images, one for the vanilla SD training and one for the controlnet training.
It is possible that there is a way to format the directories so it can be used with the imagefolder format and two images, but I was unable to figure that out (happy to be wrong here).
To get the code working for local datasets (like the fusing/fill50k example dataset), I modified the code to not assume the data is in the imagefolder format and then it can work with a setup .py file.
Reproduction
call train_controlnet.py with the "--train_data_dir" flag
Logs
No response
System Info
diffusersversion: 0.15.0.dev0- Platform: Linux-5.19.0-35-generic-x86_64-with-glibc2.35
- Python version: 3.9.13
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Huggingface_hub version: 0.13.2
- Transformers version: 4.26.1
- Accelerate version: 0.17.1
- xFormers version: 0.0.16
- Using GPU in script?: NVidia RTX 3090 Ti
- Using distributed or parallel set-up in script?: nope