Create train_dreambooth_inpaint.py #1091

thedarkzeno · 2022-11-01T00:01:15Z

train_dreambooth.py adapted to work with the inpaint model, generating random masks during the training

HuggingFaceDocBuilderDev · 2022-11-01T00:06:12Z

The documentation is not available anymore as the PR was closed or merged.

refactored train_dreambooth_inpaint with black

patrickvonplaten · 2022-11-02T17:26:04Z

Interesting! This would be a cool addition if it works well :-)

patrickvonplaten · 2022-11-16T16:33:58Z

Gentle ping here @patil-suraj

Fix prior preservation

thedarkzeno · 2022-11-18T19:26:27Z

Hey guys, I'm thinking of adding the option to create the mask with clipseg instead of just using random masks, what do you think?
I believe it could improve training by masking the area of interest.
@patil-suraj
@patrickvonplaten

patrickvonplaten · 2022-11-20T19:49:49Z

Reping @patil-suraj

loboere · 2022-11-21T20:53:47Z

can you please adapt this to this colab https://github.com/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb ,this colab trains based only on the name of the images, without class images and that complicated stuff

patrickvonplaten · 2022-11-28T12:25:28Z

Hey @loboere,

Note that this colab is not part of the diffusers repo - could you please leave an issue on https://github.com/TheLastBen/fast-stable-diffusion/ ?

0xdevalias · 2022-11-28T20:37:28Z

Note that this colab is not part of the diffusers repo - could you please leave an issue on https://github.com/TheLastBen/fast-stable-diffusion/ ?

@TheLastBen FYI

patrickvonplaten · 2022-12-01T16:03:37Z

@patil-suraj ping again

patrickvonplaten · 2022-12-01T16:08:25Z

@thedarkzeno @patil-suraj seems to busy to review the PR at the moment.

Let's just go for it :-) Could you however please add a section to https://github.com/huggingface/diffusers/tree/main/examples/dreambooth explaining how to use your script?

thedarkzeno · 2022-12-02T01:39:55Z

Hey @patrickvonplaten, sure. I think I'll just have to make a few adjustments to support stable diffusion v2.

patrickvonplaten · 2022-12-02T17:06:50Z

Awesome let's merge it :-)

@williamberman @patil-suraj it would be great if you could give it a spin :-)

0xdevalias · 2022-12-08T23:55:07Z

Was just looking, and this doesn't seem to be available at the following:

https://github.com/huggingface/diffusers/tree/main/examples/dreambooth

Why not/where did it go?

Edit: Digging into the commit history, I see the following that seem to have touched it:

https://github.com/huggingface/diffusers/commits/main/examples/dreambooth/train_dreambooth_inpaint.py

Specifically, it seems that #1553 was the one that moved it, and it now lives at:

https://github.com/huggingface/diffusers/tree/main/examples/research_projects/dreambooth_inpaint

loboere · 2022-12-09T00:09:57Z

can you please create a colab to test this and have it work on a t4 gpu

* Create train_dreambooth_inpaint.py train_dreambooth.py adapted to work with the inpaint model, generating random masks during the training * Update train_dreambooth_inpaint.py refactored train_dreambooth_inpaint with black * Update train_dreambooth_inpaint.py * Update train_dreambooth_inpaint.py * Update train_dreambooth_inpaint.py Fix prior preservation * add instructions to readme, fix SD2 compatibility

loboere · 2023-01-06T19:19:26Z

I tried to train in collaboration with the same photos of cat toy but the results are a disaster, it seems to corrupt the original inpainting model, I don't know what's wrong.

!accelerate launch train_dreambooth_inpaint.py
--pretrained_model_name_or_path="runwayml/stable-diffusion-inpainting"
--instance_data_dir="./my_concept"
--output_dir="dreambooth-concept"
--instance_prompt="skdfklsdlfksd"
--resolution=512
--train_batch_size=1
--gradient_accumulation_steps=1
--learning_rate=5e-6
--lr_scheduler="constant"
--lr_warmup_steps=100
--gradient_checkpointing
--use_8bit_adam
--mixed_precision="fp16"
--max_train_steps=600

after training load the model

#load pipeline inpainting

from diffusers import StableDiffusionInpaintPipeline
import torch

pipe = StableDiffusionInpaintPipeline.from_pretrained(
     "/content/dreambooth-concept",
     torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")

a skdfklsdlfksd photo

also with normal objects

dog photo

thedarkzeno · 2023-01-07T17:43:18Z

Hello @loboere I tryed with --use_8bit_adam and got bad results as well, but with different params my results were better.

accelerate launch dreambooth_inpaint.py ^
--pretrained_model_name_or_path="runwayml/stable-diffusion-inpainting" ^
--instance_data_dir="./toy_cat" ^
--output_dir="./dreambooth_ad_inpaint_toy_cat" ^
--instance_prompt="toy cat" ^
--resolution=512 ^
--train_batch_size=1 ^
--learning_rate=5e-6 ^
--lr_scheduler="constant" ^
--lr_warmup_steps=0 ^
--max_train_steps=1000 ^
--gradient_accumulation_steps=2 ^
--gradient_checkpointing ^
--train_text_encoder

Training with those params this was my result.

maybe something with the 8bit_adam is not working as intended.

Aldo-Aditiya · 2023-01-10T08:33:54Z

Hey @thedarkzeno I tried using your script (and using the related requirements), but ran into this error related to CrossAttnDownBlock2D

Traceback (most recent call last):
  File "train_dreambooth_inpaint.py", line 799, in <module>
    main()
  File "train_dreambooth_inpaint.py", line 493, in main
    vae = AutoencoderKL.from_pretrained(args.pretrained_model_name_or_path, subfolder="vae")
  File "/usr/local/lib/python3.8/dist-packages/diffusers/modeling_utils.py", line 483, in from_pretrained
    model = cls.from_config(config, **unused_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/configuration_utils.py", line 210, in from_config
    model = cls(**init_dict)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/configuration_utils.py", line 567, in inner_init
    init(self, *args, **init_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/models/vae.py", line 539, in __init__
    self.encoder = Encoder(
  File "/usr/local/lib/python3.8/dist-packages/diffusers/models/vae.py", line 94, in __init__
    down_block = get_down_block(
  File "/usr/local/lib/python3.8/dist-packages/diffusers/models/unet_2d_blocks.py", line 83, in get_down_block
    raise ValueError("cross_attention_dim must be specified for CrossAttnDownBlock2D")
ValueError: cross_attention_dim must be specified for CrossAttnDownBlock2D

Did you use a specific release of runwayml/stable-diffusion-inpainting by any chance?

kunalgoyal9 · 2023-02-05T19:11:18Z

Hello @loboere I tryed with --use_8bit_adam and got bad results as well, but with different params my results were better.

accelerate launch dreambooth_inpaint.py ^
--pretrained_model_name_or_path="runwayml/stable-diffusion-inpainting" ^
--instance_data_dir="./toy_cat" ^
--output_dir="./dreambooth_ad_inpaint_toy_cat" ^
--instance_prompt="toy cat" ^
--resolution=512 ^
--train_batch_size=1 ^
--learning_rate=5e-6 ^
--lr_scheduler="constant" ^
--lr_warmup_steps=0 ^
--max_train_steps=1000 ^
--gradient_accumulation_steps=2 ^
--gradient_checkpointing ^
--train_text_encoder

Training with those params this was my result.

maybe something with the 8bit_adam is not working as intended.

@thedarkzeno I was following the same settings you have mentioned but my loss is not decreasing. Any help would be appriciated.

thedarkzeno · 2023-02-05T21:04:47Z

Hello @kunalgoyal9, sometimes the loss doesn't decrease, but you can still get good results, did you check the outputs from your model?

kunalgoyal9 · 2023-02-06T10:41:22Z

@thedarkzeno Thanks for your reply... output is also not good.. I used four toy_cat images and testing using prompt "a toy cat sitting on a bench"

thedarkzeno · 2023-02-06T21:34:34Z

can you try using just "toy cat" as prompt?

dai-ichiro · 2023-02-07T03:39:20Z

Hello @Aldo-Aditiya

If there is a file named "config.json" under the cloned "stable-diffusion-inpainting" repository, the following error will occur.

ValueError: cross_attention_dim must be specified for CrossAttnDownBlock2D

I removed "config.json", and the error disappeared.

Hope this helps.

williamberman · 2023-02-13T01:52:30Z

Hey folks! We're trying to encourage the forum for open ended discussion :) Might be good to make a thread there for future dreambooth inpainting discussion https://discuss.huggingface.co/c/discussion-related-to-httpsgithubcomhuggingfacediffusers/63

JANGSOONMYUN · 2023-12-04T09:32:25Z

Hello @loboere I tryed with --use_8bit_adam and got bad results as well, but with different params my results were better.

accelerate launch dreambooth_inpaint.py ^
--pretrained_model_name_or_path="runwayml/stable-diffusion-inpainting" ^
--instance_data_dir="./toy_cat" ^
--output_dir="./dreambooth_ad_inpaint_toy_cat" ^
--instance_prompt="toy cat" ^
--resolution=512 ^
--train_batch_size=1 ^
--learning_rate=5e-6 ^
--lr_scheduler="constant" ^
--lr_warmup_steps=0 ^
--max_train_steps=1000 ^
--gradient_accumulation_steps=2 ^
--gradient_checkpointing ^
--train_text_encoder

Training with those params this was my result.

maybe something with the 8bit_adam is not working as intended.

Hi, I have tried as following your script and it works. However I would like to know if I have many pairs of images and their captions, how I can train the dataset correctly? You set the 'instance_prompt' just one string value "toy_cat", but I want to train many prompts. For example, "bed", "chair", "sofa", "wardrobe", ... for each image.
Do you have any idea for it?

thedarkzeno · 2023-12-05T14:10:41Z

@JANGSOONMYUN you have to modify the code to support your data, I suggest you take a look at the text-to-image script
here

JANGSOONMYUN · 2023-12-06T02:06:49Z

@JANGSOONMYUN you have to modify the code to support your data, I suggest you take a look at the text-to-image script here

Ok thank you!

Create train_dreambooth_inpaint.py

d394eaa

train_dreambooth.py adapted to work with the inpaint model, generating random masks during the training

Update train_dreambooth_inpaint.py

3d7729e

refactored train_dreambooth_inpaint with black

thedarkzeno mentioned this pull request Nov 1, 2022

Separating legacy inpaint pipeline #920

Closed

thedarkzeno added 2 commits October 31, 2022 23:15

Update train_dreambooth_inpaint.py

1a6f6ff

Update train_dreambooth_inpaint.py

a58d6aa

patil-suraj self-assigned this Nov 2, 2022

Update train_dreambooth_inpaint.py

6dd054b

Fix prior preservation

thedarkzeno mentioned this pull request Nov 20, 2022

train dreambooth with runway inpainting model #1345

Closed

thedarkzeno added 2 commits December 1, 2022 22:45

Merge branch 'huggingface:main' into patch-1

b8964d6

add instructions to readme, fix SD2 compatibility

107f440

patrickvonplaten merged commit 2b30b10 into huggingface:main Dec 2, 2022

loboere mentioned this pull request Dec 28, 2022

train dreambooth on model inpainting runwayml TheLastBen/fast-stable-diffusion#1186

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create train_dreambooth_inpaint.py #1091

Create train_dreambooth_inpaint.py #1091

thedarkzeno commented Nov 1, 2022

HuggingFaceDocBuilderDev commented Nov 1, 2022 •

edited

patrickvonplaten commented Nov 2, 2022

patrickvonplaten commented Nov 16, 2022 •

edited

thedarkzeno commented Nov 18, 2022

patrickvonplaten commented Nov 20, 2022

loboere commented Nov 21, 2022

patrickvonplaten commented Nov 28, 2022

0xdevalias commented Nov 28, 2022

patrickvonplaten commented Dec 1, 2022

patrickvonplaten commented Dec 1, 2022

thedarkzeno commented Dec 2, 2022

patrickvonplaten commented Dec 2, 2022

0xdevalias commented Dec 8, 2022 •

edited

loboere commented Dec 9, 2022

loboere commented Jan 6, 2023

thedarkzeno commented Jan 7, 2023

Aldo-Aditiya commented Jan 10, 2023 •

edited

kunalgoyal9 commented Feb 5, 2023

thedarkzeno commented Feb 5, 2023

kunalgoyal9 commented Feb 6, 2023

thedarkzeno commented Feb 6, 2023

dai-ichiro commented Feb 7, 2023

williamberman commented Feb 13, 2023

JANGSOONMYUN commented Dec 4, 2023

thedarkzeno commented Dec 5, 2023

JANGSOONMYUN commented Dec 6, 2023

Create train_dreambooth_inpaint.py #1091

Create train_dreambooth_inpaint.py #1091

Conversation

thedarkzeno commented Nov 1, 2022

HuggingFaceDocBuilderDev commented Nov 1, 2022 • edited

patrickvonplaten commented Nov 2, 2022

patrickvonplaten commented Nov 16, 2022 • edited

thedarkzeno commented Nov 18, 2022

patrickvonplaten commented Nov 20, 2022

loboere commented Nov 21, 2022

patrickvonplaten commented Nov 28, 2022

0xdevalias commented Nov 28, 2022

patrickvonplaten commented Dec 1, 2022

patrickvonplaten commented Dec 1, 2022

thedarkzeno commented Dec 2, 2022

patrickvonplaten commented Dec 2, 2022

0xdevalias commented Dec 8, 2022 • edited

loboere commented Dec 9, 2022

loboere commented Jan 6, 2023

thedarkzeno commented Jan 7, 2023

Aldo-Aditiya commented Jan 10, 2023 • edited

kunalgoyal9 commented Feb 5, 2023

thedarkzeno commented Feb 5, 2023

kunalgoyal9 commented Feb 6, 2023

thedarkzeno commented Feb 6, 2023

dai-ichiro commented Feb 7, 2023

williamberman commented Feb 13, 2023

JANGSOONMYUN commented Dec 4, 2023

thedarkzeno commented Dec 5, 2023

JANGSOONMYUN commented Dec 6, 2023

HuggingFaceDocBuilderDev commented Nov 1, 2022 •

edited

patrickvonplaten commented Nov 16, 2022 •

edited

0xdevalias commented Dec 8, 2022 •

edited

Aldo-Aditiya commented Jan 10, 2023 •

edited