Need some help with DreamBooth training

[FurkanGozukara]

Could you make a raw example of how to train? How much VRAM it takes full training both UNET + Text Encoder

I want to make a tutorial video but simply there aren't enough example

How do we provide training dataset? Is training supporting BF16?

Do you have regularization images support during training? Which is the core of the DreamBooth

I have done 100+ trainings with SDXL

For example for SDXL we use ohwx as rare token and man as class token to train a certain person. We are also giving regularization images as man class. And when we don't provide reg images it becomes full fine tuning.

Actually it would be super amazing if you could make a pull request to Kohya Stable Diffusion trainer : https://github.com/kohya-ss/sd-scripts

I also opened an issue there regarding PixArt : kohya-ss/sd-scripts#979

[lawrence-cj]

For training and dataset, there is nothing else https://github.com/PixArt-alpha/PixArt-alpha/blob/master/asset/docs/pixart-dreambooth.md.
For BF16, we don't have A100 GPUs, so no implemtation here.
We are not expert at DreamBooth, I will take a look at regularization images.
and can you provide more details about:

For example for SDXL we use ohwx as rare token and man as class token to train a certain person. We are also giving regularization images as man class. And when we don't provide reg images it becomes full fine tuning.

Thank you for your advice.

Have to say that, we watched through the video you provided, really promising. Shall we put the video in the readme maybe later?

[lawrence-cj]

Just to be clear in advance. The training of base model is our goal, Dreambooth is not our contribution. We won't spend more time on it since we have other improvements need to study.
Then, answers to some of your questions:

1. We put the prompt here: https://github.com/PixArt-alpha/PixArt-alpha/blob/master/configs/pixart_config/PixArt_xl2_img1024_dreambooth.py#L4C47-L4C47, we just implement a very simple version of Dreambooth where each class only has the same prompt.
2. The official .pth file is here: https://huggingface.co/PixArt-alpha/PixArt-alpha/resolve/main/PixArt-XL-2-1024-MS.pth. We provide a curl link for 1024MS here: https://github.com/PixArt-alpha/PixArt-alpha#-download-models. and we may change the .pth name for clearification.
3. We only train the diffusion model.
4. At the beginning of the base model training, set fp16 will cause unstable. But during fine-tuning, I'm not sure if it's the same.
5. it is not a single .safetensors file like SDXL. Not sure about your meaning. We have official diffusers pipeline and ckpts in our huggingface models: https://huggingface.co/PixArt-alpha. I'm not sure what kind of safetensors is needed.
6. window_block_index, window_size, use_rel_pos are related to window attention, we used them b4, though there is not useful anymore. We put them there in case someone wants an example for where to add window attention related function. multi_scale is for training with multi-resolution images during training, which is the same as SDXL.
7. lewei_scale as the content written in the config file, which's a trick for position embedding adapting when changing the image resolution from e.g. 512 to 1024.

we need to train text encoder and we need to be able to train with certain captions we want. For example training with ohwx man myself so I can consistently get myself after training

Good applications and model enhancements are emerging every single day, and we can't apply them all. We call for users to implement the function themselves, we have a lot of other applications that we haven't released, e.g. ControlNet. We still need to work on them. We hope you understand.

At last, the video is promising, however, there is some content someone needs to pay for. I understand the effort and time needed for this video and the payment for learning and guidance is reasonable. I may not put the video in our readme file, I would twitter it instead. Thank you for your video and effort again.

[FurkanGozukara]

@lawrence-cj you are really not following Stable Diffusion community and that is rather surprising to me

For example SDXL repo : https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/tree/main

You see it has sd_xl_base_1.0.safetensors file there. This file contains VAE, Text Encoder and U-NET. All in 1 package and since it is safetensors compared to ckpt it can't contain malicious code.

Currently community is looking for this packaged files. It is also so easy to move around share etc.

Here my questions if you could answer

1 : Do you have such .safetensors file for PixArt?

• I understood your DreamBooth logic. So training only diffusion model for a single class

2 : Is your training script (not DreamBooth) training Text Encoder (T5) or still only diffusion model file trained?

Believe me community will be really interested in once it is easy to train. This model is certainly undertrained atm and has huge potential

I already contacted Kohya and OneTrainer to implement PixArt training into their pipeline

But .safetensors packaged model file is really mandatory

And your discord invite is broken :)

[lawrence-cj]

Oh, now I follow up with your point. I lack experience in community building, that's why we need you guys for good suggestions.

1. I will certainly make a .safetensors soon.
2. As I said, we only train the diffusion model. T5 is frozen everywhere.
3. Updated the Discord link.

We would really appreciate it if we could build the project better with your advice.

[YotamNitzan]

The discord link seems to be broken again @lawrence-cj

[lawrence-cj]

Seems one link is forgot to change. Updated already.

[lawrence-cj]

First of all, thank you for your dedication to this very promising diffusion model. Maybe that you're looking for visibility, but just don't understand why you encouraging FurkanGozukara.

Have you really seen his video ? He's basically asking people to go to his patreon to download a batch file he made (with probably some few pip install and few python call inside) that he calls "installer". Obviously during his video, he never explain how to install PIXART-alpha and how to launch the application, without his holly "installer" files and launcher (that you have to be patreon member to download).

One of his viewers commented "Isn't it possible to follow your tutorial to runpod without becoming your patreon subscriber?" and FurkanGozukara's answer was "Possible. You can look at their GitHub repo and install it yourself. But their Gradio app is a bit weaker than my improved app." He didn't bother to give more details on how to install pixelart-alpha for people who don't have access to his patreon. even putting a link to your repository under his video was too much for him.

I may be wrong, but I think that this kind of behavior don't deserve your attention, and adding his video to your readme is promoting for paid content over your open source research project.

That being said, I won't bother you anymore.

Adding a paid tutorial to readme may be a bit ill-considered. Actually, we have a full tutorial here: https://github.com/PixArt-alpha/PixArt-alpha/tree/master/notebooks. Then, we will just leave the video in the discussion. Thx for your noticing. @NonaBuhtig

[axel578]

Everyone gave up on dreambooth training ?