Should I add the class images for "style/Object" training?

[terrificdm]

If I click "Training Wizard (Object/Style)" of DreamBooth plugin, the number of "Class Images per Instance Image" is automaticlly set to "0", so does that mean there is no need to add class images for "style/object" training? Actually I tried both(adding/removing class images) before, sometimes adding class images got better results against removing class images, but sometimes not... Can someone elaborate it a little bit further?

Another question is if I set 0 for "Class Images per Instance Image", do I still need to fill out "Class Prompt"? In my option there is no need because of no class images during training, am I right?

Thanks.

[78Alpha]

Your class images are what it compares your instance images to. So, if you have a face and want it to not look like someone else, you can have class images of other faces. This also keeps the old faces intact, or is supposed to help ensure you don't wipe out what it already knew.

You can do it with styles but that's a little harder. If you have a particular style that looks similar to another style, you can use that other style to get it to where you want without drifting too far away. It's trial and error just like usual.

[terrificdm]

Thanks

[Arilziem]

Class images make sense when you have a small dataset and you want to prevent overfitting. Overfitting usually means that unrelated things will start looking like your dataset, e.g. if you have 5 pictures of a face and train it on "a person", all prompts containing "a person" (and related concepts) will start to look like those 5 pictures, including being zoomed in to the face.

Similarly, if your 5 pictures are very grainy and badly lit, your whole model will start to look grainy and badly lit after a while. There are some counter-measures like adding "grainy" / "blurry" / "bad quality" etc. to your prompt / filewords, but your model will likely still take on that look after enough training epochs.

Class images will generate images from the model as it is and add them to your dataset. This will somewhat preserve the status quo and can help guide the training process as well.

For example, if you want cats with hats, you might train with a picture of a cat with a hat. This will however lead to all cats wearing hats after a while. Now if you e.g. use the instance prompt "a cat with a hat" and the class prompt "a cat", a number of "normal" cat images will be generated and added. This will make it easier for the optimizer to figure out that it is the hat that is different from other cat pictures, and preserve the model's capability to generate cats without hats.

This is basically the same for people and objects. I recon the main difference is that the base model will usually have seen way more images of people which are all very closely conceptually related, while objects are more varied. So strategies are a bit different, but the basic principle is the same. Some objects may take little training to be integrated and therefore lead to little overfitting, others may take longer and need class images (or manually selected preservation images). If you are training a variation of an existing object, class images can make it easier for the optimizer to figure out what's so special about the instance.

As a rule of thumb, if you have a couple of hundred pretty varied images in your dataset, you will likely not need to generate class images (although they might still help).

For style, the same principle applies. The model may still overfit on the content you use for training, so as @78Alpha suggests, it may help to use a similar style as the class to preserve variation while still pushing things into the direction of the intended look.

About needing a class prompt when not generating class images: no, you don't need it then.
I wrote something about how the instance and class tokens work here. While instance and class token will do something even if you don't generate class images, the class prompt will do nothing in that case.

[terrificdm]

Thanks, will look into it.