Failed to finetune the pre-trained model of 'stable-diffusion-v1-4' on image inpainting task

[micklexqg]

I finetuned the pre-trained model of 'stable-diffusion-inpainting' on image inpainting task, and all work well as the model is trained on image inpainting. But when I finetuned with the pre-trained model of 'stable-diffusion-v1-4' which is trained on text-to-image, the loss is NaN and the result is pure black.

As the two models have different input channels for unet, I have changed the unet input channels of 'stable-diffusion-v1-4' to be fit for image inpainting task. So far, the code can run but the loss is NaN. I do not know where is the problem, how to finetune the pre-trained model of 'stable-diffusion-inpainting' on image inpainting task ? should I change some hyparameters? Any help will be appreciated, thanks!

[asomoza]

AFAIK inpainting models are always finetuned from the base model. What size is your dataset? You should keep in mind that the base models are trained in several millions or billions of images, is not something you can switch from a finetune to train a model from scratch, the time, costs and infrastructure for them are are completely different, that's why only there's just a bunch of base models and a lot of finetunes.

If you can find an inpainting model that was trained from scratch we can look into it, but also inpainting models are something of the past which is not really required anymore to do inpainting since SDXL and newer models.

[micklexqg]

AFAIK inpainting models are always finetuned from the base model. What size is your dataset? You should keep in mind that the base models are trained in several millions or billions of images, is not something you can switch from a finetune to train a model from scratch, the time, costs and infrastructure for them are are completely different, that's why only there's just a bunch of base models and a lot of finetunes.

If you can find an inpainting model that was trained from scratch we can look into it, but also inpainting models are something of the past which is not really required anymore to do inpainting since SDXL and newer models.

@asomoza Thanks for your message, I want to finetune an model which maybe 'stable-diffusion-v1-4' for inpainting task and the dataset is ImageNet. I run the 'train_dreambooth_inpaint.py' demo using an inpainting model which is 'stable-diffusion-inpainting' from huggingface and all work well. However, when I just change the from_pretrained model from 'stable-diffusion-inpainting' to 'stable-diffusion-v1-4' , all crashed, the loss is NaN and the validation result is pure black. I run thousands of iterations and the loss is alwayls NaN. 'stable-diffusion-inpainting' is for inpainting and 'stable-diffusion-v1-4' is for text-to-image. As I know, most inpainting models are finetuned from text-to-image models such as stable diffusion or SD3. Why here failed, I wonder how to finetune a model for other tasks such as inpainting using a text-to-image model but not using one pretrained inpainting model? Are the two hyparameters heavily different?

[asomoza]

Oh I see, sorry, I don't know why I interpreted what you were doing as to train a model from scratch.

So to be clear, you have success in finetuning an inpainting model but you fail finetuning a base model with an inpainting dataset right?

Sadly as far as I know, there isn't a public script of this, all the finetuned inpainting models are open weights but not open source. You mentioned SD3 but there isn't an inpainting SD3 model, that's why I said that SDXL was the last one and was superseded by the controlnet ones.

I run the 'train_dreambooth_inpaint.py' demo using an inpainting model...

Dreambooth is a very fast method of training a model, it won't work for a different task like you're using it for.

Why here failed, I wonder how to finetune a model for other tasks such as inpainting using a text-to-image model but not using one pretrained inpainting model? Are the two hyparameters heavily different?

I haven't investigated or try it myself, but my guess is yes, or at least, you should not use dreambooth and just a normal finetuning with a low LR to test it.

Maybe you can try training a controlnet instead?

[micklexqg]

Oh I see, sorry, I don't know why I interpreted what you were doing as to train a model from scratch.

So to be clear, you have success in finetuning an inpainting model but you fail finetuning a base model with an inpainting dataset right?

Sadly as far as I know, there isn't a public script of this, all the finetuned inpainting models are open weights but not open source. You mentioned SD3 but there isn't an inpainting SD3 model, that's why I said that SDXL was the last one and was superseded by the controlnet ones.

I run the 'train_dreambooth_inpaint.py' demo using an inpainting model...

Dreambooth is a very fast method of training a model, it won't work for a different task like you're using it for.

Why here failed, I wonder how to finetune a model for other tasks such as inpainting using a text-to-image model but not using one pretrained inpainting model? Are the two hyparameters heavily different?

I haven't investigated or try it myself, but my guess is yes, or at least, you should not use dreambooth and just a normal finetuning with a low LR to test it.

Maybe you can try training a controlnet instead?

Hi @asomoza , OK, Thanks a lot, how wish there could be one open source but not open weights for finetuning inpainting models using text-to-image models or training from scratch, Haha...