controlnet-xs-inpaint

[xinli2008]

Describe the bug

Hi, thank you for your nice implement in conbiming controlnet-xs with diffusers. I try the code here to try the image outpaint (similiar to inpaint task), but i found a Strange thing, I don't know why, can you give me some advice?
When I train with 10,000 images, when I train 10W steps, the result is approximately:

When I train to 15W steps, the generated image looks like this:
The resulting picture in step 15W is all surrounded by some colorful dots as follow, which is strange.

Reproduction

I use the code here : https://github.com/UmerHA/diffusers/tree/cnxs-training/examples/controlnet_xs

Logs

No response

System Info

Who can help?

@sayakpaul @yiyixuxu @DN6 @patrickvonplaten

[xinli2008]

I don't know why when I continue to train, the resulting result is surrounded by a strange circle, and it doesn't seem to be an overfitting issue

[xinli2008]

My base model is RealVisXL v3.0

[xinli2008]

If necessary, i can share the wandb logs to see the all validation logs and images

[patrickvonplaten]

Hey @xinli2008,

Could you try to use the forum for such questions: https://github.com/huggingface/diffusers/discussions ?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.