New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Stable Diffusion Inpainting] Allow standard text-to-img checkpoints to be useable for SD inpainting #3533
Conversation
The documentation is not available anymore as the PR was closed or merged. |
nice, was looking for this functionality yesterday. |
@@ -137,6 +136,13 @@ def __init__( | |||
): | |||
super().__init__() | |||
|
|||
deprecation_message = ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's fully deprecate this class. Now "normal" inpaint has all features that are needed.
Will update README of: https://huggingface.co/lllyasviel/control_v11p_sd15_inpaint once this PR is merged |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm!
…e_sigma when pure noise updated this commit w.r.t the latest merge here: huggingface#3533
* Throw error if strength adjusted num_inference_steps < 1 * Added new fast test to check ValueError raised when num_inference_steps < 1 when strength adjusts the num_inference_steps then the inpainting pipeline should fail * fix #3487 initial latents are now only scaled by init_noise_sigma when pure noise updated this commit w.r.t the latest merge here: #3533 * fix --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Looking at the example, the control image is just the masked original. How can I use e.g., normalbae with inpainting? |
I tried using the pipeline with |
…to be useable for SD inpainting (huggingface#3533) * Add default to inpaint * Make sure controlnet also works with normal sd for inpaint * Add tests * improve * Correct encode images function * Correct inpaint controlnet * Improve text2img inpanit * make style * up * up * up * up * fix more
…gface#3532) * Throw error if strength adjusted num_inference_steps < 1 * Added new fast test to check ValueError raised when num_inference_steps < 1 when strength adjusts the num_inference_steps then the inpainting pipeline should fail * fix huggingface#3487 initial latents are now only scaled by init_noise_sigma when pure noise updated this commit w.r.t the latest merge here: huggingface#3533 * fix --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
…to be useable for SD inpainting (huggingface#3533) * Add default to inpaint * Make sure controlnet also works with normal sd for inpaint * Add tests * improve * Correct encode images function * Correct inpaint controlnet * Improve text2img inpanit * make style * up * up * up * up * fix more
…gface#3532) * Throw error if strength adjusted num_inference_steps < 1 * Added new fast test to check ValueError raised when num_inference_steps < 1 when strength adjusts the num_inference_steps then the inpainting pipeline should fail * fix huggingface#3487 initial latents are now only scaled by init_noise_sigma when pure noise updated this commit w.r.t the latest merge here: huggingface#3533 * fix --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
This PR allows to use
StableDiffusionControlNetInpaintPipeline
andStableDiffusionInpaintPipeline
for both inpainting models and normal text-to-image models. This has two purposes:1.)
We can completely remove
**StableDiffusionInpaintLegacyPipeline**
. People will still want to use one and the same model weights for text2img, img2img, and inpainting so we need to allow the following scenario:It's also stated in our official docs, here:
https://huggingface.co/docs/diffusers/main/en/api/diffusion_pipeline#diffusers.DiffusionPipeline.components.example
With this PR the standard inpainting pipeline is augmented to allow for inpainting with sd-v1-5 (the text2image checkpoint, not the inpainting) which I think removes some barries. It was also very much a requested feature:
2.)
We need this PR to fully support the ControlNet inpainting model: https://huggingface.co/lllyasviel/control_v11p_sd15_inpaint
This checkpoint works very well with the new inpaint pipeline:
becomes:
Also, I also played around with other SAM to generate masks and then use our SAM ControlNet model: https://colab.research.google.com/drive/1D6mBtne_m-3E9R-cl_ZCRY87Fq1OjkyI?usp=sharing
cc @sayakpaul this could be useful for a
diffusers
tool, but had very limited success here tbh.=> very limited success, the model doesn't seem to work super well. It seems like we either need a ControlNet checkpoint that is purely made for inpainting or use the 9-channel inpainting models.