RePaint pipeline and stable-diffusion-inpaint pipeline legacy off-by-one error #1581
Description
Describe the bug
When I am using diffuser library to merge repaint pipeline logic with the SD-inpaint-pipeline-legacy, I observed an obvious performance drop which is very curious to me. I believe it is caused by two off-by-one errors.
The repaint paper introduced a pipeline to merge the known part of the image and unknown part of the image together. However, I believe the original paper might have some typo or error in its algorithm section. The same error I believe is also carried over to the diffuser implementation:
The known part of x_{t-1} should be sqrt_alpha_bar_{t-1} * x_0 + sqrt_one_minus_alpha_bar_{t-1} * epsilon.
There are two errors in original paper in line 5 of Algorithm 1. First the coefficient of epsilon is not sqrt-ed, second the timestep is off by one.
Though the implementation of diffusers lib in https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_repaint.py#L290 fixed the first error, I believe the second error is not fixed.
Another related and similar erorr is in SD-pipleine-inpaint-legacy:
https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint_legacy.py#L566
I see no reason to merge a latents from t-1 timestep with latents from timeste t.
I might be wrong that this is not a mismatch and is actually an intended design. Please kindly let me know if I am wrong.
If this two mismatch is actually unwanted, I have provided two naive fix unless more sophiscated test suites is required.
They can be found in #1583, #1582
Thanks : )
Reproduction
N/A
Logs
No response
System Info
N/A