MPS: models require an initial pass for reproducibility

[pcuenca]

I have no idea what's causing this, I suspect something related to RNGs, as we know they don't work properly in mps yet.

This is a pending task from our initial mps support (#355) that we need to investigate. For now, we have adopted two workarounds:

• Perform an extra pass as appropriate so tests pass.
• Recommend users that care about reproducibility / accuracy to do the same.

[pcuenca]

In the second part of this comment I documented a similar behaviour using einsum vs matmul. I used @patil-suraj's branch to run the tests without the warmup pass, but they still failed. So there must be other ops that have a similar behaviour.

[Birch-san]

I found that this mitigation works, but I assume it only works if you're consistent in the size of x and context that you submit to CrossAttention (e.g. same image dimensions, number of samples, number of conditions):
Birch-san/stable-diffusion@9b1be38

it's not a cheap mitigation. unless we get lucky and there's a fast-path for zeroes or something.

[patrickvonplaten]

cc @pcuenca :-)

[patrickvonplaten]

Is this still relevant (gently pinging @pcuenca )

[pcuenca]

Is this still relevant

Yes, unfortunately, still an issue with the RC of PyTorch 1.13 from the test wheel.

@Birch-san In my experience this is only required the first time you use the model. Is this negatively impacting your use case in a non-transient mode?

[Birch-san]

personally I run pytorch stable 1.12.1 (because it's faster than 1.13RC or the nightlies pytorch/pytorch#85297, pytorch/pytorch#87010), so I don't encounter the einsum reproducibility problem.

my use-case is almost always "launch txt2img just to generate 1 image" (i.e. transient), so if I were on an exposed version of pytorch: this issue would hit me every time.

it's more than just non-determinism; the first einsum result is nonsense. so quality will be strictly lower (I expect it means the highest sigma gets denoised to some mess, but subsequent denoising brings the image back on track, albeit undersampled and with the composition a bit randomized).

I guess the real questions are:

• which versions of pytorch does HF support?
• will this bug be promoted to pytorch 1.13 stable?
  • from @pcuenca's testing: sounds like it is

[pcuenca]

Thanks @Birch-san! Great work posting all those details in PyTorch's github, it looks like they are really helping!

I'll do some more testing about the non-determinism to try to isolate what ops are still affected.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.