flux:different results on different machines.

[D222097]

Hi！
During the debugging phase, I've noticed that training the FLUX-Controlnet-Inpainting model yields different results on different machines.

Specifically, on one machine, everything trains fine,
but on another, I'm getting an error that says "Gradient for module.xxx contains NaN!".

I've double-checked and confirmed that my code is exactly the same on both machines, and use the same seed in set_seed().

Has anyone else encountered this same issue? I'm curious to know if there are any potential causes for this discrepancy in training results across different machines. Any insights or suggestions would be greatly appreciated.

[D222097]

machine1：

machine2：

[D222097]

machine1：

machine2：

use the same seed in set_seed()

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.