research: results reproducability and sdpa

[ghost]

finding 1) xformers never producess identical results

even if both torch.backends.cudnn.deterministic and torch.use_deterministic_algorithms are set to true

this makes comparison between 2 loras trained with xformers essentially meaningless.

(unless they are settings that drastically change results, even then their effects can be uncertain)

finding 2) torch.use_deterministic_algorithms=true does not make a difference, as it seems to be the default behavior.

11 and 22 were trained identiclly, 24 was trained with use_deterministic_algorithms=true and 25 was trained with same settings as 11 and 22 but after a computer restart, for sanity check.

finding 3) sdpa results are also not reproducable

even with torch.backends.cudnn.deterministic=true

or both torch.backends.cudnn.deterministic=true and torch.use_deterministic_algorithms=true

i also tried disabling memory efficient attention by setting both "torch.backends.cuda.enable_mem_efficient_sdp" and "enable_mem_efficient_sdp" to false, but still got different results.

tl;dr: it seems impossible to get reproducible results with xformers or sdpa.

24gb vram is barely enough to train sdxl lora without them, training time is also slightly increased (56 mins -> 64 mins on 1200 step DORA)

finding 3) sdpa slightly worsens results

i also trained a version of 22 with sdpa, comparing to no sdpa, here are some grids

unsurprisingly, results without sdpa are better.

i plan to train and test xformers later aswell.

tl;dr: it seems impossible to reproduce models with xformers/sdpa, at least i couldn't. and generally you should avoid them if you have enough vram.