fp32 training failures in timm_models after enabling optimizer

[desertfire]

The problem appeared after #90956.

Repro:

python benchmarks/dynamo/timm_models.py --accuracy --device cuda --backend aot_eager --float32  --training --only   volo_d1_224

The failure is consistent for volo_d1_224.

for i in {1..10}; do python benchmarks/dynamo/timm_models.py --accuracy --device cuda --backend aot_eager --float32  --training --only   fbnetv3_b; done

The failure is random for fbnetv3_b.

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh

[mlazos]

I think the work we did removing the non-deterniminism at least fixed fbnetv3_b, looking at volo now

[ezyang]

fixed now

$ python benchmarks/dynamo/timm_models.py --accuracy --device cuda --backend aot_eager --float32  --training --only   volo_d1_224

cuda train volo_d1_224                         PASS