Sporadic Bad Alloc Failures on CI

[drisspg]

Summary

Examples:

1. https://github.com/pytorch/ao/actions/runs/11632989138/job/32397205129
2. https://github.com/pytorch/ao/actions/runs/11695718655/job/32571629582

cc @jerryzh168

[jerryzh168]

this is fixed now?

[drisspg]

Yeah lets tentatively close but re-open if other workflows show up