AMD ROCm: torch.backends.cudnn.benchmark should be set to False by default on ROCm

[asumagic]

Describe the bug

With AMD HIP, torch.backends.cudnn.benchmark defaults to True. On CUDA, it defaults to False.

While this is an upstream PyTorch decision (or bug), having benchmark default to True by default is probably a bad idea in our case, because we have wildly different input shapes. Since CNN kernel benchmarking is performed on a per-shape basis, it performs those slow benchmarks very often, so training times are worsened rather than improved.

Probably should add some common per-backend quirks file for this kind of change in SB?

Expected behaviour

On AMD ROCm, it should be set to False, probably with a warning since we break the default.

To Reproduce

No response

Environment Details

No response

Relevant Log Output

No response

Additional Context

No response