[rocm7.0_internal_testing] Prevent static initialization of at::cuda::warp_size()

[ethanwee1]

Fixes SWDEV-540240, SWDEV-540309, SWDEV-539989

Error

#24 437.7   what():  HIP error: no ROCm-capable device is detected
#24 437.7 HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
#24 437.7 For debugging consider passing AMD_SERIALIZE_KERNEL=3
#24 437.7 Device-side assertions were explicitly omitted for this error check; the error probably arose while initializing the DSA handlers.
#24 437.7 Exception raised from c10_hip_check_implementation at /pytorch/c10/hip/HIPException.cpp:44 (most recent call first):
#24 437.7 frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x88 (0x7f272de18738 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
#24 437.7 frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x55 (0x7f272ddb42ed in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
...
#24 437.7 frame #7: at::cuda::getCurrentDeviceProperties() + 0x9 (0x7f270b5874e9 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so)
#24 437.7 frame #8: at::cuda::warp_size() + 0x9 (0x7f270b587509 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so)
#24 437.7 frame #9: <unknown function> + 0x81ac8b (0x7f2709c27c8b in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so)

Explanation

80cca70 created a static global variable that used at::cuda::warp_size() to initialize its value, which needs GPUs to be visible to query device properties. However, GPUs are not present on CPU-only build systems.

Solution

Convert static variable into a static function, thus preventing static initialization.

Validation

http://rocm-ci.amd.com/job/pyt_whl_docker_mainline/1461/artifact/build_artifacts.txt/*view*/

Ran microbenchmark to confirm basic functionality:

root@ubb4-rack-22:/var/lib/jenkins/pytorch-micro-benchmarking# python3 micro_benchmarking_pytorch.py --network resnet50
INFO: running forward and backward for warmup.
INFO: running the benchmark..
OK: finished running benchmark..
--------------------SUMMARY--------------------------
Microbenchmark for network : resnet50
Num devices: 1
Dtype: FP32
Mini batch size [img] : 64
Time per mini-batch : 0.10158218145370483
Throughput [img/sec] : 630.0317544289736=

[rocm-repo-management-api]

Jenkins build for 9991022d48d5480423fc3dc1d3b0fb93cdaa638a commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

[rocm-repo-management-api]

Jenkins build for 9991022d48d5480423fc3dc1d3b0fb93cdaa638a commit is in progress
Links: Blue Ocean view / Build artifacts

[jeffdaily]

I like it. Please submit upstream PR as well.

[jeffdaily]

I like it. Please submit upstream PR as well.

Actually, can you upstream this in combination with 80cca70.