[SYCL][ROCm] Add HIP NVIDIA support

[npmiller]

This PR enables building the PI ROCm plugin for NVIDIA rather than AMD, has ROCm has some NVIDIA support.

It should allow doing some testing of the ROCm PI plugin on NVIDIA hardware.

This PR includes:

• Refactoring of the ROCm PI plugin CMake and wiring for ROCm NVIDIA
• Buildbot configure script update to add --rocm-platform flag
• Changes to the PI ROCm plugin that was using HIP in an AMD specific way in some places

We are still running some testing on this, however it looks to be similar to the current ROCm plugin on AMD, and a bit better on NVIDIA since the SYCL compiler for it is more mature.

[bader]

Changes in buildbot/configure.py and sycl/doc/GetStartedGuide.md look good to me.

[bader]

@malixian, FYI.

[bader]

@smaslov-intel, @intel/llvm-reviewers-runtime, ping.

[bader]

@npmiller, how can I run sycl/test/on-device tests using ROCm plug-in on NVIDIA HW?
NOTE: there is a special CMake target for running in-tree LIT tests on CUDA back-end - https://github.com/intel/llvm/blob/sycl/sycl/test/CMakeLists.txt#L66-L79 (on-device tests are added to this target here: https://github.com/intel/llvm/blob/sycl/sycl/test/on-device/CMakeLists.txt#L30-L42).

I would expect that there should be CMake target like this for running lit tests with ROCm plug-in on AMD/NVIDIA HW. Right?
Is there another way to run these tests?

[npmiller]

@npmiller, how can I run sycl/test/on-device tests using ROCm plug-in on NVIDIA HW?
NOTE: there is a special CMake target for running in-tree LIT tests on CUDA back-end - https://github.com/intel/llvm/blob/sycl/sycl/test/CMakeLists.txt#L66-L79 (on-device tests are added to this target here: https://github.com/intel/llvm/blob/sycl/sycl/test/on-device/CMakeLists.txt#L30-L42).

I would expect that there should be CMake target like this for running lit tests with ROCm plug-in on AMD/NVIDIA HW. Right?
Is there another way to run these tests?

This is not wired up for the ROCm backend yet. It's definitely something we need however this PR was already getting pretty massive, so I was thinking about adding it in a follow up PR. It'll take a bit of time but if you prefer I can add it in here instead.

[bader]

@npmiller, how can I run sycl/test/on-device tests using ROCm plug-in on NVIDIA HW?
NOTE: there is a special CMake target for running in-tree LIT tests on CUDA back-end - https://github.com/intel/llvm/blob/sycl/sycl/test/CMakeLists.txt#L66-L79 (on-device tests are added to this target here: https://github.com/intel/llvm/blob/sycl/sycl/test/on-device/CMakeLists.txt#L30-L42).
I would expect that there should be CMake target like this for running lit tests with ROCm plug-in on AMD/NVIDIA HW. Right?
Is there another way to run these tests?

This is not wired up for the ROCm backend yet. It's definitely something we need however this PR was already getting pretty massive, so I was thinking about adding it in a follow up PR. It'll take a bit of time but if you prefer I can add it in here instead.

Separate PR sounds good to me. I just wanted to make sure that there is some way to test the ROCm backend. Thanks!

[alexbatashev]

Just curious. @npmiller would it make sense to build rocm plugin for both platforms at the same time and present them as different backends?

[npmiller]

Just curious. @npmiller would it make sense to build rocm plugin for both platforms at the same time and present them as different backends?

It could be, although I think that might make the CMake more awkward and require a bunch of glue in the C++ as well, like all the rocm_pi functions would probably need different prefixes depending on the platform and so on.

Ultimately the NVIDIA support through ROCm is mostly useful for debugging and testing at the moment, for running actual applications on NVIDIA GPUs the CUDA backend should definitely be preferred, HIP goes right back to CUDA anyway.

And building both the CUDA and ROCm backend (for either platform) works fine, so you can have a build that can target both platforms already.

I guess being able to build both of the platforms at the same time would make it easier to validate that a patch doesn't break building either, but overall I'm not too sure it's worth the extra complexity in the plugin or in the CMake.

[bader]

@smaslov-intel, @intel/llvm-reviewers-runtime, ping.

[bader]

@smaslov-intel, ping.

[smaslov-intel]

This PR includes:

• Refactoring of the ROCm PI plugin CMake and wiring for ROCm NVIDIA
• Buildbot configure script update to add --rocm-platform flag
• Changes to the PI ROCm plugin that was using HIP in an AMD specific way in some places

I approve the changes as is, but would prefer these be broken into multiple change-sets.