SYCL: Filter GPU devices

[masterleinad]

The discussion in https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1705683106903069 shows that users might accidentally use SYCL backends that we don't actively support. This pull request filters the sycl::devices so that only the sycl::backend::ext_oneapi_* ones are visible if a device architecture is enabled in the configuration.

[masterleinad]

https://github.com/pzehner Would you mind having a look if this works for you?

[pzehner]

I tested it successfully on a GPU Max 1550. Everything is good for me.

[masterleinad]

Retest this please.

[masterleinad]

@dalg24 and I discussed this pull request offline.

@dalg24's point of view is:

• this change messes with the visible devices
• we should consider what the runtime command line tool gives us when selecting the device to use (sycl-ls)
• we should print all devices in print_configuration and that should match sycl-ls
• we should discuss how we want to handle initialization when the user doesn't specify a device id. For other backends, we could take the currently active device and for SYCL possibly the first usable one.

My point of view is:

• we only care about ext_oneapi_* devices anyway (if we request a GPU device). Only those are maintained by Intel for Aurora (intel/llvm might maintain opencl as well but we know that our unit tests are not passing for example).
• other Kokkos backends can't target multiple low-level backends for the same architecture. Hence, there is no inconsistency.
• we can still print all devices but again we should only consider ext_oneapi_* ones since only those are feasible.
• we are already using the first usable device for SYCL if the user doesn't specify a device id.

We agreed that we should check in the backward initialization that we are given an ext_oneapi_* backend anyway. Even with this pull request, execution space instances created from sycL::queues could use an opencl:gpu device.

For reference, on Sunspot/Aurora compute nodes the default for ONEAPI_DEVICE_SELECTOR is level_zero:gpu and sycl-ls shows

[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:1] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:2] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:3] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:4] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:5] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]

unsetting that environment variable gives

[opencl:gpu:0] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO  [23.17.26241.22]
[opencl:gpu:1] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO  [23.17.26241.22]
[opencl:gpu:2] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO  [23.17.26241.22]
[opencl:gpu:3] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO  [23.17.26241.22]
[opencl:gpu:4] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO  [23.17.26241.22]
[opencl:gpu:5] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO  [23.17.26241.22]
[opencl:cpu:6] Intel(R) OpenCL, Intel(R) Xeon(R) CPU Max 9470C OpenCL 3.0 (Build 0) [2023.16.10.0.09_181109.xmain-eng]
[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:1] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:2] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:3] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:4] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:5] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]

so everything discussed here only makes a difference if ONEAPI_DEVICE_SELECTOR isn't already specified.

@dalg24 Please correct this comment if I misrepresented something.

[dalg24]

Would prefer if you applied my suggestions but I am fine with the PR as is if you do not agree with them.

[dalg24]

Ignoring HIP ROCm 5.2 failure

[aelovikov-intel]

FYI: I've merged intel/llvm#12719 in the SYCL RT for a different take on this issue.