GPU index to compute unit mapping on Slurm cluster

[ncreddine]

Hello @mahendrapaipuri,

I have a question about the proper way to set up GPU mapping with ceems.

We're running ceems_exporter 0.12.3 on a heterogeneous SLURM cluster. We noticed ceems_compute_unit_gpu_index_flag reporting wrong GPU UUIDs for some jobs, causing incorrect GPU usage % in the dashboard.

On nodes where SLURM's gres.conf uses AutoDetect=nvml, SLURM allocates GPUs in NVML/PCI order, and SLURM_STEP_GPUS already contains NVML indices, so ceems maps correctly without any gpu-order-map. We verified this across our cluster.

However, on nodes where gres.conf uses explicit File=/dev/nvidiaN entries (without AutoDetect), SLURM gives kernel device indices in SLURM_STEP_GPUS. When kernel minor numbers differ from NVML indices (which happens on several of our multi-GPU nodes), the mapping breaks.

The philosophy page states:

"DCGM/Device metric exporters exposes the GPU metrics of each GPU and the current exporter takes care of the GPU index to compute unit mapping."

Based on this, I assumed the GPU order was guessed automatically and that gpu-order-map was only involved when using MIG instances.

Is the broken mapping with File= expected behavior, and is gpu-order-map the intended solution for this case?

[mahendrapaipuri]

Hello @ncreddine

As mentioned in the docs and the bug report, I noticed this incosistencies between SLURM and NVIDIA ordering. The case where I noticed this behaviour is when there is a mix of physical and MIG instances. But maybe there are more cases.

Yes, --collector.slurm.gpu-order-map should be the preferable way to handle these cases. Can you provide more details on your case? What is the order reported by nvidia-smi and what is the order "assumed" by SLURM?

Is the broken mapping with File= expected behavior, and is gpu-order-map the intended solution for this case?

I did not have any response from SLURM folks on my bug report. As we are not paying customers, they did not care. So, I cannot really comment on the SLURM's behaviour. If you think --collector.slurm.gpu-order-map can work in your case, yes, it should be the solution!

[ncreddine]

Hi @mahendrapaipuri,

Ah, ok, sorry, I hadn't noticed at first that you already referenced it in the docs. For the whole story: I upgraded to SLURM 24.11 and was using AutoDetect=nvml, but I had to disable it because of a NUMA node mismatch that put the some node into an INVAL state. Enabling SlurmdParameters=l3cache_as_socket fixed that, so I'm keeping that setting. With AutoDetect=nvml SLURM enumerates the GPUs in NVML order and the mapping is correct without any gpu-order-map.

Here are the details of my case. It's on a node with 4× H100 NVL, File=/dev/nvidia[0-3]

Order reported by nvidia-smi :

GPU 0   00000000:02:00.0   GPU-62428057   
GPU 1   00000000:64:00.0   GPU-49d749a2   
GPU 2   00000000:82:00.0   GPU-2b75541b
GPU 3   00000000:E3:00.0   GPU-eafa2310

Order assumed by SLURM (with File=/dev/nvidia[0-3]

SLURM index	/dev	kernel minor	PCI bus	UUID	= nvidia-smi index
0	/dev/nvidia0	0	64:00.0	GPU-49d749a2	1
1	/dev/nvidia1	1	02:00.0	GPU-62428057	0
2	/dev/nvidia2	2	e3:00.0	GPU-eafa2310	3
3	/dev/nvidia3	3	82:00.0	GPU-2b75541b	2

So the kernel assigns /dev/nvidiaN minor numbers in PCIe probe order, while nvidia-smi/NVML sorts by PCI bus address.

One question: can this binding change after a reboot (e.g. if the kernel probes the PCIe devices in a different order)?

Also, since the mismatch is fully derivable at runtime, would it make sense for ceems to build this map automatically when File= is used? The kernel minor → PCI bus mapping is available under /proc/driver/nvidia/gpus/*/information (Device Minor + Bus Location), and the NVML order is just those buses sorted by address. ceems already parses nvidia-smi --query --xml-format (which gives the bus IDs), so it could cross-reference the two and reindex the devices without needing the operator to pass gpu-order-map manually.

[mahendrapaipuri]

Hello @ncreddine

Thanks for the detailed report. Appreciate your time and effort!!

One question: can this binding change after a reboot (e.g. if the kernel probes the PCIe devices in a different order)?

I hope they dont change, but I cant be sure. I assume you might have multiple nodes with the same configuration! Did you check this numbering on different nodes? Do they have all the same mapping between device minors and PCI adresses? That would be a good test

Also, since the mismatch is fully derivable at runtime, would it make sense for ceems to build this map automatically when File= is used?

There are too many different cases that can go wrong. For example, in the bug report that I linked above, the ordering of GPUs changed depending on the order they have been declared in partitions.conf (not gres.conf). Maybe in 24.x, they might have improved it. It is going to be very hard to cover all the cases and different versions of SLURM. That is how I ended up provided this CLI argument which can be used based on the user's need.

Is it going to be a big issue to configure this on your infrastructure?

[mahendrapaipuri]

Hello @ncreddine

Seems like we can use slurmd -G command which lists GPUs on the node. On my demo instance (with fake GPUs), I have this output:

slurmd: Gres Name=gpu Type=a100 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=0,1 CoreCnt=4 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
slurmd: Gres Name=gpu Type=a100 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Cores=0,1 CoreCnt=4 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
slurmd: Gres Name=gpu Type=a100 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=0,1 CoreCnt=4 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
slurmd: Gres Name=gpu Type=a100 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Cores=2,3 CoreCnt=4 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
slurmd: Gres Name=gpu Type=a100 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Cores=2,3 CoreCnt=4 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT

Could you check on your nodes where ordering between NVML and SLURM is different and check if Index reported by SLURM in the output matches with its internal ordering? I would also appreciate if you could do some tests using mix of physical and MIG instances and check if we can rely on the output of slurmd -G. If that works, we can detect the ordering at the runtime and manage this mapping automatically!

Cheers!

[ncreddine]

Hi @mahendrapaipuri,

Thank you for your reply.

I tested both the slurmd -G idea and a real MIG mix. Here's what came out.

slurmd -G only gives useful per-device info when AutoDetect=nvml is on, which is the case where ceems already maps correctly without a map. In the case that actually breaks, plain File=/dev/nvidia[0-3], it collapses to one line:

slurmd: warning: A line in gres.conf for GRES gpu:h100 has 4 more configured than expected in slurm.conf. Ignoring extra GRES.
slurmd: Gres Name=gpu Type=(null) Count=4 Index=0 ID=7696487 File=/dev/nvidia[0-3] Links=(null) Flags=HAS_FILE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT

With one File= line per GPU you do get per-device lines, but the Index is just the device-file order, not NVML order:

slurmd: Gres Name=gpu Type=(null) Count=1 Index=0 File=/dev/nvidia0 Links=(null) Flags=HAS_FILE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
slurmd: Gres Name=gpu Type=(null) Count=1 Index=1 File=/dev/nvidia1 Links=(null) Flags=HAS_FILE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
slurmd: Gres Name=gpu Type=(null) Count=1 Index=2 File=/dev/nvidia2 Links=(null) Flags=HAS_FILE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
slurmd: Gres Name=gpu Type=(null) Count=1 Index=3 File=/dev/nvidia3 Links=(null) Flags=HAS_FILE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT

Index 0-3 just map to /dev/nvidia0-3, i.e. kernel-minor order, which on this box is still wrong: /dev/nvidia0 (minor 0) is bus 64:00.0 = NVML 1, /dev/nvidia1 (minor 1) is bus 02:00.0 = NVML 0, and so on. So you still need the same 0:1,1:0,2:3,3:2, the per-line form doesn't change anything.

The PCI bus approach works better. Each minor's bus is in /proc/driver/nvidia/gpus/*/, and since nvidia-smi/NVML/DCGM enumerate in PCI-bus order, sorting the buses ascending gives the NVML index. No SLURM needed, and it's stable across reboots because the bus addresses are fixed.

~# ls /proc/driver/nvidia/gpus/*/
'/proc/driver/nvidia/gpus/0000:02:00.0/':
information  power  registry

'/proc/driver/nvidia/gpus/0000:64:00.0/':
information  power  registry

'/proc/driver/nvidia/gpus/0000:82:00.0/':
information  power  registry

'/proc/driver/nvidia/gpus/0000:e3:00.0/':
information  power  registry

~# cat /proc/driver/nvidia/gpus/0000\:02\:00.0/information|grep "Device Minor"
Device Minor:    1

Two references for this: the CUDA guide says CUDA_DEVICE_ORDER=PCI_BUS_ID enumerates "by PCI bus ID in ascending order" (https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/environment-variables.html), and the NVML reference for nvmlDeviceGetHandleByIndex says the index itself isn't reliable: "The order in which NVML enumerates devices has no guarantees of consistency between reboots... it is recommended that devices be looked up by their PCI ids or board serial numbers" (https://docs.nvidia.com/deploy/nvml-api/group__nvmlDeviceQueries.html). Sorting by bus reproduces our exact maps on two topologies (4xH100 gives 0:1,1:0,2:3,3:2, 3xA100 gives 0:0,1:2,2:1), and it's the same across all nodes of a given chassis. DCGM also already exposes a pci_bus_id label, so the join could be done on that or the UUID directly.

For the MIG test on the 4xH100 node I enabled MIG on GPU 0, made 4x 1g.24gb, and left GPUs 1/2/3 full:

GPU 0: NVIDIA H100 NVL (UUID: GPU-62428057-ac39-91fd-e36f-8f178e20ae34)
  MIG 1g.24gb     Device  0: (UUID: MIG-f2aa346e-b98b-53c2-b3f3-d28aaed76949)
  MIG 1g.24gb     Device  1: (UUID: MIG-a077e7f7-6627-559b-8590-c58ea2d2124e)
  MIG 1g.24gb     Device  2: (UUID: MIG-cee5952a-0611-539b-adb5-85c48ee21fae)
  MIG 1g.24gb     Device  3: (UUID: MIG-eb71d091-c399-5281-b967-89f60e2adf5e)
GPU 1: NVIDIA H100 NVL (UUID: GPU-49d749a2-7288-11bc-47e4-4af638b39c14)
GPU 2: NVIDIA H100 NVL (UUID: GPU-2b75541b-9e39-8c71-3c5b-810ed64acee9)
GPU 3: NVIDIA H100 NVL (UUID: GPU-eafa2310-9431-91c8-ff0c-d7b3af52475a)

And the full nvidia-smi view, MIG enabled only on GPU 0 (bus 02:00.0) with its four instances:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.71.05              Driver Version: 595.71.05      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 NVL                Off |   00000000:02:00.0 Off |                   On |
| N/A   45C    P0             63W /  400W |      58MiB /  95830MiB |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 NVL                Off |   00000000:64:00.0 Off |                    0 |
| N/A   45C    P0             66W /  400W |       0MiB /  95830MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA H100 NVL                Off |   00000000:82:00.0 Off |                    0 |
| N/A   45C    P0             64W /  400W |       0MiB /  95830MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA H100 NVL                Off |   00000000:E3:00.0 Off |                    0 |
| N/A   44C    P0             64W /  400W |       0MiB /  95830MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| MIG devices:                                                                            |
+------------------+----------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |              Shared Memory-Usage |        Vol|        Shared         |
|      ID  ID  Dev |                Shared BAR1-Usage | SM     Unc| CE ENC  DEC  OFA  JPG |
|                  |                                  |        ECC|                       |
|==================+==================================+===========+=======================|
|  0    3   0   0  |              15MiB / 22144MiB    | 26      0 |  1   0    1    0    1 |
|                  |               0MiB /  8809MiB    |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0    4   0   1  |              15MiB / 22144MiB    | 26      0 |  1   0    1    0    1 |
|                  |               0MiB /  8809MiB    |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0    5   0   2  |              15MiB / 22144MiB    | 26      0 |  1   0    1    0    1 |
|                  |               0MiB /  8809MiB    |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0    6   0   3  |              15MiB / 22144MiB    | 26      0 |  1   0    1    0    1 |
|                  |               0MiB /  8809MiB    |           |                       |
+------------------+----------------------------------+-----------+-----------------------+

With File=, slurmd -G doesn't see the MIG instances at all. With AutoDetect=nvml it sees them, but the indexing isn't usable. If the GRES count is too low it silently drops devices, so I set node53 to Gres=gpu:7 to match the 4 MIG + 3 full:

slurmd: Gres Name=gpu Type=(null) Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=4-7 CoreCnt=64 Links=12,-1,0,0 Flags=HAS_FILE,ENV_NVML
slurmd: Gres Name=gpu Type=(null) Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=20-23 CoreCnt=64 Links=0,0,12,-1 Flags=HAS_FILE,ENV_NVML
slurmd: Gres Name=gpu Type=(null) Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Cores=28-31 CoreCnt=64 Links=0,0,-1,12 Flags=HAS_FILE,ENV_NVML
slurmd: Gres Name=gpu Type=(null) Count=1 Index=166 ID=7696487 File=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap165,/dev/nvidia-caps/nvidia-cap166 Cores=12-15 CoreCnt=64 Links=(null) Flags=HAS_FILE,ENV_NVML
slurmd: Gres Name=gpu Type=(null) Count=1 Index=175 ID=7696487 File=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap174,/dev/nvidia-caps/nvidia-cap175 Cores=12-15 CoreCnt=64 Links=(null) Flags=HAS_FILE,ENV_NVML
slurmd: Gres Name=gpu Type=(null) Count=1 Index=184 ID=7696487 File=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap183,/dev/nvidia-caps/nvidia-cap184 Cores=12-15 CoreCnt=64 Links=(null) Flags=HAS_FILE,ENV_NVML
slurmd: Gres Name=gpu Type=(null) Count=1 Index=193 ID=7696487 File=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap192,/dev/nvidia-caps/nvidia-cap193 Cores=12-15 CoreCnt=64 Links=(null) Flags=HAS_FILE,ENV_NVML

Now all 7 show up, but the index space mixes two schemes: the full GPUs keep their kernel-minor indices (0, 2, 3, nvidia1 is the MIG parent), and the MIG instances get their CI-access cap minors (166, 175, 184, 193). The normal output doesn't include the MIG-UUID either.

Cheers!

[ncreddine]

Btw, DCGM handles the same mix fine. The three full GPUs come through on DCGM_FI_DEV_GPU_UTIL with their UUID and pci_bus_id:

DCGM_FI_DEV_GPU_UTIL{gpu="1",UUID="GPU-49d749a2-7288-11bc-47e4-4af638b39c14",pci_bus_id="00000000:64:00.0",device="nvidia1",modelName="NVIDIA H100 NVL",Hostname="node53",DCGM_FI_DEV_UUID="GPU-49d749a2-7288-11bc-47e4-4af638b39c14",DCGM_FI_DRIVER_VERSION="595.71.05"} 0
DCGM_FI_DEV_GPU_UTIL{gpu="2",UUID="GPU-2b75541b-9e39-8c71-3c5b-810ed64acee9",pci_bus_id="00000000:82:00.0",device="nvidia2",...} 0
DCGM_FI_DEV_GPU_UTIL{gpu="3",UUID="GPU-eafa2310-9431-91c8-ff0c-d7b3af52475a",pci_bus_id="00000000:E3:00.0",device="nvidia3",...} 0

The MIG instances don't appear on GPU_UTIL. They show up on the clock/profiling metrics, and each line carries the parent GPU UUID, the MIG-UUID and the GPU instance ID together:

DCGM_FI_DEV_SM_CLOCK{gpu="0",UUID="GPU-62428057-ac39-91fd-e36f-8f178e20ae34",pci_bus_id="00000000:02:00.0",device="nvidia0",modelName="NVIDIA H100 NVL",GPU_I_PROFILE="1g.22gb",GPU_I_ID="3",Hostname="node53",DCGM_FI_DEV_UUID="MIG-f2aa346e-b98b-53c2-b3f3-d28aaed76949",DCGM_FI_DRIVER_VERSION="595.71.05"} 345
DCGM_FI_DEV_SM_CLOCK{gpu="0",...,GPU_I_ID="4",...,DCGM_FI_DEV_UUID="MIG-a077e7f7-6627-559b-8590-c58ea2d2124e",...} 345
DCGM_FI_DEV_SM_CLOCK{gpu="0",...,GPU_I_ID="5",...,DCGM_FI_DEV_UUID="MIG-cee5952a-0611-539b-adb5-85c48ee21fae",...} 345
DCGM_FI_DEV_SM_CLOCK{gpu="0",...,GPU_I_ID="6",...,DCGM_FI_DEV_UUID="MIG-eb71d091-c399-5281-b967-89f60e2adf5e",...} 345

So DCGM_FI_DEV_UUID is the MIG-UUID (matches nvidia-smi and slurmd's UniqueId), UUID/pci_bus_id are the parent's, and GPU_I_ID 3-6 match nvidia-smi mig -lgi. Worth noting DCGM's device="nvidia0" is the NVML index, not the kernel minor (bus 02:00.0 is /dev/nvidia1 here), so DCGM follows the same NVML/PCI order as nvidia-smi.

Cheers!

[mahendrapaipuri]

Hello @ncreddine

Thanks a lot for all this effort!! Really appreciate your time!

Index 0-3 just map to /dev/nvidia0-3, i.e. kernel-minor order, which on this box is still wrong: /dev/nvidia0 (minor 0) is bus 64:00.0 = NVML 1, /dev/nvidia1 (minor 1) is bus 02:00.0 = NVML 0, and so on.

This is fine! What we need is how SLURM orders the GPU and we dont really care which order it follows. If it is strictly following kernel ordering, we can always fetch the PCI bus ID of each device file and eventually build a map of indices.

Actually NVIDIA drivers should probe the devices in PCI bus ID order and assign kernel minors in the same order. Not sure why they do this strange ordering on this box. Anyways, my takeaway here is SLURM only respects kernel minor ordering. Can you confirm this?

A quick question: On this box where PCI bus IDs and kernel minors are not matching, what does slurmd -G outputs when you have AutoDetect=nvml?

The PCI bus approach works better. Each minor's bus is in /proc/driver/nvidia/gpus/*/, and since nvidia-smi/NVML/DCGM enumerate in PCI-bus order, sorting the buses ascending gives the NVML index. No SLURM needed, and it's stable across reboots because the bus addresses are fixed.

Not necessarily. Imagine a case where kernel minors and PCI bus IDs match but if someone has their gres.conf like below:

Name=gpu Type=tesla File=/dev/nvidia0 COREs=0,1
Name=gpu Type=tesla  File=/dev/nvidia2 COREs=2,3
Name=gpu Type=tesla File=/dev/nvidia1 COREs=4,5
Name=gpu Type=tesla  File=/dev/nvidia3 COREs=6,7

What happens in this case? Does /dev/nvidia2 becomes GPU 1 for SLURM and /dev/nvidia1 becomes GPU 2? Need to test. I do not think we can have strong assumptions on how SLURM ordering its GPUs. We still need some realiable information from it to build our mapping.

If the GRES count is too low it silently drops devices, so I set node53 to Gres=gpu:7 to match the 4 MIG + 3 full:

This is normal. If we do not declare the GRES count for the node definition, there is no reason for SLURM to use them. More over in the production, we do not use Gres=gpu:7 in this case as not all GPUs are the same. We need to use a more appropriate declaration something like Gres=gpu:h100:3,gpu:nvidia_h100_1g.24gb:4. In my tests I noticed that the ordering of SLURM changes between declaring the GRES as Gres=gpu:h100:3,gpu:nvidia_h100_1g.24gb:4 and Gres=gpu:nvidia_h100_1g.24gb:4,gpu:h100:3. If you can could you please try these combinations and see if ordering changes or not.

I will also do some tests on my side to see if there are any "realiable" ways to get SLURM ordering.

Thanks again for your help!!

[mahendrapaipuri]

Hello @ncreddine

I have made some tests and as I suspected, SLURM orders the GPUs based on the order declared in gres.conf. And within a job, the order respects device minors. For example if I have gres.conf as follows:

Name=gpu Type=a100 File=/dev/nvidia2
Name=gpu Type=a100  File=/dev/nvidia0
Name=gpu Type=a100 File=/dev/nvidia3
Name=gpu Type=a100  File=/dev/nvidia1

In this case the ordering will be:

SLURM Index	Device File	NVML Index
0	/dev/nvidia2	2
1	/dev/nvidia0	0
2	/dev/nvidia3	3
3	/dev/nvidia1	1

And more interesting thing is if I ask for two GPUs in a job and if SLURM allocates GPU 0 and 1, i.e., /dev/nvidia2 and /dev/nvidia0, within the job GPUs are ordered based on their device minors. In other words, within the job GPU 0 is /dev/nvidia0 and GPU 1 is /dev/nvidia2. Maybe you can do some tests and confirm this?

My bottom line is it is going to be very complicated to get SLURM ordering of GPUs at runtime unless we "reimplement" their logic in the exporter code.

What we can do is to use an alternative technique to get GPU UUIDs of a job reliably. This can be "easily" done by executing the command nvidia-smi or amd-smi within the job cgroup. Although I had this idea since long time and it is technically possible to do so, I never really had motivation to implement it as it is bit too "intrusive" running commands in job cgroup. We will run it only once when job starts so it is not very expensive. So, maybe it is worth providing this option as well.

What we can do is to implement this and put it behind a feature flag so that users can use it if they do not want to fiddle with GPU mapping or in some cases it might be even practical. What do you think?