Use https://docs.nvidia.com/deploy/nvml-api/group__nvmlDeviceStructs.html#group__nvmlDeviceStructs_1g5fd9890987e4b5768817dd64438e24f4 to preferentially choose GPUs with good communication between them.
When allocating single GPUs, try to leave room for multi-GPU allocation.
Use https://docs.nvidia.com/deploy/nvml-api/group__nvmlDeviceStructs.html#group__nvmlDeviceStructs_1g5fd9890987e4b5768817dd64438e24f4 to preferentially choose GPUs with good communication between them.
When allocating single GPUs, try to leave room for multi-GPU allocation.