The GPU ID in the gpu-admission and gpu-manager sorting functions is inconsistent

[fighterhit]

Hi @mYmNeo , today I found that the GPU ID in the gpu-manager and gpu-admission sorting functions seem to be inconsistent.

From the judgment logic of the final GPU selection, gpu admission considers the ID as the GPU Minor Number,

gpu-manager/pkg/services/allocator/nvidia/allocator.go

Lines 444 to 452 in b0770c7

	if idxStr, ok := pod.ObjectMeta.Annotations[types.PredicateGPUIndexPrefix+strconv.Itoa(containerIndex)]; ok {
	if _, err := strconv.Atoi(idxStr); err != nil {
	return nil, fmt.Errorf("predicate idx %s invalid for pod %s ", idxStr, pod.UID)
	}
	devStr = types.NvidiaDevicePrefix + idxStr
	if !utils.IsValidGPUPath(devStr) {
	return nil, fmt.Errorf("predicate idx %s invalid", devStr)
	}
	} else {

gpu-manager/pkg/services/allocator/nvidia/allocator.go

Lines 462 to 465 in b0770c7

	if predicateNode.MinorName() != nodes[0].MinorName() {
	return nil, fmt.Errorf("Nvidia node mismatch for pod %s(%s), pick up:%s predicate: %s",
	pod.Name, container.Name, nodes[0].MinorName(), predicateNode.MinorName())
	}

However, the gpu-manager sorting function uses the GPU index. For example, in share mode, the Meta.ID used by the sorting algorithm nvidia.ByID is actually the GPU index, as is the link and fragment mode.

gpu-manager/pkg/algorithm/nvidia/share.go

Lines 43 to 48 in b0770c7

	func (al *shareMode) Evaluate(cores int64, memory int64) []*nvidia.NvidiaNode {
	var (
	nodes []*nvidia.NvidiaNode
	tmpStore = make([]*nvidia.NvidiaNode, al.tree.Total())
	sorter = shareModeSort(nvidia.ByAllocatableCores, nvidia.ByAllocatableMemory, nvidia.ByPids, nvidia.ByID)
	)

gpu-manager/pkg/device/nvidia/sort.go

Lines 41 to 43 in b0770c7

	ByID = func(p1, p2 *NvidiaNode) bool {
	return p1.Meta.ID < p2.Meta.ID
	}

I observed that when all GPUs are healthy, the GPU index and GPU Minor Number of 1080ti and 2080ti seem to be the same, but GPU Index and GPU Minor Number of 3090 are inconsistent.

• 1080ti:

root@ai-1080ti-25:/var/lib/kubelet# nvidia-smi --query|grep "UUID\|Minor Number"
    GPU UUID                              : GPU-03f50d8b-8228-247c-b186-860762e31dd1
    Minor Number                          : 0
    GPU UUID                              : GPU-c8c2e7f6-b738-ed64-3c65-6ae3e489e4c3
    Minor Number                          : 1
    GPU UUID                              : GPU-5572a570-a180-c4ff-298c-8dd4d4e66ffb
    Minor Number                          : 2
    GPU UUID                              : GPU-df1fa0de-0161-9bc1-eac7-825e9846fa53
    Minor Number                          : 3
    GPU UUID                              : GPU-12c05a23-d8d7-aca7-d0c5-1df2e8d1f6a9
    Minor Number                          : 4
    GPU UUID                              : GPU-cc1f9466-61bd-7c60-1f13-93bdcf5bd9c4
    Minor Number                          : 5
    GPU UUID                              : GPU-38896278-98b0-645f-1fb8-b76bf875cce2
    Minor Number                          : 6
    GPU UUID                              : GPU-f3d43c0f-d9eb-cce4-b8ec-8001bc0ee412
    Minor Number                          : 7
root@ai-1080ti-25:/var/lib/kubelet# nvidia-smi -L
GPU 0: GeForce GTX 1080 Ti (UUID: GPU-03f50d8b-8228-247c-b186-860762e31dd1)
GPU 1: GeForce GTX 1080 Ti (UUID: GPU-c8c2e7f6-b738-ed64-3c65-6ae3e489e4c3)
GPU 2: GeForce GTX 1080 Ti (UUID: GPU-5572a570-a180-c4ff-298c-8dd4d4e66ffb)
GPU 3: GeForce GTX 1080 Ti (UUID: GPU-df1fa0de-0161-9bc1-eac7-825e9846fa53)
GPU 4: GeForce GTX 1080 Ti (UUID: GPU-12c05a23-d8d7-aca7-d0c5-1df2e8d1f6a9)
GPU 5: GeForce GTX 1080 Ti (UUID: GPU-cc1f9466-61bd-7c60-1f13-93bdcf5bd9c4)
GPU 6: GeForce GTX 1080 Ti (UUID: GPU-38896278-98b0-645f-1fb8-b76bf875cce2)
GPU 7: GeForce GTX 1080 Ti (UUID: GPU-f3d43c0f-d9eb-cce4-b8ec-8001bc0ee412)

• 2080ti:

root@ai-2080ti-27:/var/lib/kubelet/device-plugins/checkpoint# nvidia-smi --query|grep "UUID\|Minor Number"
    GPU UUID                        : GPU-a6f07a02-ad51-6e97-025e-8114c2606f4e
    Minor Number                    : 0
    GPU UUID                        : GPU-3c45debc-8d71-11de-cb8b-daf9c6f21aa6
    Minor Number                    : 1
    GPU UUID                        : GPU-fe3f1b7b-9e32-6afc-ba34-bc1213b1d642
    Minor Number                    : 2
    GPU UUID                        : GPU-02664d0a-75e5-787d-51d8-8b949bb503cc
    Minor Number                    : 3
    GPU UUID                        : GPU-fe23b4e7-38a9-df47-20db-c60daa61bac2
    Minor Number                    : 4
    GPU UUID                        : GPU-bb5ce0f7-199f-c572-7da1-6286fa9decde
    Minor Number                    : 5
    GPU UUID                        : GPU-d397629c-4a90-f09b-3021-b32e5abe5267
    Minor Number                    : 6
    GPU UUID                        : GPU-71c7c670-2fa2-c284-f1ec-34ec8fc81f0f
    Minor Number                    : 7
root@ai-2080ti-27:/var/lib/kubelet/device-plugins/checkpoint# nvidia-smi -L
GPU 0: GeForce RTX 2080 Ti (UUID: GPU-a6f07a02-ad51-6e97-025e-8114c2606f4e)
GPU 1: GeForce RTX 2080 Ti (UUID: GPU-3c45debc-8d71-11de-cb8b-daf9c6f21aa6)
GPU 2: GeForce RTX 2080 Ti (UUID: GPU-fe3f1b7b-9e32-6afc-ba34-bc1213b1d642)
GPU 3: GeForce RTX 2080 Ti (UUID: GPU-02664d0a-75e5-787d-51d8-8b949bb503cc)
GPU 4: GeForce RTX 2080 Ti (UUID: GPU-fe23b4e7-38a9-df47-20db-c60daa61bac2)
GPU 5: GeForce RTX 2080 Ti (UUID: GPU-bb5ce0f7-199f-c572-7da1-6286fa9decde)
GPU 6: GeForce RTX 2080 Ti (UUID: GPU-d397629c-4a90-f09b-3021-b32e5abe5267)
GPU 7: GeForce RTX 2080 Ti (UUID: GPU-71c7c670-2fa2-c284-f1ec-34ec8fc81f0f)

• 3090:

root@nvidia-device-plugin-daemonset-cnqx9:/# nvidia-smi --query|grep "UUID\|Minor Number"
    GPU UUID                              : GPU-2990c222-1329-0435-cea0-22d7e95e95cc
    Minor Number                          : 2
    GPU UUID                              : GPU-b6f053bd-13ed-83d4-bfdf-42484b1cf651
    Minor Number                          : 3
    GPU UUID                              : GPU-9cfb2674-f388-2261-a678-bcc78c9c0a91
    Minor Number                          : 1
    GPU UUID                              : GPU-d0d02634-c47c-dec4-6e2e-9e36d8b5d9eb
    Minor Number                          : 0
    GPU UUID                              : GPU-d0631c2e-1f72-7a42-b099-ac119bc828fd
    Minor Number                          : 6
    GPU UUID                              : GPU-a5ae7a84-9efc-249f-282f-293c20c7832e
    Minor Number                          : 7
    GPU UUID                              : GPU-6ca66a67-cd42-c25e-d4a8-23fb90cb4bc2
    Minor Number                          : 5
    GPU UUID                              : GPU-d37e92f9-b1ea-c9f0-24b9-c9814b0da5c6
    Minor Number                          : 4
root@nvidia-device-plugin-daemonset-cnqx9:/# nvidia-smi -L
GPU 0: GeForce RTX 3090 (UUID: GPU-2990c222-1329-0435-cea0-22d7e95e95cc)
GPU 1: GeForce RTX 3090 (UUID: GPU-b6f053bd-13ed-83d4-bfdf-42484b1cf651)
GPU 2: GeForce RTX 3090 (UUID: GPU-9cfb2674-f388-2261-a678-bcc78c9c0a91)
GPU 3: GeForce RTX 3090 (UUID: GPU-d0d02634-c47c-dec4-6e2e-9e36d8b5d9eb)
GPU 4: GeForce RTX 3090 (UUID: GPU-d0631c2e-1f72-7a42-b099-ac119bc828fd)
GPU 5: GeForce RTX 3090 (UUID: GPU-a5ae7a84-9efc-249f-282f-293c20c7832e)
GPU 6: GeForce RTX 3090 (UUID: GPU-6ca66a67-cd42-c25e-d4a8-23fb90cb4bc2)
GPU 7: GeForce RTX 3090 (UUID: GPU-d37e92f9-b1ea-c9f0-24b9-c9814b0da5c6)

So I think the nvidia.ByID in the gpu-manager sorting function may also be sorted according to GPU Minor Number to reduce the occurrence of UnexpectedAdmissionError caused by inconsistent GPU selection. Is this right? Looking forward to your reply.

[mYmNeo]

Yes, this is reasonable, I'll fix ASAP

[fighterhit]

Yes, this is reasonable, I'll fix ASAP

Thanks for your attention @mYmNeo . In addition to nvidia.ByID, the nvidia.ByMemory sort function may also be consistent with gpu-admission, that is, use nvidia.ByAllocatableMemory instead. Although this sort function may not work in theory when the number of GPUs used is an integer (the value of tencent.com/vcuda-core is the multiple of 100), use nvidia. ByAllocatableMemory seems more reasonable.

• gpu-admission ExclusiveMode:
  https://github.com/tkestack/gpu-admission/blob/master/pkg/algorithm/exclusive.go#L48-L51

• gpu-manager fragment and link mode:
  

  gpu-manager/pkg/algorithm/nvidia/fragment.go

  Line 47 in b0770c7

  sorter = fragmentSort(nvidia.ByAvailable, nvidia.ByMemory, nvidia.ByPids, nvidia.ByID)

  
  

  gpu-manager/pkg/algorithm/nvidia/link.go

  Line 42 in b0770c7

  sorter = linkSort(nvidia.ByType, nvidia.ByAvailable, nvidia.ByMemory, nvidia.ByPids, nvidia.ByID)

[mYmNeo]

Fix it #137