Update `k8s-cc-manager` image to support Blackwell GPUs

[msanft]

The default image for the k8s-cc-manager is over 2 years old:

gpu-operator/deployments/gpu-operator/values.yaml

Lines 500 to 511 in 75d1b96

	ccManager:
	enabled: false
	defaultMode: "off"
	repository: nvcr.io/nvidia/cloud-native
	image: k8s-cc-manager
	version: v0.1.1
	imagePullPolicy: IfNotPresent
	imagePullSecrets: []
	env:
	- name: CC_CAPABLE_DEVICE_IDS
	value: "0x2339,0x2331,0x2330,0x2324,0x2322,0x233d"
	resources: {}

It would be great if it could be bumped to one that supports Blackwell GPUs. I tested ghcr.io/nvidia/k8s-cc-manager:d80a3a8a-distroless, which is the latest release. I can confirm this to work with B200s.

Along that, the CC_CAPABLE_DEVICE_IDS environment variable needs to be updated.

[zvonkok]

@msanft We have new images that lift this restriction. We're going to publish a new deployment guide soon-ish. In the meantime, you can update the image to point to nvcr.io/nvidia/cloud-native/k8s-cc-manager:v0.2.0 via ClusterPolicy.

[manuelh-dev]

@msanft - some documentation for this is on the way, see the current draft here: kata-containers/kata-containers#12257 - might be good to give this a try.

[msanft]

Thank you @zvonkok @manuelh-dev!

This worked quite well. The only thing we did is strip the P_GPU_ALIAS environment variable since we need per-GPU annotations for heterogenous clusters (i.e. nvidia.com/GB100_B200 instead of nvidia.com/pgpu). Is there another recommended way of achieving this in your reference deployment guide?

Furthermore, we had to do these two little patches for Blackwell support:

# Support Blackwell GPUs in nvidia-cc-manager
current_ids=$(kubectl get daemonset nvidia-cc-manager -n gpu-operator -o jsonpath='{.spec.template.spec.containers[?(@.name=="nvidia-cc-manager")].env[?(@.name=="CC_CAPABLE_DEVICE_IDS")].value}')
kubectl set env daemonset/nvidia-cc-manager -n gpu-operator -c nvidia-cc-manager CC_CAPABLE_DEVICE_IDS="${current_ids},0x2901"

# Deploy nvidia-cc-manager to nodes with Blackwell GPUs
kubectl patch nodefeaturerule nvidia-nfd-nodefeaturerules --type='json' -p='[
  {
    "op": "add",
    "path": "/spec/rules/10",
    "value": {
      "name": "NVIDIA B200",
      "labels": {
        "nvidia.com/gpu.B200": "true",
        "nvidia.com/gpu.family": "blackwell"
      },
      "matchFeatures": [
        {
          "feature": "pci.device",
          "matchExpressions": {
            "device": {"op": "In", "value": ["2901"]},
            "vendor": {"op": "In", "value": ["10de"]}
          }
        }
      ]
    }
  },
  {
    "op": "add",
    "path": "/spec/rules/11/matchAny/0/matchFeatures/0/matchExpressions/nvidia.com~1gpu.family/value/-",
    "value": "blackwell"
  },
  {
    "op": "add",
    "path": "/spec/rules/11/matchAny/1/matchFeatures/0/matchExpressions/nvidia.com~1gpu.family/value/-",
    "value": "blackwell"
  }
]'

The latter one should be solved by #1973 if my comment there is taken into account.