feat: add support for nvidia MIG

[piyush-jena]

Issue number:

Related:

• feat: add nvidia MIG Settings bottlerocket-settings-sdk#63
• feat: add nvidia MIG bottlerocket-core-kit#258
• Support NVIDIA MIG for more secure multi-tenant GPU sharing / allocation bottlerocket#4252

Description of changes:
Adding nvidia-migmanager service and binary that configures the instance with nvidia mig.

Testing done:

1. Instance joined the cluster

NAME                                           STATUS   ROLES    AGE   VERSION
ip-XXXX.us-west-2.compute.internal   Ready    <none>   15h   v1.29.5-eks-1109419

2. Model Default:

bash-5.1# apiclient get settings.kubelet-device-plugin
{
  "settings": {
    "kubelet-device-plugins": {
      "nvidia": {
        "device-id-strategy": "index",
        "device-list-strategy": "volume-mounts",
        "device-partitioning-strategy": "none",
        "device-sharing-strategy": "none",
        "pass-device-specs": true
      }
    }
  }
}

3. Model Updates:

bash-5.1#: apiclient set settings.kubelet-device-plugins.nvidia.device-partitioning-strategy="mig"
apiclient apply <<EOF
[settings.kubelet-device-plugins.nvidia.mig.profile]
"a100.40gb"="1g.5gb"
"h100.80gb"="4"
EOF
bash-5.1# apiclient get settings.kubelet-device-plugin
{
  "settings": {
    "kubelet-device-plugins": {
      "nvidia": {
        "device-id-strategy": "index",
        "device-list-strategy": "volume-mounts",
        "device-partitioning-strategy": "mig",
        "device-sharing-strategy": "none",
        "mig": {
          "profile": {
            "a100.40gb": "1g.5gb",
            "h100.80gb": "4",
          }
        },
        "pass-device-specs": true
      }
    }
  }
}

kubectl describe node shows 56 gpus post instance reboot.

4. Bounded check:

bash-5.1# apiclient apply <<EOF
> [settings.kubelet-device-plugins.nvidia.mig.profile]
> "hello"="1g.5gb"
> EOF
Failed to apply settings: Failed to PATCH settings from '-' to '/settings?tx=apiclient-apply-7NsnlaurtHEacSYL': Status 400 when PATCHing /settings?tx=apiclient-apply-7NsnlaurtHEacSYL: Json deserialize error: Unable to deserialize into NvidiaGPUModel: NVIDIA GPU Model must match '^([a-z])(\d+)\.(\d+)gb$', given: hello at line 1 column 62
bash-5.1# apiclient apply <<EOF
> [settings.kubelet-device-plugins.nvidia.mig.profile]
> "a100.40gb"="2"
> EOF
bash-5.1# apiclient apply <<EOF
> [settings.kubelet-device-plugins.nvidia.mig.profile]
> "a100.40gb"="5"
> EOF
Failed to apply settings: Failed to PATCH settings from '-' to '/settings?tx=apiclient-apply-GzUHB0axGlWNPzGw': Status 400 when PATCHing /settings?tx=apiclient-apply-GzUHB0axGlWNPzGw: Json deserialize error: Unable to deserialize into MIGProfile: MIG Profile must match '^[0-9]g\.\d+gb$', given: 5 at line 1 column 71

5. Files generated:

bash-5.1# cat /etc/nvidia-migmanager/nvidia-migmanager.toml
device-partitioning-strategy = "mig"
profile = { "a100.40gb" = "1g.5gb", "h100.80gb" = "4" }

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[bcressey]

nit: maybe not worth doing, but you could repeat the serde alias approach and deserialize the enum variant from the PCI device ID.

[piyush-jena]

Force push fixes all the comments.

[piyush-jena]

Force push fixes the above comments.

[bcressey]

Nice! 🥇

[piyush-jena]

Fixed the above 2 comments

[arnaldo2792]

nit:

Suggested change

	Summary: Tool to generate NVIDIA MIG Binary and config files
	Summary: Tool manage NVIDIA MIG and its config files