[EKS] [request]: vpc-resource-controller does not support .spec.runtimeClassName

[Vlatombe]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Kubernetes 1.17 and above provides RuntimeClass, which simplifies the process of using taints and tolerations.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
I set up a RuntimeClass as follow

apiVersion: node.k8s.io/v1beta1
handler: docker
kind: RuntimeClass
metadata:
  name: windows-2019
scheduling:
  nodeSelector:
    kubernetes.io/arch: amd64
    kubernetes.io/os: windows
    node.kubernetes.io/windows-build: 10.0.17763
  tolerations:
  - effect: NoSchedule
    key: kubernetes.io/os
    operator: Equal
    value: windows

then I schedule a pod referencing the RuntimeClass

apiVersion: v1
kind: Pod
metadata:
  name: windows-runtimeclass
spec:
  runtimeClassName: windows-2019
  containers:
  - name: shell
    args:
    - Start-Sleep
    - "999999"
    command:
    - powershell
    image: mcr.microsoft.com/powershell:preview-windowsservercore-1809
    imagePullPolicy: IfNotPresent

Then the pod stays in ContainerCreating, and checking events show

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7fac6d98a43b291ebb2b9b0386cc4bb5e14c5b78984c27d6dd1ba438907990b6" network for pod "windows-runtimeclass": networkPlugin cni failed to set up pod "windows-runtimeclass_vlatombe" network: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4Address

Inspecting the resulting pod spec, the resources requests and limits vpc.amazonaws.com/PrivateIPv4Address are unset.

Are you currently working around this issue?

Adding to the container

    resources:
      limits:
        vpc.amazonaws.com/PrivateIPv4Address: 1
      requests:
        vpc.amazonaws.com/PrivateIPv4Address: 1

allows the pod to be scheduled.

[tzifudzi]

This issue can be closed without requiring any further changes in https://github.com/aws/amazon-vpc-resource-controller-k8s. The workaround is no longer needed.

Tests performed to confirm

This is based on tests performed using the following YAML.

apiVersion: node.k8s.io/v1
handler: runhcs-wcow-process
kind: RuntimeClass
metadata:
  name: containerd-windows-2022
scheduling:
  nodeSelector:
    kubernetes.io/arch: amd64
    kubernetes.io/os: windows
    node.kubernetes.io/windows-build: 10.0.20348
  tolerations:
  - effect: NoSchedule
    key: kubernetes.io/os
    operator: Equal
    value: windows
---
apiVersion: v1
kind: Pod
metadata:
  name: windows-eks-runtimeclass-test
spec:
  runtimeClassName: containerd-windows-2022
  containers:
  - name: shell
    command:
      - "ping"
      - "-t"
      - "127.0.0.1"
    image: mcr.microsoft.com/windows/servercore:ltsc2022
    imagePullPolicy: IfNotPresent

Tested on

• Windows 2022 10.0.20348
• AMI: ami-0b492f4a11b2c9785
• Kubelet: v1.28.3-eks-e71965b
• EKS Version: eks.4
• Container runtime: containerd://1.6.18

Why issue was resolved without needing changes in VPC Resource Controller

• The issue was resolved with the change to automatically merge the RuntimeClass's nodeSelector is into the pod's nodeSelector in admission, thereby meaning the VPC resource controller is always aware of the pods scheduling needs and can therefore handle IP address management on behalf of the pod. This is handled by the in-tree RuntimeClass admission controller.
• I suspect that at the time the issue was raised, this mechanism may have been disabled behind an alpha feature gate which therefore required the workaround mentioned. The Runtimeclass admission controller feature went GA in 1.20 release. See references for more.

References

• https://kubernetes.io/docs/concepts/containers/runtime-class/
• https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md
• RuntimeClass GA kubernetes/kubernetes#95718
• Runtimeclass admission kubernetes/kubernetes#78484
• https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/585-runtime-class/README.md