Flatcar not supporting EKS on versions above 1.21

[EhsanLunar]

Description

When an EC2 instance boots up with a flatcar image (even the latest) the kubelet fails.
The userdata defines (and should do so) that the /etc/eks/bootstrap.sh should run, which it does.

This seems to add a ExecStartPre to the kubelet.service:
ExecStartPre=/usr/share/oem/eks/download-kubelet.sh

Both the bootstrap.sh and the download-kubelet.sh are consistent with:
https://github.com/flatcar-linux/coreos-overlay/blob/main/coreos-base/flatcar-eks/files/bootstrap.sh
https://github.com/flatcar-linux/coreos-overlay/blob/main/coreos-base/flatcar-eks/files/download-kubelet.sh

The download-kubelet.sh fails with Unsupported Kubernetes version because in the case statement on line 24->50 (https://github.com/flatcar-linux/coreos-overlay/blob/main/coreos-base/flatcar-eks/files/download-kubelet.sh#L25) only has values for kubernetes version 1.15 -> 1.21

If I manually alter the file and add 1.22 (when I test this on 1.22.9 kubernetes version deployment) and re-run the bootsrap.sh it works fine as far as I can see, the node than joins the cluster and shows up as Ready and pods starting running on the node.

The last PR I can see on this particular thing was done about a year ago flatcar-archive/coreos-overlay@f0da7f8

Impact

No EKS support for kubernetes versions higher than 1.21

Environment and steps to reproduce

1. Set-up: Create an EKS cluster with the latest flatcar AMI in the worker nodes
2. Task: SSH into the node (probably through a Bastion)
3. Action(s): No actions needed
4. Error: kubelet.service fails because the download-kubelet.sh doesn't have download locations for kubernetes version above 1.21

Expected behavior

Download locations for kubernetes versions 1.22 and 1.23 (EKS doesn't have support for 1.24 yet it seems) should be located inside the download-kubelet.sh

Additional information

By running aws s3 ls s3://amazon-eks/ you can list the available locations of the other versions, so for it should result in this:

case $CLUSTER_VERSION in
  1.23)
    S3_PATH="1.23.9/2022-07-27/"
    ;;
  1.22)
    S3_PATH="1.22.12/2022-07-27/"
    ;;
  1.21)
    S3_PATH="1.21.2/2021-07-05"
    ;;
  1.20)
    S3_PATH="1.20.4/2021-04-12"
    ;;
  1.19)
    S3_PATH="1.19.6/2021-01-05"
    ;;
  1.18)
    S3_PATH="1.18.9/2020-11-02"
    ;;
  1.17)
    S3_PATH="1.17.12/2020-11-02"
    ;;
  1.16)
    S3_PATH="1.16.15/2020-11-02"
    ;;
  1.15)
    S3_PATH="1.15.12/2020-11-02"
    ;;
  *)
    echo "Unsupported Kubernetes version"
    exit 1
    ;;
esac

[tormath1]

Hi @EhsanLunar and thanks for the investigation on this issue.

If I manually alter the file and add 1.22

Would you be interested to fill a PR to add the EKS support for 1.22 and 1.23, that would be great!

[EhsanLunar]

Hi @EhsanLunar and thanks for the investigation on this issue.

If I manually alter the file and add 1.22

Would you be interested to fill a PR to add the EKS support for 1.22 and 1.23, that would be great!

@tormath1 Sure thing, here it is :) flatcar-archive/coreos-overlay#2110

[krnowak]

That sounds like a candidate for a github action to keep that list up to date.