Add kubernetes-1.28 package and variants #3329

etungsten · 2023-08-04T22:39:53Z

Issue number:

Resolves #3274

Description of changes:
This adds the package for K8s 1.28 for building kubelet and new 1.28 Bottlerocket variants.

The new variants will use 6.1 kernel, include secure boot, and use XFS for the data volume filesystem.

Testing done:

aarch64 IPv4 conformance tests
aarch64 IPv6 conformance tests
aarch64 NVIDIA conformance tests
aarch64 NVIDIA smoke tests
x86_64 IPv4 conformance tests
x86_64 IPv6 conformance tests
x86_64 NVIDIA conformance tests
x86_64 NVIDIA smoke tests

$ testsys status 
 NAME                                                     TYPE   STATE     PASSED   SKIPPED   FAILED 
 beta-aarch64-aws-k8s-128-conformance-test                Test       passed      382      7007      2      
 beta-aarch64-aws-k8s-128-conformance-test-retry-1        Test       passed      4        7387      0      
 beta-aarch64-aws-k8s-128-instances                       Resource   completed                             
 beta-aarch64-aws-k8s-128-ipv6-conformance-test           Test       passed      381      7007      3      
 beta-aarch64-aws-k8s-128-ipv6-conformance-test-retry-1   Test       passed      4        7387      0      
 beta-aarch64-aws-k8s-128-ipv6-instances                  Resource   completed                             
 beta-aarch64-aws-k8s-128-nvidia-ipv6-conformance-test    Test       passed      384      7007      0      
 beta-aarch64-aws-k8s-128-nvidia-ipv6-instances           Resource   completed                             
 beta-aarch64-aws-k8s-128-nvidia-ipv6-nvidia-smoke-test   Test       passed      11       0         0      
 beta-x86-64-aws-k8s-128-conformance-test                 Test       passed      384      7007      0      
 beta-x86-64-aws-k8s-128-instances                        Resource   completed                             
 beta-x86-64-aws-k8s-128-ipv6-conformance-test            Test       passed      384      7007      0      
 beta-x86-64-aws-k8s-128-ipv6-instances                   Resource   completed                             
 beta-x86-64-aws-k8s-128-nvidia-ipv6-conformance-test     Test       passed      384      7007      0      
 beta-x86-64-aws-k8s-128-nvidia-ipv6-instances            Resource   completed                             
 beta-x86-64-aws-k8s-128-nvidia-ipv6-nvidia-smoke-test    Test       passed      11       0         0

metal-k8s-1.28

$ kubectl --kubeconfig br-128-eks-a-cluster.kubeconfig get nodes -o wide
NAME                                 STATUS   ROLES           AGE    VERSION               INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                  KERNEL-VERSION   CONTAINER-RUNTIME
br-128-dfw9b                         Ready    control-plane   124m   v1.28.0-eks-bb809b9   10.80.50.26   <none>        Bottlerocket OS 1.15.0 (metal-k8s-1.28)   6.1.41           containerd://1.6.20+bottlerocket
br-128-md-0-64954d44f5xczblw-59hw7   Ready    <none>          109m   v1.28.0-eks-bb809b9   10.80.50.32   <none>        Bottlerocket OS 1.15.0 (metal-k8s-1.28)   6.1.41           containerd://1.6.20+bottlerocket
br-128-md-0-64954d44f5xczblw-nw8n6   Ready    <none>          109m   v1.28.0-eks-bb809b9   10.80.50.30   <none>        Bottlerocket OS 1.15.0 (metal-k8s-1.28)   6.1.41           containerd://1.6.20+bottlerocket

$ sonobuoy --kubeconfig br-128-eks-a-cluster.kubeconfig status 
   PLUGIN     STATUS   RESULT   COUNT                                PROGRESS
      e2e   complete   passed       1   Passed:380, Failed:  0, Remaining:  0

Sonobuoy has completed. Use `sonobuoy retrieve` to get results.

vmware-k8s-1.28

$ kubectl --kubeconfig br-eksa-128-eks-a-cluster.kubeconfig get nodes
NAME                                      STATUS   ROLES           AGE   VERSION
br-eksa-128-cgn4r                         Ready    control-plane   15m   v1.28.0-eks-bb809b9
br-eksa-128-j7vzq                         Ready    control-plane   14m   v1.28.0-eks-bb809b9
br-eksa-128-md-0-6bb48f8cb5x4h8gw-5z2k2   Ready    <none>          14m   v1.28.0-eks-bb809b9
br-eksa-128-md-0-6bb48f8cb5x4h8gw-z8lvw   Ready    <none>          14m   v1.28.0-eks-bb809b9
...
Server Version: version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.0-eks-bb809b9", GitCommit:"bb809b97e4d30ba0c4f4e7905e559cb20e0ada33", GitTreeState:"archive", BuildDate:"2023-08-15T10:15:49Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"}
$ sonobuoy --kubeconfig br-eksa-128-eks-a-cluster.kubeconfig run --plugin=e2e --mode=certified-conformance --wait
19:13:14       e2e   global   complete   passed   Passed:380, Failed:  0, Remaining:  0
19:13:14 Sonobuoy has completed. Use `sonobuoy retrieve` to get results.

EBS driver testing

$ kubectl --kubeconfig beta-aarch64-aws-k8s-128-ipv6.kubeconfig get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver
NAME                                  READY   STATUS    RESTARTS   AGE
ebs-csi-controller-55d4d546d8-9c6qz   5/5     Running   0          20m
ebs-csi-controller-55d4d546d8-xxzr8   5/5     Running   0          20m
ebs-csi-node-5l6b9                    3/3     Running   0          20m
ebs-csi-node-tkcgc                    3/3     Running   0          20m

ECR Credential Provider testing (thanks @stmcginnis )

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

packages/kubernetes-1.28/dockershim-symlink.conf

variants/Cargo.toml

stmcginnis · 2023-08-07T13:49:14Z

Added aws-k8s-1.28 to a 1.27 EKS cluster:

$ k get nodes
NAME                                           STATUS   ROLES    AGE    VERSION
ip-192-168-4-253.us-east-2.compute.internal    Ready    <none>   4m7s   v1.28.0-rc.0-eks-506ffcd
ip-192-168-41-249.us-east-2.compute.internal   Ready    <none>   4m6s   v1.28.0-rc.0-eks-506ffcd

Attempted to deploy pod with image hosted in private registry of a different account:

$ k get pod
NAME     READY   STATUS         RESTARTS   AGE
brtest   0/1     ErrImagePull   0          4s

Applied credential-provider settings:

apiclient apply <<EOF
[settings.kubernetes.credential-providers.ecr-credential-provider]
enabled = true
cache-duration = "30m"
image-patterns = [
  "*.dkr.ecr.us-east-2.amazonaws.com",
  "*.dkr.ecr.us-west-2.amazonaws.com"
]

[settings.aws]
profile = "ecr"
config = “W3...o=”
EOF

Checked kubelet status and logs and verified image able to be pulled and run:

$ k get pod
NAME     READY   STATUS    RESTARTS   AGE
brtest   1/1     Running   0          23m

Ran sonobuoy run --mode=certified-conformance --wait and verified tests passed:

15:34:34             e2e                                         global   complete   passed   Passed:378, Failed:  0, Remaining:  0
15:34:34    systemd-logs    ip-192-168-4-253.us-east-2.compute.internal   complete   passed

jpculp · 2023-08-09T23:40:43Z

We'll probably want to add seccomp-default to kubelet-config if #3334 merges first.

bcressey · 2023-08-14T17:55:40Z

Will need to rebase on the changes in #3259 also which changes the 76-oci-defaults-capabilities.toml symlink.

This adds the K8s 1.28 package.

stmcginnis

Everything looks good. Not much has changed since my last test, but built and added node to a 1.27 cluster.

NAME                                           STATUS   ROLES    AGE   VERSION
ip-192-168-66-218.us-east-2.compute.internal   Ready    <none>   63s   v1.28.0-eks-bb809b9

# sonobuoy run --mode=quick --wait
...
19:42:46             e2e                                         global   complete   passed   Passed:  1, Failed:  0, Remaining:  0
19:42:46    systemd-logs   ip-192-168-66-218.us-east-2.compute.internal   complete   passed

packages/kubernetes-1.28/Cargo.toml

Starting with K8s 1.28, the supported skew between the node and control plane componenets expands by one minor version from n-2 to n-3. See https://kubernetes.io/blog/2023/08/15/kubernetes-v1-28-release/#changes-to-supported-skew-between-control-plane-and-node-versions

etungsten · 2023-08-29T22:04:29Z

EKS-D is planning to do another release for 1.28.1 sometime tomorrow(?) but there are no kubelet changes and the only change is a windows CVE fix. I'll grab the update if it's available and if this doesn't merge by tomorrow.

etungsten · 2023-08-29T23:08:57Z

If we update to the next EKS-D release, we can drop the 0001-Make-gomaxprocs-install-optional-limit-to-tests.patch patch since it's included in 1.28.1: kubernetes/kubernetes@cce3f6f

bcressey

It looks like the RotateKubeletServerCertificate feature gate was dropped - is this on by default in 1.28?

stmcginnis · 2023-08-30T17:00:01Z

It looks like the RotateKubeletServerCertificate feature gate was dropped - is this on by default in 1.28?

Yeah, this was actually enabled by default since K8s 1.12 (!), but I think it's good to only make the change for the new variant.

bcressey · 2023-08-30T18:15:03Z

Yeah, this was actually enabled by default since K8s 1.12 (!), but I think it's good to only make the change for the new variant.

Is there any output that can be used to confirm that the feature gate is still enabled?

etungsten · 2023-08-30T18:34:04Z

Yeah, this was actually enabled by default since K8s 1.12 (!), but I think it's good to only make the change for the new variant.

Is there any output that can be used to confirm that the feature gate is still enabled?

There apparently is no way to query kubelet at runtime to see what feature gates are enabled/disabled.
We can however check the feature gate defaults in k8s source: https://github.com/kubernetes/kubernetes/blob/v1.28.0/pkg/features/kube_features.go#L1136

stmcginnis reviewed Aug 5, 2023

View reviewed changes

packages/kubernetes-1.28/dockershim-symlink.conf Show resolved Hide resolved

variants/Cargo.toml Show resolved Hide resolved

etungsten force-pushed the k8s-1.28 branch 3 times, most recently from e5eb49b to a229c40 Compare August 9, 2023 22:55

etungsten force-pushed the k8s-1.28 branch 7 times, most recently from ec7979d to 236b3c6 Compare August 21, 2023 17:53

etungsten mentioned this pull request Aug 22, 2023

variants: add aws-iam-authenticator to metal variants #3357

Merged

etungsten added 4 commits August 22, 2023 13:28

packages: add kubernetes-1.28

bf9e579

This adds the K8s 1.28 package.

sources, variants: add aws-k8s-1.28 variant

3021604

sources, variants: add aws-k8s-1.28-nvidia variant

5b3fe5c

sources, variants: add vmware-k8s-1.28 variant

4618eaa

etungsten force-pushed the k8s-1.28 branch from 236b3c6 to 7e018b5 Compare August 22, 2023 20:28

sources, variants: add metal-k8s-1.28 variant

e4b4187

etungsten force-pushed the k8s-1.28 branch from 7e018b5 to e4b4187 Compare August 22, 2023 20:31

etungsten marked this pull request as ready for review August 28, 2023 18:54

etungsten requested review from bcressey and stmcginnis August 28, 2023 18:55

stmcginnis approved these changes Aug 28, 2023

View reviewed changes

packages/kubernetes-1.28/Cargo.toml Show resolved Hide resolved

bcressey reviewed Aug 30, 2023

View reviewed changes

bcressey approved these changes Aug 30, 2023

View reviewed changes

etungsten merged commit 006afce into bottlerocket-os:develop Aug 31, 2023
48 checks passed

etungsten deleted the k8s-1.28 branch August 31, 2023 16:48

gthao313 mentioned this pull request Dec 28, 2023

Add kubernetes-1.29 package and variants #3628

Merged

12 tasks

ginglis13 mentioned this pull request May 2, 2024

Add kubernetes 1.30 sources #3936

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kubernetes-1.28 package and variants #3329

Add kubernetes-1.28 package and variants #3329

etungsten commented Aug 4, 2023 •

edited

stmcginnis commented Aug 7, 2023 •

edited

jpculp commented Aug 9, 2023

bcressey commented Aug 14, 2023

stmcginnis left a comment

etungsten commented Aug 29, 2023 •

edited

etungsten commented Aug 29, 2023

bcressey left a comment

stmcginnis commented Aug 30, 2023

bcressey commented Aug 30, 2023

etungsten commented Aug 30, 2023 •

edited

Add kubernetes-1.28 package and variants #3329

Add kubernetes-1.28 package and variants #3329

Conversation

etungsten commented Aug 4, 2023 • edited

stmcginnis commented Aug 7, 2023 • edited

jpculp commented Aug 9, 2023

bcressey commented Aug 14, 2023

stmcginnis left a comment

Choose a reason for hiding this comment

etungsten commented Aug 29, 2023 • edited

etungsten commented Aug 29, 2023

bcressey left a comment

Choose a reason for hiding this comment

stmcginnis commented Aug 30, 2023

bcressey commented Aug 30, 2023

etungsten commented Aug 30, 2023 • edited

etungsten commented Aug 4, 2023 •

edited

stmcginnis commented Aug 7, 2023 •

edited

etungsten commented Aug 29, 2023 •

edited

etungsten commented Aug 30, 2023 •

edited