Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

☂️-Issue for "Support for Kubernetes v1.23” #5102

Closed
20 tasks done
rfranzke opened this issue Dec 1, 2021 · 6 comments
Closed
20 tasks done

☂️-Issue for "Support for Kubernetes v1.23” #5102

rfranzke opened this issue Dec 1, 2021 · 6 comments
Assignees
@BeckerMax @rfranzke
Labels
area/open-source Open Source (community, enablement, contributions, conferences, CNCF, etc.) related kind/enhancement Enhancement, improvement, extension priority/3 Priority (lower number equals higher priority) topology/garden Affects Garden clusters topology/seed Affects Seed clusters topology/shoot Affects Shoot clusters

Comments

@rfranzke
Copy link
Member

rfranzke commented Dec 1, 2021 •
edited

/area open-source
/kind enhancement
/priority 3
/topology garden seed shoot

Kubernetes v1.23 is about to be released (CW49), find the changelog here: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md

Work can be started and pushed to the kubernetes-v1.23 branch. There should be similar branches in the provider extension repositories in case the work has been started already.

gardener/hyperkube

  • Build new images for v1.23

gardener/gardener

gardener/gardener-extension-provider-alicloud

gardener/gardener-extension-provider-aws

gardener/gardener-extension-provider-azure

gardener/gardener-extension-provider-gcp

gardener/gardener-extension-provider-openstack

gardener/gardener-extension-provider-equinix-metal

gardener/gardener-extension-provider-vsphere
@rfranzke rfranzke added the kind/enhancement Enhancement, improvement, extension label Dec 1, 2021
@gardener-robot gardener-robot added area/open-source Open Source (community, enablement, contributions, conferences, CNCF, etc.) related priority/3 Priority (lower number equals higher priority) topology/garden Affects Garden clusters topology/seed Affects Seed clusters topology/shoot Affects Shoot clusters labels Dec 1, 2021
@ialidzhikov
Copy link
Member

ialidzhikov commented Dec 7, 2021 •
edited

One minor comment wrt:

  • ❌ Revendor gardener/cloud-provider-azure with v1.23.0 sources

There was the agreement to start using the out-of-tree CCM for Azure starting with K8s v1.23 - ref gardener/gardener-extension-provider-azure#376 (comment). In such case, I guess we can skip the update of gardener/cloud-provider-azure with v1.23.0 sources.

@vpnachev
Copy link
Member

vpnachev commented Jan 6, 2022

What do you think about changing the cgroup driver from cgroupfs to systemd for clusters on k8s >= 1.23?
I am asking because:

  • systemd is the recommended cgroup driver, ref

    We have seen cases in the field where nodes that are configured to use cgroupfs for kubelet and docker and systemd for the rest can become unstable under resource pressure. Changing the settings such that docker and kubelet use systems as a cgroup-driver stabilized the systems.

  • cgroupfs does not support cgroup v2 which the OS vendors start to consider to be the default

@rfranzke
Copy link
Member Author

rfranzke commented Jan 7, 2022

I think your proposal sounds reasonable @vpnachev

@rfranzke
Copy link
Member Author

/assign @BeckerMax @rfranzke

rfranzke added a commit that referenced this issue Jan 12, 2022
@rfranzke
Use systemd as cgroup driver for shoots >= 1.23
53d1952 
Ref #5102 (comment)
rfranzke added a commit that referenced this issue Jan 12, 2022
@rfranzke
Use systemd as cgroup driver for shoots >= 1.23
ef5763b 
Ref #5102 (comment)
@rfranzke rfranzke mentioned this issue Jan 12, 2022
Merged
rfranzke added a commit that referenced this issue Jan 12, 2022
@rfranzke
Use systemd as cgroup driver for shoots >= 1.23
b16cc48 
Ref #5102 (comment)
rfranzke added a commit that referenced this issue Jan 17, 2022
@rfranzke
Support for Kubernetes v1.23 (#5255)
5f0ae8d 
* Extend docs for 1.23 and allow Kubernetes client creation

* Maintain Kubernetes feature gates

$ ./hack/compare-k8s-feature-gates.sh 1.22 1.23
Feature gates added in 1.23 compared to 1.22:
CPUManagerPolicyAlphaOptions
CPUManagerPolicyBetaOptions
CSIMigrationPortworx
CustomResourceValidationExpressions
DisableKubeletCloudCredentialProviders
GRPCContainerProbe
GracefulNodeShutdownBasedOnPodPriority
HonorPVReclaimPolicy
IdentifyPodOS
InTreePluginPortworxUnregister
InTreePluginRBDUnregister
JobMutableNodeSchedulingDirectives
JobReadyPods
MigrationRBD
OpenAPIEnums
OpenAPIV3
PodAndContainerStatsFromCRI
RecoverVolumeExpansionFailure
ServerSideFieldValidation
StatefulSetAutoDeletePVC

Feature gates removed in 1.23 compared to 1.22:
AllowInsecureBackendProxy
BoundServiceAccountTokenVolume
CronJobControllerV2
NodeLease
ServiceAccountIssuerDiscovery
StartupProbe
SupportNodePidsLimit
SupportPodPidsLimit
Sysctls

* Adapt TLS cipher suites for Kubernetes v1.23

according to #4823

* Use `systemd` as cgroup driver for shoots >= 1.23

Ref #5102 (comment)

* Adapt to deprecated seccomp annotations

* Use v1beta3 version for kube-scheduler config

* Do no longer use ineffective flags for shoots >= 1.23

* Switch to autoscaling/v2 or add TODOs

* Bump `kindest/node` image to `v1.23.1`

* Use 1.23 for local shoot and local e2e test

* Revert "Bump `kindest/node` image to `v1.23.1`"

This reverts commit 7c31fa8.

* Address PR review feedback

* Address PR review feedback
This was referenced Jan 20, 2022
Closed
Merged
Merged
Merged
Merged
@voelzmo
Copy link
Member

voelzmo commented Jan 25, 2022

Note: 1.23.2 has a regression regarding x-kubernets-preserve-unknown-fields when being used on arrays: kubernetes/kubernetes#107688 This is fixed in >= 1.23.3

This was referenced Jan 25, 2022
Merged
Merged
Merged
Merged
Closed
@rfranzke
Copy link
Member Author

/close since all related PRs have been merged

@rfranzke rfranzke mentioned this issue Feb 17, 2022
Merged
krgostev pushed a commit to krgostev/gardener that referenced this issue Apr 21, 2022
@rfranzke
Support for Kubernetes v1.23 (gardener#5255)
644f1c2 
* Extend docs for 1.23 and allow Kubernetes client creation

* Maintain Kubernetes feature gates

$ ./hack/compare-k8s-feature-gates.sh 1.22 1.23
Feature gates added in 1.23 compared to 1.22:
CPUManagerPolicyAlphaOptions
CPUManagerPolicyBetaOptions
CSIMigrationPortworx
CustomResourceValidationExpressions
DisableKubeletCloudCredentialProviders
GRPCContainerProbe
GracefulNodeShutdownBasedOnPodPriority
HonorPVReclaimPolicy
IdentifyPodOS
InTreePluginPortworxUnregister
InTreePluginRBDUnregister
JobMutableNodeSchedulingDirectives
JobReadyPods
MigrationRBD
OpenAPIEnums
OpenAPIV3
PodAndContainerStatsFromCRI
RecoverVolumeExpansionFailure
ServerSideFieldValidation
StatefulSetAutoDeletePVC

Feature gates removed in 1.23 compared to 1.22:
AllowInsecureBackendProxy
BoundServiceAccountTokenVolume
CronJobControllerV2
NodeLease
ServiceAccountIssuerDiscovery
StartupProbe
SupportNodePidsLimit
SupportPodPidsLimit
Sysctls

* Adapt TLS cipher suites for Kubernetes v1.23

according to gardener#4823

* Use `systemd` as cgroup driver for shoots >= 1.23

Ref gardener#5102 (comment)

* Adapt to deprecated seccomp annotations

* Use v1beta3 version for kube-scheduler config

* Do no longer use ineffective flags for shoots >= 1.23

* Switch to autoscaling/v2 or add TODOs

* Bump `kindest/node` image to `v1.23.1`

* Use 1.23 for local shoot and local e2e test

* Revert "Bump `kindest/node` image to `v1.23.1`"

This reverts commit 7c31fa8.

* Address PR review feedback

* Address PR review feedback
krgostev pushed a commit to krgostev/gardener that referenced this issue Jul 5, 2022
@rfranzke
Support for Kubernetes v1.23 (gardener#5255)
e33a34b 
* Extend docs for 1.23 and allow Kubernetes client creation

* Maintain Kubernetes feature gates

$ ./hack/compare-k8s-feature-gates.sh 1.22 1.23
Feature gates added in 1.23 compared to 1.22:
CPUManagerPolicyAlphaOptions
CPUManagerPolicyBetaOptions
CSIMigrationPortworx
CustomResourceValidationExpressions
DisableKubeletCloudCredentialProviders
GRPCContainerProbe
GracefulNodeShutdownBasedOnPodPriority
HonorPVReclaimPolicy
IdentifyPodOS
InTreePluginPortworxUnregister
InTreePluginRBDUnregister
JobMutableNodeSchedulingDirectives
JobReadyPods
MigrationRBD
OpenAPIEnums
OpenAPIV3
PodAndContainerStatsFromCRI
RecoverVolumeExpansionFailure
ServerSideFieldValidation
StatefulSetAutoDeletePVC

Feature gates removed in 1.23 compared to 1.22:
AllowInsecureBackendProxy
BoundServiceAccountTokenVolume
CronJobControllerV2
NodeLease
ServiceAccountIssuerDiscovery
StartupProbe
SupportNodePidsLimit
SupportPodPidsLimit
Sysctls

* Adapt TLS cipher suites for Kubernetes v1.23

according to gardener#4823

* Use `systemd` as cgroup driver for shoots >= 1.23

Ref gardener#5102 (comment)

* Adapt to deprecated seccomp annotations

* Use v1beta3 version for kube-scheduler config

* Do no longer use ineffective flags for shoots >= 1.23

* Switch to autoscaling/v2 or add TODOs

* Bump `kindest/node` image to `v1.23.1`

* Use 1.23 for local shoot and local e2e test

* Revert "Bump `kindest/node` image to `v1.23.1`"

This reverts commit 7c31fa8.

* Address PR review feedback

* Address PR review feedback
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BeckerMax BeckerMax

@rfranzke rfranzke

Labels
area/open-source Open Source (community, enablement, contributions, conferences, CNCF, etc.) related kind/enhancement Enhancement, improvement, extension priority/3 Priority (lower number equals higher priority) topology/garden Affects Garden clusters topology/seed Affects Seed clusters topology/shoot Affects Shoot clusters
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@voelzmo @vpnachev @BeckerMax @ialidzhikov @rfranzke @gardener-robot