Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Kubernetes v1.26 #7275

Merged
merged 17 commits into from Jan 16, 2023
Merged

Support for Kubernetes v1.26 #7275

merged 17 commits into from Jan 16, 2023

Conversation

ialidzhikov
Copy link
Member

@ialidzhikov ialidzhikov commented Dec 30, 2022

How to categorize this PR?

/area open-source usability
/kind enhancement
/topology garden seed shoot

What this PR does / why we need it:
Support for Kubernetes v1.26

Which issue(s) this PR fixes:
Part of #6773

Special notes for your reviewer:

  • I didn't vendor new k8s.io/* Golang sources, this item is tracked separately in Upgrade k8s.io/* to v0.26, sigs.k8s.io/controller-runtime to v0.14 #7230.
  • ⚠️ This PR only contains the Gardener part for supporting 1.26 - there will/must be follow-up PRs for each extension repository where individual support gets added.
  • Code changes have been tested in the following scenarios:
    • Create/Delete a new Shoot cluster with 1.26.0
    • Upgrade an existing Shoot cluster to 1.26.0
    • Register a 1.26.0 as a Seed cluster (with HVPA feature gate disabled)

Release note:

Gardener can now support Shoot clusters (and Seed clusters with `HVPA` feature gate disabled) with Kubernetes version 1.26. In order to allow creation/update of 1.26 clusters you will have to update the version of your provider extension(s) to a version that supports 1.26 as well. Please consult the respective releases and notes in the provider extension's repository.
Gardener can now support Shoot clusters (and Seed clusters with `HVPA` feature gate disabled) with Kubernetes version 1.26. Extension developers have to prepare individual extensions as well to work with 1.26.
The `spec.kubernetes.kubeControllerManager.podEvictionTimeout` field in the `Shoot` API is deprecated in favor of the newly introduced fields `spec.kubernetes.kubeAPIServer.default{NotReady,Unreachable}TolerationSeconds`. The `podEvictionTimeout` field does not have effect when the taint-based eviction is enabled. The taint-based eviction is beta (enabled by default) since Kubernetes 1.13 and GA since Kubernetes 1.18. The newly introduced `defaultNotReadyTolerationSeconds` and `defaultUnreachableTolerationSeconds` fields should be used instead.
The `gardener-apiserver` defaults the `.spec.kubernetes.enableStaticTokenKubeconfig` field to `false` for `Shoot`s with Kubernetes version >= `1.26`. For `Shoot`s with Kubernetes version < `1.26`, the `gardener-apiserver` continues to default the field to `true`. The static token kubeconfig is deprecated. The [`shoots/adminkubeconfig` subresource](https://github.com/gardener/gardener/blob/master/docs/usage/shoot_access.md#shootsadminkubeconfig-subresource) should be used instead.

@gardener-prow
Copy link
Contributor

gardener-prow bot commented Dec 30, 2022

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@gardener-prow gardener-prow bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. area/open-source Open Source (community, enablement, contributions, conferences, CNCF, etc.) related area/usability Usability related kind/enhancement Enhancement, improvement, extension cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. labels Dec 30, 2022
@gardener-prow gardener-prow bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Dec 30, 2022
@ialidzhikov
Copy link
Member Author

/test all

@ialidzhikov ialidzhikov force-pushed the kubernetes-v1.26 branch 2 times, most recently from 8110869 to a9c5e99 Compare January 4, 2023 14:22
@gardener-prow gardener-prow bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 9, 2023
Copy link
Contributor

@shafeeqes shafeeqes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the well structured PR.
Had a combined look with @ary1992 and left some comments and questions.

README.md Outdated Show resolved Hide resolved
pkg/apis/core/v1alpha1/defaults_test.go Outdated Show resolved Hide resolved
pkg/apis/core/v1alpha1/defaults_test.go Outdated Show resolved Hide resolved
pkg/apis/core/v1beta1/defaults_test.go Outdated Show resolved Hide resolved
@gardener-prow gardener-prow bot added cla: no Indicates the PR's author has not signed the cla-assistant.io CLA. cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. cla: no Indicates the PR's author has not signed the cla-assistant.io CLA. labels Jan 13, 2023
@gardener-prow gardener-prow bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 13, 2023
@ialidzhikov
Copy link
Member Author

/test all

@ialidzhikov ialidzhikov marked this pull request as ready for review January 13, 2023 11:54
@ialidzhikov
Copy link
Member Author

/test pull-gardener-e2e-kind-migration

@rfranzke
Copy link
Member

@ialidzhikov Sounds reasonable! We don't have to wait for the next release, though. You can change https://github.com/gardener/gardener/blob/master/Makefile#L36 after this PR got merged. :)

$ ./hack/compare-k8s-feature-gates.sh 1.25 1.26
Feature gates added in 1.26 compared to 1.25:
APISelfSubjectReview
AggregatedDiscoveryEndpoint
ConsistentHTTPGetHandlers
CrossNamespaceVolumeDataSource
DynamicResourceAllocation
EventedPLEG
LegacyServiceAccountTokenTracking
MinimizeIPTablesRestore
PDBUnhealthyPodEvictionPolicy
PodSchedulingReadiness
StatefulSetStartOrdinal
TopologyManagerPolicyAlphaOptions
TopologyManagerPolicyBetaOptions
TopologyManagerPolicyOptions
ValidatingAdmissionPolicy
WindowsHostNetwork

Feature gates removed in 1.26 compared to 1.25:
CSIMigrationOpenStack
CSRDuration
DefaultPodTopologySpread
DynamicKubeletConfig
IndexedJob
NonPreemptingPriority
PodAffinityNamespaceSelector
PodOverhead
PreferNominatedNode
ServiceLBNodePortControl
ServiceLoadBalancerClass
SuspendJob

Feature gates locked to default in 1.26 compared to 1.25:
CPUManager
CSIMigrationvSphere
DelegateFSGroupToCSIDriver
DevicePlugins
DryRun
EndpointSliceTerminatingCondition
JobTrackingWithFinalizers
KubeletCredentialProviders
MixedProtocolLBService
ServerSideApply
ServiceIPStaticSubrange
ServiceInternalTrafficPolicy
WindowsHostProcessContainers
$ ./hack/compare-k8s-admission-plugins.sh 1.25 1.26
Admission plugins added in 1.26 compared to 1.25:
ValidatingAdmissionPolicy

Admission plugins removed in 1.26 compared to 1.25:
…troller-manager`

$ ./hack/compare-k8s-controllers.sh 1.25 1.26

kube-controller-manager controllers added in 1.26 compared to 1.25:
resource-claim-controller

kube-controller-manager controllers removed in 1.26 compared to 1.25:
… kube-apiserver fields

The kube-controller-manager flag `--pod-eviction-timeout` is deprecated in favor of the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `--pod-eviction-timeout` flag does not have effect when the taint besed eviction is enabled. The taint  based eviction is beta (enabled by default) since Kubernetes 1.13 and GA since Kubernetes 1.18. For more details, see kubernetes/kubernetes#74651.

This commit allows configuring the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `podEvictionTimeout` field is deprecated in favor of the newly introduced fields. gardener-apiserver no longer defaults the `podEvictionTimeout` field. gardener-apiserver also returns a warning when the `podEvictionTimeout` field is set.
…server_storage_db_total_size_in_bytes`

The metric `etcd_db_total_size_in_bytes` is renamed to `apiserver_storage_db_total_size_in_bytes`. Ref kubernetes/kubernetes#113310.
Test runs of the integration test that uses this template prints the following warning about the issue in the template:
```
{"level":"info","ts":"2022-12-28T19:36:29.043+0200","logger":"KubeAPIWarningLogger","msg":"unknown field \"spec.template.spec.containers[0].nodeName\""}
```
After the removal of support for Kubernetes < 1.20 Shoot clusters (ref #6987), the kubeconfig Secret no longer has the `token` field. Basic auth cannot be enabled for K8s 1.19+ clusters, hence the kubeconfig Secret cannot contain the `username`/`password` fields anymore.
…rsion >= 1.26

This commit also adapts most of the testmachinery integration tests to use the `shoots/adminkubeconfig` subresource instead of the static kubeconfig.
The Shoot creation intergration is still using the static kubeconfig and it is downloading it to `$TM_KUBECONFIG_PATH/shoot.config`. This commit sets `enableStaticTokenKubeconfig=true` until we figure out which tests/components are using this downloaded kubeconfig.
The constraint `ConstraintK8sLess126` is currently not used by gardener/gardener but it is introduced for usage from the extensions.
@gardener-prow
Copy link
Contributor

gardener-prow bot commented Jan 13, 2023

@ialidzhikov: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-gardener-apidiff 89e4f98 link false /test pull-gardener-apidiff

Full PR test history. Your PR dashboard. Command help for this repository.
Please help us cut down on flakes by linking this test failure to an open flake report or filing a new flake report if you can't find an existing one. Also see our testing guideline for how to avoid and hunt flakes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Copy link
Contributor

@shafeeqes shafeeqes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@gardener-prow gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Jan 16, 2023
@gardener-prow
Copy link
Contributor

gardener-prow bot commented Jan 16, 2023

LGTM label has been added.

Git tree hash: f07a5e5e448e2d838373d4a2ec866a6ae0fc50f5

@rfranzke
Copy link
Member

/approve

@gardener-prow
Copy link
Contributor

gardener-prow bot commented Jan 16, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rfranzke

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@gardener-prow gardener-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 16, 2023
@gardener-prow gardener-prow bot merged commit f263358 into master Jan 16, 2023
@gardener-prow gardener-prow bot deleted the kubernetes-v1.26 branch January 16, 2023 06:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/open-source Open Source (community, enablement, contributions, conferences, CNCF, etc.) related area/usability Usability related cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. kind/enhancement Enhancement, improvement, extension lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants