Upgrade to Kubernetes 1.23

[lemeurherve]

Previous upgrade: #2930

• Upgrade kubectl within docker-helmfile

  • First upgrade the updatecli manifest : https://github.com/jenkins-infra/docker-helmfile/blob/d4932dabcc997ead5c93ec5858c6b5554fd5fc0b/updatecli/updatecli.d/kubectl.yml#L29
  • Then apply the updatecli PR that is produced by this change : [updatecli] [kubectl] Update version (to 1.23.x) docker-helmfile#175
  • Then Apply the Kubernetes-Management PR that will use this new version : Bump jenkins-infra helmfile docker image to 2.5.0 kubernetes-management#3015

• Upgrade DOKS (doks and doks-public)

  • Disable kubernetes management (disable job on infra.ci.jenkins.io)
  • Bump Kubernetes version in digitalocean /variables.tf
  • Ensure cluster is re-created by terraform in jenkins-infra/digitalocean
  • Re-Enable kubernetes management (and update kubeconfigs in the infra.ci's SOPS credentials (private repo), if kubernetes management cannot connect)
  • Add the credential of the service account jenkins-agents in ci.jenkins.io if needed (manual step) doks only not doks-public

• Upgrade EKS

  • Open datetime announcement on status.jenkins.io
  • Disable cluster in ci.jenkins.io configuration as code
  • Upgrade cluster through Terraform (control plane + node groups)
  • Upgrade add-ons if needed
    • If done via the UI, select the checkbox "replace existing configuration"
    • For vpc-cni select the inheritance from node.
  • Ensure the upgraded cluster is still OK (check kubernetes-management) ⚠️ ensure that the security group eks-cluster-sg-<cluster name>-<random ID> that is managed directly by EKS doesn't contain the tag kubernetes.io/cluster/ (the tag is provided by our terraform)
  • Enable it again in ci.jenkins.io config as code

• Upgrade AKS

  • Check previous AKS upgrades and create one for 1.23
  • Open datetime announcement on status.jenkins.io
  • Upgrade temp-privatek8s first. ⚠️ it will put infra.ci out of service so might need manual steps to re-boostrap the kubernetes-management job
  • Upgrade prodpublick8s
  • Close announcement

[github-actions]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 92% Upgrade to Kubernetes 1.21 #2866
2. 77% [INFRA-3118] Upgrade to Kubernetes 1.20 #2664

[lemeurherve]

According to DigitalOcean supported Kubernetes release page, the version v1.22 will be "end of support" on the 28th of October, and automatic upgrades will begin 27th of November.

And for Azure, it will be EOL when the version 1.25 will be GA, planned for November.

For AWS, Amazon EKS guarantees support for at least four production-ready versions of Kubernetes at any given time.

To keep in mind for the next milestones.

[smerle33]

Here is a link to hackdays notes for the changelog from 1.22 : https://hackmd.io/z0sMiMMpReenvVvheoZCMA

A link to the dedicated changelog for DigitalOcean : https://docs.digitalocean.com/products/kubernetes/details/changelog/#1-23-x

[dduportal]

• https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md#changelog-since-v1220-1

[dduportal]

Note related to AKS clusters: as per https://github.com/Azure/AKS/blob/master/CHANGELOG.md#release-2022-10-17, there are 2 changes we have to look for, after the upgrade, about our Windows node pools:

• Switching to containerd
• Ensuring that the labels/taints/tolerations are not deprecated

[smerle33]

AKS upgrade checklist : https://github.com/jenkins-infra/documentation/blob/main/maintenance/kubernetes/aks-1.23.md

[lemeurherve]

Forgot the most important, the release logo: