Upgrade to Kubernetes 1.21

[lemeurherve]

Previous upgrade: #2664

• Upgrade kubectl (ref: feat: upgrade kubectl [INFRA-3120] docker-helmfile#40)

• Upgrade DOKS

  • Ensure cluster is re-created by terraform in jenkins-infra/digitalocean
    • Initial Kubernetes 1.21 upgrade: fix: upgrade kubernetes to 1.21 digitalocean#31 (Revert fix: hardcode k8s version until the switch to 1.21 digitalocean#24)
    • hotfix: Fixup upgrade to kubernetes 1.21 as per helpdesk #2866 digitalocean#32
  • Add admin's kubeconfig in the infra.ci's SOPS credentials: (private repo) https://github.com/jenkins-infra/charts-secrets/commit/09587b02aa6adac2078b483e1e48adaa8f92b4bd
  • Ensure that cluster is re-provisionned, including namespace jenkins-agents and the service accounts to be used by ci.jenkins.io: feat(doks) enable DOKS cluster again kubernetes-management#2275
  • Add the credential of the service account jenkins-agents in ci.jenkins.io (manual step)
  • Update ci.jenkins.io configuration as code to specify the cluster doks again (as found by @smerle33 : reverting jenkins-infra/jenkins-infra@1b7efb2)

• Upgrade EKS (reprioritized due to https://unit42.paloaltonetworks.com/aws-log4shell-hot-patch-vulnerabilities/)

  • Disable cluster in ci.jenkins.io's config as code (+ increase digitalocean capacity temporarly)
  • Upgrade cluster through Terraform (control plane + node groups)
  • Upgrade add-ons (weird: their version are not upgraded by the EKS module. Need to be specified as terraform variable + updatecli-zed from the tables at https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html, https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html and https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html)
  • Ensure that upgraded cluster is still ok (check kubernetes-management)
  • Enable it again in ci.jenkins.io config as code

• Upgrade AKS

  • Check previous AKS upgrades and create one for 1.21
  • Open datetime announcement
  • Upgrade temp-privatek8s first. ⚠️ it will put infra.ci out of service so might need manual steps to re-boostrap the kubernetes-management job
  • Upgrade prodpublick8s
  • Close announcement

[dduportal]

Note: this task blocks #2884

[lemeurherve]

Upgrade of Kubernetes to v1.21 complete