kubeadm upgrade apply phases

[MarkDeckert]

FEATURE REQUEST

kubeadm init recently included a "--skip-phases" option for skipping a particular phase. However, kubeadm upgrade does not appear to have this option. This specifically causes issues when doing a kubeadm upgrade on a cluster that uses kube-router to replace the functionality of kube-proxy. Since upgrade will always redeploy kube-proxy, it will immediately begin altering iptables and/or ipvs underneath kube-router, and this can cause outages if the daemonset isn't deleted quickly enough.

Additionally (please ignore my ignorance if this is the case, as this is not the main issue) I should mention the documentation is not clear whether it's even possible to skip a particular addon. It appears --skip-phases addons will skip coredns as well, which would be undesireable.

Versions

1.13.0

---

(EDIT by neolit123:)

SUMMARY:

• we want to break the command "kubeadm upgrade apply" into sub-commands (also called "phases")
• kubeadm supports a phase runner that is already used in commands like "kubeadm init"
• this requires breaking the command process is a data structure and a number of phases
• the phases are defined under cmd/phases
• "kubeadm upgrade apply" needs to be adapted to the same approach as "kubeadm init".
• currently the upgrade code requires cleanup before such a change can be applied to it.
• there are some closed attempts of cleaning up the upgrade code:
  [WIP] Upgrade apply phase cleanup kubernetes#87715
  cleaned up of EnforceRequirements kubernetes#87636
  Cleanup apply data kubernetes#87638
• once the cleanup is done creating the phases should be relatively simple.

proposal:
kubernetes/enhancements#1298
https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/2501-kubeadm-phases-to-beta

"apply" phase naming and other ideas:
https://docs.google.com/document/d/1Nicy27rt9jm9tOzZ_MEcQsnmyOFl2ZB6k110aRblv2M/edit?usp=sharing

---

TODO: remove the change added in this PR, once phases are supported:
kubernetes/kubernetes#89593

[neolit123]

kubeadm init recently included a "--skip-phases" option for skipping a particular phase. However, kubeadm upgrade does not appear to have this option. This specifically causes issues when doing a kubeadm upgrade on a cluster that uses kube-router to replace the functionality of kube-proxy.
Since upgrade will always redeploy kube-proxy, it will immediately begin altering iptables and/or ipvs underneath kube-router, and this can cause outages if the daemonset isn't deleted quickly enough.

yes, there is currently now way to skip phases during upgrade.
kube-proxy is considered an "essential addon" and kubeadm assumes that all clusters would use it.
also the daemon set is created in memory and passed to the client directly. it will not be written to disk.

this feature needs discussion.

/assign @timothysc @fabriziopandini

Additionally (please ignore my ignorance if this is the case, as this is not the main issue) I should mention the documentation is not clear whether it's even possible to skip a particular addon. It appears --skip-phases addons will skip coredns as well, which would be undesireable.

you can skip both and then apply the dns addon manually.

[MarkDeckert]

Thanks for taking a look. Also, I think if I had a choice, phase control during an upgrade would be handled through the config file/kubeadm-config configmap so as to be something inherent to the existing cluster, rather than a flag that has to be remembered at upgrade time.

[fabriziopandini]

@MarkDeckert you can skip only the proxy addon by using --skip-phases addons/proxy

First considerations on this topic, that deserve discussion at sig-level and commitment for execution:

• "Officially supported variations" of Kuberenets cluster should be supported via options in the config file, like e.g. to the choice between kube-dns and coredns
• If the user chooses to go with phases and builds a "custom variation", it should get the tools/phases for making also join/upgrade workflows working, but combining/skipping those tools should be under his responsibility, because IMO it is practically impossible to have a reliable CI signal for the almost infinite number of combinations of phases.

[rmb938]

Having the ability to skip kube-proxy re-installing during an upgrade is required. I am using kube-router to manage my services and network policies. When running a kubeadm upgrade once kube-proxy is reinstalled the cluster is unusable until a kube-proxy --cleanup is ran. This leads to downtime of services running on the cluster.

[timothysc]

Let's discuss during the next call.
I do think we should allow folks to skip but the SLO's from kubeadm side are gone then b/c of the permutations.

[bincyber]

There is a hackish workaround for kube-router users:

Create a kube-proxy daemonset and use nodeAffinity to ensure it cannot be scheduled on any node:

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: kube-proxy
 namespace: kube-system
 labels:
   k8s-app: kube-proxy
   purpose: upgrade-placeholder
spec:
 minReadySeconds: 10
 revisionHistoryLimit: 1
 updateStrategy:
   type: RollingUpdate
 selector:
   matchLabels:
     k8s-app: kube-proxy
     purpose: upgrade-placeholder
 template:
   metadata:
     labels:
       k8s-app: kube-proxy
       purpose: upgrade-placeholder
   spec:
     affinity:
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
           - matchExpressions:
             - key: kube-proxy
               operator: In
               values:
               - CantSchedule
     containers:
     - name: kube-proxy
       image: busybox:1.30
       command:
       - "/bin/sleep"
       - "300"
       resources:
         requests:
           cpu: 10m
           memory: 10Mi

During the upgrade of the first Kubernetes master node, it will fail on the final post-upgrade task in which the kube-proxy daemonset is updated with this error:
[upgrade/postupgrade] FATAL post-upgrade error: unable to update daemonset: DaemonSet.apps "kube-proxy" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"k8s-app":"kube-proxy"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

If you are using automation to perform the upgrade, it will need to cope with this error.

Upgrading additional Kubernetes master nodes with kubeadm upgrade node experimental-control-plane requires that the kube-proxy ConfigMap exist to avoid errors:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-proxy
  namespace: kube-system
  labels:
    app: kube-proxy
data:
  config.conf: |-
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    kind: KubeProxyConfiguration
    bindAddress: 0.0.0.0
    clientConnection:
      acceptContentTypes: ""
      burst: 10
      contentType: application/vnd.kubernetes.protobuf
      kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
      qps: 5
    clusterCIDR: 10.0.0.0/16
    configSyncPeriod: 15m0s
    conntrack:
      max: null
      maxPerCore: 32768
      min: 131072
      tcpCloseWaitTimeout: 1h0m0s
      tcpEstablishedTimeout: 24h0m0s
    enableProfiling: false
    healthzBindAddress: 0.0.0.0:10256
    hostnameOverride: ""
    iptables:
      masqueradeAll: false
      masqueradeBit: 14
      minSyncPeriod: 0s
      syncPeriod: 30s
    ipvs:
      excludeCIDRs: null
      minSyncPeriod: 0s
      scheduler: ""
      syncPeriod: 30s
    metricsBindAddress: 127.0.0.1:10249
    mode: ""
    nodePortAddresses: null
    oomScoreAdj: -999
    portRange: ""
    resourceContainer: /kube-proxy
    udpIdleTimeout: 250ms
  kubeconfig.conf: |-
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: "https://URL-of-KUBEMASTERS"
      name: default
    contexts:
    - context:
        cluster: default
        namespace: default
        user: default
      name: default
    current-context: default
    users:
    - name: default
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

[yagonobre]

We should promote upgrade to phase to allow the --skip-phases flag.

[neolit123]

it's a matter of bandwidth for the release cycle and who can work on it.
currently we are fixing some aspects of upgrades related to certs renewal, so it might be a better idea to leave it for the next cycle so that we don't have conflicting PRs.
cc @fabriziopandini for an opinion.

[yagonobre]

I'll work on reset phases this week, probably I'll be able to work on reset phase next week.

[timothysc]

Current plan is to re-evaluate execution of this item in the 1.16 timeframe.

[yvespp]

Maybe another use case for upgrade phases with immutable infrastructure to consider: #1511 (comment)

[fabriziopandini]

Rif #1658, we should ensure that kubeadm-upgrade phases should allow atomic upgrade of etcd

[pacoxu]

/assign

[lvii]

kubeadm upgrade apply unsupport --skip-phases option, when upgraded the installing without kube-proxy

# ks get nodes
NAME              STATUS   ROLES                  AGE   VERSION
deb-k8s-main-01   Ready    control-plane,master   30d   v1.23.6
deb-k8s-node-01   Ready    <none>                 27d   v1.23.5
deb-k8s-node-02   Ready    <none>                 27d   v1.23.5

# kubeadm upgrade apply v1.23.6 --skip-phases=addon/kube-proxy --v=5
unknown flag: --skip-phases
To see the stack trace of this error execute with --v=5 or higher

[MarkusTeufelberger]

The only thing I'm aware that's implemented from this ticket is that you can delete the coredns or kube-proxy configmap(s) and kubeadm will only complain a bit during upgrade, but not re-create them any more since about half a dozen versions. Making it intentional/explicit to kubeadm ("Hey, just skip this, I know what I'm doing!") is still not implemented.

[chenk008]

@pacoxu I was just wondering if you continue to improve this PR kubernetes/kubernetes#108767 ? Is there anything else that I can do to help?

[pacoxu]

@pacoxu I was just wondering if you continue to improve this PR kubernetes/kubernetes#108767 ? Is there anything else that I can do to help?

I may pick this up if I have time in v1.28.

[bh185140]

Is this likely to go in for 1.31? Noticed the related PRs haven't had much activity for a long time. Happy to help contribute if there's any help needed.

[neolit123]

Is this likely to go in for 1.31? Noticed the related PRs haven't had much activity for a long time. Happy to help contribute if there's any help needed.

as per the existing kubeadm maintainer roster bandwidth, unlikely.
i think the existing PR was closed. it requires a lot of work and addition of unit and e2e tests, but if you think you can manage that please open a new PR.