kubeadm: add --feature-gates flag for kubeadm upgrade node

[SataQiu]

What type of PR is this?

/kind bug
/kind feature

What this PR does / why we need it:

kubeadm: add --feature-gates flag for kubeadm upgrade node

Which issue(s) this PR fixes:

Ref kubernetes/kubeadm#2346

Special notes for your reviewer:

This flag is needed to upgrade a cluster with UpgradeAddonsBeforeControlPlane feature gate enabled.
Also align with kubeadm upgrade apply/plan.

Does this PR introduce a user-facing change?

None

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: SataQiu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cmd/kubeadm/OWNERS [SataQiu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[neolit123]

/lgtm
/priority important-soon

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 22e3472c1e0ff2731e653dbea3a3758cd6e7ee71

[pacoxu]

/lgtm
/triage accepted

[SataQiu]

Sorry. After a further analysis, I find that kubeadm upgrade apply will update the featureGates field of kube-system/kubeadm-config ConfigMap, and kubeadm upgrade node should use the configuration loaded from cluster rather than the --feature-gates flag, so this flag doesn't seem to be necessary.
I think it's better to revert this, what do you think? @neolit123

[pacoxu]

Sorry. After a further analysis, I find that kubeadm upgrade apply will update the featureGates field of kube-system/kubeadm-config ConfigMap, and kubeadm upgrade node should use the configuration loaded from cluster rather than the --feature-gates flag, so this flag doesn't seem to be necessary.

I think it's better to revert this, what do you think? @neolit123

but this can override that if specified. No harm？

[SataQiu]

but this can override that if specified. No harm？

It shouldn't hurt too much, but we need to consider whether forcing kubeadm upgrade node use the cluster configuration is necessary. It looks like we only support --feature-gates flag in init and upgrade apply/plan, join and reset will only use the cluster configuration. Not sure if this breaks some principles.

IMO, kubeadm upgrade node should respect the featureGates set by kubeadm upgrade apply.
What do you think? @pacoxu @neolit123

[pacoxu]

For NonRoot FG， this can allow user to have a control-plane using not-root and another one using root. For this case, it makes a little sense.

For upgrade-addon-after-control-plane FG, you can choose when to upgrade kube-proxy/coredns. But it is not that necessary IMO.

[neolit123]

i was wandering if this is right or not.
we again face the node specific config problem.
currently feature gates are only cluster wide in clusterconfiguration. this means we should revert.

if there are node specific feature gates we must store them somewhere per node. this is missing design bits in the api and kep work.

if we keep this override per "upgrade node" it becomes weird.

[pacoxu]

currently feature gates are only cluster wide in clusterconfiguration. this means we should revert.

So clusterconfiguration says it is a cluster-wide configuration that should not be overwritten.
Agree.