Don't use strategic merge patch on Node.Status.Addresses

[danwinship]

What type of PR is this?
/kind bug

What this PR does / why we need it:
On a node with multiple IP addresses of the same type (eg, multiple InternalIP addresses), when the set of active addresses changed, kubelet would sometimes randomly reorder them. This could cause the node to suddenly declare a different primary node IP, causing various breakage.

This is because the kubelet sync code uses strategic merge patch to update Node.Status, but Node.Status.Addresses is incorrectly annotated with a patchStrategy that only works when there is only a single address of the same type.

Even though the annotation is wrong for the API, fixing it is an API break, particularly since changing it would result in even worse behavior when the clients and servers were different versions. (The client would generate a strategic patch according to one interpretation which the server would then apply according to a different interpretation.)

So this fixes PatchNodeStatus() by having it edit the automatically-generated strategic merge patch to force it to use a "replace" strategy on Status.Addressess.

Which issue(s) this PR fixes:
none, but the newly-added unit test demonstrates the bug. (It fails against the old code.)
(Downstream bug is https://bugzilla.redhat.com/show_bug.cgi?id=1696628 but it's mostly private)

Does this PR introduce a user-facing change?:

Kubelet should now more reliably report the same primary node IP even if the set of node IPs reported by the CloudProvider changes.

/priority important-soon

[RobertKrawitz]

/cc @sttts @liggitt

[andrewsykim]

I think I agree with Dan, though - we have a case where a bug exists. If that bug exists in 2 components, they both need to be fixed.

I agree with this too, but would prefer to take a pragmatic approach for the out-of-tree case. Maybe we can isolate a commit that fixes the patch strategy for only the CCM case and backport that to previous versions? Forcing every out-of-tree provider to upgrade to v1.16 Kubernetes will not be easy because of the way things are vendored now (vendoring k8s.io/kubernetes is required).

[andrewsykim]

@kfox1111 for the vSphere case, it would only apply if there is more than 1 network interface with the matching prefix.

[thockin]

Does CCM have this equivalent loop in a new controller? Can we just make a different PR for that?

…

On Tue, Jul 2, 2019 at 4:49 PM Andrew Sy Kim ***@***.***> wrote: I think I agree with Dan, though - we have a case where a bug exists. If that bug exists in 2 components, they both need to be fixed. I agree with this too, but would prefer to take a pragmatic approach for the out-of-tree case. Maybe we can isolate a commit that fixes the patch strategy for only the CCM case and backport that to previous versions? Forcing every out-of-tree provider to upgrade to v1.16 Kubernetes will not be easy because of the way things are vendored now (vendoring k8s.io/kubernetes is required). — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#79391?email_source=notifications&email_token=ABKWAVCCZ6CYDFA2XEYS2MTP5PSR7A5CNFSM4H3MURFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZC3KCA#issuecomment-507884808>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABKWAVH5NEQWSW6LEZ45KWTP5PSR7ANCNFSM4H3MURFA> .

[thockin]

Thanks!

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danwinship, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• api/OWNERS [thockin]
• pkg/OWNERS [thockin]
• staging/src/k8s.io/api/OWNERS [thockin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andrewsykim]

Does CCM have this equivalent loop in a new controller? Can we just make a different PR for that?

It shares nodeutil.PatchNodeStatus function with the kubelet, but yeah good point we can duplicate the function locally worse-case scenario. Thanks!

[kfox1111]

@kfox1111 for the vSphere case, it would only apply if there is more than 1 network interface with the matching prefix.

Yeah. I see that on my cluster for sure. I'm just trying to understand how this bug will effect me until its fixed. Is restarting a kubelet (during a crash, say) enough to trigger node update failures due to interface reordering?

Just how frequent it happens and what triggers it, dictates how to respond to it. If its only when an interface is removed or added, this won't effect any of my production systems as I don't do that. If its on kubelet restart, then its fairly serious and I need to tread very lightly as I am not in full control of that... I'd maybe have to even patch it out of band until upstreamed in that case to be safe.

[danwinship]

Restarting won't trigger the bug

[kfox1111]

Ok. Thank you.

[cheftako]

On Tue, Jul 2, 2019 at 8:51 AM Tim Hockin ***@***.***> wrote: When CCM has been rolled out but some of the nodes are not yet updated?

For the cloud node controller to set the address on the node, the node must have the cloud taint. The node is given the cloud taint when it is created by the kubelet. The kubelet will only create with the cloud taint when the kubelet is started with the external cloud provider. So I believe we are ok.

[NeilW]

I'll add the fault I've just seen in here in case it adds any light to the discussion.

The Brightbox Cloud Controller throws up the following error occasionally in the Node Controller when we have multiple external IPs mapped to a node

E0821 10:47:34.158656       1 node_controller.go:188] Error patching node with cloud ip addresses = [failed to patch status "{\"status\":{\"$setElementOrder/addresses\":[{\"type\":\"Hostname\"},{\"type\":\"InternalDNS\"},{\"type\":\"InternalIP\"},{\"type\":\"InternalIP\"},{\"type\":\"ExternalIP\"},{\"type\":\"ExternalDNS\"},{\"type\":\"ExternalIP\"},{\"type\":\"ExternalDNS\"}],\"addresses\":[{\"address\":\"109.107.37.240\",\"type\":\"ExternalIP\"},{\"address\":\"109.107.35.7\",\"type\":\"ExternalIP\"},{\"address\":\"cip-8fp4v.gb1.brightbox.com\",\"type\":\"ExternalDNS\"},{\"address\":\"cip-2mse0.gb1.brightbox.com\",\"type\":\"ExternalDNS\"}]}}" for node "srv-tryop": The order in patch list:
[map[address:109.107.37.240 type:ExternalIP] map[address:109.107.35.7 type:ExternalIP] map[address:cip-8fp4v.gb1.brightbox.com type:ExternalDNS] map[address:cip-2mse0.gb1.brightbox.com type:ExternalDNS]]
 doesn't match $setElementOrder list:
[map[type:Hostname] map[type:InternalDNS] map[type:InternalIP] map[type:InternalIP] map[type:ExternalIP] map[type:ExternalDNS] map[type:ExternalIP] map[type:ExternalDNS]]
]