Cloud controller is unable to patch node with IP addresses from cloud provider #266

ivanfilippov · 2019-10-11T06:02:13Z

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:
Cloud controller is unable to patch node with IP addresses from cloud provider on a multi-NIC VM.

I1011 05:22:49.646435       1 instances.go:158] instances.InstanceExistsByProviderID() CACHED with 42348b4e-4ba1-e452-e946-6d0e9a323160
I1011 05:22:49.646457       1 instances.go:78] instances.NodeAddressesByProviderID() CACHED with 42348b4e-4ba1-e452-e946-6d0e9a323160
E1011 05:22:49.656899       1 node_controller.go:188] Error patching node with cloud ip addresses = [failed to patch status "{\"status\":{\"$setElementOrder/addresses\":[{\"type\":\"ExternalIP\"},{\"type\":\"InternalIP\"},{\"type\":\"Hostname\"},{\"type\":\"ExternalIP\"},{\"type\":\"InternalIP\"},{\"type\":\"ExternalIP\"},{\"type\":\"InternalIP\"}],\"addresses\":[{\"address\":\"172.16.202.44\",\"type\":\"ExternalIP\"},{\"address\":\"172.16.201.44\",\"type\":\"ExternalIP\"},{\"address\":\"172.16.45.44\",\"type\":\"ExternalIP\"},{\"address\":\"172.16.202.44\",\"type\":\"InternalIP\"},{\"address\":\"172.16.201.44\",\"type\":\"InternalIP\"},{\"address\":\"172.16.45.44\",\"type\":\"InternalIP\"}]}}" for node "k8s-prod5-worker-2": The order in patch list:
[map[address:172.16.202.44 type:ExternalIP] map[address:172.16.201.44 type:ExternalIP] map[address:172.16.45.44 type:ExternalIP] map[address:172.16.202.44 type:InternalIP] map[address:172.16.201.44 type:InternalIP] map[address:172.16.45.44 type:InternalIP]]
 doesn't match $setElementOrder list:
[map[type:ExternalIP] map[type:InternalIP] map[type:Hostname] map[type:ExternalIP] map[type:InternalIP] map[type:ExternalIP] map[type:InternalIP]]
]

What you expected to happen:
Nodes are patched with IP addresses from cloud provider.

How to reproduce it (as minimally and precisely as possible):
Build a worker node with 3 NICs and attempt to deploy the cloud controller

Anything else we need to know?:
This was encountered in the openstack provider as well and was fixed there (kubernetes/cloud-provider-openstack#407).

Environment:

vsphere-cloud-controller-manager version: latest from github
OS (e.g. from /etc/os-release): RancherOS v1.5.4
Kernel (e.g. uname -a): 4.14.138-rancher
Install tools: cluster installed via RKE
Others:

The text was updated successfully, but these errors were encountered:

frapposelli · 2019-10-11T08:11:06Z

/assign @andrewsykim @dvonthenen

k8s-ci-robot · 2019-10-11T08:11:08Z

@frapposelli: GitHub didn't allow me to assign the following users: vonthenend.

Note that only kubernetes members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @andrewsykim @vonthenend

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dvonthenen · 2019-10-11T14:28:59Z

/assign

andrewsykim · 2019-10-15T20:23:11Z

This is the upstream fix for this bug kubernetes/kubernetes#79391, should be fixed when we pull that in.

dvonthenen · 2019-10-21T14:21:09Z

It looks like that was merged and possibly made it into 1.16. Any confirmation on that?

andrewsykim · 2019-10-21T15:33:35Z

yeah its in v1.16

dvonthenen · 2019-10-21T15:40:07Z

yeah its in v1.16

Since I need to rebuild my dev env, will be trying out 1.16 today or tomorrow. Does the CPI need to pull in 1.16 for this to work? or just on the k8s side?

andrewsykim · 2019-10-21T15:48:47Z

It seems like just k8s.io/kubernetes needs to be updated

dvonthenen · 2019-10-24T16:04:38Z

Looks like it's only on the k8s side. I tested with the latest CPI binary without updating the k8s.io/kubernetes that the CPI is built on and it works just fine with 1.16.2. Closing this out.

/close

k8s-ci-robot · 2019-10-24T16:04:39Z

@dvonthenen: Closing this issue.

In response to this:

Looks like it's only on the k8s side. I tested with the latest CPI binary without updating the k8s.io/kubernetes that the CPI is built on and it works just fine with 1.16.2. Closing this out.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

andrewsykim · 2019-10-24T16:08:51Z

@dvonthenen you used multiple vNICs in your tests? At first glance it sounds like we do need to update k8s.io/kubernetes in the controller manager since PatchNodeStatus is still called by the cloud node controllerl

dvonthenen · 2019-10-24T16:10:45Z

ahhhh i missed that condition... will re-test right now

andrewsykim · 2019-10-24T17:22:45Z

/reopen

k8s-ci-robot · 2019-10-24T17:22:49Z

@andrewsykim: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dvonthenen · 2019-10-25T16:01:55Z

Yep, I re-tested with 3 NICs with 3 different IPs one on each NIC and I don't see that error. Works with the latest CPI binary without updating the k8s.io/kubernetes that the CPI is built on.

Keep in mind, we also made the change that only the first IP is patched remember. That change isn't in the 1.0.0 release but only on master.

Also keep in mind that this PR #271 is trying to re-introduce multiple IPs.

EDIT: I should say selecting a single IP from multiple ones available.

andrewsykim · 2019-10-25T16:37:06Z

So sounds like this is already fixed in master but we're going to allow users to configure which networks the VM addresses should come from in #271? So a VM can have multiple NICs, but there can only be 1 InternalIP and 1 ExternalIP on the Node, both of which will be configurable after #271.

Is that correct @dvonthenen?

dvonthenen · 2019-10-28T15:08:00Z

@andrewsykim that is correct

frapposelli · 2019-12-04T17:14:11Z

@dvonthenen can you verify and close?

dvonthenen · 2019-12-10T21:58:52Z

Same as #270

frapposelli · 2019-12-18T14:08:04Z

closed via #284

/close

k8s-ci-robot · 2019-12-18T14:08:07Z

@frapposelli: Closing this issue.

In response to this:

closed via #284

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

deshui123 · 2020-04-03T09:19:49Z

hit the same issue for pure IPv6, related ticket #312

deshui123 · 2020-04-13T06:10:48Z

When using vSphere VM Network names, the error disappeared.

andrewsykim · 2020-07-21T16:05:37Z

FYI we also need to backport kubernetes/kubernetes#79391

lubronzhan · 2023-03-15T17:10:05Z

Hit this issue with k8s 1.18 + CCM 1.1.0

k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Oct 11, 2019

k8s-ci-robot assigned andrewsykim Oct 11, 2019

k8s-ci-robot assigned dvonthenen Oct 11, 2019

k8s-ci-robot closed this as completed Oct 24, 2019

k8s-ci-robot reopened this Oct 24, 2019

andrewsykim mentioned this issue Oct 28, 2019

Locate InternalIP and ExternalIP by vSphere Network name #271

Merged

frapposelli added this to the v1.1.0 milestone Dec 4, 2019

dvonthenen mentioned this issue Dec 12, 2019

Allow Internal/External status.Address Filtering by VM Network Name #284

Merged

k8s-ci-robot closed this as completed Dec 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cloud controller is unable to patch node with IP addresses from cloud provider #266

Cloud controller is unable to patch node with IP addresses from cloud provider #266

ivanfilippov commented Oct 11, 2019

frapposelli commented Oct 11, 2019 •

edited

k8s-ci-robot commented Oct 11, 2019

dvonthenen commented Oct 11, 2019

andrewsykim commented Oct 15, 2019

dvonthenen commented Oct 21, 2019

andrewsykim commented Oct 21, 2019

dvonthenen commented Oct 21, 2019 •

edited

andrewsykim commented Oct 21, 2019

dvonthenen commented Oct 24, 2019

k8s-ci-robot commented Oct 24, 2019

andrewsykim commented Oct 24, 2019

dvonthenen commented Oct 24, 2019

andrewsykim commented Oct 24, 2019

k8s-ci-robot commented Oct 24, 2019

dvonthenen commented Oct 25, 2019 •

edited

andrewsykim commented Oct 25, 2019 •

edited

dvonthenen commented Oct 28, 2019

frapposelli commented Dec 4, 2019

dvonthenen commented Dec 10, 2019

frapposelli commented Dec 18, 2019

k8s-ci-robot commented Dec 18, 2019

deshui123 commented Apr 3, 2020

deshui123 commented Apr 13, 2020

andrewsykim commented Jul 21, 2020

lubronzhan commented Mar 15, 2023

Cloud controller is unable to patch node with IP addresses from cloud provider #266

Cloud controller is unable to patch node with IP addresses from cloud provider #266

Comments

ivanfilippov commented Oct 11, 2019

frapposelli commented Oct 11, 2019 • edited

k8s-ci-robot commented Oct 11, 2019

dvonthenen commented Oct 11, 2019

andrewsykim commented Oct 15, 2019

dvonthenen commented Oct 21, 2019

andrewsykim commented Oct 21, 2019

dvonthenen commented Oct 21, 2019 • edited

andrewsykim commented Oct 21, 2019

dvonthenen commented Oct 24, 2019

k8s-ci-robot commented Oct 24, 2019

andrewsykim commented Oct 24, 2019

dvonthenen commented Oct 24, 2019

andrewsykim commented Oct 24, 2019

k8s-ci-robot commented Oct 24, 2019

dvonthenen commented Oct 25, 2019 • edited

andrewsykim commented Oct 25, 2019 • edited

dvonthenen commented Oct 28, 2019

frapposelli commented Dec 4, 2019

dvonthenen commented Dec 10, 2019

frapposelli commented Dec 18, 2019

k8s-ci-robot commented Dec 18, 2019

deshui123 commented Apr 3, 2020

deshui123 commented Apr 13, 2020

andrewsykim commented Jul 21, 2020

lubronzhan commented Mar 15, 2023

frapposelli commented Oct 11, 2019 •

edited

dvonthenen commented Oct 21, 2019 •

edited

dvonthenen commented Oct 25, 2019 •

edited

andrewsykim commented Oct 25, 2019 •

edited