Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race between kube-proxy start up and node registration #37414

Closed
freehan opened this issue Nov 24, 2016 · 30 comments

Comments

@freehan
Copy link
Member

commented Nov 24, 2016

During kube-proxy bootstrap, kube-proxy retrieves node IP by accessing node object.
https://github.com/kubernetes/kubernetes/blob/release-1.5/cmd/kube-proxy/app/server.go#L464

Because kube-proxy is a static pod, kubelet may start kube-proxy before completing self registration. Hence kube-proxy may start up without knowing the correct node IP.

@Random-Liu

This comment has been minimized.

Copy link
Member

commented Nov 24, 2016

@yujuhong to confirm.

@calebamiles calebamiles modified the milestone: v1.6 Mar 8, 2017

@ethernetdan ethernetdan modified the milestone: v1.6 Mar 10, 2017

@ApsOps

This comment has been minimized.

Copy link
Contributor

commented Apr 13, 2017

I'm hitting this as well, for the same reason - kube-proxy comes up before node registration is complete.

E0412 12:44:15.750069   23359 server.go:388] Can't get Node "ip-10-0-8-170", assuming iptables proxy, err: nodes "ip-10-0-8-170" not found
I0412 12:44:15.750971   23359 server.go:203] Using iptables Proxier.
W0412 12:44:15.751910   23359 server.go:436] Failed to retrieve node info: nodes "ip-10-0-8-170" not found
W0412 12:44:15.751970   23359 proxier.go:226] invalid nodeIP, initialize kube-proxy with 127.0.0.1 as nodeIP
I0412 12:44:15.751988   23359 server.go:215] Tearing down userspace rules.
I0412 12:44:15.752397   23359 healthcheck.go:119] Initializing kube-proxy health checker
I0412 12:44:15.765032   23359 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_max' to 262144
I0412 12:44:15.765335   23359 conntrack.go:66] Setting conntrack hashsize to 65536
I0412 12:44:15.765595   23359 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0412 12:44:15.765625   23359 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600

I'm wondering what are the consequences of this? 🤔

@yujuhong

This comment has been minimized.

Copy link
Member

commented Apr 13, 2017

@yujuhong to confirm.

I completely missed the issue :-\

kube-proxy uses the host network, so it's not blocked by kubelet registering and getting the pod CIDR. Can't kube-proxy wait and/or retry?
@freehan is the bug still valid or has it been fixed?

@freehan

This comment has been minimized.

Copy link
Member Author

commented Apr 13, 2017

This is not fixed. We are moving kube-proxy to daemonset. I am wondering if there is some easy config to make sure kube-proxy starts up after node status becomes ready or some sort.

@yujuhong

This comment has been minimized.

Copy link
Member

commented Apr 13, 2017

This is not fixed. We are moving kube-proxy to daemonset.
yay!

This is not fixed. We are moving kube-proxy to daemonset. I am wondering if there is some easy config to make sure kube-proxy starts up after node status becomes ready or some sort.

Is making kube-proxy blocked until node is ready not an option?

@thockin thockin added this to the v1.7 milestone May 27, 2017

@thockin

This comment has been minimized.

Copy link
Member

commented Jun 1, 2017

@mspreitzer

@MikeSpreitzer

This comment has been minimized.

Copy link
Member

commented Jun 1, 2017

/assign

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2017

@MikeSpreitzer: GitHub didn't allow me to assign the following users: MikeSpreitzer.

Note that only kubernetes members can be assigned.

In response to this:

/assign

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@MikeSpreitzer

This comment has been minimized.

Copy link
Member

commented Jun 1, 2017

/assign

@thockin

This comment has been minimized.

Copy link
Member

commented Jun 9, 2017

Update?

@MikeSpreitzer

This comment has been minimized.

Copy link
Member

commented Jun 9, 2017

I got derailed by #43556 ; clues there would be nice. I will get back to this today, one way or another.

@MikeSpreitzer

This comment has been minimized.

Copy link
Member

commented Jun 9, 2017

I have been reading about tests, wondering where to add a test case that covers this problem. It is not clear to me that Kubernetes has a test suite that can test race conditions in cluster creation.

MikeSpreitzer added a commit to MikeSpreitzer/kubernetes that referenced this issue Jun 9, 2017

kube-proxy: wait, if necessary, for node to be defined
Make the kube-proxy retry fetching the local Node a few times with
exponential backoff, to resolve the race between startup of kube-proxy
and kubelet.

Resolves issue kubernetes#37414
@MikeSpreitzer

This comment has been minimized.

Copy link
Member

commented Jun 12, 2017

BTW, hack/local-up-cluster.sh always makes the kube-proxy before the kubelet, so it is easy to probe this issue in manual testing.

@bowei

This comment has been minimized.

Copy link
Member

commented Jun 13, 2017

Next milestone reason: it seems that there are still some conversations about this the PR fix around whether or not we want to make kube-proxy require the presence of kubelet. It seems warranted then to accept this fix in the next release.

@matchstick

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2017

We need to address this in a patch release, but it should not block 1.7.
@kubernetes/sig-network-misc

@matchstick matchstick modified the milestones: next-candidate, v1.7 Jun 13, 2017

MikeSpreitzer added a commit to MikeSpreitzer/kubernetes that referenced this issue Jun 13, 2017

kube-proxy: wait, if necessary, for node to be defined
Make the kube-proxy retry fetching the local Node a few times with
exponential backoff, to resolve the race between startup of kube-proxy
and kubelet.  There is a limited number of retries, and if the Node
can not be fetched within the limit then the kube-proxy exits with an
error so that a higher-level thing can deal with the problem.

Resolves issue kubernetes#37414

MikeSpreitzer added a commit to MikeSpreitzer/kubernetes that referenced this issue Aug 10, 2017

kube-proxy: wait, if necessary, for node to be defined
Make the kube-proxy retry fetching the local Node a few times with
exponential backoff, to resolve the race between startup of kube-proxy
and kubelet.  There is a limited number of retries, and if the Node
can not be fetched within the limit then the kube-proxy exits with an
error so that a higher-level thing can deal with the problem.

Resolves issue kubernetes#37414

MikeSpreitzer added a commit to MikeSpreitzer/kubernetes that referenced this issue Aug 10, 2017

kube-proxy: wait, if necessary, for node to be defined
Make the kube-proxy retry fetching the local Node a few times with
exponential backoff, to resolve the race between startup of kube-proxy
and kubelet.  There is a limited number of retries, and if the Node
can not be fetched within the limit then the kube-proxy exits with an
error so that a higher-level thing can deal with the problem.

Resolves issue kubernetes#37414

MikeSpreitzer added a commit to MikeSpreitzer/kubernetes that referenced this issue Aug 11, 2017

kube-proxy: wait, if necessary, for node to be defined
Make the kube-proxy retry fetching the local Node a few times with
exponential backoff, to resolve the race between startup of kube-proxy
and kubelet.  There is a limited number of retries, and if the Node
can not be fetched within the limit then the kube-proxy exits with an
error so that a higher-level thing can deal with the problem.

Resolves issue kubernetes#37414
@jaycoon

This comment has been minimized.

Copy link

commented Dec 22, 2017

This is impacting one of our worker nodes as well, which was working fine and then after a restart, the API server communication was impacted (for POST requests, e.g. logs, exec). We noticed the the invalid nodeIP error in the kube-proxy logs.

The workaround is to detect the failure via the logs, and restart the kube-proxy pod.

@fejta-bot

This comment has been minimized.

Copy link

commented Mar 22, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

commented Apr 21, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@thockin

This comment has been minimized.

Copy link
Member

commented Apr 29, 2018

As far as I can tell, the nodeIP is used in one particular case and nowhere else. Specifically a service of type=LoadBalancer, which also uses LoadBalancerSourceRanges field AND specifies the node itself as a valid source range for the LB.

That's a pretty corner case. I bet we could find a better way to express that, and eliminate that parameter and error message entirely. I doubt it is actually causing anyone problems unless I totally misunderstand what that field is doing

@fejta-bot

This comment has been minimized.

Copy link

commented May 29, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@paulsubrata55

This comment has been minimized.

Copy link
Contributor

commented Apr 4, 2019

@thockin Since the PR #74341 is merged to master, nodeIP is also used in ipvs/proxier.go. Now if this race condition happens, kube-proxy is not able to determine the correct nodeIP. This affects all the SCTP nodeport services with kube-proxy mode ipvs.

Please reopening the issue.

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Apr 4, 2019

@paulsubrata55: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@danwinship

This comment has been minimized.

Copy link
Contributor

commented Apr 4, 2019

looks like this shouldn't have gotten closed anyway
/reopen
/remove-lifecycle rotten
/lifecycle frozen

@k8s-ci-robot k8s-ci-robot reopened this Apr 4, 2019

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Apr 4, 2019

@danwinship: Reopened this issue.

In response to this:

looks like this shouldn't have gotten closed anyway
/reopen
/remove-lifecycle rotten
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@paulsubrata55

This comment has been minimized.

Copy link
Contributor

commented Apr 13, 2019

Can't we use the similar fix given in PR #47262.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.