Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubelet flushes KUBE-HOSTPORTS and KUBE-MARK-MASQ when pods start #32415

Closed
thockin opened this issue Sep 9, 2016 · 2 comments
Closed

kubelet flushes KUBE-HOSTPORTS and KUBE-MARK-MASQ when pods start #32415

thockin opened this issue Sep 9, 2016 · 2 comments
Assignees
@freehan
Labels
area/kubelet priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Milestone
v1.3

Comments

@thockin
Copy link
Member

thockin commented Sep 9, 2016

Observed: hitting a service nodePort fails intermittently, but the service cluster IP works 100%

Debug: tcpdump shows SYN sent, but no SYNACK returned. I noticed that in the error case, srcip was 127.0.0.1 - clearly wrong. We proved that the KUBE-MARK-MASQ chain was being flushed and so we were not getting SNAT'ed. We proved it was kubelet that was flushing, and kube-proxy that eventually restored it (yay for rectification loops!).

We found the hostport code in kubenet erroneously flushes those chains when starting a pod. After that it can take up to several minutes for kube-proxy to hit its own sync loop and fix the problem.

The fix is easy - don't flush those chains. @freehan is working on the fix right now.

@fabioy @timstclair for 1.3.x
@pwittrock for 1.4.x
@spxtr for reporting it concretely enough to repro
@thockin thockin added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. area/kubelet labels Sep 9, 2016
@thockin thockin added this to the v1.3 milestone Sep 9, 2016
@freehan freehan mentioned this issue Sep 9, 2016
Merged
@pwittrock pwittrock mentioned this issue Sep 9, 2016
Closed
@pwittrock
Copy link
Member

ACK. Created #32417 as a placeholder in the 1.4 milestone.

k8s-github-robot pushed a commit that referenced this issue Sep 9, 2016
Merge pull request #32413 from freehan/hostportfix
e43f605 
Automatic merge from submit-queue

Fix a bug in kubelet hostport logic which flushes KUBE-MARK-MASQ iptables chain

Fixes #32415
@spxtr
Copy link
Contributor

spxtr commented Sep 9, 2016

Manually confirmed this fixes my issue. Thanks so much for fixing this :)

@spxtr spxtr mentioned this issue Sep 12, 2016
Closed
@pieterlange pieterlange mentioned this issue Sep 27, 2016
Open
@spxtr spxtr mentioned this issue Nov 23, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@freehan freehan

Labels
area/kubelet priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Projects
None yet
Milestone
v1.3
Development

No branches or pull requests
5 participants
@freehan @thockin @spxtr @dchen1107 @pwittrock