New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-proxy takes 20 mins to sync ipvs rules on startup #53736

Closed
mcginne opened this Issue Oct 11, 2017 · 13 comments

Comments

Projects
None yet
6 participants
@mcginne

mcginne commented Oct 11, 2017

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:
I have enabled ipvs mode on Kube 1.8 and am having issues accessing NodePort Services. I notice if I restart kube-proxy I see the following messages every minute for ~20 minutes before it appears to actually sync the rules

Oct 11 10:33:57 worker-23.test.ibm.com kube-proxy.service[2058581]: I1011 10:33:57.095261 2058581 proxier.go:836] Not syncing ipvs rules until Services and Endpoints have been received from master
Oct 11 10:33:57 worker-23.test.ibm.com kube-proxy.service[2058581]: I1011 10:33:57.095307 2058581 proxier.go:832] syncProxyRules(EndpointsUpdate) took 63.9µs
Oct 11 10:33:57 worker-23.test.ibm.com kube-proxy.service[2058581]: I1011 10:33:57.095342 2058581 config.go:124] Calling handler.OnEndpointsAdd

We have ~35 services, so not a huge amount.

What you expected to happen:
I expect kube-proxy to update the rules immediately, or at least within a few minutes on startup.

How to reproduce it (as minimally and precisely as possible):
Start kube-proxy with the following parameters:

          --proxy-mode=ipvs \
          --ipvs-min-sync-period 60s \
          --ipvs-sync-period 300s \
          --feature-gates=SupportIPVSProxyMode=true \

View the logs.

Anything else we need to know?:
I am seeing various other issues with IPVS with it appearing to not sync correctly - i.e if I add/delete a service or delete a pod I regularly see the changes not being reflected in the IPVS rules - it appears kube-proxy is not getting notified about changes.

I compared the iptables proxy code with ipvs code at branch 1.8 and can see a difference in the NewProxier function.
iptables has the following:

	burstSyncs := 2
	glog.V(3).Infof("minSyncPeriod: %v, syncPeriod: %v, burstSyncs: %d", minSyncPeriod, syncPeriod, burstSyncs)
	proxier.syncRunner = async.NewBoundedFrequencyRunner("sync-runner", proxier.syncProxyRules, minSyncPeriod, syncPeriod, burstSyncs)
	return proxier, nil

Whereas this is missing in ipvs at the 1.8 branch. I notice this code is present in master for ipvs. Could this be causing the sync issues I'm seeing? If so is this change likely to make it into a 1.8.x release in the near future as we would like to progress testing of ipvs.

Environment:

  • Kubernetes version (use kubectl version): Kubernetes v1.8.0-1+edd813e3ac8824
  • Cloud provider or hardware configuration**: VSIs running on IBM Softlayer
  • OS (e.g. from /etc/os-release):Ubuntu 16.04.2 LTS (Xenial)
  • Kernel (e.g. uname -a):4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
  • Others:
@mcginne

This comment has been minimized.

Show comment
Hide comment
@mcginne

mcginne Oct 11, 2017

@kubernetes/sig-network-bugs

mcginne commented Oct 11, 2017

@kubernetes/sig-network-bugs

@mcginne

This comment has been minimized.

Show comment
Hide comment
@mcginne

mcginne Oct 11, 2017

/sig network

mcginne commented Oct 11, 2017

/sig network

@mcginne

This comment has been minimized.

Show comment
Hide comment
@mcginne

mcginne Oct 11, 2017

Adding kibe-proxy log showing the delay:

kube-proxyLog-delay.txt

mcginne commented Oct 11, 2017

Adding kibe-proxy log showing the delay:

kube-proxyLog-delay.txt

@mcginne

This comment has been minimized.

Show comment
Hide comment
@mcginne

mcginne Oct 12, 2017

I rebuilt kube-proxy with this commit -> 44afb09 (which includes the change I mentioned above) and it now seems to behave much better, and is syncing on startup and when changes occur.
If someone could confirm this will make it into the net update for V1.8 I'm happy to close this issue

mcginne commented Oct 12, 2017

I rebuilt kube-proxy with this commit -> 44afb09 (which includes the change I mentioned above) and it now seems to behave much better, and is syncing on startup and when changes occur.
If someone could confirm this will make it into the net update for V1.8 I'm happy to close this issue

@cmluciano

This comment has been minimized.

Show comment
Hide comment
@cmluciano
Member

cmluciano commented Oct 17, 2017

/assign @m1093782566

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566

m1093782566 Oct 18, 2017

Member

I rebuilt kube-proxy with this commit -> 44afb09 (which includes the change I mentioned above) and it now seems to behave much better, and is syncing on startup and when changes occur.

That's true.

If someone could confirm this will make it into the net update for V1.8 I'm happy to close this issue

I would like to cherry-pick the bug fix into 1.8 but don't have much experience. @cmluciano any consult?

Member

m1093782566 commented Oct 18, 2017

I rebuilt kube-proxy with this commit -> 44afb09 (which includes the change I mentioned above) and it now seems to behave much better, and is syncing on startup and when changes occur.

That's true.

If someone could confirm this will make it into the net update for V1.8 I'm happy to close this issue

I would like to cherry-pick the bug fix into 1.8 but don't have much experience. @cmluciano any consult?

@cmluciano

This comment has been minimized.

Show comment
Hide comment
@cmluciano

cmluciano Oct 18, 2017

Member

@m1093782566 Take a look here and let me know if anything is still unclear.

Member

cmluciano commented Oct 18, 2017

@m1093782566 Take a look here and let me know if anything is still unclear.

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566

m1093782566 Oct 19, 2017

Member

@cmluciano thanks a lot!

Member

m1093782566 commented Oct 19, 2017

@cmluciano thanks a lot!

@jsravn

This comment has been minimized.

Show comment
Hide comment
@jsravn

jsravn Nov 1, 2017

Contributor

@m1093782566 Did you cherry pick it? :) I'm itching to try some ipvs goodness. Or do you think we should wait for 1.9?

Contributor

jsravn commented Nov 1, 2017

@m1093782566 Did you cherry pick it? :) I'm itching to try some ipvs goodness. Or do you think we should wait for 1.9?

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566

m1093782566 Nov 1, 2017

Member

I would suggest wait for 1.9. There are some known bugs in 1.8 and should be fixed in 1.9.

Member

m1093782566 commented Nov 1, 2017

I would suggest wait for 1.9. There are some known bugs in 1.8 and should be fixed in 1.9.

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566

m1093782566 Nov 20, 2017

Member

/area ipvs

Member

m1093782566 commented Nov 20, 2017

/area ipvs

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566

m1093782566 Dec 19, 2017

Member

I believe it's fixed in v1.9

I am going to close this issue now. Please re-open if it still persist.

Member

m1093782566 commented Dec 19, 2017

I believe it's fixed in v1.9

I am going to close this issue now. Please re-open if it still persist.

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566

m1093782566 Dec 19, 2017

Member

/close

Member

m1093782566 commented Dec 19, 2017

/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment