Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

performance enhance for kube-proxy/ipvs syncProxyRules #84906

Closed
njuicsgz opened this issue Nov 7, 2019 · 4 comments · Fixed by #84924

Comments

@njuicsgz
Copy link
Contributor

@njuicsgz njuicsgz commented Nov 7, 2019

What would you like to be added:
performance enhance for kube-proxy/ipvs syncProxyRules

Why is this needed:
For kube-proxy/ipvs, syncProxyRules will take about 10s+ when the number of k8s Service with external ip > 1500+. It has those problem:

  • The ipvs rules of endpoints change may delay 10s after a pod is terminated, within the period, all requests route to this endpoint will fail.
  • The ipvs rules of Services change may delay 10s. It means that, a service may not be ready after created successfully until 10s later.

After some debug tests, I found that most of time spend on utilproxy.IsLocalIP(externalIP)

I1104 16:34:01.051111   14602 proxier.go:980] [debug] check externalIP open port: 4.5211ms
I1104 16:34:01.066563   14602 proxier.go:980] [debug] check externalIP open port: 6.349011ms
I1104 16:39:51.858951   11581 proxier.go:744] [debug] syncProxyRules took 12.045648906s

And fortunately, we could simply move the logic of geting net interface addrs from the for loop to enhance performance safely. After this change, the sync time could reduce from 12s to 730ms.

I1105 10:18:30.002215   31985 proxier.go:1002] [debug] check externalIP open port: 156ns
I1105 10:18:30.687643   31985 proxier.go:793] [debug] syncProxyRules, took 731.416349ms

I will push a PR soon for this.

@athenabot

This comment has been minimized.

Copy link

@athenabot athenabot commented Nov 7, 2019

/sig network

These SIGs are my best guesses for this issue. Please comment /remove-sig <name> if I am incorrect about one.

🤖 I am a bot run by vllry. 👩‍🔬

@k8s-ci-robot k8s-ci-robot added sig/network and removed needs-sig labels Nov 7, 2019
@athenabot

This comment has been minimized.

Copy link

@athenabot athenabot commented Nov 7, 2019

/triage unresolved

Comment /remove-triage unresolved when the issue is assessed and confirmed.

🤖 I am a bot run by vllry. 👩‍🔬

@andrewsykim

This comment has been minimized.

Copy link
Member

@andrewsykim andrewsykim commented Nov 8, 2019

Thanks for finding this

@andrewsykim

This comment has been minimized.

Copy link
Member

@andrewsykim andrewsykim commented Nov 8, 2019

/assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.