Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Leader election causes a lot of endpoint updates #23812

Closed
hanikesn opened this issue Apr 4, 2016 · 12 comments
Closed

Leader election causes a lot of endpoint updates #23812

hanikesn opened this issue Apr 4, 2016 · 12 comments
Assignees

Comments

@hanikesn
Copy link

@hanikesn hanikesn commented Apr 4, 2016

After enabling leader election for scheduler and controller-manager there's a lot of noise on /api/v1/endpoints?watch=true:

{"type":"MODIFIED","object":{"kind":"Endpoints","apiVersion":"v1","metadata":{"name":"kube-scheduler","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/endpoints/kube-scheduler","uid":"817fe020-fa4f-11e5-b3f8-9457a55ed624","resourceVersion":"1141419","creationTimestamp":"2016-04-04T10:25:08Z","annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"kubetest2\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2016-04-04T10:25:08Z\",\"renewTime\":\"2016-04-04T11:33:38Z\",\"leaderTransitions\":0}"}},"subsets":[]}}
{"type":"MODIFIED","object":{"kind":"Endpoints","apiVersion":"v1","metadata":{"name":"kube-controller-manager","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/endpoints/kube-controller-manager","uid":"8143a91d-fa4f-11e5-b3f8-9457a55ed624","resourceVersion":"1141421","creationTimestamp":"2016-04-04T10:25:08Z","annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"kubetest2\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2016-04-04T10:25:08Z\",\"renewTime\":\"2016-04-04T11:33:39Z\",\"leaderTransitions\":0}"}},"subsets":[]}}

This noise is also hard to filter out because there are no labels set for these two endpoints.

@zhouhaibing089

This comment has been minimized.

Copy link
Contributor

@zhouhaibing089 zhouhaibing089 commented Apr 4, 2016

I think each renew would cause a MODIFY.

@bgrant0607

This comment has been minimized.

Copy link
Member

@bgrant0607 bgrant0607 commented Apr 6, 2016

@j-vizcaino

This comment has been minimized.

Copy link

@j-vizcaino j-vizcaino commented Jun 1, 2016

Endpoints being modified all the time, kube-proxy syncs (ie. flush/restore) iptables entries very often (about every 500ms on a 3 node install).
This is really annoying if you want to tackle iptables related problems (and maybe lead to performance problems for large clusters)

@mikedanese

This comment has been minimized.

Copy link
Member

@mikedanese mikedanese commented Jun 1, 2016

You can reduce the frequency of renew at the expense of a speedier master failover. In larger clusters this will likely be dwarfed by pods flucuating. If this causes performance issues in large clusters, we need to fix kube proxy or apiserver or whatever component is affected. We need to support frequent object updates. There isn't evidence that this is the case and we run scalability tests that apply significantly more load in large clusters.

This is different then the issue described by OP. For that we need to add labels to the master endpoints.

@j-vizcaino

This comment has been minimized.

Copy link

@j-vizcaino j-vizcaino commented Jun 1, 2016

@mikedanese Do you want me to file a new issue regarding the kube-proxy flushing iptables way too often?

@mikedanese

This comment has been minimized.

Copy link
Member

@mikedanese mikedanese commented Jun 1, 2016

Yes please

@jonaz

This comment has been minimized.

Copy link

@jonaz jonaz commented Jun 9, 2016

The only thing changeing is renewTime so we get those events every second.

@timothysc timothysc self-assigned this Jul 28, 2016
@timothysc

This comment has been minimized.

Copy link
Member

@timothysc timothysc commented Jul 28, 2016

Is this still an issue?

@j-vizcaino

This comment has been minimized.

Copy link

@j-vizcaino j-vizcaino commented Jul 29, 2016

I haven't had the time to update to 1.3, so I cannot say.

On Thu, Jul 28, 2016, 21:12 Timothy St. Clair notifications@github.com
wrote:

Is this still an issue?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#23812 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEfKlf1KHRiqCmaKa7uyds_NMtuveRY8ks5qaP8lgaJpZM4H_A2q
.

@obeattie

This comment has been minimized.

Copy link

@obeattie obeattie commented Aug 22, 2016

This is definitely an issue in 1.3. I see iptables-restore on all my nodes very frequently, and I can see the elected masters causing an endpoint update every second.

@timothysc

This comment has been minimized.

Copy link
Member

@timothysc timothysc commented Aug 22, 2016

We should try to filter updates to the proxy when the only change is an annotation that it's not watching. This seems to be an unintended consequence of leveraging endpoints.

@mikedanese

This comment has been minimized.

Copy link
Member

@mikedanese mikedanese commented Aug 22, 2016

Do the frequent restores cause any problems? Agree we shouldn't restore if the state hasn't changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.