-
Notifications
You must be signed in to change notification settings - Fork 38.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adjust watch channel sizes in watchcache #109708
Adjust watch channel sizes in watchcache #109708
Conversation
Please note that we're already in Test Freeze for the |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: wojtek-t The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
||
const ( | ||
// minWatchChanSize is the min size of channels used by the watch. | ||
minWatchChanSize = 3 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would not change min size till we have proper tracking of terminating watchers in load test, WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm fine with blocking on perf-test change on that.
I wanted to change that, because technically we're risking increasing some of channels and wanted to compensate for that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Issue created for it: kubernetes/perf-tests#2054
/hold |
/triage accepted |
68d37ae
to
0db5c05
Compare
/test pull-kubernetes-e2e-kind-ipv6 |
/retest |
/lgtm |
/retest |
And we don't see visible change in memory usage - which is great. |
…!882) 1.20:移植社区启发式设置watch通道大小的特性,并支持通过env设置某些资源watch通道大小 1 51d81e2 移植 kubernetes#109708 把社区currentCapacity()中的写锁改成了读锁 2 8995bf4 增加RESOURCE_WATCH_CHAN_SIZE env,精细控制资源的watch通道大小
Currently if there are types where we have very high churn (say leases in 5k-node clusters generate 500 QPS) for which we don't define trigger function in watchcache (so leases are the best example), we observe that re-instantiating watch may suffer from problems, because processing the init events may take more time than filling in the incomming channel.
This PR is switching the way those channels are initialized based on some heuristic around throughput of those objects.
/kind bug
/sig api-machinery
/priority important-longterm
/assign @mborsz @marseel