-
Notifications
You must be signed in to change notification settings - Fork 6.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update memory tracker periodically with cgroup memory usage #59516
Merged
rschu1ze
merged 10 commits into
ClickHouse:master
from
rschu1ze:cgroup-observing-memory-tracker
Feb 26, 2024
Merged
Update memory tracker periodically with cgroup memory usage #59516
rschu1ze
merged 10 commits into
ClickHouse:master
from
rschu1ze:cgroup-observing-memory-tracker
Feb 26, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
This is an automated comment for commit d4b1517 with description of existing statuses. It's updated for the latest CI running ❌ Click here to open a full report in a separate page Successful checks
|
665dbde
to
e292c97
Compare
This comment was marked as outdated.
This comment was marked as outdated.
e292c97
to
c2cd4e7
Compare
c2cd4e7
to
50bdb87
Compare
50bdb87
to
42ba6ca
Compare
2b18995
to
3a1ba69
Compare
3a1ba69
to
f6f1fd4
Compare
hanfei1991
reviewed
Feb 19, 2024
alesapin
reviewed
Feb 26, 2024
alesapin
reviewed
Feb 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
pr-backports-created-cloud
pr-must-backport-cloud
pr-not-for-changelog
This PR should not be mentioned in the changelog
pr-synced-to-cloud
The PR is synced to the cloud repo
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When the system runs out of memory (as determined by system-wide settings or cgroups memory thresholds), the OOM killer stops memory-intensive processes. Due to its nature, ClickHouse will often be the victim. To avoid this, the upper limit of the (global) memory tracker in ClickHouse can be configured to a value smaller than the "kill threshold", see setting max_server_memory_usage_ram_ratio.
There are some problems with this approach:
The maximum memory threshold of the processes's cgroup can be changed (increased, decreased) while ClickHouse runs. An earlier fix (Respect cgroup memory limit when reloading configuration #53455) ensured that new memory limits were at least considered when ClickHouse's configuration changes (a follow-up fix Consider nested cgroup memory limits for the total memory tracker #59435 resolved problems with nested cgroups). Of course, users need to make real or dummy configuration changes for this to happen and it may already be too late at this point.
The memory tracking in ClickHouse is not 100% precise. For example, ClickHouse may think it uses less memory than the OS thinks. In that case, there is a risk of getting OOM-killed. Or the other way round, ClickHouse produces OOMs even if there is sufficient memory.
This PR tries to address this by reading the current memory consumption as seen by cgroups periodically and running two possible actions:
The read interval (setting
cgroups_memory_usage_observer_wait_time
) is 15 sec by default and it can be reduced to 1 sec if needed. 0 means disabled. The parameter cannot be changed after database start.The hard and soft limits are controlled by parameters
cgroup_memory_watcher_hard_limit_ratio
andcgroup_memory_watcher_soft_limit_ratio
(defaults: 0.95, 0.9). They each refer to the initial maximum allowed global memory consumption computed astotal system memory x max_server_memory_usage_to_ram_ratio
. Both parameters can be changed at runtime.Changelog category (leave one):