New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider nested cgroup memory limits for the total memory tracker #59435
Consider nested cgroup memory limits for the total memory tracker #59435
Conversation
This is an automated comment for commit ad13a57 with description of existing statuses. It's updated for the latest CI running ✅ Click here to open a full report in a separate page Successful checks
|
This fixes the edge case that cgroup memory limit "0" is no longer interpreted as "no limit" but as actual zero available memory, see (*). The cgroup default setting "max" continues to be interpreted as "no limit". (*) https://docs.kernel.org/admin-guide/cgroup-v2.html
b204809
to
531a9e4
Compare
531a9e4
to
80a0e05
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it is not particularly relevant to this pr, but in general seems like there are quite a few places when any logging might be useful
I have a problem, I think, related to this change. I'm running the Clickhouse docker image version 24.2.2.71 (official build) on Ubuntu 22.04 (cgroup v2 enabled by default, no cgroup v1) with error in log. This error did not occur when I ran the previous version 24.1 of Clickhouse. Application: Disabling cgroup memory observer because of an error during initialization: Code: 107. DB::Exception: Cannot find cgroups v1 or v2 current memory file. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):
Additional information about my HOST system
Additional information from the inside of running container
|
As you can see, there are no nested cgroups in /sys/fs/cgroup Due to problem of "Disabling cgroup memory observer", the available memory is incorrectly determined. Clickhouse believes that all HOST memory is available without respect of cgroup limit. This causes container memory overruns and triggers OOM killer. |
@Stanislav-Vlasov First, can you please open a GitHub issue for that? That would be a better place for discussion than the original PR. As a workaround, you could set a sensible value for server setting Exception For completeness, I tested the latest official Docker image (v24.3, Ubuntu 22.04 LTS) and it worked for me, i.e. no errors. |
Review note: Contains commits
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
At startup and whenever config files are changed, ClickHouse updates the hard memory limits of its total memory tracker. These limits are computed based on various server settings and cgroups limits (on Linux). Previously, setting
/sys/fs/cgroup/memory.max
(for cgroups v2) was hard-coded. As a result, cgroup v2 memory limits configured for nested groups (hierarchies), e.g./sys/fs/cgroup/my/nested/group/memory.max
were ignored. This is now fixed. The behavior of v1 memory limits remains unchanged.