Clickhouse nodes OOM Kill

[Magicbeanbuyer]

Company or project name

DeepL SE

Describe the unexpected behaviour

Our Clickhouse nodes are suffering from OOM Kill frequently, and we have observed that the Clickhouse process's RSS is higher than our max_server_memory_usage_to_ram_ratio = 0.9 configuration.

How to reproduce

I conducted some experiments with a Clickhouse Server running in a Docker container and scraped the Clickhouse process's RSS using top command.

Prepare test data

client.query(
"""CREATE OR REPLACE TABLE test_merge_parts 
(
  id UInt64,
  value String
)
ENGINE=MergeTree 
ORDER BY id"""
)

client.query("SYSTEM STOP MERGES test_merge_parts")

for i in range(680):
    client.query(
        f"""INSERT INTO test_merge_parts 
        SELECT
            number as id,
            toString(number) as value
        FROM numbers(700000)"""
    )

Run lightweight delete

client.command("SET optimize_throw_if_noop=1")
client.command("SYSTEM START MERGES test_merge_parts")
client.query("DELETE FROM test_merge_parts WHERE modulo(id, 2) != 0;")

This would lead to Docker container killed by the OS.

Plot the Clickhouse memory consumption with data from top command. It is observed that Clickhouse RSS was frequently above the 90% max_server_memory_usage_to_ram_ratio.

Expected behavior

The Clickhouse process should honor the max_server_memory_usage_to_ram_ratio and kill memory-intensive queries or background processes to control its memory consumption.

Error message and/or stacktrace

Following are error logs from our production cluster.

{"CODE_FILE":"src/core/unit.c","CODE_FUNC":"unit_log_failure","CODE_LINE":"5564","INVOCATION_ID":"4baa083c21b8431b84bc6c10261f651b","MESSAGE_ID":"d9b373ed55a64feb8242e02dbe79a49c","PRIORITY":"4","SYSLOG_FACILITY":"3","SYSLOG_IDENTIFIER":"systemd","TID":"1","UNIT":"clickhouse-server.service","UNIT_RESULT":"oom-kill","_BOOT_ID":"f9ad45b850bc4ff5b0ccf0fe309d1394","_CAP_EFFECTIVE":"1ffffffffff","_CMDLINE":"/sbin/init","_COMM":"systemd","_EXE":"/usr/lib/systemd/systemd","_GID":"0","_MACHINE_ID":"0aabf60b01e14472a0138d6cae5fc8d5","_PID":"1","_SELINUX_CONTEXT":"unconfined\n","_SOURCE_REALTIME_TIMESTAMP":"1742086974123395","_SYSTEMD_CGROUP":"/init.scope","_SYSTEMD_SLICE":"-.slice","_SYSTEMD_UNIT":"init.scope","_TRANSPORT":"journal","_UID":"0","__MONOTONIC_TIMESTAMP":"2112531994843","__REALTIME_TIMESTAMP":"1742086974123678","message":"clickhouse-server.service: Failed with result 'oom-kill'."}

{"CODE_FILE":"src/core/unit.c","CODE_FUNC":"unit_log_failure","CODE_LINE":"5564","INVOCATION_ID":"4baa083c21b8431b84bc6c10261f651b","MESSAGE_ID":"d9b373ed55a64feb8242e02dbe79a49c","PRIORITY":"4","SYSLOG_FACILITY":"3","SYSLOG_IDENTIFIER":"systemd","TID":"1","UNIT":"clickhouse-server.service","UNIT_RESULT":"oom-kill","_BOOT_ID":"f9ad45b850bc4ff5b0ccf0fe309d1394","_CAP_EFFECTIVE":"1ffffffffff","_CMDLINE":"/sbin/init","_COMM":"systemd","_EXE":"/usr/lib/systemd/systemd","_GID":"0","_MACHINE_ID":"0aabf60b01e14472a0138d6cae5fc8d5","_PID":"1","_SELINUX_CONTEXT":"unconfined\n","_SOURCE_REALTIME_TIMESTAMP":"1742086974123395","_SYSTEMD_CGROUP":"/init.scope","_SYSTEMD_SLICE":"-.slice","_SYSTEMD_UNIT":"init.scope","_TRANSPORT":"journal","_UID":"0","__MONOTONIC_TIMESTAMP":"2112531994843","__REALTIME_TIMESTAMP":"1742086974123678","message":"clickhouse-server.service: Failed with result 'oom-kill'."}

Additional context

Clickhouse Version: 24.12.5.81

[Magicbeanbuyer]

Log from our production cluster shortly before an OOM Kill

Out of memory: Killed process 203738 (clickhouse-serv) total-vm:13753086420kB, anon-rss:472957280kB, file-rss:114688kB, shmem-rss:0kB, UID:995 pgtables:23658068kB oom_score_adj:0

[alexey-milovidov]

ClickHouse does best effort to prevent the use of memory over the limit, but on low-memory systems it can overrun, especially in the presence of more applications on the same machine.

[Magicbeanbuyer]

Hi @alexey-milovidov @den-crane ,

The 1 GiB is just for the easy of reproducing the issue.

Our dedicated production nodes have 503Gi each with max_server_memory_usage_to_ram_ratio = 0.9 configuration, and we have also observed clickhouse memory over consumption, and the nodes are frequently OOM killed.

Would appreciate some input from you.

[nickitat]

please check if async metric CGroupMemoryTotal displays the correct value

[IngaFeick]

Hi @nickitat I'm working with @Magicbeanbuyer on this. Interestingly neither ClickHouseAsyncMetrics_CGroupMemoryTotal nor ClickHouseAsyncMetrics_CGroupMemoryUsed are being exported on our production nodes. If you happen to have an idea for any settings or circumstances that might cause this to happen then I'd welcome any suggestions. Otherwise I'll go through the code to figure this out. Thank you!

[nickitat]

First of all, make sure that the CH process runs under cgroups. Then, I recommend taking the most recent patch release (the last fix in this area, to my memory, is #78566) and double-checking that the problem persists.
What version you're on right now?

[IngaFeick]

We're on 25.1.8.25 and the cgroup is 0::/system.slice/clickhouse-server.service

[filimonov]

Our dedicated production nodes have 503Gi each with max_server_memory_usage_to_ram_ratio = 0.9 configuration, and we have also observed clickhouse memory over consumption, and the nodes are frequently OOM killed.

If the memory is allocated very rapidly by many concurrent thread clickhouse may allocate more ram than the limit allows.

Memory allocation is a 'critical' path, and the accounting is not 100% accurate due to lack of locks / acid guarantees there, it uses lightweight atomic counters.

For example if you are 1GB till the limit, and 5 threads simultnatiosly will check that '1GB is still allowed' they all can request 1GB simultanioulsy and the memory usage will become bigger that it should.

In usual scenarios the chances of such race are minimal. It's better to contol concurrency level and memory usage of the queries. Or just shift down the limit, to avoid workind very close to the place where oomkiller will start to operate.