self-hosting: ClickHouse idle CPU usage at ~140% due to log file spin loop

[SeanPedersen]

Bug Report

Summary

The op-ch (ClickHouse) container in the self-hosting stack consumes ~140% CPU indefinitely even when completely idle. This is caused
by a silent spin loop in ClickHouse's async logging thread.

Environment

• OpenPanel self-hosting (Docker Compose)
• ClickHouse 25.10.2.65
• Observed on a deployment left running for several weeks

Root Cause

The issue is a conflict between the default ClickHouse config and the op-config override:

1. config.xml (default, shipped with the ClickHouse image) configures file logging:

   <log>/var/log/clickhouse-server/clickhouse-server.log</log>
   <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
   <size>1000M</size>

2. clickhouse-config.xml (the OpenPanel override) adds true but does not disable or override the file log paths.
3. At some point after startup, ClickHouse loses the ability to write to the log file (e.g. host-side logrotate replaces the inode,
   breaking the open file handle).
4. ClickHouse's OwnRunnableForChannel async logging thread enters an infinite tight loop:
   - Attempts to write a log message → Poco::RotateBySizeStrategy::mustRotate() fails with File access error
   - That failure generates another log message → loop repeats endlessly at full speed
5. The process silently accumulates CPU. In the reported case the container had been spinning for 20+ days (28 days of accumulated CPU
   time) with no visible symptoms other than high CPU.

The log file gets frozen at exactly 8192 bytes and stops updating — observable on the host:
-rw-r----- 1 clickhouse clickhouse 8.0K Mar 5 23:27 clickhouse-server.log

docker logs shows nothing but a wall of repeated identical stack traces:
Cannot log message in OwnAsyncSplitChannel channel: Poco::Exception. Code: 1000, e.code() = 0,
File access error: /var/log/clickhouse-server/clickhouse-server.log
0. Poco::RotateBySizeStrategy::mustRotate(Poco::LogFile*)
1. Poco::FileChannel::log(Poco::Message const&)
2. DB::OwnFormattingChannel::logExtended(...)
3. DB::OwnRunnableForChannel::run()

Fix

Explicitly redirect file log paths to /dev/null in clickhouse-config.xml:

warning true + /dev/null + /dev/null

This also makes the op-ch-logs volume mount unnecessary.

Impact

• Wasted CPU on every self-hosted deployment once the log file access breaks
• No error visible to the user — container appears healthy in docker ps
• Likely affects all self-hosted deployments running for more than a few days

[lindesvard]

Hopefully fixed 🤞