ClickHouse leaks urandom fds during startup, crashes, with "Too many open files"

[vvenkateshgupta]

Describe your question

Title

Environment

ClickHouse leaks urandom fds during startup, crashes, with "Too many open files"

• Image:

bitnami/clickhouse (bitnamilegacy), tag

25.2.1-debian-12-r0

• Kubernetes, amd64, cgroup v2, StatefulSet, 3 replicas, gp3 PVC each
• Pod: requests/limits. cpu: 6 (host has 8 cores)

Summary

Shortly after clickhouse-server starts (10s-7min in), open fd count grows ~150-170/sec. Sampling /proc/<pid>/fd shows growth is nearly 100% urandom (no socket/pipe growth). Eventually crashes:

libc++abi: terminating due to uncaught exception of type std::_1::system_error: random_device failed to open /dev/urandom: Too many open files

Intermittent — one replica can run 25+ min fine while a sibling crashes almost every restart.

Sample fd growth

  |t+10s 1623 fds (1600 urandom)
-- | -- | --
  | t+26s 4262 fds (4242 urandom)
  | t+52s 8622 fds (8604 urandom)
  |t+95s 15586 fds (15572 urandom)
  | t+165s 26971 fds
  | t+282s crash |  

Ruled out

• ZooKeeper enabled/disabled - no change

• THREAD_FUZZER_* - unset

• send_crash_reports - disabled

• Corrupted/stale PVC — reproduces on fresh empty PVC too

• Node-specific - reproduces on multiple nodes

• CPU/nproc thread-pool auto-sizing - pinned background_pool_size , background_schedule_pool_size, background_fetches_pool_size, background_common_pool_size, background_move_pool_size background_message_broker_schedule_pool_size,
background_distributed_schedule_pool_size, background_buffer_flush_schedule_pool_size, max_thread_pool_size - leak identical

• Replica ordinal - reproduces on 0, 1, 2

Config in use

<send_crash_reports>false</send_crash_reports>
<max_open_files>10000</max_open_files>
<background_pool_size>4</background_pool_size>
<background_schedule_pool_size>4</background_schedule_pool_size> <background_fetches_pool_size>2<|background_fetches_pool_size> <background_common_pool_size>2</background_common_pool_size>
<background_move_pool_size>2</background_move_pool_size>
<max_thread_pool_size>2000</max_thread_pool_size>

Expected

clickhouse-server should not continuously leak urandom fds.

Actual
Fd count grows unbounded until crash via uncaught
std:: random_device exception.

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

This appears to be a ClickHouse-specific bug rather than a Langfuse issue. The urandom file descriptor leak you're experiencing is happening within ClickHouse itself, not in Langfuse's integration with it.

A few points based on your setup:

Support level for your deployment: The Bitnami ClickHouse Helm Chart is community-supported, not officially supported by Langfuse. (1) For an officially supported self-managed setup on Kubernetes, consider the ClickHouse Kubernetes Operator instead. (1)

Recommended actions:

1. Report to ClickHouse: Since this is a file descriptor leak in std::random_device within ClickHouse itself, this should be reported to the ClickHouse GitHub repository rather than Langfuse.

2. Try a different ClickHouse version: Langfuse v3 supports ClickHouse versions >= 24.3. (1) You're using 25.2.1 — try testing with a different version to see if this is a regression in that specific release.

3. Consider ClickHouse Cloud/BYOC: For larger deployments, ClickHouse Cloud or BYOC is recommended as they provide cloud-native scaling primitives and are officially supported. (1)

4. Try the official ClickHouse Kubernetes Operator: The official ClickHouse Kubernetes Operator automates deployment and management of self-managed ClickHouse clusters on Kubernetes and is officially supported by Langfuse. (1)

For Docker setups, the example configuration includes setting ulimits: (1)

docker run --name clickhouse-server \
  -e CLICKHOUSE_DB=default \
  -e CLICKHOUSE_USER=clickhouse \
  -e CLICKHOUSE_PASSWORD=clickhouse \
  -d --ulimit nofile=262144:262144 \
  -p 8123:8123 \
  -p 9000:9000 \
  -p 9009:9009 \
  clickhouse/clickhouse-server

(1)

However, since your issue is a leak (not just hitting a limit), raising ulimits would only delay the crash, not fix the root cause.

📚 Sources:

1. ClickHouse

---

Have another question? Just tag @inkeep.