BE remote scanner thread pool (rs_normal) never shrinks — thread count grows to 27k+ and BE freezes

[tracy1014-hub]

BE remote scanner thread pool (rs_normal) never shrinks — thread count grows to 27k+ and BE freezes

Environment

• Doris version: 4.1.1-rc01
• Cluster size: 4 BE + 3 FE, deployed via DorisOperator on Kubernetes
• BE image: official apache/doris:doris-4.1.1-rc01-be (x86_64)
• Workload: BE queries external tables backed by Paimon on S3 (Doris × Paimon catalog), mixed with normal internal OLAP queries
• BE resources: cpu 8 core, memory limit 40G (cgroup), mem_limit=32G

Symptom

BE pod RSS climbs monotonically from ~3G to 6–7G over ~4h, then either:

• keeps growing until the pod is OOM-ish and gets killed by cgroup, or
• at ~27k threads the BE process freezes: it stops responding to FE heartbeats, SHOW BACKENDS shows Alive=false with SocketTimeoutException: Read timed out, CPU drops to ~1m, readiness probe times out. Not OOMKilled — internal lock/resource exhaustion.

The only recovery is kubectl delete pod (restart the BE process).

Misleading signals

All conventional memory metrics stay green, which made this look like a "memory leak" for a while:

• JVM heap used — normal GC sawtooth, returns to baseline
• jemalloc resident — stable at ~2G
• jvm non-heap — stable at ~130M
• mem_tracker — stable

Root cause

/proc/$BE_PID/status shows Threads=16000–27000, of which ~90% are rs_normal [worker] threads in S (sleeping) state. Each thread has a 512K stack → 7–12GB just for thread stacks, which is what pushes RSS up.

Thread census command:

BE_PID=$(pgrep -f doris_be)
ls /proc/$BE_PID/task/ | while read t; do cat /proc/$BE_PID/task/$t/comm; done \
  | sort | uniq -c | sort -rn | head

Typical output on a frozen BE:

27012 rs_normal
   48 olap_scan
   24 brpc
   ...

The rs_normal pool (remote scanner, used for querying external catalogs like Paimon/Hive/Iceberg) only grows, never shrinks. Idle keep-alive eviction either isn't implemented or isn't effective in 4.1.1-rc01.

Configs I tried to cap the pool (all ineffective on rc01)

#	Where	Setting	Result
1	be.conf	doris_scanner_thread_pool_thread_num=256	Only caps ls_normal (local scanner). rs_normal still linear-grows.
2	be.conf	doris_max_remote_scanner_thread_pool_thread_num=256	No effect. rs_normal keeps growing past 256.
3	FE	ALTER WORKLOAD GROUP normal PROPERTIES ('max_remote_scan_thread_num'='256', 'min_remote_scan_thread_num'='48');	No effect.

Related defaults on this BE:

• doris_remote_scanner_thread_pool_thread_num=48 (initial)
• doris_max_remote_scanner_thread_pool_thread_num=-1 (unlimited)

Current workaround

CronJob that rotates one BE pod every 4h (kubectl delete pod on the oldest ready BE, with guards: ready BE >= 3 and pod age > 6h). This keeps the leak from ever reaching the freeze threshold, but it's purely symptomatic.

YAML for the workaround is available if useful.

Expected behavior

At least one of these should actually work on rc01:

1. doris_max_remote_scanner_thread_pool_thread_num=N should hard-cap rs_normal size, excess idle threads should be reaped when the cap is hit and load drops.
2. Idle rs_normal workers should be evicted after a keep-alive timeout (like ls_normal does), so the pool shrinks back to doris_remote_scanner_thread_pool_thread_num when external-scan load stops.
3. ALTER WORKLOAD GROUP … max_remote_scan_thread_num=N should bound concurrent remote scans per workload group — currently appears to be ignored or not enforced against the rs_normal pool.

Reproduction

1. Fresh BE 4.1.1-rc01 with a Paimon catalog on S3.
2. Run a sustained mixed workload that issues external scans against Paimon (a few QPS is enough; the pool grows even under modest load).
3. Within 4–8h, Threads= in /proc/$BE_PID/status crosses 20k and RSS climbs correspondingly.
4. Eventually (usually 24k–28k threads) the BE stops answering FE heartbeats and has to be killed externally.

Happy to provide thread dumps, be.conf, SHOW BACKENDS/SHOW WORKLOAD GROUPS output, or a heap profile if it helps. Just let me know what's most useful.

Question

Is this a known issue in 4.1.1-rc01? Is there a fix in a later 4.1.x or 4.2.x release? If a fix exists, I'd like to upgrade instead of running the rotator CronJob forever.