Automatic MergeTree merge remains at progress=1 indefinitely and background merge scheduling stops

[staryuM]

Company or project name

No response

Describe what's wrong

On a single-node ClickHouse server, automatic MergeTree background merge
scheduling can become permanently stalled. A very small merge reaches
progress = 1 with all rows read and written, but it remains in
system.merges for hours and does not disappear. While this state persists,
many MergeTree tables accumulate active parts and inserts are delayed or
rejected with TOO_MANY_PARTS.

This is not an OPTIMIZE ... FINAL operation. It is an automatic background
merge. There was no manual SYSTEM STOP MERGES or SYSTEM START MERGES around
the incident.

Does it reproduce on the most recent release?

No

How to reproduce

ClickHouse version: 26.6.1.1193-stable.

Interface: native client / HTTP ingestion. The failure is in automatic server-side background merge scheduling, not in a client query.

Non-default settings:

background_pool_size = 16
background_merges_mutations_concurrency_ratio = 2

Representative affected table, with identifiers anonymized:

CREATE TABLE default.scheduler_hosts
(
    `cluster_name` String,
    `sampling_time` DateTime,
    `host_name` String,
    `host_status` String,
    `jl_u` Nullable(Decimal(10, 0)),
    `max_slot` Nullable(Decimal(10, 0)),
    `njobs` Nullable(Decimal(10, 0)),
    `run_slot` Nullable(Decimal(10, 0)),
    `sys_susp_slot` Nullable(Decimal(10, 0)),
    `user_susp_slot` Nullable(Decimal(10, 0)),
    `rsv_slot` Nullable(Decimal(10, 0)),
    `interval_period` Nullable(Decimal(13, 0)),
    `sampling_time_long` Nullable(Decimal(13, 0)),
    `halt_num` Nullable(Decimal(13, 0)),
    `gpus` Nullable(Decimal(13, 0)),
    `gpu_slices` Nullable(Decimal(13, 0)),
    `gpu_occ_rate` Nullable(Decimal(15, 4)) DEFAULT -1,
    `gpu_num` Nullable(Decimal(15, 0)),
    `gpu_used_num` Nullable(Decimal(15, 0))
)
ENGINE = MergeTree
PARTITION BY toDate(sampling_time)
ORDER BY (cluster_name, host_name, sampling_time)
SETTINGS index_granularity = 8192, parts_to_delay_insert = 200, parts_to_throw_insert = 500;

Observed triggering workload, not yet a deterministic minimal reproducer:

1. Use one ClickHouse server and the table above, together with several other MergeTree tables receiving small batches once per minute.
2. Continuously insert scheduler-host sampling data into the daily partition.
3. Monitor system.merges and active parts in system.parts.
4. After an undetermined runtime, an automatic merge can remain at progress = 1 although rows_read = rows_written; active parts then accumulate across tables.

We can run a proposed minimal reproducer, enable a recommended diagnostic setting, or test a patch.

Expected behavior

Once an automatic merge reports progress = 1 and has written all output rows, it should finish and release the merge executor so background merge scheduling continues. Small, minute-level inserts should not accumulate parts indefinitely because one completed merge remains present.

Error message and/or stacktrace

system.merges snapshot:

database: default
table: scheduler_hosts
partition_id: 20260717
result_part_name: 20260717_16810_17066_64
num_parts: 5
total_size_bytes_compressed: approximately 13 KiB
rows_read: 257
rows_written: 257
progress: 1
elapsed: 19405 seconds
thread_id: 20520

The target merge thread (LWP 20520) from pstack:

pthread_mutex_lock
std::mutex::lock
DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction

Several background scheduler threads concurrently showed this path:

DB::LockGuardWithStopWatch
DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::trySchedule
DB::BackgroundJobsAssignee::scheduleMergeMutateTask
DB::StorageMergeTree::scheduleDataProcessingJob

The full pstack, system.merges, system.parts, system.events, and server log excerpts are attached.

Related issues and pull requests

No response

Additional context

OS: CentOS/RHEL 7 compatible, Linux kernel 3.10.0-123.el7.x86_64
Deployment: single ClickHouse server, no replication
Resources: 4 vCPU, 7.6 GiB RAM

Impact during this incident:

• A separate high-frequency state table reached about 501 active parts.
• Inserts began to be delayed and rejected: DelayedInserts = 2038, RejectedInserts = 146; errors included TOO_MANY_PARTS.
• CPU, memory, and disk were not saturated.
• SQL-level recovery attempts did not unblock the executor. A server restart recovered previous incidents, but this incident was intentionally left running while evidence was collected.

Questions:

1. Is this a known issue in 26.6.1.1193-stable or fixed in a later stable/LTS release?
2. Does the threadFunction and trySchedule mutex contention indicate a known deadlock/stall condition in MergeTreeBackgroundExecutor?
3. What additional diagnostic output is useful before restart, and is there a supported online recovery action for this executor state?