IngestSST concurrency is limited to 16 under v8.5

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Deploy a tidb cluster with tiflash, using 32c256G machine for tiflash instance
2. Run br restore for a large dataset
3. After the "download sst stage" is finished, tiflash enter the "ingest sst stage", upgrade the cluster version to v8.5.4 including tiflash

2. What did you expect to see? (Required)

tiflash should use 32 concurrency at max for ingest sst after upgraded to v8.5.4

3. What did you see instead (Required)

tiflash only use 16 concurrency at max for ingest sst after upgraded to v8.5.4

4. What is your TiFlash version? (Required)

v8.5.4

[JaySon-Huang]

In a env that use v7.5.6 running br restore. during the ingest sst stage after 17:00, tiflash run ingest sst with 32 concurrency at max.

---

In this env, between 04:54~05:09, tiflash run ingest sst with 32 concurrency at max. But after upgraded to v8.5.4, tiflash only run ingest sst with 16 concurrency at max. Metrics and logging show that the concurrency should be limited by PreHandlingTrace.

[2025/12/11 02:21:59.723 +00:00] [INFO] [PrehandleSnapshot.cpp:128] ["Prehandle resource acquired after 2.018 seconds, region_id=19480292 parallel=1"] [source=PreHandlingTrace] [thread_id=7818]
[2025/12/11 02:21:59.988 +00:00] [INFO] [PrehandleSnapshot.cpp:128] ["Prehandle resource acquired after 1.593 seconds, region_id=19482446 parallel=1"] [source=PreHandlingTrace] [thread_id=7812]
[2025/12/11 02:22:00.029 +00:00] [INFO] [PrehandleSnapshot.cpp:128] ["Prehandle resource acquired after 1.452 seconds, region_id=19506459 parallel=1"] [source=PreHandlingTrace] [thread_id=7815]
[2025/12/11 02:22:00.071 +00:00] [INFO] [PrehandleSnapshot.cpp:128] ["Prehandle resource acquired after 0.493 seconds, region_id=19506557 parallel=1"] [source=PreHandlingTrace] [thread_id=7805]

[JaySon-Huang]

Root casue

The concurrency is controlled by PreHandlingTrace::waitForSubtaskResources

tiflash/dbms/src/Storages/KVStore/MultiRaft/PrehandleSnapshot.cpp

Lines 82 to 130 in e7d70a3

	void PreHandlingTrace::waitForSubtaskResources(uint64_t region_id, size_t parallel, size_t parallel_subtask_limit)
	{
	{
	auto current = ongoing_prehandle_subtask_count.load();
	if (current + parallel <= parallel_subtask_limit
	&& ongoing_prehandle_subtask_count.compare_exchange_weak(current, current + parallel))
	{
	LOG_DEBUG(
	log,
	"Prehandle resource meet, limit={} current={} region_id={}",
	parallel_subtask_limit,
	ongoing_prehandle_subtask_count.load(),
	region_id);
	return;
	}
	}
	Stopwatch watch;
	LOG_DEBUG(
	log,
	"Prehandle resource wait begin, limit={} current={} parallel={} region_id={}",
	parallel_subtask_limit,
	ongoing_prehandle_subtask_count.load(),
	parallel,
	region_id);
	CurrentMetrics::add(CurrentMetrics::RaftNumWaitedParallelPrehandlingTasks);
	while (true)
	{
	std::unique_lock<std::mutex> cpu_resource_lock{cpu_resource_mut};
	cpu_resource_cv.wait(cpu_resource_lock, [&]() {
	return ongoing_prehandle_subtask_count.load() + parallel <= parallel_subtask_limit;
	});
	auto current = ongoing_prehandle_subtask_count.load();
	if (current + parallel <= parallel_subtask_limit
	&& ongoing_prehandle_subtask_count.compare_exchange_weak(current, current + parallel))
	{
	break;
	}
	}
	GET_METRIC(tiflash_raft_command_duration_seconds, type_apply_snapshot_predecode_parallel_wait)
	.Observe(watch.elapsedSeconds());
	LOG_INFO(
	log,
	"Prehandle resource acquired after {:.3f} seconds, region_id={} parallel={}",
	watch.elapsedSeconds(),
	region_id,
	parallel);
	CurrentMetrics::sub(CurrentMetrics::RaftNumWaitedParallelPrehandlingTasks);
	}

The concurrency is controlled by KVStore::getMaxParallelPrehandleSize, which use the "raftstore.snap-handle-pool-size". And in order to avoid too many apply snapshot affecting the raftlog write speed, the max concurrency of apply snapshot is limited to 16 even if the cpu size is 32.

tiflash/dbms/src/Storages/KVStore/MultiRaft/PrehandleSnapshot.cpp

Lines 304 to 318 in e7d70a3

	size_t KVStore::getMaxParallelPrehandleSize() const
	{
	const auto & proxy_config = getProxyConfigSummay();
	size_t total_concurrency = 0;
	if (proxy_config.valid)
	{
	total_concurrency = proxy_config.snap_handle_pool_size;
	}
	else
	{
	auto cpu_num = std::thread::hardware_concurrency();
	total_concurrency = static_cast<size_t>(std::clamp(cpu_num * 0.7, 2.0, 16.0));
	}
	return total_concurrency;
	}

Affect versions

Introduced by #8184

affects LTS versions:

• v8.1.0~v8.1.2
• v8.5.0~v8.5.4

Workaround under affect versions

If tiflash instance is deployed on a machine with more than 16 logical cores, change the tiflash-proxy (tiflash-learner.toml) config "raftstore.snap-handle-pool-size" to number of cpu could increase the BR restore speed during ingest sst.