The lock in RegionPersister::doPersist slow down raft apply when meets jitter

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

tiflash/dbms/src/Storages/KVStore/MultiRaft/RegionPersister.cpp

Lines 99 to 125 in 5980908

	void RegionPersister::doPersist(
	RegionCacheWriteElement & region_write_buffer,
	const RegionTaskLock & region_task_lock,
	const Region & region)
	{
	auto & [region_id, buffer, region_size, applied_index] = region_write_buffer;
	std::lock_guard lock(mutex);
	auto entry = page_reader->getPageEntry(region_id);
	if (entry.isValid() && entry.tag > applied_index)
	return;
	if (region.isPendingRemove())
	{
	LOG_DEBUG(log, "no need to persist {} because of pending remove", region.toString(false));
	return;
	}
	auto read_buf = buffer.tryGetReadBuffer();
	RUNTIME_CHECK_MSG(read_buf != nullptr, "failed to gen buffer for {}", region.toString(true));
	DB::WriteBatchWrapper wb{run_mode, getWriteBatchPrefix()};
	wb.putPage(region_id, applied_index, read_buf, region_size);
	page_writer->write(std::move(wb), global_context.getWriteLimiter());
	region.updateLastCompactLogApplied(region_task_lock);
	}

RegionPersister::doPersist acquires a lock and calls PageStorage::write with the lock. This means when we're persisting Region meta, all other threads can not do the persistence for other Region meta. Which is not reasonable.

4. What is your TiFlash version? (Required)

4.0~master