[Dynamic Regions] Tablet GC tasks are accumulated overtime and incorrect store size reported

[tonyxuqqi]

Bug Report

What version of TiKV are you using?

nightly

What operating system and CPU are you using?

Steps to reproduce

tpcc 25k warehouse prepare

What did you expect?

Tablet GC could run timely

What did happened?

Pending tablet gc tasks are accumulated overtime and the store size exceeds the capacity.

Grafana: http://54.189.73.225:33762/d/RDVQiEzZz/tpcc-tikv-details?orgId=1&from=1675200847691&to=1675204064867&var-k8s_cluster=&var-tidb_cluster=&var-db=All&var-command=All&var-instance=172.31.15.134:20180&var-titan_db=All

[tonyxuqqi]

There're three tikv nodes and one of them is normal, but the other two are abnormal----basically there's no "destroy tablet" or "failed to check whether the tablet path is locked" in the two abnormal tikv logs.

[tonyxuqqi]

The same stack is found in the two abnormal nodes. I think we may run into some dead lock .

#0  0x00007f97b2d0f377 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055b967f813cd in rocksdb::port::CondVar::Wait() ()
#2  0x000055b967e7ec15 in rocksdb::InstrumentedCondVar::Wait() ()
#3  0x000055b967d3f1cf in rocksdb::DBImpl::RunManualCompaction(rocksdb::ColumnFamilyData*, int, int, rocksdb::CompactRangeOptions const&, rocksdb::Slice const*, rocksdb::Slice const*, bool, bool, unsigned long) ()
#4  0x000055b967d3fede in rocksdb::DBImpl::CompactRangeInternal(rocksdb::CompactRangeOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const*, rocksdb::Slice const*) ()
#5  0x000055b967d403e8 in rocksdb::DBImpl::CompactRange(rocksdb::CompactRangeOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const*, rocksdb::Slice const*) ()
#6  0x000055b967c4444e in crocksdb_compact_range_cf_opt ()
#7  0x000055b964e0c326 in engine_rocks::compact::<impl engine_traits::compact::CompactExt for engine_rocks::engine::RocksEngine>::compact_range_cf ()
#8  0x000055b9660907e0 in engine_traits::compact::CompactExt::compact_range ()
#9  0x000055b965d117c5 in raftstore_v2::worker::tablet_gc::Runner<EK>::trim ()
#10 0x000055b965cf1f4b in <raftstore_v2::worker::tablet_gc::Runner<EK> as tikv_util::worker::pool::Runnable>::run ()
#11 0x000055b966cd9fb1 in <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll ()
#12 0x000055b967a1cbab in <yatp::task::future::Runner as yatp::pool::runner::Runner>::handle ()
#13 0x000055b9662dd8fa in <tikv_util::yatp_pool::YatpPoolRunner<T> as yatp::pool::runner::Runner>::handle ()
#14 0x000055b966786eae in yatp::pool::worker::WorkerThread<T,R>::run ()
#15 0x000055b9664ce805 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#16 0x000055b9660ff2dc in core::ops::function::FnOnce::call_once{{vtable-shim}} ()
#17 0x000055b967180735 in std::sys::unix::thread::Thread::new::thread_start ()
#18 0x00007f97b2d0944b in start_thread () from /lib64/libpthread.so.0
#19 0x00007f97b270452f in clone () from /lib64/libc.so.6