wait_mutex duration reach 400ms when the number of total SST file reach 40 thousands

[zhangjinpeng87]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

Expected behavior

Write duration and get duration should not be impacted so much by compaction.

Actual behavior

The compaction impacts the Get duration and the Write duration a lot when the number of total SST files is about 40K. After we open the perf_context feature, we found the wait_mutex duration is very large, this is the root cause for the high latency.

Steps to reproduce the behavior

• Set target_file_size = 8MB, and insert data to let the number of the total SST files reach about 40K
• Run read and write workload

[wolfkdy]

I wonder why DBImpl::Get duration will be affected by db_impl._mutex，As DBImpl::Get only takes superversion, which is mostly lock-free. unless superversion is changed when mem/imm flushed or new ssts generated.
Did you take a deep look at why _mutex holds for so long or who holds it for that long?
AFAIK, FindObsoleteFiles may hold mutex for a long time. which is an known issue.

[ramvadiv]

@zhangjinpeng1987 - this scenario should have improved after #7412

Can you pls retest and provide an update?