Bound MaintainPoolSize loops by the LRU list and abort on corrupt pool accounting

[yxshwanth]

Summary

Fixes #15128, where an LRU block cache shared across ~128 DBs wedged every one of them because a thread was spinning in LRUCacheShard::MaintainPoolSize() holding the shard mutex.

Both loops there terminated only on a size_t usage counter while walking a circular list. The invariant tying those counters to the list is checked only by assert, which is compiled out in release. So a counter that disagreed with the list sent the walk past its own region and around the list, subtracting charges never counted into that pool. The counter underflowed, and since low_pri_pool_ratio defaults to 0 the capacity is 0, leaving usage > capacity true forever with mutex_ held.

This bounds each loop by the list it walks. The low-pri loop takes two bounds: the pool it drains is the entries in (lru_bottom_pri_, lru_low_pri_], and separately it stops at the end of the list, which matters once the markers are out of order as they were in the report. Without the second one the walk runs off the end and reads the sentinel's uninitialized total_charge. If a loop runs out of entries while its counter still claims the pool is over capacity, that counter no longer describes the shard, so it prints the accounting state and aborts. @xingbowang asked for a crash with a core dump rather than silent recovery, which is what this does. port::Crash() is not usable for it because SIGTERM does not dump core, so it calls std::abort().

Rather than detect a loop that has run too long, it detects the state that makes the loop unable to end. That fires on the first bad iteration and has nothing to tune.

I could not find what makes the counters diverge in the first place. 40M randomized operations against a shard, checking both counters against the list after each one, produced no divergence, and I found no unsynchronized path: LRUHandle appears only under cache/, every public LRUCacheShard method takes mutex_, and NotifyEvicted frees each handle once after unlinking. So this guards the amplification, not the trigger. It turns a permanent process-wide hang into a crash with the evidence attached.

Test plan

make check passes, other than failures that reproduce on unmodified HEAD (see below).

A standalone program against a release librocksdb.a, differing only in the loop bounds, corrupts low_pri_pool_usage_ and calls MaintainPoolSize(). Unbounded it runs at 99.4% CPU until killed, with all 1699 stack samples in MaintainPoolSize(). Bounded:

rocksdb: corrupt low-pri pool accounting in LRUCacheShard 0x74101c000; aborting.
capacity=10 usage=5 lru_usage=5 high_pri_pool_usage=5 high_pri_pool_capacity=5
low_pri_pool_usage=18446744073709551615 low_pri_pool_capacity=0

Before trusting an abort in production code I ran 40M randomized operations with it in place, across every ratio combination, both metadata charge policies, and strict and non-strict. No spurious abort. None of the bounds are reachable while the counters agree with the list, so consistent shards are unaffected.

Two smaller things worth flagging:

ValidateLRUList checked list order and per-entry flags but never the usage counters, which are what these loops run on. The third commit adds a stress test that checks them after every operation. It also asserts on average LRU list length, because an earlier version passed while averaging 0.3 entries per list and exercised almost none of the pool logic.

The death test only exercises the guard indirectly in debug builds: without the bounds the walk trips assert(lru_bottom_pri_->InLowPriPool()) long before the counter wraps. The spin is specific to release builds, and RocksDB's test infrastructure cannot build in release at all, since SyncPoint compiles out under NDEBUG and db_test_util.cc fails. That is why the release evidence above needed a standalone harness, and it may be worth a separate look: the checks for this class of bug run only in builds where the bug cannot occur.

On macOS, make check also fails two DBSSTTest cases, three EventListenerTest cases, and rocksdb_dump_test, all of which reproduce on unmodified HEAD. $TMPDIR ends in / there, so TEST_TMPDIR contains a doubled slash and SstFileManager tracks each file under two spellings; the same tests pass with a TEST_TMPDIR that has no trailing slash. Unrelated to this change, happy to file separately.