Add memtable MultiGet finger search optimization

[anand1976]

Add memtable batch lookup optimization with finger search

Summary

Optimize memtable MultiGet by using a finger search on the skip list. After finding key[i], the search path (Splice) is retained as a "finger" for key[i+1]. The next search walks up the finger until the forward pointer overshoots, then descends -- costing O(log d) where d is the distance between consecutive sorted keys, rather than O(log N) from the head each time.

Controlled by the new memtable_batch_lookup_optimization column family option (default: false).

Changes

• Add FindGreaterOrEqualWithFinger() and MultiGet() to InlineSkipList with optional paranoid validation support (key ordering checks and per-key checksum verification via default parameters)
• Add virtual MultiGet() to MemTableRep, override in SkipListRep
• Add memtable_batch_lookup_optimization CF option
• Integrate finger search into MemTable::MultiGet -- sorts keys, performs batched finger search, then unscrambles results
• Add db_bench, db_stress, and crash test support
• Add unit tests for InlineSkipList::MultiGet (5 tests) and integration tests at the DB level (25 BatchLookup tests), including paranoid validation tests

Benchmark Results

Setup: 2M keys in memtable, release build (DEBUG_LEVEL=0).

Single-threaded multireadrandom

Batch Size	Baseline	BatchOpt	vs Base
2	363,593	376,562	+3.6%
8	385,204	386,082	+0.2%
32	360,339	375,105	+4.1%
64	352,696	378,497	+7.3%

Multithreaded multireadrandom (batch_size=64)

Threads	Baseline	BatchOpt	vs Base
4	171,356	185,633	+8.3%
8	161,478	171,392	+6.1%

multireadwhilewriting (batch_size=64)

Threads	Baseline	BatchOpt	vs Base
1	163,938	175,068	+6.8%
4	109,698	120,790	+10.1%
8	116,623	123,658	+6.0%

No regression at any batch size or thread count. The finger is stack-allocated per MultiGet call, so there is no shared state or cache-line contention between threads. Concurrent writes don't invalidate the finger's bracket since it only reads via acquire-load Next() pointers.

Small Batch Size Regression Check: memtable_batch_lookup_optimization

All values are ops/sec. Measured with db_bench (release build).
2M keys in memtable, value_size=100, duration=30s per run.

Multithreaded multireadrandom — small batch sizes

Threads | Batch Size | Baseline | BatchOpt | Change
4 | 2 | 532,302 | 540,682 | +1.6%
8 | 2 | 1,044,046 | 1,046,920 | +0.3%
4 | 8 | 633,733 | 630,039 | -0.6%
8 | 8 | 1,260,818 | 1,241,792 | -1.5%

multireadwhilewriting — small batch sizes

Threads | Batch Size | Baseline | BatchOpt | Change
1 | 2 | 107,284 | 105,217 | -1.9%
4 | 2 | 428,508 | 423,747 | -1.1%
8 | 2 | 854,654 | 864,923 | +1.2%
1 | 8 | 114,788 | 118,496 | +3.2%
4 | 8 | 467,995 | 473,437 | +1.2%
8 | 8 | 935,636 | 960,791 | +2.7%

No regression at small batch sizes. Variations at batch_size=2 are
within noise (~1-2%). At batch_size=8, modest positive trend (+1-3%
in read-while-writing).

Test Plan

• make check -- all tests pass
• ASSERT_STATUS_CHECKED=1 make check -- all tests pass
• 2-hour db_crashtest.py blackbox with --memtable_batch_lookup_optimization=1 -- passed
• 2-hour db_crashtest.py blackbox with --memtable_batch_lookup_optimization=1 --paranoid_memory_checks=1 -- passed

[meta-codesync]

@anand1976 has imported this pull request. If you are a Meta employee, you can view this in D95813786.

[github-actions]

✅ clang-tidy: No findings on changed lines

Completed in 361.0s.

[meta-codesync]

@anand1976 has imported this pull request. If you are a Meta employee, you can view this in D95813786.

[xingbowang]

I don't see test coverage on the skiplist height growing handling. I guess it would involve with concurrent read/write. Could we some add test to validate read after write consistency with concurrent read/write operation with some randomly generated keys?

Some other Findings from AI:

1. Finger not updated after callback loop walk-forward

After the callback loop walks forward through level-0 nodes, finger->next_[0] still points to the search result rather than the last visited node. On the next FindGreaterOrEqualWithFinger call, the walk-up starts from this stale position, ascending more levels than necessary. For the typical case (callback returns false after 1 entry), this is negligible. For merge operations consuming multiple entries per key, the next lookup degrades toward O(log N).

2. key_validation_callback not invoked during callback loop

In InlineSkipList::MultiGet, the callback loop at level 0 (walking forward through matching entries) only checks key ordering but does not invoke key_validation_callback:

inlineskiplist.h
Lines 1330-1337
for (; node != nullptr && callback_func(callback_args[i], node->Key());) {
Node* prev_node = node;
node = node->Next(0);
if (detect_key_out_of_order && prev_node != head_ && node != nullptr &&
compare_(prev_node->Key(), node->Key()) >= 0) {
return Corruption(prev_node, node, allow_data_in_errors);
}
}
During the search phase, FindSpliceForLevelValidated calls key_validation_callback on every traversed node. But nodes visited by the callback loop escape checksum/corruption validation. If paranoid_memory_checks is enabled, this is a gap compared to the per-key path (which validates every node via GetAndValidate → SeekAndValidate + NextAndValidate).

[xingbowang]

I would also like to see benchmark with smaller batch size to make sure there is no regression on smaller batch size.

[xingbowang]

The improvement is mostly from reducing cache miss by reusing the same key from last search. With high batch, I felt we could do parallel lookup with multiple prefetch stages to further hide the latency. I claude coded it for insert and seen up to 4X throughput improvement. I guess it would work well for read as well. We can do that later. Feel free to explore it, if you have passion pursuing that direction.
main...xingbowang:rocksdb:experiment_skiplist_perf_up

[xingbowang]

This test runs one writer on the main thread and four readers on background threads, all operating on the same skip list simultaneously. This is good improvement. However, this is slightly off my goal. My goal is to validate read after write consistency. I would suggest to

1. generate a long sequence of unique keys for insertion.
2. split the sequence into different chunks.
3. spin up different threads to insert different chunks of keys into the skiplist, and then immediately query a batch of random key including the key just inserted. Validate the key just inserted is visible.
   Since each key is unique, we don't need worry about key override, the key read from which it just inserted should be easy to validate its value to validate read after write consistency.
   To push this further, maybe there is some way to also have it read recent keys inserted by other threads through some other synchronization mechanism, although it is a bit more complicate. Maybe AI could help.

[pdillinger]

I felt we could do parallel lookup with multiple prefetch stages to further hide the latency.

My inclination would be to binary search + partition the (sub)batch at each height of the skip list into sub-batches to send further down the skip list. This approach would "visit" each skip list node only once (per batch). Nodes pushed on the BFS queue to visit could be prefetched to be hot in cache. (A DFS stack might not hide latency as well.)

[pdillinger]

Thanks Xingbo for first round reviews. Overall looks good to me. I trust the crash test to find bugs here if there are any.

[meta-codesync]

@anand1976 has imported this pull request. If you are a Meta employee, you can view this in D95813786.

[anand1976]

Thanks Xingbo and Peter for the review.

My inclination would be to binary search + partition the (sub)batch at each height of the skip list into sub-batches to send further down the skip list. This approach would "visit" each skip list node only once (per batch). Nodes pushed on the BFS queue to visit could be prefetched to be hot in cache. (A DFS stack might not hide latency as well.)

That's a nice idea. To Xingbo's point, hiding cache latency would give a significant boost. I'll explore this when I have some time. The BFS approach could also be applied to batched index lookups in SSTs (though impact may be less due to smaller batches).

[meta-codesync]

@anand1976 merged this pull request in c66c142.