Tracking Issue: In-memory Engine

[SpadeA-Tang]

RFC: tikv/rfcs#111

Summary:
In-memory engine works as a range cache, selectively caching certain ranges to enhance read performance.

Tasks:

• Integrate In-memory Engine with TiKV
  • In-memory Engine: defining hybrid engine #16114
  • In-memory Engine: integrate hybrid engine with TiKV #16132
  • In-memory Engine: refactor from region based to range based #16383
  • In-memory Engine: enable HybridEngine only for storage and coprocessor modules #17181
• Implement Write
  • In-Memory Engine: write path #16323
  • In-memory Engine: optimize write batch to selectively cache writes #16619
• Implement Read
  • In-memory Engine: implement read flow -- memory engine part #16163
  • In-memory Engine: implement read flow -- hybrid engine part #16181
  • In-memory Engine: implement read flow -- support sequence number #16193
  • In-memory Engine: implement read flow -- prefix read #16237
• Implement garbage collection
  • In-memory Engine: implement garbage collection -- runner part #16238
  • In-memory Engine: implement garbage collection -- worker part #16461
• Implement SST ingestion (skipped)
• Implement Load and Evict
  • In-memory Engine: implement range load #16500
  • [in-memory Engine] Provide a manual way to cache or evict a specific region #16509
• Implement memory controller
  • In-memory Engine: introduce memory controller #16671
• Implement A High Performant Skiplist
  • In-memory Engine: use a more crossbeam-like skiplist #16652
  • In-memory Engine: merge tikv/skiplist-rs repo #17166
• Integration tests framework
  • In-memory Engine: integrate HybridEngine with integration test framework #16186
  • In-memory Engine: make integration test work #16474
  • In-memory engine: make integration tests reusable for hybrid engine #16725
• Observability and diagnosibility
  • In-memory-engine: add metrics for acquire snapshot and gc #16696
  • In-memory engine: collector iteration related metrics #16735
  • In-memory engine: add metrics for memory usage #16822
  • In-memory engine: implement core local array and collect read flow statistics #16945
  • In-memory engine: add metrics for range count #16971
  • In-memory engine: add metrics for different types of iteration operations #16997
  • In-memory engine: add seek duration metrics #17160
  • In-memory engine: add metrics for prepare for write duration #17161

Enhancement

• In-memory engine: RangeManger is designed too complicated to be implemented correctly #17276

Bugs

• in-memory-engine: tikv crash due to an assert failure #16994
• In-memory engine: assertion failed during prewrite #17074
• In-memory Engine: sysbench oltp_read_write fails with "assertion failed" #17252
• In-memory engine: default not found in long tpcc run #17060
• In-memory engine: wrong-total in jepsen test #17018
• in-memory-engine: OLTP-insert has more than 10% regression when in-memory-engine is enabled in a test environment #16947
• In-memory Engine: assertion failed: self.direction == Direction::Backward #17079
• In-memory Engine: u64 endian in key encoding is wrong #17082
• In-memory Engine: Deadlock during TPCC prepare  #17100
• In-memory Engine: cached_write_batch is not cleared after load failure #17103
• In-memory Engine: race condition among delete range and write to memory engine #17104
• In-memory Engine: memory usage goes to negative number #17106
• In-memory Engine: cache engine's sequence number may differ from rocksdb  #17114
• In-memory Engine: safe_point in gc_range should be obtained from PD #17123
• In-memory Engine: range to evict may contain multiple cached range #17131
• In-memory Engine: lock should not be deleted directly #17127
• In-memory Engine: eviction should also consider loading ranges #17139
• In-memory Engine: encoding keys for lock cf should not contain mvcc in delete range #17140
• In-memory Engine: the tag in CacheRange may be inaccurate  #17147
• In-memory Engine: entered unreachable code in next_to_match #17148
• In-memory Engine: SELECT command denied to user 'root'@'% #17163
• In-memory Engine: loading range should be considered in eviction #17153
• In-memory Engine: range should be evicted after sst ingestion  #17182
• In-memory engine: manual load table does work properly #16966
• In-memory engine: evict in delete_range is too late #17243
• In-memory engine: panic in cross check task #17216
• In-memory engine: failed to start raft_server in jepsen test #17053
• In-memory Engine: Duplicate error when range cache enabled #17245
• In-memory Engine: delete range task has been scheduled multiple times #17259
• In-memory Engine: Deadlock during TPCC run #17260