Support in-memory compaction in rocksdb and use it for lock CF

[yiwu-arbug]

Feature Request

Is your feature request related to a problem? Please describe:

For rocksdb (or any LSM tree implementation) in use cases when data get deleted soon after they being inserted, when memtable flush to disk, the end result could be significant smaller than the memtable size. This will result in inefficient compaction since L0 file sizes become too small. One way to work it around is compacting memtables: instead of flushing memtable when they are full, we can run compaction on them to reduce their size in memory. The technique reduces IO and also keep hot data in memory as much as possible.

One challenge to implement in-memory compaction is handling WAL. Normally when memtable get flushed, corresponding WAL file can be deleted since data is persisted to SSTs. However, after in-memory compaction such WAL files cannot be deleted. They will grow indefinitely. A workaround is to force memtable flush when WAL size reach certain threshold. RocksDB already provide such functionality. The WAL wouldn't be a problem if we remove the use of WAL from KvDB.

We can use in-memory compaction on lock CF to reduce its IO cost and keep its content in memory for query.

This is a common optimization for LSM implementations. We can probably sync with rocksdb team to know their views and plan.

[jyizheng]

Is this task to change RocksDB code? Is the in-memory compaction can be controlled by ti TiKV code? It would be helpful if anybody gives me a pointer to the relevant code.

[yiwu-arbug]

@jyizheng Yes, this task require changing rocksdb code, and for a warning this task is a hard task which could easily take > 1 month work. If you are still interested you can join #sig-engine channel at tikv-wg.slack.com and ping us. Thanks.

[jyizheng]

@yiwu-arbug Yes, I still want to work on this issue. I want to learn more about RocksDB. Can this issue be assigned to me?

[yiwu-arbug]

Discussed with @jyizheng offline and he will be working on the task.

[Connor1996]

From my test, l0 flow of lock CF is high.

With in-memory compaction, I think it would save some unnecessary write IO.

[dwangxxx]

From my test, l0 flow of lock CF is high.

With in-memory compaction, I think it would save some unnecessary write IO.

Hi, how should I get the compaction flow like this image? I'm not familiar with RocksDB, could you give me some guidance? Thanks!