PCP-27: level compaction skip some sst to reduce write amplification

[zhangjinpeng87]

Description

Ln sst1[a...b e...f]
Ln+1 sst2[a...b] sst3[c...d] sst4[e...f]
Assume a level compaction picks sst1, sst2, sst3 and sst4 as source ssts, we can see sst3's key range is a gap in sst1, but their ranges overlaps. And sst3's content will be unnecessarily re-written in level n+1, this will cause write amplification.
In TiDB the gap between index keys and row keys for the same row is broad, and will cause extra write amplification.

Difficulty

• Hard

Score

• 3600

Mentor(s)

• @zhangjinpeng1987

Recommended Skills

• Algorithm

[zhangjinpeng87]

@huachaohuang Seems you have finish this issue facebook/rocksdb#6021, Do you have interest to participate our performance challenge match? https://pingcap.com/community-cn/tidb-performance-challenge/

[huachaohuang]

Well, I'm not sure, I just do that for fun. Thanks for letting me know.

[huachaohuang]

How's the progress of this issue, are you still working on facebook/rocksdb#5201?
I'm interested on using this feature now.

[yiwu-arbug]

How's the progress of this issue, are you still working on facebook/rocksdb#5201?
I'm interested on using this feature now.

I'm restarting the work.