Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use deleteFilesInRanges instead of deleteRange in RocksDBLogStorage #768

Closed
killme2008 opened this issue Feb 21, 2022 · 4 comments
Closed
Labels
enhancement New feature or request high priority
Milestone

Comments

@killme2008
Copy link
Contributor

killme2008 commented Feb 21, 2022

According to https://github.com/facebook/rocksdb/wiki/Delete-A-Range-Of-Keys

The command will remove all SST files only containing keys in the range to delete. For a large chunk, it will immediately reclaim most space, so it is a good solution to release disk pressure under writing heavy load.

@killme2008
Copy link
Contributor Author

killme2008 commented Feb 21, 2022

It' also noted by TIKV https://tikv.github.io/deep-dive-tikv/key-value-engine/rocksdb.html

DeleteRange turns out to be unstable and can not release disk space fast enough.

A faster way to release disk space is to delete some files directly, which leads us to the DeleteFilesInRange feature. 
But this feature is not perfect, it is quite dangerous because it breaks the snapshot consistency. 
If you acquire a snapshot from RocksDB, use DeleteFilesInRange to delete some files, 
then try to read that data you will find that some of it is missing. So we should use this feature carefully.

In RocksDBLogStorage we don't do any snapshot, so there is no consistency problem.

fengjiachun pushed a commit that referenced this issue Feb 25, 2022
#769)

* (feat) Use deleteFilesInRange/compactRange to replace deleteRange in RocksdbLogStorage#truncatePrefix, #768

* (feat) Improve logs in RocksDBLogStorage
@killme2008 killme2008 added this to the 1.4.0 milestone Mar 8, 2022
@fengjiachun fengjiachun mentioned this issue Mar 17, 2022
11 tasks
@18013631263
Copy link

升级到1.3.10后,deleteFilesInRanges 之后再 compactRange 并没有立即释放磁盘,rockdb空间磁盘占用还是居高不下

@killme2008
Copy link
Contributor Author

killme2008 commented Mar 27, 2022

看看你的 rocksdb 占用是什么吧,这个只会在启用 snapshot,加快 raft log 的磁盘占用回收,对状态机的数据没有帮助的

@cassvin
Copy link

cassvin commented Mar 29, 2022

在某些场景下可能会有数据残留:

  1. deleteFilesInRanges不会删除L0文件;
  2. 如果文件里面只有部分key ranges。比如触发snapshot时index是1000,等到实际调用删除接口时最新的sst文件里面包含900 - 1100范围的key,那么这个sst也不会被选中删除,下一轮也删不了。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request high priority
Projects
None yet
Development

No branches or pull requests

4 participants