Level Compaction with TTL #3591
Conversation
There was a problem hiding this comment.
@sagar0 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@sagar0 has updated the pull request. View: changes, changes since last import |
There was a problem hiding this comment.
@sagar0 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@sagar0 has updated the pull request. |
There was a problem hiding this comment.
@sagar0 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@sagar0 has updated the pull request. |
|
Addressed @ajkr 's comments, and rebased. |
There was a problem hiding this comment.
@sagar0 is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
@sagar0 is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
Summary: Level Compaction with TTL. As of today, a file could exist in the LSM tree without going through the compaction process for a really long time if there are no updates to the data in the file's key range. For example, in certain use cases, the keys are not actually "deleted"; instead they are just set to empty values. There might not be any more writes to this "deleted" key range, and if so, such data could remain in the LSM for a really long time resulting in wasted space. Introducing a TTL could solve this problem. Files (and, in turn, data) older than TTL will be scheduled for compaction when there is no other background work. This will make the data go through the regular compaction process and get rid of old unwanted data. This also has the (good) side-effect of all the data in the non-bottommost level being newer than ttl, and all data in the bottommost level older than ttl. It could lead to more writes while reducing space. This functionality can be controlled by the newly introduced column family option -- ttl. TODO for later: - Make ttl mutable - Extend TTL to Universal compaction as well? (TTL is already supported in FIFO) - Maybe deprecate CompactionOptionsFIFO.ttl in favor of this new ttl option. Closes #3591 Differential Revision: D7275442 Pulled By: sagar0 fbshipit-source-id: dcba484717341200d419b0953dafcdf9eb2f0267
Level Compaction with TTL.
As of today, a file could exist in the LSM tree without going through the compaction process for a really long time if there are no updates to the data in the file's key range. For example, in certain use cases, the keys are not actually "deleted"; instead they are just set to empty values. There might not be any more writes to this "deleted" key range, and if so, such data could remain in the LSM for a really long time resulting in wasted space.
Introducing a TTL could solve this problem. Files (and, in turn, data) older than TTL will be scheduled for compaction when there is no other background work. This will make the data go through the regular compaction process and get rid of old unwanted data.
This also has the (good) side-effect of all the data in the non-bottommost level being newer than ttl, and all data in the bottommost level older than ttl. It could lead to more writes while reducing space.
This functionality can be controlled by the newly introduced column family option -- ttl.
TODO for later:
Test Plan:
Added a new unit test.
LOG file from the unit test: https://gist.github.com/sagar0/93e86a41dab29a4f5d5f1ad511663a4f .