HDDS-6962. [Snapshot] Background Service to delete irrelevant SST files in a snapshot.#3883
Merged
sadanand48 merged 17 commits intoapache:HDDS-6517-Snapshotfrom Dec 6, 2022
Merged
Conversation
sadanand48
changed the title
smengcl
added a commit
to smengcl/hadoop-ozone
that referenced
this pull request
Nov 2, 2022
…ST files in a snapshot apache#3883 Last commit e1501e3
Contributor
This surprises me. I always thought the first level was the bulkiest. Is there any documentation you can point me to? |
GeorgeJahad
reviewed
Nov 2, 2022
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SstFilteringService.java
Outdated
Show resolved
Hide resolved
Contributor
Author
Compaction is triggered when number of files in a level reaches a threshold size and the threshold size increases as the levels increase so the last level files will be biggest in size |
added 7 commits
November 4, 2022 01:20
(cherry picked from commit d4bf31123a243b4569bbd0ecd66a382ce13c126c) (cherry picked from commit ebb11917f0a08732e366bf60ab811740cc9d355d)
hemantk-12
reviewed
Nov 8, 2022
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabase.java
Outdated
Show resolved
Hide resolved
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabase.java
Outdated
Show resolved
Hide resolved
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabase.java
Outdated
Show resolved
Hide resolved
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabase.java
Outdated
Show resolved
Hide resolved
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabase.java
Show resolved
Hide resolved
...ksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java
Outdated
Show resolved
Hide resolved
...op-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestOMSnapshotDAG.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SstFilteringService.java
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SstFilteringService.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SstFilteringService.java
Outdated
Show resolved
Hide resolved
added 2 commits
November 9, 2022 17:12
smengcl
added a commit
to smengcl/hadoop-ozone
that referenced
this pull request
Nov 9, 2022
…elete irrelevant SST files in a snapshot https: //github.com/apache/pull/3883
...op-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestSstFilteringService.java
Show resolved
Hide resolved
Contributor
|
@sadanand48 can we resolve the pending comments and merge this PR ? |
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabase.java
Outdated
Show resolved
Hide resolved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR introduces a background service
SSTFilteringServiceto delete irrelevant SST files of a snapshot, It makes use of RocksDB#deleteFileAPI to do so which only permits the deletion of last level of SST files ie if there are n levels of SST's at a given point , the service will delete only nth level SST's. This is still beneficial as the last level SST's are the bulkiest and deletion of these can help save space. On successful deletion , it will update a marker file
filtered-snapshotsand write the snapshot ID of the processed Snapshot to it. Only thekeyTable,fileTable&directoryTablekeys are deleted as they can grow very large compared to other tables.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6962
How was this patch tested?
Unit tests