HDDS-4307.Start Trash Emptier in Ozone Manager #1507
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mukul1987
requested changes
Oct 20, 2020
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/TrashDeletingService.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/TrashDeletingService.java
Outdated
Show resolved
Hide resolved
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
Outdated
Show resolved
Hide resolved
|
Can you address the CI failures ? |
rakeshadr
reviewed
Oct 22, 2020
...one/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystem.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/TrashDeletingService.java
Outdated
Show resolved
Hide resolved
rakeshadr
reviewed
Oct 22, 2020
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/TrashDeletingService.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/TrashDeletingService.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/TrashDeletingService.java
Outdated
Show resolved
Hide resolved
|
Thanks @sadanand48 for the contribution. I've added few comments, please go through it. |
sadanand48
changed the title
|
Thanks @rakeshadr for the review. Addressed your comments . Will incorporate the changes you suggested on the TrashDeletingService when i include it in the next patch. |
rakeshadr
reviewed
Oct 27, 2020
There was a problem hiding this comment.
Thanks @sadanand48 for the contribution. I've added few minor comments.
Apart from these +1 LGTM.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
Outdated
Show resolved
Hide resolved
|
+1 LGTM |
mukul1987
approved these changes
Nov 11, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The change here is to implement a Trash emptier for Trash Deletion. This feature basically introduces Trash for Ozone File System by leveraging the hadoop code for Trash (
Trash.java) . When a client does a delete on the filesystem, it calls thefs.rename( src,dst,Options)api which does a rename to the Trash Path. As an initial approach all operations are on theKeyTableitself.This pr starts a TrashEmptier and deletes all the files obtained by
fs.getTrashRoots(). By passing an OFS uri pointing to the root to thisfsinstance covering all volumes and buckets will be ensured. Thus the background service starts the emptier thread which basically iterates through the trash roots and does the checkpointing and then deletion from checkpointed directories.bin/ozone fs -rm o3fs://buck1.vol1/Test2
2020-10-20 18:42:12,779 [main] INFO fs.TrashPolicyDefault: Moved: 'o3fs://buck1.vol1/Test2' to trash at: /.Trash/sadanand.shenoy/Current/Test2bin/ozone fs -ls -R o3fs://buck1.vol1/
-rw-rw-rw- 1 sadanand.shenoy sadanand.shenoy 0 2020-10-20 18:42 o3fs://buck1.vol1/.Trash/sadanand.shenoy/Current/Test2after checkpoint interval
-rw-rw-rw- 1 sadanand.shenoy sadanand.shenoy 0 2020-10-20 18:45 o3fs://buck1.vol1/.Trash/sadanand.shenoy/201020184500/Test2The key gets deleted after the trash interval.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4307
How was this patch tested?
Manually as well as few unit Tests.