HBASE-27843 If moveAndClose fails HFileArchiver should delete any incomplete archive side changes #5240
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Precommit test failures are not related. The failing tests are related to TLS RPC. I filed HBASE-27868 for these. |
Apache9
approved these changes
May 17, 2023
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
Show resolved
Hide resolved
sunhelly
approved these changes
May 17, 2023
wchevreuil
approved these changes
May 17, 2023
virajjasani
reviewed
May 17, 2023
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
Outdated
Show resolved
Hide resolved
…omplete archive side changes When HFiles are placed on a filesystem other than HDFS a rename operation can be a non-atomic file copy operation. It can take a long time to copy a large hfile and if interrupted there may be a partially copied file present at the destination. If we fail to “rename” the files into the archive we will continue to fail indefinitely. Before larger changes are considered, perhaps to StoreFileTracker, we should mitigate this problem.
|
Addressed review feedback, and rebased. |
virajjasani
approved these changes
May 18, 2023
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
asfgit
pushed a commit
that referenced
this pull request
May 25, 2023
…omplete archive side changes (#5240) When HFiles are placed on a filesystem other than HDFS a rename operation can be a non-atomic file copy operation. It can take a long time to copy a large hfile and if interrupted there may be a partially copied file present at the destination. If we fail to “rename” the files into the archive we will continue to fail indefinitely. Before larger changes are considered, perhaps to StoreFileTracker, we should mitigate this problem. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org> Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
asfgit
pushed a commit
that referenced
this pull request
May 25, 2023
…omplete archive side changes (#5240) When HFiles are placed on a filesystem other than HDFS a rename operation can be a non-atomic file copy operation. It can take a long time to copy a large hfile and if interrupted there may be a partially copied file present at the destination. If we fail to “rename” the files into the archive we will continue to fail indefinitely. Before larger changes are considered, perhaps to StoreFileTracker, we should mitigate this problem. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org> Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When HFiles are placed on a filesystem other than HDFS a rename operation can be a non-atomic file copy operation. It can take a long time to copy a large hfile and if interrupted there may be a partially copied file present at the destination. If we fail to “rename” the files into the archive we will continue to fail indefinitely. Before larger changes are considered, perhaps to StoreFileTracker, we should mitigate this problem.