HDFS-16964. Improve processing of excess redundancy after failover. #5510
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Hexiaoqiao
reviewed
Mar 24, 2023
There was a problem hiding this comment.
Thanks to report and patch. IIUC, this improvement try to prepose to clean excess replicas exclude ones reside at stale storage, rather than wait to all storage leave stale status, right? If so, I think it makes sense to me. Leave some review comments inline, FYI.
| @@ -4020,12 +4014,22 @@ public void setReplication( | |||
| } | |||
| } | |||
|
|
|||
| private void processExtraRedundancyBlock(final BlockInfo block, | |||
There was a problem hiding this comment.
Suggest to add some java doc for this new method.
| /** | ||
| * Find how many of the containing nodes are "extra", if any. | ||
| * If there are any extras, call chooseExcessRedundancies() to | ||
| * mark them in the excessRedundancyMap. | ||
| * @return if all redundancy replicas are removed |
There was a problem hiding this comment.
@return if all redundancy replicas are removed
->
@return true if all redundancy replicas are removed.
| @@ -4035,17 +4039,17 @@ private void processExtraRedundancyBlock(final BlockInfo block, | |||
| Collection<DatanodeStorageInfo> nonExcess = new ArrayList<>(); | |||
| Collection<DatanodeDescriptor> corruptNodes = corruptReplicas | |||
| .getNodes(block); | |||
| boolean hasStaleStorage = false; | |||
| DatanodeStorageInfo staleStorage = null; | |||
There was a problem hiding this comment.
this should be one set about DatanodeStorageInfo because it could be more than one stale storage here.
|
💔 -1 overall
This message was automatically generated. |
|
@Hexiaoqiao Thanks for your review, I've updated the PR based on your suggestion. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Hexiaoqiao
approved these changes
Mar 27, 2023
There was a problem hiding this comment.
LGTM. +1 from my side. Let's wait if other folks are interested about this improvement.
BTW, the failed unit test seems un-relate to this PR. @zhangshuyan0 Please take another check.
|
@Hexiaoqiao Thanks. I've checked it out. |
|
Committed to trunk. @zhangshuyan0 thanks for your contribution! |
ferdelyi
pushed a commit
to ferdelyi/hadoop
that referenced
this pull request
May 26, 2023
…pache#5510). Contributed by Shuyan Zhang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
After failover, the block with excess redundancy cannot be processed until all replicas are not stale, because the stale ones may have been deleted. That is to say, we need to wait for the FBRs of all datanodes on which the block resides before deleting the redundant replicas. This is unnecessary, we can bypass stale replicas when dealing with excess replicas, and delete non-stale excess replicas in a more timely manner.