HADOOP-18056. DistCp: Filter duplicates in the source paths. #3825
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ayushtkn
force-pushed
the
HADOOP-18056
branch
from
9766389
to
565b1f8
Compare
|
🎊 +1 overall
This message was automatically generated. |
ayushtkn
force-pushed
the
HADOOP-18056
branch
from
565b1f8
to
a232e11
Compare
|
🎊 +1 overall
This message was automatically generated. |
tomscut
reviewed
Dec 28, 2021
| Set<Path> uniquePaths = new LinkedHashSet<>(); | ||
| for (Path path : srcPaths) { | ||
| if (!uniquePaths.add(path)) { | ||
| LOG.warn("Path: {} added multiple times, Ignoring the redundant entry.", path); |
There was a problem hiding this comment.
LGTM. Do we need to change Ignoring to ignoring.
There was a problem hiding this comment.
Thanx @tomscut, I have changed it.
|
🎊 +1 overall
This message was automatically generated. |
tomscut
approved these changes
Jan 3, 2022
|
LGTM. |
| Set<Path> uniquePaths = new LinkedHashSet<>(); | ||
| for (Path path : srcPaths) { | ||
| if (!uniquePaths.add(path)) { | ||
| LOG.warn("Path: {} added multiple times, ignoring the redundant entry.", path); |
There was a problem hiding this comment.
Should it log at info? I'm just worried because WARN Messages are often interpreted as a sign of serious problems; if this is happening due to some workflow then it's not going to go away.
There was a problem hiding this comment.
Yes, makes sense.
Thanx @steveloughran for the review. I have changed it to INFO
|
🎊 +1 overall
This message was automatically generated. |
asfgit
pushed a commit
that referenced
this pull request
Jan 5, 2022
… Contributed by Ayush Saxena. Reviewed-by: tomscut <litao@bigo.sg> Reviewed-by: Steve Loughran <stevel@apache.org>
sunchao
pushed a commit
that referenced
this pull request
Jan 15, 2022
… Contributed by Ayush Saxena. Reviewed-by: tomscut <litao@bigo.sg> Reviewed-by: Steve Loughran <stevel@apache.org>
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
…3825). Contributed by Ayush Saxena. Reviewed-by: tomscut <litao@bigo.sg> Reviewed-by: Steve Loughran <stevel@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Filter duplicates from source paths
How was this patch tested?
UT
For code changes: