HDFS-16316.Improve DirectoryScanner: add regular file check related block. #3861
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
jianghuazhu
force-pushed
the
HDFS-16316
branch
from
112122d
to
3a0edea
Compare
|
💔 -1 overall
This message was automatically generated. |
jianghuazhu
force-pushed
the
HDFS-16316
branch
from
3a0edea
to
2929261
Compare
|
💔 -1 overall
This message was automatically generated. |
|
Here are some unit tests that failed: It seems that these failed tests have little to do with the content I submitted. |
jojochuang
reviewed
Feb 9, 2022
| @@ -2812,6 +2816,10 @@ public void checkAndUpdate(String bpid, ScanInfo scanInfo) | |||
| + memBlockInfo.getNumBytes() + " to " | |||
| + memBlockInfo.getBlockDataLength()); | |||
| memBlockInfo.setNumBytes(memBlockInfo.getBlockDataLength()); | |||
| } else if (!isRegular) { | |||
There was a problem hiding this comment.
Unrelated: the checkAndUpdate() is way too long. We should refactor it in the future.
There was a problem hiding this comment.
Thanks for your comment and review, @jojochuang .
I'll add some unit tests later.
This is indeed a bit too long for the checkAndUpdate() method.
I'll be fine-tuning this later if I can, but will handle this as a new jira.
There was a problem hiding this comment.
Absolutely. Let's not worry about the refactor now. Thanks.
jianghuazhu
force-pushed
the
HDFS-16316
branch
2 times, most recently
from
adede86
to
39177f5
Compare
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
Here are some examples of online clusters. This file is non-standard. Then DataNode will tell NameNode that there are some unqualified blocks through NameNodeRpcServer#reportBadBlocks(). After the NameNode gets the data, it will process it further. Can you help review this pr again, @jojochuang . |
jojochuang
reviewed
Feb 15, 2022
| @Test(timeout = 600000) | ||
| public void testRegularBlock() throws Exception { | ||
| // add a logger stream to check what has printed to log | ||
| ByteArrayOutputStream loggerStream = new ByteArrayOutputStream(); |
There was a problem hiding this comment.
Can you use the Hadoop utility class LogCapturer
for this purpose?There was a problem hiding this comment.
Yes, it was my mistake.
jojochuang
reviewed
Feb 15, 2022
jianghuazhu
force-pushed
the
HDFS-16316
branch
from
39177f5
to
ca61b8b
Compare
|
🎊 +1 overall
This message was automatically generated. |
jianghuazhu
force-pushed
the
HDFS-16316
branch
from
ca61b8b
to
f5e27e4
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
Thanks for the suggestion, @jojochuang . When I remove the fix, the newly added unit test does not succeed, which is expected and does not affect the execution of other unit tests. Here is an example during normal testing: |
Thanks @jianghuazhu for reminding me. This check makes sense to me. And what I'm curious about is what was done before the metaFile became a device file? Or what is the root cause of this problem? |
tomscut
reviewed
Feb 17, 2022
| @@ -2812,6 +2816,9 @@ public void checkAndUpdate(String bpid, ScanInfo scanInfo) | |||
| + memBlockInfo.getNumBytes() + " to " | |||
| + memBlockInfo.getBlockDataLength()); | |||
| memBlockInfo.setNumBytes(memBlockInfo.getBlockDataLength()); | |||
| } else if (!isRegular) { | |||
| corruptBlock = new Block(memBlockInfo); | |||
| LOG.warn("Block:{} is not a regular file.", corruptBlock.getBlockId()); | |||
There was a problem hiding this comment.
Maybe we should print the absolute path so that we can deal with these abnormal files?
There was a problem hiding this comment.
Thanks @tomscut for the comment and review.
This happens occasionally, I've been monitoring it for a long time and still haven't found the root cause.
But I think this situation may be related to the Linux environment. When the normal data flow is working, no exception occurs. (I will continue to monitor this situation)
Here are some more canonical checks to prevent further worse cases on the cluster. This is a good thing for clusters.
When the file is actually cleaned up, the specific path will be printed. Here are some examples of online clusters:
2022-02-15 11:24:12,856 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-xxxx blk_xxxx file /mnt/dfs/11/data/current/BP-xxxx.xxxx.xxxx/current/finalized/subdir0/subdir0/blk_xxxx
There was a problem hiding this comment.
Thank you for explaining this. It looks like the operation related to mount has been performed. Did HDFS successfully clean the abnormal file on your online cluster you mentioned after this change?
There was a problem hiding this comment.
Yes, these exception files are cleaned up.
When NameNode obtains such abnormal files, it treats them as invalid Blocks.
When the DataNode sends a heartbeat to the NameNode, the NameNode notifies the DataNode to clean up.
The specific cleaning action is performed by FsDatasetAsyncDiskServic
ferhui
reviewed
Feb 18, 2022
...oject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
Show resolved
Hide resolved
|
@jianghuazhu Thanks for your contribution. @jojochuang @tomscut Thanks for your reviews! Merged. |
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
When blk_xxxx and blk_xxxx.meta are not regular files, they will have adverse effects on the cluster, such as errors in the calculation space and the possibility of failure to read data.
For this type of block, it should not be used as a normal block file.
Details:
HDFS-16316
How was this patch tested?
Need to verify whether a file is a real regular file.