Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-18726. Set the locale to avoid printing useless logs. #5612

Merged
merged 1 commit into from
May 2, 2023
Merged

HADOOP-18726. Set the locale to avoid printing useless logs. #5612

merged 1 commit into from
May 2, 2023

Conversation

zhangshuyan0
Copy link
Contributor

In our production environment, if the hadoop process is started in a non-English environment, many unexpected error logs will be printed. The following is the error message printed by datanode.

2023-05-01 09:10:50,299 ERROR org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op transferToSocketFully : 断开的管道
2023-05-01 09:10:50,299 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() exception: 
java.io.IOException: 断开的管道
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
        at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
        at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
        at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:242)
        at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:260)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:559)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:801)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:755)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:580)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:258)
        at java.lang.Thread.run(Thread.java:745)
2023-05-01 09:10:50,298 ERROR org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op transferToSocketFully : 断开的管道
2023-05-01 09:10:50,298 ERROR org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op transferToSocketFully : 断开的管道
2023-05-01 09:10:50,298 ERROR org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op transferToSocketFully : 断开的管道
2023-05-01 09:10:50,298 ERROR org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op transferToSocketFully : 断开的管道
2023-05-01 09:10:50,302 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() exception: 
java.io.IOException: 断开的管道
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
        at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
        at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
        at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:242)
        at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:260)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:559)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:801)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:755)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:580)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:258)
        at java.lang.Thread.run(Thread.java:745)
2023-05-01 09:10:50,303 ERROR org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op transferToSocketFully : 断开的管道
2023-05-01 09:10:50,303 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() exception: 
java.io.IOException: 断开的管道
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
        at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
        at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
        at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:242)
        at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:260)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:568)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:801)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:755)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:580)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:258)
        at java.lang.Thread.run(Thread.java:745)

The reason for this situation is that the code uses the message of IOException to determine whether to print Exception logs, but different locales will change the content of the message.

hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java

Lines 654 to 666 in 87e17b2

if (ioem.startsWith(EIO_ERROR)) {
throw new DiskFileCorruptException("A disk IO error occurred", e);
}
String causeMessage = e.getCause() != null ? e.getCause().getMessage() : "";
causeMessage = causeMessage != null ? causeMessage : "";
if (!ioem.startsWith("Broken pipe")
&& !ioem.startsWith("Connection reset")
&& !causeMessage.startsWith("Broken pipe")
&& !causeMessage.startsWith("Connection reset")) {
LOG.error("BlockSender.sendChunks() exception: ", e);
datanode.getBlockScanner().markSuspectBlock(
ris.getVolumeRef().getVolume().getStorageID(), block);
}

This large number of error logs is very misleading, so this patch sets the environment variable LANG in hadoop-env.sh.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 36s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 40m 12s trunk passed
+1 💚 mvnsite 1m 29s trunk passed
+1 💚 shadedclient 19m 22s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 58s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 1m 18s the patch passed
+1 💚 shellcheck 0m 0s No new issues.
+1 💚 shadedclient 19m 17s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 1m 44s hadoop-common in the patch passed.
+1 💚 asflicense 0m 39s The patch does not generate ASF License warnings.
88m 26s
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5612/1/artifact/out/Dockerfile
GITHUB PR #5612
Optional Tests dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs
uname Linux 0ba13099fc75 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 7bcd58a
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5612/1/testReport/
Max. process+thread count 554 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5612/1/console
versions git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn
ayushtkn approved these changes May 2, 2023
View reviewed changes
Copy link
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me. Will be good if we can have another pair of eyes from another committer as well

Hexiaoqiao
Hexiaoqiao approved these changes May 2, 2023
View reviewed changes
Copy link
Contributor

@Hexiaoqiao Hexiaoqiao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. +1.

@Hexiaoqiao Hexiaoqiao merged commit fddc976 into apache:trunk May 2, 2023
@Hexiaoqiao
Copy link
Contributor

Committed to trunk. Thanks @zhangshuyan0 for your contribution! Thanks @ayushtkn for your reviews!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Hexiaoqiao Hexiaoqiao Hexiaoqiao approved these changes

@ayushtkn ayushtkn ayushtkn approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@zhangshuyan0 @hadoop-yetus @Hexiaoqiao @ayushtkn