Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDFS-15643. EC: Fix checksum computation in case of native encoders. #2424

Merged
merged 1 commit into from Nov 4, 2020

Conversation

ayushtkn
Copy link
Member

https://issues.apache.org/jira/browse/HDFS-15643

This is the reason for failure of TestFileChecksum when ISA-L is loaded

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 49m 33s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 ❌ mvninstall 4m 40s /branch-mvninstall-root.txt root in trunk failed.
-1 ❌ compile 1m 25s /branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.
-1 ❌ compile 0m 44s /branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.
-0 ⚠️ checkstyle 0m 21s /buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt The patch fails to run checkstyle in hadoop-hdfs
-1 ❌ mvnsite 0m 22s /branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in trunk failed.
+1 💚 shadedclient 1m 9s branch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 23s /branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.
-1 ❌ javadoc 0m 22s /branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.
+0 🆗 spotbugs 2m 19s Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌ findbugs 0m 23s /branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in trunk failed.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 22s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ compile 0m 22s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.
-1 ❌ javac 0m 22s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.
-1 ❌ compile 0m 23s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.
-1 ❌ javac 0m 23s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.
-0 ⚠️ checkstyle 0m 20s /buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt The patch fails to run checkstyle in hadoop-hdfs
-1 ❌ mvnsite 0m 23s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 0m 22s patch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 23s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.
-1 ❌ javadoc 0m 22s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.
-1 ❌ findbugs 0m 23s /patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
_ Other Tests _
-1 ❌ unit 16m 21s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+0 🆗 asflicense 0m 23s ASF License check generated no output?
83m 52s
Reason Tests
Failed junit tests hadoop.fs.contract.hdfs.TestHDFSContractMkdir
hadoop.hdfs.TestExternalBlockReader
hadoop.hdfs.TestFSOutputSummer
hadoop.hdfs.TestClose
hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/1/artifact/out/Dockerfile
GITHUB PR #2424
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux f0383d1cb2ed 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8ee6bc2
Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/1/testReport/
Max. process+thread count 242 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/1/console
versions git=2.17.1 maven=3.6.0
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 2m 15s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 ❌ mvninstall 33m 31s /branch-mvninstall-root.txt root in trunk failed.
-1 ❌ compile 0m 31s /branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.
-1 ❌ compile 0m 31s /branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.
+1 💚 checkstyle 0m 52s trunk passed
-1 ❌ mvnsite 1m 2s /branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in trunk failed.
-1 ❌ shadedclient 27m 9s branch has errors when building and testing our client artifacts.
+1 💚 javadoc 1m 2s trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 1m 30s trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+0 🆗 spotbugs 3m 23s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 21s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 1m 15s the patch passed
+1 💚 compile 1m 16s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
-1 ❌ javac 1m 16s /diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 generated 602 new + 0 unchanged - 0 fixed = 602 total (was 0)
+1 💚 compile 1m 10s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
-1 ❌ javac 1m 10s /diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu118.04-b10 with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu118.04-b10 generated 586 new + 0 unchanged - 0 fixed = 586 total (was 0)
+1 💚 checkstyle 0m 40s the patch passed
+1 💚 mvnsite 1m 12s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 26s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 48s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 1m 20s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 findbugs 3m 4s the patch passed
_ Other Tests _
-1 ❌ unit 118m 20s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 43s The patch does not generate ASF License warnings.
218m 35s
Reason Tests
Failed junit tests hadoop.hdfs.TestGetFileChecksum
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/2/artifact/out/Dockerfile
GITHUB PR #2424
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 90a7e59baa37 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8ee6bc2
Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/2/testReport/
Max. process+thread count 3553 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/2/console
versions git=2.17.1 maven=3.6.0 findbugs=4.1.3
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@ayushtkn
Copy link
Member Author

The build is failing -
[ERROR] unable to create new native thread -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError

This is something weird, Luckily I had one successful build on the jira, post that seems the Jenkins isn't happy with me. :-(

@amahussein
Copy link
Contributor

The build is failing -
[ERROR] unable to create new native thread -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError

This is something weird, Luckily I had one successful build on the jira, post that seems the Jenkins isn't happy with me. :-(

@ayushtkn , this error used to show up from time to time. It was one of the things we discussed on the mailing list.
Can you make any small change to the TestFileChecksum to trigger yetus running the junit? That way we can verify if the code change actually has an effect on the success of the test class.

@ayushtkn
Copy link
Member Author

Thanx @amahussein I have touched the test file. Well the test didn't run in this PR due to OOM. The only time it ran was yesterday.
Ref https://issues.apache.org/jira/browse/HDFS-15643?focusedCommentId=17223456&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17223456

I have a patch too, seems the build crashed there as well. Will try a couple of times, If no help, I think we need to reach Infra guys.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 6s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 33m 43s trunk passed
+1 💚 compile 1m 19s trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 compile 1m 9s trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 checkstyle 0m 50s trunk passed
+1 💚 mvnsite 1m 18s trunk passed
+1 💚 shadedclient 17m 34s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 54s trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 1m 26s trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+0 🆗 spotbugs 3m 13s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 10s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 1m 14s the patch passed
+1 💚 compile 1m 19s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javac 1m 19s the patch passed
+1 💚 compile 1m 3s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 javac 1m 3s the patch passed
+1 💚 checkstyle 0m 37s the patch passed
+1 💚 mvnsite 1m 10s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 44s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 49s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 1m 21s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 findbugs 3m 5s the patch passed
_ Other Tests _
-1 ❌ unit 133m 45s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 42s The patch does not generate ASF License warnings.
224m 17s
Reason Tests
Failed junit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
hadoop.hdfs.TestGetFileChecksum
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
hadoop.hdfs.server.datanode.TestBPOfferService
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/3/artifact/out/Dockerfile
GITHUB PR #2424
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux fb269eb03fb5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8ee6bc2
Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/3/testReport/
Max. process+thread count 3536 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/3/console
versions git=2.17.1 maven=3.6.0 findbugs=4.1.3
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 23s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 0m 0s test4tests The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 34m 47s trunk passed
+1 💚 compile 1m 19s trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 compile 1m 11s trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 checkstyle 0m 46s trunk passed
+1 💚 mvnsite 1m 15s trunk passed
+1 💚 shadedclient 18m 37s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 3s trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 1m 35s trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+0 🆗 spotbugs 3m 44s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 42s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 1m 22s the patch passed
+1 💚 compile 1m 24s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javac 1m 24s the patch passed
+1 💚 compile 1m 12s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 javac 1m 12s the patch passed
+1 💚 checkstyle 0m 45s the patch passed
+1 💚 mvnsite 1m 19s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 17m 31s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 56s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 1m 34s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 findbugs 3m 42s the patch passed
_ Other Tests _
-1 ❌ unit 165m 29s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 33s The patch does not generate ASF License warnings.
263m 42s
Reason Tests
Failed junit tests hadoop.hdfs.TestFileAppend2
hadoop.hdfs.TestDFSStripedOutputStream
hadoop.hdfs.TestFileAppendRestart
hadoop.hdfs.server.namenode.TestFsck
hadoop.fs.TestFcHdfsSetUMask
hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy
hadoop.hdfs.server.namenode.TestINodeAttributeProvider
hadoop.hdfs.TestGetFileChecksum
hadoop.fs.TestResolveHdfsSymlink
hadoop.hdfs.server.namenode.web.resources.TestWebHdfsCreatePermissions
hadoop.hdfs.TestFileAppend
hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant
hadoop.hdfs.server.namenode.TestStripedINodeFile
hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory
hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader
hadoop.hdfs.TestRollingUpgradeRollback
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
hadoop.hdfs.web.TestHttpsFileSystem
hadoop.fs.TestUrlStreamHandler
hadoop.fs.TestEnhancedByteBufferAccess
hadoop.hdfs.TestSetTimes
hadoop.fs.contract.hdfs.TestHDFSContractRename
hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
hadoop.hdfs.TestHdfsAdmin
hadoop.fs.TestFcHdfsCreateMkdir
hadoop.hdfs.web.TestWebHDFSXAttr
hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter
hadoop.hdfs.server.namenode.TestFSImage
hadoop.hdfs.server.namenode.TestNameNodeRecovery
hadoop.hdfs.TestReadStripedFileWithDNFailure
hadoop.hdfs.web.TestWebHDFS
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/4/artifact/out/Dockerfile
GITHUB PR #2424
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 224d9d1b4819 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / c47c9fd
Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/4/testReport/
Max. process+thread count 3501 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/4/console
versions git=2.17.1 maven=3.6.0 findbugs=4.1.3
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@ayushtkn
Copy link
Member Author

ayushtkn commented Oct 30, 2020

Oh..now the other fixed flaky tests fail again :)

Happens sometimes. :-)
In the recent build too, there are some test failures but they are due to OOM. But still our tests are safe.
TestGetFileChecksum looked little suspicious but that is due to NODE_TOO_BUSY, Akira's jira should get this sorted.

On the jira too, We have the test passed. So, If you are +1 on the changes, I will revert the last commit(The change in test file)

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 33m 49s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 ❌ mvninstall 0m 21s /branch-mvninstall-root.txt root in trunk failed.
-1 ❌ compile 0m 22s /branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.
-1 ❌ compile 0m 22s /branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.
-0 ⚠️ checkstyle 0m 19s /buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt The patch fails to run checkstyle in hadoop-hdfs
-1 ❌ mvnsite 0m 22s /branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in trunk failed.
+1 💚 shadedclient 1m 6s branch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 22s /branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.
-1 ❌ javadoc 0m 22s /branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.
+0 🆗 spotbugs 2m 13s Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌ findbugs 0m 22s /branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in trunk failed.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 22s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ compile 0m 22s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.
-1 ❌ javac 0m 22s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.
-1 ❌ compile 0m 22s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.
-1 ❌ javac 0m 22s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.
-0 ⚠️ checkstyle 0m 19s /buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt The patch fails to run checkstyle in hadoop-hdfs
-1 ❌ mvnsite 0m 22s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 0m 20s patch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 21s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.
-1 ❌ javadoc 0m 22s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.
-1 ❌ findbugs 0m 22s /patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
_ Other Tests _
-1 ❌ unit 0m 23s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
+1 💚 asflicense 0m 31s The patch does not generate ASF License warnings.
45m 1s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/5/artifact/out/Dockerfile
GITHUB PR #2424
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 5141cacec223 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 4890b85
Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/5/testReport/
Max. process+thread count 51 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/5/console
versions git=2.17.1 maven=3.6.0
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 29s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 40m 15s trunk passed
+1 💚 compile 1m 44s trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 compile 1m 27s trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 checkstyle 0m 58s trunk passed
+1 💚 mvnsite 1m 39s trunk passed
+1 💚 shadedclient 21m 58s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 3s trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 1m 42s trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+0 🆗 spotbugs 4m 10s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 4m 7s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 1m 26s the patch passed
+1 💚 compile 1m 37s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javac 1m 37s the patch passed
+1 💚 compile 1m 25s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 javac 1m 25s the patch passed
+1 💚 checkstyle 0m 46s the patch passed
+1 💚 mvnsite 1m 26s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 19m 18s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 59s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 1m 38s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 findbugs 4m 6s the patch passed
_ Other Tests _
-1 ❌ unit 126m 1s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 42s The patch does not generate ASF License warnings.
238m 13s
Reason Tests
Failed junit tests hadoop.hdfs.TestWriteConfigurationToDFS
hadoop.hdfs.TestStateAlignmentContextWithHA
hadoop.hdfs.TestFileChecksumCompositeCrc
hadoop.hdfs.TestGetFileChecksum
hadoop.hdfs.TestErasureCodingPolicies
hadoop.hdfs.TestFSOutputSummer
hadoop.hdfs.server.datanode.TestBPOfferService
hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor
hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
hadoop.hdfs.TestDFSOutputStream
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/6/artifact/out/Dockerfile
GITHUB PR #2424
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 50dd2cbc19e6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 4890b85
Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/6/testReport/
Max. process+thread count 3480 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/6/console
versions git=2.17.1 maven=3.6.0 findbugs=4.1.3
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@amahussein
Copy link
Contributor

Oh..now the other fixed flaky tests fail again :)

Happens sometimes. :-)
In the recent build too, there are some test failures but they are due to OOM. But still our tests are safe.
TestGetFileChecksum looked little suspicious but that is due to NODE_TOO_BUSY, Akira's jira should get this sorted.

On the jira too, We have the test passed. So, If you are +1 on the changes, I will revert the last commit(The change in test file)

+1 (non binding)

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 6s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 33m 49s trunk passed
+1 💚 compile 1m 20s trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 compile 1m 12s trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 checkstyle 0m 48s trunk passed
+1 💚 mvnsite 1m 20s trunk passed
+1 💚 shadedclient 17m 10s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 55s trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 1m 26s trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+0 🆗 spotbugs 3m 4s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 2s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 1m 8s the patch passed
+1 💚 compile 1m 11s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javac 1m 11s the patch passed
+1 💚 compile 1m 5s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 javac 1m 5s the patch passed
+1 💚 checkstyle 0m 39s the patch passed
+1 💚 mvnsite 1m 10s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 34s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 49s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 1m 20s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 findbugs 3m 0s the patch passed
_ Other Tests _
+1 💚 unit 118m 52s hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 42s The patch does not generate ASF License warnings.
208m 43s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/7/artifact/out/Dockerfile
GITHUB PR #2424
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 3d4f309d5aef 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / deea5d8
Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/7/testReport/
Max. process+thread count 3953 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2424/7/console
versions git=2.17.1 maven=3.6.0 findbugs=4.1.3
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Copy link
Member

@aajisaka aajisaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Thank you @ayushtkn and @amahussein

@ayushtkn ayushtkn merged commit 7ac7840 into apache:trunk Nov 4, 2020
asfgit pushed a commit that referenced this pull request Nov 4, 2020
asfgit pushed a commit that referenced this pull request Nov 4, 2020
asfgit pushed a commit that referenced this pull request Nov 4, 2020
@ayushtkn
Copy link
Member Author

ayushtkn commented Nov 4, 2020

Thanx @aajisaka and @amahussein for the reviews!!!

@amahussein it would be good to give a check to the tests that failed again, post getting fixed recently. I will try to figure out from the jenkins report, you to give a check

@amahussein
Copy link
Contributor

Thanks @ayushtkn ..
Yes, I was looking into that. I started looking into HDFS-15659 to set the dfs.namenode.redundancy.considerLoad in MiniDFSCluster. This should deal with all Junit that has NODE_TOO_BUSY in their stack trace.

asfgit pushed a commit that referenced this pull request Nov 6, 2020
Xushaohong pushed a commit to Xushaohong/hadoop that referenced this pull request Jul 20, 2022
jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023
…pache#2424). Contributed by Ayush Saxena.

(cherry picked from commit 76a7c0c)
Change-Id: Id1dc3c49eaca9d6c0fdd250d6a75444ccf46d507
(cherry picked from commit 1b8a9fde30a160dbc8c8810623716ea67549c36b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants