HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false #2443

amahussein · 2020-11-06T17:55:53Z

This PR does the following:

MiniDFSCluster.Builder sets dfs.namenode.redundancy.considerLoad to false by default.
a new method Builder.setNNRedundancyConsiderLoad() can be used to overwrite the default value in the conf.
removed explicit disabling of the flag in some Junits. Those were safe place to remove the disabling the flag, as the MiniDFSCluster is initialized directly after setting the configuration.
supercede changes done in HDFS-9776. TestHAAppend#testMultipleAppendsDuringCatchupTailing is flaky #2410 HDFS-15461. TestDFSClientRetries testGetFileChecksum fails. #2404

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

hadoop-yetus · 2020-11-06T21:29:39Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 10s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 13 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	36m 13s		trunk passed
+1 💚	compile	1m 18s		trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	compile	1m 12s		trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚	checkstyle	0m 53s		trunk passed
+1 💚	mvnsite	1m 18s		trunk passed
+1 💚	shadedclient	18m 53s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 54s		trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javadoc	1m 21s		trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+0 🆗	spotbugs	3m 11s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 9s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	1m 13s		the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javac	1m 13s		the patch passed
+1 💚	compile	1m 5s		the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚	javac	1m 5s		the patch passed
+1 💚	checkstyle	0m 48s		the patch passed
+1 💚	mvnsite	1m 11s		the patch passed
+1 💚	whitespace	0m 1s		The patch has no whitespace issues.
+1 💚	shadedclient	16m 10s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 48s		the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javadoc	1m 18s		the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚	findbugs	3m 13s		the patch passed
			_ Other Tests _
-1 ❌	unit	116m 53s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		212m 36s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancer

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2443/1/artifact/out/Dockerfile
GITHUB PR	#2443
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux bdd0a0aaca54 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `4b31281`
Default Java	Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2443/1/testReport/
Max. process+thread count	2974 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2443/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.1.3
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2020-11-07T04:19:41Z

Thanx for the update, the approach and changes looks good. But I think Jenkins won't have run all the tests? If not, you can touch a line in hadoop-project/pom.xml and increase the timeout in Jenkinsfile from 20 to 30 hrs, if post that everything is good we can merge the current change.

Out of curiosity: You don't need a new PR for a new change, you could have force-pushed to your previous branch itself, it would have updated the same PR.

hadoop-yetus · 2020-11-10T08:36:13Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 11s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 13 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 2s		Maven dependency ordering for branch
+1 💚	mvninstall	23m 9s		trunk passed
+1 💚	compile	21m 49s		trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	compile	18m 30s		trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚	checkstyle	2m 59s		trunk passed
+1 💚	mvnsite	26m 5s		trunk passed
+1 💚	shadedclient	16m 49s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 26s		trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javadoc	8m 24s		trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+0 🆗	spotbugs	38m 6s		Used deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗	findbugs	0m 23s		branch/hadoop-project no findbugs output file (findbugsXml.xml)
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	24m 21s		the patch passed
+1 💚	compile	21m 18s		the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javac	21m 18s		the patch passed
+1 💚	compile	19m 21s		the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚	javac	19m 21s		the patch passed
+1 💚	checkstyle	3m 23s		the patch passed
+1 💚	mvnsite	22m 46s		the patch passed
+1 💚	shellcheck	0m 0s		There were no new shellcheck issues.
+1 💚	shelldocs	0m 14s		There were no new shelldocs issues.
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	shadedclient	16m 35s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	8m 8s		the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javadoc	7m 50s		the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+0 🆗	findbugs	0m 17s		hadoop-project has no data from findbugs
			_ Other Tests _
-1 ❌	unit	614m 58s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 25s		The patch does not generate ASF License warnings.
		969m 24s

Reason	Tests
Failed junit tests	hadoop.tools.dynamometer.TestDynamometerInfra
	hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
	hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
	hadoop.fs.azure.TestNativeAzureFileSystemMocked
	hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
	hadoop.fs.azure.TestOutOfBandAzureBlobOperations
	hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
	hadoop.fs.azure.TestBlobMetadata
	hadoop.fs.azure.TestWasbFsck
	hadoop.yarn.applications.distributedshell.TestDistributedShell
	hadoop.yarn.client.api.impl.TestAMRMClient
	hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem
	hadoop.hdfs.TestDFSShell
	hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2443/2/artifact/out/Dockerfile
GITHUB PR	#2443
Optional Tests	dupname asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 1aee3e9136d3 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ae7b00a`
Default Java	Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2443/2/testReport/
Max. process+thread count	2941 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-hdfs-project/hadoop-hdfs . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2443/2/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.1.3
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

amahussein · 2020-11-10T21:59:38Z

Thanx for the update, the approach and changes looks good. But I think Jenkins won't have run all the tests? If not, you can touch a line in hadoop-project/pom.xml and increase the timeout in Jenkinsfile from 20 to 30 hrs, if post that everything is good we can merge the current change.

Thanks @ayushtkn, I made the change to the PR.
The failing unit tests are the usual suspects. Even {{TestDynamometerInfra}} started to show up in all commits since yesterday.

Out of curiosity: You don't need a new PR for a new change, you could have force-pushed to your previous branch itself, it would have updated the same PR.

Yeah, thanks. I know I could force commits but I did not want to lose track of the previous implementation.

ayushtkn · 2020-11-11T02:46:54Z

Thanx @amahussein the update. Changes looks good. Can you revert the last commit, I think the tests are stable. We can proceed here

…lt to false

hadoop-yetus · 2020-11-11T19:15:29Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 13 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	36m 51s		trunk passed
+1 💚	compile	1m 21s		trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	compile	1m 10s		trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚	checkstyle	0m 55s		trunk passed
+1 💚	mvnsite	1m 20s		trunk passed
+1 💚	shadedclient	19m 32s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 56s		trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javadoc	1m 22s		trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+0 🆗	spotbugs	3m 15s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 13s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 11s		the patch passed
+1 💚	compile	1m 13s		the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javac	1m 13s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚	javac	1m 6s		the patch passed
+1 💚	checkstyle	0m 48s		the patch passed
+1 💚	mvnsite	1m 11s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	16m 39s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 49s		the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javadoc	1m 22s		the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚	findbugs	3m 13s		the patch passed
			_ Other Tests _
-1 ❌	unit	116m 13s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		213m 58s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestListCorruptFileBlocks

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2443/3/artifact/out/Dockerfile
GITHUB PR	#2443
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 36d402169250 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e6d2dcc`
Default Java	Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2443/3/testReport/
Max. process+thread count	2653 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2443/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.1.3
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

amahussein · 2020-11-18T19:43:38Z

I checked hadoop.hdfs.server.namenode.TestListCorruptFileBlocks and it is passing.
I will monitor it and file a new lira if I see it failing frequently.
@ayushtkn , I think we can go ahead and merge this.

ayushtkn

Changes LGTM
Will merge by EOD, if no further comments

…lt to false (#2443). Contributed by Ahmed Hussein. (cherry picked from commit 747883a)

…lt to false (#2443). Contributed by Ahmed Hussein.

amahussein force-pushed the hdfs-15659-easy-approach branch from 10d83b9 to b9e548a Compare November 9, 2020 16:25

HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad defau…

ab17df9

…lt to false

amahussein force-pushed the hdfs-15659-easy-approach branch from b9e548a to ab17df9 Compare November 11, 2020 15:40

ayushtkn approved these changes Nov 19, 2020

View reviewed changes

ayushtkn merged commit 747883a into apache:trunk Nov 20, 2020

asfgit pushed a commit that referenced this pull request Jun 14, 2021

HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad defau…

840612b

…lt to false (#2443). Contributed by Ahmed Hussein. (cherry picked from commit 747883a)

asfgit pushed a commit that referenced this pull request Jun 25, 2021

HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad defau…

9731dfb

…lt to false (#2443). Contributed by Ahmed Hussein.

asfgit pushed a commit that referenced this pull request Jun 25, 2021

HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad defau…

7d231f4

…lt to false (#2443). Contributed by Ahmed Hussein.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false #2443

HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false #2443

amahussein commented Nov 6, 2020 •

edited

Loading

hadoop-yetus commented Nov 6, 2020

ayushtkn commented Nov 7, 2020

hadoop-yetus commented Nov 10, 2020

amahussein commented Nov 10, 2020

ayushtkn commented Nov 11, 2020

hadoop-yetus commented Nov 11, 2020

amahussein commented Nov 18, 2020

ayushtkn left a comment

HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false #2443

HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false #2443

Conversation

amahussein commented Nov 6, 2020 • edited Loading

NOTICE

hadoop-yetus commented Nov 6, 2020

ayushtkn commented Nov 7, 2020

hadoop-yetus commented Nov 10, 2020

amahussein commented Nov 10, 2020

ayushtkn commented Nov 11, 2020

hadoop-yetus commented Nov 11, 2020

amahussein commented Nov 18, 2020

ayushtkn left a comment

Choose a reason for hiding this comment

amahussein commented Nov 6, 2020 •

edited

Loading