HDFS-16838. Fix NPE in testAddRplicaProcessorForAddingReplicaInMap by ZanderXu · Pull Request #5125 · apache/hadoop

ZanderXu · 2022-11-11T03:04:47Z

Description of PR

There is a NPE in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList#testAddRplicaProcessorForAddingReplicaInMap if we run this UT individually. And the related bug as bellow:

public void testAddRplicaProcessorForAddingReplicaInMap() throws Exception {
  // BUG here
  BlockPoolSlice.reInitializeAddReplicaThreadPool();
  Configuration cnf = new Configuration();
  int poolSize = 5; 
  ...
}

hadoop-yetus · 2022-11-11T08:57:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 42s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 21s		trunk passed
+1 💚	compile	1m 42s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 37s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 20s		trunk passed
+1 💚	mvnsite	1m 41s		trunk passed
+1 💚	javadoc	1m 16s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 43s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 32s		trunk passed
+1 💚	shadedclient	23m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 18s		the patch passed
+1 💚	compile	1m 24s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 24s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 1s		the patch passed
+1 💚	mvnsite	1m 27s		the patch passed
+1 💚	javadoc	0m 54s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 25s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 23s		the patch passed
+1 💚	shadedclient	22m 55s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	238m 25s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 10s		The patch does not generate ASF License warnings.
		351m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5125/1/artifact/out/Dockerfile
GITHUB PR	#5125
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 5c440fb5a00e 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `097fc87`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5125/1/testReport/
Max. process+thread count	3245 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5125/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hotcodemacha

+1. Changes makes sense.

xinglin · 2022-11-12T17:12:39Z

Instead of the change proposed in this PR, shouldn't we actually fix reInitializeAddReplicaThreadPool() itself as below?

Current:

  @VisibleForTesting
  public static void reInitializeAddReplicaThreadPool() {
    addReplicaThreadPool.shutdown();
    addReplicaThreadPool = null;
  }

Fixed as following:

  @VisibleForTesting
  public static void reInitializeAddReplicaThreadPool() {
    if (addReplicaThreadPool != null) {
      addReplicaThreadPool.shutdown();
      addReplicaThreadPool = null;
    }
  }

aajisaka · 2022-11-21T07:00:51Z

I'm +1 for the @xinglin 's proposal. @ZanderXu could you update the PR?

ZanderXu · 2022-11-22T13:08:18Z

Sorry for the late update, @xinglin @aajisaka please help me review it again. Thanks so much.

xinglin

Thanks for making the change/fix. lgtm.

hadoop-yetus · 2022-11-22T19:49:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 48s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 13s		trunk passed
+1 💚	compile	1m 37s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 30s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 19s		trunk passed
+1 💚	mvnsite	1m 42s		trunk passed
+1 💚	javadoc	1m 15s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 49s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 34s		trunk passed
+1 💚	shadedclient	23m 22s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 18s		the patch passed
+1 💚	compile	1m 21s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 21s		the patch passed
+1 💚	compile	1m 18s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 56s		the patch passed
+1 💚	mvnsite	1m 28s		the patch passed
+1 💚	javadoc	0m 53s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 39s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
-1 ❌	spotbugs	4m 11s	/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html	hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	25m 58s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	284m 8s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 8s		The patch does not generate ASF License warnings.
		397m 56s

Reason	Tests
SpotBugs	module:hadoop-hdfs-project/hadoop-hdfs
	Incorrect lazy initialization of static field org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.addReplicaThreadPool in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.reInitializeAddReplicaThreadPool() At BlockPoolSlice.java:field org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.addReplicaThreadPool in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.reInitializeAddReplicaThreadPool() At BlockPoolSlice.java:[lines 1142-1144]
Failed junit tests	hadoop.hdfs.TestLeaseRecovery2
	hadoop.hdfs.TestDFSInotifyEventInputStream

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5125/2/artifact/out/Dockerfile
GITHUB PR	#5125
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 3ac68c2dd87d 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3ff22f2`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5125/2/testReport/
Max. process+thread count	2819 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5125/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

tomscut · 2022-11-23T00:24:16Z

Hi @ZanderXu , could you please fix the SpotBugs warn.

hadoop-yetus · 2022-11-23T08:06:08Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 38s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 27s		trunk passed
+1 💚	compile	1m 34s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	1m 33s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	1m 24s		trunk passed
+1 💚	mvnsite	1m 41s		trunk passed
+1 💚	javadoc	1m 15s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	1m 43s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	3m 38s		trunk passed
+1 💚	shadedclient	23m 21s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 23s		the patch passed
+1 💚	compile	1m 22s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	1m 22s		the patch passed
+1 💚	compile	1m 19s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	1m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 56s		the patch passed
+1 💚	mvnsite	1m 23s		the patch passed
+1 💚	javadoc	0m 54s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	1m 30s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	3m 23s		the patch passed
+1 💚	shadedclient	22m 44s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	303m 1s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 3s		The patch does not generate ASF License warnings.
		412m 53s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestRollingUpgrade
	hadoop.hdfs.server.balancer.TestBalancerService
	hadoop.hdfs.TestLeaseRecovery2

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5125/3/artifact/out/Dockerfile
GITHUB PR	#5125
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 0915b660eb8f 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `7f040a1`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5125/3/testReport/
Max. process+thread count	3399 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5125/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

tomscut · 2022-11-24T01:07:09Z

The failed unit tests are unrelated to the change.

tomscut · 2022-11-24T01:08:52Z

Thanks @ZanderXu for your contribution! Thanks @aajisaka , @xinglin and @ashutoshcipher for your review!

…pache#5125) Reviewed-by: Xing Lin <xinglin@linkedin.com> Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com> Signed-off-by: Tao Li <tomscut@apache.org>

hotcodemacha approved these changes Nov 11, 2022

View reviewed changes

HDFS-16838. Fix NPE in testAddRplicaProcessorForAddingReplicaInMap

3ff22f2

ZanderXu force-pushed the HDFS-16838 branch from 097fc87 to 3ff22f2 Compare November 22, 2022 13:07

xinglin approved these changes Nov 22, 2022

View reviewed changes

tomscut approved these changes Nov 23, 2022

View reviewed changes

tomscut self-requested a review November 23, 2022 00:24

HDFS-16838. Fix code

7f040a1

tomscut approved these changes Nov 24, 2022

View reviewed changes

tomscut merged commit bcc3d2a into apache:trunk Nov 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16838. Fix NPE in testAddRplicaProcessorForAddingReplicaInMap#5125

HDFS-16838. Fix NPE in testAddRplicaProcessorForAddingReplicaInMap#5125
tomscut merged 2 commits intoapache:trunkfrom
ZanderXu:HDFS-16838

ZanderXu commented Nov 11, 2022

Uh oh!

hadoop-yetus commented Nov 11, 2022

Uh oh!

hotcodemacha left a comment

Uh oh!

xinglin commented Nov 12, 2022

Uh oh!

aajisaka commented Nov 21, 2022

Uh oh!

ZanderXu commented Nov 22, 2022

Uh oh!

xinglin left a comment

Uh oh!

hadoop-yetus commented Nov 22, 2022

Uh oh!

tomscut commented Nov 23, 2022

Uh oh!

hadoop-yetus commented Nov 23, 2022

Uh oh!

tomscut commented Nov 24, 2022

Uh oh!

tomscut commented Nov 24, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

ZanderXu commented Nov 11, 2022

Description of PR

Uh oh!

hadoop-yetus commented Nov 11, 2022

Uh oh!

hotcodemacha left a comment

Choose a reason for hiding this comment

Uh oh!

xinglin commented Nov 12, 2022

Uh oh!

aajisaka commented Nov 21, 2022

Uh oh!

ZanderXu commented Nov 22, 2022

Uh oh!

xinglin left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Nov 22, 2022

Uh oh!

tomscut commented Nov 23, 2022

Uh oh!

hadoop-yetus commented Nov 23, 2022

Uh oh!

tomscut commented Nov 24, 2022

Uh oh!

tomscut commented Nov 24, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants