HDFS-16832. [SBN READ] Fix NPE when check the block location of empty… #5099

zhengchenyu · 2022-11-02T10:38:51Z

Description of PR

https://issues.apache.org/jira/browse/HDFS-16832

How was this patch tested?

unit test
check in our cluster manually

For code changes:

check null point

hadoop-yetus · 2022-11-02T18:51:46Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 0s		trunk passed
+1 💚	compile	1m 33s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 36s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 19s		trunk passed
+1 💚	mvnsite	1m 38s		trunk passed
+1 💚	javadoc	1m 15s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 46s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 43s		trunk passed
+1 💚	shadedclient	23m 15s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 25s		the patch passed
+1 💚	compile	1m 25s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 25s		the patch passed
+1 💚	compile	1m 13s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 0s		the patch passed
+1 💚	mvnsite	1m 24s		the patch passed
+1 💚	javadoc	0m 54s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 27s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 24s		the patch passed
+1 💚	shadedclient	22m 55s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	381m 29s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		491m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5099/1/artifact/out/Dockerfile
GITHUB PR	#5099
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 7984bffd2633 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `83146db`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5099/1/testReport/
Max. process+thread count	2763 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5099/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

zhengchenyu · 2022-11-10T06:27:31Z

@xkrogen @shvachko @ZanderXu Can you please review this PR? NPE is introduced by HDFS-16732. NPE will reproduce by below:

hive -hiveconf "hive.execution.engine=tez" -e "select 1 as a1 UNION ALL select 2 as a2";

ZanderXu

@zhengchenyu Thanks for your report. LGTM

ZanderXu · 2022-11-17T09:44:45Z

@zhengchenyu Can you rebase trunk and trigger the yetus?

… directory

hadoop-yetus · 2022-11-17T19:14:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 15s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 7s		trunk passed
+1 💚	compile	1m 40s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 39s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 15s		trunk passed
+1 💚	mvnsite	1m 44s		trunk passed
+1 💚	javadoc	1m 24s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 44s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 44s		trunk passed
+1 💚	shadedclient	23m 10s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 30s		the patch passed
+1 💚	compile	1m 27s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 27s		the patch passed
+1 💚	compile	1m 21s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 58s		the patch passed
+1 💚	mvnsite	1m 30s		the patch passed
+1 💚	javadoc	0m 56s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 33s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 40s		the patch passed
+1 💚	shadedclient	22m 46s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	438m 42s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 4s		The patch does not generate ASF License warnings.
		550m 52s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5099/3/artifact/out/Dockerfile
GITHUB PR	#5099
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 15011cc5ae8b 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `bea887a`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5099/3/testReport/
Max. process+thread count	2834 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5099/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-11-17T19:14:45Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 19s		trunk passed
+1 💚	compile	1m 38s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 35s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 15s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 25s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 41s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 42s		trunk passed
+1 💚	shadedclient	23m 22s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 25s		the patch passed
+1 💚	compile	1m 29s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 29s		the patch passed
+1 💚	compile	1m 19s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 58s		the patch passed
+1 💚	mvnsite	1m 25s		the patch passed
+1 💚	javadoc	0m 57s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 32s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 37s		the patch passed
+1 💚	shadedclient	23m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	438m 51s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		551m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5099/4/artifact/out/Dockerfile
GITHUB PR	#5099
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 23e530bcec40 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `bea887a`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5099/4/testReport/
Max. process+thread count	2793 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5099/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ZanderXu · 2022-11-18T02:39:18Z

@zhengchenyu Thanks for your update. I will merge it tomorrow if there are no other comments.

zhengchenyu · 2022-11-18T02:48:22Z

@zhengchenyu Thanks for your update. I will merge it tomorrow if there are no other comments.

Okay, Thanks for your review again! It had better to package HDFS-16832 and HDFS-16732 into same release. Sorry for this bug, I should fix in HDFS-16732. I found this after HDFS-16732 was online some weeks.

xkrogen

Thanks for fixing this @zhengchenyu !

...s-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

hadoop-yetus · 2022-11-19T11:37:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 33s		trunk passed
+1 💚	compile	1m 34s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 29s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 15s		trunk passed
+1 💚	mvnsite	1m 47s		trunk passed
+1 💚	javadoc	1m 17s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 46s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 44s		trunk passed
+1 💚	shadedclient	23m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 20s		the patch passed
+1 💚	compile	1m 21s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 21s		the patch passed
+1 💚	compile	1m 15s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 55s		the patch passed
+1 💚	mvnsite	1m 25s		the patch passed
+1 💚	javadoc	0m 53s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 28s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 22s		the patch passed
+1 💚	shadedclient	23m 4s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	400m 6s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 4s		The patch does not generate ASF License warnings.
		510m 7s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestLeaseRecovery2

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5099/5/artifact/out/Dockerfile
GITHUB PR	#5099
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 5a7955358aff 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f51bf94`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5099/5/testReport/
Max. process+thread count	2793 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5099/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

xkrogen · 2022-11-21T16:19:25Z

TestLeaseRecovery2 has been flaky; I am not worried about the failure.

Merging to trunk. Thank you for the contribution @zhengchenyu !

…e block location of empty directory (#5099) Signed-off-by: Erik Krogen <xkrogen@apache.org> Reviewed-by: Zengqiang Xu <xuzq_zander@163.com> (cherry picked from commit dc2fba4)

xkrogen · 2022-11-21T16:41:26Z

Merged to branch-3.3 as well since that has HDFS-16732 also.

…e block location of empty directory (apache#5099) Signed-off-by: Erik Krogen <xkrogen@apache.org> Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>

…e block location of empty directory (#5099) Signed-off-by: Erik Krogen <xkrogen@apache.org> Reviewed-by: Zengqiang Xu <xuzq_zander@163.com> (cherry picked from commit dc2fba4) (cherry picked from commit 4addf31)

…e block location of empty directory (apache#5099) Signed-off-by: Erik Krogen <xkrogen@apache.org> Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>

ZanderXu approved these changes Nov 17, 2022

View reviewed changes

zhengchenyu closed this Nov 17, 2022

zhengchenyu deleted the HDFS-16832 branch November 17, 2022 09:56

HDFS-16832. [SBN READ] Fix NPE when check the block location of empty…

bea887a

… directory

zhengchenyu reopened this Nov 17, 2022

zhengchenyu force-pushed the HDFS-16832 branch from ec991cf to bea887a Compare November 17, 2022 10:02

xkrogen reviewed Nov 18, 2022

View reviewed changes

...s-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java Outdated Show resolved Hide resolved

fix code

f51bf94

xkrogen merged commit dc2fba4 into apache:trunk Nov 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16832. [SBN READ] Fix NPE when check the block location of empty… #5099

HDFS-16832. [SBN READ] Fix NPE when check the block location of empty… #5099

zhengchenyu commented Nov 2, 2022

hadoop-yetus commented Nov 2, 2022

zhengchenyu commented Nov 10, 2022

ZanderXu left a comment

ZanderXu commented Nov 17, 2022

hadoop-yetus commented Nov 17, 2022

hadoop-yetus commented Nov 17, 2022

ZanderXu commented Nov 18, 2022

zhengchenyu commented Nov 18, 2022 •

edited

xkrogen left a comment

hadoop-yetus commented Nov 19, 2022

xkrogen commented Nov 21, 2022

xkrogen commented Nov 21, 2022

HDFS-16832. [SBN READ] Fix NPE when check the block location of empty… #5099

HDFS-16832. [SBN READ] Fix NPE when check the block location of empty… #5099

Conversation

zhengchenyu commented Nov 2, 2022

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Nov 2, 2022

zhengchenyu commented Nov 10, 2022

ZanderXu left a comment

Choose a reason for hiding this comment

ZanderXu commented Nov 17, 2022

hadoop-yetus commented Nov 17, 2022

hadoop-yetus commented Nov 17, 2022

ZanderXu commented Nov 18, 2022

zhengchenyu commented Nov 18, 2022 • edited

xkrogen left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Nov 19, 2022

xkrogen commented Nov 21, 2022

xkrogen commented Nov 21, 2022

zhengchenyu commented Nov 18, 2022 •

edited