HDFS-17297. The NameNode should remove block from the BlocksMap if the block is marked as deleted #6369

haiyang1987 · 2023-12-19T04:24:03Z

Description of PR

https://issues.apache.org/jira/browse/HDFS-17297

When call internalReleaseLease method:

boolean internalReleaseLease(
...
int minLocationsNum = 1;
if (lastBlock.isStriped()) {
  minLocationsNum = ((BlockInfoStriped) lastBlock).getRealDataBlockNum();
}

if (uc.getNumExpectedLocations() < minLocationsNum &&
    lastBlock.getNumBytes() == 0) {
  // There is no datanode reported to this block.
  // may be client have crashed before writing data to pipeline.
  // This blocks doesn't need any recovery.
  // We can remove this block and close the file.
  pendingFile.removeLastBlock(lastBlock);
  finalizeINodeFileUnderConstruction(src, pendingFile,
      iip.getLatestSnapshotId(), false); 
...
}

if the condition uc.getNumExpectedLocations() < minLocationsNum && lastBlock.getNumBytes() == 0 is met during the execution of UNDER_RECOVERY logic, the block is removed from the block list in the inode file and marked as deleted.
However it is not removed from the BlocksMap, it may cause memory leak.

Therefore it is necessary to remove the block from the BlocksMap at this point as well.

…e block is marked as deleted

hadoop-yetus · 2023-12-19T04:48:57Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	8m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
-1 ❌	mvninstall	0m 16s	/branch-mvninstall-root.txt	root in trunk failed.
-1 ❌	compile	0m 17s	/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt	hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.
-1 ❌	compile	0m 8s	/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt	hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.
-0 ⚠️	checkstyle	3m 4s	/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	The patch fails to run checkstyle in hadoop-hdfs
-1 ❌	mvnsite	0m 22s	/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in trunk failed.
-1 ❌	javadoc	0m 22s	/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt	hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.
-1 ❌	javadoc	0m 21s	/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt	hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.
-1 ❌	spotbugs	0m 21s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in trunk failed.
+1 💚	shadedclient	5m 13s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 21s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	compile	0m 21s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt	hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.
-1 ❌	javac	0m 21s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt	hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.
-1 ❌	compile	0m 10s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt	hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.
-1 ❌	javac	0m 10s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt	hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 10s	/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	The patch fails to run checkstyle in hadoop-hdfs
-1 ❌	mvnsite	0m 21s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	javadoc	0m 21s	/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt	hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.
-1 ❌	javadoc	0m 30s	/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt	hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.
-1 ❌	spotbugs	0m 11s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	shadedclient	6m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 21s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+0 🆗	asflicense	0m 21s		ASF License check generated no output?
		23m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6369/1/artifact/out/Dockerfile
GITHUB PR	#6369
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 0f3a5c9425ba 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a4c6137`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6369/1/testReport/
Max. process+thread count	51 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6369/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-12-19T07:01:53Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 21s		trunk passed
+1 💚	compile	0m 45s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 41s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 41s		trunk passed
+1 💚	mvnsite	0m 42s		trunk passed
+1 💚	javadoc	0m 40s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 2s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 55s		trunk passed
-1 ❌	shadedclient	34m 52s		branch has errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 41s		the patch passed
+1 💚	compile	0m 41s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 41s		the patch passed
+1 💚	compile	0m 36s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 36s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 31s		the patch passed
+1 💚	mvnsite	0m 40s		the patch passed
+1 💚	javadoc	0m 30s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 56s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 53s		the patch passed
-1 ❌	shadedclient	23m 0s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 42s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		107m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6369/2/artifact/out/Dockerfile
GITHUB PR	#6369
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux c33f92e02368 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `dc032c9`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6369/2/testReport/
Max. process+thread count	686 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6369/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-12-19T16:05:52Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	31m 39s		trunk passed
+1 💚	compile	0m 40s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 36s		trunk passed
+1 💚	mvnsite	0m 43s		trunk passed
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 4s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 41s		trunk passed
+1 💚	shadedclient	21m 22s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 40s		the patch passed
+1 💚	compile	0m 38s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 38s		the patch passed
+1 💚	compile	0m 38s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 38s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 32s		the patch passed
+1 💚	mvnsite	0m 38s		the patch passed
+1 💚	javadoc	0m 29s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 0s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 51s		the patch passed
+1 💚	shadedclient	21m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	180m 5s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 22s		The patch does not generate ASF License warnings.
		268m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6369/3/artifact/out/Dockerfile
GITHUB PR	#6369
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux bdd4cfe9ca69 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `28ca3c4`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6369/3/testReport/
Max. process+thread count	4133 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6369/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

haiyang1987 · 2023-12-20T12:00:13Z

Hi @ayushtkn @Hexiaoqiao @ZanderXu @zhangshuyan0 @tomscut Could you please help me review this pr when you have free time ? Thank you very much.

haiyang1987 · 2023-12-26T11:38:47Z

Hi @tasanuma Could you please help me review this pr when you have free time ? Thank you very much.

ZanderXu

LGTM

tasanuma · 2023-12-28T13:13:58Z

Merged. Thanks for fixing the issue, @haiyang1987. Thanks for your review, @ZanderXu.

…e block is marked as deleted (#6369) Reviewed-by: ZanderXu <zanderxu@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org> (cherry picked from commit 3aceec7)

tasanuma · 2023-12-28T13:59:23Z

Cherry-picked to branch-3.3.

haiyang1987 · 2023-12-29T02:31:25Z

Thanks @tasanuma @ZanderXu for your review and merge !!!

…e block is marked as deleted (apache#6369) Reviewed-by: ZanderXu <zanderxu@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>

HDFS-17297. The NameNode should remove block from the BlocksMap if th…

a4c6137

…e block is marked as deleted

github-actions bot added HDFS trunk labels Dec 19, 2023

trigger ci

dc032c9

trigger ci

28ca3c4

ZanderXu approved these changes Dec 27, 2023

View reviewed changes

tasanuma approved these changes Dec 27, 2023

View reviewed changes

tasanuma merged commit 3aceec7 into apache:trunk Dec 28, 2023
1 of 3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-17297. The NameNode should remove block from the BlocksMap if the block is marked as deleted #6369

HDFS-17297. The NameNode should remove block from the BlocksMap if the block is marked as deleted #6369

haiyang1987 commented Dec 19, 2023

hadoop-yetus commented Dec 19, 2023

hadoop-yetus commented Dec 19, 2023

hadoop-yetus commented Dec 19, 2023

haiyang1987 commented Dec 20, 2023

haiyang1987 commented Dec 26, 2023

ZanderXu left a comment

tasanuma commented Dec 28, 2023

tasanuma commented Dec 28, 2023

haiyang1987 commented Dec 29, 2023

HDFS-17297. The NameNode should remove block from the BlocksMap if the block is marked as deleted #6369

HDFS-17297. The NameNode should remove block from the BlocksMap if the block is marked as deleted #6369

Conversation

haiyang1987 commented Dec 19, 2023

Description of PR

hadoop-yetus commented Dec 19, 2023

hadoop-yetus commented Dec 19, 2023

hadoop-yetus commented Dec 19, 2023

haiyang1987 commented Dec 20, 2023

haiyang1987 commented Dec 26, 2023

ZanderXu left a comment

Choose a reason for hiding this comment

tasanuma commented Dec 28, 2023

tasanuma commented Dec 28, 2023

haiyang1987 commented Dec 29, 2023