HDFS-17398. [FGL] Implement the FGL lock for FSNLockManager #6599

ZanderXu · 2024-02-29T06:29:50Z

HDFS-17398. [FGL] Implement the FGL lock for FSNLockManager.

hadoop-yetus · 2024-03-04T18:24:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	16m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ HDFS-17384 Compile Tests _
+1 💚	mvninstall	48m 27s		HDFS-17384 passed
+1 💚	compile	1m 25s		HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	1m 15s		HDFS-17384 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	1m 12s		HDFS-17384 passed
+1 💚	mvnsite	1m 22s		HDFS-17384 passed
+1 💚	javadoc	1m 10s		HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 35s		HDFS-17384 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 17s		HDFS-17384 passed
+1 💚	shadedclient	40m 0s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 10s		the patch passed
+1 💚	compile	1m 16s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	1m 16s		the patch passed
+1 💚	compile	1m 8s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	1m 8s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 3s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚	mvnsite	1m 14s		the patch passed
-1 ❌	javadoc	0m 56s	/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt	hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.
+1 💚	javadoc	1m 31s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 18s		the patch passed
+1 💚	shadedclient	39m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	259m 23s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		428m 47s

Reason	Tests
Failed junit tests	hadoop.hdfs.tools.TestDFSAdmin
	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.protocol.TestBlockListAsLongs

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6599/2/artifact/out/Dockerfile
GITHUB PR	#6599
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux b57cdf3721f1 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HDFS-17384 / `ce0e01f`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6599/2/testReport/
Max. process+thread count	2968 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6599/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ferhui

It introduces a simple FGL which splits the global lock into 2 locks, fsLock and bmLock. It is clear to me. Left some minor comments.

...fs/src/main/java/org/apache/hadoop/hdfs/server/namenode/fgl/FineGrainedFSNamesystemLock.java

...rc/test/java/org/apache/hadoop/hdfs/server/namenode/fgl/TestFineGrainedFSNamesystemLock.java

ferhui

+1

hadoop-yetus · 2024-03-05T10:45:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ HDFS-17384 Compile Tests _
+1 💚	mvninstall	49m 5s		HDFS-17384 passed
+1 💚	compile	1m 23s		HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	1m 15s		HDFS-17384 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	1m 12s		HDFS-17384 passed
+1 💚	mvnsite	1m 21s		HDFS-17384 passed
+1 💚	javadoc	1m 9s		HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 36s		HDFS-17384 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 17s		HDFS-17384 passed
+1 💚	shadedclient	40m 5s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 10s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	1m 16s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	1m 6s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 2s		the patch passed
+1 💚	mvnsite	1m 14s		the patch passed
+1 💚	javadoc	0m 56s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 33s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 17s		the patch passed
+1 💚	shadedclient	40m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	264m 22s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		418m 51s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestRollingUpgrade
	hadoop.hdfs.tools.TestDFSAdmin
	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.protocol.TestBlockListAsLongs

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6599/3/artifact/out/Dockerfile
GITHUB PR	#6599
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 111c75fcc935 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HDFS-17384 / `142e80d`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6599/3/testReport/
Max. process+thread count	2969 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6599/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-03-05T14:08:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ HDFS-17384 Compile Tests _
+1 💚	mvninstall	49m 12s		HDFS-17384 passed
+1 💚	compile	1m 25s		HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	1m 16s		HDFS-17384 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	1m 13s		HDFS-17384 passed
+1 💚	mvnsite	1m 24s		HDFS-17384 passed
+1 💚	javadoc	1m 9s		HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 37s		HDFS-17384 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 20s		HDFS-17384 passed
+1 💚	shadedclient	40m 34s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 11s		the patch passed
+1 💚	compile	1m 16s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	1m 16s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	1m 6s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 3s		the patch passed
+1 💚	mvnsite	1m 13s		the patch passed
+1 💚	javadoc	0m 58s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 33s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 17s		the patch passed
+1 💚	shadedclient	40m 37s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	265m 6s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 50s		The patch does not generate ASF License warnings.
		438m 24s

Reason	Tests
Failed junit tests	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.tools.TestDFSAdmin
	hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6599/4/artifact/out/Dockerfile
GITHUB PR	#6599
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux c133c37a03b3 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HDFS-17384 / `a39fa75`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6599/4/testReport/
Max. process+thread count	3050 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6599/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ferhui · 2024-03-05T14:10:23Z

Failed cases are unrelated.

ferhui · 2024-03-05T14:11:10Z

Thanks. Merged

)

github-actions bot added HDFS trunk labels Feb 29, 2024

HDFS-17398. [FGL] Implement the FGL lock for FSNLockManager

ce0e01f

ZanderXu force-pushed the HDFS-17398 branch from c4ed9bc to ce0e01f Compare March 4, 2024 11:13

ZanderXu requested a review from ferhui March 4, 2024 11:15

HDFS-17398. fix check style

142e80d

ferhui reviewed Mar 5, 2024

View reviewed changes

...fs/src/main/java/org/apache/hadoop/hdfs/server/namenode/fgl/FineGrainedFSNamesystemLock.java Show resolved Hide resolved

...rc/test/java/org/apache/hadoop/hdfs/server/namenode/fgl/TestFineGrainedFSNamesystemLock.java Outdated Show resolved Hide resolved

HDFS-17398. Add some comments

a39fa75

ferhui approved these changes Mar 5, 2024

View reviewed changes

ferhui merged commit c19db05 into apache:HDFS-17384 Mar 5, 2024
1 of 4 checks passed

ZanderXu added a commit to ZanderXu/hadoop that referenced this pull request Mar 25, 2024

HDFS-17398. [FGL] Implement the FGL lock for FSNLockManager (apache#6599

87287dc

)

ZanderXu added a commit that referenced this pull request Mar 25, 2024

HDFS-17398. [FGL] Implement the FGL lock for FSNLockManager (#6599)

b688ac8

ZanderXu added a commit that referenced this pull request Apr 23, 2024

HDFS-17398. [FGL] Implement the FGL lock for FSNLockManager (#6599)

0317088

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-17398. [FGL] Implement the FGL lock for FSNLockManager #6599

HDFS-17398. [FGL] Implement the FGL lock for FSNLockManager #6599

ZanderXu commented Feb 29, 2024

hadoop-yetus commented Mar 4, 2024

ferhui left a comment

ferhui left a comment

hadoop-yetus commented Mar 5, 2024

hadoop-yetus commented Mar 5, 2024

ferhui commented Mar 5, 2024

ferhui commented Mar 5, 2024

HDFS-17398. [FGL] Implement the FGL lock for FSNLockManager #6599

HDFS-17398. [FGL] Implement the FGL lock for FSNLockManager #6599

Conversation

ZanderXu commented Feb 29, 2024

hadoop-yetus commented Mar 4, 2024

ferhui left a comment

Choose a reason for hiding this comment

ferhui left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Mar 5, 2024

hadoop-yetus commented Mar 5, 2024

ferhui commented Mar 5, 2024

ferhui commented Mar 5, 2024