HDFS-16791. Add getEnclosingRoot() API to filesystem interface and im… #6262

mccormickt12 · 2023-11-09T01:11:21Z

…plementations (#6198)

The enclosing root path is a common ancestor that should be used for temp and staging dirs as well as within encryption zones and other restricted directories.

Contributed by Tom McCormick

Description of PR

Cherry pick of #6198 (review)

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…plementations (apache#6198) The enclosing root path is a common ancestor that should be used for temp and staging dirs as well as within encryption zones and other restricted directories. Contributed by Tom McCormick

mccormickt12 · 2023-11-09T01:22:06Z

@steveloughran Nearly clean cherry pick. Just a couple of things that exist in trunk but not in 3.3, so all deletes.

Unmerged paths:
  (use "git add <file>..." to mark resolution)
        both modified:   hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        both modified:   hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/

hadoop-yetus · 2023-11-09T06:49:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	5m 10s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	14m 3s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 46s		branch-3.3 passed
+1 💚	compile	11m 38s		branch-3.3 passed
+1 💚	checkstyle	1m 47s		branch-3.3 passed
+1 💚	mvnsite	3m 59s		branch-3.3 passed
+1 💚	javadoc	3m 38s		branch-3.3 passed
+1 💚	spotbugs	6m 33s		branch-3.3 passed
+1 💚	shadedclient	23m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s		Maven dependency ordering for patch
-1 ❌	mvninstall	0m 19s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch failed.
-1 ❌	compile	3m 13s	/patch-compile-root.txt	root in the patch failed.
-1 ❌	cc	3m 13s	/patch-compile-root.txt	root in the patch failed.
-1 ❌	javac	3m 13s	/patch-compile-root.txt	root in the patch failed.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 32s		the patch passed
-1 ❌	mvnsite	0m 22s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch failed.
-1 ❌	javadoc	0m 43s	/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-project_hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌	spotbugs	0m 21s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch failed.
+1 💚	shadedclient	23m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	15m 23s		hadoop-common in the patch passed.
+1 💚	unit	1m 53s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	179m 50s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
-1 ❌	unit	0m 32s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch failed.
+1 💚	asflicense	0m 40s		The patch does not generate ASF License warnings.
		336m 37s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
	hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/1/artifact/out/Dockerfile
GITHUB PR	#6262
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint cc buflint bufcompat
uname	Linux 064de127f595 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `82285ce`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/1/testReport/
Max. process+thread count	3227 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2023-11-10T19:34:37Z

looks like the hadoop-rbf stuff didn't take. just did a local build of branch-3.3 and it is happy, so this probably really is a regression

hadoop-yetus · 2023-11-10T23:14:06Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 26s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	14m 0s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 12s		branch-3.3 passed
+1 💚	compile	11m 45s		branch-3.3 passed
+1 💚	checkstyle	1m 49s		branch-3.3 passed
+1 💚	mvnsite	3m 57s		branch-3.3 passed
+1 💚	javadoc	3m 39s		branch-3.3 passed
+1 💚	spotbugs	6m 32s		branch-3.3 passed
+1 💚	shadedclient	23m 29s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 35s		the patch passed
+1 💚	compile	11m 10s		the patch passed
+1 💚	cc	11m 10s		the patch passed
+1 💚	javac	11m 10s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 52s		the patch passed
+1 💚	mvnsite	3m 56s		the patch passed
+1 💚	javadoc	3m 38s		the patch passed
+1 💚	spotbugs	7m 11s		the patch passed
+1 💚	shadedclient	23m 38s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	15m 54s		hadoop-common in the patch passed.
+1 💚	unit	2m 7s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	178m 50s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	unit	15m 14s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 55s		The patch does not generate ASF License warnings.
		359m 19s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/2/artifact/out/Dockerfile
GITHUB PR	#6262
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint cc buflint bufcompat
uname	Linux be5ac791eb92 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `d74f8cb`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/2/testReport/
Max. process+thread count	3140 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-11-11T00:52:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	13m 51s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 44s		branch-3.3 passed
+1 💚	compile	11m 48s		branch-3.3 passed
+1 💚	checkstyle	1m 47s		branch-3.3 passed
+1 💚	mvnsite	3m 58s		branch-3.3 passed
+1 💚	javadoc	3m 44s		branch-3.3 passed
+1 💚	spotbugs	6m 40s		branch-3.3 passed
+1 💚	shadedclient	23m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 50s		Maven dependency ordering for patch
-1 ❌	mvninstall	0m 30s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt	hadoop-hdfs-client in the patch failed.
-1 ❌	mvninstall	0m 30s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	mvninstall	0m 19s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch failed.
-1 ❌	compile	1m 28s	/patch-compile-root.txt	root in the patch failed.
-1 ❌	cc	1m 28s	/patch-compile-root.txt	root in the patch failed.
-1 ❌	javac	1m 28s	/patch-compile-root.txt	root in the patch failed.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 30s		the patch passed
-1 ❌	mvnsite	0m 32s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt	hadoop-hdfs-client in the patch failed.
-1 ❌	mvnsite	0m 32s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	mvnsite	0m 22s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch failed.
-1 ❌	javadoc	0m 58s	/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project_hadoop-hdfs generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1)
-1 ❌	spotbugs	0m 32s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt	hadoop-hdfs-client in the patch failed.
-1 ❌	spotbugs	0m 30s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	spotbugs	0m 21s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch failed.
-1 ❌	shadedclient	7m 19s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	15m 25s		hadoop-common in the patch passed.
-1 ❌	unit	0m 33s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt	hadoop-hdfs-client in the patch failed.
-1 ❌	unit	0m 31s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	unit	0m 21s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch failed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		131m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/3/artifact/out/Dockerfile
GITHUB PR	#6262
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint cc buflint bufcompat
uname	Linux 729255df80a4 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `9603a9e`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/3/testReport/
Max. process+thread count	1637 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/3/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-11-16T00:15:52Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	4m 25s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	13m 38s		Maven dependency ordering for branch
+1 💚	mvninstall	23m 4s		branch-3.3 passed
+1 💚	compile	12m 4s		branch-3.3 passed
+1 💚	checkstyle	1m 28s		branch-3.3 passed
+1 💚	mvnsite	3m 11s		branch-3.3 passed
+1 💚	javadoc	2m 48s		branch-3.3 passed
+1 💚	spotbugs	5m 38s		branch-3.3 passed
+1 💚	shadedclient	22m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
-1 ❌	mvninstall	0m 28s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt	hadoop-hdfs-client in the patch failed.
-1 ❌	mvninstall	0m 28s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	mvninstall	0m 17s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch failed.
-1 ❌	compile	1m 34s	/patch-compile-root.txt	root in the patch failed.
-1 ❌	cc	1m 34s	/patch-compile-root.txt	root in the patch failed.
-1 ❌	javac	1m 34s	/patch-compile-root.txt	root in the patch failed.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 16s		the patch passed
-1 ❌	mvnsite	0m 29s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt	hadoop-hdfs-client in the patch failed.
-1 ❌	mvnsite	0m 29s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	mvnsite	0m 19s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch failed.
-1 ❌	javadoc	0m 59s	/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project_hadoop-hdfs generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1)
-1 ❌	spotbugs	0m 27s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt	hadoop-hdfs-client in the patch failed.
-1 ❌	spotbugs	0m 28s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	spotbugs	0m 18s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch failed.
-1 ❌	shadedclient	6m 10s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	15m 52s		hadoop-common in the patch passed.
-1 ❌	unit	0m 32s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt	hadoop-hdfs-client in the patch failed.
-1 ❌	unit	0m 31s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	unit	0m 19s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch failed.
+1 💚	asflicense	0m 22s		The patch does not generate ASF License warnings.
		126m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/4/artifact/out/Dockerfile
GITHUB PR	#6262
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint cc buflint bufcompat
uname	Linux 0404f4720b88 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `9603a9e`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/4/testReport/
Max. process+thread count	3152 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/4/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

mccormickt12 · 2023-11-18T19:49:29Z

@steveloughran This kind of seems like yetus issues to me now.
#6262 (comment) This check was good after I fixed the path import (you can tell from the revision)

I also locally saw one other failure locally which was the protobuf static import thats not in 3.3.
From a local build i get this (i get this vectorized failure locally even on trunk and 3.3 branch, as do others at LinkedIn)

[INFO] Reactor Summary for Apache Hadoop Main 3.3.9-SNAPSHOT:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  2.175 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.702 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.783 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  4.200 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.440 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  2.311 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  4.932 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  1.763 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 14.290 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  3.420 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 52.162 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  5.820 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  4.902 s]
[INFO] Apache Hadoop Registry ............................. SUCCESS [  4.801 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.221 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 27.815 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:03 min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  6.074 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [  7.388 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  8.744 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 16.316 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.110 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.115 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 16.592 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 19.048 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.151 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 18.680 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 16.008 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  5.339 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  6.741 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  5.562 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 28.425 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 12.229 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 11.407 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  6.134 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  6.123 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  0.100 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [  6.191 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [  8.402 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [  0.223 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2  SUCCESS [  8.135 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [  6.883 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  6.092 s]
[INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [  6.275 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.202 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  6.363 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  4.921 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  3.996 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 12.177 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  8.662 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 16.345 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 35.711 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  7.221 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 16.106 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  6.607 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [  0.121 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [  9.997 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [ 11.396 s]
[INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [  0.164 s]
[INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [ 30.373 s]
[INFO] Apache Hadoop YARN Application Catalog Docker Image  SUCCESS [  1.290 s]
[INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [  0.109 s]
[INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [  4.645 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.161 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  3.270 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.213 s]
[INFO] Apache Hadoop YARN CSI ............................. SUCCESS [ 18.946 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 40.277 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  4.629 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [  6.116 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [  4.928 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  7.062 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  9.783 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  7.484 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  7.579 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [  6.449 s]
[INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [  5.438 s]
[INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [  7.000 s]
[INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [  5.220 s]
[INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [ 14.487 s]
[INFO] Apache Hadoop Dynamometer .......................... SUCCESS [  0.112 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  5.620 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  5.841 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  6.825 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 24.614 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 15.760 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  6.101 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.132 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 17.450 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [  3.482 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 11.015 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  4.691 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  6.107 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [  4.900 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  4.188 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [  6.270 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 34.632 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  3.371 s]
[INFO] Apache Hadoop Common Benchmark ..................... FAILURE [ 19.398 s]
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Client API ........................... SKIPPED
[INFO] Apache Hadoop Client Runtime ....................... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Tencent COS Support .................. SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15:36 min
[INFO] Finished at: 2023-11-18T11:43:02-08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.4:single (make-assembly) on project hadoop-benchmark: Failed to create assembly: Error creating assembly archive uber: Problem creating jar: jar:file:/Users/tmccormi/projects/hadoop/hadoop-tools/hadoop-benchmark/target/hadoop-benchmark-3.3.9-SNAPSHOT.jar!/org/apache/hadoop/benchmark/generated/VectoredReadBenchmark_jmhType_B1.class -> [Help 1]
[ERROR]

Im gonna try reverting the last commit (although it seems needed) to see if yetus likes it more

This reverts commit 9603a9e.

mccormickt12 · 2023-11-18T20:08:14Z

After the revert I get building locally

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-hdfs-client: Compilation failure
[ERROR] /Users/tmccormi/projects/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java:[2098,13] cannot find symbol
[ERROR]   symbol:   method getRemoteException(org.apache.hadoop.thirdparty.protobuf.ServiceException)
[ERROR]   location: class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB

hadoop-yetus · 2023-11-19T01:28:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+0 🆗	buf	0m 1s		buf was not available.
+0 🆗	buf	0m 1s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	13m 39s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 42s		branch-3.3 passed
+1 💚	compile	11m 3s		branch-3.3 passed
+1 💚	checkstyle	1m 31s		branch-3.3 passed
+1 💚	mvnsite	3m 11s		branch-3.3 passed
+1 💚	javadoc	2m 56s		branch-3.3 passed
+1 💚	spotbugs	5m 51s		branch-3.3 passed
+1 💚	shadedclient	21m 9s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 23s		the patch passed
+1 💚	compile	10m 42s		the patch passed
+1 💚	cc	10m 42s		the patch passed
+1 💚	javac	10m 42s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 25s		the patch passed
+1 💚	mvnsite	3m 11s		the patch passed
+1 💚	javadoc	2m 52s		the patch passed
+1 💚	spotbugs	6m 11s		the patch passed
+1 💚	shadedclient	21m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	15m 42s		hadoop-common in the patch passed.
+1 💚	unit	1m 55s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	169m 22s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	unit	14m 43s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		335m 54s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/5/artifact/out/Dockerfile
GITHUB PR	#6262
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint cc buflint bufcompat
uname	Linux c32133654be4 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `6a7e834`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/5/testReport/
Max. process+thread count	4699 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/5/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

mccormickt12 · 2023-11-27T17:30:30Z

@steveloughran Let me know if theres any concerns with the current failure, it seems transient / expected to me, but not 100%
cc @simbadzina

mccormickt12 · 2023-11-29T22:25:09Z

@goiri Thoughts? Does this backport look ok. The failure doesn't seem related to my backport right?

goiri · 2023-11-30T21:38:04Z

@goiri Thoughts? Does this backport look ok. The failure doesn't seem related to my backport right?

LGTM.

@steveloughran do you mind taking a look?

steveloughran · 2023-12-01T14:10:19Z

I'm going to say unrelated. HDFS-16740/#4835 addresses flakiness in trunk, but it's not been backported. as usual: volunteers welcome.

steveloughran · 2023-12-01T14:12:06Z

merged. If you are on common-dev you can see we've started a discussion about pushing out a branch-3.3 release...

add path import for build failure

d74f8cb

fix static import difference from trunk

9603a9e

Revert "fix static import difference from trunk"

6a7e834

This reverts commit 9603a9e.

goiri approved these changes Nov 30, 2023

View reviewed changes

steveloughran merged commit e82d83e into apache:branch-3.3 Dec 1, 2023
0 of 3 checks passed

mccormickt12 mentioned this pull request Dec 6, 2023

Add getEnclosingRoot API to FileSystem interface and all implementations #6196

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16791. Add getEnclosingRoot() API to filesystem interface and im… #6262

HDFS-16791. Add getEnclosingRoot() API to filesystem interface and im… #6262

mccormickt12 commented Nov 9, 2023

mccormickt12 commented Nov 9, 2023

hadoop-yetus commented Nov 9, 2023

steveloughran commented Nov 10, 2023

hadoop-yetus commented Nov 10, 2023

hadoop-yetus commented Nov 11, 2023

hadoop-yetus commented Nov 16, 2023

mccormickt12 commented Nov 18, 2023

mccormickt12 commented Nov 18, 2023

hadoop-yetus commented Nov 19, 2023

mccormickt12 commented Nov 27, 2023

mccormickt12 commented Nov 29, 2023

goiri commented Nov 30, 2023

steveloughran commented Dec 1, 2023

steveloughran commented Dec 1, 2023

HDFS-16791. Add getEnclosingRoot() API to filesystem interface and im… #6262

HDFS-16791. Add getEnclosingRoot() API to filesystem interface and im… #6262

Conversation

mccormickt12 commented Nov 9, 2023

Description of PR

How was this patch tested?

For code changes:

mccormickt12 commented Nov 9, 2023

hadoop-yetus commented Nov 9, 2023

steveloughran commented Nov 10, 2023

hadoop-yetus commented Nov 10, 2023

hadoop-yetus commented Nov 11, 2023

hadoop-yetus commented Nov 16, 2023

mccormickt12 commented Nov 18, 2023

mccormickt12 commented Nov 18, 2023

hadoop-yetus commented Nov 19, 2023

mccormickt12 commented Nov 27, 2023

mccormickt12 commented Nov 29, 2023

goiri commented Nov 30, 2023

steveloughran commented Dec 1, 2023

steveloughran commented Dec 1, 2023