HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS #4750

ashutoshcipher · 2022-08-16T21:44:24Z

Description of PR

Add GETFILEBLOCKLOCATIONS operation to HttpFS.

How was this patch tested?

UT

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2022-08-17T13:25:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 2s		codespell was not available.
+0 🆗	detsecrets	0m 2s		detect-secrets was not available.
+0 🆗	xmllint	0m 2s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 31s		Maven dependency ordering for branch
+1 💚	mvninstall	27m 6s		trunk passed
+1 💚	compile	6m 45s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	6m 10s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 37s		trunk passed
+1 💚	mvnsite	3m 48s		trunk passed
+1 💚	javadoc	2m 53s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	3m 8s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	7m 32s		trunk passed
+1 💚	shadedclient	22m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 32s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 49s		the patch passed
+1 💚	compile	6m 41s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	6m 41s		the patch passed
+1 💚	compile	6m 18s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	6m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 23s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 5 new + 419 unchanged - 1 fixed = 424 total (was 420)
+1 💚	mvnsite	3m 13s		the patch passed
+1 💚	javadoc	2m 16s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 37s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	7m 26s		the patch passed
+1 💚	shadedclient	21m 22s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 39s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	242m 20s		hadoop-hdfs in the patch passed.
+1 💚	unit	7m 1s		hadoop-hdfs-httpfs in the patch passed.
+1 💚	asflicense	1m 11s		The patch does not generate ASF License warnings.
		408m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/1/artifact/out/Dockerfile
GITHUB PR	#4750
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 9222d08d5290 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b78a873`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/1/testReport/
Max. process+thread count	3263 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ashutoshcipher · 2022-08-17T16:34:39Z

@amahussein @jojochuang - Can you please help in reviewing this PR? Thanks.

ashutoshcipher · 2022-09-13T03:33:38Z

@ferhui - Can you please help in reviewing this. Thanks.

ferhui · 2022-09-13T03:46:24Z

@ashutoshcipher Thanks for involving me. Will look into it when I am available. One or two days later.

ashutoshcipher · 2022-09-13T03:50:29Z

@ashutoshcipher Thanks for involving me. Will look into it when I am available. One or two days later.

@ferhui Thank you so much. Please take your time.

ZanderXu

Thanks @ashutoshcipher for your good job. Left some comments, please fix them. Thanks~

...-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java

...oop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java

...-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java

...ject/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java

...op-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestJsonUtilClient.java

hadoop-yetus · 2022-09-23T05:08:14Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 52s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 57s		trunk passed
+1 💚	compile	7m 4s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	6m 22s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 32s		trunk passed
+1 💚	mvnsite	3m 32s		trunk passed
+1 💚	javadoc	2m 36s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	3m 0s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	7m 18s		trunk passed
+1 💚	shadedclient	23m 21s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 40s		the patch passed
+1 💚	compile	6m 49s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	6m 49s		the patch passed
+1 💚	compile	6m 16s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	6m 16s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 18s		hadoop-hdfs-project: The patch generated 0 new + 417 unchanged - 1 fixed = 417 total (was 418)
+1 💚	mvnsite	2m 55s		the patch passed
+1 💚	javadoc	2m 3s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 26s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	7m 17s		the patch passed
+1 💚	shadedclient	23m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 29s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	333m 39s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
-1 ❌	unit	0m 50s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt	hadoop-hdfs-httpfs in the patch failed.
+1 💚	asflicense	0m 56s		The patch does not generate ASF License warnings.
		496m 18s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.mover.TestMover

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/2/artifact/out/Dockerfile
GITHUB PR	#4750
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 8305deff00da 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f28c025`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/2/testReport/
Max. process+thread count	2327 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-09-23T20:28:36Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 59s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 51s		trunk passed
+1 💚	compile	6m 47s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	6m 26s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 33s		trunk passed
+1 💚	mvnsite	3m 33s		trunk passed
+1 💚	javadoc	2m 39s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	3m 10s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	7m 34s		trunk passed
+1 💚	shadedclient	23m 52s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 52s		the patch passed
+1 💚	compile	7m 6s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	7m 6s		the patch passed
+1 💚	compile	6m 40s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	6m 40s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 29s		hadoop-hdfs-project: The patch generated 0 new + 417 unchanged - 1 fixed = 417 total (was 418)
+1 💚	mvnsite	3m 29s		the patch passed
+1 💚	javadoc	2m 10s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 34s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	8m 6s		the patch passed
+1 💚	shadedclient	24m 15s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 28s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	352m 25s		hadoop-hdfs in the patch passed.
+1 💚	unit	9m 2s		hadoop-hdfs-httpfs in the patch passed.
+1 💚	asflicense	0m 55s		The patch does not generate ASF License warnings.
		527m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/3/artifact/out/Dockerfile
GITHUB PR	#4750
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 0ed9f32fb78e 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b8d0d52`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/3/testReport/
Max. process+thread count	2313 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ashutoshcipher · 2022-09-23T21:17:22Z

@ferhui - Please help with review. Thanks.

ashutoshcipher · 2022-10-01T17:00:06Z

@ZanderXu @ferhui - Please help in review. Thanks.

ashutoshcipher · 2022-10-04T21:06:17Z

@ZanderXu @ferhui - Please help in review whenever you get chance. Thanks.

ZanderXu

Thanks @ashutoshcipher for your reminder and sorry for the later response.

Leave some minor comments.

BTW, the new checkstyle has increased the maximum number of characters on a single line from 80 to 100, so some modifications can be moved to one single line.

...-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java

ZanderXu · 2022-09-29T11:48:55Z

...hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java

+    long length = ((Number) m.get("length")).longValue();
+    long offset = ((Number) m.get("offset")).longValue();
+    boolean corrupt = Boolean.getBoolean(m.get("corrupt").toString());
+    String[] storageIds = toStringArray(getList(m, "storageIds"));


If you want use this storageIds, why not modify the serialize code to add it? here

Thanks for suggestion. I think current way looks fine too. Any specific reason to modify?

I just think that it's an invalidate code line. Because the serialized code didn't serialize it. Not block this PR.

...s-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java

ZanderXu · 2022-10-05T01:14:25Z

...s-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java

+      }
+    } catch (RemoteException e) {
+      // parsing the exception is needed only if the client thinks the service is compatible
+      if (isServerHCFSCompatible && isGetFileBlockLocationsException(e)) {


Can add one UT to test this fallback?

...s-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java

ZanderXu · 2022-10-05T01:31:19Z

...ct/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java

-import org.eclipse.jetty.webapp.WebAppContext;
-
-import java.io.File;
-import java.io.FileOutputStream;


Rollback this modification?

Its just import optimization. may be we can keep this. What do you think?

It's ok. But I like to minimize modification.

...ct/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java

ZanderXu · 2022-10-05T01:33:45Z

...ct/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java

+    if (!this.isLocalFS()) {
+      FileSystem fs = this.getHttpFSFileSystem();
+      testFile = new Path(getProxiedFSTestDir(), "singleBlock.txt");
+      DFSTestUtil.createFile(fs, testFile, (long) 1, (short) 1, 0L);


Can remove the redundant cast (long).

...ject/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java

hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java

hadoop-yetus · 2022-10-06T21:37:04Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	16m 11s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 11s		trunk passed
+1 💚	compile	6m 12s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	5m 54s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 34s		trunk passed
+1 💚	mvnsite	3m 50s		trunk passed
+1 💚	javadoc	2m 43s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	3m 29s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	7m 26s		trunk passed
+1 💚	shadedclient	20m 32s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 43s		the patch passed
+1 💚	compile	6m 5s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	6m 5s		the patch passed
+1 💚	compile	5m 43s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	5m 43s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 13s		hadoop-hdfs-project: The patch generated 0 new + 417 unchanged - 1 fixed = 417 total (was 418)
+1 💚	mvnsite	2m 59s		the patch passed
+1 💚	javadoc	2m 8s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 39s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
-1 ❌	spotbugs	3m 27s	/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html	hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	20m 24s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 36s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	249m 24s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
-1 ❌	unit	6m 44s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt	hadoop-hdfs-httpfs in the patch passed.
+1 💚	asflicense	1m 12s		The patch does not generate ASF License warnings.
		410m 32s

Reason	Tests
SpotBugs	module:hadoop-hdfs-project/hadoop-hdfs
	There is an apparent infinite recursive loop in org.apache.hadoop.hdfs.web.JsonUtil.toJsonString(BlockLocation[]) At JsonUtil.java:recursive loop in org.apache.hadoop.hdfs.web.JsonUtil.toJsonString(BlockLocation[]) At JsonUtil.java:[line 712]
Failed junit tests	hadoop.fs.TestSWebHdfsFileContextMainOperations
	hadoop.hdfs.web.TestWebHdfsFileSystemContract
	hadoop.hdfs.web.TestWebHDFS
	hadoop.hdfs.web.TestWebHdfsUrl
	hadoop.hdfs.server.namenode.ha.TestObserverNode
	hadoop.fs.TestWebHdfsFileContextMainOperations
	hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/4/artifact/out/Dockerfile
GITHUB PR	#4750
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 8d1a6cac5c99 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `d19311b`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/4/testReport/
Max. process+thread count	2869 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-10-08T03:46:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 55s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	16m 31s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 45s		trunk passed
+1 💚	compile	7m 3s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	6m 26s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 33s		trunk passed
+1 💚	mvnsite	3m 33s		trunk passed
+1 💚	javadoc	2m 36s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 57s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	7m 23s		trunk passed
+1 💚	shadedclient	23m 29s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 45s		the patch passed
+1 💚	compile	6m 39s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	6m 39s		the patch passed
+1 💚	compile	6m 12s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	6m 12s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 17s		hadoop-hdfs-project: The patch generated 0 new + 417 unchanged - 1 fixed = 417 total (was 418)
+1 💚	mvnsite	2m 56s		the patch passed
+1 💚	javadoc	2m 2s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 31s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	7m 18s		the patch passed
+1 💚	shadedclient	23m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 26s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	356m 40s		hadoop-hdfs in the patch passed.
-1 ❌	unit	9m 32s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt	hadoop-hdfs-httpfs in the patch passed.
+1 💚	asflicense	0m 55s		The patch does not generate ASF License warnings.
		529m 33s

Reason	Tests
Failed junit tests	hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/5/artifact/out/Dockerfile
GITHUB PR	#4750
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux b1f609b50d6e 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c6761af`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/5/testReport/
Max. process+thread count	2199 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ashutoshcipher · 2022-10-08T11:23:58Z

hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem

Is passing in local. Will trigger jenkins once again after getting current code being reviewed

ashutoshcipher · 2022-10-08T11:25:44Z

Hi @ZanderXu - Can you please help in reviewing ? Thank you so much.

ZanderXu

LGTM. Thanks @ashutoshcipher .

ZanderXu · 2022-10-09T02:22:50Z

...hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java

+    long length = ((Number) m.get("length")).longValue();
+    long offset = ((Number) m.get("offset")).longValue();
+    boolean corrupt = Boolean.getBoolean(m.get("corrupt").toString());
+    String[] storageIds = toStringArray(getList(m, "storageIds"));


I just think that it's an invalidate code line. Because the serialized code didn't serialize it. Not block this PR.

ZanderXu · 2022-10-09T02:27:36Z

...ct/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java

-import org.eclipse.jetty.webapp.WebAppContext;
-
-import java.io.File;
-import java.io.FileOutputStream;


It's ok. But I like to minimize modification.

ashutoshcipher · 2022-10-09T07:27:36Z

Thanks @ZanderXu for review.

I just think that it's an invalidate code line. Because the serialized code didn't serialize it. Not block this PR.

Thanks for mentioning not being a blocker in this PR.

It's ok. But I like to minimize modification.

Reverted the import changes in my latest commit.

ashutoshcipher · 2022-10-09T08:33:56Z

Thanks @ZanderXu for your review and final approval :)

hadoop-yetus · 2022-10-09T16:14:46Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 16s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 49s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 51s		trunk passed
+1 💚	compile	6m 56s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	6m 26s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 34s		trunk passed
+1 💚	mvnsite	3m 29s		trunk passed
+1 💚	javadoc	2m 31s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 58s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	7m 19s		trunk passed
+1 💚	shadedclient	23m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 38s		the patch passed
+1 💚	compile	6m 44s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	6m 44s		the patch passed
+1 💚	compile	6m 17s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	6m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 17s		hadoop-hdfs-project: The patch generated 0 new + 418 unchanged - 1 fixed = 418 total (was 419)
+1 💚	mvnsite	2m 54s		the patch passed
+1 💚	javadoc	2m 2s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 27s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	7m 17s		the patch passed
+1 💚	shadedclient	23m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 27s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	342m 42s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	unit	8m 31s		hadoop-hdfs-httpfs in the patch passed.
+1 💚	asflicense	0m 56s		The patch does not generate ASF License warnings.
		528m 28s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.ha.TestObserverNode

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/6/artifact/out/Dockerfile
GITHUB PR	#4750
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 75e69096caae 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `91a5192`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/6/testReport/
Max. process+thread count	2287 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4750/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ashutoshcipher · 2022-10-09T17:19:59Z

TestObserverNode failures are not related and didnt fail trying multiple times in local.

ZanderXu · 2022-10-10T01:55:44Z

TestObserverNode failures are not related and didnt fail trying multiple times in local.

Yeah, hadoop.hdfs.server.namenode.ha.TestObserverNode also failed in other PRs. And I'm trying to locate the root cause. Just ignore it here.

ashutoshcipher · 2022-10-10T02:02:51Z

Thanks @ZanderXu for confirming. I have created JIRA for it - HDFS-16801. Please feel to take it over incase you find the root cause. Meanwhile I will also try looking into it. Thanks.

ashutoshcipher · 2022-10-12T16:20:20Z

Many congratulations @ZanderXu for committing a committer 🎉 Well deserved and great work. 😃

ashutoshcipher · 2022-10-17T10:53:04Z

@ZanderXu - Can we commit/merge this?

ZanderXu · 2022-10-17T10:56:50Z

Merged into trunk. Thanks @ashutoshcipher for your contribution.

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS

b78a873

ashutoshcipher marked this pull request as ready for review August 17, 2022 16:32

ashutoshcipher changed the title ~~[WIP] HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS~~ HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS Aug 18, 2022

ZanderXu reviewed Sep 21, 2022

View reviewed changes

Fixes

f28c025

fixed

b8d0d52

ZanderXu reviewed Oct 5, 2022

View reviewed changes

Addressing comments

d19311b

Addressing failures

c6761af

ashutoshcipher requested a review from ZanderXu October 8, 2022 11:25

ZanderXu reviewed Oct 9, 2022

View reviewed changes

Reverting import changes

91a5192

ZanderXu approved these changes Oct 9, 2022

View reviewed changes

ZanderXu merged commit 9a8aff6 into apache:trunk Oct 17, 2022

aajisaka mentioned this pull request Nov 21, 2022

HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS. #3322

Closed

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS (apache#4750)

fe335ad

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS #4750

HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS #4750

Conversation

ashutoshcipher commented Aug 16, 2022 • edited Loading

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Aug 17, 2022

ashutoshcipher commented Aug 17, 2022

ashutoshcipher commented Sep 13, 2022

ferhui commented Sep 13, 2022

ashutoshcipher commented Sep 13, 2022

ZanderXu left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Sep 23, 2022

hadoop-yetus commented Sep 23, 2022

ashutoshcipher commented Sep 23, 2022

ashutoshcipher commented Oct 1, 2022

ashutoshcipher commented Oct 4, 2022

ZanderXu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hadoop-yetus commented Oct 6, 2022

hadoop-yetus commented Oct 8, 2022

ashutoshcipher commented Oct 8, 2022

ashutoshcipher commented Oct 8, 2022

ZanderXu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashutoshcipher commented Oct 9, 2022

ashutoshcipher commented Oct 9, 2022

hadoop-yetus commented Oct 9, 2022

ashutoshcipher commented Oct 9, 2022

ZanderXu commented Oct 10, 2022

ashutoshcipher commented Oct 10, 2022

ashutoshcipher commented Oct 12, 2022

ashutoshcipher commented Oct 17, 2022

ZanderXu commented Oct 17, 2022

ashutoshcipher commented Aug 16, 2022 •

edited

Loading