HADOOP-17402. Add GCS config to the core-site #2638

ravwojdyla · 2021-01-21T15:56:36Z

hadoop-common-project/hadoop-common/src/main/resources/core-default.xml

steveloughran · 2021-01-31T16:06:51Z

Assuming the gcs init code initialises all its properties with meaningful defaults, I'd rather leave out everything gcs-related except for the base gcs & viewfs bindings, and that list of excluded properties.

putting it into core-default means that when the gcs team change any default value in their own code, the FS will still be inited with whatever was the default when that version of hadoop was built.

let's keep this to a minimum

ravwojdyla · 2021-01-31T22:46:46Z

@steveloughran sounds good, updated in 16a487e

medb · 2021-02-01T01:08:08Z

...roject/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java

@@ -142,6 +141,7 @@ public void initializeMemberVariables() {
    xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.webhdfs.impl");
    xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.wasb.impl");
    xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.swift.impl");
+    xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.gs.impl");


Seems like this list should be sorted alphabetically?

@medb I can certainly change this, but it's already not sorted due to swift.

yeah, but lets try not to make things worse. It's like how we try and follow ordering/layout rules on imports. Strive to to better than your predecessors

@steveloughran done, also fixed the swift order.

jojochuang · 2021-02-22T08:49:28Z

hadoop-common-project/hadoop-common/src/main/resources/core-default.xml

+<property>
+  <name>fs.viewfs.overload.scheme.target.gs.impl</name>
+  <value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS</value>
+  <description>The HttpFileSystem for view file system overload scheme


nit: change "HttpFileSystem" to "Google Cloud Storage file system"

@jojochuang done.

hadoop-yetus · 2021-06-29T08:33:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	29m 49s		trunk passed
+1 💚	compile	21m 41s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	18m 36s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 4s		trunk passed
+1 💚	mvnsite	1m 38s		trunk passed
+1 💚	javadoc	1m 11s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 43s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 27s		trunk passed
+1 💚	shadedclient	15m 46s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 56s		the patch passed
+1 💚	compile	21m 16s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	21m 16s		the patch passed
+1 💚	compile	18m 48s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	18m 48s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 2s		the patch passed
+1 💚	mvnsite	1m 31s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 6s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 42s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 38s		the patch passed
+1 💚	shadedclient	16m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 5s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		178m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2638/6/artifact/out/Dockerfile
GITHUB PR	#2638
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle
uname	Linux d09e20fc7b3d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a57700c`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2638/6/testReport/
Max. process+thread count	1266 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2638/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran

+1

steveloughran · 2021-07-05T20:08:06Z

Merged to trunk. @ravwojdyla : if you cherrypick this to branch-3.3 and submit a new PR there, I'll merge as soon as yetus is happy. Leaving the JIRA open for now

thanks!

Contributed by Rafal Wojdyla

ravwojdyla · 2021-07-06T18:10:45Z

@steveloughran thanks! Done in: #3180.

Contributed by Rafal Wojdyla

ravwojdyla commented Jan 21, 2021

View reviewed changes

hadoop-common-project/hadoop-common/src/main/resources/core-default.xml Outdated Show resolved Hide resolved

ravwojdyla force-pushed the rav/add_gcs_core branch from e18c220 to 8b4ca91 Compare January 21, 2021 21:52

ravwojdyla force-pushed the rav/add_gcs_core branch from 8b4ca91 to 16a487e Compare January 31, 2021 22:46

medb suggested changes Feb 1, 2021

View reviewed changes

medb approved these changes Feb 1, 2021

View reviewed changes

jojochuang reviewed Feb 22, 2021

View reviewed changes

ravwojdyla force-pushed the rav/add_gcs_core branch from 16a487e to 9c4af4d Compare February 23, 2021 00:18

ravwojdyla force-pushed the rav/add_gcs_core branch from 9c4af4d to 838adbb Compare March 12, 2021 06:51

Add GCS config to the core-site

a57700c

ravwojdyla force-pushed the rav/add_gcs_core branch from 838adbb to a57700c Compare June 29, 2021 05:34

jojochuang approved these changes Jun 29, 2021

View reviewed changes

ravwojdyla requested a review from steveloughran June 29, 2021 16:09

apache deleted a comment from hadoop-yetus Jul 5, 2021

steveloughran approved these changes Jul 5, 2021

View reviewed changes

steveloughran merged commit f639fbc into apache:trunk Jul 5, 2021

ravwojdyla added a commit to ravwojdyla/hadoop that referenced this pull request Jul 6, 2021

HADOOP-17402. Add GCS config to the core-site (apache#2638)

091a7ba

Contributed by Rafal Wojdyla

ravwojdyla mentioned this pull request Jul 6, 2021

HADOOP-17402. Add GCS config to the core-site (#2638) #3180

Merged

ravwojdyla deleted the rav/add_gcs_core branch July 6, 2021 18:10

steveloughran pushed a commit that referenced this pull request Jul 7, 2021

HADOOP-17402. Add GCS config to the core-site (#2638)

e3fb63f

Contributed by Rafal Wojdyla

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HADOOP-17402. Add GCS config to the core-site (apache#2638)

950c721

Contributed by Rafal Wojdyla

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-17402. Add GCS config to the core-site #2638

HADOOP-17402. Add GCS config to the core-site #2638

ravwojdyla commented Jan 21, 2021 •

edited

steveloughran commented Jan 31, 2021

ravwojdyla commented Jan 31, 2021

medb Feb 1, 2021

ravwojdyla Feb 1, 2021

steveloughran Mar 9, 2021

ravwojdyla Mar 12, 2021

jojochuang Feb 22, 2021

ravwojdyla Feb 23, 2021

hadoop-yetus commented Jun 29, 2021

steveloughran left a comment

steveloughran commented Jul 5, 2021

ravwojdyla commented Jul 6, 2021

HADOOP-17402. Add GCS config to the core-site #2638

HADOOP-17402. Add GCS config to the core-site #2638

Conversation

ravwojdyla commented Jan 21, 2021 • edited

steveloughran commented Jan 31, 2021

ravwojdyla commented Jan 31, 2021

medb Feb 1, 2021

Choose a reason for hiding this comment

ravwojdyla Feb 1, 2021

Choose a reason for hiding this comment

steveloughran Mar 9, 2021

Choose a reason for hiding this comment

ravwojdyla Mar 12, 2021

Choose a reason for hiding this comment

jojochuang Feb 22, 2021

Choose a reason for hiding this comment

ravwojdyla Feb 23, 2021

Choose a reason for hiding this comment

hadoop-yetus commented Jun 29, 2021

steveloughran left a comment

Choose a reason for hiding this comment

steveloughran commented Jul 5, 2021

ravwojdyla commented Jul 6, 2021

ravwojdyla commented Jan 21, 2021 •

edited