HADOOP-18328. Add documentation for S3A support on S3 Outposts #5976

yutingc · 2023-08-23T01:06:55Z

Description of PR

As follow up for #4533, we're providing more documentation around S3 on Outposts.

How was this patch tested?

Did both manual testing on an outpost and integration tests for #4533.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

Add a section for configure AP on Outpost

Update index.md to include S3A on Outpost

hadoop-yetus · 2023-08-23T03:06:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 22s		trunk passed
+1 💚	mvnsite	0m 44s		trunk passed
+1 💚	shadedclient	78m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 30s		the patch passed
-1 ❌	blanks	0m 1s	/blanks-eol.txt	The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	mvnsite	0m 32s		the patch passed
+1 💚	shadedclient	33m 45s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		117m 57s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5976/1/artifact/out/Dockerfile
GITHUB PR	#5976
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint
uname	Linux 0f4dd386bf40 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b48a7fe`
Max. process+thread count	555 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5976/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-08-23T07:01:02Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 20s		trunk passed
+1 💚	mvnsite	0m 45s		trunk passed
+1 💚	shadedclient	78m 51s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 32s		the patch passed
+1 💚	shadedclient	33m 37s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 41s		The patch does not generate ASF License warnings.
		118m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5976/2/artifact/out/Dockerfile
GITHUB PR	#5976
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint
uname	Linux c6e1e9925e47 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e40a13f`
Max. process+thread count	557 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5976/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran

seems good, some minor changes

was any of the text lifted from the AWS docs? we aren't allowed to do that.
is there any extra troubleshooting needed to be covered here/in the troubleshooting doc?

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

Address comments. Shorten the description and add imp[rove formatting.

yutingc · 2023-08-23T18:09:59Z

seems good, some minor changes

was any of the text lifted from the AWS docs? we aren't allowed to do that.

is there any extra troubleshooting needed to be covered here/in the troubleshooting doc?

Thanks @steveloughran for the review.
We revised the introduction part, cut the text from AWS docs and make it more concise.
No extra troubleshooting is needed at the moment.

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

hadoop-yetus · 2023-08-23T20:07:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 41s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 7s		trunk passed
+1 💚	mvnsite	0m 44s		trunk passed
+1 💚	shadedclient	78m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
-1 ❌	blanks	0m 1s	/blanks-eol.txt	The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	mvnsite	0m 31s		the patch passed
+1 💚	shadedclient	33m 48s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 40s		The patch does not generate ASF License warnings.
		117m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5976/3/artifact/out/Dockerfile
GITHUB PR	#5976
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint
uname	Linux fcdaa7970acc 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `62dba79`
Max. process+thread count	555 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5976/3/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-08-23T22:30:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 13s		trunk passed
+1 💚	mvnsite	0m 46s		trunk passed
+1 💚	shadedclient	77m 44s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	mvnsite	0m 31s		the patch passed
+1 💚	shadedclient	33m 50s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 41s		The patch does not generate ASF License warnings.
		118m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5976/4/artifact/out/Dockerfile
GITHUB PR	#5976
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint
uname	Linux 5f66956a02e5 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `14bcf8c`
Max. process+thread count	554 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5976/4/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-08-23T23:29:03Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 27s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 22s		trunk passed
+1 💚	mvnsite	0m 33s		trunk passed
+1 💚	shadedclient	52m 49s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 22s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 24s		the patch passed
+1 💚	shadedclient	20m 22s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		77m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5976/5/artifact/out/Dockerfile
GITHUB PR	#5976
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint
uname	Linux 6de97ba37ccd 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2e267e5`
Max. process+thread count	560 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5976/5/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran

LGTM
+1

Contributed by Sotetsu Suzugamine HADOOP-18328. Add documentation for S3A support on S3 Outposts (#5976) Contributed by Yuting Chen

…e#5976) Contributed by Yuting Chen

yutingc added 2 commits August 22, 2023 17:59

Update index.md to include S3A on Outpost

273afb5

Add a section for configure AP on Outpost

Merge pull request #1 from yutingc/yutingc-patch-1

b48a7fe

Update index.md to include S3A on Outpost

github-actions bot added trunk TOOLS AWS labels Aug 23, 2023

Update index.md

e40a13f

steveloughran requested changes Aug 23, 2023

View reviewed changes

Update index.md

62dba79

Address comments. Shorten the description and add imp[rove formatting.

simbadzina reviewed Aug 23, 2023

View reviewed changes

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md Outdated Show resolved Hide resolved

Update index.md to address comment

14bcf8c

simbadzina approved these changes Aug 23, 2023

View reviewed changes

remove white space

2e267e5

simbadzina approved these changes Aug 23, 2023

View reviewed changes

steveloughran approved these changes Aug 24, 2023

View reviewed changes

steveloughran merged commit ce5bc48 into apache:trunk Aug 24, 2023
2 checks passed

asfgit pushed a commit that referenced this pull request Aug 31, 2023

HADOOP-18328. S3A to support S3 on Outposts (#4533)

a3bbeba

Contributed by Sotetsu Suzugamine HADOOP-18328. Add documentation for S3A support on S3 Outposts (#5976) Contributed by Yuting Chen

jiajunmao pushed a commit to jiajunmao/hadoop-MLEC that referenced this pull request Feb 6, 2024

HADOOP-18328. Add documentation for S3A support on S3 Outposts (apach…

6921e51

…e#5976) Contributed by Yuting Chen

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-18328. Add documentation for S3A support on S3 Outposts #5976

HADOOP-18328. Add documentation for S3A support on S3 Outposts #5976

yutingc commented Aug 23, 2023

hadoop-yetus commented Aug 23, 2023

hadoop-yetus commented Aug 23, 2023

steveloughran left a comment

yutingc commented Aug 23, 2023

hadoop-yetus commented Aug 23, 2023

hadoop-yetus commented Aug 23, 2023

hadoop-yetus commented Aug 23, 2023

steveloughran left a comment

HADOOP-18328. Add documentation for S3A support on S3 Outposts #5976

HADOOP-18328. Add documentation for S3A support on S3 Outposts #5976

Conversation

yutingc commented Aug 23, 2023

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Aug 23, 2023

hadoop-yetus commented Aug 23, 2023

steveloughran left a comment

Choose a reason for hiding this comment

yutingc commented Aug 23, 2023

hadoop-yetus commented Aug 23, 2023

hadoop-yetus commented Aug 23, 2023

hadoop-yetus commented Aug 23, 2023

steveloughran left a comment

Choose a reason for hiding this comment