HADOOP-18234. Fix s3a access point xml examples #4309

ashutoshcipher · 2022-05-13T11:21:37Z

Description of PR

Fixed s3a access point xml examples

JIRA: HADOOP-18234

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?

dannycjones

Looks good - I see two other misses to address in this doc.

hadoop/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

Lines 1618 to 1622 in 3d46c69

    
           The `fs.s3a.accesspoint.required` property can also require all access to S3 to go through Access 
        
           Points. This has the advantage of increasing security inside a VPN / VPC as you only allow access 
        
           to known sources of data defined through Access Points. In case there is a need to access a bucket 
        
           directly (without Access Points) then you can use per bucket overrides to disable this setting on a 
        
           bucket by bucket basis i.e. `fs.s3a.{YOUR-BUCKET}.accesspoint.required`.

hadoop/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

Lines 1630 to 1634 in 3d46c69

    
           <!-- Disable it on a per-bucket basis if needed --> 
        
           <property> 
        
               <name>fs.s3a.example-bucket.accesspoint.required</name> 
        
               <value>false</value> 
        
           </property>

hadoop-yetus · 2022-05-13T12:53:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 47s		trunk passed
+1 💚	mvnsite	0m 55s		trunk passed
+1 💚	shadedclient	63m 23s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 36s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 40s		the patch passed
+1 💚	shadedclient	23m 4s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		90m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4309/1/artifact/out/Dockerfile
GITHUB PR	#4309
Optional Tests	dupname asflicense mvnsite codespell markdownlint
uname	Linux 568a338a477e 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3d46c69`
Max. process+thread count	523 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4309/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ashutoshcipher · 2022-05-13T13:47:32Z

Thanks @dannycjones for review. I have addressed your comment.

dannycjones

Thanks, looks good to me!

hadoop-yetus · 2022-05-13T15:19:55Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 29s		trunk passed
+1 💚	mvnsite	0m 54s		trunk passed
+1 💚	shadedclient	64m 13s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 37s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 40s		the patch passed
+1 💚	shadedclient	23m 34s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		91m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4309/2/artifact/out/Dockerfile
GITHUB PR	#4309
Optional Tests	dupname asflicense mvnsite codespell markdownlint
uname	Linux caf6f2a03c55 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3c9a638`
Max. process+thread count	530 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4309/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran

example changes are good. i think a whole paragraph and one of the examples can be cut. the original PR would let you declare an AP globally as well as locally, but the final implementation only supports per-bucket binding. there' s no need for the "you can also" section as it ~duplicates the paragraph and example above

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

hadoop-yetus · 2022-05-16T13:30:48Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 19s		trunk passed
+1 💚	mvnsite	0m 56s		trunk passed
+1 💚	shadedclient	63m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 37s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 39s		the patch passed
+1 💚	shadedclient	22m 34s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		90m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4309/3/artifact/out/Dockerfile
GITHUB PR	#4309
Optional Tests	dupname asflicense mvnsite codespell markdownlint
uname	Linux bc3b73402fe2 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `da52f86`
Max. process+thread count	575 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4309/3/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-05-16T13:40:46Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	13m 5s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+1 💚	mvninstall	37m 29s		trunk passed
+1 💚	mvnsite	1m 4s		trunk passed
+1 💚	shadedclient	58m 43s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 39s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 41s		the patch passed
+1 💚	shadedclient	20m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 45s		The patch does not generate ASF License warnings.
		95m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4309/4/artifact/out/Dockerfile
GITHUB PR	#4309
Optional Tests	dupname asflicense mvnsite codespell markdownlint
uname	Linux 49b51e81b86d 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2557b4c`
Max. process+thread count	549 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4309/4/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran · 2022-05-16T16:46:47Z

+1, merging

Contributed by Ashutosh Gupta Change-Id: Ib99a4c7113d60690c1301d80cd5c27a43c463db0

Contributed by Ashutosh Gupta

HADOOP-18234. Fixed s3a access point xml examples

3d46c69

dannycjones suggested changes May 13, 2022

View reviewed changes

fixed misses in doc

3c9a638

dannycjones approved these changes May 13, 2022

View reviewed changes

steveloughran requested changes May 16, 2022

View reviewed changes

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md Outdated Show resolved Hide resolved

Ashutosh Gupta added 2 commits May 16, 2022 12:59

Removed duplicate example for accesspoint.arn

da52f86

removing extra line

2557b4c

ashutoshcipher requested a review from steveloughran May 16, 2022 12:04

steveloughran changed the title ~~HADOOP-18234. Fixed s3a access point xml examples~~ HADOOP-18234. Fix s3a access point xml examples May 16, 2022

steveloughran merged commit a46ef5f into apache:trunk May 16, 2022

asfgit pushed a commit that referenced this pull request May 16, 2022

HADOOP-18234. Fix s3a access point xml examples (#4309)

c207194

Contributed by Ashutosh Gupta Change-Id: Ib99a4c7113d60690c1301d80cd5c27a43c463db0

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HADOOP-18234. Fix s3a access point xml examples (apache#4309)

222cf55

Contributed by Ashutosh Gupta

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-18234. Fix s3a access point xml examples #4309

HADOOP-18234. Fix s3a access point xml examples #4309

ashutoshcipher commented May 13, 2022

dannycjones left a comment

hadoop-yetus commented May 13, 2022

ashutoshcipher commented May 13, 2022

dannycjones left a comment

hadoop-yetus commented May 13, 2022

steveloughran left a comment

hadoop-yetus commented May 16, 2022

hadoop-yetus commented May 16, 2022

steveloughran commented May 16, 2022

	The `fs.s3a.accesspoint.required` property can also require all access to S3 to go through Access
	Points. This has the advantage of increasing security inside a VPN / VPC as you only allow access
	to known sources of data defined through Access Points. In case there is a need to access a bucket
	directly (without Access Points) then you can use per bucket overrides to disable this setting on a
	bucket by bucket basis i.e. `fs.s3a.{YOUR-BUCKET}.accesspoint.required`.

	<!-- Disable it on a per-bucket basis if needed -->
	<property>
	<name>fs.s3a.example-bucket.accesspoint.required</name>
	<value>false</value>
	</property>

HADOOP-18234. Fix s3a access point xml examples #4309

HADOOP-18234. Fix s3a access point xml examples #4309

Conversation

ashutoshcipher commented May 13, 2022

Description of PR

dannycjones left a comment

Choose a reason for hiding this comment

hadoop-yetus commented May 13, 2022

ashutoshcipher commented May 13, 2022

dannycjones left a comment

Choose a reason for hiding this comment

hadoop-yetus commented May 13, 2022

steveloughran left a comment

Choose a reason for hiding this comment

hadoop-yetus commented May 16, 2022

hadoop-yetus commented May 16, 2022

steveloughran commented May 16, 2022