Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-17480. Document that AWS S3 is consistent and that S3Guard is not needed #2636

Merged

Conversation

steveloughran
Copy link
Contributor

  • s3guard docs have an intro saying "no need"
  • move to past tense in architecture docs in describing problems
  • committer docs to emphasise they are still needed
  • troubleshooting cuts out some of the observed consistency problems as they
    won't happen any more

…not needed

Change-Id: I2b1c2aa16205cf39cb271bf8ea348ee287530d97
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 26m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ trunk Compile Tests _
+1 💚 mvninstall 33m 9s trunk passed
+1 💚 mvnsite 0m 44s trunk passed
+1 💚 shadedclient 49m 25s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 37s the patch passed
+1 💚 mvnsite 0m 33s the patch passed
-1 ❌ whitespace 0m 0s /whitespace-eol.txt The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚 shadedclient 14m 56s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 asflicense 0m 35s The patch does not generate ASF License warnings.
94m 27s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2636/1/artifact/out/Dockerfile
GITHUB PR #2636
Optional Tests dupname asflicense mvnsite markdownlint
uname Linux c3432eeb2f21 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 06fef5e
Max. process+thread count 706 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2636/1/console
versions git=2.17.1 maven=3.6.0
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Copy link
Member

@liuml07 liuml07 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

How to move off S3Guard, given it is no longer needed.

1. Unset the option `fs.s3a.metadatastore.impl` globally/for all buckets for which it
was selected.
Copy link
Member

@liuml07 liuml07 Jan 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to remind people to delete those S3Guard DynamoDB tables as provisioned IPOS will still cost money. I think we can also use the s3guard command line tool to destroy, e.g. hadoop s3guard destroy s3a://guarded-table/.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

once you remove the s3guard settings it becomes harder to work out the table to destroy, you need to use dynamodb URLs. Easiest just to use the console, especially as it lets you see if there are any orphan tables around.

@steveloughran steveloughran merged commit 06a5d34 into apache:trunk Jan 25, 2021
asfgit pushed a commit that referenced this pull request Jan 25, 2021
…not needed (#2636)

Contributed by Steve Loughran.

Change-Id: I775e3ee7b60665240ec621859c337b053f747a49
@steveloughran steveloughran deleted the s3/HADOOP-17480-s3guard-deprecated branch January 25, 2021 15:08
jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023
… S3Guard is not needed (apache#2636)

Contributed by Steve Loughran.

Change-Id: I775e3ee7b60665240ec621859c337b053f747a49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants