Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-15563 S3Guard to create on-demand DDB tables #879

Closed
wants to merge 4 commits into from
Closed

HADOOP-15563 S3Guard to create on-demand DDB tables #879

wants to merge 4 commits into from

Commits on Jun 6, 2019

  1. HADOOP-15563 Full S3Guard Support for on-demand DDB tables 

    * If you create a table with read capacity == write capacity == 0, you get an on-demand table.
* change the defaults in Constants and core-defaults to be zero. You get an on-demand table by default.
* S3Guard docs reworked to cover the topic, including the updated defaults and why you should go to on-demand; also mentioned in the troubleshooting section.

It'd be nice if set-capacity could switch a table to on-demand, but the latest AWS SDK doesn't let you do that, even though the low level REST API does.

ITestS3GuardToolDynamoDB l
* ifecycle test creates an on-demand table and skips changing its capacity.
* Fix HADOOP-16187 ITestS3GuardToolDynamoDB test failures: test wasn't clearing enough per-bucket options.

ITestDynamoDBMetadataStore: review of every test case in so that
* Capacity is always set to on-demand before table creation
* Every test always cleans up its tables, even on assert failures.

Those tests have tended to leak tables in the past; these changes should eliminate it on uninterrupted test runs, and with on-demand capacity you don't get billed much until the next time you do an audit and delete of tables.

Also:

- removed @unstable attribute from those Constants which I consider having been shipping too long to be changed.
- moved test-only S3GUARD_DDB_TEST_TABLE_NAME_KEY key to S3ATestConstants. It was tagged Unstable, and should be in the right file.

Change-Id: Ia44559354666db8027e574eab97983167ed930bf
(cherry picked from commit d695ba4e9935928c211e4fd52d635c3f03f8c5d7)
    @steveloughran
    steveloughran committed Jun 6, 2019
    Copy the full SHA
    a70a679 View commit details
    Browse the repository at this point in the history

  2. HADOOP-15563 address gabor's comments 

    Change-Id: I62422be23b8a30d2222e3c4d6354cb20d765260c
(cherry picked from commit 9904c9a8a885a4c4aa715058b0a1294105b6b0d9)
    @steveloughran
    steveloughran committed Jun 6, 2019
    Copy the full SHA
    80c3292 View commit details
    Browse the repository at this point in the history

  3. HADOOP-15183 Tuning the ITestDynamoDBMetaStore tests 

    +DynamoDBMetaStore provisionTable() has the r=0 w=0 capacity check pushed down as it is visible for testing.

Change-Id: Ide149515f8b212381087e84b842c5582c5cdab02
(cherry picked from commit 476d9c72c0f04de37d85893b5efdba24f9d232cf)
    @steveloughran
    steveloughran committed Jun 6, 2019
    Copy the full SHA
    a17aaef View commit details
    Browse the repository at this point in the history

  4. HADOOP-15563 address checkstyle complaints (line length, mostly) 

    Change-Id: I6a7a104b6fe2bb041109c6055d2245636ca42067
    @steveloughran
    steveloughran committed Jun 6, 2019
    Copy the full SHA
    74dab46 View commit details
    Browse the repository at this point in the history