Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

r/glue_catalog_table - add support for specifying schema #17335

Merged
merged 8 commits into from
Feb 4, 2021
Merged

r/glue_catalog_table - add support for specifying schema #17335

merged 8 commits into from
Feb 4, 2021

Conversation

DrFaust92
Copy link
Collaborator

@DrFaust92 DrFaust92 commented Jan 28, 2021 •
edited
Loading

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #17302

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAWSGlueCatalogTable_'
--- PASS: TestAccAWSGlueCatalogTable_disappears_database (33.80s)
--- PASS: TestAccAWSGlueCatalogTable_disappears (37.91s)
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_EmptyConfigurationBlock (38.69s)
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_EmptyConfigurationBlock (38.69s)
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_SkewedInfo_EmptyConfigurationBlock (40.98s)
--- PASS: TestAccAWSGlueCatalogTable_partitionIndexesSingle (45.24s)
--- PASS: TestAccAWSGlueCatalogTable_full (46.48s)
--- PASS: TestAccAWSGlueCatalogTable_columnParameters (46.78s)
--- PASS: TestAccAWSGlueCatalogTable_basic (47.16s)
--- PASS: TestAccAWSGlueCatalogTable_partitionIndexesMultiple (48.08s)
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReferenceArn (60.65s)
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference (61.27s)
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_UpdateValues (75.00s)
--- PASS: TestAccAWSGlueCatalogTable_update_addValues (75.84s)
--- PASS: TestAccAWSGlueCatalogTable_update_replaceValues (75.95s)

@ghost ghost added size/XL Managed by automation to categorize the size of a PR. documentation Introduces or discusses updates to documentation. service/glue Issues and PRs that pertain to the glue service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. labels Jan 28, 2021
@DrFaust92 DrFaust92 marked this pull request as ready for review January 28, 2021 17:21
@DrFaust92 DrFaust92 requested a review from a team as a code owner January 28, 2021 17:21
@DrFaust92 DrFaust92 added the enhancement Requests to existing resources that expand the functionality or scope. label Jan 28, 2021
ewbankkit
ewbankkit approved these changes Jan 29, 2021
View reviewed changes
Copy link
Contributor

@ewbankkit ewbankkit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSGlueCatalogTable_' ACCTEST_PARALLELISM=2
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 2 -run=TestAccAWSGlueCatalogTable_ -timeout 120m
=== RUN   TestAccAWSGlueCatalogTable_basic
=== PAUSE TestAccAWSGlueCatalogTable_basic
=== RUN   TestAccAWSGlueCatalogTable_columnParameters
=== PAUSE TestAccAWSGlueCatalogTable_columnParameters
=== RUN   TestAccAWSGlueCatalogTable_full
=== PAUSE TestAccAWSGlueCatalogTable_full
=== RUN   TestAccAWSGlueCatalogTable_update_addValues
=== PAUSE TestAccAWSGlueCatalogTable_update_addValues
=== RUN   TestAccAWSGlueCatalogTable_update_replaceValues
=== PAUSE TestAccAWSGlueCatalogTable_update_replaceValues
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_EmptyConfigurationBlock
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_EmptyConfigurationBlock
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_EmptyConfigurationBlock
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_EmptyConfigurationBlock
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_UpdateValues
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_UpdateValues
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_SkewedInfo_EmptyConfigurationBlock
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_SkewedInfo_EmptyConfigurationBlock
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReferenceArn
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReferenceArn
=== RUN   TestAccAWSGlueCatalogTable_partitionIndexesSingle
=== PAUSE TestAccAWSGlueCatalogTable_partitionIndexesSingle
=== RUN   TestAccAWSGlueCatalogTable_partitionIndexesMultiple
=== PAUSE TestAccAWSGlueCatalogTable_partitionIndexesMultiple
=== RUN   TestAccAWSGlueCatalogTable_disappears_database
=== PAUSE TestAccAWSGlueCatalogTable_disappears_database
=== RUN   TestAccAWSGlueCatalogTable_disappears
=== PAUSE TestAccAWSGlueCatalogTable_disappears
=== CONT  TestAccAWSGlueCatalogTable_basic
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_SkewedInfo_EmptyConfigurationBlock
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_SkewedInfo_EmptyConfigurationBlock (15.62s)
=== CONT  TestAccAWSGlueCatalogTable_disappears
--- PASS: TestAccAWSGlueCatalogTable_basic (17.67s)
=== CONT  TestAccAWSGlueCatalogTable_disappears_database
--- PASS: TestAccAWSGlueCatalogTable_disappears (12.42s)
=== CONT  TestAccAWSGlueCatalogTable_partitionIndexesMultiple
--- PASS: TestAccAWSGlueCatalogTable_disappears_database (10.54s)
=== CONT  TestAccAWSGlueCatalogTable_partitionIndexesSingle
--- PASS: TestAccAWSGlueCatalogTable_partitionIndexesMultiple (17.16s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReferenceArn
--- PASS: TestAccAWSGlueCatalogTable_partitionIndexesSingle (17.03s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReferenceArn (26.63s)
=== CONT  TestAccAWSGlueCatalogTable_update_replaceValues
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference (26.68s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_UpdateValues
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_UpdateValues (29.48s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_EmptyConfigurationBlock
--- PASS: TestAccAWSGlueCatalogTable_update_replaceValues (29.66s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_EmptyConfigurationBlock
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_EmptyConfigurationBlock (14.59s)
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_EmptyConfigurationBlock (14.51s)
=== CONT  TestAccAWSGlueCatalogTable_full
=== CONT  TestAccAWSGlueCatalogTable_update_addValues
--- PASS: TestAccAWSGlueCatalogTable_full (16.98s)
=== CONT  TestAccAWSGlueCatalogTable_columnParameters
--- PASS: TestAccAWSGlueCatalogTable_update_addValues (28.95s)
--- PASS: TestAccAWSGlueCatalogTable_columnParameters (15.60s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	148.678s

@breathingdust
Copy link
Member

LGTM 🚀 Thanks @DrFaust92!

Verified Acceptance Tests in Commercial (us-west-2)

make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSGlueCatalogTable_' ACCTEST_PARALLELISM=2
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 2 -run=TestAccAWSGlueCatalogTable_ -timeout 120m
=== RUN   TestAccAWSGlueCatalogTable_basic
=== PAUSE TestAccAWSGlueCatalogTable_basic
=== RUN   TestAccAWSGlueCatalogTable_columnParameters
=== PAUSE TestAccAWSGlueCatalogTable_columnParameters
=== RUN   TestAccAWSGlueCatalogTable_full
=== PAUSE TestAccAWSGlueCatalogTable_full
=== RUN   TestAccAWSGlueCatalogTable_update_addValues
=== PAUSE TestAccAWSGlueCatalogTable_update_addValues
=== RUN   TestAccAWSGlueCatalogTable_update_replaceValues
=== PAUSE TestAccAWSGlueCatalogTable_update_replaceValues
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_EmptyConfigurationBlock
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_EmptyConfigurationBlock
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_EmptyConfigurationBlock
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_EmptyConfigurationBlock
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_UpdateValues
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_UpdateValues
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_SkewedInfo_EmptyConfigurationBlock
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_SkewedInfo_EmptyConfigurationBlock
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReferenceArn
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReferenceArn
=== RUN   TestAccAWSGlueCatalogTable_partitionIndexesSingle
=== PAUSE TestAccAWSGlueCatalogTable_partitionIndexesSingle
=== RUN   TestAccAWSGlueCatalogTable_partitionIndexesMultiple
=== PAUSE TestAccAWSGlueCatalogTable_partitionIndexesMultiple
=== RUN   TestAccAWSGlueCatalogTable_disappears_database
=== PAUSE TestAccAWSGlueCatalogTable_disappears_database
=== RUN   TestAccAWSGlueCatalogTable_disappears
=== PAUSE TestAccAWSGlueCatalogTable_disappears
=== CONT  TestAccAWSGlueCatalogTable_basic
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_SkewedInfo_EmptyConfigurationBlock
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_SkewedInfo_EmptyConfigurationBlock (13.70s)
=== CONT  TestAccAWSGlueCatalogTable_disappears
--- PASS: TestAccAWSGlueCatalogTable_basic (15.10s)
=== CONT  TestAccAWSGlueCatalogTable_disappears_database
--- PASS: TestAccAWSGlueCatalogTable_disappears_database (11.19s)
=== CONT  TestAccAWSGlueCatalogTable_partitionIndexesMultiple
--- PASS: TestAccAWSGlueCatalogTable_disappears (13.39s)
=== CONT  TestAccAWSGlueCatalogTable_partitionIndexesSingle
--- PASS: TestAccAWSGlueCatalogTable_partitionIndexesMultiple (14.79s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReferenceArn
--- PASS: TestAccAWSGlueCatalogTable_partitionIndexesSingle (14.09s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReferenceArn (20.68s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_EmptyConfigurationBlock
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference (21.10s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_UpdateValues
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_EmptyConfigurationBlock (12.14s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_EmptyConfigurationBlock
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_EmptyConfigurationBlock (11.80s)
=== CONT  TestAccAWSGlueCatalogTable_update_replaceValues
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_UpdateValues (23.83s)
=== CONT  TestAccAWSGlueCatalogTable_columnParameters
--- PASS: TestAccAWSGlueCatalogTable_columnParameters (13.10s)
=== CONT  TestAccAWSGlueCatalogTable_update_addValues
--- PASS: TestAccAWSGlueCatalogTable_update_replaceValues (24.55s)
=== CONT  TestAccAWSGlueCatalogTable_full
--- PASS: TestAccAWSGlueCatalogTable_update_addValues (23.62s)
--- PASS: TestAccAWSGlueCatalogTable_full (13.48s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	125.314s

Verified Acceptance Tests in GovCloud (us-gov-west-1), there is a failing test due to a permissions issue which only occurs in GovCloud for TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference. Will create a followup issue and link to this issue.

make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSGlueCatalogTable_' ACCTEST_PARALLELISM=2
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 2 -run=TestAccAWSGlueCatalogTable_ -timeout 120m
=== RUN   TestAccAWSGlueCatalogTable_basic
=== PAUSE TestAccAWSGlueCatalogTable_basic
=== RUN   TestAccAWSGlueCatalogTable_columnParameters
=== PAUSE TestAccAWSGlueCatalogTable_columnParameters
=== RUN   TestAccAWSGlueCatalogTable_full
=== PAUSE TestAccAWSGlueCatalogTable_full
=== RUN   TestAccAWSGlueCatalogTable_update_addValues
=== PAUSE TestAccAWSGlueCatalogTable_update_addValues
=== RUN   TestAccAWSGlueCatalogTable_update_replaceValues
=== PAUSE TestAccAWSGlueCatalogTable_update_replaceValues
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_EmptyConfigurationBlock
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_EmptyConfigurationBlock
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_EmptyConfigurationBlock
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_EmptyConfigurationBlock
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_UpdateValues
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_UpdateValues
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_SkewedInfo_EmptyConfigurationBlock
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_SkewedInfo_EmptyConfigurationBlock
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference
=== RUN   TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReferenceArn
=== PAUSE TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReferenceArn
=== RUN   TestAccAWSGlueCatalogTable_partitionIndexesSingle
=== PAUSE TestAccAWSGlueCatalogTable_partitionIndexesSingle
=== RUN   TestAccAWSGlueCatalogTable_partitionIndexesMultiple
=== PAUSE TestAccAWSGlueCatalogTable_partitionIndexesMultiple
=== RUN   TestAccAWSGlueCatalogTable_disappears_database
=== PAUSE TestAccAWSGlueCatalogTable_disappears_database
=== RUN   TestAccAWSGlueCatalogTable_disappears
=== PAUSE TestAccAWSGlueCatalogTable_disappears
=== CONT  TestAccAWSGlueCatalogTable_basic
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_SkewedInfo_EmptyConfigurationBlock
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_SkewedInfo_EmptyConfigurationBlock (9.41s)
=== CONT  TestAccAWSGlueCatalogTable_partitionIndexesMultiple
--- PASS: TestAccAWSGlueCatalogTable_basic (10.60s)
=== CONT  TestAccAWSGlueCatalogTable_update_replaceValues
--- PASS: TestAccAWSGlueCatalogTable_partitionIndexesMultiple (10.67s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_UpdateValues
--- PASS: TestAccAWSGlueCatalogTable_update_replaceValues (17.43s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_EmptyConfigurationBlock
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_EmptyConfigurationBlock (8.54s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_EmptyConfigurationBlock
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_SerDeInfo_UpdateValues (17.95s)
=== CONT  TestAccAWSGlueCatalogTable_disappears
--- PASS: TestAccAWSGlueCatalogTable_StorageDescriptor_EmptyConfigurationBlock (7.79s)
=== CONT  TestAccAWSGlueCatalogTable_disappears_database
--- PASS: TestAccAWSGlueCatalogTable_disappears (8.35s)
=== CONT  TestAccAWSGlueCatalogTable_full
--- PASS: TestAccAWSGlueCatalogTable_disappears_database (6.32s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReferenceArn
    resource_aws_glue_catalog_table_test.go:459: Step 1/2 error: Error running apply: 2021/02/03 20:06:26 [DEBUG] Using modified User-Agent: Terraform/0.12.29 HashiCorp-terraform-exec/0.12.0

        Error: error creating Glue Registry: AccessDeniedException: Operation not allowed


--- FAIL: TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReferenceArn (3.51s)
=== CONT  TestAccAWSGlueCatalogTable_partitionIndexesSingle
--- PASS: TestAccAWSGlueCatalogTable_full (9.65s)
=== CONT  TestAccAWSGlueCatalogTable_columnParameters
--- PASS: TestAccAWSGlueCatalogTable_partitionIndexesSingle (12.37s)
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference
--- PASS: TestAccAWSGlueCatalogTable_columnParameters (11.32s)
=== CONT  TestAccAWSGlueCatalogTable_update_addValues
=== CONT  TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference
    resource_aws_glue_catalog_table_test.go:429: Step 1/2 error: Error running apply: 2021/02/03 20:06:42 [DEBUG] Using modified User-Agent: Terraform/0.12.29 HashiCorp-terraform-exec/0.12.0

        Error: error creating Glue Registry: AccessDeniedException: Operation not allowed


--- FAIL: TestAccAWSGlueCatalogTable_StorageDescriptor_schemaReference (3.24s)
--- PASS: TestAccAWSGlueCatalogTable_update_addValues (19.01s)
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	88.028s
FAIL
make: *** [testacc] Error 1

@breathingdust breathingdust merged commit bcd6b6a into hashicorp:main Feb 4, 2021
@github-actions github-actions bot added this to the v3.27.0 milestone Feb 4, 2021
@breathingdust breathingdust mentioned this pull request Feb 4, 2021
Closed
@ghost
Copy link

ghost commented Feb 5, 2021

This has been released in version 3.27.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@ewbankkit ewbankkit mentioned this pull request Feb 18, 2021
Closed
@ghost
Copy link

ghost commented Mar 6, 2021

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked as resolved and limited conversation to collaborators Mar 6, 2021
@DrFaust92 DrFaust92 deleted the r/glue_table_schema branch April 15, 2021 10:36
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@ewbankkit ewbankkit ewbankkit approved these changes

Assignees

@ewbankkit ewbankkit

Labels
documentation Introduces or discusses updates to documentation. enhancement Requests to existing resources that expand the functionality or scope. service/glue Issues and PRs that pertain to the glue service. size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Milestone
v3.27.0
Development

Successfully merging this pull request may close these issues.

Terraform Feature Request - Allow creation of Glue Table to use schema from Schema Registry
3 participants
@DrFaust92 @breathingdust @ewbankkit