Add data source for listing S3 objects #6968

YakDriver · 2018-12-22T04:39:28Z

Changes proposed in this pull request:

Add new data source called aws_s3_bucket_objects to list objects in an S3 bucket
Add documentation / tests for new data source

Example usage:

data "aws_s3_bucket_objects" "yesh" {
  bucket    = "my_bucket"
  prefix    = "prefix/of/the/objects/"
}

The result is a list of object keys: data.aws_s3_bucket_objects.yesh.keys

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccDataSourceAWSS3BucketObjects_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./... -v -parallel 20 -run=TestAccDataSourceAWSS3BucketObjects_ -timeout 120m
?   	github.com/terraform-providers/terraform-provider-aws	[no test files]
=== RUN   TestAccDataSourceAWSS3BucketObjects_basic
=== PAUSE TestAccDataSourceAWSS3BucketObjects_basic
=== RUN   TestAccDataSourceAWSS3BucketObjects_all
=== PAUSE TestAccDataSourceAWSS3BucketObjects_all
=== RUN   TestAccDataSourceAWSS3BucketObjects_prefixes
=== PAUSE TestAccDataSourceAWSS3BucketObjects_prefixes
=== RUN   TestAccDataSourceAWSS3BucketObjects_encoded
=== PAUSE TestAccDataSourceAWSS3BucketObjects_encoded
=== RUN   TestAccDataSourceAWSS3BucketObjects_maxKeys
=== PAUSE TestAccDataSourceAWSS3BucketObjects_maxKeys
=== RUN   TestAccDataSourceAWSS3BucketObjects_startAfter
=== PAUSE TestAccDataSourceAWSS3BucketObjects_startAfter
=== RUN   TestAccDataSourceAWSS3BucketObjects_fetchOwner
=== PAUSE TestAccDataSourceAWSS3BucketObjects_fetchOwner
=== CONT  TestAccDataSourceAWSS3BucketObjects_basic
=== CONT  TestAccDataSourceAWSS3BucketObjects_maxKeys
=== CONT  TestAccDataSourceAWSS3BucketObjects_all
=== CONT  TestAccDataSourceAWSS3BucketObjects_prefixes
=== CONT  TestAccDataSourceAWSS3BucketObjects_encoded
=== CONT  TestAccDataSourceAWSS3BucketObjects_startAfter
=== CONT  TestAccDataSourceAWSS3BucketObjects_fetchOwner
--- PASS: TestAccDataSourceAWSS3BucketObjects_startAfter (10.91s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_all (11.15s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_basic (11.22s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_maxKeys (11.23s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_encoded (11.27s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_fetchOwner (11.31s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_prefixes (11.37s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	11.442s

YakDriver · 2019-01-05T19:16:50Z

@bflad Is there any adjust you'd like to see on this?

AndresCidoncha · 2019-05-11T12:31:37Z

Any chance this will be merged soon? That will allow us to improve our code a lot.

YakDriver · 2019-07-31T19:18:24Z

@bflad I've done the following on this PR:

Rebase
Add note about performance if you retrieve too many keys
Add a default value for max_keys of 1000
Re-run acceptance tests

$ make testacc TESTARGS='-run=TestAccDataSourceAWSS3BucketObjects_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./... -v -parallel 20 -run=TestAccDataSourceAWSS3BucketObjects_ -timeout 120m
?   	github.com/terraform-providers/terraform-provider-aws	[no test files]
=== RUN   TestAccDataSourceAWSS3BucketObjects_basic
=== PAUSE TestAccDataSourceAWSS3BucketObjects_basic
=== RUN   TestAccDataSourceAWSS3BucketObjects_all
=== PAUSE TestAccDataSourceAWSS3BucketObjects_all
=== RUN   TestAccDataSourceAWSS3BucketObjects_prefixes
=== PAUSE TestAccDataSourceAWSS3BucketObjects_prefixes
=== RUN   TestAccDataSourceAWSS3BucketObjects_encoded
=== PAUSE TestAccDataSourceAWSS3BucketObjects_encoded
=== RUN   TestAccDataSourceAWSS3BucketObjects_maxKeys
=== PAUSE TestAccDataSourceAWSS3BucketObjects_maxKeys
=== RUN   TestAccDataSourceAWSS3BucketObjects_startAfter
=== PAUSE TestAccDataSourceAWSS3BucketObjects_startAfter
=== RUN   TestAccDataSourceAWSS3BucketObjects_fetchOwner
=== PAUSE TestAccDataSourceAWSS3BucketObjects_fetchOwner
=== CONT  TestAccDataSourceAWSS3BucketObjects_basic
=== CONT  TestAccDataSourceAWSS3BucketObjects_fetchOwner
=== CONT  TestAccDataSourceAWSS3BucketObjects_startAfter
=== CONT  TestAccDataSourceAWSS3BucketObjects_encoded
=== CONT  TestAccDataSourceAWSS3BucketObjects_all
=== CONT  TestAccDataSourceAWSS3BucketObjects_maxKeys
=== CONT  TestAccDataSourceAWSS3BucketObjects_prefixes
--- PASS: TestAccDataSourceAWSS3BucketObjects_startAfter (33.67s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_encoded (33.79s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_fetchOwner (33.92s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_all (34.07s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_prefixes (34.20s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_basic (34.38s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_maxKeys (34.39s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	34.677s

bflad

Hey again @YakDriver 👋 Excited to see this making progress! Left some feedback items below -- please reach out with any questions or if you do not have time to implement them.

aws/data_source_aws_s3_bucket_objects.go

aws/data_source_aws_s3_bucket_objects_test.go

website/docs/d/s3_bucket_objects.html.markdown

YakDriver · 2019-08-01T17:03:34Z

@bflad Eventual consistency is killing the acceptance tests. The S3 service doesn't return the correct responses until a while after s3 resource creation. The s3 object should have a wait state but that's a separate issue. 😞

bflad · 2019-08-01T17:43:57Z

@YakDriver This could be related to https://github.com/terraform-providers/terraform-provider-aws/pull/6968/files#r309787429 as well -- a workaround pattern you can use is a "two step" apply in the testing, which ensures all the resources are applied first and introduces a slight delay in the testing before trying to call the data source, e.g.

Steps: []resource.TestStep{
  {
    Config: testAccAWSDataSourceS3ObjectsConfigResources(rInt), // NOTE: contains no data source
    // Does not need Check
  },
  {
    Config: testAccAWSDataSourceS3ObjectsConfigBasic(rInt), // NOTE: contains data source
    Check: /* ... */,
  },
}

YakDriver · 2019-08-01T18:38:44Z

@bflad Genius!

$ make testacc TESTARGS='-run=TestAccDataSourceAWSS3BucketObjects_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./... -v -parallel 20 -run=TestAccDataSourceAWSS3BucketObjects_ -timeout 120m
?   	github.com/terraform-providers/terraform-provider-aws	[no test files]
=== RUN   TestAccDataSourceAWSS3BucketObjects_basic
=== PAUSE TestAccDataSourceAWSS3BucketObjects_basic
=== RUN   TestAccDataSourceAWSS3BucketObjects_all
=== PAUSE TestAccDataSourceAWSS3BucketObjects_all
=== RUN   TestAccDataSourceAWSS3BucketObjects_prefixes
=== PAUSE TestAccDataSourceAWSS3BucketObjects_prefixes
=== RUN   TestAccDataSourceAWSS3BucketObjects_encoded
=== PAUSE TestAccDataSourceAWSS3BucketObjects_encoded
=== RUN   TestAccDataSourceAWSS3BucketObjects_maxKeys
=== PAUSE TestAccDataSourceAWSS3BucketObjects_maxKeys
=== RUN   TestAccDataSourceAWSS3BucketObjects_startAfter
=== PAUSE TestAccDataSourceAWSS3BucketObjects_startAfter
=== RUN   TestAccDataSourceAWSS3BucketObjects_fetchOwner
=== PAUSE TestAccDataSourceAWSS3BucketObjects_fetchOwner
=== CONT  TestAccDataSourceAWSS3BucketObjects_basic
=== CONT  TestAccDataSourceAWSS3BucketObjects_maxKeys
=== CONT  TestAccDataSourceAWSS3BucketObjects_all
=== CONT  TestAccDataSourceAWSS3BucketObjects_encoded
=== CONT  TestAccDataSourceAWSS3BucketObjects_prefixes
=== CONT  TestAccDataSourceAWSS3BucketObjects_fetchOwner
=== CONT  TestAccDataSourceAWSS3BucketObjects_startAfter
--- PASS: TestAccDataSourceAWSS3BucketObjects_basic (51.04s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_prefixes (51.05s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_startAfter (51.30s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_maxKeys (51.31s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_encoded (51.39s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_fetchOwner (51.52s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_all (51.70s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	51.768s

bflad

Looks great, @YakDriver, thanks! 🚀

--- PASS: TestAccDataSourceAWSS3BucketObjects_basic (18.48s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_maxKeys (18.69s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_fetchOwner (18.80s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_encoded (18.81s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_prefixes (18.85s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_startAfter (18.97s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_all (19.02s)

ghost · 2019-08-07T17:38:30Z

This has been released in version 2.23.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

ghost · 2019-11-02T14:53:35Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

YakDriver force-pushed the s3-buck-objs branch from fdbcd13 to 56be4b7 Compare December 26, 2018 19:58

YakDriver force-pushed the s3-buck-objs branch from 56be4b7 to 0746877 Compare January 5, 2019 19:16

bflad added the new-data-source Introduces a new data source. label Jan 14, 2019

aeschright requested a review from a team June 25, 2019 21:47

bflad self-assigned this Jun 26, 2019

YakDriver force-pushed the s3-buck-objs branch from 0746877 to e19b001 Compare July 31, 2019 18:30

bflad suggested changes Aug 1, 2019

View reviewed changes

bflad added the waiting-response Maintainers are waiting on response from community or contributor. label Aug 1, 2019

YakDriver added 4 commits August 1, 2019 10:44

Add data source for listing s3 objects

e381543

d/aws_s3_bucket_objects: Documentation

a4f742b

d/aws_s3_bucket_objects: Add acc tests

0ed98f4

Clarify the docs, add default max keys

6f9ac1b

YakDriver force-pushed the s3-buck-objs branch from fe3a3f0 to e206791 Compare August 1, 2019 17:00

ghost removed the waiting-response Maintainers are waiting on response from community or contributor. label Aug 1, 2019

YakDriver force-pushed the s3-buck-objs branch from e206791 to 661f07f Compare August 1, 2019 17:16

YakDriver force-pushed the s3-buck-objs branch from 661f07f to f2b56cf Compare August 1, 2019 18:34

Fix adjust issues

c69f642

YakDriver force-pushed the s3-buck-objs branch from f2b56cf to c69f642 Compare August 1, 2019 18:44

bflad self-requested a review August 1, 2019 20:29

bflad added this to the v2.23.0 milestone Aug 1, 2019

bflad approved these changes Aug 1, 2019

View reviewed changes

bflad merged commit 1194fe8 into hashicorp:master Aug 1, 2019

bflad added a commit that referenced this pull request Aug 1, 2019

Update CHANGELOG for #6968

eddb052

YakDriver deleted the s3-buck-objs branch August 1, 2019 21:44

ghost locked and limited conversation to collaborators Nov 2, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add data source for listing S3 objects #6968

Add data source for listing S3 objects #6968

YakDriver commented Dec 22, 2018

YakDriver commented Jan 5, 2019

AndresCidoncha commented May 11, 2019

YakDriver commented Jul 31, 2019

bflad left a comment

YakDriver commented Aug 1, 2019 •

edited

bflad commented Aug 1, 2019

YakDriver commented Aug 1, 2019

bflad left a comment

ghost commented Aug 7, 2019

ghost commented Nov 2, 2019

Add data source for listing S3 objects #6968

Add data source for listing S3 objects #6968

Conversation

YakDriver commented Dec 22, 2018

YakDriver commented Jan 5, 2019

AndresCidoncha commented May 11, 2019

YakDriver commented Jul 31, 2019

bflad left a comment

Choose a reason for hiding this comment

YakDriver commented Aug 1, 2019 • edited

bflad commented Aug 1, 2019

YakDriver commented Aug 1, 2019

bflad left a comment

Choose a reason for hiding this comment

ghost commented Aug 7, 2019

ghost commented Nov 2, 2019

YakDriver commented Aug 1, 2019 •

edited