Proposal: Enable HTTP Response Recording for long running Acceptance Tests

[breathingdust]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Some AWS services can take considerable time to provision, leading to Acceptance Test run times that can take in some cases nearly an hour for a single fixture. This creates a very slow development cycle time and slow development cycles and CI suite runs.

Prototype the integration of HTTP Response Recording ie (go-vcr) into the provider Acceptance Test suite to see if this would help us improve development cycle times. A balance will need to be struck in when to use the recorded responses, and when to require a "real" response from the AWS API's. We need to improve the development cycle, but not at the expense of detecting regressions.

This library is already in use within the google provider.

Tasks

• Prototype Integration of go-vcr
• Propose usage pattern for development
• Propose usage pattern in CI

Relates #18419

[ewbankkit]

Phase 1 implemented the infrastructure required to VCR-enable a resource's acceptance tests and did so for 2 resources, aws_cloudwatch_log_group (which uses AWS SDK for Go v1) and aws_scheduler_schedule (which uses AWS SDK for Go v1).

As most acceptance testing configurations contain more than 1 resource type (e.g. many configurations contains aws_iam_role and aws_vpc resources), to get any significant value from go-vcr we need to enable VCR everywhere.
This will be the focus of Phase 2:

• Complete Graham's work converting all CRUD handlers to WithoutTimeout(context.Context, ...) variants: provider: Switch Resources To Terraform Plugin SDK V2 WithoutTimeout Signatures #15090
• Complete migration of all resources to self-registration mechanism to allow CRUD handler wrapping and custom context: Proposal: Service and resource (and data source) self-registration #26626
• Handle alternate providers in acceptance tests (map[string]map[string]*conns.AWSClient) with the default having a key of ""?
• Add VCR flags to the custom context
• Replace stateConf.WaitForState with our own version that doesn't sleep if VCR is replaying
• Swap all acctest Random functions
• Swap all acctest.Provider.Meta().(*conns.AWSClient) to acctest.ProviderMeta(t), especially testAccCheck...Destroy and testAccCheck...Exists
• Look at acctest.PreCheck(), particularly when Provider.Configure is called
• cassette.ErrInteractionNotFound error handling (when a request doesn't match a recorded interaction) for all AWS services
• Change all resource.Test and resource.ParallelTest to acctest.Test and acctest.ParallelTest
• Delayed creation of all AWS clients and store in map[string]any
• Restore non-standard Region test providers replaced with acctest.PreCheckRegion() in Tech debt: Replace service-specific Region acceptance tests with acctest.PreCheckRegion #29964