Make provider cache work with parallelized test runs

[melinath]

Current Terraform Version

Terraform v0.13.5

Use-cases

We have a large suite of tests for the terraform provider for google, many of which share one or more external providers. We need to run the tests in parallel so that they take a reasonable amount of time to complete. Currently, each test re-downloads the external providers. While this is currently fast, we are not able to consider adding new external providers (especially large ones) to the test suite since they could end up being downloaded N times, which means:

• more bandwidth usage
• more possibilities for a network blip to break a test intermittently
• more time for the overall test run

Attempted Solutions

There is a lot of discussion in #25849 about potential workarounds and how they went for folks. It sounds like there might be a viable workaround via prewarming a cache; however, it seems like it could be a useful feature for folks to have concurrency handled out of the box as well.

Proposal

Many of the commenters on #25849 say that the provider cache wasn't designed with concurrency in mind. However, concurrent test runs are a common practice at scale, and using a shared cache of external dependencies is a common way to reduce network usage and network dependencies. It would be great to have support for this in the provider cache feature.

I think this would probably all be under the hood, rather than causing any modifications to configuration files or CLI UX.

References

• parallel init failure when using plugin cache #25849