Historically the testing framework didn't have any way to run acceptance tests in parallel. We used a separate tool to achieve this and we still use it today. This worked & works in context of official providers (github.com/terraform-providers/*), but not in others and most importantly not on developer's workstations.
The goal is to document this with some examples from provider which already use this helper function.
There are clear benefits we can gain by parallel testing:
It provides a way of ensuring the provider implementation & SDK & API works correctly under real load without having to wait until end users report misbehaviours to us.
Terraform defaults to parallelism=10 which means that it will issue up to 10 requests in parallel by default. Parallel testing therefore reflects how users would use the provider in the wild.
This saves users & machines costly time and is one of essential features of Terraform.
There are however some (side) effects of this:
Many APIs aren't ready to process more than 1 (or N) requests at the same time, without documenting this, or even knowing this. Maintainers often learn this hard-way through duplicate IDs returned for different resources, HTTP 500 responses, or otherwise misbehaving APIs.
This then leads to maintainers reaching to API vendors to fix their APIs, and/or implement various workarounds in providers, e.g. mutexkv to practically prevent Terraform from sending parallel requests to the affected API.
Many APIs implement some form of rate limiting, to prevent abuse.
Many APIs have shared namespaces which expect unique resource names. Provider developers may use static names in tests, which then makes tests to fail as a result of name clash.
The text was updated successfully, but these errors were encountered: