Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Document best practices for parallel testing #1

Open
radeksimko opened this issue Jun 11, 2019 · 0 comments
Open

docs: Document best practices for parallel testing #1

radeksimko opened this issue Jun 11, 2019 · 0 comments
Assignees
Labels
documentation Improvements or additions to documentation subsystem/tests Issues and feature requests related to the testing framework.

Comments

@radeksimko
Copy link
Member

Historically the testing framework didn't have any way to run acceptance tests in parallel. We used a separate tool to achieve this and we still use it today. This worked & works in context of official providers (github.com/terraform-providers/*), but not in others and most importantly not on developer's workstations.

This is one of the reasons @bflad introduced ParallelTest in hashicorp/terraform#18688

The goal is to document this with some examples from provider which already use this helper function.

There are clear benefits we can gain by parallel testing:

  • It provides a way of ensuring the provider implementation & SDK & API works correctly under real load without having to wait until end users report misbehaviours to us.
  • Terraform defaults to parallelism=10 which means that it will issue up to 10 requests in parallel by default. Parallel testing therefore reflects how users would use the provider in the wild.
  • This saves users & machines costly time and is one of essential features of Terraform.

There are however some (side) effects of this:

  • Many APIs aren't ready to process more than 1 (or N) requests at the same time, without documenting this, or even knowing this. Maintainers often learn this hard-way through duplicate IDs returned for different resources, HTTP 500 responses, or otherwise misbehaving APIs.
    • This then leads to maintainers reaching to API vendors to fix their APIs, and/or implement various workarounds in providers, e.g. mutexkv to practically prevent Terraform from sending parallel requests to the affected API.
  • Many APIs implement some form of rate limiting, to prevent abuse.
  • Many APIs have shared namespaces which expect unique resource names. Provider developers may use static names in tests, which then makes tests to fail as a result of name clash.
@radeksimko radeksimko added documentation Improvements or additions to documentation testing labels Jun 11, 2019
kmoe added a commit that referenced this issue Nov 2, 2020
@paddycarver paddycarver added subsystem/tests Issues and feature requests related to the testing framework. and removed testing labels Dec 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation subsystem/tests Issues and feature requests related to the testing framework.
Projects
None yet
Development

No branches or pull requests

4 participants