Multi-reader single writer state locking

[ashleydavies]

I believe what I am suggesting here would involve provider-level implementation, however I believe it would also require changes to Terraform itself so I thought it best to suggest it here first.

In addition, I should prefix this with a note that I don't think this is a particularly important or urgent feature, it's just something I think would be a nice-to-have based on my current understanding of how terraform state locks work.

Current Terraform Version

Terraform v0.13.0-rc1

Use-cases

Many Terraform plans inevitably are ignored or result in no diff; particularly regular automated drift check plan s that tend to have no diff. In addition, it's not uncommon to have automation running terraform plan on e.g. pull requests submitted against Terraform code to see if it caused a diff.

This has the problem that, if you're using locking, you either have to:

1. Turn off read locking (which seems safe but the only discussion I can find about it (https://discuss.hashicorp.com/t/why-does-a-plan-lock-the-state-file/3520) trails off in uncertainty)
2. Set a sufficient timeout on the lock that should allow other plans to complete.

One of the disadvantages of disabling read locking is that any ongoing writes (which may take a while in some cases) won't be reflected in the plan, as I understand it, so it would be nice to avoid option one, however option two is also unsatisfactory because it is (as I understand it) a waste of time to wait for a read operation to complete before doing another read operation.

Proposal

I propose an addition that would allow backends to use a multi-reader single writer locking scheme, where multiple reads can be happening in parallel, and a write operation would cause all new read requests to block until the write completes.

A timeline for how I would expect this to work would look something like:

Plan A starts, acquiring a read lock
Plan B starts, acquiring a read lock
Apply C is executed, acquiring a write lock and blocking on the read locks
Plan B completes, releasing a read lock
Plan D is executed, blocking due to the write lock
Plan A completes, releasing a read lock
Apply C starts executing
Apply C completes, releasing the write lock
Plan D starts, acquiring a read lock
Plan D completes, releasing a read lock

Here, 'a read lock' could likely be implemented as a incrementing or decrementing a counter

https://en.wikipedia.org/wiki/Readers%E2%80%93writer_lock goes into great detail on the subject

References

I had a dig around and couldn't find any similar suggestions or discussions, although apologies in advance if this is a case of my search-fu not being strong enough.