Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Implement a Google Cloud Datastore backend. #16431
This backend leverages Google Cloud Datastore for remote state and locking.
Note that datastore blob properties are limited to just shy of 1MB of data. The backend zlib compresses state to avoid this limitation.
Note also that Datastore queries are eventually consistent. In the context of this backend that means the workspaces returned by
You can test this backend by running the following:
$ TF_DATASTORE_TEST=true \ TF_DATASTORE_TEST_PROJECT=REDACTED \ go test \ -timeout=60s \ -parallel=4 \ -cover \ -race \ github.com/hashicorp/terraform/backend/remote-state/datastore ok github.com/hashicorp/terraform/backend/remote-state/datastore 15.454s coverage: 82.7% of statements
In the above example
added a commit
this pull request
Oct 25, 2017
Ping. I wonder if a Terraform maintainer could help set my expectations around this PR.
Specifically I'm wondering:
I'm a little worried about getting stuck in vendor-rebase-hell if this PR sits around too long. :)
Hi @negz! Thanks for working on this and sorry for the delay here.
We're currently getting into the 0.11 release period and so we've been less active here on GitHub while we work on some release-time tasks.
With that said, I think the first question most users would have is how to choose between the Cloud Storage backend and the Datastore backend given that they seem to have very similar functionality. I assume you were motivated to implement this due to a limitation of the existing Google Cloud Storage provider, so I'd love to hear more about that.
The 1MB object limit seems like a bit of a gotcha here, though it could be reasonable if this backend has other compelling benefits over the GCS backend. The Consul backend has a similar limitation, and it's particularly troublesome if you happen to hit the limit while in the middle of an operation since it's difficult to recover; there's no real alternative when working with Consul as a backend, but I'd hesitate to introduce a new provider with a similar limitation unless it has a benefit to offset it.
Another possible approach is to try to mix GCS and Datastore in a single backend, similarly to how the
Curious to hear your thoughts on this! I'm open to including this but I want to make sure we can clearly explain the tradeoff between the two Google Cloud backends and that there's a compelling reason to introduce maintenance overhead for two backends for the same vendor.
As for timeline to review: we unfortunately can't promise specific timelines since each change is different and changes like this require some background study (to understand what Datastore does and how it works) before we can give a good review. Due to us currently being engaged in release tasks this would most likely be delayed until after 0.11 is released just due to workload.
To be honest at the time I started working on this I missed that there was concurrent work happening to build a GCS backend with support for workspaces and locking, so I was simply motivated by wanting to use workspaces and locking in GCP without running my own infrastructure.
I chose the pure Datastore route because I was unaware that it was possible to do locking using GCS alone, and an S3-backend-style combination of GCS and Datastore felt overcomplicated. I had suspected the 1MB limitation might be a problem, but had assumed it was acceptable given the Consul backend faced similar issues.
To be honest I'm not sure there is much of a compelling reason. Intuitively I would expect Datastore to be the better system for storing and locking state, but in practice I think GCS is ahead here because:
In other ways they're mostly equivalent.
I'm fine with closing this given the updated GCS backend is now merged, unless anyone really wants to store their state in Datastore rather than GCS for some reason.
Thanks for this additional context @negz!
I'm sorry for the accidental colliding work here, and appreciate the time you spent working on this.
Based on what you said here, I think I agree that it would be best to stick with the existing GCS provider for now, until (and if!) we find a reason to support both. The existing provider is already well-used and tested, and to be honest we don't really have Datastore expertise on the team to be able to review this without a lot of prior research.
I'm going to close this, but again I'm grateful that you took the time to work on this, and I'm sorry for the collision of work. Thanks!