New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a Google Cloud Datastore backend. #16431

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
2 participants
@negz
Contributor

negz commented Oct 24, 2017

This backend leverages Google Cloud Datastore for remote state and locking.

Note that datastore blob properties are limited to just shy of 1MB of data. The backend zlib compresses state to avoid this limitation.

Note also that Datastore queries are eventually consistent. In the context of this backend that means the workspaces returned by terraform workspace list will be eventually consistent, and thus newly added or deleted workspaces may take a short time to propagate. This backend uses strongly consistent operations for all other functionality, so selecting, locking, reading, and updating state will be strongly consistent.

You can test this backend by running the following:

$ TF_DATASTORE_TEST=true \
  TF_DATASTORE_TEST_PROJECT=REDACTED \
  go test \
    -timeout=60s \
    -parallel=4 \
    -cover \
    -race \
    github.com/hashicorp/terraform/backend/remote-state/datastore
ok      github.com/hashicorp/terraform/backend/remote-state/datastore   15.454s coverage: 82.7% of statements

In the above example REDACTED must be a Google Cloud Platform project with Datastore enabled. The caller must have read/write Datastore access to said project and be authenticated.

negz added a commit to negz/terraform-provider-terraform that referenced this pull request Oct 25, 2017

negz added some commits Oct 22, 2017

Implement a Datastore backend.
Uses Google Cloud Datastore for remote state and locking. Note that datastore
blob properties are limited to just shy of 1MB of data. The backend zlib
compresses the state to avoid this limitation.

Note also that Datastore queries are eventually consistent. In the context of
this backend that means the list of available workspaces will be eventually
consistent. Explicitly referencing a workspace by name will always be strongly
consistent.
Update Google OAuth2 Client
This is necessary for compatibility with the 0.15.0 release of the Datastore
client used by the Datastore backend.
@negz

This comment has been minimized.

Contributor

negz commented Oct 31, 2017

Ping. I wonder if a Terraform maintainer could help set my expectations around this PR.

Specifically I'm wondering:

  • Is a Datastore backend something people feel is useful given the GCS backend was just updated to support state and locking?
  • What's the approximate timeline to review a new backend like this?

I'm a little worried about getting stuck in vendor-rebase-hell if this PR sits around too long. :)

@apparentlymart

This comment has been minimized.

Contributor

apparentlymart commented Oct 31, 2017

Hi @negz! Thanks for working on this and sorry for the delay here.

We're currently getting into the 0.11 release period and so we've been less active here on GitHub while we work on some release-time tasks.

With that said, I think the first question most users would have is how to choose between the Cloud Storage backend and the Datastore backend given that they seem to have very similar functionality. I assume you were motivated to implement this due to a limitation of the existing Google Cloud Storage provider, so I'd love to hear more about that.

The 1MB object limit seems like a bit of a gotcha here, though it could be reasonable if this backend has other compelling benefits over the GCS backend. The Consul backend has a similar limitation, and it's particularly troublesome if you happen to hit the limit while in the middle of an operation since it's difficult to recover; there's no real alternative when working with Consul as a backend, but I'd hesitate to introduce a new provider with a similar limitation unless it has a benefit to offset it.

Another possible approach is to try to mix GCS and Datastore in a single backend, similarly to how the s3 backend actually blends S3 (for storage) and DynamoDB (for locking). But with that said, the GCS backend does configure locking (as you saw) so the use-case here may not be so compelling. 🤔

Curious to hear your thoughts on this! I'm open to including this but I want to make sure we can clearly explain the tradeoff between the two Google Cloud backends and that there's a compelling reason to introduce maintenance overhead for two backends for the same vendor.

As for timeline to review: we unfortunately can't promise specific timelines since each change is different and changes like this require some background study (to understand what Datastore does and how it works) before we can give a good review. Due to us currently being engaged in release tasks this would most likely be delayed until after 0.11 is released just due to workload.

@negz

This comment has been minimized.

Contributor

negz commented Oct 31, 2017

I assume you were motivated to implement this due to a limitation of the existing Google Cloud Storage provider, so I'd love to hear more about that.

To be honest at the time I started working on this I missed that there was concurrent work happening to build a GCS backend with support for workspaces and locking, so I was simply motivated by wanting to use workspaces and locking in GCP without running my own infrastructure.

I chose the pure Datastore route because I was unaware that it was possible to do locking using GCS alone, and an S3-backend-style combination of GCS and Datastore felt overcomplicated. I had suspected the 1MB limitation might be a problem, but had assumed it was acceptable given the Consul backend faced similar issues.

I'm open to including this but I want to make sure we can clearly explain the tradeoff between the two Google Cloud backends and that there's a compelling reason to introduce maintenance overhead for two backends for the same vendor.

To be honest I'm not sure there is much of a compelling reason. Intuitively I would expect Datastore to be the better system for storing and locking state, but in practice I think GCS is ahead here because:

  • It lacks the 1MB size limit on blob data, and thus does not require compression.
  • It's strongly consistent for all operations (I believe).

In other ways they're mostly equivalent.

  • For Terraform state purposes both backends would almost certainly fall into the free pricing tier, with GCS allowing for more free storage and Datastore allowing for more free R/W operations.
  • Both backends require no real setup or maintenance aside enabling GCS or Datastore APIs for your project. I believe the GCS backend automatically creates its bucket, and the Datastore backend has no such need.
  • Both backends support storing state for multiple Terraform configs (by this I mean disparate directories of Terraform config - not workspaces). GCS via prefix and Datastore via namespace.
  • Both backends offer similar levels of redundancy - i.e. multi-region replication.

I'm fine with closing this given the updated GCS backend is now merged, unless anyone really wants to store their state in Datastore rather than GCS for some reason.

@apparentlymart

This comment has been minimized.

Contributor

apparentlymart commented Oct 31, 2017

Thanks for this additional context @negz!

I'm sorry for the accidental colliding work here, and appreciate the time you spent working on this.

Based on what you said here, I think I agree that it would be best to stick with the existing GCS provider for now, until (and if!) we find a reason to support both. The existing provider is already well-used and tested, and to be honest we don't really have Datastore expertise on the team to be able to review this without a lot of prior research.

I'm going to close this, but again I'm grateful that you took the time to work on this, and I'm sorry for the collision of work. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment