Support for GCS history backend #23

ethernetdan · 2021-04-26T16:07:32Z

This PR attempts to add support for Google Cloud Storage (GCS) as a backend for tfmigrate history. Process went pretty well but had some questions about testing and hit an issue that I could use some help on.

Planning to add integration tests but wanted to get your thoughts before proceeding. It doesn't appear that there's a localstack equivalent for Google Cloud that mocks the entire platform. My initial thought is use fake-gcs-server to mock just storage. What do you think?
Currently debugging an issue setting up test-fixtures/storage_gcs. It appears to setup the backend correctly but seems to expect that the migration already has been applied:

➜  storage_gcs git:(gcs-history) ✗ go run ../../ apply
2021/04/26 08:54:07 [INFO] [runner] unapplied migration files: [mv_bar1.hcl]
2021/04/26 08:54:07 [INFO] [runner] load migration file: tfmigrate/mv_bar1.hcl
2021/04/26 08:54:07 [INFO] [migrator] start state migrator plan phase for apply
2021/04/26 08:54:07 [INFO] [migrator@.] terraform version: 0.14.9
2021/04/26 08:54:07 [INFO] [migrator@.] initialize work dir
2021/04/26 08:54:08 [INFO] [migrator@.] get the current remote state
2021/04/26 08:54:09 [INFO] [migrator@.] override backend to local
2021/04/26 08:54:09 [INFO] [executor@.] create an override file
2021/04/26 08:54:09 [INFO] [executor@.] switch backend to local
2021/04/26 08:54:09 [INFO] [migrator@.] compute a new state
2021/04/26 08:54:09 [INFO] [migrator@.] check diffs
2021/04/26 08:54:10 [ERROR] [migrator@.] unexpected diffs
2021/04/26 08:54:10 [INFO] [executor@.] remove the override file
2021/04/26 08:54:10 [INFO] [executor@.] switch back to remote
2021/04/26 08:54:11 [ERROR] [runner] failed to apply: mv_bar1.hcl
terraform plan command returns unexpected diffs: failed to run command (exited 2): terraform plan -state=/tmp/tmp150138163 -out=/tmp/tfplan981121270 -input=false -no-color -detailed-exitcode
stdout:
null_resource.bar1: Refreshing state... [id=4206823204168622562]
null_resource.foo4: Refreshing state... [id=7147799551067636043]
null_resource.foo3: Refreshing state... [id=6422014241050749061]
null_resource.foo2: Refreshing state... [id=3049368332131422781]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # null_resource.bar1 will be destroyed
  - resource "null_resource" "bar1" {
      - id = "4206823204168622562" -> null
    }

  # null_resource.foo1 will be created
  + resource "null_resource" "foo1" {
      + id = (known after apply)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

------------------------------------------------------------------------

This plan was saved to: /tmp/tfplan981121270

To perform exactly these actions, run the following command to apply:
    terraform apply "/tmp/tfplan981121270"


stderr:

exit status 1

Any thoughts on where I might look to troubleshoot?

minamijoyo · 2021-04-27T14:46:12Z

@ethernetdan Thank you for working on this!

The fake-gcs-server looks good to me. I prefer to use it as a docker container rather than a go library as the same as the localstack so that we can use it outside of go test.
I'm not sure what you are trying now, but you can get more detailed debug logs with exporting TFMIGRATE_LOG=DEBUG or TRACE. If it doesn't resolve the problem, please let me know how can I reproduce the error, I'd be happy to debug with you.

minamijoyo · 2021-04-27T15:19:43Z

For 1, looking at S3 history storage tests, I wrote only unit tests. It's ok to add integration tests if you want, but I think it's not required to merge this feature.

w1mvy · 2021-08-20T01:02:15Z

@minamijoyo @ethernetdan
Hello. I wanted to use tfmigrate and found this PR!
I read comment and tried to check the operation using fake-gcs-server.
There was a problem on the terraform side, but I confirmed that tfmigrate itself works.

https://gist.github.com/w1mvy/308b1a9be16b2ec28a7089611c978a01

If there's anything else I can do to help, just let me know.

minamijoyo · 2021-08-23T13:20:39Z

@w1mvy thank you for helping debug. I've been just waiting for a response from @ethernetdan.

ethernetdan · 2021-08-23T15:34:54Z

@w1mvy Thanks for verifying this!

@minamijoyo Sorry for the delay in responding, wanted to make sure the issue wasn't on the tfmigrate side but seems like that's not the case. I'm for merging if you're happy with where it is now.

minamijoyo · 2021-08-24T08:46:07Z

@ethernetdan Thank you for your reply. It seems that the Terraform core has a problem with the fake-gcs-server, but as I said before, I think adding acceptance tests for a new storage type is not required to merge.

This branch looks stale and conflicts with the latest master branch. Could you rebase it on the master? I'll review it later.

ethernetdan · 2021-08-24T16:19:43Z

Just rebased, let me know if there's any changes you'd like.

w1mvy · 2021-08-25T00:47:49Z

Thank you both very much.

minamijoyo · 2021-08-27T08:59:29Z

history/storage_gcs.go

+	Bucket string `hcl:"bucket"`
+
+	// Prefix is the directory where state files will be saved inside the bucket.
+	Prefix string `hcl:"prefix"`


I tested with the real gcs backend and noticed that there is an inconsistent behavior. The prefix attribute in the tfmigrate gcs storage doesn't actually work as prefix and looks just a name of object. It saves a history file as prefix, on the other hand the prefix attribute in the terraform gcs backend saves a state file as prefix/default.tfstate. I guess default is a name of workspace, right?

Ideally it should behave the same, but referencing the workspace name from the current storage implementation is not so easy. As a compromise, how about just using the name attribute so that users doesn't expect the same behavior?

minamijoyo · 2021-08-27T09:02:35Z

history/storage_gcs.go

+
+	// EncryptionKey is a 32 byte base64 encoded 'customer supplied encryption key' used to encrypt all state.
+	EncryptionKey string `hcl:"encryption_key,optional"`
+}


Please add documents for gcs storage type in README.
https://github.com/minamijoyo/tfmigrate/tree/v0.2.7#storage-block

minamijoyo · 2021-08-27T09:50:33Z

history/storage_gcs.go

+}
+
+// adapted from https://github.com/hashicorp/terraform/blob/2635b3b/backend/remote-state/gcs/backend.go#L113-L228
+func (s *GCSStorage) init(ctx context.Context) error {


I think the code you copied from the upstream violates the MPL2.0 license. The upstream hashicorp/terraform is distributed under the terms of MPL2.0, but the tfmigrate is currently distributed under MIT.

I understand that tfmigrate users will expect to be authenticated cloud providers with the same options and precedences as terraform backend. This was not a problem in the s3 storage (I think) because the authentication logic for aws is implemented in an external library hashicorp/aws-sdk-go-base.

Can I use your patch as MPL2.0? If you agree it, I'll refactor and split storage related code into a new package to clarify that the code in the new package uses MPL2.0. If you don't agree it, I (or someone else) need to create a new external library with MPL2.0, this will need additional works. I would like to confirm your thoughts before moving this forward.

minamijoyo · 2021-08-27T09:55:25Z

test-fixtures/storage_gcs/tfmigrate/mv_bar2.hcl.bk

+  actions = [
+    "mv null_resource.foo2 null_resource.bar2"
+  ]
+}


Can we remove this file? Is it enough mv_bar1.hcl?

minamijoyo · 2021-08-27T09:56:07Z

test-fixtures/storage_gcs/.tfmigrate.hcl

+  history {
+    storage "gcs" {
+      bucket = "tfmigrate-gcs"
+      prefix    = "tfmigrate/history.json"


Please align the indent

minamijoyo · 2021-08-27T14:25:51Z

@ethernetdan Thank you for your work! I reviewed this PR and left some comments. Please check them.

minamijoyo · 2022-08-25T14:41:39Z

The gcs history storage has been finally added in v0.3.7 via #103. Thank you all for working on this!

I originally wrote the local and s3 storage implementation for tfmigrate as the MIT license. Next, someone sent me a patch for gcs storage implementation minamijoyo/tfmigrate#23, copied from the upstream terraform backend implementation. I was concerned it would violate the MPL2 license because the tfmigrate' license is MIT, and terraform's is MPL2. So I split the storage implementations minamijoyo/tfmigrate#80 into a separate https://github.com/minamijoyo/tfmigrate-storage repository and relicensed it as MPL2 so that someone could easily copy snippets from the upstream. However, we could not contact that patch's author and the initial patch was not merged finally. Time passed, KengoTODA sent me another patch #3 for gcs support which contained minimal functionality and was not copied from upstream. This patch has finally been accepted and merged. That is, I've changed the license of the storage implementation with the intention that it will contain a derivative work of terraform that can be easily copied from terraform's backend implementation to extend various storage types, but this has yet to happen. It's clearly far from the terraform's implementation at the time of writing. Recently, HashiCorp announced that it would change the license of its products, including Terraform, from MPL2 to BSL1.1. https://www.hashicorp.com/blog/hashicorp-adopts-business-source-license If, as I originally intended, we allow contributors to copy code from upstream terraform, the tfmigrate-storage repository must also be changed to BSL. This means that the tfmigrate source can continue to be distributed as MIT, but the tfmigrate's binary will be bounded by BSL. Even though HashiCorp's BSL is a reasonably permissive license with no restrictions other than HashiCorp's competitors, it is no longer open-source by OSI definition. I want to continue to keep tfmigrate as an open-source project. Technically, it is possible to keep the tfmigrate-storage repository as MPL2. Still, I no longer have a solid reason to do so; it is just inconvenient to have a separate repository. Fortunately, KengoTODA and I are still the only contributors to the tfmigrate-storage repository, and I got permission from KengoTODA to re-license his patch as MIT in minamijoyo/tfmigrate#146. So, I'm going to change the license of the tfmigrate-storage repository to MIT and will merge it into the tfmigrate repository.

ethernetdan force-pushed the gcs-history branch from 4a049e1 to 50bdc27 Compare April 26, 2021 16:33

ethernetdan force-pushed the gcs-history branch from 50bdc27 to b2618fc Compare August 24, 2021 16:14

Add support for GCS history backend

5c32151

ethernetdan force-pushed the gcs-history branch from b2618fc to 5c32151 Compare August 24, 2021 16:18

minamijoyo reviewed Aug 27, 2021

View reviewed changes

GoodmanBen mentioned this pull request Feb 24, 2022

[Feature Request] GCP Cloud Storage History Backend #72

Closed

minamijoyo mentioned this pull request Mar 20, 2022

Move storage implementations to a new package #79

Merged

This was referenced Aug 19, 2022

Add a simple GCS storage impl minamijoyo/tfmigrate-storage#3

Merged

Support GCS as a history storage #103

Merged

minamijoyo closed this Aug 25, 2022

minamijoyo mentioned this pull request Aug 14, 2023

Change tfmigrate-storage license from MPL2 to MIT #146

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for GCS history backend #23

Support for GCS history backend #23

ethernetdan commented Apr 26, 2021

minamijoyo commented Apr 27, 2021 •

edited

Loading

minamijoyo commented Apr 27, 2021

w1mvy commented Aug 20, 2021

minamijoyo commented Aug 23, 2021

ethernetdan commented Aug 23, 2021 •

edited

Loading

minamijoyo commented Aug 24, 2021

ethernetdan commented Aug 24, 2021

w1mvy commented Aug 25, 2021

minamijoyo Aug 27, 2021

minamijoyo Aug 27, 2021

minamijoyo Aug 27, 2021

minamijoyo Aug 27, 2021

minamijoyo Aug 27, 2021

minamijoyo commented Aug 27, 2021

minamijoyo commented Aug 25, 2022

Support for GCS history backend #23

Support for GCS history backend #23

Conversation

ethernetdan commented Apr 26, 2021

minamijoyo commented Apr 27, 2021 • edited Loading

minamijoyo commented Apr 27, 2021

w1mvy commented Aug 20, 2021

minamijoyo commented Aug 23, 2021

ethernetdan commented Aug 23, 2021 • edited Loading

minamijoyo commented Aug 24, 2021

ethernetdan commented Aug 24, 2021

w1mvy commented Aug 25, 2021

minamijoyo Aug 27, 2021

Choose a reason for hiding this comment

minamijoyo Aug 27, 2021

Choose a reason for hiding this comment

minamijoyo Aug 27, 2021

Choose a reason for hiding this comment

minamijoyo Aug 27, 2021

Choose a reason for hiding this comment

minamijoyo Aug 27, 2021

Choose a reason for hiding this comment

minamijoyo commented Aug 27, 2021

minamijoyo commented Aug 25, 2022

minamijoyo commented Apr 27, 2021 •

edited

Loading

ethernetdan commented Aug 23, 2021 •

edited

Loading