[Feature] allow force creation of consul_key_prefix resource with existing keys

[plaisted]

Terraform Version

Terraform v0.11.10

• provider.consul v2.2.0

Affected Resource(s)

consul_key_prefix

Enhancement

I use terraform to manage configuration stored in consul using the consul_key_prefix resource. Terraform configuration is stored in a git repo and promotion of config changes are managed using pull requests. The git repo is meant to the the source of truth.

With the current setup of the consul provider, if the terraform state is ever lost or corrupted, it requires a complete key purge of the consul_key_prefix resources in order for terraform to control this prefix again. If keys exist, the following error is given:

consul_key_prefix.example: X keys already exist under config/example/; delete them before managing this prefix with Terraform

I understand this is done to prevent accidentally destroying existing consul KV's. I also understand TF state is important and shouldn't be lost. However with a consul_key_prefix resource it is very disruptive to have to purge all existing keys in a live environment because the TF state was lost / corrupt. It would be very useful to have the ability to force the resource creation even with existing keys. Alternatively a way to "bootstrap" tfstate with an existing consul_keys_prefix would solve this as well.

[remilapeyre]

Hi @plaisted, thanks for opening an issue.

It is good to keep and version Terraform scenarios in git. As you have noted, loosing or corrupting the terraform state will lead to messy situations.

To avoid such situations, I usually use the S3 backend (https://www.terraform.io/docs/backends/types/s3.html) with locking in DynamoDB. Sharing the state with your team can simplify operations and the locking with DynamoDB will prevent multiple person to apply changes at the same time. Activating versionning in the S3 bucket should also help you to recover a previous state when an issue arise.

If you use a local state file, backups should also be made and could be use to recover from such situations without having to delete the whole Consul KV path.

Also, the Terraform state should not usually get corrupted, does this happen often?

it is very disruptive to have to purge all existing keys in a live environment because the TF state was lost / corrupt

If you want to delete all at once, you can use consul kv delete -recurse config/example just before terraform apply. I understand this can be problematic when your applications are watching for changes in the Consul KV store.

Alternatively a way to "bootstrap" tfstate with an existing consul_keys_prefix would solve this as well

This is what terraform import is for when resources supports it, I will open a new PR to add this feature to consul_key_prefix.

[plaisted]

Thanks for the comments. I can't recall TF state getting corrupt without someone doing something dumb, but we are new to TF so have been experimenting with a lot of different setups which have led to state issues (lost state, multiple sets of state from different groups, etc). Most of these issues were part of our operational learning curve.

I'm planning out our disaster recovery strategy for the consul configuration which is why I raised this issue. Theoretically losing the TF state shouldn't be a hard scenario to recover from if just using consul_keys_prefix as all the data is redundant. It sounds like terraform import would solve this problem.

We use consul-template with lots of "keyOrDefault" so purging keys isn't really an option as it'd have fallout with applications as you mentioned. For now I think we'd have to create a clean consul environment and push the terraform values there and then manually copy the state created doing that to our real production setup (would take some research as the state version/lineage/serial may need to be adjusted).

[remilapeyre]

Hi @plaisted, the new changeset in #78 should do what you are looking for.

EDIT: By the way, having the Terraform state is a start but some keys may not appear in it, like Vault's data if you are using Consul as the backend or the data from consul-esm. Have a look at https://www.consul.io/docs/commands/snapshot.html to make periodic backups of Consul data and catalog.

Here is an example:

➜  terraform-provider-consul git:(master) ✗ cat example.tf
resource "consul_key_prefix" "myapp_config" {
  path_prefix = "config/example/"

  subkeys = {
    "foo"         = "bar"
  }
}
➜  terraform-provider-consul git:(master) ✗ consul kv put config/example/test/coucou
Success! Data written to: config/example/test/coucou
➜  terraform-provider-consul git:(master) ✗ terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + consul_key_prefix.myapp_config
      id:          <computed>
      datacenter:  <computed>
      path_prefix: "config/example/"
      subkeys.%:   "1"
      subkeys.foo: "bar"


Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

consul_key_prefix.myapp_config: Creating...
  datacenter:  "" => "<computed>"
  path_prefix: "" => "config/example/"
  subkeys.%:   "" => "1"
  subkeys.foo: "" => "bar"

Error: Error applying plan:

1 error(s) occurred:

* consul_key_prefix.myapp_config: 1 error(s) occurred:

* consul_key_prefix.myapp_config: 1 keys already exist under config/example/; delete them before managing this prefix with Terraform

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.


➜  terraform-provider-consul git:(master) ✗ terraform import consul_key_prefix.myapp_config config/example/
consul_key_prefix.myapp_config: Importing from ID "config/example/"...
consul_key_prefix.myapp_config: Import complete!
  Imported consul_key_prefix (ID: config/example/)
consul_key_prefix.myapp_config: Refreshing state... (ID: config/example/)

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

➜  terraform-provider-consul git:(master) ✗ terraform apply
consul_key_prefix.myapp_config: Refreshing state... (ID: config/example/)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

-/+ consul_key_prefix.myapp_config (new resource required)
      id:                  "config/example/" => <computed> (forces new resource)
      datacenter:          "dc1" => <computed>
      path_prefix:         "" => "config/example/" (forces new resource)
      subkeys.%:           "1" => "1"
      subkeys.foo:         "" => "bar"
      subkeys.test/coucou: "" => ""


Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

consul_key_prefix.myapp_config: Destroying... (ID: config/example/)
consul_key_prefix.myapp_config: Destruction complete after 0s
consul_key_prefix.myapp_config: Creating...
  datacenter:  "" => "<computed>"
  path_prefix: "" => "config/example/"
  subkeys.%:   "" => "1"
  subkeys.foo: "" => "bar"
consul_key_prefix.myapp_config: Creation complete after 0s (ID: config/example/)

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.