Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terraform changes a lot of resources when removing an element from the middle of a list #14275

Closed
joshuaspence opened this issue May 8, 2017 · 17 comments

Comments

Projects
None yet
@joshuaspence
Copy link
Contributor

commented May 8, 2017

We have a lot of AWS Route53 zones which are setup in exactly the same way. As such, we are using count and a list variable to manage these. The code basically looks like this:

variable "zone_names" {
  type    = "list"
  default = []
}

resource "aws_route53_zone" "ctld" {
  name  = "${var.zone_names[count.index]}"
  count = "${length(var.zone_names)}"
}

resource "aws_route53_record" "www" {
  zone_id = "${var.zone_names[count.index]}"
  name    = "www"
  type    = "A"

  alias {
    # ...
  }

  count = "${length(var.zone_names)}"
}

However, the problem with this is that Terraform references resources using a numeric identifier. As such, if we remove an element from the middle of the list then Terraform will want to recreate all resources with a larger numeric index. This can be minimally reproduce with the following code snippet:

variable "names" {
  type    = "list"
  default = ["foo", "bar", "baz"]
}

resource "null_resource" "test" {
  triggers {
    name = "${var.names[count.index]}"
  }

  count = "${length(var.names)}"
}

Run terraform apply and then remove "bar" from var.names. The subsequent plan is as follows:

-/+ null_resource.test.1
    triggers.%:    "1" => "1"
    triggers.name: "bar" => "baz" (forces new resource)

- null_resource.test.2

I thought that one possible way to fix this would be if Terraform could index a list of resources by its identifier rather than its sequence in a list.

@apparentlymart

This comment has been minimized.

Copy link
Member

commented May 8, 2017

Hi @joshuaspence!

This is an issue we've had on our radar for a while now and have some ideas to deal with it, which will hopefully be included as part of a set of configuration language improvements coming in the future. The most promising idea so far is to add a new for_each argument alongside count which can use a stable identifier for the resource identifiers, similar to what you suggested.

Thanks for reporting it!

@catsby catsby added core bug labels May 8, 2017

@ewbankkit

This comment has been minimized.

Copy link
Contributor

commented May 10, 2017

@apparentlymart Until said functionality is available, any thoughts around adding a CLI command to compact a list in the state file (e.g. terraform state compact)?
You could then:

  • remove a list element in the code - e.g. at index 1
  • remove the associated resource from state - terraform state rm aws_route53_zone.ctld[1]
  • compact the counted resource in state - terraform state compact aws_route53_zone.ctld

At this point terraform plan should show no changes.

@apparentlymart

This comment has been minimized.

Copy link
Member

commented May 10, 2017

That's an interesting idea, @ewbankkit!

My first reaction to it is being a little nervous about it, since it's a single command that could create a lot of disruption in a state that would be hard to recover from if it's a mistake.

However, perhaps as one of a suite of operations it would be okay, so that mistakes would be no more costly than accidentally running terraform state rm on the wrong resource...

  • As you proposed, an operation to remove an instance from a counted list of instances
  • Another operation to "insert" an instance at an index, which would really just be renumbering all of the indices >= to the specified one to make a space for a new instance to exist in the next plan.
  • An operation to swap two instances as a single operation, so to help with recovery if instances end up in the wrong order.

This way if I make a mistake with the "remove one instance" operation I can use the "insert" operation to renumber everything back again, and then hopefully terraform import the thing I removed back into its previous place.

I expect no matter how we were to present it these commands would be a bit arcane and edge-casey, but this would still be superior to having users manually edit the state file to renumber things. I'm a little unsure as to what to name these three operations so that they are self-explaining and make sense together as a set. If you have any more thoughts here, maybe let's discuss this in a separate issue.

@giuliano108

This comment has been minimized.

Copy link

commented Jul 5, 2017

The way we worked around this is by scripting a .tfstate transform. It moves to the end of a list the resources we want to remove...

@goffinf

This comment has been minimized.

Copy link

commented Aug 1, 2017

We also use count to create a lot of instances of various resource types (aws_ecr_repository is one example). We provide a list of repo names and use count to iterate. But, as mentioned here if you either add a new entry to the list in any position other than the end, or, if you delete an entry from anywhere other than the last, Terraform we destroy and recreate every resource from that point.

Doesn't this sound like a case for passing a map into a resource so that resource creation / deletion is based on the explicit key rather than rely on the ordinal position in a list ?

I suppose if we could interpolate the resource name itself that might do it, but I'm not sure how practical that is ?

Anyway, whatever the underlaying implementation, this problem is really hurting us.

We currently use Terraform 0.9.11. Is a solution likely in the near term ?

Can you suggest any other work-arounds other than manipulating the data backend state file (not something I'm especially attracted towards) ?

Kind Regards

Fraser.

@coingraham

This comment has been minimized.

Copy link

commented Aug 15, 2017

I wrote a utility for editing count in the state file, but would like to see a fix as well.

@pschuegr

This comment has been minimized.

Copy link

commented Oct 10, 2017

@apparentlymart Do you have any idea of what the timeframe might be for a fix to this issue? Thanks!

@Farhie

This comment has been minimized.

Copy link

commented Nov 14, 2017

I am running into this issue with generation of ECR repos. I'd be interested to know where this is on your radar @apparentlymart? I assume the current workaround is to utilise a map as per @goffinf's suggestion?

@sebglon

This comment has been minimized.

Copy link

commented Nov 21, 2017

If we replace Index by the values on tfstate, this can solve the problem if value is unique on list and this work perfectly on map. On this case, we can create iterator and use value to not change count function.

If we want to use more complexe date we can store position on tfstate and manage rewrire index data with diff on variable list or map.

@goffinf

This comment has been minimized.

Copy link

commented Nov 21, 2017

@sebglon could you show an example of your first suggestion please

@sebglon

This comment has been minimized.

Copy link

commented Nov 25, 2017

yes on tfstate you put index on resources.
For examplel: "google_compute_address.collectd_ip.1
actualy, i use count with map.
But if i replace index by map Key, this will work.

For list if i use value if it is not duplicated, i can use the same.

instead of using count, you can create iterate function to make it work.

@rkul

This comment has been minimized.

Copy link

commented Jan 24, 2018

Could anybody help to figure out timeframe or any workaround, except removing resources from state and importing them back?

@pdecat

This comment has been minimized.

Copy link
Contributor

commented Jan 24, 2018

Hi @rkul, as a workaround, you can use terraform state mv <resource-name>.<resource-id>[<i>] <resource-name>.<resource-id>[<j>] to move elements one by one in the list, upwards or downwards depending on the values you set for i & j.

@rkul

This comment has been minimized.

Copy link

commented Jan 24, 2018

@pdecat Thank you, I appreciate your response, but it might be too hard to move half of list elements every time.

@nqbao

This comment has been minimized.

Copy link

commented Jan 29, 2018

I also worked around by reimporting all the resources (terraform import). However, @pdecat approach is much better, because i don't have to manually inspect the target id.

@apparentlymart

This comment has been minimized.

Copy link
Member

commented Jan 29, 2018

Hi all! Sorry for the long silence here... there's a handful of overlapping issues on this topic and so it's often hard to keep up with updates in all of them. 😖

I think the best sum-up of progress here is over in #17179. To summarize: we'd like to address this not by adding new commands to manipulate the state but instead by creating the possibility for the multiple instances of a resource block to be identified by map keys (strings) rather than indexes.

For @joshuaspence's original use-case, for example:

# NOT YET IMPLEMENTED, and details may change before release

variable "zone_names" {
  type    = "list"
  default = []
}

resource "aws_route53_zone" "ctld" {
  # this is the new "for expression" syntax included in the improved configuration language, coming soon.
  # turns a list like ["example.com"] into a map like {"example.com" => "example.com"}
  for_each = {for name in var.zone_names: name => name}

  name = each.value
}

resource "aws_route53_record" "www" {
  for_each = {for name in var.zone_names: name => name}

  zone_id = aws_route53_zone.ctld[each.key].id
  name    = "www"
  type    = "A"

  alias {
    # ...
  }
}

Since the resource instances in the above example would be identified by the strings in zone_names, rather than by their indexes, adding and removing items from that list would have the intended effect of creating and removing zones and their associated records, rather than updating existing numbered resources.

Our expectation is that for_each will be used in preference to count in most situations, and thus the original problem here will become moot. count will still be retained and for_each will also support iterating over lists, so index-based instances will still be possible in rare cases where they are preferable.

Since this issue was (due to my early comments) primarily focused on ways to work around the problem via new plumbing commands, I'm going to close this one just to consolidate ongoing discussion in #17179. I'll post more updates there when we're ready to finalize the details and do the final implementation of the for_each feature.

Thanks for sharing your workarounds here, everyone!

@mssaisandeep

This comment has been minimized.

Copy link

commented Mar 15, 2019

Hi @rkul, as a workaround, you can use terraform state mv <resource-name>.<resource-id>[<i>] <resource-name>.<resource-id>[<j>] to move elements one by one in the list, upwards or downwards depending on the values you set for i & j.

This worked for me. Whenever there is a destroy of a specific instance. I manually moved the ids as mentioned above using "terraform state mv" command and updated my vars files (vars as an input to terraform scripts(.tf files)) accordingly.

Temporary fix until we get a solution. But that's ok. We rarely do delete

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.