Globally apply lifecycle.ignore_changes

[devonbleak]

Current Terraform Version

$ terraform -v
Terraform v0.11.11
+ provider.aws v2.3.0

Use-cases

1. Environments with "guard rails" or other automated remediation - this would be a great solution for being able to globally ignore changes that are caused by some form of automation doing things like applying tags to resources. Since terraform removes tags that aren't in its configurations this would stop terraform and that kind of automation from fighting each other. There may be other types of automated remediation beyond tag updates that this might also be useful for.
2. No longer needing to share sensitive variables across systems. This would allow for setting the value of sensitive variables in TFE or other automation systems and still allow users to run local plans without having to have that value set locally. Currently every resource that leverages that variable must ignore the changes in those parameters.

Attempted Solutions

Checked the docs, nothing exists that would address this use case in either 0.11 or 0.12.

Proposal

I propose adding a new lifecycle parameter to the terraform {} block similar to what exists for resources:

terraform {
  lifecycle {
    ignore_changes = ["tags:BackupPlan", "tags:%", "rds_master_password"]
  }
}

This would basically append the indicated fields to every resource's lifecycle.ignore_changes

References

[apparentlymart]

Thanks for sharing this use-case, @devonbleak!

We've generally advised that conventions like you describe be managed by applying policy to Terraform plans rather than by external processes that intentionally create drift, but I understand that in some cases that is impractical, particularly if certain separate tooling creates tags as a side-effect of its work.

A global ignore_changes across all resource types is tricky because of course each resource type has its own set of attributes. Setting an ignore for tags["BackupPlan"] would not be effective for aws_autoscaling_group, for example, because it represents tags in a different way.

In the shorter term I think I'd suggest a more localized solution for tags in particular in the AWS provider, where perhaps the provider itself could accept a configuration for global tags to ignore across all resource types:

provider "aws" {
  ignore_tags = ["BackupPlan"]
}

By doing this at the provider level we can avoid problems caused by slight differences in representations of tags between resource types, because the provider understands AWS tagging natively while Terraform does not. I expect that such a feature would cause the tags in question to be invisible to Terraform entirely, and the various resources would return an error if you attempt to set them with Terraform, to make it clear that those tags are the responsibility of some other system.

I understand that this doesn't entirely meet what you suggested because it would be for tags specifically, but I think the two situations you describe feel different to me and may deserve different solutions. For example, the Terraform Enterprise team has been working on the remote backend so that local development can be tested via speculative plans run remotely, where the configured credentials can be used automatically without needing to sprawl them onto developer machines.

I'm going to label this "thinking" so we can more easily find it later when considering potential architectural changes. We've been considering some changes to how ignore_changes works to address some other limitations and inconsistencies with it, and I think we should think about these use-cases as part of that work so that we can think about the problem holistically.

In the meantime, you might consider opening an issue with the AWS provider (if there isn't one already) about the idea of globally ignoring tags in particular; the AWS provider team may have other thoughts on how best to address that, but we tend to try to solve problems at the provider level first before considering entirely new Terraform Core features because fully-general solutions are much more challenging to get right than localized solutions, and real-world experience with a solution in a provider gives us valuable additional information when considering a Terraform Core generalization of it.

Thanks again for sharing this!

[devonbleak]

@apparentlymart Provider-scoped ignore_changes seems like a reasonable alternative to global ignore_changes.

provider "aws" {
  lifecycle {
    ignore_changes = ["tags.foo", ...]
  }
}

I've been digging into the code around this and at this point looks like the lifecycle metaparameters are all handled within terraform - the providers are basically just defining their schema and some client metadata but the actual diff and application of ignore_changes isn't happening there. Given this, if we were to push the entire implementation of this down to the providers, they would have to implement functionality to hide the ignored changes from terraform in every resource's CRUD. Let me know if I'm missing something but this doesn't sound like the right architecture.

In terms of a path forward, it seems like this would require at least some implementation in terraform core to either merge in the ignore_changes metaparameters directly from the provider definition (would automatically provide an implementation of this functionality for every provider), or providing a callback for the providers to inject those values (would allow for more provider control and dynamic values to be injected), or even both. I'd gravitate toward the first option.

Let me know what you think!

cc @bflad @radeksimko

[apparentlymart]

Hi @devonbleak,

I was not suggesting it being a lifecycle block at all, but rather just a normal in the AWS provider's configuration block, as I showed in my earlier comment:

provider "aws" {
  ignore_tags = ["BackupPlan"]
}

That does not require any changes to Terraform Core, though I expect it will require some substantial changes to the AWS provider itself, given that for AWS the tagging APIs tend to be quite different between various services even though their UX is similar.

[devonbleak]

@apparentlymart you're right that in your description of the implementation there is no change required to Terraform Core, however I think it's still worth asking the question "does it make sense to do it this way?" IMO extending the existing ignore_changes functionality in a centralized place makes way more sense than putting duplicate code all over all of the providers in order to modify the data being returned to Core just for the sake of not changing Core.

[apparentlymart]

Hi @devonbleak,

As I mentioned before, making localized changes to providers to solve specific problems is both generally easier to do in the short term (because providers can make assumptions that Terraform cannot, such as knowing how AWS tags work) and also important so that we can ensure that the eventual Terraform Core feature is needed and can be designed properly. Without that practical experience with real examples it's difficult to know what sort of general solution is appropriate.

I'm not suggesting placing duplicate code over all providers. I'm suggesting a single specific feature to ignore particular tags within the AWS provider only, to directly address one of the use-cases you shared and in turn to learn from that experience whether a generalization is warranted and, if so, how that generalization should be designed within Terraform Core itself.

Specific solutions are easier to design, implement, and test than general ones, so we need the experience from specific solutions in order to inform the design of general solutions.

[phinze]

@devonbleak and I were able to have a valuable face-to-face conversation on this issue last week. I have followed up with a few internal chats with @apparentlymart and others.

Here are a few higher level points of context coming out of this discussion:

• The two use-cases outlined above (playing nicely with external tag manipulation and sensitive value handling) are both ones we want to support. It's possible that ignore_changes is not the tool we decide is best, but regardless we'll work towards supporting them one way or another!
• One of Terraform's overarching design goals is for its configuration to be as explicit as possible. The behavior of a given resource should be clearly traceable to its explicit inputs. This means we typically avoid features that allow for "action at a distance" or anything that implicitly changes the behavior of a given piece of configuration. This may be a case for us to make an exception to that general posture, but it would definitely be an exception.
• The ignore_changes subsystem is due for a holistic overhaul. It currently does not support dynamic inputs and it has a fair amount of under-defined and potentially surprising edge case behavior. This is something we'd like to go fix, but it's a sizable undertaking that we have yet to land on our roadmap. Until we get a chance to really dig in, we're hesitant to drop any new functionality or tweaks into this subsystem beyond bugfixes.

Given all of the above, our current tentative stance is that the best way to handle the first use case is via AWS provider functionality, and that the second use case should get addressed as we start to work on strengthening Terraform's handling of sensitive values as a whole.

So let's proceed to further explore the AWS-provider-specific solution for now.

A few follow up questions for you @devonbleak:

1. We are inclined to treat any ignore_tags type feature strictly. "Terraform does not see or touch these tags." We might even go so far as to make it an error message if any configuration tries to set tags using the ignored keys. This helps us avoid the more ambiguous space where we need to do an initial set and then ignore updates. Does the strict definition of ignore meet your needs?
2. We are evaluating implementing this as a tag prefix that Terraform would ignore everywhere. Would prefix-matching be sufficient to meet your needs?

[devonbleak]

Thanks @phinze.

For 1 I think that may be acceptable if we provide an alternative way to specify values for the ignored tags. Something like a separate resource that can create the tag outside of the tags map on the resource and not trigger the error.

For 2 a tag prefix is an interesting idea but I would say if it's going to cause a delay to the core functionality of ignoring exact matches we could probably live without it initially.

[phinze]

Thanks for these updates @devonbleak. We're planning to map out the post-0.12 releases over the coming 2-3 weeks and we'll include this functionality under consideration. I'll follow up with the results our conversations with an update for our plans here.

[bflad]

The discussion happening over here in this Terraform AWS Provider codebase issue regarding the similar but opposite problem of having default tags in the provider is most likely relevant for this feature request as well: hashicorp/terraform-provider-aws#7926 (comment)

[bflad]

For those interested in an update from the Terraform AWS Provider functionality for ignoring resource tags at the provider level, we are settling in on certain details that are in an effort to support this functionality. For example, see the refactoring effort to make this much functionality easier to implement within each resource (hashicorp/terraform-provider-aws#10688), a proof of concept implementation that will land shortly (hashicorp/terraform-provider-aws#10418), and a specific issue for tracking the provider-level ignore tagging enhancement (hashicorp/terraform-provider-aws#10689). 👍

[hiimbex]

Hi, just tracked down this issue which describes my exact use case! It looks like it was released in hashicorp/terraform-provider-aws#10689 in version 2.60.0 of the Terraform AWS provider, so this issue can probably be closed?

Just wanted to comment to say thanks for working on this! Super excited to go give it a spin!

Edit: works like a charm!! thanks again!

[zpriddy]

@hiimbex I would probably leave this open or create another issue.

Yes the provider level lets you ignore tags, but only works for those tags that were not generated via terraform. Lets say we use a tag ProvisionDate, and set it to timestamp(). If we tag every resource with this, then we want to ignore changes to this tag..

If you try to do that at the provider level, the aws provider actually removes the tag before sending it to terraform.. at that point it thinks it needs to add the now missing tag and does so on apply...

This can be seen in the debug log:

2020/07/10 12:17:57 [WARN] Provider "registry.terraform.io/-/aws" produced an unexpected new value for module.xxxxxx, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .tags: element "ProvisionDate" has vanished

Having some way to globally apply a base lifecycle per provider at the terraform level would be extremely useful

[pniederlag]

hashicorp/terraform-provider-azurerm#7034 is a clone of this issue on scope of the provider azurerem where it is not possible yet