Proposal: Terragrunt hooks

[brikis98]

A proposal to discuss with the Terragrunt community: adding "hooks" to Terragrunt that can be used to execute arbitrary shell commands. The idea would be to follow the basic structure of extra_arguments, where you can specify shell commands that will be executed either before or after specific Terraform commands. Here's the rough idea:

terragrunt = {
  terraform {
    # This hook configures Terragrunt to copy /foo to /bar before executing apply or plan
    before_hook "copy-file" {
      commands = ["apply", "plan"]
      execute = ["cp", "/foo", "/bar"]
    }

    # This hook configures Terragrunt to do a simple echo statement after executing any Terraform command
    after_hook "copy-file" {
      commands = ["${all_terraform_commands()}"]
      execute = ["echo", "Hello World"]
    }
  }
}

This idea has come up before, but I pushed back against it, recommending the use of Terraform's local-exec provisioner instead. However, I realize now that if you need to execute some code before or after terraform runs, local-exec doesn't help you.

A pattern that seems particularly interesting is to copy files into the working directory before running Terraform. This is especially useful for cases where Terraform doesn't support interpolation or effective code reuse. Examples:

1. Copy a version.tf file into your working directory. This file could define the required_version setting for your whole company so you don't have to copy/paste that setting into every single Terraform module.

2. Copy a common-vars.tf file into your working directory. This file could define common input variables that every single module should define: e.g., aws_region, aws_account_id, etc.

3. Copy a remote-state.tf file into your working directory and dynamically fill in the proper key and bucket values. Instead of relying on Terragrunt for keeping your remote state configuration DRY, you could use this approach to execute arbitrary code, and have a more flexible/customizable system.

Thoughts? Feedback? Concerns?

[antonbabenko]

That is exactly what I am implementing using another shell wrapper script and I really don't like my approach. My use case is described in example 1 and 2.

My feedback is that hooks in child terraform.tfvars and in a parent's one should be possible to combine, because the one in a child sometimes has some specific steps, while the one in parent should be applied for all cases.

[john-mcgowan-wowza]

I'm interested in the hooks pattern too. But for the purpose of halting an operation if some condition isn't met.

For instance. If we could assign a hook that validates that the git repo is totally up to date with origin and there are no uncommitted/unpushed changes that would help us enforce good practices when working with our -live repo.

[brikis98]

@john-mcgowan-wowza If a hook exits with a non-zero exit code, it would halt Terragrunt execution. This would work very simply & intuitively for a "pre" hook to check for uncommitted changes.

[john-mcgowan-wowza]

I like the idea of starting out with the generic hook functionality. And then if a particular hook pattern is fundamental enough, we could add it as part of terragrunt itself. So initially it might look like this...

before_hook "check-git-clean" {
      commands = ["apply", "plan"]
      execute = ["check-git-clean.sh"]
    }

but then when everybody ends up relying on the same functionality provided by check-git-clean.sh it could change to this?

before_hook "check-git-clean" {
      commands = ["apply", "plan"]
      execute = ["terragrunt --check-git-clean"]
    }

Not sure if there is any precedent for a terragrunt cli param that doesn't do normal terragrunt stuff but this would be pretty cool so that people wouldn't have to maintain their own check-git-clean.sh if all they want is basic best practice functionality?

[brikis98]

If functionality becomes so common that we decide to build it directly into Terragrunt, we could also express it with a syntax like:

before_hook "check-git-clean" {
  commands = ["apply", "plan"]
  built_in = ["check-git-clean"]
}

Alternatively, we could have a collection of user-contributed "hooks" in a hooks folder in the Terragrunt repo, and you could just download the ones you need. Not sure what would work better. Probably worth adding hooks, seeing how people start using them, and to try to extract some best practices from that.

[ebarault]

Would that be possible from a terragrunt workflow point of view that the before_hook is run before *.tfvars are parsed and injected in the temp folder with the terraform module files?

i'm thinking about resolving external references such as arns, ids, urls, secrets, etc. and injecting them in *.tfvars files through 3rd party templating tools.

# *.tfvars template file before before_hook
database_password = "${DATABASE_PASSWORD}"
some_other_key = "some_other_value"

• before_hook script runs (e.g. fetch some refs, and envsubst them)

# resolved *.tfvars file after before_hook
database_password = "some_password_fetch_by_hook_script"
some_other_key = "some_other_value"

• terragrunt apply

In some cases, it's very hard to inject remote data sources outputs in terraform modules when one wants to use them nested in other resources' blocks.
It's easier to inject those refs directly at terragrunt *.tfvars files level. I currently deal with that with a custom script that prepares the main *.tfvars before I run terragrunt apply

[brikis98]

Would that be possible from a terragrunt workflow point of view that the before_hook is run before *.tfvars are parsed and injected in the temp folder with the terraform module files?

Since the Terragrunt configuration, including hooks, are defined in a .tfvars file, we couldn't do it before parsing that file. But we could execute the hook before the files are copied. Or even have multiple stages: before_init, before, after.

i'm thinking about resolving external references such as arns, ids, urls, secrets, etc. and injecting them in *.tfvars files through 3rd party templating tools.

Ah, interesting use case. Actually, you'd probably want this hook to run after copying the .tfvars files so that the references are resolved in the tmp files and not the original source files. Otherwise, your source files would have modifications in them—such as secrets—that you might not want to accidentally check in!

[ebarault]

Actually, you'd probably want this hook to run after copying the .tfvars files so that the references are resolved in the tmp files and not the original source files

yes, you're right 👍

[brikis98]

Implemented in #439 and available in https://github.com/gruntwork-io/terragrunt/releases/tag/v0.14.4.

[mauricioscastro]

Is there a way to add an extra argument to "hooks" for "changing directory"/chdir before executing it?

Would be great, I am very very new to Go, but I could give it a go. Or is there already a proposal for something like that?

Regards.

[queglay]

Can these hooks be used to source a bash script, or only use output? The reason why is in cloud9, most of the variables I need are implicit to that environment, and I'd love to be able to source them at different stages. Currently I have to source the env vars before any terragrunt operations, but that means they can't really change between modules.

Here is an example of the types of vars I might evaluate in bash, but there are many more:

# Instance and vpc data
export TF_VAR_deployer_ip_cidr="$(curl http://169.254.169.254/latest/meta-data/public-ipv4)/32" # Initially there will be no remote ip onsite, so we use the cloud 9 ip.
export TF_VAR_remote_cloud_public_ip_cidr="$(curl http://169.254.169.254/latest/meta-data/public-ipv4)/32" # The cloud 9 IP to provision with.
export TF_VAR_remote_cloud_private_ip_cidr="$(curl http://169.254.169.254/latest/meta-data/local-ipv4)/32"
macid=$(curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/)
export TF_VAR_vpc_id_main_cloud9=$(curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/${macid}/vpc-id) # Aquire the cloud 9 instance's VPC ID to peer with Main VPC
export TF_VAR_cloud9_instance_name="$(aws ec2 describe-tags --filters Name=resource-id,Values=$TF_VAR_instance_id_main_cloud9 --out=json|jq '.Tags[]| select(.Key == "Name")|.Value' --raw-output)"
export TF_VAR_account_id=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | grep -oP '(?<="accountId" : ")[^"]*(?=")')
export TF_VAR_owner="$(aws s3api list-buckets --query Owner.DisplayName --output text)"

[brikis98]

The hooks do not modify the environment Terragrunt is running in, so sourcing variables will not work. You'll have to source the variables before running Terragrunt.