Interpolation inside .terragrunt file

[robkinyon]

We have a bunch of different Terraform state files, each handling .tf files in different directories. There's a .terragrunt file in each directory that looks something like:

dynamoDbLock = { stateFileId = "my_project_terraform_db_lock" }

remoteState = {
  backend = "s3"
  backendConfigs = {
    encrypt = "true"
    bucket = "my-bucket"
    key = "terragrunt/<dirname>.tfstate"
    region = "us-east-1"
  }
}

I'd really like to be able to use a single .terragrunt file for all the directories and be able do something like:

# Skip a bunch of common code shown above
remoteState = {
  backendConfigs = {
    key = "terragrunt/${dirname()}.tfstate"
  }
}

That way, I can symlink to a common .terragrunt in the parent directory and be assured I'm not screwing anything up when I copy the .terragrunt file into a new directory.

[brikis98]

Agreed this may be a good feature to have to reduce duplication. However, the tricky bit is figuring out what exactly would dirname() do? Apologies for the long reply, but here is a stream of consciousness to think through this question:

1. Does it return the absolute path? That obviously doesn't work since different developers on your team will check the code out into different directories on their computers.
2. Does it return the name of the parent directory of the .terragrunt file? This is likely to cause conflicts. For example, we have stage/vpc and prod/vpc folders, each with a .terragrunt file in them, so if we just used parent directory, they would both get vpc, which would cause the stage and prod VPCs to overwrite each other.
3. Does it navigate up the directory tree until it finds a version control root? For example, it could navigate up until it finds the first .git folder, indicating it's at the root of a git directory. That might work, but only for Git users.
4. Perhaps it can navigate up the directory tree until it finds another .terragrunt file, which indicates a root directory. I suppose that root .terragrunt file could even define some shared settings to reduce duplication further. But we still have another problem with all this directory climbing: if you refactor your folder structure in any way, you'll lose your Terraform state. Well, it'll obviously still be there in an S3 bucket, but Terragrunt will end up configuring it to look in the wrong folder, and it'll be as if you're starting with a blank slate. So after a refactor, you'd have to set these back to "manual" mode for a while, which just feels messy.
5. One final idea: perhaps we take a step back and reconsider the need to have .terragrunt in each folder. Instead, perhaps if there isn't such a file in the current directory, terragrunt should navigate up the folder structure to find one. Inside of that file could be the settings to use for all subfolders below that file that don't have their own .terragrunt config. That file could declare either a general purpose rule remoteState and dynamoDbLock rule to use for each folder (using dirname or similar helper function) or it could simply list the settings for each one while reusing common variables. This approach is more elegant, but still has the drawback that a simple folder move could break things and you'd have to remember to insert a custom .terragrunt file to work around that.

Having gone through this thought process, I'm honestly not convinced any of these options are inherently better than the simple, no-frills, no-magic, 100% declarative, WYSIWYG approach we have now. Yes, you have to do some copy/paste. But it's a one-time cost you do for each set of templates. Perhaps the real solution is to make even this act simpler by having terragrunt check for a .terragrunt file in the current folder, and if it doesn't find one, to interactively prompt you for the settings to create one, filling in lots of reasonable defaults automatically (e.g. by looking at the current folder path). That would largely automate the creation of these files, largely removing manual error from the equation.

Thoughts?

[cosmopetrich]

I've got the same terragrunt use case as @robkinyon.

Another option might be to make terragrunt capable of reading variables from a separate file (either its own, or *.tf) and interpolating them with the usual syntax, e.g.

remoteState = {
  backend = "s3"
  backendConfigs = {
    key = "terragrunt/${var.terragrunt_key}.tfstate"
    [...]
  }
}

That would allow symlinking a 'shared' config like the one below into each directory while keeping the per-directory settings in a separate non-linked file. There'd still be the need to create that regular file for each directory, but it would make things a bit simpler.

[brikis98]

OK, I have a proposal. We add two new features to Terragrunt:

1. The ability to pass a new parameter, --templates-path, which specifies the path where Terragrunt should run the terraform command. This allows you to have the .terragrunt file in one location and your Terraform templates somewhere else. Update: I just realized that, technically, we don't even need this. As of v0.1.2, Terragrunt supports a --terragrunt-config option (or TERRAGRUNT_CONFIG env var) to specify a custom location of your .terragrunt file. See the other "update" below for what this means.
2. Add basic interpolation support to .terragrunt files. For now, just one interpolation function would be supported, relative_path(), which returns the relative path between the --templates-path parameter and your .terragrunt file. This allows you to derive all the state_file_id and key configurations from the folder structure of your templates rather than having to duplicate it all by hand.

With the changes above, you could have the following folder structure:

my-terraform-repo
  └ .terragrunt
  └ qa
      └ my-app
          └ main.tf
  └ stage
      └ my-app
          └ main.tf
  └ prod
      └ my-app
          └ main.tf

Notice how there is only a single .terragrunt file for the whole repo, rather than one for each of the sets of Terraform folders. This .terragrunt file could contain the following:

# Configure Terragrunt to use DynamoDB for locking
lock = {
  backend = "dynamodb"
  config {
    state_file_id = "terragrunt-locks/${relative_path()}"
  }
}

# Configure Terragrunt to automatically store tfstate files in S3
remote_state = {
  backend = "s3"
  config {
    encrypt = "true"
    bucket = "my-terraform-bucket"
    key = "terraform-state/${relative_path()}"
    region = "us-east-1"
  }
}

Now, let's say you want to make a change in the stage/my-app folder. The Terragrunt command, which you would run from the root of the repo (which is where the .terragrunt file is) would look like this:

> cd my-terraform-repo
> terragrunt apply --templates-path=stage/my-app

Update: an alternative that's already supported today is to go into the stage/my-app folder and use the --terragrunt-config flag to point at the root .terragrunt file:

> cd my-terraform-repo/stage/my-app
> terragrunt apply --terragrunt-config=../../.terragrunt

When you run this command, Terragrunt would:

1. Go into the stage/my-app folder
2. Configure remote state in S3 with the key set to terraform-state/stage/my-app
3. Obtain a lock in DynamoDb with the name terragrunt-locks/stage/my-app
4. Run terraform apply
5. Release the lock

This way, the folder layout of your state files in S3 and the names of your locks in DynamoDB will automatically match the folder layout of your Terraform templates. Of course, if you need to override it for some reason, such as when you rename a folder but want it to keep using the old state file, you could include a .terragrunt file in just that folder, and it would override the root .terragrunt file.

Thoughts?

[josh-padnick]

First, it's great we're discussing this. This solves a real issue I've encountered with multiple software teams.

I generally like the idea but I have a few concerns. I'm going to think out loud here, so these thoughts might be a little messy:

1. One of the original goals of terragrunt was to create a dead-easy workflow that would ensure that users don't accidentally forget to configure terraform remote state. Today, the only rule the user has to remember is "use terragrunt on top of terraform," which seems pretty manageable.

   But now we're adding another rule. "Only do a terraform apply using terragrunt from the root folder." That's not too onerous, but I can imagine an unsuspecting user going to a subfolder, running terragrunt apply, seeing an error that no .terragrunt file exists, and then being confused.

   So my preference is still to have a .terragrunt file in each repo, but perhaps we can make the .terragrunt file copy-and-pasteable without having to change anything.

2. Following up on that last sentence, one alternative is to have a .terragrunt file in, say, /stage/my-app that looks like this:

   # Define the root folder explicitly
   root_folder = "/repos/infrastructure-live"  
   
   # Configure Terragrunt to use DynamoDB for locking
   lock = {
     backend = "dynamodb"
     config {
      state_file_id = "terragrunt-locks/${path_relative_to_root_folder()}"
     }
   }
   
   # Configure Terragrunt to automatically store tfstate files in S3
   remote_state = {
     backend = "s3"
     config {
        encrypt = "true"
        bucket = "my-terraform-bucket"
        key = "terraform-state/${path_relative_to_root_folder()}"
        region = "us-east-1"
    }
   }

   I'm not loving the name path_relative_to_root_folder and now we have a new problem where different users may have different root_folder values (although perhaps that's solved by using a relative value like root_folder = "../../../").

   But at least now we have our guardrail in place that communicates to a user they have to use the explicitly declared terraform remote state.

3. Another possibility is to have a root .terragrunt file that specifies common settings, as you suggest (possibly called .terragrunt.root to further distinguish it), and a simplified .terragrunt file in each folder, similar to this:

   lock = ${root.lock}
   remote_state = ${root.remote_state}

   This seems to be a pretty clean setup and solves the above problem. Also as you suggested, we could enhance terragrunt to traverse parent directories until it finds a .terragrunt.root file (the first such one), where it can read these file settings.

4. In any of these approaches, we still have the problem you outlined above, which is how to handle moving folders? There are a few possibilities here:

   1. Just let users manually manage this by leveraging the local .terraform state. For example, you could:

      1. terragrunt apply in the old location
      2. move the folder to the new location
      3. terragrunt apply in the new folder.

      I'm pretty sure Terraform will automatically write the local state to the remote state, though this needs to be verified.

   2. We could actually build migration functionality directly into terragrunt. For example, maybe we add a feature where terragrunt will get the latest remote_state, and then automatically push it to a new location based on properties in a .terragrunt file.

Lots of ideas here, and still sorting through them myself, but right now I'm most attracted to .terragrunt.root with a simpler .terragrunt file, though even this feels a little complex.

[brikis98]

Only do a terraform apply using terragrunt from the root folder

As I mentioned above, you can actually run Terragrunt from the terraform folders themselves. You just need to provide a --terragrunt-config parameter (or env var) that specifies where your .terragrunt file is.

In fact, since there is really only one .terragrunt file at the root of each project, if Terragrunt doesn't find one in the current folder, it would make sense to walk up the folder structure until you find one (and error out if you don't find one). That way, the usage pattern for Terragrunt would be identical to what it is today!

root_folder = "/repos/infrastructure-live"

This can't be an absolute path, and if it's a relative path, you have to tediously update it if you ever refactor your code and move files around.

but right now I'm most attracted to .terragrunt.root with a simpler .terragrunt file, though even this feels a little complex.

What's the purpose of the "local" .terragrunt files? Why create (and manage) this extra code?

I think it's a better idea to only have local .terragrunt files as overrides.

[robkinyon]

... if Terragrunt doesn't find one in the current folder, it would make sense to walk up the folder structure until you find one (and error out if you don't find one).

This sounds very good, especially when combined with relative_path().

I would change the --templates-path parameter to --workdir and it would default to pwd (preserving current behavior).

As for multiple .terragrunt files, I think that overriding is a bad idea unless you have specified some other .terragrunt file as a base. Then, Terragrunt would do a hash-merge, overriding values from base file with the current file. Overriding shouldn't happen as an automagical activity.

I do see a value in multiple .terragrunt files. For example, if I am managing a bunch of different things, including several DNS zones. I might want all the DNS zones to have the same lockfile so that all DNS changes are single-threaded. But, a DNS change could happen at the same time as a database change. So, my root .terragrunt file would use relative_path() to derive the lockfile key, but my DNS zones would have a .terragrunt file that would look something like:

basefile = "../.terragrunt"
lock = {
  config {
    state_file_id = "terragrunt-locks/dns"
  }
}

That said, if I don't set a basefile, then don't keep looking for .terragrunt files.

Also, I would terminate the .terragrunt parent-directory search once you find a .git directory.

[josh-padnick]

Also, I would terminate the .terragrunt parent-directory search once you find a .git directory.

This is a great idea, but it makes me realize the end user may be left wondering when terragrunt stops searching for .terragrunt files.

Another concern is what if there's a non-root .terragrunt file in a parent folder. How would the template know it's not the root without a basefile attribute? Or should .terragrunt files be "hierarchical" so that the root .terragrunt sets the defaults, and any other .terragrunt file can override for his directory and his descendants?

[robkinyon]

This is a great idea, but it makes me realize the end user may be left wondering when terragrunt stops searching for .terragrunt files.

I think this is solved by two actions:

1. documentation
2. Adding a --debug option which outputs (among other things) which directories terragrunt searches for a .terragrunt file and why it stops searching when it stops.

... should .terragrunt files be "hierarchical" so that the root .terragrunt sets the defaults, and any other .terragrunt file can override for his directory and his descendants?

I think this is a very logical extension and is almost simpler to code than the parent-child-only option.

[brikis98]

@josh-padnick and @robkinyon, please check out #59 and share your feedback!