What are the pros/cons of using Terragrunt versus Terraform Workspaces when instantiating modules?

[marijakstrazdas]

What are the pros/cons of using Terragrunt versus Terraform Workspaces when instantiating modules?

[brikis98]

Updated answer (Aug, 2022)

I took the original answer below and flushed out the details in a blog post series called How to manage multiple environments with Terraform to answer this question. This series walks through how to use workspaces, branches, and Terragrunt to set up multiple environments, switch between environments, use different configurations, backends, and versions in each environment, and how to work with multiple modules. At the end of each post, you’ll see a list of the advantages and drawbacks of using that option. And for those of you looking for the short version, there's a handy comparison table in that series to help you pick:

Original answer (Dec, 2021)

There are a bunch of trade-offs. The short version is that workspaces are native to Terraform, so they are available out of the box and easy to use, but Terragrunt handles the separation you need between environments (minimizing "blast radius") better.

Here is the more in-depth version:

Advantages of workspaces vs Terragrunt

1. Workspaces are natively built into Terraform, so you don't need to learn/use/maintain/etc an external tool like Terragrunt.

2. Workspaces are natively used by Terraform Cloud (TFC) and Enterprise (TFE). Terragrunt is not natively supported by TFC or TFE (though there are some workarounds to use them together).

3. Workspaces are natively supported in the Terraform code itself: you can get the name of the workspace using terraform.workspace and have the code behave differently based on that name (e.g., instance_type == terraform.workspace === "dev" ? "t2.micro" : "m4.large").

4. Workspaces allow you to keep one copy of your code, but to have the state of the infrastructure captured in separate state files. This reduces code duplication. With Terragrunt, you do need to create extra files/folders to define each environment, so you do end up with a bit more duplication. Note that this is intentional, as the idea behind Terragrunt is to make those environments visible in your repo (you'll see more about that in the next section), but it does mean having more files/folders to manage.

Drawbacks of workspaces vs Terragrunt

1. The most common way people try to use workspaces is to define separate environments: e.g., dev, stage, prod. It turns out that even HashiCorp itself does NOT recommend this. Workspaces don't have good enough support for the separation of configuration and backends that you need between different environments:

   In particular, organizations commonly want to create a strong separation between multiple deployments of the same infrastructure serving different development stages (e.g. staging vs. production) or different internal teams. In this case, the backend used for each deployment often belongs to that deployment, with different credentials and access controls. Named workspaces are not a suitable isolation mechanism for this scenario.

   Terragrunt is specifically designed to help you define and manage multiple environments in a DRY manner, as explained here, and this is an effective way to keep configurations separate across environments and using separate backends (as explained here).

2. Terragrunt is natively built to not only support multiple environments, but to use different versions of your Terraform modules in each environment by setting different source URLs: e.g., you might use v0.1.0 of your vpc module in the prod environment and v0.2.0 in the stage environment. This is very useful for immutable infrastructure practices, where you roll out a new version of an immutable artifact from one environment to the next, always knowing exactly what's deployed where, and having the ability to roll back to older versions. With workspaces, there's no native support for this (the source URL in Terraform modules doesn't allow any variables or interpolation).

3. To create or switch to a workspace, you run a command: e.g., terraform workspace new or terraform workplace select. That means the list of workspaces that exist is not visible in the code or on disk. So just by browsing the repo, you can't tell what workspaces or environments actually exist; you'd have to look at state files or run terraform workspace list to figure that out. Moreover, workspaces are often created for testing or temporary work (especially in TFC/TFE), so it's very hard to get a picture of what you intended to deploy. With Terragrunt, the idiomatic way to define environments is to define them in files/folders on disk, so just by browsing a repo, you have a really good idea of what's deployed and the intention behind it (e.g., see the example live repo layout here).

4. To create or switch to a workspace, you run a command, which is quite error prone. It's a little too easy to think you're in the dev environment, make some changes, run destroy, and only then realize you were actually in prod all along, all because you accidentally forgot to run terraform workspace select. With Terragrunt, since environments are defined in files/folders on disk, it's a lot easier to tell that you're in the dev or prod folder, which minimizes that sort of mistake.

[adamkendall1]

I was on the fence about whether or not to use Terragrunt after hearing some opinions that recent improvements to Terraform make Terragrunt less useful than it had been in the past.

Your argument here regarding how Terragrunt makes environment separation, isolation, and intent clearly visible just by browsing the repo has ultimately won me over. Using workspaces or branches for this purpose is not nearly as transparent, and much more prone to things getting messy at scale I think. The time invested in setting up Terragrunt will be worth it for this benefit alone. Thanks for writing this up.

[evbo]

@brikis98 You wrote a great blog and I'm very grateful for it listing ideas I was unaware of (particularly using git branches, that was new to me).

But I'm a little confused.. For close to a decade I've been using Terraform modules like so (folder structure):

environment:
  prod:
    ec2:
    lambda:
  dev
    ec2:
    lambda:
modules:
  ec2
  lambda

Whereby main.tf, outputs.tf, and vars.tf exist in each folder respectfully. prod and dev each provide their own backend but otherwise simply calls the respective module.

Why didn't you consider this sort of approach in your blog? It's nearly identical to terragrunt, albeit a few extra lines of code... Workspaces aren't a solution for environments, but they are a solution for colocated deployments, such as "blue - green" strategies, etc.

Just curious why this isn't compared, am I too outdated? :)

[ekostjuk]

If you consider immutable artifacts approach - the act of promoting dev/staging to prod should not be involving changing any code, only switching environments and clicking deploy.

In your case, say you add/change an ec2 instance. Your flow would be:
Change it in dev -> test -> merge -> deploy to staging -> change for prod(separate folder) -> merge -> deploy to prod. Or variation of it. Does not matter how modularized and light your env is - you are still making extra commits specific to prod to be able to deploy.

What terragrunt allows you to do on top of TF.

1. Commit/test/merge once, deploy multiple times different envs.
2. Have a dependency links between sub-folders in your env. I.e. frontend depending on backend and you just running run-all apply. Parallel runs. (all account setup in parallel -> all DBs in parallel -> all backends in parallel -> all frontends in parallel)
3. Separating out config layering aspects of environments out into yaml files. (merge(common.yaml, <env>.yaml)). So DRY config, re-used env structure.
4. Extra to No2: graph separation. I think TF still suffers from non-linear growth of refresh times with more resources in tfstate.
5. Allows multiple-pass stuff, like generating config files from data blocks to pass into subsequent modules. Can't do that in one command in vanilla TF. Can generate stuff like k8s helm values override file per component per env runtime.
6. Allows more than dev/staging split. You can do dynamic environments and multi-tenancy environments, all config driven.
7. Avoids everyone re-inventing the wheel with ever-growing bash/python wrappers of TF for doing above. LIke for config layering, env switching, bootstraping backends, swapping auth keys, ordering sub-dir runs in a single env.
8. Allows different CLI args for each terragrunt component. I.e. I can run "-parallelism=1" for grafana or onepass apis, while maintaining high paralelism for aws/gcp. Without either component impacting each other in terms of speed. Or I can validate active mfa/sdk sessions or args before TF even runs.

Note: I did step away from Terragrunts suggested directory structure towards same code for all envs and config layering.

[Warns]

same code for all envs and config layering

Can you please elaborate a bit more about this?

[ekostjuk]

Soz, late.
Idiomatic TG example is modules + <env>/<component>/ dir structure.
I have modules + <component>/ code and switch env using TF_VAR_env=<env> tg run-all init -reconfigure with state repoint happening underneath.
For layered config - have yaml files for common + <env> with a section for each <component> which I merge on runs using a re-usable include (effect.config.hcl).
This means the only difference between prod and others is difference between config.prod.yaml and config.staging.yaml (3rd party urls, account ids, assume role stuff) and nothing else. Most things go into commons. ie. secrets wiring, networking, image tags(overridable).

So when you promote things between envs - you dont move code around, but click deploy to qa/prod from commit.