Terragrunt parsing support #843

zischknall · 2023-07-14T17:06:03Z

zischknall
Jul 14, 2023

Feature Overview

Terragrunt is a Terraform wrapper to handle DRY configuration of multiple environments managed through Terraform.

Terragrunt runs multiple modules one after another and has next to no output of it's own, it would be nice to also handle the outputs of those as though they were one Terraform plan output.

Why is the feature needed?

Currently runs of terragrunt plan can cause the parser to crash with an index out of bounds if it contains blocks multiple times. For example when the output contains two change blocks the parser crashes.

Does the feature include Breaking Changes?

By always looking for multiple blocks Terragrunt could be supported at the same time as continuing support for Terraform.
I currently have a fork with what I believe to be working Terragrunt plan parsing (no apply as of yet).
The only breaking change would be that I changed the formatting of the resulting output blocks slightly, while I might argue that it is know formatted in a more controlled manner.

Example Code

No small code example as it would change the internal parsing behavior.

Reference

No response

Answered by suzuki-shunsuke

Jul 15, 2023

I found that we can post the result of terragrunt with tfcmt by terragrunt --terragrunt-tfpath option.

Create a wrapper script of terraform and make it executable

$ vi tfwrapper.sh
$ chmod a+x tfwrapper.sh

tfwrapper.sh

#!/bin/bash

set -euo pipefail

command=$1

base_dir=$(git rev-parse --show-toplevel) # Please fix if necessary
target=${PWD#"$base_dir"/}

if [ "$command" == "plan" ]; then
  tfcmt -var "target:${target}" plan -- terraform "$@"
elif [ "$command" == "apply" ]; then
  tfcmt -var "target:${target}" apply -- terraform "$@"
else
  terraform "$@"
fi

Run terragrunt run-all with --terragrunt-tfpath

$ terragrunt run-all plan --terragrunt-tfpath "<absolute path of tfwrapper.sh>"

T…

View full answer

suzuki-shunsuke · 2023-07-14T22:14:29Z

suzuki-shunsuke
Jul 14, 2023
Maintainer

Thank you for your proposal.
I'm not familiar with Terragrunt at all, so first I'll take a look at it and reproduce the issue to understand the issue.

3 replies

suzuki-shunsuke Jul 15, 2023
Maintainer

I could reproduce the issue.

$ terraform version
Terraform v1.5.3
on darwin_arm64

$ tfcmt version
tfcmt version 4.4.2 (49d366319ec5d3170ede4076f3a4c79d1f035638)

$ terragrunt --version
terragrunt version v0.48.1

$ tree
.
├── frontend-app
│   ├── main.tf
│   └── terragrunt.hcl
├── mysql
│   ├── main.tf
│   └── terragrunt.hcl
└── terragrunt.hcl

3 directories, 5 files

terragrunt.hcl

# empty

frontend-app/main.tf, mysql/main.tf

resource "null_resource" "foo" {}

terragrunt run-all plan succeeded.

https://terragrunt.gruntwork.io/docs/features/execute-terraform-commands-on-multiple-modules-at-once/#the-run-all-command

$ terragrunt run-all plan
INFO[0000] The stack at /Users/shunsukesuzuki/Documents/test/tfcmt/terragrunt/843/stage will be processed in the following order for command plan:
Group 1
- Module /Users/shunsukesuzuki/Documents/test/tfcmt/terragrunt/843/stage/frontend-app
- Module /Users/shunsukesuzuki/Documents/test/tfcmt/terragrunt/843/stage/mysql
 

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:

  # null_resource.foo will be created
  + resource "null_resource" "foo" {
      + id = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.
  + create

Terraform will perform the following actions:

  # null_resource.foo will be created
  + resource "null_resource" "foo" {
      + id = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

tfcmt plan -- terragrunt run-all plan failed.

$ tfcmt plan -- terragrunt run-all plan
time=2023-07-15T09:37:14+09:00 level=info msg=The stack at /Users/shunsukesuzuki/Documents/test/tfcmt/terragrunt/843/stage will be processed in the following order for command plan:
Group 1
- Module /Users/shunsukesuzuki/Documents/test/tfcmt/terragrunt/843/stage/frontend-app
- Module /Users/shunsukesuzuki/Documents/test/tfcmt/terragrunt/843/stage/mysql



Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # null_resource.foo will be created
  + resource "null_resource" "foo" {
      + id = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # null_resource.foo will be created
  + resource "null_resource" "foo" {
      + id = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.
panic: runtime error: slice bounds out of range [25:19]

goroutine 1 [running]:
github.com/suzuki-shunsuke/tfcmt/pkg/terraform.(*PlanParser).Parse(0x140000b8370, {0x14000310000, 0x72d})
	/home/runner/work/tfcmt/tfcmt/pkg/terraform/parser.go:183 +0x1130
github.com/suzuki-shunsuke/tfcmt/pkg/notifier/github.(*NotifyService).Plan(0x140000aa198, {0x101100f40, 0x14000036e80}, 0x14000312000)
	/home/runner/work/tfcmt/tfcmt/pkg/notifier/github/plan.go:24 +0xb4
github.com/suzuki-shunsuke/tfcmt/pkg/controller.(*Controller).Plan(0x1400014f710, {0x101100f40, 0x14000036e80}, {{0x16f362f2e, 0xa}, {0x140000a2d20, 0x2, 0x2}})
	/home/runner/work/tfcmt/tfcmt/pkg/controller/plan.go:49 +0xa2c
github.com/suzuki-shunsuke/tfcmt/pkg/cli.cmdPlan(0x140000a8400)
	/home/runner/work/tfcmt/tfcmt/pkg/cli/plan.go:36 +0x2a4
github.com/urfave/cli/v2.(*Command).Run(0x140000a0000, 0x140000a8400, {0x140000b8050, 0x5, 0x5})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:274 +0x73c
github.com/urfave/cli/v2.(*Command).Run(0x140000a0580, 0x140000a8040, {0x1400001e1e0, 0x6, 0x6})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:267 +0x948
github.com/urfave/cli/v2.(*App).RunContext(0x14000094000, {0x101100f40?, 0x14000036e80}, {0x1400001e1e0, 0x6, 0x6})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:332 +0x568
main.core()
	/home/runner/work/tfcmt/tfcmt/cmd/tfcmt/main.go:30 +0x120
main.main()
	/home/runner/work/tfcmt/tfcmt/cmd/tfcmt/main.go:19 +0x1c

suzuki-shunsuke Jul 15, 2023
Maintainer

📝 I understood the cause of the panic.
startChangeOutput is the last line of Terraform will perform the following actions:,
but endChangeOutput is the first line starting with Plan:.
So the index of startChangeOutput is larger than endChangeOutput, so lines[startChangeOutput:endChangeOutput] gets a panic.

tfcmt/pkg/terraform/parser.go

Lines 182 to 184 in a2dd26e

    
           if startChangeOutput != -1 { 
        
           	changeResult = strings.Join(lines[startChangeOutput:endChangeOutput], "\n") 
        
           }

tfcmt/pkg/terraform/parser.go

Lines 133 to 135 in a2dd26e

    
           if line == "Terraform will perform the following actions:" { // https://github.com/hashicorp/terraform/blob/332045a4e4b1d256c45f98aac74e31102ace7af7/internal/command/views/plan.go#L252 
        
           	startChangeOutput = i + 1 
        
           }

tfcmt/pkg/terraform/parser.go

Lines 136 to 138 in a2dd26e

    
           if startChangeOutput != -1 && endChangeOutput == -1 && strings.HasPrefix(line, "Plan: ") { // https://github.com/hashicorp/terraform/blob/dfc12a6a9e1cff323829026d51873c1b80200757/internal/command/views/plan.go#L306 
        
           	endChangeOutput = i + 1 
        
           }

zischknall Jul 15, 2023
Author

I think there are three ways to go about this:

Ignore it and say that terragrunt is not supported (would be totally fine)
Make sure it does not crash but don't change the parsing. Causing only one change block to be commented in a terragrunt run (so somewhat partial support)
Change the parsing so it can handle both Terraform and Terragrunt. Officially supporting both

I think all of the above are totally fine and it's ultimately up to you.

suzuki-shunsuke · 2023-07-15T00:51:02Z

suzuki-shunsuke
Jul 15, 2023
Maintainer

What's the desirable hahaviour of this case? 🤔

Put together all modules' result in one comment
Post comments per module

2 replies

suzuki-shunsuke Jul 15, 2023
Maintainer

I prefer 2, but ideally we can choose both by configuration file and command line argument.

suzuki-shunsuke Jul 15, 2023
Maintainer

Idea 1

$ tfcmt -terragrunt-run-all -split-comment plan

tfcmt.yaml

terragrunt:
  run-all:
    split_comment: true

Idea 2

$ tfcmt -terragrunt-run-all -one-comment plan

tfcmt.yaml

terragrunt:
  run-all:
    one_comment: true

suzuki-shunsuke · 2023-07-15T01:38:01Z

suzuki-shunsuke
Jul 15, 2023
Maintainer

📝 We need to get the list of modules from the following output.

Group 1
- Module /Users/shunsukesuzuki/Documents/test/tfcmt/terragrunt/843/stage/frontend-app
- Module /Users/shunsukesuzuki/Documents/test/tfcmt/terragrunt/843/stage/mysql

https://github.com/gruntwork-io/terragrunt/blob/eda265f2dc6b64dba46f1f5e618cd9ec28cd8c97/configstack/stack.go#L34-L52

1 reply

suzuki-shunsuke Jul 15, 2023
Maintainer

We need to the relative path from the current directory.

/Users/shunsukesuzuki/Documents/test/tfcmt/terragrunt/843/stage/frontend-app => frontend-app

Sometimes we need to convert the directory file path to a target.

zischknall · 2023-07-15T07:23:27Z

zischknall
Jul 15, 2023
Author

Sorry at first only saw your first answer. Never worked with the discussions feature before and the mobile UI is a bit clunky. From my point of view at first I was fine with just concatenating all of the modules output blocks.
But I like the idea of showing it per module. But if I remember correctly Terragrunt shows at the beginning what modules it applies but in the following output it is difficult to discern which output belongs to which module.

0 replies

suzuki-shunsuke · 2023-07-15T13:47:25Z

suzuki-shunsuke
Jul 15, 2023
Maintainer

I found that we can post the result of terragrunt with tfcmt by terragrunt --terragrunt-tfpath option.

Create a wrapper script of terraform and make it executable

$ vi tfwrapper.sh
$ chmod a+x tfwrapper.sh

tfwrapper.sh

#!/bin/bash

set -euo pipefail

command=$1

base_dir=$(git rev-parse --show-toplevel) # Please fix if necessary
target=${PWD#"$base_dir"/}

if [ "$command" == "plan" ]; then
  tfcmt -var "target:${target}" plan -- terraform "$@"
elif [ "$command" == "apply" ]; then
  tfcmt -var "target:${target}" apply -- terraform "$@"
else
  terraform "$@"
fi

Run terragrunt run-all with --terragrunt-tfpath

$ terragrunt run-all plan --terragrunt-tfpath "<absolute path of tfwrapper.sh>"

Then the result of terraform plan is posted per module.

12 replies

zischknall Jul 18, 2023
Author

Terragrunt is run with run-all from each project folder so, once for qa and once for pp and so on.
Ideally I would want to have one comment on the Pull Request for each of those environments. Because we are running a plan on all four environments for each module. It would get quite overwhelming if there was a comment for each module.

Also I will try the DEBUG outputs and get back to you with that as well.

zischknall Jul 18, 2023
Author

Here is some of the DEBUG output:

[DEBUG] PWD: /home/runner/work/xxx/xxx/projects/ci/buckets/terraform-state/.terragrunt-cache/iH1mOlqPdLCgP0MJTfatAPSEYfs/ufzFrCkDUnPpPM2hdbErkJoUWvU
[DEBUG] target: /home/runner/work/xxx/xxx/projects/ci/buckets/terraform-state/.terragrunt-cache/iH1mOlqPdLCgP0MJTfatAPSEYfs/ufzFrCkDUnPpPM2hdbErkJoUWvU
[DEBUG] base_dir: /home/runner/work/xxx/xxx/projects/ci/wif-gh/.terragrunt-cache/BcrPECXlqn7U-5Fotql61tZ7Ejo/Z980WgVAL1mpXzVQ-9hVUsPNkOs
[DEBUG] PWD: /home/runner/work/xxx/xxx/projects/ci/wif-gh/.terragrunt-cache/BcrPECXlqn7U-5Fotql61tZ7Ejo/Z980WgVAL1mpXzVQ-9hVUsPNkOs/wif-gh
[DEBUG] target: wif-gh
[DEBUG] base_dir: /home/runner/work/xxx/xxx/projects/ci/project-cfg/.terragrunt-cache/-36JXAhoYXLANqQK-lAYxNcoalw/Nd3sy7QfqxLOiPv0uS3AyL_BjyM
[DEBUG] PWD: /home/runner/work/xxx/xxx/projects/ci/project-cfg/.terragrunt-cache/-36JXAhoYXLANqQK-lAYxNcoalw/Nd3sy7QfqxLOiPv0uS3AyL_BjyM
[DEBUG] target: /home/runner/work/xxx/xxx/projects/ci/project-cfg/.terragrunt-cache/-36JXAhoYXLANqQK-lAYxNcoalw/Nd3sy7QfqxLOiPv0uS3AyL_BjyM
[DEBUG] base_dir: /home/runner/work/xxx/xxx/projects/ci/project-cfg/.terragrunt-cache/-36JXAhoYXLANqQK-lAYxNcoalw/Nd3sy7QfqxLOiPv0uS3AyL_BjyM
[DEBUG] PWD: /home/runner/work/xxx/xxx/projects/ci/project-cfg/.terragrunt-cache/-36JXAhoYXLANqQK-lAYxNcoalw/Nd3sy7QfqxLOiPv0uS3AyL_BjyM
[DEBUG] target: /home/runner/work/xxx/xxx/projects/ci/project-cfg/.terragrunt-cache/-36JXAhoYXLANqQK-lAYxNcoalw/Nd3sy7QfqxLOiPv0uS3AyL_BjyM

suzuki-shunsuke Jul 18, 2023
Maintainer

Terragrunt is run with run-all from each project folder so, once for qa and once for pp and so on.

Yeah, that is what I expect, so I can't understand why target was /home/runner/work/xxx/xxx/projects/ci/buckets/pipeline/.terragrunt-cache/ZQKh0uHBw_1pS35ve2ew8rzhJgs/ufzFrCkDUnPpPM2hdbErkJoUWvU.

Ideally I would want to have one comment on the Pull Request for each of those environments.

I understand your needs.
But tfcmt doesn't support it for now.

My concern is that if results are put together in one comment, the comment is so large that it is difficult to read and the length may reach the limitation of the comment length.

suzuki-shunsuke Jul 18, 2023
Maintainer

The debug log looks weird.

$ pbpaste | grep PWD   
[DEBUG] PWD: /home/runner/work/xxx/xxx/projects/ci/buckets/terraform-state/.terragrunt-cache/iH1mOlqPdLCgP0MJTfatAPSEYfs/ufzFrCkDUnPpPM2hdbErkJoUWvU
[DEBUG] PWD: /home/runner/work/xxx/xxx/projects/ci/wif-gh/.terragrunt-cache/BcrPECXlqn7U-5Fotql61tZ7Ejo/Z980WgVAL1mpXzVQ-9hVUsPNkOs/wif-gh
[DEBUG] PWD: /home/runner/work/xxx/xxx/projects/ci/project-cfg/.terragrunt-cache/-36JXAhoYXLANqQK-lAYxNcoalw/Nd3sy7QfqxLOiPv0uS3AyL_BjyM
[DEBUG] PWD: /home/runner/work/xxx/xxx/projects/ci/project-cfg/.terragrunt-cache/-36JXAhoYXLANqQK-lAYxNcoalw/Nd3sy7QfqxLOiPv0uS3AyL_BjyM

The command was run twice in the same directory /home/runner/work/xxx/xxx/projects/ci/project-cfg/.terragrunt-cache/-36JXAhoYXLANqQK-lAYxNcoalw/Nd3sy7QfqxLOiPv0uS3AyL_BjyM.
base_dir was outputted only three times.

$ pbpaste | grep base_dir
[DEBUG] base_dir: /home/runner/work/xxx/xxx/projects/ci/wif-gh/.terragrunt-cache/BcrPECXlqn7U-5Fotql61tZ7Ejo/Z980WgVAL1mpXzVQ-9hVUsPNkOs
[DEBUG] base_dir: /home/runner/work/xxx/xxx/projects/ci/project-cfg/.terragrunt-cache/-36JXAhoYXLANqQK-lAYxNcoalw/Nd3sy7QfqxLOiPv0uS3AyL_BjyM
[DEBUG] base_dir: /home/runner/work/xxx/xxx/projects/ci/project-cfg/.terragrunt-cache/-36JXAhoYXLANqQK-lAYxNcoalw/Nd3sy7QfqxLOiPv0uS3AyL_BjyM

zischknall Jul 18, 2023
Author

I think I will continue to maintain my fork then and improve on it. I have not run into any comment body limits yet. I will deal with those as they appear.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Terragrunt parsing support #843

{{title}}

Replies: 5 comments 18 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Terragrunt parsing support #843

zischknall Jul 14, 2023

Feature Overview

Why is the feature needed?

Does the feature include Breaking Changes?

Example Code

Reference

Replies: 5 comments · 18 replies

suzuki-shunsuke Jul 14, 2023 Maintainer

suzuki-shunsuke Jul 15, 2023 Maintainer

suzuki-shunsuke Jul 15, 2023 Maintainer

zischknall Jul 15, 2023 Author

suzuki-shunsuke Jul 15, 2023 Maintainer

suzuki-shunsuke Jul 15, 2023 Maintainer

suzuki-shunsuke Jul 15, 2023 Maintainer

Idea 1

Idea 2

suzuki-shunsuke Jul 15, 2023 Maintainer

suzuki-shunsuke Jul 15, 2023 Maintainer

zischknall Jul 15, 2023 Author

suzuki-shunsuke Jul 15, 2023 Maintainer

zischknall Jul 18, 2023 Author

zischknall Jul 18, 2023 Author

suzuki-shunsuke Jul 18, 2023 Maintainer

suzuki-shunsuke Jul 18, 2023 Maintainer

zischknall Jul 18, 2023 Author

zischknall
Jul 14, 2023

Replies: 5 comments 18 replies

suzuki-shunsuke
Jul 14, 2023
Maintainer

suzuki-shunsuke Jul 15, 2023
Maintainer

suzuki-shunsuke Jul 15, 2023
Maintainer

zischknall Jul 15, 2023
Author

suzuki-shunsuke
Jul 15, 2023
Maintainer

suzuki-shunsuke Jul 15, 2023
Maintainer

suzuki-shunsuke Jul 15, 2023
Maintainer

suzuki-shunsuke
Jul 15, 2023
Maintainer

suzuki-shunsuke Jul 15, 2023
Maintainer

zischknall
Jul 15, 2023
Author

suzuki-shunsuke
Jul 15, 2023
Maintainer

zischknall Jul 18, 2023
Author

zischknall Jul 18, 2023
Author

suzuki-shunsuke Jul 18, 2023
Maintainer

suzuki-shunsuke Jul 18, 2023
Maintainer

zischknall Jul 18, 2023
Author