feat: Add `terragrunt_providers_lock` hook #632

wzooff · 2024-02-21T20:25:47Z

Put an x into the box if that apply:

This PR introduces breaking change.
This PR fixes a bug.
This PR adds new functionality.
This PR enhances existing functionality.

Description of your changes

Add hook that generates .terrafrom.lock.hcl using the terragrunt providers lock command

Fixes #630

How can we test changes

git clone git@github.com:wzooff/cats-dogs-infra.git
cd cats-dogs-infra/dogs/test/cluster
pre-commit install

Hook pointed to the latest commit in this PR

You can

remove .terraform.lock.hlc file and try to commit
edit modules/dummy/main.tf and set required version for null to 3.2.2 and then edit dogs/test/cluster/terragrunt.hcl name attribute (simulates version bump for module)

All above will trigger lock file regeneration.

.pre-commit-hooks.yaml

README.md

yermulnik · 2024-02-21T21:05:15Z

hooks/terragrunt_providers_lock.sh

I'm also wondering whether it is worth of 1) re-using tf providers lock per-dir function from tf providers lock hook since tg provider lock just relays this to underlying terraform and hence avoid sort of code duplication and 2) (re-)implementing the same check for lockfile integrity (to check hashes for all requested platforms) as what was implemented to tf providers lock recently (https://github.com/antonbabenko/pre-commit-terraform/blob/master/hooks/terraform_providers_lock.sh#L32)? 🤔

I also noticed that all three terragrunt hook scripts are the same file except few lines of code. Just different terragrunt commands. So room for improvement for sure.

As I know in v2.0 we'll get a bit cleaner code and hope till then @Tensho will provide feedback on actual usage

Co-authored-by: George L. Yermulnik <yz@yz.kiev.ua>

This reverts commit 3c974c4.

Tensho · 2024-02-22T11:25:56Z

Thank you for the rapid contribution 🙇 Can we have this feature merged and refactor all the code related to Terragrunt realms later?

yermulnik

LGTM apart from this thread: #632 (comment)
Let's see what @MaxymVlasov will say in this regards (and maybe Max will have other comments).

MaxymVlasov · 2024-02-22T13:27:45Z

After

git clone git@github.com:wzooff/cats-dogs-infra.git
cd cats-dogs-infra/dogs/test/cluster
pre-commit install

rm .terraform.lock.hcl
git add -A
pre-commit run -a

All these errors in outputs, it's expected?

[INFO] Initializing environment for https://github.com/wzooff/pre-commit-terraform.
Terragrunt providers lock................................................Failed
- hook id: terragrunt_providers_lock
- exit code: 1

ERRO[0000] Did not find any Terraform files (*.tf) in /tmp/cats-dogs-infra/cats/prod/cluster 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 
ERRO[0000] Did not find any Terraform files (*.tf) in /tmp/cats-dogs-infra/cats/test/storage/s3 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 
INFO[0000] Downloading Terraform configurations from git::ssh://git@github.com/terraform-aws-modules/terraform-aws-s3-bucket.git?ref=3.7.0 into /tmp/cats-dogs-infra/dogs-ua/test/storage/s3/.terragrunt-cache/wOb2gGSOIRhzEnVBnIOKou4IY40/9PaYfi3-KkZZxN2E0HJDpIPM9XI 
- Fetching hashicorp/null 3.2.2 for darwin_arm64...
- Retrieved hashicorp/null 3.2.2 for darwin_arm64 (signed by HashiCorp)
- Fetching hashicorp/null 3.2.2 for darwin_amd64...
- Retrieved hashicorp/null 3.2.2 for darwin_amd64 (signed by HashiCorp)
- Fetching hashicorp/null 3.2.2 for linux_amd64...
ERRO[0002] downloading source url git::ssh://git@github.com/terraform-aws-modules/terraform-aws-s3-bucket.git?ref=3.7.0
1 error occurred:
        * error downloading 'ssh://git@github.com/terraform-aws-modules/terraform-aws-s3-bucket.git?ref=3.7.0': /usr/bin/git exited with 1: error: pathspec '3.7.0' did not match any file(s) known to git

 
ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 
- Retrieved hashicorp/null 3.2.2 for linux_amd64 (signed by HashiCorp)
- Obtained hashicorp/null checksums for darwin_arm64; This was a new provider and the checksums for this platform are now tracked in the lock file
- Obtained hashicorp/null checksums for darwin_amd64; This was a new provider and the checksums for this platform are now tracked in the lock file
- Obtained hashicorp/null checksums for linux_amd64; This was a new provider and the checksums for this platform are now tracked in the lock file

Success! Terraform has updated the lock file.

Review the changes in .terraform.lock.hcl and then commit to your
version control system to retain the new checksums.

ERRO[0000] Did not find any Terraform files (*.tf) in /tmp/cats-dogs-infra/cats/test/cluster 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 
ERRO[0000] Did not find any Terraform files (*.tf) in /tmp/cats-dogs-infra/dogs-ua/test/cluster 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 
ERRO[0000] Did not find any Terraform files (*.tf) in /tmp/cats-dogs-infra/cats/prod/storage/s3 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 
ERRO[0000] Did not find any Terraform files (*.tf) in /tmp/cats-dogs-infra/dogs/test/storage/s3 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

yermulnik · 2024-02-22T13:32:40Z

Can we have this feature merged

@Tensho If you're in an urgency you can temporary point your local pre-commit-terraform to use latest commit from this PR via rev arg: #632 (comment)

MaxymVlasov · 2024-02-22T13:39:06Z

Okay, one of the errors comes from the invalid example

module version should be v3.7.0, not 3.7.0

https://github.com/wzooff/cats-dogs-infra/blob/5760e3280066d18aef8ba371391c8bb79f974857/dogs-ua/test/storage/s3/terragrunt.hcl#L2

But

ERRO[0000] Did not find any Terraform files (*.tf) in /tmp/cats-dogs-infra/cats/prod/storage/s3 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

still in place

➜ pre-commit run -a 
Terragrunt providers lock................................................Failed
- hook id: terragrunt_providers_lock
- exit code: 1

ERRO[0000] Did not find any Terraform files (*.tf) in /tmp/cats-dogs-infra/cats/prod/cluster 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 
ERRO[0000] Did not find any Terraform files (*.tf) in /tmp/cats-dogs-infra/cats/test/storage/s3 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 
- Fetching hashicorp/null 3.2.2 for darwin_arm64...
- Fetching hashicorp/aws 5.37.0 for darwin_arm64...
- Retrieved hashicorp/null 3.2.2 for darwin_arm64 (signed by HashiCorp)
- Fetching hashicorp/null 3.2.2 for darwin_amd64...
- Retrieved hashicorp/null 3.2.2 for darwin_amd64 (signed by HashiCorp)
- Fetching hashicorp/null 3.2.2 for linux_amd64...
- Retrieved hashicorp/null 3.2.2 for linux_amd64 (signed by HashiCorp)
- Obtained hashicorp/null checksums for darwin_arm64; This was a new provider and the checksums for this platform are now tracked in the lock file
- Obtained hashicorp/null checksums for darwin_amd64; This was a new provider and the checksums for this platform are now tracked in the lock file
- Obtained hashicorp/null checksums for linux_amd64; This was a new provider and the checksums for this platform are now tracked in the lock file

Success! Terraform has updated the lock file.

Review the changes in .terraform.lock.hcl and then commit to your
version control system to retain the new checksums.

- Retrieved hashicorp/aws 5.37.0 for darwin_arm64 (signed by HashiCorp)
- Fetching hashicorp/aws 5.37.0 for darwin_amd64...
- Retrieved hashicorp/aws 5.37.0 for darwin_amd64 (signed by HashiCorp)
- Fetching hashicorp/aws 5.37.0 for linux_amd64...
- Retrieved hashicorp/aws 5.37.0 for linux_amd64 (signed by HashiCorp)
- Obtained hashicorp/aws checksums for darwin_arm64; Additional checksums for this platform are now tracked in the lock file
- Obtained hashicorp/aws checksums for darwin_amd64; Additional checksums for this platform are now tracked in the lock file
- Obtained hashicorp/aws checksums for linux_amd64; All checksums for this platform were already tracked in the lock file

Success! Terraform has updated the lock file.

Review the changes in .terraform.lock.hcl and then commit to your
version control system to retain the new checksums.

ERRO[0000] Did not find any Terraform files (*.tf) in /tmp/cats-dogs-infra/dogs-ua/test/cluster 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 
ERRO[0000] Did not find any Terraform files (*.tf) in /tmp/cats-dogs-infra/cats/test/cluster 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 
ERRO[0000] Did not find any Terraform files (*.tf) in /tmp/cats-dogs-infra/dogs/test/storage/s3 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 
ERRO[0000] Did not find any Terraform files (*.tf) in /tmp/cats-dogs-infra/cats/prod/storage/s3 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

When it should looks like

➜ terragrunt run-all providers lock
INFO[0000] The stack at /tmp/cats-dogs-infra will be processed in the following order for command providers:
Group 1
- Module /tmp/cats-dogs-infra/dogs-ua/test/storage/s3
- Module /tmp/cats-dogs-infra/dogs/test/cluster
 
INFO[0000] Downloading Terraform configurations from git::ssh://git@github.com/terraform-aws-modules/terraform-aws-s3-bucket.git?ref=v3.7.0 into /tmp/cats-dogs-infra/dogs-ua/test/storage/s3/.terragrunt-cache/wOb2gGSOIRhzEnVBnIOKou4IY40/9PaYfi3-KkZZxN2E0HJDpIPM9XI  prefix=[/tmp/cats-dogs-infra/dogs-ua/test/storage/s3] 
- Fetching hashicorp/null 3.2.2 for linux_amd64...
- Retrieved hashicorp/null 3.2.2 for linux_amd64 (signed by HashiCorp)
- Obtained hashicorp/null checksums for linux_amd64; All checksums for this platform were already tracked in the lock file

Success! Terraform has validated the lock file and found no need for changes.
- Fetching hashicorp/aws 5.37.0 for linux_amd64...
- Retrieved hashicorp/aws 5.37.0 for linux_amd64 (signed by HashiCorp)
- Obtained hashicorp/aws checksums for linux_amd64; All checksums for this platform were already tracked in the lock file

Success! Terraform has validated the lock file and found no need for changes.

MaxymVlasov · 2024-02-22T13:48:56Z

You can test it locally without committing changes by

# run inside cats-dogs-infra/
/usr/bin/bash <path-to-wzoof-repo>/hooks/terragrunt_providers_lock.sh dogs-ua/test/cluster/.

wzooff · 2024-02-22T17:29:26Z

I'm afraid that this repo not ready for running hooks in all folders :) coz there a lot of modules that require proper aws creds. It's for playing with gha

wzooff · 2024-02-22T19:46:23Z

@MaxymVlasov try now. I've made all modules dummy in the repo

My results

➜  cats-dogs-infra git:(main) ✗ pre-commit run --all-files
Terragrunt providers lock................................................Passed
➜  cats-dogs-infra git:(main) ✗ git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	cats/prod/cluster/.terraform.lock.hcl
	cats/prod/storage/s3/.terraform.lock.hcl
	cats/test/cluster/.terraform.lock.hcl
	cats/test/storage/s3/.terraform.lock.hcl
	dogs-ua/test/cluster/.terraform.lock.hcl
	dogs-ua/test/storage/s3/.terraform.lock.hcl
	dogs/test/storage/s3/.terraform.lock.hcl

nothing added to commit but untracked files present (use "git add" to track)

➜  cats-dogs-infra git:(main) ✗ terragrunt run-all providers lock
INFO[0000] The stack at /Users/wzooff/projects/cats-dogs-infra will be processed in the following order for command providers:
Group 1
- Module /Users/wzooff/projects/cats-dogs-infra/cats/prod/cluster
- Module /Users/wzooff/projects/cats-dogs-infra/cats/prod/storage/s3
- Module /Users/wzooff/projects/cats-dogs-infra/cats/test/cluster
- Module /Users/wzooff/projects/cats-dogs-infra/cats/test/storage/s3
- Module /Users/wzooff/projects/cats-dogs-infra/dogs-ua/test/cluster
- Module /Users/wzooff/projects/cats-dogs-infra/dogs-ua/test/storage/s3
- Module /Users/wzooff/projects/cats-dogs-infra/dogs/test/cluster
- Module /Users/wzooff/projects/cats-dogs-infra/dogs/test/storage/s3

- Fetching hashicorp/null 3.2.2 for darwin_arm64...
- Fetching hashicorp/null 3.2.2 for darwin_arm64...
- Fetching hashicorp/null 3.2.2 for darwin_arm64...
- Fetching hashicorp/null 3.2.2 for darwin_arm64...
- Fetching hashicorp/null 3.2.2 for darwin_arm64...
- Fetching hashicorp/null 3.2.2 for darwin_arm64...
- Fetching hashicorp/null 3.2.2 for darwin_arm64...
- Fetching hashicorp/null 3.2.2 for darwin_arm64...
- Retrieved hashicorp/null 3.2.2 for darwin_arm64 (signed by HashiCorp)
- Obtained hashicorp/null checksums for darwin_arm64; All checksums for this platform were already tracked in the lock file

Success! Terraform has validated the lock file and found no need for changes.
- Retrieved hashicorp/null 3.2.2 for darwin_arm64 (signed by HashiCorp)
- Obtained hashicorp/null checksums for darwin_arm64; All checksums for this platform were already tracked in the lock file

Success! Terraform has validated the lock file and found no need for changes.
- Retrieved hashicorp/null 3.2.2 for darwin_arm64 (signed by HashiCorp)
- Obtained hashicorp/null checksums for darwin_arm64; All checksums for this platform were already tracked in the lock file

Success! Terraform has validated the lock file and found no need for changes.
- Retrieved hashicorp/null 3.2.2 for darwin_arm64 (signed by HashiCorp)
- Obtained hashicorp/null checksums for darwin_arm64; All checksums for this platform were already tracked in the lock file

Success! Terraform has validated the lock file and found no need for changes.
- Retrieved hashicorp/null 3.2.2 for darwin_arm64 (signed by HashiCorp)
- Obtained hashicorp/null checksums for darwin_arm64; All checksums for this platform were already tracked in the lock file

Success! Terraform has validated the lock file and found no need for changes.
- Retrieved hashicorp/null 3.2.2 for darwin_arm64 (signed by HashiCorp)
- Obtained hashicorp/null checksums for darwin_arm64; All checksums for this platform were already tracked in the lock file

Success! Terraform has validated the lock file and found no need for changes.
- Retrieved hashicorp/null 3.2.2 for darwin_arm64 (signed by HashiCorp)
- Obtained hashicorp/null checksums for darwin_arm64; All checksums for this platform were already tracked in the lock file

Success! Terraform has validated the lock file and found no need for changes.
- Retrieved hashicorp/null 3.2.2 for darwin_arm64 (signed by HashiCorp)
- Obtained hashicorp/null checksums for darwin_arm64; All checksums for this platform were already tracked in the lock file

Success! Terraform has validated the lock file and found no need for changes.

README.md

# [1.88.0](v1.87.1...v1.88.0) (2024-02-22) ### Features * Add `terragrunt_providers_lock` hook ([#632](#632)) ([77940fd](77940fd))

antonbabenko · 2024-02-22T21:47:40Z

This PR is included in version 1.88.0 🎉

wzooff added 4 commits February 21, 2024 00:10

feat: add terragrunt_providers_lock hook

d1abce1

make executable

78fb7a8

update readme

5b9e33a

adjust include exclude files

582d87b

wzooff requested review from MaxymVlasov and yermulnik as code owners February 21, 2024 20:25

wzooff changed the title ~~feat: add terragrunt_providers_lock hook~~ feat: Add terragrunt_providers_lock hook Feb 21, 2024

wzooff added 2 commits February 21, 2024 21:51

remove trailing spaces

3c974c4

amend table of contents

8ccb5ef

yermulnik reviewed Feb 21, 2024

View reviewed changes

wzooff and others added 4 commits February 21, 2024 22:14

optimize hook regexp

796df1a

Co-authored-by: George L. Yermulnik <yz@yz.kiev.ua>

Revert "remove trailing spaces"

056083a

This reverts commit 3c974c4.

amend available hooks section

75a05e8

amend hook usage section

97a352f

Minor docs changes

2fda000

yermulnik reviewed Feb 22, 2024

View reviewed changes

MaxymVlasov approved these changes Feb 22, 2024

View reviewed changes

MaxymVlasov reviewed Feb 22, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

Apply suggestions from code review

4828847

MaxymVlasov merged commit 77940fd into antonbabenko:master Feb 22, 2024
5 checks passed

antonbabenko pushed a commit that referenced this pull request Feb 22, 2024

chore(release): version 1.88.0 [skip ci]

4bf2502

# [1.88.0](v1.87.1...v1.88.0) (2024-02-22) ### Features * Add `terragrunt_providers_lock` hook ([#632](#632)) ([77940fd](77940fd))

MaxymVlasov added the feature New feature or request label Feb 22, 2024

MaxymVlasov added the hook/terragrunt_providers_lock Bash hook label Feb 22, 2024

MaxymVlasov added this to the Features implementing and cleanup without breaking changes milestone Feb 22, 2024

wzooff deleted the feat/add-terragrunt-providers-lock branch February 23, 2024 08:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add `terragrunt_providers_lock` hook #632

feat: Add `terragrunt_providers_lock` hook #632

wzooff commented Feb 21, 2024 •

edited

yermulnik Feb 21, 2024

wzooff Feb 21, 2024

Tensho commented Feb 22, 2024

yermulnik left a comment

MaxymVlasov commented Feb 22, 2024

yermulnik commented Feb 22, 2024

MaxymVlasov commented Feb 22, 2024

MaxymVlasov commented Feb 22, 2024

wzooff commented Feb 22, 2024 •

edited

wzooff commented Feb 22, 2024 •

edited

antonbabenko commented Feb 22, 2024

feat: Add terragrunt_providers_lock hook #632

feat: Add terragrunt_providers_lock hook #632

Conversation

wzooff commented Feb 21, 2024 • edited

Description of your changes

How can we test changes

yermulnik Feb 21, 2024

Choose a reason for hiding this comment

wzooff Feb 21, 2024

Choose a reason for hiding this comment

Tensho commented Feb 22, 2024

yermulnik left a comment

Choose a reason for hiding this comment

MaxymVlasov commented Feb 22, 2024

yermulnik commented Feb 22, 2024

MaxymVlasov commented Feb 22, 2024

MaxymVlasov commented Feb 22, 2024

wzooff commented Feb 22, 2024 • edited

wzooff commented Feb 22, 2024 • edited

antonbabenko commented Feb 22, 2024

feat: Add `terragrunt_providers_lock` hook #632

feat: Add `terragrunt_providers_lock` hook #632

wzooff commented Feb 21, 2024 •

edited

wzooff commented Feb 22, 2024 •

edited

wzooff commented Feb 22, 2024 •

edited