Added Terraform lock file generation with provider caching

[levkohimins]

Description

Fixes #3089.

TODOs

Read the Gruntwork contribution guidelines.

• Update the docs.
• Run the relevant tests successfully, including pre-commit checks.
• Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable.
• Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

Added Terraform lock file generation with provider caching.

Migration Guide

[brikis98]

Just to make sure I understand, when TG provider caching is enabled, what this code does is:

1. [NEW] TG generates a different .terraformrc file for each TF instance, including a UUID in the cache server URL so it knows which requests are coming from which TF instances.
2. The TF instance(s) run init and make requests to the cache server for providers.
3. The cache server, as before, downloads the providers in the background and returns 409s.
4. [NEW] The cache server tracks which providers were requested by each TF instance, and from this data, generates a .terraform.lock.hcl file for those instances.
5. Those instances run init again, and since all have a .terraform.lock.hcl file, all will fetch their data from the cache.

Is that right? If so, we should update the docs to mention the UUID thing.

Also, do we have some sort of automated test that verifies that instances without lock file will make use of the cache now?

[levkohimins]

Just to make sure I understand, when TG provider caching is enabled, what this code does is:

1. [NEW] TG generates a different .terraformrc file for each TF instance, including a UUID in the cache server URL so it knows which requests are coming from which TF instances.
2. The TF instance(s) run init and make requests to the cache server for providers.
3. The cache server, as before, downloads the providers in the background and returns 409s.
4. [NEW] The cache server tracks which providers were requested by each TF instance, and from this data, generates a .terraform.lock.hcl file for those instances.
5. Those instances run init again, and since all have a .terraform.lock.hcl file, all will fetch their data from the cache.

Is that right? If so, we should update the docs to mention the UUID thing.

Yes, this is correct. I updated the docs.

Also, do we have some sort of automated test that verifies that instances without lock file will make use of the cache now?

Yes, we have. I improved the existing integration test.

terragrunt/test/integration_test.go

Line 271 in fb18445

providerBlock := lockfile.Body().FirstMatchingBlock("provider", []string{filepath.Dir(provider)})

Here we check that the lock file has a provider entry in the .terraform.lock.hcl.
We also run all our integration tests twice, with TERRAGRUNT_PROVIDER_CACHE=1 enabled and not, is also a check for lock file generation.

[brikis98]

LGTM!

[lorengordon]

@levkohimins Looks like a couple steps failed in the release pipeline for this one.... https://github.com/gruntwork-io/terragrunt/runs/24641974865

[levkohimins]

@lorengordon, thanks, these tests are flaky, we will improve them soon, but in the meantime I'll restart the circleci job.

[lorengordon]

Yep, totally understand, wasn't criticizing, just wanted to make you aware. Was trying to install latest and of course the assets weren't available! 😁 😅

[levkohimins]

@lorengordon, I should have set pre-release until the assets are built. Now all assets are available, you can try the new version.

[levkohimins]

Resolved in v0.58.3 release.