Terragrunt Provider Cache not working properly when using it with Atlantis

[amontalban]

Describe the bug

As described in #2920 (comment) we are having mixed results while using Terragrunt Provider Cache running 0.57.2 inside Atlantis, in the same PR some plans succeed and some don't.

We use Atlantis in ECS with EFS for storage, we run multiple instances of it with Redis as backend.

Recently we upgraded to Terragrunt 0.57.2 and Atlantis 0.27.2 to use the Terragrunt Provider Cache so we can speed up the PR planning/applying because we have over 700 terragrunt.hcl files in our code.

When doing a PR that includes an upgrade for Terraform AWS Provider it will trigger a huge number of plans and given we have Atlantis configured with --parallel-plan and --parallel-apply it starts multiple terragrunt plan processes at the same time.

To Reproduce
I do not have a step by step to reproduce but this is our env vars when running inside Atlantis:

TERRAGRUNT_DOWNLOAD="$HOME/.cache/terragrunt/modules"
TERRAGRUNT_FETCH_DEPENDENCY_OUTPUT_FROM_STATE="true"
TERRAGRUNT_PROVIDER_CACHE=1
TERRAGRUNT_NON_INTERACTIVE="true"
TERRAGRUNT_INCLUDE_EXTERNAL_DEPENDENCIES="true"

Atlantis server args:

--enable-diff-markdown-format  --hide-prev-plan-comments --automerge --hide-unchanged-plan-comments --parallel-apply --parallel-plan

Atlantis env vars:

ATLANTIS_PARALLEL_POOL_SIZE=10
ATLANTIS_ALLOW_REPO_CONFIG=true

Expected behavior
Be able to use Terragrunt Provider Cache on Atlantis using parallel plan and apply.

Nice to have

• Terminal output

time=2024-04-18T23:16:12Z level=info msg=Terragrunt Cache server is listening on 127.0.0.1:39541
time=2024-04-18T23:16:12Z level=info msg=Start Terragrunt Cache server
time=2024-04-18T23:16:16Z level=info msg=Downloading Terraform configurations from git::ssh://git@github.com/terraform-aws-modules/terraform-aws-iam.git?ref=v5.37.1 into /home/atlantis/.cache/terragrunt/modules/CZJOkESJyj2a2n17Ph1lBtPy7p8/Z_nexO2qqCg5RPmJa_gkAX4ynAY prefix=[/home/atlantis/.atlantis/repos/ACME/terraform/838/provider_aws_dev__global_iam_roles_sre-role/provider/aws/security/_global/iam/policies/sre-assume-role] 
time=2024-04-18T23:16:16Z level=info msg=Downloading Terraform configurations from git::ssh://git@github.com/ACME/tf-aws-iam-saml-provider.git?ref=v1.0.1 into /home/atlantis/.cache/terragrunt/modules/1-d3ZTASqfksjn_orsKPDUGD7ks/lPrDQ0wT1dtsjgNOnzZoq0oCtyE prefix=[/home/atlantis/.atlantis/repos/ACME/terraform/838/provider_aws_dev__global_iam_roles_sre-role/provider/aws/security/_global/iam/identity_providers/X] 
╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider
│ hashicorp/aws: host registry.terraform.io rejected the given authentication
│ credentials

time=2024-04-18T22:43:57Z level=info msg=Terragrunt Cache server is listening on 127.0.0.1:38393
time=2024-04-18T22:43:57Z level=info msg=Start Terragrunt Cache server
time=2024-04-18T22:43:58Z level=info msg=Downloading Terraform configurations from git::ssh://git@github.com/terraform-aws-modules/terraform-aws-iam.git?ref=v5.30.0 into /home/atlantis/.cache/terragrunt/modules/BBCsdcAtPBWyKxcKxrWGSmPnyMI/Z_nexO2qqCg5RPmJa_gkAX4ynAY

Error: Could not retrieve providers for locking

Terraform failed to fetch the requested providers for cache_provider in order
to calculate their checksums: some providers could not be installed:
- registry.terraform.io/hashicorp/aws: host registry.terraform.io rejected
the given authentication credentials.

Versions

• Terragrunt version: 0.57.2
• Terraform version: 1.4.6
• Environment details (Ubuntu 20.04, Windows 10, etc.): ghcr.io/runatlantis/atlantis:v0.27.2

Additional context
Atlantis configuration: https://gist.github.com/amontalban/dcd106a55390203bd30cb706e96698e6

We do not have any CLI Configuration or credentials configured.

[lev-ok]

Hi @amontalban, Unfortunately, this is not enough information to understand the cause of the issue.

│ Could not retrieve the list of available versions for provider
│ hashicorp/aws: host registry.terraform.io rejected the given authentication
│ credentials

This error is in no way related to the number of concurrent processes.
Please review how caching works https://terragrunt.gruntwork.io/docs/features/provider-cache/
Most likely, the values, TF_TOKEN_*, are somehow overwritten, which is why you cannot connect to the cache server and get rejected the given authentication credentials.

[lev-ok]

I would suggest also checking the credentials file.

[amontalban]

Hi @levkohimins,

Thanks for looking this issue, I have built a test case for easier testing.

You can find it here: https://github.com/amontalban/terragrunt-issue-3076

If you set your machine with those requirements (Mainly GNU Parallel, Docker and the specified Terraform/Terragrunt versions) then if you run make parallel you will see the error I specified in the report, on the other side if you run make sequentially it will work properly because it is executed one at a time.

Let me know if that helps or you need anything else.

Thanks once again!

[amontalban]

@levkohimins after building this test case and running some tests, I think you were on the right track.

The problem is that when I run multiple parallel plans the generated token which is an UUID will be different for each of the processes and therefore causing this authentication issue.

If I set the TERRAGRUNT_PROVIDER_CACHE_TOKEN env var or --terragrunt-provider-cache-token to be the same for all processes then this works in my test case.

I'm going to test it out in Atlantis to confirm and get back to you to confirm that solves my problem.

[lev-ok]

Hi @amontalban,

Thanks for looking this issue, I have built a test case for easier testing.

You can find it here: https://github.com/amontalban/terragrunt-issue-3076

Thank you, this should really help, I will try it out.

The problem is that when I run multiple parallel plans the generated token which is an UUID will be different for each of the processes and therefore causing this authentication issue.

Each Terragrunt process runs its own cache server (starts listening on a free port) and generates its own token, they shouldn't conflict. I suppose that this rejected the given authentication credentials error occurs when Terraform tries to connect to a server other than the one whose process run the terraform command.
The logic for determining a free port is implemented in Golang code, so the likelihood of a bug is unlikely.

That's all thoughts for now, I'll try your test if I encounter this error I will be able to say more.

[amontalban]

Hi @amontalban,

Thanks for looking this issue, I have built a test case for easier testing.
You can find it here: https://github.com/amontalban/terragrunt-issue-3076

Thank you, this should really help, I will try it out.

The problem is that when I run multiple parallel plans the generated token which is an UUID will be different for each of the processes and therefore causing this authentication issue.

Each Terragrunt process runs its own cache server (starts listening on a free port) and generates its own token, they shouldn't conflict. I suppose that this rejected the given authentication credentials error occurs when Terraform tries to connect to a server other than the one whose process run the terraform command. The logic for determining a free port is implemented in Golang code, so the likelihood of a bug is unlikely.

That's all thoughts for now, I'll try your test if I encounter this error I will be able to say more.

Hi @levkohimins,

Thank you for your explanation! I have now tested this in Atlantis and the rejected the given authentication credentials. error disappeared after using a fixed TERRAGRUNT_PROVIDER_CACHE_TOKEN value for all processes.

However I am now getting this error in some of the plans (Here are the stats 42 failing, 1 pending, and 652 successful checks):

╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider
│ hashicorp/null: could not query provider registry for
│ registry.terraform.io/hashicorp/null: the request failed after 2 attempts,
│ please try again later: Get
│ "http://127.0.0.1:36847/v1/providers/registry.terraform.io/hashicorp/null/versions":
│ dial tcp 127.0.0.1:36847: connect: connection refused

Here is the full log just in case: https://gist.github.com/amontalban/a5dbe6f5831d9ea79fd2923793a53323

Just in case I'm going to test with v0.57.6 and report back because I have been testing with v0.57.2.

Thanks once again for the help!

[lev-ok]

Hi @amontalban,
Thanks for the test case. I figured out why it doesn't work, --terragrunt-provider-cache conflicts with --terragrunt-download-dir. I have started working on fixing this issue and will update you once it is done.

[lev-ok]

Hi @amontalban @nsb413, the issue should be resolved in v0.57.9 release, could you please confirm this?

[amontalban]

Hey @levkohimins, sorry I missed your previous message. I will try to test it out tomorrow and report back.

Thank you for the quick fix!

[sshepel]

Still an issue for me.

TG: 0.57.9
TF: 1.5.6

TERRAGRUNT_PROVIDER_CACHE_DIR="somepath"
TERRAGRUNT_PROVIDER_CACHE="true"
TF_PLUGIN_CACHE_DIR is not set
TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE should be set automatically based on the caching docs.

Might be some different problem, because cache is there after the first run, but during the second run on terrgarunt run-all init -upgrade I can see that provides been installed again even though that caching server is running from what I can see in logs.

[lev-ok]

@sshepel, I could not reproduce the issue.

[sshepel]

@sshepel, I could not reproduce the issue.

I think the init logs are missleading (so it's working). Because with terragrunt cache enabled I can see this:

Initializing provider plugins...
- Finding latest version of hashicorp/template...
....
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (unauthenticated)

Where as with TF_PLUGIN_CACHE_DIR var set I saw something like: used from cache for the same provider plugin.
But I have checked the .terraform folder under .terragrunt-cache for specific module after the init step with cache and I can see that the binaries for provider plugins are linked to the cached folder as mentioned in Provider Caching docs.

Also there is a message about incomplete lock file information for providers (which seems to be fine, but very annoying).