Provider Cache Server clarification (and possible bug)

[dave-v]

The Provider Cache Server feature ensures that each provider is only ever downloaded and stored on disk exactly once¹

I work on a large Terragrunt/Terraform project with hundreds of terragrunt.hcl files, and no matter how much I fiddle with the provider cache server feature, providers are being duplicated on disk. This means I often end up with ~200GB of providers in my cache directory. I've read through all the pre-existing issues on the subject I could find, but that has not yielded any success.

I am running commands such terragrunt run --all apply from various subdirectories with the following environment variables:

TF_PLUGIN_CACHE_DIR=C:\.terraform-cache
TG_DOWNLOAD_DIR=C:\.terragrunt-cache
TG_PROVIDER_CACHE=1
TG_PROVIDER_CACHE_DIR=C:\.terragrunt-cache\providers

When I start up, log messages indicate the provider cache server is running successfully.

Every unit in the run has log messages such as:

[s3] Downloading Terraform configurations from {repo-directory} into C:/.terragrunt-cache/IaOznkz2RQHNlRPWSzBWj26t9gc/u_SjJeLIAzG6p_jjY4-Rcxi8KCo
[s3] Caching terraform providers for C:/.terragrunt-cache/IaOznkz2RQHNlRPWSzBWj26t9gc/u_SjJeLIAzG6p_jjY4-Rcxi8KCo/modules/s3

The second message only appears if the cache directory didn't already exist.

And C:/.terragrunt-cache/IaOznkz2RQHNlRPWSzBWj26t9gc/u_SjJeLIAzG6p_jjY4-Rcxi8KCo contains a copy of my entire directory structure for the project with the provider binary for the unit in question copied into the relevant module's subtree.

The entire directory tree being duplicated for every unit seems excessive already, but that's not such an issue on its own compared to the large provider binaries being duplicated everywhere too. My understanding of the provider cache server mechanism is that this should not happen, but I'm not entirely convinced I understand how it is supposed to behave, therefore I didn't open this as a bug.

I do have a single copy of all the providers located in my TG_PROVIDER_CACHE_DIR directory, and I would have expected these to be used rather than being copied each time.

Since I'm on Windows, I'm wondering if it might be an issue with symbolic links?

I am running on:
Windows 10
Terragrunt v0.93.0 (I was also seeing the same behaviour on v0.57.9)
Terraform 1.13.4

TL;DR: Despite using the provider cache server, provider binaries are being duplicated once per terragrunt.hcl file that gets initialised. Have I misunderstood the provider cache server feature or does it look like a probable bug?

Many thanks for any help!

Footnotes

1. https://terragrunt.gruntwork.io/docs/features/provider-cache-server/ ↩

[coderabbitai]

Caution

The CodeRabbit agent's plans did not produce any file changes.

[yhakbar]

Hey @dave-v ,

I think we need the ability to reproduce the behavior you're seeing to give you any feedback on it.

The Provider Cache Server should be deduplicating instances of providers on your machine, so I would recommend you create a small fixture (e.g. three Terragrunt units that just do something simple with a local file provider or null resource, etc.) to reproduce the behavior you're seeing. We can then reproduce this on our machines to better understand the issue you're facing. It might also give you a clue as to what is different in your set up vs the expected behavior of the feature.

[dave-v]

Hi @yhakbar, I've put together a small repo that reproduces the issue for me: https://github.com/QuickRelease/terragrunt-cache-test

[yhakbar]

@dave-v ,

I've had a chance to look into this a bit further, and I think I know what's happening here. I think you're dealing with OpenTofu/Terraform not symlinking the provider into the .terraform directory because of a Windows permissions issue.

Are you using Developer Mode or Administrator privileges in Windows while using Terragrunt and Terraform/OpenTofu? If not, OpenTofu/Terraform won't have permissions to create symbolic links and will fallback to copying the provider from your provider cache.

See the following for more information:
https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/security-policy-settings/create-symbolic-links

[yhakbar]

We've added a tip to call out that folks might be dealing with permissions issues in Windows environments when a test symlink fails on Windows environments while the provider cache is active:

#5778

It'll be included in the next patch release. If you want to test it, you can test out a local build of Terragrunt.

We're working on building out a tip build system to make testing out the latest changes easier.