Replies: 1 comment 1 reply
-
|
Check out this change coming soon in If you want to try it out, you can install a tip build: |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
I’m really relieved that you’ve already taken a proactive approach to this issue. Many thanks — your solution exactly addresses my problem. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hello,
this is a discussion as follow up of #5986
@yhakbar , according to your last comment I kindly ask for a good strategy regarding the changed behaviour.
Note: For clarification - I have no concerns regarding the change on how the lock file is treated. The new behaviour is in my opinion the correct way to use the lock file.
In my last comment on issue 5986 I wrote, the easiest solution is to wait for OpenTofu 1.12 and everything is fine then. Now, as 1.12 is released, I found things not as easy as I hoped.
Let me explain my challenge. My setup is Terragrunt + OpenTofu + overall only few (<10) different provider. My local environment is darwin_arm64, while the build machines are linux_amd64.
The one project with my hardest challenge contains 135 terragrunt modules. The task is to get the hashes for at least both platforms in all lock files.
Maybe we start with the straight forward approach and just run
terragrunt run --all -- init [-upgrade]in root directory. Because of changes OpenTofu did with version 1.12, as a result, I will have 135 lock files containingzhandh1hashes for all supported platforms. This fulfils the desired outcome. However, the resulting data transfer is huge, because each module re‑downloads the providers it needs. Additionally, I'm frequently affected by rate limiting.To mitigate the waste of bandwidth, I thought a good idea is to enable Terragrunt Provider Cache. On first sight things are much better now. Init (upgrade) is incredible fast and each provider is downloaded only once. Unfortunately the lockfiles contain only the
h1hash for the current platform (in my case darwin_arm64).This was okay for Terragrunt prior to version 1, as the lock files were not used in a read-only manner. When executed on linux_amd64 pipeline, when the cache is empty, the provider is downloaded (+added to the cache), the
zhhash is verified, and the new/missingh1hash is added to the lock file.Now with Terragrunt 1, as OpenTofu init is invoked with
-lockfile=readonly, the required linux_amd64 hash is not found in the lock file and further processing is stopped.The solution/mitigation is to explicitly add all required platform hashes:
terragrunt run --all -- providers lock -platform=linux_amd64 -platform=darwin_arm64. As a result all lock files will again contain allh1hashes from all supported platforms (and not only the two provided as argument). I don't know from where all theses hashes (here I only talking about the hashes and not the provider for the other platform itself) are taken. But either they are again retrieved from remote registry (and in this case again 135 times) or they are somehow extracted from the provider present in TG provider cache. Wherever the information is taken from, it is done over and over again and nothing is cached (or added back to cached data).Here you can see for a very small project tree, how it looks like:
Now back to my initial request, what is the best strategy to not download and/or calculate the same piece of information multiple times? Even across project boundaries. Other Terragrunt projects usually use the same versions of the same provider and should only work with data cached in the provider cache. But for me it look like the
h1hashes are not scope in no cache and therefore require re-download/re-calculation for every terragrunt module again. Or do I miss essential things here?Beta Was this translation helpful? Give feedback.
All reactions