feat(terraform): defined nearly all supported Terraform platforms for locking #4784

sebbrandt87 · 2022-03-02T06:55:54Z

For the purpose of Terraform locking, we now have defined nearly all supported Terraform platforms in the terraform providers lock command.

Except linux_arm64, as not all providers seem to be present yet for that platform.

This makes sure that we have all h1 sums in the .terraform.lock.hcl so that we keep compatability and do not run in errors on runs at other platforms than Linux.

Quote Terraform docs

If you wish to avoid ongoing additions of new h1: hashes as you work with your configuration on new target platforms, or if you are installing providers from a mirror that therefore can't provide official signed checksums, you can ask Terraform to pre-populate hashes for a chosen set of platforms using the terraform providers lock command:

terraform providers lock \
  -platform=linux_arm64 \
  -platform=linux_amd64 \
  -platform=darwin_amd64 \
  -platform=windows_amd64

The above command will download and verify the official packages for all of the required providers across all four of the given platforms, and then record both zh: and h1: checksums for each of them in the lock file, thus avoiding the case where Terraform will learn about a h1: equivalent only at a later time. See the terraform providers lock documentation for more information on this command.

resolves: #4042

jurre · 2022-03-02T07:47:50Z

Thanks for looking into this! One possible concern I have is that for folks that don't run terraform with these options locally, this would result in large diffs (and possibly merge conflicts because of those) with changes made locally vs by Dependabot

Curious to get your thoughts on this

sebbrandt87 · 2022-03-04T11:46:14Z

Hey @jurre

I wouldn't expect merge conflicts, but for sure larger diffs initially.
We might see if we really get concerns about merge conflicts. I have not seen any in the past, as most of the time a user is not explicitly locking the providers but terraform just does for us automatically depending on the platform we work on.

billinghamj · 2022-03-04T13:52:14Z

FWIW, the current situation is likely far worse, in terms of conflicts with changes made locally

Currently every Windows + Mac user has dependabot continuously removing the right architectures, resulting in a lot of people always having uncommitted changes from running terraform init locally

So every time there's a change and they pull, they have to stash/discard those changes

With this, people will no longer need to routinely change their local .terraform.lock.hcl files, so merge conflicts are far less likely

sebbrandt87 · 2022-03-04T13:59:05Z

correct @billinghamj 😄

sebbrandt87 · 2022-03-10T10:51:01Z

@jurre anything still open to be clarified?

Nishnha · 2022-03-10T17:26:56Z

Hi @sebbrandt87 and @billinghamj,

This is something that's been on our radar for a bit. I agree that Dependabot continuously removing the right architectures is pretty frustrating, so I'm open to merging this change in for the time being.

I ran some numbers on the time it took for Terraform to download hashes for mac, linux, and windows (x64) versus just one platform, and it looks like it takes ~60% longer, which makes sense, but isn't ideal. It also results in more outbound calls (where it could error) and bloats the lockfile, like mentioned before.

We discussed a couple solutions:

We include a platform key in dependabot.yml for Terraform to specify which architecture(s) to fetch. We can leave the default as is (i.e. linux x64)

version: 2
updates:
  - package-ecosystem: 'terraform'
    platform: 'linux'
    directory: '/'
    schedule:
      interval: 'daily'

We guess what architecture hashes to include based on the existing hashes in the lockfile. This would be slightly slower than the first option and is more prone to breaking.

What do y'all think?

sebbrandt87 · 2022-03-10T18:08:26Z

Hey @Nishnha
thank you very much 🥰
I am all in for option 1.

billinghamj · 2022-03-10T22:00:04Z

I'd be very, very happy with either of those options :) (though agree #1 is best, much clearer)

That said, in the temporary absence of either of those - if they wouldn't be realistic in the near term, I'd very much appreciate having this more "brute force" approach in the meantime

… locking For the purpose of Terraform locking, we now have defined nearly all supported Terraform platforms in the `terraform providers lock` command. Except linux_arm64, as not all providers seem to be present yet for that platform. This makes sure that we have all h1 sums in the `.terraform.lock.hcl` so that we keep compatability and do not run in errors on runs at other platforms than Linux. resolves: dependabot#4042

Nishnha · 2022-03-11T22:13:07Z

Sounds like consensus.

I'll work with the Dependabot team to track this work and draft a release plan!
Unfortunately, I will have to ship this PR next week since it's Friday, and this introduces a significant change to most Terraform users's lockfiles. We don't want to surprise everyone first thing Monday morning 😆

Hope we can resolve this pain point soon!

jurre · 2022-03-18T18:37:43Z

Could we maybe talk to the terraform folks and see if we can get something like a .tfrc (like .npmrc, but for tf) config file that you can commit to the repo? That way you can configure terraform to do the right thing, and Dependabot just reads that config file?

billinghamj · 2022-03-18T18:43:21Z

@jurre I think that is very unlikely to be considered. Terraform is developed quite slowly, and having now reached 1.0, is no longer changing significantly in how it works. There is also already good support for this use-case, and it automatically chooses the right options for the architecture you're using. Adding a feature just for a team inside GitHub seems unlikely

jurre · 2022-03-18T18:44:28Z

@jurre I think that is very unlikely to be considered. Terraform is developed quite slowly, and having now reached 1.0, is no longer changing significantly in how it works. There is also already good support for this use-case, and it automatically chooses the right options for the architecture you're using. Adding a feature just for a team inside GitHub seems unlikely

Surely users run into this running terraform locally as well, right? 😕 If you run on linux_arm64 and darwin_amd64 now you need to always pass those config options to the CLI, having this stored in a config file seems sensible to me for all users, not just us

sebbrandt87 · 2022-03-25T11:07:36Z

@jurre I think that is very unlikely to be considered. Terraform is developed quite slowly, and having now reached 1.0, is no longer changing significantly in how it works. There is also already good support for this use-case, and it automatically chooses the right options for the architecture you're using. Adding a feature just for a team inside GitHub seems unlikely

Surely users run into this running terraform locally as well, right? 😕 If you run on linux_arm64 and darwin_amd64 now you need to always pass those config options to the CLI, having this stored in a config file seems sensible to me for all users, not just us

I bet we can also create a PR to Terraform, so that we maybe get something like a .tfrc implemented. But it could be that it is not considered to be a valid point as this is more a workflow / convention thing in each team and if once the .terraform.lock.hcl is written, it is respected.

But as dependabot does atomic updates to the providers, without an option for this convention to be set and respected, we still have this issue.

Can't we proceed as @Nishnha said a few days ago for now, and we come back to remove this "hard" locking in the next iterations of releases / features being implemented? Please ❤️ 😊

EDIT

Friendly ping @jurre

billinghamj · 2022-04-05T10:49:07Z

terraform/lib/dependabot/terraform/file_updater.rb

@@ -14,6 +14,7 @@ class FileUpdater < Dependabot::FileUpdaters::Base
      PRIVATE_MODULE_ERROR = /Could not download module.*code from\n.*\"(?<repo>\S+)\":/.freeze
      MODULE_NOT_INSTALLED_ERROR =  /Module not installed.*module\s*\"(?<mod>\S+)\"/m.freeze
      GIT_HTTPS_PREFIX = %r{^git::https://}.freeze
+      TF_PLATFORMS = "-platform=linux_amd64 -platform=darwin_amd64 -platform=windows_amd64"


Could we make sure to include Apple Silicon support too?

Suggested change

TF_PLATFORMS = "-platform=linux_amd64 -platform=darwin_amd64 -platform=windows_amd64"

TF_PLATFORMS = "-platform=linux_amd64 -platform=darwin_amd64 -platform=darwin_arm64 -platform=windows_amd64"

I am unsure if we can handle this atm, we would need to make sure that for all basic terraform providers we have the darwin_arm64 build available. I can tell that we don't have the linux_arm64 nor darwin_arm64 available for all, that is why I excluded it for now from the static TF_PLATFORMS. To check some you can have a look at https://releases.hashicorp.com

Mmm this probably needs to be configurable rather than hardcoded then I think - unless e.g. it can just add all available platforms

For this reason we talked about #4784 (comment).
But for now we should have at least something to drop pain points most of us share on a daily / weekly basis 😉

The number of people using Apple Silicon MacBooks now is pretty huge - so I don't think this prevents the pain for "most" at this point really

I think I'll look into making a quick workflow which just updates the dependabot PRs to add on the other providers

billinghamj · 2022-04-13T13:57:14Z

For the benefit of anyone remaining frustrated at this problem still not getting any traction, you can make an actions workflow to deal with this short-term:

https://gist.github.com/billinghamj/2a127b9cbd475e6b210d10431ccdde30

@sebbrandt87 in case you find it helpful

To GitHub - this still needs to be fixed. It's very disappointing that you agreed to work with us to figure this out, that you said you'd release this PR, and that a month later you have gone silent. As time goes on, it gets increasingly more difficult to recommend Dependabot to other engineers.

Nishnha · 2022-04-13T20:54:27Z

Hi all, sorry for my general lack of response on this issue. I should have posted this update sooner.

I wanted to try out automatically detecting platform architectures by checking against the existing hashes in .terraform.lock.hcl. This work is in #4905, and it's proving to be to a reasonable solution to this problem--since it would "just work" for users, and it only slows down the update job by a few seconds (updates usually operate on the scale of minutes).

I realized, however, that if we merge in this PR, we would have already written other architectures to a user's lockfile, and we would need to reverse that if we try to automatically detect architectures. The only way I can think of reversing this is if we force users manually specify them again with --platform.

Given the "just work" nature that we are going for with Dependabot, we will need to think of a way to reverse this PR or we may unintentionally break Terraform projects for many users. Please let me know if you all think of a solution so we can get this PR merged in.

The code for automatically detecting Terraform architectures (#4905) is still failing one of its tests and hasn't been fully optimized, but the Dependabot team would like to move forward with that approach if possible.

We also would like to eventually work with Hashicorp to bring a .tfrc config file to Terraform like Jurre stated, since it would solve the problem for the greater community. If you know any Terraform maintainers, we would love to work with them on scoping out this feature!

Sorry for changing things from what I had originally stated and thank you for the Action @billinghamj.
Please let me know what you all think.

billinghamj · 2022-04-14T11:16:14Z

Thank you @Nishnha - it's great to understand how the work has changed, and makes sense that this could conflict with it. Will follow the other PR longer term :)

Nishnha · 2022-04-15T06:32:51Z

Opened #4905 for review

sebbrandt87 · 2022-04-19T17:39:13Z

Closing in favor of #4905

sebbrandt87 requested a review from a team as a code owner March 2, 2022 06:55

sebbrandt87 force-pushed the feat/terraform-providers-locking-platforms branch 2 times, most recently from 5c24eba to 0a4d2ba Compare March 2, 2022 07:28

sebbrandt87 force-pushed the feat/terraform-providers-locking-platforms branch 2 times, most recently from 6f0d528 to de3f846 Compare March 4, 2022 11:31

sebbrandt87 force-pushed the feat/terraform-providers-locking-platforms branch from de3f846 to 5f0d559 Compare March 11, 2022 09:18

sebbrandt87 changed the title ~~feat(terraform): defined all supported Terraform platforms for locking~~ feat(terraform): defined nearly all supported Terraform platforms for locking Mar 11, 2022

Merge branch 'main' into feat/terraform-providers-locking-platforms

5b68646

sebbrandt87 added 2 commits March 15, 2022 14:05

Merge branch 'main' into feat/terraform-providers-locking-platforms

2a8eae0

Merge branch 'main' into feat/terraform-providers-locking-platforms

8dea5dc

sebbrandt87 added 2 commits March 25, 2022 12:07

Merge branch 'main' into feat/terraform-providers-locking-platforms

ceae4b2

Merge branch 'main' into feat/terraform-providers-locking-platforms

0c3d774

billinghamj reviewed Apr 5, 2022

View reviewed changes

Nishnha mentioned this pull request Apr 19, 2022

Automate Terraform Platform Detection for Lockfile Hashes #4905

Merged

sebbrandt87 closed this Apr 19, 2022

Nishnha mentioned this pull request Jul 12, 2022

fix(terraform): correctly keep platform hashes when multiple providers are installed #5341

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(terraform): defined nearly all supported Terraform platforms for locking #4784

feat(terraform): defined nearly all supported Terraform platforms for locking #4784

sebbrandt87 commented Mar 2, 2022 •

edited

jurre commented Mar 2, 2022

sebbrandt87 commented Mar 4, 2022 •

edited

billinghamj commented Mar 4, 2022

sebbrandt87 commented Mar 4, 2022

sebbrandt87 commented Mar 10, 2022

Nishnha commented Mar 10, 2022

sebbrandt87 commented Mar 10, 2022

billinghamj commented Mar 10, 2022 •

edited

Nishnha commented Mar 11, 2022 •

edited

jurre commented Mar 18, 2022

billinghamj commented Mar 18, 2022

jurre commented Mar 18, 2022 •

edited

sebbrandt87 commented Mar 25, 2022 •

edited

billinghamj Apr 5, 2022

sebbrandt87 Apr 10, 2022

billinghamj Apr 11, 2022

sebbrandt87 Apr 13, 2022

billinghamj Apr 13, 2022 •

edited

billinghamj commented Apr 13, 2022 •

edited

Nishnha commented Apr 13, 2022 •

edited

billinghamj commented Apr 14, 2022

Nishnha commented Apr 15, 2022

sebbrandt87 commented Apr 19, 2022

	TF_PLATFORMS = "-platform=linux_amd64 -platform=darwin_amd64 -platform=windows_amd64"
	TF_PLATFORMS = "-platform=linux_amd64 -platform=darwin_amd64 -platform=darwin_arm64 -platform=windows_amd64"

feat(terraform): defined nearly all supported Terraform platforms for locking #4784

feat(terraform): defined nearly all supported Terraform platforms for locking #4784

Conversation

sebbrandt87 commented Mar 2, 2022 • edited

jurre commented Mar 2, 2022

sebbrandt87 commented Mar 4, 2022 • edited

billinghamj commented Mar 4, 2022

sebbrandt87 commented Mar 4, 2022

sebbrandt87 commented Mar 10, 2022

Nishnha commented Mar 10, 2022

sebbrandt87 commented Mar 10, 2022

billinghamj commented Mar 10, 2022 • edited

Nishnha commented Mar 11, 2022 • edited

jurre commented Mar 18, 2022

billinghamj commented Mar 18, 2022

jurre commented Mar 18, 2022 • edited

sebbrandt87 commented Mar 25, 2022 • edited

billinghamj Apr 5, 2022

Choose a reason for hiding this comment

sebbrandt87 Apr 10, 2022

Choose a reason for hiding this comment

billinghamj Apr 11, 2022

Choose a reason for hiding this comment

sebbrandt87 Apr 13, 2022

Choose a reason for hiding this comment

billinghamj Apr 13, 2022 • edited

Choose a reason for hiding this comment

billinghamj commented Apr 13, 2022 • edited

Nishnha commented Apr 13, 2022 • edited

billinghamj commented Apr 14, 2022

Nishnha commented Apr 15, 2022

sebbrandt87 commented Apr 19, 2022

sebbrandt87 commented Mar 2, 2022 •

edited

sebbrandt87 commented Mar 4, 2022 •

edited

billinghamj commented Mar 10, 2022 •

edited

Nishnha commented Mar 11, 2022 •

edited

jurre commented Mar 18, 2022 •

edited

sebbrandt87 commented Mar 25, 2022 •

edited

billinghamj Apr 13, 2022 •

edited

billinghamj commented Apr 13, 2022 •

edited

Nishnha commented Apr 13, 2022 •

edited