feat: terraform autodiscovery

[mcwarman]

Fix #534

Add terraform autodiscovery support for lock files.

Test

To test this pull request, you can run the following commands:

cd pkg/plugins/autodiscovery/terraform/
go test ./...

name: Terraform Auto Discovery

autodiscovery:
  crawlers:
    terraform:
      platforms:
        - linux_amd64
        - linux_arm64
        - darwin_amd64
        - darwin_arm64

Additional Information

Tradeoff

Potential improvement

• Support terraform/providers

[olblak]

While this a great PR, I have one important change and a few questions

Jsonschema

The variable AutodiscoverySpecsMapping:

updatecli/pkg/core/pipeline/autodiscovery/main.go

Line 36 in 91b2be1

AutodiscoverySpecsMapping = map[string]interface{}{

needs the mapping for Terraform similarly to the other entries, that variable is used to generate the Jsonschema spec which is ultimately used to generate the website documentation and IDE.

Default Autodiscovery

The variable DefaultCrawlerSpecs could have terraform listed there, that variable contains the default autodiscovery plugins we want to execute when we run updatecli diff --experimental. Note that every autodiscovery plugins shouldn't necessary be there as we may have collusion between plugin but I think it could be interesting in the current situation.

Platform

I did a quick test with the following manifest on https://github.com/jenkins-infra/azure

autodiscovery:
  crawlers:
    terraform:

and it complained about missing platform, I had to use

autodiscovery:
  crawlers:
    terraform:
      platforms:
        - linux_amd64
        - linux_arm64
        - darwin_amd64
        - darwin_arm64

It's been a while since I had to maintain Terraform and I don't remember the purpose of that platform setting, could it be detected? If not that's ok but then we shouldn't add the terraform plugin to the default list of plugin execute as it will always fail

[mcwarman]

AutodiscoverySpecsMapping

I'll add that.

Default Autodiscovery

It can be added, but only if we solve Platform

Platform

When running terraform init it will detect the platform you are running on you can then add hashes for that. You can add additional platforms using terraform providers lock command.

An option is provide a sensible default maybe just linux_amd64 if nothing is specified.

[olblak]

Thanks for the information,

An option is provide a sensible default maybe just linux_amd64 if nothing is specified.

That's what I initially tried but it was confusing because Updatecli reported some changes such as bump version from "1.0.0" to "1.0.0" basically I had to specify the four platforms to show me the correct result.

If it's not possible to detect what platform is in used within the lock then I guess the best approach is to not run the terraform autodiscovery by default

[mcwarman]

That's what I initially tried but it was confusing because Updatecli reported some changes such as bump version from "1.0.0" to "1.0.0" basically I had to specify the four platforms to show me the correct result.

Yeah that would be because the hashes would have been removed for the other platforms, there's probably some additional logging required for when that happens.

If it's not possible to detect what platform is in used within the lock then I guess the best approach is to not run the terraform autodiscovery by default

Detecting platforms would be expensive as I don't think there's anyway to look up hashes origin without downloading all of the artefacts and cross checking them.

Should there be a default added so the empty config works?

autodiscovery:
  crawlers:
    terraform:

[olblak]

I think I would do something like

if len(platforms) == 0 { 
  logrus.Warningf("No platforms specified, we fallback to linux_amd64, linux_arm64, darwin_amd64, darwin_arm64")
  set default value
}

[mcwarman]

Also added log message:

⚠ - "hashicorp/random" already set to "3.5.1", hashes for the provider have been updated in file ".terraform.lock.hcl"

[olblak]

Thanks for the quick feedback, I just did some testing and it works great

[olblak]

Thanks @mcwarman for the PR

[olblak]

@mcwarman I almost forgot but that would be great to have some documentation on https://github.com/updatecli/website/tree/master/content/en/docs/plugins/autodiscovery

[mcwarman]

@olblak Added the following:

• docs: update terraform autodiscovery documentation #1627 - The curse of copy and paste
• docs: add terraform autodiscovery website#1162