Support source URLs for GitLab-provided terraform modules

[secustor]

Discussed in #25002

^{Originally posted by colinodell October 3, 2023}
TL;DR: I would like Renovate's terraform-module datasource to always capture the source URL field if provided by the registry, as some registries (like GitLab's) do expose this non-standard field.

Details

The Terraform Module Registry Protocol defines a minimal specification that all implementations must follow. Notably, the "list available versions" endpoint must return responses like this:

{
   "modules": [
      {
         "versions": [
            {"version": "1.0.0"},
            {"version": "1.1.0"},
            {"version": "2.0.0"}
         ]
      }
   ]
}

However, implementations are allowed to provided a superset of the API:

The public Terraform Registry implements a superset of the API described on this page, in order to capture additional information used in the registry UI. For information on those extensions, see Terraform Registry HTTP API. Third-party registry implementations may choose to implement those extensions if desired, but Terraform CLI itself does not use them.

Both the public Terraform Registry and GitLab's Terraform Registry provide an additional source field which may (or may not) contain a valid URL.

Currently, Renovate will only check for and use that source field if the registry URL contains terraform.io:

renovate/lib/modules/datasource/terraform-module/index.ts

Lines 61 to 73 in 2554a5e

	if (this.extendedApiRegistryUrls.includes(registryUrlNormalized)) {
	return await this.queryRegistryExtendedApi(
	serviceDiscovery,
	registryUrlNormalized,
	repository
	);
	}
	return await this.queryRegistryVersions(
	serviceDiscovery,
	registryUrlNormalized,
	repository
	);

(This next code snippet is only present in the queryRegistryExtendedApi() method)

renovate/lib/modules/datasource/terraform-module/index.ts

Lines 114 to 116 in 2554a5e

	if (res.source) {
	dep.sourceUrl = res.source;
	}

I propose that Renovate should always check for that source field (for other registries via queryRegistryVersions()), and use that value as the sourceUrl if it indeed contains a URL. The proposed implementation would look something like this:

  /**
   * this version uses the Module Registry Protocol that all registries are required to implement
   * https://www.terraform.io/internals/module-registry-protocol
   */
  private async queryRegistryVersions(
    serviceDiscovery: ServiceDiscoveryResult,
    registryUrl: string,
    repository: string
  ): Promise<ReleaseResult | null> {
    let res: TerraformModuleVersions;

    // ...

+   // Add the source URL if given
+   if (res.modules[0].source && validateUrl(res.modules[0].source)) {
+     dep.sourceUrl = res.modules[0].source;
+   }
+
    return dep;
  }

(Some other minor tweaks are also needed; this is just the important bit)

Basically, if a source key is found and it looks like a URL, let's assume it must be a source URL.

I have this working locally (along with an automated test) and would be glad to push up a PR if you're open to this change :)

[renovate-release]

🎉 This issue has been resolved in version 37.6.0 🎉

The release is available on:

• GitHub release
• 37.6.0

Your semantic-release bot 📦🚀