feat(modules): optional Terraform module registry with pull-through caching

[pascalinthecloud]

Closes the last open roadmap item. Adds MODULES_ENABLED (off by default), serving Terraform's module registry protocol at /.well-known/terraform.json and /v1/modules/.

Module version lists, resolved locations, and archives are cached in the same two-layer cache as provider zips, reusing the existing TTL revalidation, serve-stale-on-outage, and singleflight coalescing.

The catch, up front

Unlike providers this is a registry, not a mirror — Terraform defines no module mirror protocol. Consumers must rewrite each module's source:

module "regions" {
  source  = "tf-mirror.internal/claranet/regions/azurerm"  # was: claranet/regions/azurerm
  version = "8.0.6"
}

Module archives are also cached unverified — the protocol publishes no checksums, so they get no equivalent of the SHA-256 check provider zips receive. Defenses are an https-only fetch and a 512 MiB cap. This is documented prominently in the README.

Three things the spec doesn't prepare you for

All found against the live registry and the real CLI, not assumed:

1. The docs are stale. registry.terraform.io returns git::https://github.com/OWNER/REPO?ref=<sha> for every module sampled, not the api.github.com/…//*?archive=tar.gz tarball the spec shows. Without handling this, nothing from the public registry would be cached at all. These are mapped onto the equivalent codeload.github.com tarball — no git client, no new dependency.
2. Terraform does not expand the go-getter //* subdir glob for registry modules. It records the literal path .terraform/modules/<name>/* and fails with "Unreadable module subdirectory". So the tarball's single wrapper directory is stripped on the way through (repack.go) rather than papered over with a glob. Deriving the wrapper name from REPO-REF would have been simpler but wrong — GitHub strips a leading v from tag refs.
3. Credentials never reach the archive. Terraform's ModuleLocation calls addRequestCreds only on the registry request; the go-getter fetch that follows carries no Authorization header. The archive endpoint is therefore mounted outside bearer auth — otherwise terraform init breaks whenever AUTH_TOKEN is set.

Also: registering the module and provider route patterns on one ServeMux panics at startup (they overlap with neither more specific). Providers stay on mirrorMux, modules on the root mux, with a regression test guarding the split.

Sources that can't be fetched (non-GitHub git::, ssh://, s3::) pass through verbatim with X-Cache: BYPASS rather than failing — visible via the new terrastrata_module_downloads_total{outcome} metric.

Refactors

internal/pathsafe (traversal-proof validation) and internal/freshness (TTL envelope) extracted from internal/mirror and shared by both protocols. Pure moves, no behavior change — the existing mirror tests pass untouched.

Verification

• Unit tests across modules, pathsafe, config, and cmd — protocol parsing, repacking (including the pax_global_header GitHub emits), coalescing, size cap, auth boundary, and the mux-conflict guard.
• Real terraform init against the live registry, run locally end to end: module installs at its own root, second init serves everything from cache, module_downloads_total{outcome="cached"} 2 with zero bypass.
• New E2E CI step does the same with the real CLI behind the existing TLS proxy. It's the only test that can catch the subdir and archive-shape problems above.
• make test, lint, go vet, and govulncheck all clean.

Chart bumped to 0.4.0 (modules.enabled, modules.upstreamBase).

Known limitations

• Only GitHub-hosted git:: sources are cacheable; others pass through.
• No module pre-warming, and no private/multi-upstream module registry support.