Configurable download URLs for corporate/air-gapped environments

[linkvt]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Summary: It seems that the download URLs of pulumi providers are not easily configurable from a user which causes problems in our air-gapped/isolated environments.

We want to use Pulumi in a large corporate environment that has all kinds of restrictions (network segments without internet access, proxies, ...) making the developer life a bit harder but "more secure".
We have a pull through cache (Artifactory) set up which can be configured/used to download NPM dependencies or artifacts from HTTP sources.
Some network segments have public access but will see errors with pulumi, as rate limits for these public egress IPs are usually reached quickly.

It seems that the URLs used for downloading providers/plugins cannot be overwritten through a configuration file - at least not during runtime.
After looking at the plugins.go file it seems that this overwrite is only supported during build time?

I think a generic solution would be to allow overwriting a download URL in a similar fashion by letting the user provide a regex for either a download URL or a package name and a new URL that will get used instead of the original base URL.
The new URL can contain a path that should be used as a base, pulumi should only add path segments instead of replacing the path.

A workaround we use looks as follows:

# list the providers and versions we use
❯ npm list --omit dev | grep -v 'pulumi/pulumi'
pulumi-project@ /home/me/repo
├── @pulumi/aws@6.34.1
└── @pulumi/random@4.16.1
 
# run per provider in a script or manually
PROVIDER=aws
VERSION=6.34.1
pulumi plugin install resource ${PROVIDER} ${VERSION} --server "https://internal.example.com/some/paths/pulumi/pulumi-${PROVIDER}/releases/download/v${VERSION}/"

While this definitely works, it is not a great dev experience, especially with regular updates to the dependencies.

I'm definitely up for providing a PR here, but have no idea what the ideal solution is, some concept work would need to be done first (configurable per user in ~/.config, overwritable in projects/stacks, match by package name or URL, ...).

Affected area/feature

Download of Plugins/Providers

Related Issues

• Add environment variable to only get providers from get.pulumi.com #14459
  • too specific, would not solve our use case but overwritable download URLs as requested here would solve it, too

[justinvp]

Thanks for the suggestion and writing up this issue!

After looking at the plugins.go file it seems that this overwrite is only supported during build time?

That's right.

This is also related to #13484, which is one place where such config could live.