fix(host/go): Download external plugins even if they're not imported

[abhinav]

We previously changed how the Go language host retrieves information
about a Go module:

before: go list -m -json -mod=mod all
after:  go list -m -json $importPath1 $importPath1

This change made it possible for us to supprot running in vendored mode
which allowed for use of private plugins that are not go-get-able.

This uncovered a corner case in the go command's behavior
when running in module mode:

If a package is listed in the go.mod
but it's not imported by the current module,
go list -m -json $importPath will not list its Dir in the output
even if it's present in the module cache.
For example, given a go.mod file that declares a dependency
but code that doesn't import it, as of Go 1.20.5,

export GOMODCACHE=$(mktemp -d)
go mod download -json
go list -m -json $importPath

The output of go mod download will include information
about the dependency and the Dir where it was downloaded,
but go list -m will not.

Unfortunately, we can't just use go mod download
because that breaks vendoring:
vendored dependencies cannot always be redownloaded.

To resolve this issue,
we switch to a two-pass variant to gather this information:

• Run go list -m to get whatever information we can locally.
  This will be sufficient for the majority of use cases.
• For the handful of cases where the dependency isn't imported,
  we'll use go mod download -json to download them on-demand
  and get their location from that instead.

The go mod download step will take place only if we're in module mode.
In vendor mode, we'll work with what we have.

Resolves #13301

[pulumi-bot]

Changelog

[uncommitted] (2023-07-11)

Bug Fixes

• [sdk/go] Fix downloading of unimported external plugins.
  #13455

[abhinav]

Repro: https://github.com/pulumi/pulumi/actions/runs/5525559114/jobs/10079407337?pr=13455

=== FAIL: integration TestAutomation_externalPluginDownload_issue13301 (3.04s)
    environment.go:97: Created new test environment:  /tmp/test-env2781205917
    integration_go_test.go:964: Running command pulumi login --cloud-url file:///tmp/test-env2781205917
    integration_go_test.go:992: Running command pulumi plugin install
    integration_go_test.go:1024: 
        	Error Trace:	/home/runner/work/pulumi/pulumi/tests/integration/integration_go_test.go:1024
        	Error:      	Received unexpected error:
        	            	failed to run preview: exit status 255
        	            	code: 255
        	            	stdout: Previewing update (foo):
        	            	
        	            	 +  pulumi:pulumi:Stack issue-13301-foo create 
        	            	    pulumi:providers:hcp hcp  error: could not find latest version for provider hcp: GetLatestVersion is not supported for plugins from http sources
        	            	 +  pulumi:pulumi:Stack issue-13301-foo create 
        	            	    pulumi:providers:hcp hcp  1 error
        	            	
        	            	Diagnostics:
        	            	  pulumi:providers:hcp (hcp):
        	            	    error: could not find latest version for provider hcp: GetLatestVersion is not supported for plugins from http sources
        	            	
        	            	
        	            	
        	            	stderr: 
        	Test:       	TestAutomation_externalPluginDownload_issue13301
    integration_go_test.go:985: grpc debug log:
        {"method":"/pulumirpc.LanguageRuntime/GetPluginInfo","request":{},"response":{},"metadata":{"kind":"language","mode":"client","runtime":"go"}}
        {"method":"/pulumirpc.LanguageRuntime/GetRequiredPlugins","request":{"project":"issue-13301","pwd":"/tmp/test-env2781205917","program":"."},"response":{},"metadata":{"kind":"language","mode":"client","runtime":"go"}}
        {"method":"/pulumirpc.LanguageRuntime/GetRequiredPlugins","request":{"project":"issue-13301","pwd":"/tmp/test-env2781205917","program":"."},"response":{},"metadata":{"address":"127.0.0.1:[355](https://github.com/pulumi/pulumi/actions/runs/5525559114/jobs/10079407337?pr=13455#step:34:356)33","kind":"language","mode":"client"}}
        {"method":"/pulumirpc.ResourceMonitor/SupportsFeature","request":{"id":"resourceReferences"},"response":{"hasSupport":true},"metadata":{"mode":"server"}}
        {"method":"/pulumirpc.ResourceMonitor/SupportsFeature","request":{"id":"outputValues"},"response":{"hasSupport":true},"metadata":{"mode":"server"}}
        {"method":"/pulumirpc.ResourceMonitor/SupportsFeature","request":{"id":"deletedWith"},"response":{"hasSupport":true},"metadata":{"mode":"server"}}
        {"method":"/pulumirpc.ResourceMonitor/SupportsFeature","request":{"id":"aliasSpecs"},"response":{"hasSupport":true},"metadata":{"mode":"server"}}
        {"method":"/pulumirpc.ResourceMonitor/RegisterResource","request":{"type":"pulumi:pulumi:Stack","name":"issue-13301-foo","object":{},"acceptSecrets":true,"customTimeouts":{},"acceptResources":true},"response":{"urn":"urn:pulumi:foo::issue-13301::pulumi:pulumi:Stack::issue-13301-foo","object":{}},"metadata":{"mode":"server"}}

[abhinav]

Note that this fix depends on #13447.
Thanks to @Frassle for fixing.

[Frassle]

LGTM

[abhinav]

bors merge

[bors]

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

• bors-ok