Cache Go builds with rebuild: true runtime option

[t0yv0]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Consider introducing a rebuild: true runtime option for Go programs.
The option is to be used in conjunction with binary: $dest option and
will instruct Pulumi to rebuild the binary with go build -o $dest
prior to executing:

name: mygoproject
description: A minimal Go Pulumi program
runtime:
  name: go
  options:
   binary: ./my-program
   rebuild: true

In version v3.38.0 Pulumi commands such as pulumi up used go run to execute Go code, but is going to use go build -o $tempfile with a temp file after #10375

Pulumi also supports binary option. If set, Pulumi does not build code at all but executes a prebuilt code at the location given by binary.

As it turns out, go build -o $dest can be much more efficient than go build -o $tempfile on repeat invocations. The rebuild: true opens a way to rebuild the binary efficiently as part of normal pulumi up.

It should be an error to specify rebuild: true without also specifying binary.

Alternatives

This feature requires explicit user opt-in to result in speedups. We
considered an alternative design that would speed up all user programs
transparently upon upgrade: a go compilation cache. Pulumi would
manage a LRU binary cache under ~/.pulumi. We decided against this
given some complexity in dealing with locking and managing disk space.

Rollout

• Implement the feature
• Release
• Document in https://www.pulumi.com/docs/reference/pulumi-yaml/#runtime-options
• Advertise and collect feedback
• Migrate pulumi/templates to use the feature
• Migrate pulumi/examples to use the feature

Potential speedup

$ pulumi new azure-native-go
$ go build -o proj # initial build

$ hyperfine -i 'go run .'
Benchmark 1: go run .
  Time (mean ± σ):      1.276 s ±  0.036 s    [User: 1.732 s, System: 0.963 s]
  Range (min … max):    1.233 s …  1.333 s    10 runs

  Warning: Ignoring non-zero exit code.

$ hyperfine 'go build -o proj'
Benchmark 1: go build -o proj
  Time (mean ± σ):     228.1 ms ±   2.5 ms    [User: 762.7 ms, System: 782.0 ms]
  Range (min … max):   224.5 ms … 233.2 ms    12 runs

$ hyperfine -i './proj' -n 20
Benchmark 1: 20
  Time (mean ± σ):      12.3 ms ±   1.7 ms    [User: 7.0 ms, System: 3.3 ms]
  Range (min … max):    11.6 ms …  34.1 ms    175 runs

  Warning: Ignoring non-zero exit code.
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

So the new method would result in a warm start (after initial build)
of 250ms for a Pulumi project using azure-native compared to the 1.2s
baseline with go run. It's common for users to run several
operations that don't result in relinking the binary so these savings
can be meaningful.

Affected area/feature

Go provider.

[t0yv0]

Noting idea from @AaronFriel to add an option with the path to the binary so users can opt-in to a workspace-local binary path. This seems reasonable.

#10375 (review)