Enable single binary Go Automation API programs

[EvanBoyle]

Hello!

• Vote on this issue by adding a 👍 reaction
• To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already)

Issue details

Currently, the Automation API requires a CLI to be installed locally. But since the Pulumi engine is written in Go, it is possible (with some refactoring) to import the Pulumi engine directly if you're writing a Go program. Here's a demo that does just that. Notice there is no CLI installation:

In this demo we build a single 50MB binary that is self contained. It is able to run without a Pulumi CLI installed. This has a number of upsides for building tooling, and services around pulumi.

Demo code: https://github.com/EvanBoyle/singleBinaryAutomationAPI
Pulumi PoC branch: https://github.com/pulumi/pulumi/compare/evan/singleBinaryAuto

The general strategy here is to refactor logic out of the pulumi cobra commands into a separate package (in the PoC case pkg/cmd). Then cobra commands are refactored to consume this common logic. In addition, the Go Automation API is moved to a top level module github.com/pulumi/pulumi/auto/v3 so it can take a dependency on pulumi/pkg. Next anywhere that automation api shells out to the CLI, we make a update that calls into pkg/cmd directly.

I only went through this exercise (very roughly) for pulumi stack init, pulumi stack select, pulumi verison, and pulumi up for the purposes of the PoC. Generally, it seems like this strategy can work without any breaking changes to the API shape.

While this only offers single binary programs for go, the approach could be extended to benefit other languages as well. Refactoring the command interface to be consumed directly by automation api would make it far easier to add a gRPC layer that other Automation API SDKs could consume (you could imagine then generating a java automation api client from protobufs).

There is still a little work to do here to truly get to single binary. We need to explore what it would look like to link in a provider statically, but I believe there is a path forward there.

[EvanBoyle]

In the meantime while design continues on this issue, I wanted to point folks towards Pulumi Deployments: https://www.pulumi.com/blog/pulumi-deployments/

A good way to think of deployments is "Automation API as a Service" behind a REST API. We are already finding many users building on top of it to create infrastructure APIs, IaaS products, and more. There is a #pulumi-deployments channel in community slack if you'd like to learn more and plenty of demos to be found at github.com/pulumi/deploy-demos