Automation API - Go SDK #4977

EvanBoyle · 2020-07-07T21:30:03Z

This PR is for an alpha version of the Automation API Go SDK. This API is designed to address the feedback and use cases raised in #3901, primarily orchestration, programmatic usage, and building higher-level tools and services on top of Pulumi.

The Automation API provides a way to drive existing Pulumi programs programmatically. These are separate programs that are typically driven by the CLI, and contain a pulumi.yaml and various pulumi.stack.yaml in their directories. In addition to driving these traditional programs, Automation API enables Inline programs. These are programs where the source code lives alongside the Automation API code driving the update:

// just another normal go program, can be run via `go run main.go`
func main(){
 stack, err := NewStackInlineSource(ctx, "myOrg/myProj/myStack", func(pCtx *pulumi.Context) error {
        // this is an inline Pulumi program
	bucket, err := s3.NewBucket(pCtx, "bucket", nil)
	if err != nil {
		return err
	}
	pCtx.Export("bucketName", bucket.Bucket)
	return nil
 })
}

This is accomplished by adding a new CLI command pulumi host. This command starts up an instance of the engine and resource monitor, leaving an update running while waiting for an interrupt signal and printing the addresses to stderr. Normally, pulumi invokes the language runtime with these addresses as environment variables. For inline functions, the automation API invokes pulumi host, managing the lifecycle of the engine, and populating appropriate environment variables before executing the user-defined inline program. In this manner, everything can run from a single main.go.

At a high level this SDK is a driver that shells out the Pulumi CLI. Over time we'll continue to put in more effort to defining well-structured error messages, and structured output to CLI commands. This model based on improving structured output in and out of the CLI provides a foundation to eventually deliver Automation API across all Pulumi supported languages.

As for the API itself, it defines a few important concepts:

A Workspace interface: the execution context containing a single Pulumi project, a program, and multiple stacks. Workspaces are used to manage the execution environment, providing various utilities such as plugin installation, environment configuration ($PULUMI_HOME), and creation, deletion, and listing of Stacks. This interface has some escape hatch methods that enable future workspace implementations that don't rely on pulumi.yaml or local filesystems (think config, and project settings stored in SQL or cloud object storage).
LocalWorkspace: A concrete implementation of the Workspace interface based on the way that Pulumi workspaces are currently exposed by the CLI. You can think of this as a directory, with a pulumi program, a pulumi.yaml, and zero or more pulumi.stack.yaml files.
Stack: a struct with methods for stack management and Pulumi lifecycle (up/refresh/destroy/preview). A Stack is created with a Workspace that it maintains a reference to. It delegates some operations like getting and setting config to the workspace.

The goal of this Workspace/Stack design is to model Pulumi programs as they are exposed today via the CLI, while still leaving the door open for Workspace improvements that may be necessary as users begin shifting workloads into more automated settings (think web services, Kubernetes operator pattern, etc).

The package has comprehensive godocs including examples for most types and methods, and this is probably the best way to get started reviewing this changeset:

Enlist this branch in your $GOPATH
cd $GOPATH/src/github.com/pulumi/pulumi/sdk/go/x/auto
godoc -http=:6060
Navigate to http://localhost:6060/pkg/github.com/pulumi/pulumi/sdk/v2/go/x/auto/

There are a number of known issues that need to be addressed before this comes out of alpha. These issues will not be addressed in this PR. The goal is to get this checked in to (1) allow other members of the team to contribute to this effort in parallel, and (2) allow the community to kick the tires and provide feedback. A list of issues that will be addressed as continued work:

Improved strongly-typed output for all Stack lifecycle methods Add structured output to Stack.Up #5218 Add structured output to Stack.Destory #5219 Add structured output to Stack.Refresh #5220
Add Stack lifecycle APIs that support streaming: Add streaming Stack lifecycle APIs #5224
Finish wiring up flags in pulumi host command Wire up various parameters to pulumi host command #5225
Add missing commands to Stack/Workspace Add missing methods to Stack/Workspace #5226

If you'd like to try this out locally and take it for a spin yourself, make sure to build the CLI so you have the new pulumi host command.

jaxxstorm · 2020-08-25T19:19:58Z

Makefile

-travis_push: only_build publish_tgz only_test publish_packages
-travis_pull_request: all
-travis_api: all
+travis_cron: install dist all


Why were these new targets added? I think all already runs these

The automation API tests need the other language SDKs, and PaC analyzers to be installed. So I added a full pass of build first.

pkg/cmd/pulumi/host.go

sdk/go/x/auto/cmd.go

pgavlin · 2020-08-27T22:13:52Z

sdk/go/x/auto/errors.go

+	return strings.Contains(ae.stderr, "[409] Conflict: Another update is currently in progress.")
+}
+
+// IsCompilationError returns true if the program failed at the build/run step (only Typescript, Go, C#)


How do you imagine these errors being used? For example, what action is expected of a consumer of the API in the case of a compilation or runtime error?

I implemented IsCompilationError for mostly demonstrative purposes. The only error here in this package that I think is truly useful is IsConflictingUpdateError.

I need more real usage like @metral's to identify what cases to handle. He had good thoughts about having an AutoError that indicates if stack creation failed because one already exists: https://github.com/pulumi/pulumi-kubernetes-operator/pull/86/files#diff-e31cd52d2ed37a9f588a46e7ca7e52b6R551-R556

With that said, IsCompilationError, and likely UnexpectedEngineError and IsRuntimeError can probably be removed.

pgavlin · 2020-08-27T22:17:26Z

sdk/go/common/resource/plugin/langruntime_plugin.go

+// HostOnlyRuntimeMode can be used to launch the engine in "host"-only mode. That is to say, the engine and
+// resource monitors will be spawned, but no language runtime will be created. This allows automated
+// scenarios that drive the engine from outside of the engine itself, such as, for example, a Node.js
+// program spawning, deploying, and tearing down the engine's resources.
+const HostOnlyRuntimeMode = "host"


I'd prefer to see a more complete design for host mode. Offhand, the way I would have expected this feature to work is for a language host to launch that waits for a connection from the actual host once Run is called. Though this is a bit more complex in terms of its mechanics, it simplifies the lifecycle of the update, as the lifetime remains bounded by the call to Run (this approach may also remove the need for the host command, as any of the usual update commands could be launched in host mode).

That's a great idea. I've been worried that the host lifecycle being decoupled from the execution of the program could lead to leaked hosts and updates that don't get closed when the program encounters an exception.

I've set up time for us to discuss design in more detail next week, and opened the following issue to track: #5249

It would be my preference to not design it in this PR though. To that end, I've hidden the host command via aba397a. With the host command hidden, and the fact that the underlying implementation of host mode isn't exposed to users via automation API (just the presence of workspace.program is all), I think we should be able to swap this out without any user-facing impact.

Can we merge this with host mode as is? This allows @metral and others to start contributing while I work on this in parallel.

sdk/go/x/auto/local_workspace.go

pgavlin · 2020-08-27T23:27:36Z

sdk/go/x/auto/local_workspace.go

+// LocalWorkspaceOption is used to customize and configure a LocalWorkspace at initialization time.
+// See Workdir, Program, PulumiHome, Project, Stacks, and Repo for concrete options.
+type LocalWorkspaceOption interface {
+	applyLocalWorkspaceOption(*localWorkspaceOptions)


Might as well export this method and the localWorkspaceOptions type s.t. third parties can author their own options

Do you mean author their own version of NewStackLocalSource or equivalent that takes custom options?

pgavlin · 2020-08-27T23:28:26Z

sdk/go/x/auto/local_workspace.go

+	return l, nil
+}
+
+type localWorkspaceOptions struct {


I think that we should make this type an option in and of itself. This would allow easy manipulation of options after the fact.

Not sure I grok. Would you mind explaining more or perhaps provide an example?

komalali

Tried to do a fairly thorough review of the docs, so the changes are largely for clarity or cosmetic. The godocs are awesome, especially all the examples.

Some overall nits:

Inconsistent docstring endings (some end with ., others do not)
Inconsistent casing of Pulumi ([Pp]ulumi)

sdk/go/x/auto/stack.go

sdk/go/x/auto/test/errors/runtime_error/javascript/index.js

sdk/go/x/auto/test/errors/runtime_error/python/__main__.py

sdk/go/x/auto/test/errors/runtime_error/typescript/index.ts

sdk/go/x/auto/test/errors/compilation_error/typescript/index.ts

sdk/go/x/auto/local_workspace.go

sdk/go/x/auto/optdestroy/optdestroy.go

sdk/go/x/auto/workspace.go

metral

Great work overall! 🎉

Sprinkled usability and dev experience related comments through out, but overall this worked as expected!

pkg/cmd/pulumi/host.go

pkg/cmd/pulumi/util.go

sdk/go/Makefile

sdk/go/common/resource/plugin/langruntime_plugin.go

sdk/go/x/auto/test/errors/compilation_error/typescript/index.ts

sdk/go/x/auto/test/errors/runtime_error/javascript/index.js

sdk/go/x/auto/test/errors/runtime_error/python/__main__.py

sdk/go/x/auto/test/errors/runtime_error/typescript/index.ts

sdk/go/x/auto/stack.go

mikhailshilkov

This makes a lot of sense, looks great! I haven't used it yet, but I'm planning to pick that up for several things once it hits master.

sdk/go/x/README.md

sdk/go/x/auto/cmd.go

sdk/go/x/auto/errors.go

sdk/go/x/auto/local_workspace.go

sdk/go/x/auto/stack.go

sdk/go/x/auto/test/errors/compilation_error/dotnet/Pulumi.yaml

sdk/go/x/auto/test/errors/conflict_error/main.go

sdk/go/x/auto/test/errors/runtime_error/dotnet/MyStack.cs

sdk/go/x/auto/test/testproj/Pulumi.yaml

pkg/cmd/pulumi/host.go

sdk/go/x/auto/errors.go

sdk/go/x/auto/local_workspace.go

EvanBoyle added 30 commits July 7, 2020 14:25

prototype

2504db3

init/select stack, improvements

e08e362

expose GetUser GetOutputs

ae465a1

add utils to round trip pulumi.yaml and pulumi.stack.yaml

ea6cd84

set config and secrets upon stack initialization

13746f9

capture last event from history for UpResult

c09a1cb

preview

30f0962

destroy

102dbad

pulumi stack rm

7ac7041

refresh, e2e test

96b9c74

tighten up the interface

5b9f1b6

lint

5faf54c

typo

99d769b

init error strategy

cf4e02d

clean up test

f6bc545

IsCompilationError

ff81dd8

Merge branch 'master' of https://github.com/pulumi/pulumi into evan/auto

c360e4d

IsRuntimeError()

0500aae

add some comments based on findings

be42f75

IsUnexpectedEngineError()

acc719e

update test

e789ae9

update error formatting

6e7ce72

add ability to specify git URL instead of local source

0d3203c

init inline prototype

b0468d4

inline preview

b21f4e0

use ts seeds for random test names

ae76944

enable e2e up/pre/ref/dest/rm test for inline source

15b8f03

host error handling, inline concurrent update detection

41d6767

handle and detect runtime errors for inline source

45ed471

provide defaults for pului.yaml for inline source

114e3c2

EvanBoyle added 2 commits August 24, 2020 17:51

Merge branch 'master' of https://github.com/pulumi/pulumi into evan/auto

463c7b6

automation api godoc

ab03c53

EvanBoyle changed the title ~~Prototype~~ Automation API - Go SDK Aug 25, 2020

changelog

7fcf833

EvanBoyle requested review from pgavlin, justinvp, komalali and mikhailshilkov August 25, 2020 19:14

EvanBoyle marked this pull request as ready for review August 25, 2020 19:14

EvanBoyle requested review from lukehoban and metral August 25, 2020 19:15

jaxxstorm reviewed Aug 25, 2020

View reviewed changes

justinvp reviewed Aug 25, 2020

View reviewed changes

pkg/cmd/pulumi/host.go Outdated Show resolved Hide resolved

sdk/go/x/auto/cmd.go Show resolved Hide resolved

metral mentioned this pull request Aug 25, 2020

Refactor controller to use Automation API pulumi/pulumi-kubernetes-operator#85

Closed

4 tasks

update file headers

cf247c1

pgavlin reviewed Aug 27, 2020

View reviewed changes

komalali reviewed Aug 27, 2020

View reviewed changes

metral suggested changes Aug 28, 2020

View reviewed changes

mikhailshilkov approved these changes Aug 28, 2020

View reviewed changes

EvanBoyle mentioned this pull request Aug 28, 2020

Improve design of host mode #5249

Closed

EvanBoyle and others added 2 commits August 28, 2020 08:43

hide host command

aba397a

Merge branch 'master' into evan/auto

ed2e5c6

justinvp reviewed Aug 28, 2020

View reviewed changes

docs updates

b0d9147

EvanBoyle mentioned this pull request Aug 29, 2020

Audit the non-saas automation api experience #5254

Closed

EvanBoyle and others added 3 commits August 28, 2020 19:10

docs and style updates

c23f26b

Merge branch 'master' into evan/auto

09964d2

add expectnochanges to stack.refresh

5458e2f

EvanBoyle merged commit 766bf10 into master Aug 29, 2020

pulumi-bot deleted the evan/auto branch August 29, 2020 03:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automation API - Go SDK #4977

Automation API - Go SDK #4977

EvanBoyle commented Jul 7, 2020 •

edited

Loading

jaxxstorm Aug 25, 2020

EvanBoyle Aug 25, 2020

pgavlin Aug 27, 2020

EvanBoyle Aug 27, 2020

EvanBoyle Aug 27, 2020

pgavlin Aug 27, 2020

EvanBoyle Aug 28, 2020

pgavlin Aug 27, 2020

EvanBoyle Aug 28, 2020

pgavlin Aug 27, 2020

EvanBoyle Aug 28, 2020

komalali left a comment

metral left a comment

mikhailshilkov left a comment

Automation API - Go SDK #4977

Automation API - Go SDK #4977

Conversation

EvanBoyle commented Jul 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

komalali left a comment

Choose a reason for hiding this comment

metral left a comment

Choose a reason for hiding this comment

mikhailshilkov left a comment

Choose a reason for hiding this comment

EvanBoyle commented Jul 7, 2020 •

edited

Loading