[Automation API] - C# Implementation #5761

orionstudt · 2020-11-13T18:40:03Z

Here is what I've got so far.

Got kind of busy so wasn't able to get as much done as I liked this week - but will be trying to start the RunPulumiCmd portion as well as project settings deserialization early next week.

I will be placing some thoughts / questions in the applicable files...

…lementation

sdk/dotnet/Pulumi/X/Automation/IWorkspace.cs

sdk/dotnet/Pulumi/X/Automation/StackSettings.cs

sdk/dotnet/Pulumi/X/Automation/IWorkspace.cs

sdk/dotnet/Pulumi/X/Automation/LocalWorkspace.cs

sdk/dotnet/Pulumi/X/Automation/PluginInfo.cs

sdk/dotnet/Pulumi/X/Automation/IWorkspace.cs

EvanBoyle

This is shaping up nicely!

sdk/dotnet/Pulumi/X/Automation/IWorkspace.cs

sdk/dotnet/Pulumi/X/Automation/LocalWorkspace.cs

sdk/dotnet/Pulumi/X/Automation/StackInfo.cs

sdk/dotnet/Pulumi/X/Automation/StackSettings.cs

StefanXhunga

Thank you for your assistance 18/19 viewed

…ion, cleanup run pulumi cmd implementation and make it an instance so it is mockable/testable, separate serialization in prep for custom converters

orionstudt · 2020-11-20T22:14:33Z

@EvanBoyle @justinvp

Alright so latest commit has:

RunPulumiCmd implementation. For .NET I made this into an internal interface / instance so that it could be dropped in and out for testing (which IDK how we're doing yet)
Fleshed out all of the LocalWorkspace methods
Started on Stack implementation which is currently called XStack pending naming decision
Started to separate out LocalWorkspace's serialization into an internal LocalSerializer service because I am expecting that once I start testing I will need to write some custom converters and this means it can be tested independently + serialization settings can be configured in 1 place
I believe I have the "idiomatic structured error handling" completed in here as well

I want to note that some internal API changes were made to the abstract Workspace. This is because I needed to separate the XStack implementation from the local implementation of Workspace (it looks like you guys want XStack to be workspace-agnostic). If you look in the TypeScript implementation you'll see that stack.ts remains workspace-agnostic with the exception of the fact that it has its own private runPulumiCmd which reaches into the local implementation.. I don't know if that was intentional but am curious to hear any thoughts on this.

Big next step for me right now is getting any custom serialization converters written and then testing each of the methods on LocalWorkspace, at which point I will move to the up, preview, & destroy methods on XStack.

Current questions:

Original issue mentions the Import, Export, & Cancel methods on Workspace in GO. I found code for import & export but couldn't find cancel?
Currently the TypeScript implementation of static LocalWorkspace.Create : LocalWorkspace doesn't set default project settings if they are not provided, but the static LocalWorkspace.(Create or Select)Stack : Stack methods do set default project settings if not provided. Should we make sure the static method that just creates a workspace does so as well?
The TypeScript implementation of GetProjectSettings and GetStackSettings both throw an exception if it is unable to find a settings file, while the XML comment describing the methods seem to indicate that it could or should return null if settings are not found. Should we be throwing if there is no settings file or just returning null?

orionstudt · 2020-11-24T13:48:55Z

I will return to this after the holiday.

EvanBoyle · 2020-12-04T18:49:15Z

I made my way through the recent changes, functionally it looks great. I'll leave it to @justinvp to comment on dotnet-isms.

To answer your questions:

I found code for import & export but couldn't find cancel?

cancel is an operation on stack, similar to up/preview/destroy/refresh: https://github.com/pulumi/pulumi/blob/master/sdk/go/x/auto/stack.go#L579-L595

Should we make sure the static method that just creates a workspace does so as well?

A stack has to have a project settings file, but a workspace does not (it could be empty).

static LocalWorkspace.(Create or Select)Stack : Stack methods do set default project settings if not provided.

We only set a default project settings if none is provided in the inline program case:
https://github.com/pulumi/pulumi/blob/master/sdk/nodejs/x/automation/localWorkspace.ts#L171-L187

For stacks using local programs, we assume an existing pulumi.yaml already exists in the directory.

Should we be throwing if there is no settings file or just returning null?

Yes, this sounds reasonable. The existing design of throwing in go/typescript isn't ideal and we should probably change it.

…value yaml serialization

…alization so that it is in the commit history

language server implementation to be wired into stack implementation

…since we are defering a complete serialization implementation until a later date

github-actions · 2021-02-03T21:39:29Z

PR is now waiting for a maintainer to run the acceptance tests.

Note for the maintainer: To run the acceptance tests, please comment /run-acceptance-tests on the PR

github-actions · 2021-02-12T14:19:43Z

PR is now waiting for a maintainer to run the acceptance tests.

Note for the maintainer: To run the acceptance tests, please comment /run-acceptance-tests on the PR

github-actions · 2021-02-12T14:23:37Z

PR is now waiting for a maintainer to run the acceptance tests.

Note for the maintainer: To run the acceptance tests, please comment /run-acceptance-tests on the PR

github-actions · 2021-02-12T14:51:55Z

PR is now waiting for a maintainer to run the acceptance tests.

Note for the maintainer: To run the acceptance tests, please comment /run-acceptance-tests on the PR

orionstudt · 2021-02-12T14:55:59Z

alright a final recap: @EvanBoyle @mikhailshilkov

I believe the only thing we're waiting for here (that I'm aware of) is deciding on final naming of the Pulumi.Automation.XStack.

I realize now that the automation API examples are in a different repo, so I can make a PR to complete some of those once this has merged.

And finally, I noticed #6257 and went to add the --limit support to this implementation but then realized that obviously locally I can only test against released CLI versions, and that has not been released yet. Idk if your build server tests against unreleased CLI? If so we can include that here too, it's very minimal.

…lumi#6257

github-actions · 2021-02-12T16:29:00Z

PR is now waiting for a maintainer to run the acceptance tests.

Note for the maintainer: To run the acceptance tests, please comment /run-acceptance-tests on the PR

sdk/dotnet/Pulumi.Automation/WorkspaceStack.cs

…n api per pulumi#6292

github-actions · 2021-02-12T17:19:17Z

PR is now waiting for a maintainer to run the acceptance tests.

Note for the maintainer: To run the acceptance tests, please comment /run-acceptance-tests on the PR

github-actions · 2021-02-16T20:41:04Z

PR is now waiting for a maintainer to run the acceptance tests.

Note for the maintainer: To run the acceptance tests, please comment /run-acceptance-tests on the PR

mikhailshilkov

LGTM! Thank you!

github-actions · 2021-02-18T10:36:19Z

PR is now waiting for a maintainer to run the acceptance tests.

Note for the maintainer: To run the acceptance tests, please comment /run-acceptance-tests on the PR

This reverts commit 963b5ab.

* Init Workspace interface for C# Automation API * fleshing out workspace interface and beginning of local workspace implementation * initial run pulumi cmd implementation * resolve issue with pulumi cmd cleanup wrapper task after testing * flesh out local workspace implementation, flesh out stack implementation, cleanup run pulumi cmd implementation and make it an instance so it is mockable/testable, separate serialization in prep for custom converters * project settings json serialization implemented * Initial commit of language server * Add deployment from language server * Cleanup * finish json serialization * project runtime yaml serialization completed. just need stack config value yaml serialization * Remove typed argument * Limit concurrency * Rename file for consistency * final commit of a semi-working project settings & stack settings serialization so that it is in the commit history * modify workspace API so that settings accessors aren't fully exposed since we are defering a complete serialization implementation until a later date * yaml converters wrap any outgoing exceptions so resolve that * getting the beginning of inline program GRPC communication set up * stack lifecycle operations implemented, and switched to newtonsoft for JSON serialization * change back to system.text.json with a custom object converter * local workspace tests written, working on getting them passing * fix the encoding on the GO files used for testing * all tests passing except inline program, pulumi engine not available with inline * inline program engine is now running as expecting, but inline program is not recognizing local stack config * All tests passing, but no concurrency capability because of the singleton DeploymentInstance. * cleanup unnecessary usings * minor cleanup / changes after a quick review. Make sure ConfigureAwait is used where needed. Remove newtonsoft dependency from testing. Update workspace API to use existing PluginKind enum. Modify LanguageRuntimeService so that its semaphore operates process-wide. * support for parallel execution of inline program, test included * Update LocalWorkspaceTests.cs remove some redundancy from the inline program parallel execution text * flesh out some comments and make asynclocal instance readonly * Strip out instance locking since it is no longer necessary with AsyncLocal wrapping the Deployment.Instance. Modify CreateRunner method such that we are ensuring there isn't a chance of delayed synchronous execution polluting the value of Deployment.Instance across calls to Deployment.RunAsync * resolve conflicts with changes made to Deployment.TestAsync entrypoints * update changelog * write a test that fails if the CreateRunnerAndRunAsync method on Deployment is not marked async and fix test project data file ref * make resource package state share the lifetime of the deployment so that their isn't cross deployment issues with resource packages, add support and tests for external resource packages (resource packages that aren't referenced by the executing assembly) * enable parallel test collection execution in test suite, add some additional tests for deployment instance protection and ensuring that our first class stack exceptions are thrown when expected * minor inline project name arg change, and re-add xunit json to build output (whoops) * strip out concurrency changes since they are now in pulumi/pulumi#6139, split automation into separate assembly, split automation tests into separate assembly * add copyright to the top of each new file * resolve some PR remarks * inline program exception is now properly propagated to the caller on UpAsync and PreviewAsync * modify PulumiFn to allow TStack and other delegate overloads without needing multiple first class delegates. * whoops missing a copyright * resolve getting TStack into IRunner so that outputs are registered correctly and so that there isn't 2 instances of Pulumi.Stack instantiated. * resolve issue with propagation of TStack exceptions and add a test * add support for a TStack PulumiFn resolved via IServiceProvider * update automation API description * fix comment and remove unnecessary TODOs * disable packaging of automation api assembly * re-name automation api documentation file appropriately * add --limit support to dotnet automation api for stack history per pulumi/pulumi#6257 * re-name XStack as WorkspaceStack * replace --limit usage with --page-size and --page in dotnet automation api per pulumi/pulumi#6292 Co-authored-by: evanboyle <evan@pulumi.com> Co-authored-by: Josh Studt <josh.studt@figmarketing.com> Co-authored-by: Dan Friedman <dan@thefriedmans.org> Co-authored-by: David Ferretti <David.Ferretti@figmarketing.com> Co-authored-by: Mikhail Shilkov <github@mikhail.io>

EvanBoyle and others added 2 commits October 13, 2020 14:51

Init Workspace interface for C# Automation API

66b7794

fleshing out workspace interface and beginning of local workspace imp…

550e409

…lementation

orionstudt commented Nov 13, 2020

View reviewed changes

sdk/dotnet/Pulumi/X/Automation/IWorkspace.cs Outdated Show resolved Hide resolved

orionstudt commented Nov 13, 2020

View reviewed changes

sdk/dotnet/Pulumi/X/Automation/StackSettings.cs Outdated Show resolved Hide resolved

orionstudt commented Nov 13, 2020

View reviewed changes

sdk/dotnet/Pulumi/X/Automation/IWorkspace.cs Outdated Show resolved Hide resolved

orionstudt commented Nov 13, 2020

View reviewed changes

sdk/dotnet/Pulumi/X/Automation/LocalWorkspace.cs Outdated Show resolved Hide resolved

orionstudt commented Nov 13, 2020

View reviewed changes

sdk/dotnet/Pulumi/X/Automation/PluginInfo.cs Outdated Show resolved Hide resolved

justinvp reviewed Nov 14, 2020

View reviewed changes

sdk/dotnet/Pulumi/X/Automation/IWorkspace.cs Outdated Show resolved Hide resolved

EvanBoyle reviewed Nov 16, 2020

View reviewed changes

StefanXhunga reviewed Nov 16, 2020

View reviewed changes

orionstudt added 3 commits November 19, 2020 11:19

initial run pulumi cmd implementation

e7598a1

resolve issue with pulumi cmd cleanup wrapper task after testing

c8bec4f

flesh out local workspace implementation, flesh out stack implementat…

b450d8b

…ion, cleanup run pulumi cmd implementation and make it an instance so it is mockable/testable, separate serialization in prep for custom converters

orionstudt and others added 14 commits December 9, 2020 09:43

project settings json serialization implemented

bf4a1f7

Initial commit of language server

32a13a0

Add deployment from language server

7d037ae

Cleanup

902ba1d

finish json serialization

73b262d

project runtime yaml serialization completed. just need stack config …

a8953b5

…value yaml serialization

Remove typed argument

7096aeb

Limit concurrency

af68874

Rename file for consistency

c41dbcb

final commit of a semi-working project settings & stack settings seri…

6f9249d

…alization so that it is in the commit history

Merge pull request #1 from MisinformedDNA/mdna/auto/dotnet

81ca88f

language server implementation to be wired into stack implementation

modify workspace API so that settings accessors aren't fully exposed …

de913b8

…since we are defering a complete serialization implementation until a later date

yaml converters wrap any outgoing exceptions so resolve that

6252234

getting the beginning of inline program GRPC communication set up

fdba7c3

disable packaging of automation api assembly

2cbd83f

re-name automation api documentation file appropriately

7ae2ff2

merge with master

1ac5316

orionstudt added 2 commits February 12, 2021 11:25

add --limit support to dotnet automation api for stack history per pu…

6c6847d

…lumi#6257

re-name XStack as WorkspaceStack

69fd02d

EvanBoyle reviewed Feb 12, 2021

View reviewed changes

sdk/dotnet/Pulumi.Automation/WorkspaceStack.cs Outdated Show resolved Hide resolved

replace --limit usage with --page-size and --page in dotnet automatio…

b6cbd40

…n api per pulumi#6292

Merge branch 'master' into auto/dotnet

66836ed

mikhailshilkov approved these changes Feb 18, 2021

View reviewed changes

Merge branch 'master' into auto/dotnet

1741125

mikhailshilkov merged commit 963b5ab into pulumi:master Feb 18, 2021

jankeromnes added a commit to jankeromnes/pulumi that referenced this pull request Feb 18, 2021

Revert "[Automation API] - C# Implementation (pulumi#5761)"

fd41816

This reverts commit 963b5ab.

orionstudt deleted the auto/dotnet branch February 18, 2021 17:35

infin8x modified the milestones: 0.55, 0.53 Feb 25, 2021

orionstudt mentioned this pull request Mar 3, 2021

[Epic] Automation API: C# #5596

Closed

7 tasks

leezen mentioned this pull request Mar 3, 2021

Automation API C# Updates for GA #6459

Closed

lblackstone mentioned this pull request Mar 17, 2021

C# Automation API test failure #6558

Closed

orionstudt mentioned this pull request Apr 8, 2021

Method renames for dotnet automation API GA #6731

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Automation API] - C# Implementation #5761

[Automation API] - C# Implementation #5761

orionstudt commented Nov 13, 2020 •

edited

EvanBoyle left a comment

StefanXhunga left a comment

orionstudt commented Nov 20, 2020 •

edited

orionstudt commented Nov 24, 2020

EvanBoyle commented Dec 4, 2020

github-actions bot commented Feb 3, 2021

github-actions bot commented Feb 12, 2021

github-actions bot commented Feb 12, 2021

github-actions bot commented Feb 12, 2021

orionstudt commented Feb 12, 2021

github-actions bot commented Feb 12, 2021

github-actions bot commented Feb 12, 2021

github-actions bot commented Feb 16, 2021

mikhailshilkov left a comment

github-actions bot commented Feb 18, 2021

[Automation API] - C# Implementation #5761

[Automation API] - C# Implementation #5761

Conversation

orionstudt commented Nov 13, 2020 • edited

EvanBoyle left a comment

Choose a reason for hiding this comment

StefanXhunga left a comment

Choose a reason for hiding this comment

orionstudt commented Nov 20, 2020 • edited

orionstudt commented Nov 24, 2020

EvanBoyle commented Dec 4, 2020

github-actions bot commented Feb 3, 2021

github-actions bot commented Feb 12, 2021

github-actions bot commented Feb 12, 2021

github-actions bot commented Feb 12, 2021

orionstudt commented Feb 12, 2021

github-actions bot commented Feb 12, 2021

github-actions bot commented Feb 12, 2021

github-actions bot commented Feb 16, 2021

mikhailshilkov left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 18, 2021

orionstudt commented Nov 13, 2020 •

edited

orionstudt commented Nov 20, 2020 •

edited