Add split Go SDK

[danielrbradley]

1. Generate secondary go SDK into sdk/pulumi-azure-native-sdk to match new pulumi-azure-native-sdk repository
2. Insert root and leaf go.mod files as part of codegen (based on directories with init.go files).
3. Include replace directives so the modules can be used locally.
4. Incude current version number as part of go.mod references
   • Fall back to latest if not available.
   • Strip build number from codegen version to make compatible with go module versioning.
5. Build each go module using find.
6. Add go_prepublish target to strip go.sum files and go.mod replace directives
7. Adapt GitHub workflows to generate and persist new go SDK between steps
8. During publish, checkout new repository in sub-folder then tag and push

Outstanding issues:

• Generated version utility assumes the SDK is in the same repository so won't get the exact correct version.

Original investigation notes

Investigation into solving #1930 as a proof of concept and to decide a route forward.

Either approach below would be best combined with committing the new go SDK into a new repo such as pulumi-azure-sdk as ~200 tags will need to be added on each release, and it adds significantly more code.

Add go.mod files to the original structure

Pros: The core generation is almost identical, just with extra files appended

Have to reference the root sdk module of the same version. This would require publishing the root SDK first with the new version number, then pulling in the new version number into the per-module go.mod then publishing each after.

We really need the provider to be included in each sub-module directly to remove any internal dependencies between modules that have to be published together.

Split the schema and then generate independently

This results in every module having its own provider class (and config). This doesn't affect when using the default provider, but would look a little more messy with explicit providers.

Split by modifying discovery glob - makes discovery really quick for single modules.

Go package name is calculated from end of importBasePath in go language config.

authorization, storate and synapse seem to be mixins - not from specs. Could add options to skip adding these, have just filtered them out by token for now.

Generate the go.mod and add to list of files to emit.

Optional: Remove Double Nesting

Go module is at something like github.com/pulumi/pulumi-azure-native-sdk/compute. Codegen wants to create a second compute folder within that, but that's not great to have to import github.com/pulumi/pulumi-azure-native-sdk/compute/compute.

How do we avoid the double nesting of module name in Go when there's only one module? Could manually change the paths for now. Should we just live with double nesting for now?

There's separate init files in the root and module folders. The root file has construct provider and registers resource package. The module file has the resource "construct" method and registers resource module. Seems like we can just rename the module init file on moving, then fix the import used for the PkgVersion.

Opportunities to improve core codegen

• Add option to generate go.mod
• Allow option for a "default" module to be pulled to the top-level rather than every module having its own folder.

Out of scope

• Integration of the new Go SDK for ProgramTest - could continue to generate the monolithic SDK temporarily for testing.
• Publishing workflow - would need to add steps to commit to a separate repo and add relevant tags.

[Frassle]

Add option to generate go.mod

I think we need that in platform for some things anyway. We want to generate some types into the core sdk, but just the code files none of the project related stuff (i.e. no .csproj, go.mod, etc). I think that option would work for you here as well.

[danielrbradley]

Pair review with @thomas11

Document & changelog

[github-actions]

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.
Looking good! No API changes found.

[viveklak]

Should/can we add an acceptance test for this?

[viveklak]

Didn't realize we were still depending on pkg/errors...

[viveklak]

One general question would be how we update this template if versions change or we need additional dependencies added etc. Do you have thoughts on documenting a dev loop for this?

[danielrbradley]

No idea on what we actually use. This is taken from our currently manually committed go.mod in the sdk folder which we currently manually maintain. I can add something to the contributing docs around updating this. Long term, I think codegen should be generating these dependencies - so they can be upgraded based on what the generated code will need - similar in approach to this issue around Python dependencies

[danielrbradley]

Just checked and apparently we do actually use that package in the generated SDK.

[danielrbradley]

Should/can we add an acceptance test for this?

So, not entirely straightforward to figure out, but it seems like it should work!

[github-actions]

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.
Looking good! No API changes found.

[danielrbradley]

@viveklak doesn't look like program tests are likely to work at the time. Worth just doing a quiet release alongside the old SDK so we can continue testing against an actual real release?

[github-actions]

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.
Looking good! No API changes found.