Support transformations for multi-language components

[lukehoban]

Pulumi Packages support authoring components in one language and making them available to other languages. Components authored this way are presented as normal Pulumi resources in the consumer language. Most standard resource options are supported for these components. But the transformations resource option is not yet. That is because this option accepts functions as input, and those functions need to run when any child resource of the target component is created (to potentially modify the inputs to the child resource). This means that a component authored in Go, and consumed in Python, would have a transformation function written in Python that the Go implementation of the child resources needs to invoke. We will need to add support for this to the multi-language component model. At a high level, this will likely include:

1. A mechanism in each language SDK to register functions as opaque handles
2. The ability to pass these handles around (for example as the serialized arguments to transformations)
3. An engine capability to invoke one of these handles passing an args bag and receiving back and return object.
4. Support for using the engine capability in (3) as part of the implementation of transformations support in each SDK

[ringods]

I bumped into this issue by testing exactly this combination today. If I would like to equal my TS only code setup I have at the moment, the setup becomes even more complex.

Currently, I have abstractions for a k8s deployment+service and abstractions for databases, message queues and alike. In my code, I create explicit objects for connections between my abstractions. A base setup of a service doesn't contain any resources or configuration to connect it to anything else. If I connect an instance of my k8s deployment abstraction to an instance of my database abstraction in my stack code, under the hood, I leverage transformations to extend the k8s Deployment resource with database access information coming from the db abstraction. Based on the type of database, the transformation function, coming from the DB abstraction, knows exactly which info to add. This gives me a good separation of concerns.

When applying this multi-language components, the transformation functions would also be in the component provider codebase, e.g in Go. This can be in the same or in a different component provider. We could in the end have 3 codebases into play:

• Python stack consuming the multi-language components
• multi-lang component provider for my service deployments
• multi-lang component provider for my DB abstractions, also exposing & implementing the DB specific resource transformation functions

When consuming two of these abstractions in Python, Python is only used to wire the two abstractions. The calling stack should retrieve the opts & args of the base service deployment resources, pass these to the DB provider for transformation and then sent back to the deployment provider before the real resources are actually created.

[justinvp]

We're actively working on this capability. I wanted to share an update on a design challenge we ran into and our plan to address.

The plan has been to use the existing transformations APIs for the new engine-based transformation capability that works with MLCs, making it the new default behavior. In doing so, there were some known minor behavior differences, but we believed that the majority of existing transformations would just work when run with the new behavior, and we'd have a way to opt back into the old behavior for the edge cases that were problematic. (The differences being that we wouldn't be able to pass the same Resource instance to transformations as before and all properties will be serialized/deserialized as they're passed-through the engine, so they'd be fully "observed" when they might not have been previously.)

However, we've run into a design challenge in Go and other strongly-typed SDKs like .NET and Java. In those languages, transformations use typed arg values (e.g. s3.BucketArgs), but when we get sent the remote transform request we only have a property bag. We don't know what type to cast it to. Node.js and Python don't have this problem because they pass properties to transformations as untyped objects or dicts.

For example, typical usage in Go looks like the following:

err = ctx.RegisterStackTransformation(func(rta *pulumi.ResourceTransformationArgs) *pulumi.ResourceTransformationResult {
    if rta.Type == "aws:s3/bucket:Bucket" {
        var props *s3.BucketArgs
        if rta.Props == nil {
            props = &s3.BucketArgs{}
        } else {
            props = rta.Props.(*s3.BucketArgs)
        }
        // ...
    }
    return nil
})

The problem is we don't currently have a way to make it such that rta.Props is an instance of *s3.BucketArgs. We discussed options for providing such a mapping but it is expensive ot implement and has versioning challenges.

We've discussed a few different options on how to proceed:

1. Add legacyTranformations: true (break by default)
2. Add legacyTranformations array (break by default)
3. Add transforms (or some better name) (don't break by default)
   • transforms
   • asyncTransformations
   • remoteTransformations
   • patches
   • remediations

(1) is basically the plan we had been working towards until we discovered this problem, with an option to opt-in to the old behavior.

(2) is a variant where the existing API would break by default, and there'd be a new API that could be used to specify transformations that use the old behavior.

(3) is adding a new API to opt-in to the new behavior. Existing transformations would continue to work as-is. The new API would need to be used to be able to provide transformations for MLCs. We'd generally recommend moving to the new API, and eventually deprecate the old API. We therefore want to choose a name for the new API that will make sense long-term. The best name we've come up with so far is transforms, a shorter variant of transformations, but certainly open to other ideas.

We're currently looking at going with (3). The type of the Props argument to the transforms in the new API in Go will be pulumi.Map (which is map[string]Input).