Transformations

[lukehoban]

Adds the ability to provide transformations to modify the properties and resource options that will be used for any child resource of a component or stack.

This offers an "escape hatch" to modify the behaviour of a component by peeking behind it's abstraction. For example, it can be used to add a resource option (additionalSecretOutputs, aliases, protect, etc.) to a specific known child of a component, or to modify some input property to a child resource if the component does not (yet) expose the ability to control that input directly. It could also be used for more interesting scenarios - such as:

1. Automatically applying tags to all resources that support them in a stack (or component)
2. Injecting real dependencies between stringly-referenced resources in a Helm Chart
3. Injecting explicit names using a preferred naming convention across all resources in a stack
4. Injecting import onto all resources by doing a lookup into a name=>id mapping

Because this feature makes it possible to peek behind a component abstraction, it must be used with care in cases where the component is versioned independently of the use of transformations. Also, this can result in "spooky action at a distance", so should be used judiciously. That said - this can be used as an escape hatch to unblock a wide variety of common use cases without waiting on changes to be made in a component implementation, as well

Each transformation is passed the name, type, props and opts that are passed into the Resource constructor for any resource descended from the resource that has the transformation applied. The transformation callback can optionally return alternate versions of the props and opts to be used in place of the originally provided values.

Transformations are applied iteratively in order of specificity - so stack transformations are applied first, then parent transformations (in order), then child transformations (in order). The props and opts are passed through each layer of these transformations.

Questions for reviewers:

1. I'm actually not sure order of specificity is correct here - somewhat counterintuitively - this feature is actually about allow less specific code to override more specific code. Should the order of application of transformations actually be inverted so that the last added Stack transformation gets final say?
2. Is there really value in using transformations instead of just transformation? It seems exceedingly rare to provide multiple at one callsite - and in the case this is necessary, it seems the user could compose them functionally themselves.
3. transformations vs. transforms (vs. something else)?
4. The stack level version of this is dependent on the global context, which is not shared in SxS scenarios. That means it could be possible for the transformation to not apply to all children. I'm not sure what to do about that. The stack level version of this is likely to be the most common use case for casual usage ("apply tags to everything", "adding import to everything").

Fixes #2068.

[lukehoban]

@pgavlin @CyrusNajmabadi @hausdorff This is ready for review.

[CyrusNajmabadi]

I really like the simplicity and power here. Overall no real complaints. In terms of your questions:

1. I'm actually not sure order of specificity is correct here - somewhat counterintuitively - this feature is actually about allow less specific code to override more specific code. Should the order of application of transformations actually be inverted so that the last added Stack transformation gets final say?

What you have here intuitively made sense to me. Especially with how the parent transformations come into play, but the child tranforms also get 'last say' to do what they want.

2. Is there really value in using transformations instead of just transformation? It seems exceedingly rare to provide multiple at one callsite - and in the case this is necessary, it seems the user could compose them functionally themselves.

I feel like it would not be uncommon in how a top-level user might make a component, passing in a transformation, and the component could itself pass in its own transformations to other components. I'm ok with it being an array.

3. transformations vs. transforms (vs. something else)?

I don't care.

4. The stack level version of this is dependent on the global context, which is not shared in SxS scenarios. That means it could be possible for the transformation to not apply to all children. I'm not sure what to do about that. The stack level version of this is likely to be the most common use case for casual usage ("apply tags to everything", "adding import to everything").

I'm ok with that. we can document this accordingly. even in the apply-tags to everything case, you'll have several clear top-level points where you can provide the transform to the resources you care most about.

[CyrusNajmabadi]

Preemptively signing off. I'm not seeing anything here that worries me.

[lukehoban]

actually, it feels like we shoudl always be able to get this. should we throw in this method if this is called in a scenario where stackResource was not set? That would indicate this runInpulumiStack was not called, which would be weird if we were then running the same code in resource-construction couldn't access this.

Done. I'm slightly nervous about this - but okay with being strict here and seeing if we encounter places this cause problems.

Turns out this did introduce new issues - mostly because the Stack itself is a resource, and so there is code in the super construction of the Stack itself that ends up reading from this. I've reverted back to closer to what it was before - but made it a little more explicit that this gets initialized very early (and always before the init with user code is called).

[pgavlin]

So if I want to modify a property, presumably I need to do it like

pulumi.output(props).apply(...)

?

[lukehoban]

Yes - or pulumi.output(props["myprop"]).apply(myprop => ...) in the case that you just want to overwrite a single property in a way that is conditional on the value passed in.

This is slightly more cumbersome to author - but is more expressive - as it allows things like injecting new dependencies. For a low level power-user feature like this - my feeling was that providing flexibility/expressivity is more valuable than providing ease-of-authoring.

I've added a new test that is motivated by the scenario in #2068 (comment) which shows how this can be taken advantage of for a real (and fairly complex) scenario involving injecting a dependency on one child into another child.

Also - FWIW - in most common cases I've come up with - the transformation is overwriting the value or injecting a new default - so doesn't actually need to look at the existing input value.

[pgavlin]

The only high-level feedback I have is that it might be nice to have fully-resolved inputs before running transformations, but that I think that might drastically complicate the code. LGTM otherwise.

[joeduffy]

Do we envision supporting name transforms eventually? (May help with auto-naming stuff.)

[joeduffy]

Ahh, nevermind, I realize now that you can simply set the physical resource name property. No need to change the logical name.

[lukehoban]

Opened #3283 to track adding this to Python as well. We should do that ASAP as a follow-up.