Send output values to transforms for dependency tracking

[Frassle]

Description

This engine change is necessary for correct dependency tracking of properties through transform functions.

Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking.

This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value).
This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms.

After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure.

There are a couple of things to keep in mind with this change.

Firstly using a transform can cause a property that was being sent to a component provider as just a resource.NumberProperty to start being sent as an OutputProperty with the NumberProperty as its element. Component providers should handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function.

Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources:

A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])

Currently both would get upgraded to Output(Array([Number(1)]), ["dep"]). For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set.

Checklist

• I have run make tidy to update any new dependencies
• I have run make lint to verify my code passes the lint check
  • I have formatted my code using gofumpt

• I have added tests that prove my fix is effective or that my feature works

• I have run make changelog and committed the changelog/pending/<file> documenting my change

• Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version

[pulumi-bot]

Changelog

[uncommitted] (2024-03-19)

Features

• [engine] Send output values with property dependency information to transform functions
  #15637

[Frassle]

This needs a bit more set handling for the dependency arrays, going to try and get #15671 and maybe some other small changes in first to minimise diff size.

[tgummerer]

This looks good go me, with one question inline.

[tgummerer]

I'm not sure I'm following this change properly. Since we didn't upgrade the props in the case where there are no transforms, couldn't we just use the old code from https://github.com/pulumi/pulumi/pull/15637/files#diff-984b3279d61b588030474831898bb76d093d863ca2d0015bb081d842602a9ab7L1665? Or why is this change necessary?

[Frassle]

This is pretty much that code but for Set instead of slice.

[justinvp]

Firstly using a transform can cause a property that was being sent to a component provider as just a resource.NumberProperty to start being sent as an OutputProperty with the NumberProperty as its element.

Does this only happen if that property has dependencies?

[Frassle]

Does this only happen if that property has dependencies?

Yes