[engine] Add support for source positions

[pgavlin]

These changes add support for passing source position information in gRPC metadata and recording the source position that corresponds to a resource registration in the statefile.

Enabling source position information in the resource model can provide substantial benefits, including but not limited to:

• Better errors from the Pulumi CLI
• Go-to-defintion for resources in state
• Editor integration for errors, etc. from pulumi preview

Source positions are (file, line) or (file, line, column) tuples represented as URIs. The line and column are stored in the fragment portion of the URI as "line(,column)?". The scheme of the URI and the form of its path component depends on the context in which it is generated or used:

• During an active update, the URI's scheme is file and paths are absolute filesystem paths. This allows consumers to easily access arbitrary files that are available on the host.
• In a statefile, the URI's scheme is project and paths are relative to the project root. This allows consumers to resolve source positions relative to the project file in different contexts irrespective of the location of the project itself (e.g. given a project-relative path and the URL of the project's root on GitHub, one can build a GitHub URL for the source position).

During an update, source position information may be attached to gRPC calls as "source-position" metadata. This allows arbitrary calls to be associated with source positions without changes to their protobuf payloads. Modifying the protobuf payloads is also a viable approach, but is somewhat more invasive than attaching metadata, and requires changes to every call signature.

Source positions should reflect the position in user code that initiated a resource model operation (e.g. the source position passed with RegisterResource for pet in the example above should be the source position in index.ts, not the source position in the Pulumi SDK). In general, the Pulumi SDK should be able to infer the source position of the resource registration, as the relationship between a resource registration and its corresponding user code should be static per SDK.

Source positions in state files will be stored as a new registeredAt property on each resource. This property is optional.

Part of #13436

[pulumi-bot]

Changelog

[uncommitted] (2023-07-10)

[pgavlin]

I should point out that these changes don't add any code to the various language SDKs to start reporting source positions. I'll submit additional PRs with those changes.

[pgavlin]

(implementations for the core SDKs are in this branch: pgavlin/source-position...pgavlin/source-position-sdks; only the Node implementation has been tested)

[Frassle]

Have we given any consideration to how MLCs are going to work with this?
Feels like they probably need their positions stripped by the engine, or somehow marked that their a remote source, not the project source.

[Frassle]

Modifying the protobuf payloads is also a viable approach, but is somewhat more invasive than attaching metadata, and requires changes to every call signature.

I think I'd prefer that. This doesn't really feel like metadata given how used it's going to end up being. It also makes it a lot easier for us to document for anyone implementing their own language plugins.

[pgavlin]

Have we given any consideration to how MLCs are going to work with this?
Feels like they probably need their positions stripped by the engine, or somehow marked that their a remote source, not the project source.

Yes--and packages more generally. Part of the motivation behind using URIs rather than simple paths is s.t. we can support alternative schemes. I imagine that one of those schemes might be package://package-name/path#line,column.

Modifying the protobuf payloads is also a viable approach, but is somewhat more invasive than attaching metadata, and requires changes to every call signature.

I think I'd prefer that. This doesn't really feel like metadata given how used it's going to end up being. It also makes it a lot easier for us to document for anyone implementing their own language plugins.

That sounds okay to me. It's a bit of a drag to add this field to registerresource, readresource, invoke, streaminvoke, call, and perhaps registerresourceoutputs, but ultimately it's not too bad.

[pgavlin]

bors merge

[bors]

Build failed:

• bors-ok

[pgavlin]

bors retry

[bors]

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

• bors-ok