Provide association between program source and created resource

[chrsmith]

There is no association between resources created by a Pulumi program and the program itself. For example, resource X was created by "/src/database.ts@(3:10-3:26)". (i.e. the file, line/col span.)

To support this, we could provide a way for the language host to be fed language-specific diagnostic files (source maps, PDBs, etc.) and optionally include that when making resource requests to the engine.

Being able to navigate from the created resource to the original source context would be, dare I say, ✨ magical ✨ .

/cc @CyrusNajmabadi @lukehoban

[swgillespie]

Mozilla has a very cool library for dealing with source maps: https://github.com/mozilla/source-map.

[joeduffy]

Let's try to put in the foundation for this during M11, and then we can light it up in the UI during M12.

[joeduffy]

@swgillespie just FYI, spoke to @khyperia earlier today about https://github.com/pulumi/pulumi-service/issues/766, which will give us basic source viewing support. I think the source map stuff will take a bit longer to hammer out, so I'm going to move this to M12. Let's think about how these two workstreams eventually dock in together, however.

[swgillespie]

Just spoke to @joeduffy about this and I think there are two desirable sub-tasks here:

1. Given a resource, we should be able to identify the line of code (JavaScript or TypeScript) that resulted in resource creation. This would be the first frame of user code when walking the stack of a call to registerResource.
2. Given a lambda, we should be able to present the original (pre-serialized) source code of that lambda in its original language.

Some thoughts on this:

• v8 is capable of reading source maps and will use them when rendering a stack trace. This information is exposed to us here: https://github.com/v8/v8/wiki/Stack-Trace-API#customizing-stack-traces. For the registerResource case, we can construct a new Error object on entry and inspect the Error's stack trace.
• v8 is also capable of retrieving a Script object from a Function object, but this information is not directly exposed to us. We can reach into v8 debug APIs to do this, via this (deprecated, but still functional) API: https://nodejs.org/api/vm.html#vm_vm_runindebugcontext_code. We can reflect on Script objects to get a Function's filename, line and column offsets, and line count.

As part of this issue I think we need to decide on the format that we'd like to use to persist source information across service boundaries. I don't think we should be in the business of storing custom debug formats for every language we'd want to support, since we'd end up in a combinatorial explosion of formats.

[lukehoban]

Given a resource, we should be able to identify the line of code (JavaScript or TypeScript) that resulted in resource creation. This would be the first frame of user code when walking the stack of a call to registerResource.

It's interesting to ask what "first frame of user code" means - and what we think should actually be displayed here. In practice today - the frames look like:

• registerResource (@pulumi/pulumi)
• new aws.lambda.Function (@pulumi/aws)
• new aws.serverless.Function (@pulumi/aws)
• new Function (@pulumi/cloud-aws)
• <several more frames in @pulumi/cloud-aws)
• http.get("/", (req, res) => {...}) (index.ts)

I assume since we're talking about the resource creation itself, that we mean the second one above. That will effectively always be in @pulumi/aws. It's not immediately clear what value providing a link to that line of code provides.

In the general case, @pulumi/aws is some other 3rd party resource provider - but rarely if ever "user code".

We could assume that anything under @pulumi is not user code (or some other hueristic) - but then we would show the very last line above, and it's not at all clear to the user how that line maps to a Function (or more strangely, to a Role or other esoteric implementation detail).

Probably the most useful/expected would be the third item above - the place where new CustomResource is called. That will frequently be inside an abstraction/component - but that is probably what is intended in that case (with the component creations themselves being traceable back to "user" source through a similar means).

[lukehoban]

Given a lambda, we should be able to present the original (pre-serialized) source code of that lambda in its original language.

I don't quite understand this one. Where would we use this? And what do we mean by "lambda"?

[swgillespie]

@lukehoban

Perhaps the "context" of resource creation can be a full stack trace, then? In that case it can be up to the user to decide what frames they care about.

"Lambda" I meant in the AWS sense, providing some way to correlate an AWS Lambda resource with a JavaScript function that was serialized and is now running in the serverless instance. Something like, when a user discovers that their program deployed a serverless resource, they can see what code got serialized and is now running.

[lukehoban]

Perhaps the "context" of resource creation can be a full stack trace, then? In that case it can be up to the user to decide what frames they care about.

Sure - though I suppose it depends what goal we have for displaying this to users.

"Lambda" I meant in the AWS sense, providing some way to correlate an AWS Lambda resource with a JavaScript function that was serialized and is now running in the serverless instance. Something like, when a user discovers that their program deployed a serverless resource, they can see what code got serialized and is now running.

I see. There are going to in general be many function bodies from the source code which are part of the Lambda, and some subset of those which are in some definition of "user code".

For both of these scenarios, would be useful to sketch an example of what this might look like in a real world use case, to make sure that we have a target experience in mind before diving too far into particular implementation techniques - it seems very possible that the implementation strategy might differ based on what experience we think can be most useful/interesting to users.

[lukehoban]

Moving out of 0.12 so we can focus on other core engine work this week.

Given current priorities, I'm moving to 0.16, but let's discuss any need to prioritize this sooner in the context of fleshing out details of https://github.com/pulumi/home/issues/59.

[swgillespie]

This issue keeps getting punted and we have no real product-wide plan here. I'm going to close this out. If we still want a feature of this magnitude, we should come up with a new design for it.