Allow dynamic querying of resources

[joeduffy]

There may be circumstances in which we want to dynamically discover resources.

For example, when reverse engineering the LumiGL from an existing environment, we may have an object moniker and/or set of properties in hand, and want to discover a resource's ID, if any. There is no reason we couldn't support an API like the following on ResourceProvider, for examples:

rpc Find(moniker, properties) returns (id) {}

Going even further beyond that, we may eventually want a way in Lumi programs themselves to look up resources dynamically. For example, imagine our organization has set up an ECS cluster, and we simply want to plug into that. It would be reasonable to write code like this:

let cluster = aws.ecs.Cluster.find("id");
let container = new aws.ecs.Container({ cluster: cluster, ... });

It's not clear what "id" we would use for this, but presumably the object moniker would be a great place to start. In that case, the hypothetical ResourceProvider.Find API above would do the trick.

It's possible we would want to support more sophisticated queries (eventually), however we need to be careful not to lead people down a bad path. For instance, hard-coding an ARN into a Lumi program would tie that program to a specific deployment, whereas the moniker is sufficiently "logical" that it wouldn't interfere with multi-instancing, etc.

[joeduffy]

This is related to #30 and almost certainly required for #254. So pulling back to 0.3.

[joeduffy]

Here is what I am thinking.

All resources are given three new APIs (shown in TypeScript):

// get looks up a resource by its provider-assigned ID.
static get(id: string): R;
// lookup looks up a resource by the Lumi-assigned URN.
static lookup(urn: string): R;
// query finds resources of the given type with matching properties.
static query(q: Query): R[];

A few notes:

• R is the concrete type of resource these APIs are on.
• For lookup to work, we need additional context. For now, we will just let lookups function across the current stack. Eventually as part of dynamic linking, we can facilitate cross-stack lookups.
• The shape of the Query is still an open question. Both AWS and GCP expose filters, but the latter exposes richer filters with arbitrary query operators. Eventually, I could even imagine a LINQ-like style of querying and mashing up resources across multiple types (getting closer to Prototype scriptable object model #121).

Finally, all of these can bottom out on two APIs in the provider layer:

rpc Get(id string) returns (Resource) {} // get, lookup
rpc Query(q Query) returns (stream Resource) {} // find

Thanks to gRPC streams (over HTTP/2), we can get nice streaming behavior with Lookup. In theory this will allow us to hide the messy "pagination" aspects of the cloud APIs.

[lukehoban]

lookup is a bit odd, as the URN is not something that Lumi developers have ever interacted with before (nothing exposes those into the program or expects them as input). Would be good to see an example of how this is expected to be used in a Lumi program. (If it is just to support dynamic linking, presumably we don't even need to expose that directly into Lumi programs themselves, at least in the near term).

Name for Lookup should presumably be Query, to align with query?

[joeduffy]

I've gone backwards and forwards on this one myself.

The URN actually does show up in a number of places: it's in the state files, printed to the console, and, in fact, as of the output properties change it is accessible as the .urn property on all resources. That said, your point is valid, which is that most developers needn't know this.

You are right that this is a foundational piece for dynamic linking. I don't know that this is all it'll ever be useful for, however. It's possible you'd want to use this as a slightly more stable ID than the physical provider ID, for example. There are many ways I could imagine it being used in practice.

For now, I'll leave it out, since it's always easier to add later on. Best to tackle as part of #109.

Good catch on Query typo in the bug. As you probably guessed, I started out calling it Lookup 😀

Thanks for the feedback!

[joeduffy]

@bforsyth927 is carrying the query pieces forward from here. Reassigning and putting this in 0.4, since we won't have time to finish it this milestone.

[joeduffy]

@bforsyth927 did a great prototype of this during the sprint. It is archived in the PR #289. I'm removing this from the sprint since productizing this will likely wait until a later date.

[joeduffy]

Bringing this back, since we lost our ability to get along the way.

[lukehoban]

One question on this - and I'll open an issue in the pulumi-aws repo as well.

In practice, we haven't needed this in a bunch of places because we have just been projecting Terraform AWS providers with raw string based properties instead of strongly typed properties. This means it's easy to just provide the string value of the id as an input and that you don't need a get.

I actually wonder whether we want to just go back to not trying to project strong types onto the raw projections. That will simplify a lot of these cases, and give us (and users) a lot more flexibility. It's not quite as pretty, but in practice as we've built larger things, this hasn't felt like a cost (and indeed has helped ensure we don't hit too many roadblocks).

[joeduffy]

I will comment on the overlay part in the AWS repo, however, I still think want this capability, so that you can look up arbitrary property values from existing resources. This will also be a foundational capability that we use when enabling #109.

That said, we found a way to work around this in the current customer scenario, so I'm going to punt this to 0.9.

[lukehoban]

@hausdorff did some prototyping of list operations for the @pulumi/kubernetes provider in particular in M21, but focused initially on the stack context, not the target cloud provider context. We'll be looking at building upon that in future milestones.

[ellismg]

We'll land the first part of this (querying against an existing stack's state, not live resources) in #2630.

[lukehoban]

Opened #83 as a more up-to-date tracking item for the remaining piece of this.