Dynamic stack linking

[joeduffy]

Our current model acts more like static linking, in that the resources for dependencies become part of the resource topology in the consuming library. This is okay in some cases -- like when you're building a larger topology out of other smaller 1st party components -- but not in other cases -- like when you are stitching together a complex topology built out of other separately deployed pieces.

As a concrete example, imagine you've factored your overall stack into three key pieces: 1) at the very bottom, one for infrastructure; 2) in the middle, another one for persistent state (presumably parameterized so that restoration from backup state is possible); and 3) at the top, services and applications. Each is revved and deployed independently -- with increasing frequency from bottom-to-top -- ideally in a way that isolates impacts to the other layers as much as possible.

So we will definitely want the equivalent to dynamic linking. But this is pretty fundamental.

As step 1, we should model resource dependencies using URIs. This at least affords some flexibility in the resolution of them to physical resources, at least conceptually. It also gives us guidance around existing schemes for versioning (make a new URI), redirects, and so on. This presumably means we need some way of resolving inter-stack dependencies, however, e.g. a Pulumi "DNS" server.

Even after that, many complexities remain; for example, infrastructure is not always perfectly versionable without a cascading impact to dependencies. A change to the infrastructure that necessitates rebuilding and redeploying the database and/or application tier, for instance, needs to carry that knowledge in a way that at least notifies the operator, if not actually doing something automatically. Even changes that merely alter output values that might have been depended upon, versus destroying and recreating resources, will have similar cascading impacts.

I'm putting this in S10 for consideration. I'm not yet certain when to bite this off, but my inclination is "soon" since it will have some fairly fundamental architectural impacts that we want to front-load.

[joeduffy]

This could also be a premium feature.

[joeduffy]

This won't block our customer conversions this sprint, since we can just use strings and other workarounds. However, we should think about the design here, and if something obvious arises, we should do it. Either way, let's try to end the sprint with an idea of what to do in 0.9.

[joeduffy]

One idea I had here is to use JavaScript exports to define what is available across stacks.

For example, let's say that I have a stack, mystack, that defines a queue I want to consume elsewhere. To make it accessible, I simply say export:

export let queue = new aws.sqs.Queue("requests", ...);

Now in the consuming stack, I can say something like this:

let queue = aws.sqs.Queue.discover("mystack", "requests");

This may be an "abuse" of the module system, but I think I like it. If you prefer static linking, go ahead and statically link; if you prefer dynamic linking, you can do that using discover.

[lukehoban]

If you are referring to it by the urnName requests, why would you need to require exporting it from the source stack? Is that just to allow selectively exposing only a subset of names? Does feel a strange abuse of the module system (which we may want to abuse for a variety of other purposes as well in the future :-)) - especially since the export name is not even how it's being exported.

Al alternative might be just pulumi.export("exportedname", queue) which also allows using a stable export name instead of directly referencing the internal urnName.

[joeduffy]

We found a workaround for this, and so I'm pushing this to 0.9.

[briandrennan]

A change to the infrastructure that necessitates rebuilding and redeploying the database and/or application tier, for instance, needs to carry that knowledge in a way that at least notifies the operator, if not actually doing something automatically.

Isn't this a bit of an orchestration problem? Given the example of a non-compatible database migration, wouldn't it be possible to do something along the lines of create a separate Pulumi program that can handle provisioning the new infrastructure, and then setting up a task (AWS Lambda, Azure Serverless, etc.) that can be used to handle migration, and then make the relevant changes in the primary stack?

I'm not sure if this helps, but I'm dealing with something that I think is similar to what you've described @joeduffy: http://michaeljswart.com/2012/04/modifying-tables-online-part-1-migration-strategy/

[jnancel]

How about making some kind of an authorization system for stacks to read the checkpoint of other stacks ? In that case, no need to export things on the source stack, just to authorize, you then reference in the destination stack the source stack whose checkpoint you want to read, then the name of the resource, then the output you need.

[pgavlin]

Here are the proposed approaches:

Short term

We will introduce a new custom resource type, pulumi.service.StackReference, that accepts a stack name as an argument and returns the stack's output properties. Limiting the returned data to outputs maintains a useful encapsulation boundary: the specific implementation details of the upstream stack are not exposed to the downstream stack. Using a custom resource provides a number of benefits:

• Resources are stored in the checkpoint file. This gives us a simple way in the near term to tell whether or not a stack depends on other stacks from the checkpoint file alone. It does not, however, allow us to easily discover whether or not a stack is depended on by other stacks.
• Resources have lifecycles. This allows us to easily determine when new references are created and old references are removed. In the medium-term, we could perform actual CRUD operations in the resource's lifecycle methods in order to reify these stack dependencies in the Pulumi service.
• Custom resources are implemented by language-independent providers rather than language-dependent SDKs. This allows us to write the underlying code once. It can then be exposed rather easily to JS, Go, Python, etc.

We will also update the Pulumi console and Pulumi CLI to display stack references in a richer manner than normal resources.

Below is an example of three stacks that are layered using StackReference. The first stack creates a VPC, the second stack creates an EKS cluster in the VPC, and the third stack deploys a Helm chart to the EKS cluster.

base-vpc stack

import * as pulumi from "@pulumi/pulumi";
import * as awsinfra from "@pulumi/aws-infra";

// Create and export a VPC.
const vpc = new awsinfra.Network("network");
export const network = vpc;

eks-cluster stack

import * as pulumi from "@pulumi/pulumi";
import * as eks from "@pulumi/eks";

// Import the base VPC from our base stack.
const baseVpc = new pulumi.service.StackReference("myorg/base-vpc").output("network");

// Create an EKS cluster
const cluster = new eks.Cluster("cluster", {
    vpcId: baseVpc.apply(vpc => vpc.vpcId),
    subnetIds: baseVpc.apply(vpc => vpc.subnetIds),
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

app-hackmd stack

import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

// Import the EKS cluster stack
const kubeconfig = new pulumi.service.StackReference("myorg/eks").output("kubeconfig");

// Create a k8s provider that targets the EKS cluster
const k8sProvider = new k8s.Provider("eks", { kubeconfig: kubeconfig.apply(JSON.stringify) });

// Deploy the HackMD Helm chart.
const hackmd = new k8s.helm.v2.Chart("hackmd", {
    repo: "stable",
    chart: "hackmd",
    version: "0.1.1",
    values: {
        service: {
            type: "LoadBalancer",
            port: 80,
        },
    },
}, { providers: { kubernetes: k8sprovider } });

Medium term

We will flesh out the StackReference type:

• We will add support for configuring the backend used to fetch the stack (e.g. service vs. local, API token, etc.)
• We will flesh out the CRUD operations to expose reified stack dependencies in the service and update the console and CLI to display a list of a stack's dependent stacks.