diff --git a/content/docs/intro/concepts/config.md b/content/docs/intro/concepts/config.md index 40aaf14e79c9..0d9349f03ca7 100644 --- a/content/docs/intro/concepts/config.md +++ b/content/docs/intro/concepts/config.md @@ -8,6 +8,9 @@ menu: aliases: ["/docs/reference/config/"] --- +Configuration allows you to parameterize your program based on an externally managed configuration file. Configuration can be helpful if you want to, for example, have a different number of servers in your production stack than in development. + +This section describes how to programmatically interact with a configuration that has already been set. In many cases, different stacks for a single project will need differing values. For instance, you may want to use a different size for your AWS EC2 instance, or a different number of servers for your Kubernetes cluster between your development and production stacks. @@ -66,6 +69,10 @@ $ cat my_key.pub | pulumi config set publicKey Configuration can be used from within Pulumi programs by constructing an instance of the `Config` class and using it to `get` or `require` the value of a given config key. Additional details can be found in the [config]({{< relref "/docs/intro/concepts/programming-model#reading-configuration-values" >}}) section of the programming model documentation. +Configuration values can be retrieved using either [`Config.get`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Config.get" >}}) or [`Config.require`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Config.require" >}}). Using [`Config.get`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Config.get" >}}) returns None if the configuration value is not provided, and [`Config.require`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Config.require" >}}) raises an exception with an error message to prevent the deployment from continuing until the variable has been set by using the CLI. + +In this example, the name is required and a lucky number is optional: + {{< chooser language "javascript,typescript,python,go,csharp" >}} {{% choosable language javascript %}} @@ -135,6 +142,20 @@ Console.WriteLine($"Hello, {name} -- I see your lucky number is {lucky}!"); {{< /chooser >}} +Here, we have created an instance of the [`Config`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Config" >}}) class. In that instance, a number of getter functions allow us to read the currently set values. + +This code uses the simple, empty constructor. This default constructor automatically uses the current project as the namespace. If you are writing code that will be imported into another project, such as your own library of components, you should pass your library’s name to the constructor. This string is then used as a namespace for all configuration keys. The [`Config`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Config" >}}) object also provides getters that mark a value as secret. These getters ensure that the underlying raw value is encrypted no matter where it goes. Read more about this in the [`Secrets`]({{< relref "/docs/intro/concepts/programming-model#secrets" >}}) documentation. + +### Typed Configuration Values + +Configuration values are stored as strings, but can be parsed and retrieved as typed values. In addition to Config.get and Config.require , which are untyped, there is also a family of typed functions. For example, Config.get_int converts the string value to a number returns an int value instead of a string, and raises an exception if the value cannot be parsed as a number. We saw this in action above. +For richer structured data, the Config.get_object method can be used to parse JSON values which can be set on the command line with pulumi config set and the --path flag. For example: +$ pulumi config set --path data.active true +$ pulumi config set --path data.nums[0] 1 +$ pulumi config set --path data.nums[1] 2 +$ pulumi config set --path data.nums[2] 3 + + ### Encrypted Secrets {#secrets} Some configuration data is sensitive, such as database passwords or service tokens. For such cases, passing the `--secret` flag to the `config set` command encrypts the data and stores the resulting ciphertext instead of plaintext. diff --git a/content/docs/intro/concepts/inputs-outputs.md b/content/docs/intro/concepts/inputs-outputs.md new file mode 100644 index 000000000000..e0766349cc96 --- /dev/null +++ b/content/docs/intro/concepts/inputs-outputs.md @@ -0,0 +1,167 @@ +--- +title: "Inputs and Outputs" +meta_desc: An in depth look at Pulumi Inputs and Outputs and their usage. +menu: + intro: + parent: concepts + weight: 3 + +aliases: ["/docs/reference/inputs-outputs/"] +--- + +## Inputs and Outputs + +Resource properties are treated specially in Pulumi, both for purposes of input and output. + +All resource arguments accept *inputs*. Inputs are values of type [Input]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}), a type that permits either a raw value of a given type (such as string, integer, boolean, list, map, and so on), an asynchronously computed value (i.e., a `Promise` or `Task`), or an output read from another resource’s properties. + +All resource properties on the instance object itself are *outputs*. Outputs are values of type [Output]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}), which behave very much like [promises](https://en.wikipedia.org/wiki/Futures_and_promises). This is necessary because outputs are not fully known until the infrastructure resource has actually completed provisioning, which happens asynchronously. Outputs are also how Pulumi tracks dependencies between resources. + +Outputs, therefore, represent two things: + +- The eventual raw value of the output +- The dependency on the source(s) of the output value + +Pulumi automatically captures dependencies when you pass an output from one resource as an input to another resource. Capturing these dependencies ensures that the physical infrastructure resources are not created or updated until all their dependencies are available and up-to-date. + +Because outputs are asynchronous, their actual raw values are not immediately available. If you need to access an output’s raw value—for example, to compute a derived, new value, or because you want to log it—you have these options: + +- [Apply]({{< relref "/docs/intro/concepts/programming-model#apply" >}}): a callback that receives the raw value, and computes a new output +- [Lifting]({{< relref "/docs/intro/concepts/programming-model#lifting" >}}): directly read properties off an output value +- [Interpolation]){{< relref "/docs/intro/concepts/programming-model#outputs-and-strings" >}}: concatenate string outputs with other strings directly + +## Apply + +To access the raw value of an output and transform that value into a new value, use [apply]({{> relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" }}). This method accepts a callback that will be invoked with the raw value, once that value is available. + +For example, the following code creates an HTTPS URL from the DNS name (the raw value) of a virtual machine: + +```python +url = virtual_machine.dns_name.apply( + lambda dns_name: "https://" + dns_name +) +``` + +The result of the call to [apply]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) is a new Output. So in this example, the url variable is also an [Output]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}). It will resolve to the new value returned from the callback, and carries the dependencies of the original [Output]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}). If the callback itself returns an Output, the dependencies of that output are also kept in the resulting [Output]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}). + +Note: during some program executions, [apply]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) doesn’t run. For example, it won’t run during a preview, when resource output values may be unknown. Therefore, you should avoid side-effects within the callbacks. For this reason, you should not allocate new resources inside of your callbacks either, as it could lead to `pulumi preview` being wrong. + +## All + +If you have multiple outputs and need to join them, the [all]({{< relref "/docs/intro/concepts/programming-model#all": >}}) function acts like an apply over many resources. The [Output.all]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Output.all" >}}) function acts like an [apply]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) over multiple [Output.all]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Output.all" >}}). This function joins over an entire list of outputs. It waits for all of them to become available and then provides them to the supplied callback. This function can be used to compute an entirely new output value, such as by adding or concatenating outputs from two different resources together, or by creating a new data structure that uses them. Just like with [apply]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}), the result of [Output.all]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Output.all" >}}) is itself an [Output]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}). + +For example, let’s use a server and a database name to create a database connection string: + +```python +from pulumi import Output +# ... +connection_string = Output.all(sql_server.name, database.name) \ + .apply(lambda args: f"Server=tcp:{args[0]}.database.windows.net;initial catalog={args[1]}...") +``` + +Notice that [Output.all]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Output.all" >}}) works by returning an output that represents the combination of multiple outputs so that, within the callback, the raw values are available inside of a tuple. + +## Accessing Properties of an Output by Lifting + +If you just need to access a property of an [Output]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) value in order to pass that property’s value as an argument to another resource’s constructor, you can often just directly access it. + +For example, if to read a domain record from an ACM certificate, you need to drill into a resource’s property value. Because that value is an output, we would normally need to use [apply]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}): + +```pythong +certificate = aws.acm.Certificate('cert', + domain_name='example.com', + validation_method='DNS' +) + +record = aws.route53.Record('validation', + records=[ + # Need to pass along a deep subproperty of this Output + certificate.domain_validation_options.apply( + lambda domain_validation_options: domain_validation_options[0]['resourceRecordValue'] + ) + ], +... +``` + +Instead, to make it easier to access simple property and array elements, an [Output]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) lifts the properties of the underlying value, behaving very much like an instance of it. Lift allows you to access properties and elements directly from the [Output]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) itself without needing [apply]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}). If we return to the above example, we can now simplify it: + +```python +certificate = aws.acm.Certificate('cert', + domain_name='example.com', + validation_method='DNS' +) + +record = aws.route53.Record('validation', + records=[ + certificate.domain_validation_options[0].resource_record_value + ], +... +``` + +This approach is easier to read and write and does not lose any important dependency information that is needed to properly create and maintain the stack. This approach doesn’t work in all cases, but when it does, it can be a great help. + +In JavaScript and TypeScript, a ‘lifted’ property access on an `Output` that wraps undefined produces another `Output` with the undefined value instead of throwing or producing a ‘faulted’ `Output`. In other words, lifted property accesses behave like the [?. (optional chaining operator)](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-7.html#optional-chaining) in JavaScript and TypeScript. This behavior makes it much easier to form a chain of property accesses on an `Output`. + +```javascript +let certValidation = new aws.route53.Record("cert_validation", { + records: [certCertificate.domainValidationOptions[0].resourceRecordValue], + +// instead of + +let certValidation = new aws.route53.Record("cert_validation", { + records: [certCertificate.apply(cc => cc ? cc.domainValidationOptions : undefined) + .apply(dvo => dvo ? dvo[0] : undefined) + .apply(o => o ? o.resourceRecordValue : undefined)], +``` + +```typescript +let certValidation = new aws.route53.Record("cert_validation", { + records: [certCertificate.domainValidationOptions[0].resourceRecordValue], + +// instead of + +let certValidation = new aws.route53.Record("cert_validation", { + records: [certCertificate.apply(cc => cc ? cc.domainValidationOptions : undefined) + .apply(dvo => dvo ? dvo[0] : undefined) + .apply(o => o ? o.resourceRecordValue : undefined)], +``` + +## Working with Outputs and Strings + +Outputs that contain strings cannot be used directly in operations such as string concatenation. String interpolation lets you more easily build a string out of various output values, without needing [apply]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) or [Output.all]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Output.all" >}}). You can use string interpolation to export a stack output, provide a dynamically computed string as a new resource argument, or even simply for diagnostic purposes. + +For example, say you want to create a URL from hostname and port output values: + +```python +hostname: Output[str] = # get some Output +port: Output[int] = # get some Output + +# Would like to produce a string equivalent to: http://${hostname}:${port}/ +url = # ? +``` + +It is possible to use apply and all to do this, of course: + +```python +url = Output.all(hostname, port).apply(lambda l: f"http://{l[0]}:{l[1]}/") +``` + +However, this approach is verbose and unwieldy. To make this common task easier, Pulumi exposes helpers that allow you to create strings that contain outputs—internally hiding all of the messiness required to join them together: + +```python +# concat takes a list of args and concatenates all of them into a single output: +url = Output.concat("http://", hostname, ":", port, "/") +``` + +## Convert Input to Output through Interpolation + +It is possible to turn an [Input]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) into an [Output]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) value. Resource arguments already accept outputs as input values however, in some cases you need to know that a value is definitely an [Output]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) at runtime. Knowing this can be helpful because, since [Input]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) values have many possible representations—a raw value, a promise, or an output—you would normally need to handle all possible cases; by first transforming that value into an [Output]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}), you can treat it uniformly instead. + +For example, this code transforms an [Input]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) into an [Output]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) so that it can use the apply function: + +```python +def split(input): + output = Output.from_input(input); + return output.apply(lambda v: v.split()); +} +``` diff --git a/content/docs/intro/concepts/project.md b/content/docs/intro/concepts/project.md index a956d1630dce..d34b919a847e 100644 --- a/content/docs/intro/concepts/project.md +++ b/content/docs/intro/concepts/project.md @@ -1,6 +1,6 @@ --- -title: "Projects" -meta_desc: An in depth look at Pulumi Projects and their usage. +title: "Projects and Programs" +meta_desc: An in-depth look at Pulumi Projects and their usage. menu: intro: parent: concepts @@ -9,9 +9,51 @@ menu: aliases: ["/docs/reference/project/"] --- -A Pulumi project is any folder which contains a `Pulumi.yaml` file. When in a subfolder, the closest enclosing folder with a `Pulumi.yaml` file determines the current project. A new project can be created with `pulumi new`. A project specifies which runtime to use, which determines where to look for the program that should be executed during deployments. Supported runtimes are `nodejs`, `python`, `dotnet`, and `go`. +In this document, you’ll learn the basics of Pulumi projects and programs. We’ll show you how to create a project and then show you an example of a simple Pulumi program. We’ll give you an overview of a program’s structure and then show you what happens when you run a program. -## Project file {#pulumi-yaml} +## Projects {#pulumi-yaml} + +A Pulumi project is a name and instructions for running a program. You must create a project before you create a program. + +The simplest way to create a new project is with the Pulumi CLI and the `new` command. + +> Assume that all the commands in this document happen in the CLI. + +This example creates a new project for programs written in Typescript that creates infrastructure on AWS. (If you’re following along, first create a new directory and move to it.) + +```bash +$ mkdir myproject && cd myproject +$ pulumi new aws-typescript +This command will walk you through creating a new Pulumi project. + +Enter a value or leave blank to accept the (default), and press . +Press ^C at any time to quit. + +project name: (bucket-ts) +project description: (A minimal AWS TypeScript Pulumi program) +Created project 'bucket-ts' + +Please enter your desired stack name. +To create a stack in an organization, use the format / (e.g. `acmecorp/dev`). +stack name: (dev) +Created stack 'dev' + +aws:region: The AWS region to deploy into: (us-east-1) us-west-2 +Saved config + +Installing dependencies... +Your new project is ready to go! ✨ + +To perform an initial deployment, run 'pulumi up' +``` + +The command creates a project folder that contains: + +- Pulumi.yaml +- Pulumi..yaml +- package.json file for managing dependencies +- tsconfig.json file for setting compiler options +- an index.ts file with a minimal Pulumi program. The `Pulumi.yaml` project file specifies metadata about your project. @@ -48,7 +90,7 @@ A project file contains the following attributes: * `config`: (optional) directory to store stack-specific configuration files, relative to location of `Pulumi.yaml`. * `backend`: (optional) configuration for project state [backend]({{< relref "state#config-stack" >}}). Supports these options: - * `url`: explicitly specify backend url like `https://pulumi.acmecorp.com`, `file:///app/data`, etc. + * `url`: explicitly specify backend URL like `https://pulumi.acmecorp.com`, `file:///app/data`, etc. * `template`: (optional) provides configuration settings that will be used when initializing a new stack from a project file using `pulumi new`. Currently these values are *only* used by `pulumi new`, and not by `pulumi stack init` or as default configuration for existing stacks. * `description`: (optional) a description for the template itself. @@ -59,6 +101,10 @@ A project file contains the following attributes: When using JavaScript, the working directory for the project should contain a `package.json` that points to a file such as `index.js`. In Python, there should either be a `__main__.py` file or a file `setup.py` that defines the entry point. +{{< chooser language "javascript,typescript,go,python,csharp" >}} + +{{% choosable language javascript %}} + A `Pulumi.yaml` file for a `nodejs` program that does not execute TypeScript natively via `ts-node`: ```yaml @@ -70,6 +116,23 @@ runtime: typescript: false ``` +{{% /choosable %}} +{{% choosable language typescript %}} + +A `Pulumi.yaml` file for a `nodejs` program with TypeScript: + +```yaml +name: minimal +description: A minimal Pulumi program. +runtime: + name: nodejs + options: + typescript: true +``` + +{{% /choosable %}} +{{% choosable language go %}} + A `Pulumi.yaml` file for a `go` program that will only use a pre-built executable by the name `mybinary`: ```yaml @@ -81,6 +144,9 @@ runtime: description: A minimal Go Pulumi program ``` +{{% /choosable %}} +{{% choosable language csharp %}} + A `Pulumi.yaml` file for a `dotnet` program that will use a pre-built assembly `MyInfra.dll` under the `bin` directory: ```yaml @@ -92,6 +158,10 @@ runtime: description: A precompiled .NET Pulumi program ``` +{{% /choosable %}} + +{{< /chooser >}} + ### Paths When your Pulumi program references resources in the local filesystem, they are always relative to the working directory. The following example code references a subfolder `app` of the working directory, which would contain a `Dockerfile` and application code: @@ -144,7 +214,28 @@ var myTask = new Task("myTask", new TaskArgs {{< /chooser >}} -## Stack Settings Files {#stack-settings-file} +## Stacks + +A stack is an instance of a program. You can think of a stack as being similar to an environment. For instance you might have a dev stack, a test stack, and a production stack, each of them configured differently. Because you can have as many stacks as you want, a project can have multiple stack configuration files. The naming convention for these files is Pulumi.stackname.yaml. For example, the configuration file for the dev stack is Pulumi.dev.yaml. + +By default, Pulumi creates a stack named dev. In our example, this stack will contain the infrastructure necessary to set up an S3 bucket on AWS. + +To create a different stack with a different name, use this command: + +`stack name: name` + +To create an empty stack, use this command: + +```bash +stack init +``` + +For a list of all the options available with the [init command]({{< relref "/docs/reference/cli/pulumi_stack_init" >}}). + +There’s much more to learn about stacks. For more information, see [Stacks](). + + +### Stack Settings Files {#stack-settings-file} Each stack that is created in a project will have a file named `Pulumi..yaml` which contains the configuration specific to this stack. @@ -153,3 +244,70 @@ For stacks that are actively developed by multiple members of a team, the recomm For stacks that are ephemeral or are used in "inner loop" development, the stack settings are typically not checked into source control. For more information about configuration and how this file is managed using the CLI and programming model, refer to [Configuration and Secrets]({{< relref "config" >}}). + +### Resources + +A stack contains resources. Resources are the pieces that make up your infrastructure. The S3 bucket in our program is an example of a resource. What resources are available and what they’re called depends on the cloud provider you’re using. For more information about resources, see [Resources]({{< relref "" >}}). + + +## Relationships between Projects, Programs and Stacks + +This diagram shows a simplified version of the relationships between projects, programs and stacks. + +```bash +Project +| +|_ Program + | + |_ Stack + | + |_ Resources + +``` + +A program is always a part of a project. A program, when it runs, creates stacks and stacks contain resources. + +## Verify Your Program + +Before creating resources on a cloud provider, verify that your program will create the resources you want. To do this, use the preview command. You’ll see an output something like this: + +```bash +$ pulumi preview +Previewing update (dev) + +View Live: https://app.pulumi.com/spara/bucket-ts/dev/previews/32dfbf67-6a12-4616-b920-415f6be277d3 + + Type Name Plan + + pulumi:pulumi:Stack minimalist-dev create + + └─ aws:s3:Bucket my-bucket create + +Resources: + + 2 to create +``` + +You’ll see that the program will create two resources. One resource is the dev stack, which is an instance of `pulumi:pulumi:Stack`. The stacks contains an instance of the Bucket class. Each resource has a name and the plan is to create those resources on AWS. + +## Deploy to AWS + +You’re finally ready to create your infrastructure. To do that, use the `up` command (up is short for update). You should see something like this. + +```bash +$ pulumi up -f +Updating (dev) + +View Live: https://app.pulumi.com/spara/bucket-ts/dev/updates/2 + + Type Name Status + pulumi:pulumi:Stack bucket-ts-dev + +Outputs: + bucketName: "my-bucket-3247fdd" + +Resources: + 2 unchanged + +Duration: 2s +``` + > Note: To actually create the dev stack on AWS, you’ll need an AWS account and your credentials should be stored ?? Of course, if you’re using another cloud provider, the same requirements apply. + +If you want, you can go to the AWS Management Console to verify that the infrastructure exists. You can also use the Pulumi Console to, for example, keep track of your projects and the stacks you’ve created. For more information, see Pulumi Console. diff --git a/content/docs/intro/concepts/pulumi-up.png b/content/docs/intro/concepts/pulumi-up.png new file mode 100644 index 000000000000..3308197eadf8 Binary files /dev/null and b/content/docs/intro/concepts/pulumi-up.png differ diff --git a/content/docs/intro/concepts/resource-providers.md b/content/docs/intro/concepts/resource-providers.md new file mode 100644 index 000000000000..015e60505740 --- /dev/null +++ b/content/docs/intro/concepts/resource-providers.md @@ -0,0 +1,381 @@ +--- +title: "Resource Providers" +meta_desc: An in-depth look at Resource Providers and their usage. +menu: + intro: + parent: concepts + weight: 2 + +aliases: ["/docs/reference/resource-providers/"] +--- + +A resource provider handles communications with a cloud service to create, read, update, and delete the resources you define in your Pulumi programs. Pulumi passes your code to a language host such as Node.js, waits to be notified of resource registrations, assembles a model of your desired state and calls on the resource provider to produce that state. The resource provider translates those requests into API calls to the cloud service. + +A resource provider is tied to the language that you use to write your programs. For example, if your cloud provider is AWS, the following providers are available: + +- JavaScript/TypeScript: @pulumi/aws +- Python: pulumi-aws +- Go: github.com/pulumi/pulumi-aws/sdk/go/aws +- .NET: Pulumi.Aws + +Normally, since you declare the language and cloud provider you intend to use when you write a program, Pulumi installs the provider for you as a plugin, using the appropriate package manager, such as NPM for Typescript. + +The resource provider for custom resources is determined based by its package name. For example, the aws package loads a plugin named pulumi-resource-aws, and the kubernetes package loads a plugin named pulumi-resource-kubernetes. + +For more information on this and how you can manage installations yourself, see these references. + +## Default Provider Configuration + +By default, each provider uses its package’s global configuration settings, which are controlled by your stack’s configuration. You can set information such as your cloud provider credentials with environment variables and configuration files. If you store this data in standard locations, Pulumi knows how to retrieve them. + +For example, suppose you run this CLI command: + +```bash +$ pulumi config set aws:region us-west-2 +``` + +Then, suppose you deploy the following Pulumi program: + +```typescript +from pulumi_aws import ec2 + +instance = ec2.Instance("myInstance", instance_type="t2.micro", ami="myAMI") +``` + +It creates a single EC2 instance in the us-west-2 region. + +## Explicit Provider Configuration + +While this works for the majority of Pulumi programs, some programs may have special requirements. One example is a program that needs to deploy to multiple AWS regions simultaneously. Another example is a program that needs to deploy to a Kubernetes cluster, created earlier in the program, which requires explicitly creating, configuring, and referencing providers. This is typically done by instantiating the relevant package’s `Provider` type and passing in the options for each [CustomResource]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.CustomResource" >}}) or [ComponentResource]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.ComponentResource" >}}) that needs to use it. For example, the following configuration and program creates an ACM certificate in the `us-east-1` region and a load balancer listener in the `us-west-2` region. + +```typescript +import pulumi +import pulumi_aws as aws + +# Create an AWS provider for the us-east-1 region. +useast1 = aws.Provider("useast1", region="us-east-1") + +# Create an ACM certificate in us-east-1. +cert = aws.acm.Certificate("cert", + domain_name="foo.com", + validation_method="EMAIL", + __opts__=pulumi.ResourceOptions(provider=useast1)) + +# Create an ALB listener in the default region that references the ACM certificate created above. +listener = aws.lb.Listener("listener", + load_balancer_arn=load_balancer_arn, + port=443, + protocol="HTTPS", + ssl_policy="ELBSecurityPolicy-2016-08", + certificate_arn=cert.arn, + default_action={ + "target_group_arn": target_group_arn, + "type": "forward", + }) +``` + +```bash +$ pulumi config set aws:region us-west-2 +``` + +Component resources also accept a set of providers to use with their child resources. For example, the EC2 instance parented to `myResource` in the program below is created in `us-east-1`, and the Kubernetes pod parented to myResource is created in the cluster targeted by the `test-ci` context. + +```python +class MyResource(pulumi.ComponentResource): + def __init__(self, name, opts): + instance = aws.ec2.Instance("instance", ..., __opts__=pulumi.ResourceOptions(parent=self)) + pod = kubernetes.core.v1.Pod("pod", ..., __opts__=pulumi.ResourceOptions(parent=self)) + +useast1 = aws.Provider("useast1", region="us-east-1") +myk8s = kubernetes.Provider("myk8s", context="test-ci") +my_resource = MyResource("myResource", pulumi.ResourceOptions(providers={ + "aws": useast1, + "kubernetes": myk8s, +}) +``` + +## Dynamic Providers + +There are three types of resource providers. The first are the standard resource providers. These resource providers are built and maintained by Pulumi. There is a second kind, called a dynamic resource provider, which we will discuss here. These resource providers run only in the context of your program. They are not shareable. The third type of resource provider is shareable. You write it yourself and then you can distribute it so that others can use it. See Sharable Resource Providers for more information. + +Dynamic resource providers can be written in any language you choose. Because they are not shareable, dynamic resource providers don’t need a plugin . + +There are several reasons why you might want to write a dynamic resource provider. Here are some of them: + +- You want to create some new custom resource types. +- You want to use a cloud provider that Pulumi doesn’t support. For example, you might want to write a dynamic resource provider for WordPress + +All dynamic providers must conform to certain interface requirements. You must at least implement the `create` function but, in practice, you will probably also want to implement the `read`, `update`, and `delete` functions as well. + +To continue with our WordPress example, you would probably want to create new blogs, update existing blogs, and destroy them. The mechanics of how these operations happen would be essentially the same as if you used one of the standard resource providers. The difference is that the calls that would've been made on the standard resource provider by the Pulumi engine would now be made on your dynamic resource provider and it, in turn, would make the API calls to WordPress. + +Dynamic providers are defined by first implementing the `pulumi.dynamic.ResourceProvider` interface. This interface supports all CRUD operations, but only the create function is required. A minimal implementation might look like this: + +```python +from pulumi.dynamic import ResourceProvider, CreateResult + +class MyProvider(ResourceProvider): + def create(self, inputs): + return CreateResult(id_="foo", outs={}) +``` + +This dynamic resource provider is then used to create a new kind of custom resource by inheriting from the `pulumi.dynamic.Resource` base class, which is a subclass of `pulumi.CustomResource`: + +```python +from pulumi import ResourceOptions +from pulumi.dynamic import Resource +from typing import Any, Optional + +class MyResource(Resource): + def __init__(self, name: str, props: Any, opts: Optional[ResourceOptions] = None): + super().__init__(MyProvider(), name, props, opts) +``` + +We can now create instances of the new MyResource resource type in our program with new MyResource("name", args), just like we would any custom resource. Pulumi understands how to use the custom provider logic appropriately. + +Specifically: + +1. If Pulumi determines the resource has not yet been created, it will call the create method on the resource provider interface. +1. If another Pulumi deployment happens and the resource already exists, Pulumi will call the diff method to determine whether a change can be made in place or whether a replacement is needed. +1. If a replacement is needed, Pulumi will call create for the new resource and then call delete for the old resource. +1. If no replacement is needed, Pulumi will call update. +1. In all cases, Pulumi first calls the check method with the resource arguments to give the provider a chance to verify that the arguments are valid. +1. If Pulumi needs to read an existing resource without managing it directly, it will call read. + +See below for details on each of these functions. + +## How Dynamic Providers Work + +Dynamic providers are a flexible and low-level mechanism that allow you to include arbitrary code directly into the deployment process. While most code in a Pulumi program runs while the desired state of the resources is constructed (in other words, as the resource graph is built), the code inside a dynamic provider’s implementation, such as `create` or `update`, runs during resource provisioning, while the resource graph is being turned into a set of CRUD operations scheduled against the cloud provider. + +In fact, these two phases of execution actually run in completely separate processes. The construction of a `new MyResource` happens inside the JavaScript, Python, or Go process running in your Pulumi program. In contrast, your implementations of create or update are executed by a special resource provider binary called `pulumi-resource-pulumi-nodejs`. This binary is what actually implements the Pulumi resource provider gRPC interface and it speaks directly to the Pulumi engine. + +Because your implementation of the resource provider interface must be used by a different process, potentially at a different point in time, dynamic providers are built on top of the same [function serialization]({{< relref "/docs/tutorials/aws/serializing-functions" >}}) that is used for turning callbacks into AWS Lambdas or Google Cloud Functions. Because of this serialization, there are some limits on what can be done inside the implementation of the resource provider interface. You can read more about these limitations in the function serialization documentation. + +## The Resource Provider Interface + +Implementing the `pulumi.dynamic.ResourceProvider` interface requires implementing a subset of the methods listed further down in this section. Each of these methods can be asynchronous, and most implementations of these methods will perform network I/O to provision resources in a backing cloud provider or other resource model. There are several important contracts between a dynamic provider and the Pulumi CLI that inform when these methods are called and with what data. + +Though the input properties passed to a `pulumi.dynamic.Resource` instance will usually be [Input values]({{< relref "/docs/intro/concepts/programming-model#outputs" >}}), the dynamic provider’s functions are invoked with the fully resolved input values in order to compose well with Pulumi resources. Strong typing for the inputs to your provider’s functions can help clarify this. You can achieve this by creating a second interface with the same properties as your resource’s inputs, but with fully unwrapped types. + +```python +from pulumi import Input, Output, ResourceOptions +from pulumi.dynamic import * +from typing import Any, Optional + +class MyResourceInputs(object): + my_string_prop: Input[str] + my_bool_prop: Input[bool] + + def __init__(self, my_string_prop, my_bool_prop): + self.my_string_prop = my_string_prop + self.my_bool_prop = my_bool_prop + +class _MyResourceProviderInputs(object): + """ + MyResourceProviderInputs is the unwrapped version of the same inputs + from the MyResourceInputs class. + """ + my_string_prop: str + my_bool_prop: bool + + def __init__(self, my_string_prop: str, my_bool_prop: bool): + self.my_bool_prop = my_bool_prop + self.my_string_prop = my_string_prop + +class MyResourceProvider(ResourceProvider): + def create(self, inputs: _MyResourceProviderInputs) -> CreateResult: + ... + return CreateResult() + + def diff(self, id: str, oldInputs: _MyResourceProviderInputs, newInputs: _MyResourceProviderInputs) -> DiffResult: + ... + return DiffResult() + +class MyResource(Resource): + def __init__(self, name: str, props: MyResourceInputs, opts: Optional[ResourceOptions] = None): + super().__init__(MyResourceProvider(), name, {**vars(props)}, opts) +``` + +check(olds, news) + +The `check` method is invoked before any other methods. The resolved input properties that were originally provided to the resource constructor by the user are passed to it. The operation is passed both the old input properties that were stored in the *state file* after the previous update to the resource, as well as the new inputs from the current deployment. It has two jobs: + +1. Verify that the inputs (particularly the news) are valid or return useful error messages if they are not. +1. Return a set of checked inputs. + +The inputs returned from the call to `check` will be the inputs that the Pulumi engine uses for all further processing of the resource, including the values that will be passed back in to `diff`, `create`, `update`, or other operations. In many cases, the news can be returned directly as the checked inputs. But in cases where the provider needs to populate defaults, or do some normalization on values, it may want to do that in the `check` method so that this data is complete and normalized prior to being passed in to other methods. + +create(inputs) + +The `create` method is invoked when the URN of the resource created by the user is not found in the existing state of the deployment. The engine passes the provider the checked inputs returned from the call to `check`. The `create` method creates the resource in the cloud provider. It then returns two pieces of data: + +1. An id that can uniquely identify the resource in the backing provider for later lookups, and +1. A set of outputs from the backing provider that should be returned to the user code as properties on the CustomResource object. These outputs are stored in the checkpoint file. If an error occurs, an exception can be thrown from the create method that should be returned to the user. + +diff(id, olds, news) + +The `diff` method is invoked when the URN of the resource created by the user already exists. Because the resource already exists it will need to be either updated or replaced. The `diff` method is passed the `id` of the resource, as returned by `create`, as well as the old outputs from the checkpoint file, which are values returned from a previous call to either `create` or `update`. The checked inputs from the current deployment are passed to the diff method. + +It returns four optional values: + +- `changes: true` if the provider believes there is a difference between the olds and news and wants to do an update or replace to affect this change. +- `replaces`: An array of property names that have changed that should force a replacement. Returning a non-zero length array tells the Pulumi engine to schedule a replacement instead of an update. Replacements might involve downtime, so this value should only be used when a diff requested by the user cannot be implemented as an in-place update on the cloud provider. +- `stables`: An array of property names that are known not to change between updates. Pulumi will use this information to allow some [`apply`]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) calls on [`Output[T]`]({{< relref "/docs/reference/pkg/python/pulumi#outputs-and-inputs" >}}) to be processed during `previews` because it knows that the values of these property names will stay the same during an update. +- `deleteBeforeReplace`: true if the proposed replacements require that the existing resource be deleted before creating the new one. By default, Pulumi will try to create the new resource before deleting the old one to avoid downtime. If an error occurs, an exception can be thrown from the diff method to return this error to the user. + +update(id, olds, news) + +The `update` method is invoked if the call to diff indicates that a replacement is unnecessary. The method is passed the `id` of the resource as returned by `create`, and the old outputs from the checkpoint file, which are values returned from a previous call to either `create` or `update`. The new checked inputs are also passed from the current deployment. The `update` method is expected to do the work in the cloud provider to update an existing resource to the new desired state. It then returns a new set of `outputs` from the cloud provider that should be returned to the user code as properties on the [`CustomResource`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.CustomResource" >}}) object, and stored into the checkpoint file. If an error occurs, an exception can be thrown from the `update` method to return this error to the user. + +delete(id, props) + +The `delete` operation is invoked if the URN exists in the previous state but not in the new desired state, or if a replacement is needed. The method is passed the `id` of the resource as returned by `create`, and the old outputs from the checkpoint file, which are values returned from a previous call to either `create` or `update`. The method deletes the corresponding resource from the cloud provider. Nothing needs to be returned. If an error occurs, an exception can be thrown from the `delete` method to return this error to the user. + +read(id, props) + +The `read` method is invoked when the Pulumi engine needs to get data about a resource that is not managed by Pulumi. The method is passed the `id` of the resource, as tracked in the cloud provider, and an optional bag of additional properties that can be used to disambiguate the request, if needed. The `read` method looks up the requested resource, and returns the canonical `id` and output properties of this resource if found. If an error occurs, an exception can be thrown from the `read` method to return this error to the user. + +##Dynamic Resource Inputs + +The inputs to your `pulumi.dynamic.ResourceProvider`’s functions come from subclasses of `pulumi.dynamic.Resource`. These inputs include any values in the input arguments passed to the `pulumi.dynamic.Resource` constructor. This is just a map of key/value pairs however, in statically typed languages, you can declare types for these input shapes. + +For example, `props`, in the `MyResource` class shown below, defines the inputs to the resource provider functions: + +```python +from pulumi import Input, ResourceOptions +from pulumi.dynamic import Resource +from typing import Any, Optional + +class MyResourceInputs(object): + my_string_prop: Input[str] + my_bool_prop: Input[bool] + def __init__(self, my_string_prop, my_bool_prop): + self.my_string_prop = my_string_prop + self.my_bool_prop = my_bool_prop + +class MyResource(Resource): + def __init__(self, name: str, props: MyResourceInputs, opts: Optional[ResourceOptions] = None): + super().__init__(MyProvider(), name, {**vars(props)}, opts) +``` + +## Dynamic Resource Outputs + +Any outputs can be returned by your create function in the outs property of `pulumi.dynamic.CreateResult`. + +> Note: The following only applies to statically typed languages. + +If you need to access the outputs of your custom resource outside it with strong typing support, declare each output property returned in the `outs` property by your `create` function as a class member of the `pulumi.dynamic.Resource` itself. For example, in TypeScript, these outputs must be declared as `public readonly` class members in your `pulumi.dynamic.Resource` class. These class members must also have the type `pulumi.Output`. + +> Note: The name of the class member must match the names of the output properties as returned by the create function. + +```python +from pulumi import ResourceOptions, Input, Output +from pulumi.dynamic import Resource, ResourceProvider, CreateResult +from typing import Any, Optional + +... +... + +class MyProvider(ResourceProvider): + def create(self, inputs): + return CreateResult(id_="foo", outs={ 'my_number_output': 12, 'my_string_output': "some value" }) + +class MyResource(Resource): + my_string_output: Output[str] + my_number_output: Output[str] + + def __init__(self, name: str, props: MyResourceInputs, opts: Optional[ResourceOptions] = None): + super().__init__(MyProvider(), name, { 'my_string_output': None, 'my_number_output': None, **vars(props) }, opts) +``` + +## Dynamic Provider Examples + +### Example: Random + +This example generates a random number using a dynamic provider. It highlights using dynamic providers to run some code only when a resource is created, and then store the results of that in the state file so that this value is maintained across deployments of the resource. Because we want our random number to be created once, and then remain stable for subsequent updates, we cannot simply use a random number generator in our program; we need dynamic providers. The result is a provider similar to the one provided in `@pulumi/random`, just specific to our program and language. + +Implementing this example requires that we have a provider and resource type: + +```python +from pulumi import ResourceOptions +from pulumi.dynamic import Resource, ResourceProvider, CreateResult +from typing import Optional +import binascii +import os + +class RandomProvider(ResourceProvider): + def create(self, inputs): + return CreateResult(id_=binascii.b2a_hex(os.urandom(16)), outs={}) + +class Random(Resource): + def __init__(self, name: str, opts: Optional[ResourceOptions] = None): + super().__init__(RandomProvider(), name, {}, opts) +``` + +Now, with this, we can construct new `Random` resource instances, and Pulumi will drive the right calls at the right time. + +### Example: GitHub Labels REST API + +This example highlights how to make REST API calls to a backing provider to perform CRUD operations. In this case, the backing provider is the GitHub API in this case. Because the resource provider method implementations will be serialized and used in a different process, we keep all the work to initialize the REST client and to make calls to it, local to each function. + +```python +from pulumi import ComponentResource, export, Input, Output +from pulumi.dynamic import Resource, ResourceProvider, CreateResult, UpdateResult +from typing import Optional +from github import Github, GithubObject + +auth = "" +g = Github(auth) + +class GithubLabelArgs(object): + owner: Input[str] + repo: Input[str] + name: Input[str] + color: Input[str] + description: Optional[Input[str]] + def __init__(self, owner, repo, name, color, description=None): + self.owner = owner + self.repo = repo + self.name = name + self.color = color + self.description = description + +class GithubLabelProvider(ResourceProvider): + def create(self, props): + l = g.get_user(props["owner"]).get_repo(props["repo"]).create_label( + name=props["name"], + color=props["color"], + description=props.get("description", GithubObject.NotSet)) + return CreateResult(l.name, {**props, **l.raw_data}) + def update(self, id, _olds, props): + l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id) + l.edit(name=props["name"], + color=props["color"], + description=props.get("description", GithubObject.NotSet)) + return UpdateResult({**props, **l.raw_data}) + def delete(self, id, props): + l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id) + l.delete() + +class GithubLabel(Resource): + name: Output[str] + color: Output[str] + url: Output[str] + description: Output[str] + def __init__(self, name, args: GithubLabelArgs, opts = None): + full_args = {'url':None, 'description':None, 'name':None, 'color':None, **vars(args)} + super().__init__(GithubLabelProvider(), name, full_args, opts) + +label = GithubLabel("foo", GithubLabelArgs("lukehoban", "todo", "mylabel", "d94f0b")) + +export("label_color", label.color) +export("label_url", label.url) +``` + +### Additional Examples + +- [Add a Custom Domain to an Azure CDN endpoint](https://github.com/pulumi/examples/tree/master/azure-ts-dynamicresource) + Similar to the previous example, this is another example of a shortcoming of the regular Azure resource provider available in Pulumi. However, due to the availability of a REST API, we can easily add a custom domain to an Azure CDN resource using a dynamic provider. +- [Dynamic Providers as Provisioners](https://github.com/pulumi/examples/tree/master/aws-ts-ec2-provisioners) + Provisioning a VM after it is created is a common problem. Developers have the option to run user-supplied scripts while creating the VM itself. For example, the AWS EC2 resource has a userData parameter, that allows you to specify an inline script, which EC2 will run at instance startup. However, this example of dynamic providers as provisioners allows you to copy/execute scripts on the target instance without replacing the instance itself. diff --git a/content/docs/intro/concepts/resources.md b/content/docs/intro/concepts/resources.md new file mode 100644 index 000000000000..253604ce4b38 --- /dev/null +++ b/content/docs/intro/concepts/resources.md @@ -0,0 +1,480 @@ +--- +title: "Resources" +meta_desc: An in-depth look at Resources and their usage. +menu: + intro: + parent: concepts + weight: 2 + +aliases: ["/docs/reference/resources/"] +--- + +A resource represents one or more pieces in your infrastructure. For example, a resource might represent a compute instance or an S3 bucket on AWS. All infrastructure resources are described by one of two subclasses of the [`Resource `({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Resource" >}})]class. These two subclasses are: + +- [`CustomResource`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.CustomResource" >}}): A custom resource is the most common kind of resource. A custom resource is a resource managed by a resource provider. (A resource provider creates the infrastructure you’ve described in your program in the cloud service you’ve selected. For more information about resource providers, see [Providers]().) +- [`ComponentResource`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.ComponentResource" >}}): A component resource is an aggregation of many resources that forms a larger abstraction. + +## Resource Libraries + +Pulumi has libraries for AWS, Google, Azure, with Kubernetes, as well as other services. These libraries describe all the resources that each cloud service offers. When you write a program, you import the relevant library package. Here are links to the current resource libraries: + +## Custom Resources + +A custom resource’s desired state is declared by constructing an instance: + +```python +res = Resource(name, args, options) +``` + +All resources have a required [`name`]({{< relref "/intro/concepts/programming-model#names" >}}) argument, which must be unique across resources of the same kind in a [`stack`]({{< relref "/docs/intro/concepts/stack">}}). This logical name influences the physical name assigned by your infrastructure’s cloud provider. [Pulumi auto-names physical resources]({{< relref "/docs/intro/concepts/programming-model#autonaming" >}}) by default, so the physical name and the logical name may differ. (For more information about auto-naming, see [Physical Names and Auto-Naming](), below. + +The `args` argument is an object with a set of named property input values that are used to initialize the resource. These can be normal raw values—such as strings, integers, lists, and maps—or [outputs from other resources]({{< relref "/docs/intro/concepts/programming-model/#outputs" >}}). For more information, see [Inputs and Outputs](). + +The `options` argument is optional, but [lets you control certain aspects of the resource]({{< relref "/docs/intro/concepts/programming-model#resourceoptions" >}}). For example, you can show explicit dependencies, use a custom provider configuration, or import an existing infrastructure. For more information, [Resource Options]({{< relref "/docs/intro/concepts/programming-model#resourceoptions" >}}) later in this document. + +## Declaring Infrastructure + +To declare new infrastructure in your program, allocate a resource object whose properties correspond to the desired state of your infrastructure. For example, this program creates a new AWS EC2 security group and an instance that uses it: + +```python +import pulumi_aws as aws + +group = aws.ec2.SecurityGroup('web-sg', + description='Enable HTTP access', + ingress=[ + { 'protocol': 'tcp', 'from_port': 80, 'to_port': 80, 'cidr_blocks': ['0.0.0.0/0'] } + ]) + +server = aws.ec2.Instance('web-server', + ami='ami-6869aa05', + instance_type='t2.micro', + security_groups=[group.name]) # reference the security group resource +``` + +In this example, the two resource objects, `group` and `server`, plus their logical names and properties, tell Pulumi everything it needs to create, update, or delete your infrastructure. For example, Pulumi now knows you’d like an EC2 security group named `web-sg` with a single ingress rule, and a `t2.micro`-sized EC2 instance that runs AMI `ami-8689aa05` and uses the `web-sg` security group. + +Thanks to [output properties]({{< relref "/docs/intro/concepts/programming-model/#outputs" >}}), Pulumi understands the dependencies between resources. Pulumi uses that information to create a dependency graph of resources. With that graph, it can maximize parallelism and ensure correct ordering. When you run the `pulumi up` command, Pulumi computes the desired state, compares it to the current infrastructure (if present), shows the delta, and confirms and carries out the changes. + +You can export the resulting infrastructure values if you want to access them outside your application. For example, adding this code to the example exports the server’s resulting IP address and DNS name: + +```python +# ... +pulumi.export('public_ip', server.public_ip) +pulumi.export('public_dns', server.public_dns) +``` + +The exported values are printed after you do a `pulumi up` and they are easy to access from the CLI’s pulumi stack output command. To learn more, see [stack outputs]({{< relref "/docs/intro/concepts/programming-model/#stack-outputs" >}}). + +The export call/expression stores the computed values on the stack's checkpoint (this is a state file, the JSON-serialized representation of the stack as of the last update), which makes them available as stack "outputs" -- values you can reference programmatically or with the CLI: + +![stack output](stack-outputs.png) + +You can run commands such as `pulumi stack output bucketUrl` to get a specific value, or `pulumi stack output` to get all of them. + +## Resource Names + +Every resource managed by Pulumi has a logical name that you specify as an argument to its constructor. For instance, the logical name of this IAM role is `my-role`: + +```python +role = iam.Role("my-role") +``` + +The logical name you specify during resource creation is used in two ways: + +- As a default prefix for the resource’s physical name, assigned by the cloud provider. +- To construct the [Universal Resource Name (URN)]({{< relref "/docs/intro/concepts/programming-model#urns" >}}) used to track the resource across updates. + +Pulumi uses the logical name to track the identity of a resource through multiple deployments of the same program and uses it to choose between creating new resources or updating existing ones. + +Pulumi uses the logical name to track the identity of a resource through multiple deployments of the same program and to choose between creating new resources or updating existing ones. + +> Note that the variable names assigned to resource objects aren’t used for either logical or physical resource naming. The variable only refers to that resource in that program. For example, in this code: + +```typescript +var foo = new aws.Thing("my-thing"); +``` + +The word foo has no bearing at all on the resulting infrastructure. The resources that comprise the resulting infrastructure never use the word foo. You could change it to bar, run a pulumi up, and the result would be no changes. (The only exception is if you export that variable, in which case the name of the export would change from foo to bar). + +## Physical Names and Auto-Naming + +A resource’s logical and physical names may not match. In fact, most physical resource names in Pulumi are, by default, auto-named. As a result, even if your IAM role has a logical name of `my-role`, the physical name will typically look something like `my-role-d7c2fa0`. The suffix appended to the end of the name is random. + +This random suffix serves two purposes: + +- It ensures that two stacks for the same project can be deployed without their resources colliding. The suffix helps you to create multiple instances of your project more easily, whether because you want, for example, many development or testing stacks, or to scale to new regions +- It allows Pulumi to do zero-downtime resource updates. Certain updates require replacing resources rather than updating them in place, due to the way some cloud providers work. By default, Pulumi creates replacements first, then updates the existing references to them, and finally deletes the old resources. + +For cases that require specific names, you can override auto-naming by specifying a physical name. Most resources have a `name` property that you can use to name the resource yourself. Specify your name in the argument object to the constructor. Here’s an example. + +```python +role = iam.Role('my-role', { + name='my-role-001' +}) +``` + +### Auto-naming + +Auto-naming generates physical names for your resources by adding a suffix to the resource’s logical name. Without auto-naming, you would have to manually assign names to your resources, possibly across multiple stacks. Auto-naming is also the reason that zero-downtime updates can happen. Without it, Pulumi would first need to delete the resources and then create new resources. This sequence is much more impactful and does require downtime. +For use cases that require specific physical names, you can override auto-naming by manually specifying a physical name. Most (but not all) resources offer this option by way of a `name` property that can be specified in the argument object to the constructor: + +```python +role = iam.Role('my-role', { + name='my-role-001' +}) +``` + +If `name` doesn’t work, consult the [API Reference]({{< relref "/docs/reference/pkg" >}}) for the specific resource you are creating. Someresources use a different property to override auto-naming. For instance, the `aws.s3.Bucket` type uses the property `bucket` instead of name. Other resources, such as `aws.kms.Key`, don’t have physical names and use other auto-generated IDs to uniquely identify them. + +Overriding auto-naming opens your project up to naming collisions. As a result, for resources that may need to be replaced, you should specify deleteBeforeReplace: true in the resource’s options. This option ensures that old resources are deleted before new ones are created which will prevent those collisions. + +Because physical and logical names don’t need to match, you can construct the physical name by using your project and stack names. Similarly to auto-naming, this approach protects you from naming collisions while still having meaningful names. Note that `deleteBeforeReplace` is still necessary: + +```pythin +role = iam.Role('my-role', { + name='my-role-{}-{}'.format(pulumi.get_project(), pulumi.get_stack() +}, opts=ResourceOptions(delete_before_replace=True)) +``` + +This example shows that, if you decided to override auto-naming by providing your own name property, you could use the `get_project()` and `get_stack()` runtime functions to retrieve the project and stack names. You could then use those values to assemble a physical name of your own that would remain unique across the stacks of the project. + +On the other hand, if you provided only + +```python +name='my-role' +``` + +then your first `pulumi up` (say, for the dev stack) might succeed, but creating a second stack with that same code would likely fail, because the name `my-role` would already be in use. + +Adding the project name and stack name to the string (thereby making the role’s names `my-role-my-project-dev`, `my-role-my-project-prod`, etc.) helps to make the name a little more unique and therefore more durable across multiple stacks. + +## Resource URNs + +Each resource is assigned a [Uniform Resource Name (URN)](https://en.wikipedia.org/wiki/Uniform_Resource_Name) that uniquely identifies that resource globally. Unless you are writing a tool, you will seldom need to interact with an URN directly, but it is fundamental to how Pulumi works so it’s good to have a general understanding of it. + +The URN is automatically constructed from the project name, stack name, resource name, resource type, and the types of all the parent resources (in the case of [component resources](). Here’s an example of an URN. + +``` +urn:pulumi:production::acmecorp-website::custom:resources:Resource$aws:s3/bucket:Bucket::my-bucket + ^^^^^^^^^^ ^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^ + +``` + +The URN must be globally unique. This means all of the components that go into a URN must be unique within your program. If you create two resources with the same name, type, and parent path, for instance, you will see an error: + +```bash +error: Duplicate resource URN 'urn:pulumi:production::acmecorp-website::custom:resources:Resource$aws:s3/bucket:Bucket::my-bucket'; try giving it a unique name +``` + +Any change to the URN of a resource causes the old and new resources to be treated as unrelated—the new one will be created (since it was not in the prior state) and the old one will be deleted (since it is not in the new desired state). This happens when you change the name used to construct the resource or the structure of a resource’s parent hierarchy. Both of these operations will lead to a different URN, and thus require the `create` and `delete` operations instead of an `update` or `replace` operation that you would use for an existing resource. In other words, be careful when you change a resource’s name. + +If you’d like to rename a resource without destroying the old one, refer to the [aliases]({{< relref "/docs/intro/concepts/programming-model#aliases" >}}) capability. + +Resources constructed as children of a [component]({{< relref "/docs/intro/concepts/programming-model/#components" >}}) resource should have their names that are unique across multiple instances of the component resource. In general, the name of the component resource instance itself (the `name` parameter passed into the component resource constructor) should be used as part of the name of the child resources. + +### Resource Arguments + +A resource’s argument parameters differ by resource type. Each resource has a number of named input properties that control the behavior of the resulting infrastructure. To determine what arguments a resource supports, refer to that resource’s [API documentation]({{< relref "/docs/reference/pkg/" >}}). + +### Resource Options + +All resource constructors accept an options argument that provide the following resource options: + +- [additionalSecretOutputs]({{< relref "/docs/intro/concepts/programming-model#additionalsecretoutputs" >}}): specify properties that must be encrypted as secrets. +- [aliases]({{< relref "/docs/intro/concepts/programming-model#aliases" >}}): specify aliases for this resource, so that renaming or refactoring doesn’t replace it. +- [customTimeouts]({{< relref "/docs/intro/concepts/programming-model#customtimeouts" >}}): override the default retry/timeout behavior for resource provisioning. The default value varies by resource. +- [deleteBeforeReplace]({{< relref "/docs/intro/concepts/programming-model#deletebeforereplace" >}}): override the default create-before-delete behavior when replacing a resource. +- [dependsOn]({{< relref "/docs/intro/concepts/programming-model#dependson" >}}): specify additional explicit dependencies in addition to the ones in the dependency graph. +- [ignoreChanges]({{< relref "/docs/intro/concepts/programming-model#ignorechanges >}}): declare that changes to certain properties should be ignored during a diff. +- [import]({{< relref "/docs/intro/concepts/programming-model#import" >}}: bring an existing cloud resource into Pulumi. +- [parent]({{< relref "/docs/intro/concepts/programming-model#parent" >}}): establish a parent/child relationship between resources. +- [protect]({{< relref "/docs/intro/concepts/programming-model#protect" >}}): prevent accidental deletion of a resource by marking it as protected. +- [provider]({{< relref "/docs/intro/concepts/programming-model#provider" >}}): pass an [explicitly configured provider]({{< relref "/docs/intro/concepts/programming-model#explicit-provider-configuration" >}}), instead of using the default global provider. +- [transformations]({{< relref "/docs/intro/concepts/programming-model#transformations" >}}): dynamically transform a resource’s properties on the fly. +- [version]({{< relref "/docs/intro/concepts/programming-model#version" >}}): pass a provider plugin version that should be used when operating on a resource. + +#### additionalSecretOutputs + +This option specifies a list of named output properties that should be treated as [secrets]({{< relref "/docs/intro/concepts/programming-model#secrets" >}}), which means they will be encrypted. It augments the list of values that Pulumi detects, based on secret inputs to the resource. + +This example ensures that the password generated for a database resource is an encrypted secret: + +```python +db = Database('db', + opts=ResourceOptions(additional_secret_outputs=['password'])) +``` + +Only top-level resource properties can be designated secret. If sensitive data is nested inside of a property, you must mark the entire top-level output property as secret. +aliases + +This option provides a list of aliases for a resource or component resource. If you’re changing the name, type, or parent path of a resource or component resource, you can add the old name to the list of aliases for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource. + +For example, imagine we change a database resource’s name from `old-name-for-db` to `new-name-for-db`. By default, when we run pulumi up, we see that the old resource is deleted and the new one created. If we annotate that resource with the aliases option, however, the resource is updated in-place: + +```python +db = Database('db', + opts=ResourceOptions(aliases=[Alias(name='old-name-for-db')])) +``` + +The aliases option accepts a list of old identifiers. If a resource has been renamed multiple times, it can have many aliases. The list of aliases may contain old Alias objects and/or old resource URNs. + +The above example used objects of type Alias with the old resource names. These values may specify any combination of the old name, type, parent, stack, and/or project values. Alternatively, you can just specify the URN directly: + +```python +db = Database('db', + opts=ResourceOptions(aliases=['urn:pulumi:stackname::projectname::aws:rds/database:Database::old-name-for-db'])) +``` + +#### customTimeouts + +This option provides a set of custom timeouts for `create`, `update`, and `delete` operations on a resource. These timeouts are specified using a duration string such as "5m" (5 minutes), "40s" (40 seconds), or "1d" (1 day). Supported duration units are "ns", "us" (or "µs"), "ms", "s", "m", and "h" (nanoseconds, microseconds, milliseconds, seconds, minutes, and hours, respectively). + +For the most part, Pulumi automatically waits for operations to complete and times out appropriately. In some circumstances, such as working around bugs in the infrastructure provider, custom timeouts may be necessary. + +This example specifies that the create operation should wait up to 30 minutes to complete before timing out: + +```python +db = Database('db', + opts=ResourceOptions(custom_timeouts=CustomTimeouts(create='30m'))) +``` + +#### deleteBeforeReplace + +A resource may need to be replaced if an immutable property changes. In these cases, cloud providers do not support updating an existing resource so a new instance will be created and the old one deleted. By default, to minimize downtime, Pulumi creates new instances of resources before deleting old ones. + +Setting the `deleteBeforeReplace` option to true means that Pulumi will delete the existing resource before creating its replacement. Be aware that this behavior has a cascading impact on dependencies so more resources may be replaced, which can lead to downtime. However, this option may be necessary for some resources that manage scarce resources behind the scenes, and/or resources that cannot exist side-by-side. + +This example deletes a database entirely before its replacement is created: + +```python +db = Database("db", + opts=ResourceOptions(delete_before_replace=True)) +``` + +#### dependsOn + +The `dependsOn` option creates a list of explicit dependencies between resources. + +Pulumi automatically tracks dependencies between resources when you supply an input argument that came from another resource’s output properties. In some cases, however, you may need to explicitly specify additional dependencies that Pulumi doesn’t know about but must still respect. This might happen if a dependency is external to the infrastructure itself—such as an application dependency—or is implied due to an ordering or eventual consistency requirement. The dependsOn option ensures that resource creation, update, and deletion operations are done in the correct order. + +This example demonstrates how to make res2 dependent on res1, even if there is no property-level dependency: + +```python +res1 = MyResource("res1"); +res2 = MyResource("res2", opts=ResourceOptions(depends_on=[res1])); +``` + +#### ignoreChanges + +This option specifies a list of properties that Pulumi will ignore when it updates existing resources. Any properties specified in this list that are also specified in the resource’s arguments will only be used when creating the resource. + +For instance, in this example, the resource’s prop property "new-value" will be set when Pulumi initially creates the resource, but from then on, any updates will ignore it: + +```python +res = MyResource("res", + prop="new-value", + opts=ResourceOptions(ignore_changes=["prop"])) +``` + +One reason you would use the ignoreChanges option is to ignore changes in properties that lead to diffs. Another reason is to change the defaults for a property without forcing all existing deployed stacks to update or replace the affected resource. This is common after you’ve imported existing infrastructure provisioned by another method into Pulumi. In these cases, there may be historical drift that you’d prefer to retain, rather than replacing and reconstructing critical parts of your infrastructure. + +> Note: The property names passed to ignoreChanges should always be the “camelCase” version of the property name, as used in the core Pulumi resource model. + +#### import + +This option imports an existing cloud resource so that Pulumi can manage it. Imported resources can have been provisioned by any other method, including manually in the cloud console or with the cloud CLI. + +To import a resource, first specify the `import` option with the resource’s ID. This ID is the same as would be returned by the id property for any resource created by Pulumi; the ID is resource-specific. Pulumi reads the current state of the resource with the given ID from the cloud provider. Next, you must specify all required arguments to the resource constructor so that it exactly matches the state to import. By doing this, you end up with a Pulumi program that will accurately generate a matching desired state. + +This example imports an existing EC2 security group with ID sg-04aeda9a214730248 and an EC2 instance with ID `i-06a1073de86f4adef`: + +```python +# IMPORTANT: Python appends an underscore (`import_`) to avoid conflicting with the keyword. + +import pulumi_aws as aws + +group = aws.ec2.SecurityGroup('web-sg', + name='web-sg-62a569b', + description='Enable HTTP access', + ingress=[ + { 'protocol': 'tcp', 'from_port': 80, 'to_port': 80, 'cidr_blocks': ['0.0.0.0/0'] } + ], + opts=ResourceOptions(import_='sg-04aeda9a214730248')) + +server = aws.ec2.Instance('web-server', + ami='ami-6869aa05', + instance_type='t2.micro', + security_groups=[group.name], + opts=ResourceOptions(import_='i-06a1073de86f4adef')) +``` + +For this to work, your Pulumi stack must be configured correctly. In this example, it’s important that the AWS region is correct. + +If the resource’s arguments differ from the imported state, the import will fail. You will receive this message: `warning: inputs to import do not match the existing resource; importing this resource will fail`. Select “details” in the `pulumi up` preview to learn what the differences are. If you try to proceed without correcting the inconsistencies, you will see this message: e`rror: inputs to import do not match the existing resource`. To fix these errors, make sure that your program computes a state that completely matches the resource to be imported. + +Because of auto-naming, it is common to run into this error when you import a resource’s name property. Unless you explicitly specify a name, Pulumi will auto-generate one, which is guaranteed not to match, because it will have a random hex suffix. To fix this problem, explicitly specify the resource’s name [as described here]){{< relref "/docs/intro/concepts/programming-model#autonaming" >}}. Note that, in the example for the EC2 security group, the name was specified by passing `web-sg-62a569b` as the resource’s name property. + +Once a resource is successfully imported, remove the import option because Pulumi is now managing the resource. + +#### parent + +This option specifies a parent for a resource. It is used to associate children with the parents that encapsulate or are responsible for them. Good examples of this are [component resources]){{< relref "/docs/intro/concepts/programming-model#components" >}}. The default behavior is to parent each resource to the implicitly-created `pulumi:pulumi:Stack` component resource that is a root resource for all Pulumi stacks. + +For example, this code creates two resources, a parent and child, the latter of which is a child to the former: + +```python +parent = MyResource("parent"); +child = MyResource("child", opts=ResourceOptions(parent=parent)); +``` + +Using parents can clarify causality or why a given resource was created in the first place. For example, this pulumi up output shows an AWS Virtual Private Cloud (VPC) with two subnets attached to it, and also shows that the VPC directly belongs to the implicit pulumi:pulumi:Stack resource: + +```bash +Previewing update (dev): + + Type Name Plan + pulumi:pulumi:Stack parent-demo-dev + + ├─ awsx:x:ec2:Vpc default-vpc-866580ff create + + │ ├─ awsx:x:ec2:Subnet default-vpc-866580ff-public-1 create + + │ └─ awsx:x:ec2:Subnet default-vpc-866580ff-public-0 create +``` + +#### protect + +The `protect` option marks a resource as protected. A protected resource cannot be deleted directly. Instead, you must first set `protect: false` and run `pulumi up`. Then you can delete the resource by removing the line of code or by running `pulumi destroy`. The default is to inherit this value from the parent resource, and `false` for resources without a parent. + +```python +db = Database("db", opts=ResourceOptions(protect=True)) +``` + +### provider + +The provider option sets a provider for the resource. See [Providers]){{< relref "/docs/intro/concepts/programming-model#providers" >}}. The default is to inherit this value from the parent resource, and to use the ambient provider specified by Pulumi configuration for resources without a parent. + +```pythong +provider = Provider("provider", region="us-west-2") +vpc = ec2.Vpc("vpc", opts=ResourceOptions(provider=provider)) +``` + +#### transformations + +The `transformations` option provides a list of transformations to apply to a resource and all of its children. This option is used to override or modify the inputs to the child resources of a component resource. One example is to use the option to add other resource options (such as `ignoreChanges` or `protect`). Another example is to modify an input property (such as adding to tags or changing a property that is not directly configurable). + +Each transformation is a callback that gets invoked by the Pulumi runtime. It receives the resource type, name, input properties, resource options, and the resource instance object itself. The callback returns a new set of resource input properties and resource options that will be used to construct the resource instead of the original values. + +This example looks for all VPC and Subnet resources inside of a component’s child hierarchy and adds an option to ignore any changes for tags properties (perhaps because we manage all VPC and Subnet tags outside of Pulumi): + +```python +def transformation(args: ResourceTransformationArgs): + if args.type_ == "aws:ec2/vpc:Vpc" or args.type_ == "aws:ec2/subnet:Subnet": + return ResourceTransformationResult( + props=args.props, + opts=ResourceOptions.merge(args.opts, ResourceOptions( + ignore_changes=["tags"], + ))) + +vpc = MyVpcComponent("vpc", opts=ResourceOptions(transformations=[transformation])) +``` + +Transformations can also be applied in bulk to multiple resources in a stack by using the `registerStackTransformation` function. + +#### version + +The `version` option specifies a provider version to use when operating on a resource. This version overrides the version information inferred from the current package. This option should be used rarely. + +```python +vpc = ec2.Vpc("vpc", opts=ResourceOptions(version="2.10.0")) +``` + +### Resource Getter Functions + +You can use the static `get` function, which is available on all resource types, to look up an existing resource’s ID. The `get` function is different from the `import` function. The difference is that, although the resulting resource object’s state will match the live state from an existing environment, the resource will not be managed by Pulumi. A resource read with the `get` function will never be updated or deleted by Pulumi during an update. + +You can use the `get` function to consume properties from a resource that was provisioned elsewhere. For example, this program reads an existing EC2 Security Group whose ID is `sg-0dfd33cdac25b1ec9` and uses the result as input to create an EC2 Instance that Pulumi will manage: + +```python +import pulumi_aws as aws + +group = aws.ec2.SecurityGroup.get('sg-0dfd33cdac25b1ec9') + +server = aws.ec2.Instance('web-server', + ami='ami-6869aa05', + instance_type='t2.micro', + security_groups=[group.name]) # reference the security group resource above +``` + +Importantly, Pulumi will never attempt to modify the security group in this example. It simply reads back the state from your currently configured cloud account and then uses it as input for the new EC2 Instance. + +## Component Resources + +A component resource is a logical grouping of resources. Components resources usually instantiate a set of related resources in their constructor, aggregate them as children, and create a larger, useful abstraction that encapsulates their implementation details. + +Here are a few examples of component resources: + +- A Vpc that automatically comes with built-in best practices. +- An AcmeCorpVirtualMachine that adheres to your company’s requirements, such as tagging. +- A KubernetesCluster that can create EKS, AKS, and GKE clusters, depending on the target. + +The implicit pulumi:pulumi:Stack resource is itself a component resource that contains all top-level resources in a program. + +### Authoring a New Component Resource + +To author a new component, either in a program or for a reusable library, create a subclass of [`ComponentResource`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.ComponentResource" >}}). Inside of its constructor, chain to the base constructor, passing its type string, name, arguments, and options. Also inside of its constructor, allocate any child resources, passing the [`parent`]({{< relref "/docs/intro/concepts/programming-model#parent" >}}) option as appropriate to ensure component resource children are parented correctly. + +Here’s a simple component example: + +```python +class MyComponent(pulumi.ComponentResource): + def __init__(self, name, opts = None): + super().__init__('pkg:index:MyComponent', name, None, opts) +``` + +Upon creating a new instance of MyComponent, the call to the base constructor (using `super/base`) registers the component resource instance with the Pulumi engine. This records the resource’s state and tracks it across program deployments so that you see diffs during updates just like with a regular resource (even though component resources have no provider logic associated with them). Since all resources must have a name, a component resource constructor should accept a name and pass it to super. + +If you wish to have full control over one of the custom resource’s lifecycle in your component resource—including running specific code when a resource has been updated or deleted—you should look into [`dynamic providers`]({{< re;relref "/docs/intro/concepts/programming-model#dynamicproviders" >}}). These let you create full-blown resource abstractions in your language of choice. + +A component resource must register a unique type name with the base constructor. In the example, the registration is `pkg:index:MyComponent`. To reduce the potential of other type name conflicts, this name contains the package and module name, in addition to the type: `::`. These names are namespaced alongside non-component resources, such as aws:lambda:Function. + +For more information about component resources, [see the Pulumi Components tutorial]({{< relref "/docs/tutorials/aws/s3-folder-component" >}}). + +### Creating Child Resources + +Component resources often contain child resources. The names of child resources are often derived from the component resources’s name to ensure uniqueness. For example, you might use the component resource’s name as a prefix. Also, when constructing a resource, children must be registered as such. To do this, pass the component resource itself as the parent option. + +This example demonstrates both the naming convention and how to designate the component resource as the parent: + +```python +bucket = s3.Bucket(f"{name}-bucket", + opts=pulumi.ResourceOptions(parent=self)) +``` + +### Registering Component Outputs + +Component resources can define their own output properties by using register_outputs . The Pulumi engine uses this information to display the logical outputs of the component resource and any changes to those outputs will be shown during an update. + +For example, this code registers an S3 bucket’s computed domain name, which won’t be known until the bucket is created: + +```python +self.register_outputs({ + bucketDnsName: bucket.bucketDomainName +}) +``` + +The call to `registerOutputs` typically happens at the very end of the component resource’s constructor. + +The call to `registerOutputs` also tells Pulumi that the resource is done registering children and should be considered fully constructed, so—although it’s not enforced—the best practice is to call it in all components even if no outputs need to be registered. + +### Inheriting Resource Providers + +One option all resources have is the ability to pass an [explicit resource provider]({{< relref "/docs/intro/concepts/programming-model#providers">}}) to supply explicit configuration settings. For instance, you may want to ensure that all AWS resources are created in a different region than the globally configured region. In the case of component resources, the challenge is that these providers must flow from parent to children. + +To allow this, component resources accept a `providers` option that custom resources don’t have. This value contains a map from the provider name to the explicit provider instance to use for the component resource. The map is used by a component resource to fetch the proper `provider` object to use for any child resources. This example overrides the globally configured AWS region and sets it to us-east-1. Note that `myk8s` is the name of the Kubernetes provider. + +```python +component = MyComponent('...', ResourceOptions(providers={ + 'aws': useast1, + 'kubernetes': myk8s, +})) +``` + +If a component resource is itself a child of another component resource, its set of providers is inherited from its parent by default. diff --git a/content/docs/intro/concepts/runtime-functions.md b/content/docs/intro/concepts/runtime-functions.md new file mode 100644 index 000000000000..38965193d2d4 --- /dev/null +++ b/content/docs/intro/concepts/runtime-functions.md @@ -0,0 +1,55 @@ +--- +title: "Runtime Functions" +meta_desc: An in depth look at Pulumi secrets and their usage. +menu: + intro: + parent: concepts + weight: 3 + +aliases: ["/docs/reference/secrets/"] +--- + +## Runtime Functions + +The Pulumi SDK library includes a number of helper functions for the following actions. + +- [Current Project or Stack]({{< relref "/docs/intro/concepts/programming-model#current-project-stack">}}): get information about the current deployment +- [Logging]({{< relref "/docs/intro/concepts/programming-model#logging" >}}): for diagnostics, warnings, or errors pertaining to the deployment +- [Serializing Lambdas]({{< relref "/docs/intro/concepts/programming-model#runtime" >}}): for turning JavaScript callbacks into data which can be used as application code + +## Getting the Current Project or Stack + +The [`get_project`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.get_project" >}}) and [`get_stack`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.get_stack" >}}) functions give you the currently deployed project and stack, respectively. This can be useful for naming or tagging resources. + +```python +project = pulumi.get_project() +stack = pulumi.get_stack() +``` + +## Logging + +The [`pulumi.debug [info] [warn] [error]`]({{> relref "/docs/reference/pkg/python/pulumi#logging" }}) functions allow you to log diagnostics, warnings, or errors with the Pulumi engine. These are displayed, alongside all other Pulumi output, in the CLI and in the Pulumi Console. They are also logged and stored in case you want to audit or diagnose a past event. Here are the error functions: + +```text +log.info("message") +log.info("message", resource) +log.debug("hidden by default") +log.warn("warning") +log.error("fatal error") +``` + +# Serializing Lambdas (Node.js only) + +You can create libraries and components that allow the caller to pass in JavaScript callbacks that are invoked at runtime. For example, you can create an AWS Lambda function or an Azure function by providing a JavaScript callback that serves as its implementation. + +> Note" Runtime code provided via callbacks are currently not supported in Python. See https://github.com/pulumi/pulumi/issues/1535. + +Libraries that use JavaScript callbacks as inputs that are provided as source text to resource construction—such as the Lambda that is created by the onObjectCreated function in the previous example—are built on top of the [`pulumi.runtime.serializeFunction`]({{< relref "/docs/reference/pkg/nodejs/pulumi/pulumi/runtime#serializeFunction" >}}) API, which takes a JavaScript Function object as input, and returns a Promise that contains the serialized form of that function. + +Here is what occurs when a function is serialized to text: steps: + +1. Any captured variables referenced by the function are evaluated when the function is serialized. +1. The values of those variables are serialized. +1.When the values are objects, all properties and prototype chains are serialized. When the values are functions, those functions are serialized by following these same steps. + +See [`Serializing Functions`]({{< relref "/docs/tutorials/aws/serializing-functions/" >}}) for more details. diff --git a/content/docs/intro/concepts/secrets.md b/content/docs/intro/concepts/secrets.md new file mode 100644 index 000000000000..e38a99a97420 --- /dev/null +++ b/content/docs/intro/concepts/secrets.md @@ -0,0 +1,61 @@ +--- +title: "Secrets" +meta_desc: An in depth look at Pulumi secrets and their usage. +menu: + intro: + parent: concepts + weight: 3 + +aliases: ["/docs/reference/secrets/"] +--- + +## Secrets + +All resource input and output values are recorded as [`state`]({{< relref "/docs/intro/concepts/state/" >}}), and are stored in the Pulumi Service, a file, or a pluggable provider that you choose. These raw values are usually just server names, configuration settings, and so on. In some cases, however, these values contain sensitive data, such as database passwords or service tokens. + +The Pulumi Service always transmits and stores entire state files securely; however, Pulumi also supports encrypting specific values as “secrets” for extra protection. Encryption ensures that these values never appear as plaintext in your state file. By default, the encryption method uses automatic, per-stack encryption keys provided by the Pulumi Service or you can use a [provider of your own choosing]({{< relref "/docs/intro/concepts/config#configuring-secrets-encryption" >}}) instead. + +To encrypt a configuration setting before runtime, you can use the CLI command [`config set`]({{< relref "/docs/intro/concepts/config#configuration" >}}) command with a [`--secret`]({{< relref "/docs/intro/concepts/config#secrets" >}}) flag. You can also set a secret during runtime. Any [`Output`]({{< relref "/docs/reference/pkg/python/pulumi/#outputs-and-inputs" >}}) value can be marked secret. If an output is a secret, any computed values derived from it—such as those derived through an [`apply`]({{< relref "/docs/reference/pkg/python/pulumi/#outputs-and-inputs" >}}) call —will also be marked secret. All these encrypted values are stored in your state file. + +An `Output`]({{< relref "/docs/reference/pkg/python/pulumi/#outputs-and-inputs" >}}) can be marked secret in a number of ways: + +- By reading a secret from configuration using [`Config.get_secret`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Config.get_secret" >}}) or [`Config.require_secret`]({{< relref "/docs/reference/pkg/python/pulumi/#pulumi.Config.require_secret" >}}). +- By creating a new secret value with [`Output.secret`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Output.secret" >}}), such as when generating a new random password. +- By marking a resource as having secret properties using [`additionalSecretOutputs`]({{< relref "/docs/intro/concepts/programming-model#additionalsecretoutputs" >}}). +- By computing a secret value by using [`apply`]({{< relref "/docs/reference/pkg/python/pulumi/#outputs-and-inputs" >}}) or [`Output.all`]({{< relref "/reference/pkg/python/pulumi#pulumi.Output.all" >}}) with another secret value. + +As soon as an `Output` is marked secret, the Pulumi engine will encrypt it wherever it is stored. + +> Note: Inside of an `apply` or `Output.all` call, your secret is decrypted into plaintext for use within the callback. It is up to your program to treat this value sensitively and only pass the plaintext value to code that you trust. + +## Programmatically Creating Secrets + +There are two ways to programmatically create secret values: + +- By using [`get_secret`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Config.get_secret" >}}) or [`require_secret`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Config.require_secret" >}}) when reading a value from config. +- By calling [`Output.secret`]({{< relref "/docs/reference/pkg/python/pulumi#pulumi.Output.secret" >}}) to construct a secret from an existing value. + +As an example, let’s create an AWS Parameter Store secure value. Parameter Store is an AWS service that stores strings. Those strings can either be secret or not. To create an encrypted value, we need to pass an argument to initialize the store’s `value` property. Unfortunately, the obvious thing to do —passing a raw, unencrypted value— means that the value is also stored in the Pulumi state, unencrypted We need toe ensure that the value is a secret: + +```python +cfg = pulumi.Config() +param = ssm.Parameter("a-secret-param", + type="SecureString", + value=cfg.require_secret("my-secret-value")) +``` + +The Parameter resource’s value property is encrypted in the Pulumi state file. + +Pulumi tracks the transitive use of secrets, so that your secret won’t end up accidentally leaking into the state file. Tracking includes automatically marking data generated from secret inputs as secret themselves, as well as fully encrypting any resource properties that include secrets in them. + +## How Secrets Relate to Outputs + +Secrets have the same type, `Output`, as do unencrypted resource outputs. The difference is that they are marked internally as needing encryption before persisting in the state file. When you combine an existing output that is marked as a secret using `apply` or `Output.all` , the resulting output is also marked as a secret. + +An `apply`’s callback is given the plaintext value of the underlying secret. Although Pulumi ensures that the value returned from an `apply` on a secret is also marked as secret, Pulumi cannot guarantee that the `apply` callback itself will not expose the secret value —for instance, by explicitly printing the value to the console or saving it to a file. Be careful that you do not pass this plaintext value to code that might expose it. + +Unlike regular outputs, secrets cannot be captured by the Pulumi closure serialization system for use in serverless code and doing so leads to an exception. We do plan to support this once we can ensure that the values will be persisted securely. See [pulumi/pulumi#2718](https://github.com/pulumi/pulumi/issues/2718). + +## Explicitly Marking Resource Outputs as Secrets + +It is possible to mark resource outputs as containing secrets. In this case, Pulumi will automatically treat those outputs as secrets and encrypt them in the state file and anywhere they flow to. To do so, use the [`additional secret outputs`]({{< relref "/docs/intro/concepts/programming-model#additionalsecretoutputs" >}}) option, as described above. diff --git a/content/docs/intro/concepts/stack-outputs.png b/content/docs/intro/concepts/stack-outputs.png new file mode 100644 index 000000000000..60aa89acaf01 Binary files /dev/null and b/content/docs/intro/concepts/stack-outputs.png differ diff --git a/content/docs/intro/concepts/stack.md b/content/docs/intro/concepts/stack.md index 1cdded980988..0b70585d4e4a 100644 --- a/content/docs/intro/concepts/stack.md +++ b/content/docs/intro/concepts/stack.md @@ -9,8 +9,15 @@ menu: aliases: ["/docs/reference/stack/"] --- -Every Pulumi program is deployed to a **stack**. A stack is an isolated, independently [configurable]({{< relref "config" >}}) -instance of a Pulumi program. Stacks are commonly used to denote different phases of development (such as **development**, **staging** and **production**) or feature branches (such as **feature-x-dev**, **jane-feature-x-dev**). +## Stacks + +A stack is a collection of cloud resources that is managed by a single Pulumi program. They are analogous to environments. For example, if you’ve ever created infrastructure as code, you may have created, for example, different environments for development, test, production. + +A project can have as many stacks as you need. By default, Pulumi creates a stack for you when you start a new project. The stack recognizes the current directory as the default name for your program. + +Each stack in a project has its own configuration file. A stack configuration file is named Pulumi.name.yaml. For example, the name of the dev stack configuration file is Pulumi.dev.yaml. + +Every Pulumi program is deployed to a **stack**. A stack is an isolated, independently [configurable]({{< relref "config" >}}) instance of a Pulumi program. Stacks are commonly used to denote different phases of development (such as **development**, **staging** and **production**) or feature branches (such as **feature-x-dev**, **jane-feature-x-dev**). ## Create a stack {#create-stack} @@ -102,9 +109,11 @@ Current stack outputs (2): Use `pulumi stack select` to change stack; `pulumi stack ls` lists known ones ``` -## View stack outputs {#outputs} +## Stack Outputs {#outputs} -When you use top-level exports in your Pulumi [program]({{< relref "programming-model#programs" >}}), they become [stack outputs]({{< relref "programming-model#stack-outputs" >}}). Stack outputs can be viewed via `pulumi stack output` and are shown on the stack information page on pulumi.com. +A stack can export values as [stack outputs]({{< relref "/docs/intro/concepts/stack#outputs">}}). These outputs are shown during an update, can be easily retrieved with the Pulumi CLI, and are displayed in the Pulumi Console. They can be used for important values like resource IDs and computed IP addresses and DNS names. + +Stack outputs can be viewed via `pulumi stack output` and are shown on the stack information page on app.pulumi.com. ### **JavaScript code** @@ -130,7 +139,40 @@ $ pulumi stack output publicIp 18.218.85.197 ``` -See also [Inter-Stack Dependencies]({{< relref "organizing-stacks-projects#inter-stack-dependencies" >}}), which allow one stack to reference the outputs of another stack. +The right-hand side of a stack export can be a regular value, an Output, or a Promise (effectively, a Promise is the same as an Input). The actual values are resolved at the end of `pulumi up`. + +Stack exports are effectively JSON serialized, though quotes are removed when exporting strings. +For example, this program: + +```typescript +pulumi.export("x", "hello") +pulumi.export("o", {'num': 42}) +``` + +produces the following stack outputs: + +```bash +$ pulumi stack output x +hello +$ pulumi stack output o +{"num": 42} +``` + +The full set of outputs can be rendered as JSON by using `pulumi stack output --json`: + +```bash +$ pulumi stack output --json +{ + "x": "hello", + "o": { + "num": 42 + } +} +``` + +> Note: If you export an actual resource, it too will be JSON serialized. This usually isn’t what you want, especially because some resources are quite large. If you only want to export the resource’s ID or name, for example, just export those properties directly. + +Stack outputs respect secret annotations and are encrypted appropriately. If a stack contains any secret values, their plaintext values will not be shown by default. Instead, they will be displayed as [secret]({{< relref "" >}}) in the CLI. Pass `--show-secrets` to `pulumi stack output` to see the plaintext value. ## Import and export a stack deployment @@ -163,3 +205,17 @@ Custom tags can be assigned to a stack by running [`pulumi stack tag set > **Note:** As a best practice, custom tags should not be prefixed with `pulumi:`, `gitHub:`, or `vcs:` to avoid conflicting with built-in tags that are assigned and updated with fresh values each time a stack is updated. Tags can be deleted by running [`pulumi stack tag rm `]({{< relref "/docs/reference/cli/pulumi_stack_tag_rm" >}}). + + +## Stack References + +Stack references allow you to access the outputs of one stack from another stack. [Inter-Stack Dependencies]({{< relref "organizing-stacks-projects#inter-stack-dependencies" >}}) allow one stack to reference the outputs of another stack. To reference values from another stack, create an instance of the StackReference type using the fully qualified name of the stack as an input, and then read exported stack outputs by their name: + +```typescript +from pulumi import StackReference + +other = StackReference(f"acmecorp/infra/other") +other_output = other.get_output("x"); +``` + +Stack names must be fully qualified, including the organization, project, and stack name components, in the format `//`. For individual accounts, use your account name for the organization component.