Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 1 addition & 58 deletions content/docs/iac/troubleshooting/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ We take great pride in service uptime and work rapidly to fix service interrupti

### Post-step event returned an error {#post-step-event}

If an I/O error occurs after "post-step event returned an error", you can safely re-start your update. If you see "after mutation of snapshot", you have hit a bug in Pulumi. You will possibly need to do some [manual intervention to repair your stack](#editing-your-deployment).
If an I/O error occurs after "post-step event returned an error", you can safely re-start your update. If you see "after mutation of snapshot", you have hit a bug in Pulumi. You will possibly need to do some [manual intervention to repair your stack](/docs/iac/troubleshooting/editing-state-files).

The Pulumi engine runs a small amount of code after every "step" that it performs. If this code fails for any reason, it will fail the entire update. One of the things that the Pulumi engine does before and after every step is a self-check on its internal data structures to ensure that they are in a consistent state. If they are not, Pulumi will issue an error and fail the deployment.

Expand Down Expand Up @@ -247,60 +247,3 @@ If `pulumi cancel` fails with `error: [400] Bad Request: the update has already
Then run `pulumi refresh` to remove any pending operations cleanly, allowing you to resolve any pending operations that Pulumi could not fix unaided.

At this point your stack should be valid, up-to-date, and ready to accept future updates.

## Manually editing your deployment {#editing-your-deployment}

Sometimes the only recourse for fixing a stack that is unable to complete deployments is to edit the deployment directly. We would love to hear about the issues you are experiencing that you can't resolve, both so we can assist you in fixing your stack and also to fix the issues in Pulumi that made it impossible for you to recover your stack in any other way.

The Pulumi engine uses both your program and your stack's existing state to make decisions about what resources to create, read, update, or delete. The most common problem that makes it impossible to make changes to your stack is that the stack's state has been corrupted in some way. There are a variety of ways that a stack's state could be corrupted, but in almost all cases it is possible to manually edit the stack's state to fix the issue.

This is an advanced operation and should be an absolute last resort. We recommend you check in with the [Pulumi Community Slack](https://slack.pulumi.com) first before editing your snapshot.

If you intend to unprotect or delete a resource, consider using the [`pulumi state`](/docs/cli/commands/pulumi_state) command instead of editing your state directly. `pulumi state` makes fixes to your state without requiring you to edit the JSON representation of your stack's current state.

To get a JSON representation of your stack's current state, export your current stack to a file:

```bash
$ pulumi stack export --file state.json
```

This file contains a lot of information. At the top level, this JSON object has two fields:

| Field | Description |
| - | - |
| `version` | The version of the file format. This should not be changed. |
| `deployment` | The last deployment state of the stack. |

The `deployment` object itself has three fields:

| Field | Description |
| - | - |
| `manifest` | Metadata about the previous deployment. This should not be changed. |
| `pending_operations` | List of the operations that the Pulumi engine started but has not finished. |
| `resources` | List of resources that Pulumi knows about. |

The possible fields of a resource are:

| Field | Description |
| - | - |
| `urn` | The resource URN, which is a Pulumi-specific universal resource identifier. |
| `custom` | A boolean indicating whether or not this resource is a `custom` resource, which means that it uses a resource provider to operate. Component resources are not `custom`. |
| `delete` | A boolean indicating whether or not this resource is pending deletion. |
| `id` | This resource's ID, which is a provider-specific resource identifier. This often corresponds to a cloud provider's identifier for a resource. |
| `type` | The Pulumi type of this resource. |
| `inputs` | A map of inputs for this resource. Inputs are the set of key-value pairs used as an input to a resource provider. |
| `outputs` | A map of outputs for this resource. Outputs are the set of key-value pairs that were given to Pulumi by a resource provider after a resource has been provisioned. |
| `parent` | A URN for this resource's parent resource. |
| `protect` | A boolean indicating whether or not this resource is protected. Protected resources can not be deleted. |
| `external` | A boolean indicating whether or not this resource is external to Pulumi. If a resource is external, Pulumi does not own its life cycle and it will not ever delete or update the resource. Resources that are read using the `get` function are external. |
| `dependencies` | A list of URNs indicating the resources that this resource depends on. Pulumi tracks dependencies between resources. It is important that this list be the full list of resources upon which this resource depends. |
| `initErrors` | A list of errors that occurred that prevented the resource from initializing. Some resource providers (most notably Kubernetes) populate this field to indicate that a resource was created but failed to initialize. |
| `provider` | Reference to the provider responsible for the resource. |

The `resources` field is a list, not a set. The order of resources in the list is important and is enforced by the Pulumi engine. Resources in a deployment must be in *dependency order* - if resource A depends on resource B, resource A *must* appear after resource B in the list.

Import your changes by running:

```bash
$ pulumi stack import --file state.json
```
141 changes: 141 additions & 0 deletions content/docs/iac/troubleshooting/editing-state-files.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
---
title_tag: "Editing Pulumi State Files"
meta_desc: "Learn how to safely edit Pulumi state files when necessary"
title: Editing state files
h1: Editing state files
meta_image: /images/docs/meta-images/docs-meta.png
menu:
iac:
name: Editing state Files
parent: iac-troubleshooting
weight: 30
---

The Pulumi IaC engine computes the difference between your program's code (desired state) and your stack's state file (current state) to make decisions about what resources to create, read, update, or delete, and in what order. In some cases, you may need to perform edits to your state file, either via the Pulumi CLI (preferred) or by editing the state file in a text editor (as a last resort).

This page is a guide on how to edit your state file as safely as possible.

## When you might need to edit your state file

You might need to edit your state file in the following situations:

- You want to move resources between stacks in the course of refactoring your Pulumi codebase(s)
- You need to unprotect resources from deletion
- A Pulumi command fails with an error indicating a corrupt state, for example if you see [an I/O error with the text `after mutation of snapshot`](/docs/iac/troubleshooting#post-step-event), which can occur in rare scenarios like a network partition during a state file update.

## What to try before editing your state file

Before manually editing your state file, consider these troubleshooting steps:

1. Run the `pulumi refresh` command.
1. Update to the latest version of the Pulumi CLI ([installation instructions](/docs/iac/download-install/)) and attempt your operation again.
1. If a `pulumi update` failed and a resource was created and shows in your cloud console, but Pulumi is attempting to create the resource again, use the [`pulumi import`](/docs/iac/cli/commands/pulumi_import) command instead of editing your state file.
1. If you have recently updated the Pulumi CLI, consider downgrading back to the previous known-good version and attempt your operation again.
1. Update the Pulumi SDK to the version that matches the Pulumi CLI.
1. If the problem seems to be related to a particular resource type or provider, consider updating your provider dependencies (do this work on a separate git branch so you can easily undo any changes to your Pulumi code).

These steps don’t always resolve the issue in every case, but they often help and are worth trying before manually editing your state file.

{{% notes type="info" %}}
If your Pulumi issue is causing a serious outage for your workload(s), and Pulumi CLI operations are taking too long, consider the `--target` option, which allows you to limit the refresh operation only to the resources you specify. (The flag can be specified more than once.)

The following commands support the `--target` option:

- `pulumi refresh`
- `pulumi preview`
- `pulumi up`
- `pulumi destroy`
{{% /notes %}}

## How to edit your state file safely

If you have determined that it is appropriate and necessary to edit your state file, follow these steps to do so safely:

{{% notes type="info" %}}
Ensure that your team members do not attempt to make updates to your stack while you are editing your state file, e.g., by communicating your intent over a shared chat channel. If someone performs a Pulumi operation that writes to your state file in while you are in the process of editing it, important data loss may occur: either your changes or your teammate's risk being overwritten.
{{% /notes %}}

### 1. Save a backup of your state file

First, you should take a backup of your state file to ensure that any changes you make can be easily reversed. To take a backup of your state file:

```bash
pulumi stack export --file pulumi-state-backup.json
```

If you want to undo the changes you make in subsequent steps, you can restore your state file from backup:

```bash
pulumi stack import --file pulumi-state-backup.json
```

### 2. Try targeted fixes with the `pulumi state` command

The [`pulumi state`](/docs/iac/cli/commands/pulumi_state) command allows you to make targeted, surgical changes to your state file without the risk of exposing your entire state file in an editor for hand-editing, which can cause additional errors.

The `pulumi state` command can help with the following scenarios:

- Automatically repairing your state file with [`pulumi state repair`](/docs/iac/cli/commands/pulumi_state_repair)
- Deleting resources from your state file with [`pulumi state delete`](/docs/iac/cli/commands/pulumi_state_delete)
- Moving resources between stacks with [`pulumi state move`](/docs/iac/cli/commands/pulumi_state_move)
- Unprotecting resources from deletion with [`pulumi state unprotect`](/docs/iac/cli/commands/pulumi_state_unprotect)
- Targeting resources for recreation with [`pulumi state taint`](/docs/iac/cli/commands/pulumi_state_taint)

Refer to [the `pulumi state` reference docs](/docs/iac/cli/commands/pulumi_state) for a complete list of capabilities.

### 3. If necessary, export your state file and edit

If the `pulumi state` command does not resolve the issue for you, you will need to edit your Pulumi state file in an editor to resolve the issue. First, export the state file again:

```bash
pulumi stack export --file state.json
```

{{% notes type="warning" %}}
The recommendation to export your state file a second time with a different filename from the backup you created earlier is intentional!

Do not forget to export an additional copy of your Pulumi state file. Failing to do so may make it difficult or impossible to undo your changes once you save your state file to your Pulumi state backend.
{{% /notes %}}

Then, perform any manual edits using the [State File reference](#state-file-reference) as a guide. When your edits are complete, import the edited state file with the following command:

```bash
pulumi stack import --file state.json
```

## State file reference

Your state file is represented as a JSON object. At the top level, this JSON object has two fields:

| Field | Description |
| - | - |
| `version` | The version of the file format. This should not be changed. |
| `deployment` | The last deployment state of the stack. |

The `deployment` object has the following fields:

| Field | Description |
| - | - |
| `manifest` | Metadata about the previous deployment. This should not be changed. |
| `pending_operations` | List of the operations that the Pulumi engine started but has not finished. |
| `resources` | List of resources that Pulumi knows about. |

The `resources` field is a list, not a set: The order of resources in the list is important and is enforced by the Pulumi engine. Resources in a deployment must be in *dependency order* - if resource A depends on resource B, resource A *must* appear after resource B in the list.

The possible fields of an entry in `resources` are:

| Field | Description |
| - | - |
| `urn` | The resource URN, which is a Pulumi-specific universal resource identifier. |
| `custom` | A boolean indicating whether or not this resource is a `custom` resource, which means that it uses a resource provider to operate. Component resources are not `custom`. |
| `delete` | A boolean indicating whether or not this resource is pending deletion. |
| `id` | This resource's ID, which is a provider-specific resource identifier. This often corresponds to a cloud provider's identifier for a resource. |
| `type` | The Pulumi type of this resource. |
| `inputs` | A map of inputs for this resource. Inputs are the set of key-value pairs used as an input to a resource provider. |
| `outputs` | A map of outputs for this resource. Outputs are the set of key-value pairs that were given to Pulumi by a resource provider after a resource has been provisioned. |
| `parent` | A URN for this resource's parent resource. |
| `protect` | A boolean indicating whether or not this resource is protected. Protected resources can not be deleted. |
| `external` | A boolean indicating whether or not this resource is external to Pulumi. If a resource is external, Pulumi does not own its life cycle and it will not ever delete or update the resource. Resources that are read using the `get` function are external. |
| `dependencies` | A list of URNs indicating the resources that this resource depends on. Pulumi tracks dependencies between resources. It is important that this list be the full list of resources upon which this resource depends. |
| `initErrors` | A list of errors that occurred that prevented the resource from initializing. Some resource providers (most notably Kubernetes) populate this field to indicate that a resource was created but failed to initialize. |
| `provider` | Reference to the provider responsible for the resource. |
Loading