Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--target leads to "snapshot integrity failure; refusing to use it" error #15959

Closed
blampe opened this issue Apr 17, 2024 · 1 comment · Fixed by #16247
Closed

--target leads to "snapshot integrity failure; refusing to use it" error #15959

blampe opened this issue Apr 17, 2024 · 1 comment · Fixed by #16247
Assignees
Labels
impact/snapshot-integrity Snapshot integrity failure kind/bug Some behavior is incorrect or out of spec p1 A bug severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed
Milestone

Comments

@blampe
Copy link
Contributor

blampe commented Apr 17, 2024

What happened?

I ran pulumi up --target 'urn:...::aws:ecs/service:Service::ratelimit-' --target-dependents.

The preview correctly identified the resource as being deleted, but the actual up returned an error for unrelated resources:

error: could not deserialize deployment: snapshot integrity failure; refusing to use it: resource urn:...::aws:ecs/taskDefinition:TaskDefinition::consoleTaskDefinition’s dependency urn:...::pulumi-service:infrastructure:Service$docker:buildx/image:Index::consoleImage comes after it

Subsequent ups had the same error.

I was eventually able to get past this with up --disable-integrity-checking -- once that succeeded everything worked as normal again.

Example

Steps:

Output of pulumi about

CLI
Version 3.113.0
Go Version go1.22.2
Go Compiler gc

Host
OS darwin
Version 14.1
Arch arm64

Backend
Name pulumi.com
URL https://app.pulumi.com/bryce-pulumi-corp
User bryce-pulumi-corp
Organizations bryce-pulumi-corp, pulumi
Token type personal

Additional context

Both resources have retainOnDelete set.

The targeted resource was expected to be deleted, because I had replaced it with a different type/provider in my program.

I exported a state file shortly before running the update with --target, so it might be possible to repo using that. Ping me and I can send it over.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@blampe blampe added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Apr 17, 2024
@justinvp justinvp added impact/snapshot-integrity Snapshot integrity failure p1 A bug severe enough to be the next item assigned to an engineer and removed needs-triage Needs attention from the triage team labels Apr 18, 2024
@justinvp justinvp added this to the 0.103 milestone Apr 19, 2024
@1oglop1
Copy link

1oglop1 commented May 1, 2024

I have the same problem, but different resources

const users = ['u1', 'u2']
const iamUsers = usersNames.map( u => new aws.iam.User(u, {name: u}))

// note this is not UserPolicy because it has already been attached
new aws.iam.PolicyAttachment(
  "self-management",
  {
    policyArn: selfManagePolicy.arn,
    users: iamUsers.map((u) => u.name),
  },
  {
    provider: awsFrankfurt,
  }
);

pulumi up --target '**u1**' --target-dependents won't update self-management

CLI          
Version      3.112.0
Go Version   go1.22.1
Go Compiler  gc

Plugins
NAME     VERSION
aws      6.31.0
command  0.10.0
nodejs   unknown

@justinvp justinvp modified the milestones: 0.103, 0.104 May 6, 2024
@justinvp justinvp assigned Frassle and unassigned PollRobots May 17, 2024
@justinvp justinvp modified the milestones: 0.104, 0.105 May 20, 2024
github-merge-queue bot pushed a commit that referenced this issue May 23, 2024
When using `--target` to target specific resources during an update, we
use the list of targets to decide which steps to generate given a set of
resource registrations. Specifically:

* If the registration event names a resource that is targeted, we
process it as usual.
* If the registration event names a resource that _is not_ targeted, we
emit a `SameStep` for it.

In the latter case, the emission of a `SameStep` means that the old
state for the resource will be copied across to the new state. This is
the desired behaviour -- the resource was not targeted and so the new
state should contain the resource exactly as it was prior to the update.
However, this presents a problem if the old state has references to
resources that either will not appear in the new state, or will appear
in the wrong place. Consider the following program in TypeScript-esque
pseudocode:

```typescript
const a = new Resource("a")
const b = new Resource("b", { dependency: a })
const c = new Resource("c")
```

Here, `b` depends on `a`, while `a` and `c` have no dependencies. We run
this program without specifying targets and obtain a state containing
`a`, `b` and `c`, with `a` appearing before `b` due to `b`'s dependency
on `a`. We now modify the program as follows:

```typescript
const b = new Resource("b")
const c = new Resource("c")
```

`a` has been removed from the program and consequently `b` no longer
depends on it. We once more run the program, this time with a `--target`
of `c`. That is to say, neither `a` nor `b` is targeted. The execution
proceeds as follows:

* `a` is not in the program, so no `RegisterResourceEvent` will be
emitted and processed for it.
* `b` is in the program, but it is not targeted. Its
`RegisterResourceEvent` will be turned into a `SameStep` and `b`'s _old
state will be copied as-is to the new state_.
* `c` is in the program and is targeted. It will be processed as normal.

At the end of execution when we come to write the snapshot, we take the
following actions:

* We first write the processed resources: `b`'s old state and `c`'s new
state.
* We then copy over any unprocessed resources from the base (previous)
snapshot. This includes `a` (which is again desirable since its deletion
should not be processed due to it not being targeted).

Our snapshot is now not topologically sorted and thus invalid: `b` has a
dependency on `a`, but `a` appears after it. Presently this bug will
manifest irrespective of the nature of the dependency: `.Dependencies`,
`.PropertyDependencies` and `.DeletedWith` are all affected.

This commit fixes this issue by traversing all untargeted resource
dependency relationships and ensuring that `SameStep`s (or better if
they have been targeted) are emitted before emitting the depending
resource's `SameStep`.

* Fixes #16052 
* Fixes #15959
@pulumi-bot pulumi-bot added the resolution/fixed This issue was fixed label May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
impact/snapshot-integrity Snapshot integrity failure kind/bug Some behavior is incorrect or out of spec p1 A bug severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants