Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Received panic on a failed assertion - "expected mutation to be successful" #13923

Closed
lhaskins opened this issue Sep 11, 2023 · 2 comments
Closed

Received panic on a failed assertion - "expected mutation to be successful" #13923

lhaskins opened this issue Sep 11, 2023 · 2 comments
Assignees
@Frassle
Labels
area/engine Pulumi engine impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec p1 Bugs severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed
Milestone
0.95

Comments

@lhaskins
Copy link

lhaskins commented Sep 11, 2023 •
edited by justinvp

What happened?

While performing a pulumi up on one of our stacks, we received the following error twice. Each time, a new bash window was started and pulumi cancel executed to clear the panic-ed attempt. On the 3rd attempt, the command was successful (meaning, no panic was received).

The action was to create a RDS aurora postgresql clone from a snapshot.


     Type                 Name             Status      Info
     pulumi:pulumi:Stack  ameff-wjob-prod  running..   read pulumi:pulumi:StackReference modern-energy/ameff-db/prod
panic: fatal: An assertion has failed: expected mutation to be successful

goroutine 2134 [running]:
github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)
/private/tmp/pulumi-20230811-5126-1glc60o/sdk/go/common/util/contract/failfast.go:23
github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf(0x19?, {0x1020a6cd5?, 0xc0010caf20?}, {0x0?, 0x10204f75a?, 0xc00063b8f0?})
/private/tmp/pulumi-20230811-5126-1glc60o/sdk/go/common/util/contract/assert.go:35 +0xed
github.com/pulumi/pulumi/pkg/v3/backend.(*sameSnapshotMutation).End(0xc002948fc8, {0x102654f30, 0xc0011dde60}, 0x0)
/private/tmp/pulumi-20230811-5126-1glc60o/pkg/backend/snapshot.go:289 +0x169
github.com/pulumi/pulumi/pkg/v3/engine.(*updateActions).OnResourceStepPost(0xc002487b80, {0x101cf2b00?, 0xc002948fc8}, {0x102654f30, 0xc0011dde60}, 0x0, {0x1026297e0?, 0xc001e08350})
/private/tmp/pulumi-20230811-5126-1glc60o/pkg/engine/update.go:614 +0x7f9
github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*stepExecutor).executeStep(0xc0017f62a0, 0x21?, {0x102654f30, 0xc0011dde60})
/private/tmp/pulumi-20230811-5126-1glc60o/pkg/resource/deploy/step_executor.go:376 +0x694
github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*stepExecutor).executeChain(0xc0017f62a0, 0xc001764080?, {0xc001cebb30?, 0x1, 0x0?})
/private/tmp/pulumi-20230811-5126-1glc60o/pkg/resource/deploy/step_executor.go:251 +0xe5
github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*stepExecutor).worker.func1()
/private/tmp/pulumi-20230811-5126-1glc60o/pkg/resource/deploy/step_executor.go:451 +0xd9
created by github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*stepExecutor).worker
/private/tmp/pulumi-20230811-5126-1glc60o/pkg/resource/deploy/step_executor.go:448 +0x2b1

Expected Behavior

I would have expected one of 2 things:

  • The database cluster should have been created without a panic or
  • a cleaner failure to have occurred that did not cause my bash window to become unusable without a reset

Steps to reproduce

The following is the output from the preview before the update was confirmed (with some details redacted):

$ pulumi up
Previewing update (modern-energy/prod)

View in Browser (Ctrl+O): https://app.pulumi.com/modern-energy/ameff-wjob/prod/previews/cac2ac38-de9a-44f5-bdc9-c064b312378c

     Type                                      Name                                           Plan        Info
     pulumi:pulumi:Stack                       [STACK_NAME]                                            
 +-  ├─ aws:ec2:SecurityGroupRule              [STACK_NAME]-db-ingress                     replace     [diff: ~securityGroupId]
 ~   ├─ aws:ssm:Parameter                      [STACK_NAME]-db-secret-celery_result_url    update      
 ~   ├─ aws:ssm:Parameter                      [STACK_NAME]-db-secret-w_db_root            update      
 ~   ├─ aws:ssm:Parameter                      [STACK_NAME]-db-secret-w_db_url             update      [diff: ~value]
 ~   ├─ aws:ssm:Parameter                      [STACK_NAME]-db-secret-w_db_url_migrate     update      [diff: ~value]
 ~   ├─ aws:iam:Policy                         wjob-s3-access                                 update      [diff: ~policy]
     └─ util:ECSService                        [STACK_NAME]-service-wjob                               
        ├─ aws:alb:LoadBalancer                [STACK_NAME]-service-wjob                               
 +      │  ├─ aws:alb:TargetGroup              [STACK_NAME]-service-wjob                   create      
 +      │  ├─ aws:alb:Listener                 [STACK_NAME]-service-wjob-http-redirect     create      
 +      │  ├─ aws:alb:Listener                 [STACK_NAME]-service-wjob-https             create      
 +      │  └─ aws:route53:Record               [STACK_NAME]-service-wjob                   create      
        ├─ aws:ecs:Cluster                     [STACK_NAME]-service-wjob                               
        │  ├─ util:IAMRole                     [STACK_NAME]-wjob-web-execution                           
 +      │  │  ├─ aws:iam:Role                  [STACK_NAME]-wjob-web-execution-role          create      
 +      │  │  ├─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-web-execution-attachment-0  create      
 +      │  │  └─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-web-execution-attachment-1  create      
        │  ├─ util:IAMRole                     [STACK_NAME]-wjob-web-task                              
 +      │  │  ├─ aws:iam:Role                  [STACK_NAME]-wjob-web-task-role             create      
 +      │  │  └─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-web-task-attachment-0     create      
        │  ├─ util:IAMRole                     [STACK_NAME]-wjob-migrate-task                           
 +      │  │  ├─ aws:iam:Role                  [STACK_NAME]-wjob-migrate-task-role          create      
 +      │  │  └─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-migrate-task-attachment-0  create      
        │  ├─ util:IAMRole                     [STACK_NAME]-wjob-queue-monitor-task                           
 +      │  │  ├─ aws:iam:Role                  [STACK_NAME]-wjob-queue-monitor-task-role          create      
 +      │  │  └─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-queue-monitor-task-attachment-0  create      
        │  ├─ util:IAMRole                     [STACK_NAME]-wjob-beat-task                             
 +      │  │  ├─ aws:iam:Role                  [STACK_NAME]-wjob-beat-task-role            create      
 +      │  │  └─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-beat-task-attachment-0    create      
        │  ├─ util:IAMRole                     [STACK_NAME]-wjob-worker-execution                            
 +      │  │  ├─ aws:iam:Role                  [STACK_NAME]-wjob-worker-execution-role           create      
 +      │  │  ├─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-worker-execution-attachment-0   create      
 +      │  │  └─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-worker-execution-attachment-1   create      
        │  ├─ util:IAMRole                     [STACK_NAME]-wjob-beat-execution                           
 +      │  │  ├─ aws:iam:Role                  [STACK_NAME]-wjob-beat-execution-role          create      
 +      │  │  ├─ aws:iam:Policy                [STACK_NAME]-wjob-beat-execution-policy        create      
 +      │  │  ├─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-beat-execution-attachment-0  create      
 +      │  │  └─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-beat-execution-attachment-1  create      
        │  ├─ util:IAMRole                     [STACK_NAME]-wjob-worker-task                           
 +      │  │  ├─ aws:iam:Role                  [STACK_NAME]-wjob-worker-task-role          create      
 +      │  │  └─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-worker-task-attachment-0  create      
        │  ├─ util:IAMRole                     [STACK_NAME]-wjob-queue-monitor-execution                           
 +      │  │  ├─ aws:iam:Role                  [STACK_NAME]-wjob-queue-monitor-execution-role          create      
 +      │  │  ├─ aws:iam:Policy                [STACK_NAME]-wjob-queue-monitor-execution-policy        create      
 +      │  │  ├─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-queue-monitor-execution-attachment-0  create      
 +      │  │  └─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-queue-monitor-execution-attachment-1  create      
        │  ├─ util:IAMRole                     [STACK_NAME]-wjob-migrate-execution                           
 +      │  │  ├─ aws:iam:Role                  [STACK_NAME]-wjob-migrate-execution-role          create      
 +      │  │  ├─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-migrate-execution-attachment-0  create      
 +      │  │  └─ aws:iam:RolePolicyAttachment  [STACK_NAME]-wjob-migrate-execution-attachment-1  create      
 +      │  ├─ aws:ecs:TaskDefinition           [STACK_NAME]-wjob-web                       create      
 +      │  │  └─ aws:ecs:Service               [STACK_NAME]-wjob-web                       create      
 +      │  ├─ aws:ecs:TaskDefinition           [STACK_NAME]-wjob-worker                    create      
 +      │  │  └─ aws:ecs:Service               [STACK_NAME]-wjob-worker                    create      
 +      │  ├─ aws:ecs:TaskDefinition           [STACK_NAME]-wjob-beat                      create      
 +      │  │  └─ aws:ecs:Service               [STACK_NAME]-wjob-beat                      create      
 +      │  ├─ aws:ecs:TaskDefinition           [STACK_NAME]-wjob-migrate                   create      
 +      │  └─ aws:ecs:TaskDefinition           [STACK_NAME]-wjob-queue-monitor             create      
 +      │     └─ aws:ecs:Service               [STACK_NAME]-wjob-queue-monitor             create      
 +      ├─ command:local:Command               [STACK_NAME]-run-migration-task             create      
 +      └─ command:local:Command               [STACK_NAME]-await-migration-task           create      


Resources:
    + 42 to create
    ~ 5 to update
    +-1 to replace
    48 changes. 40 unchanged

Output of pulumi about

CLI
Version 3.81.0
Go Version go1.21.1
Go Compiler gc

Plugins
NAME VERSION
aws 5.11.0
aws-tags 0.4.1
command 0.8.2
local-exec 0.3.1
python unknown
random 4.2.0
utils-py 0.5.6

Host
OS darwin
Version 12.3
Arch x86_64

This project is written in python: executable='/usr/local/bin/python3' version='3.11.5
'

TYPE URN
pulumi:providers:aws urn:pulumi:<STACK_INFO>::pulumi:providers:aws::default
pulumi:providers:aws urn:pulumi:<STACK_INFO>::pulumi:providers:aws::default_5_11_0
pulumi:providers:random urn:pulumi:<STACK_INFO>::pulumi:providers:random::default_4_2_0
pulumi:providers:local-exec urn:pulumi:<STACK_INFO>::pulumi:providers:local-exec::default_0_3_1

Backend
Name pulumi.com
URL https://app.pulumi.com/
User

Dependencies:
NAME VERSION
pip 23.2.1
pulumi-local-exec 0.3.1
pulumi-utils-py 0.5.6
setuptools 56.0.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
@lhaskins lhaskins added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Sep 11, 2023
@justinvp justinvp added area/engine Pulumi engine impact/panic This bug represents a panic or unexpected crash and removed needs-triage Needs attention from the triage team labels Sep 13, 2023
@justinvp justinvp added the p1 Bugs severe enough to be the next item assigned to an engineer label Sep 30, 2023
@justinvp justinvp added this to the 0.95 milestone Sep 30, 2023
@Frassle Frassle mentioned this issue Oct 4, 2023
Merged
6 tasks
github-merge-queue bot pushed a commit that referenced this issue Oct 13, 2023
@Frassle
SameSteps can fail (#14076)
68af26a 
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This should fix the panic in
#13923, but will probably just
cause a new error to be raised for that use case.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
@Frassle
Copy link
Member

Frassle commented Oct 13, 2023

I've just merged a fix for the panic, which will be in the next release. I am not going to close this issue off yet though as I suspect this panic was just masking another error.

Once we get the next release out if you could try this again and let us know if it works, or more likely what error it now shows we can keep investigating.

@justinvp justinvp added the resolution/fixed This issue was fixed label Oct 17, 2023
@justinvp
Copy link
Member

I'm going to close this because we addressed the panic, which shipped in v3.89.0. Now that the panic has been addressed, please open a new issue if you're hitting a new error with v3.89.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Frassle Frassle

Labels
area/engine Pulumi engine impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec p1 Bugs severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed
Projects
None yet
Milestone
0.95
Development

No branches or pull requests
3 participants
@justinvp @Frassle @lhaskins