Deleting pendingDeletes at beginning of deployment leads to stuck states
#2948
Comments
lukehoban
added
the
p1
|
Another example that is likely related - from https://pulumi-community.slack.com/archives/C84L4E3N1/p1563557008032100:
|
|
We've decided that this is too risky a change to take at this point in Q3. We will fix this ASAP post-1.0. |
pgavlin
removed
the
p1
What's the recommended solution to get out of this weird state? I'm having similar issues as described in the Slack message however can't see the responses due to the 10,000 message limit. Error log below: |
|
This would be a great quality of life improvement! I've run into both of the problems Luke mentioned in the description. |
|
Another member of the internal team hit this today. Their first updated did the create side of a replacement of a LaunchConfiguration. Then the update failed for a reasonable reason. The next update they did failed almost immediately with: |
got exactly this error too, merely changed some vpc stuff and then it decided it was time to delete the target group and now can't get rid of "completing deletion from previous update..." state |
|
any suggested workarounds for this issue? |
|
Same here... tried to manually remove the resource from the stack to no avail. Do you want to perform this update? yes View Live: https://app.pulumi.com/CLIENT/STACK/ENV/updates/NN
Diagnostics: Resources: Duration: 2s |
|
Ok, found a workaround. It's not pretty but does the job:
|
This removes all the handling of pending deletes from the start of deployments. Instead we allow resources to just be deleted as they usually would at the end of the deployment. There's a big comment in TestPendingDeleteOrder that explains the order of operations in a succesful run and how that order differs if we try and do pending deletes up-front. Fixes #2948
This removes all the handling of pending deletes from the start of deployments. Instead we allow resources to just be deleted as they usually would at the end of the deployment. There's a big comment in TestPendingDeleteOrder that explains the order of operations in a succesful run and how that order differs if we try and do pending deletes up-front. Fixes #2948
|
@ralvarez-globant You say:
Did you just remove the problematic resource itself? I would imagine that you need to remove any other resources the reference it as a dependency, as well. |
11009: Fix update plans with dependent replacements r=Frassle a=Frassle <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> We weren't correctly handling the case where a resource was marked for deletion due to one of it's dependencies being deleted. We would add an entry to it's "Ops" list, but then overwrite that "Ops" list when we came to generate the recreation step. Fixes #10924 ## Checklist <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Service, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. --> 11027: Do not execute pending deletes at the start of deployment r=Frassle a=Frassle <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This removes all the handling of pending deletes from the start of deployments. Instead we allow resources to just be deleted as they usually would at the end of the deployment. There's a big comment in TestPendingDeleteOrder that explains the order of operations in a succesful run and how that order differs if we try and do pending deletes up-front. Fixes #2948 ## Checklist <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Service, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. --> Co-authored-by: Fraser Waters <fraser@pulumi.com>
ericrudder
changed the title
pendingDeletes at beginning of deployment leads to stuck states
Today, we process any
pendingDeletesat the beginning of a deployment. This is not "correct".Two examples:
First, a program with a VPC and an Instance. A change causes the VPC to be replaced, and the Instance fails to create. This leads to a newly created VPC, and a pending delete VPC. On the next update, we try to flush the pending deletes, meaning trying to delete the old VPC. This fails, because the Instance is still running in the old VPC. It is only "correct" to delete the old VPC at the end of the deployment after all other repercussions of the replacement have been made.
Second, a Kubernetes Provider and a Kubernetes Resource. A change causes the Kubernetes Provider to be replaced, but the Kubernetes Resource fails to create. This leads to a newly created Provider in the checkpoint, and a pending delete Provider in the checkpoint. On the next update, we successfully delete the pending delete Provider from the checkpoint. However, now all of the references in the checkpoint have provider references to a provider which does not exist. When we try to process the recreate of the Kubernetes Resource, it fails with a message like:
To be correct, I believe we will need to postpone pending deletes to the end of the deployment.
The text was updated successfully, but these errors were encountered: