New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
crtl-c shows "graceful" shutdown message, then long pause, then terraform still destroys resources #30918
Comments
Hi @josh-m-sharpe! Thanks for reporting this. It does seem like the graceful shutdown began at a bad time here where Terraform wasn't prepared to deal with it; I guess it probably cancelled an operation that wasn't actually needed to continue and so Terraform didn't get its usual opportunity to see an operation fail with an error as a result of being cancelled and thus didn't stop running early as it should have. You mentioned this went on to actually apply the plan. Did Terraform fail to show the confirmation prompt in this case, or had you intentionally disabled confirmation with Since this seems to be a timing-sensitive problem I bet it will be hard to reliably reproduce, but it looks like the cancellation here might've arrived right at the end of the planning process, when Terraform had already done all of the cancelable work anyway, and so the plan operation was still active (and subject to cancellation) but was already in the process of returning the successful plan and so didn't check the cancellation state again. If so, I expect we need to find some way to propagate the cancellation channel from the plan operation into the confirmation prompt and subsequent apply operation so that once it is cancelled it stays cancelled and will not be able to start any new cancelable operations without immediately encountering the cancellation error. |
I ride the lighting -- so, yes, I did run I'm not 100% sure if it had completed planning when I ctrl-c'd. It would seem that a ctrl-c any time before the operation/execution phase should prevent that phase from starting. |
I think probably what we'll need to do here is assume my theory above is correct for the moment, and find some way to modify the code to simulate that situation by artifically triggering the cancellation sequence as just the wrong moment (after planning has completed all of the network operations but the apply walk hasn't yet started) and see if that causes the cancellation to be ignored. If it does, that would confirm that my hunch is valid and we can work on figuring out why the cancellation state doesn't survive between the plan phase and the apply phase. (My initial guess: we're creating a new context for each walk and so the fact that the context for the plan walk was cancelled doesn't affect the freshly-created context for the apply walk. But I've not checked that.) |
The behavior shown here can be explained by the combination of a couple factors. After the plan is cancelled however we hit the root cause of this bug -- the next step where the context is checked when running apply with no prior plan is during the plan confirmation, so when the confirmation is skipped, so is the check for cancellation. |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. |
Terraform Version
v1.1.7
Screenshot of destruction:
Expected Behavior
not destroyed resources
Actual Behavior
resources destroyed. site down. 💥
Steps to Reproduce
Press ctrl-c - see graceful message. wait.
Additional Context
As noted in the screenshot there was a long pause after the Graceful shutdown message showed up to when it continued on and destroyed resources. That is, it had plenty of time to think about doing things and still didn't shutdown.
Counterintuitively, I'm pretty sure if I held ctrl-c down and sent the process 10s or 100s of kill signals it would've died "ungracefully" and the resource would not have been destroyed. This would've been a much better scenario.
The text was updated successfully, but these errors were encountered: