New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terraform completely destroy local state on Ctrl+C while performing #13851

Closed
aholbreich opened this Issue Apr 21, 2017 · 8 comments

Comments

Projects
None yet
6 participants
@aholbreich
Copy link

aholbreich commented Apr 21, 2017

Terraform Version

0.9.3

This is really bad. Took me hours. If terraform stuck and while waiting for some resource beeing created and i hit [Ctr-C] (Or double) to break up. I fould nearle ampty terrafrom.tfstate under my current profile.
Instead of >90 resources only 1 was left. So That resoucer i need somehow delte in AWS manually
and other are considered to be new... :(

@aholbreich

This comment has been minimized.

Copy link

aholbreich commented Apr 21, 2017

Also it happens when cycles where found..
`Error applying plan:

1 error(s) occurred:

  • Cycle: aws_route53_record.wm-ott-reporting, module.wm-ott-reporting.aws_alb_target_group.default (destroy)

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
`

@jbardin

This comment has been minimized.

Copy link
Contributor

jbardin commented Apr 21, 2017

Hi @aholbreich,

Sorry you're having an issue with this. Are you using a remote state backend? If your state was stored locally, after terraform exited there should have been a backup file in the same directory containing the original state. Did you see a state file with a .backup extension?

If you didn't get a backup file, can you share the command line options you used to invoke terraform?

@aholbreich

This comment has been minimized.

Copy link

aholbreich commented Apr 22, 2017

@jbardin I'v used local state. Unfortunately it took time till I understood what was going on. So backup was also useless. But last checked in state from my git report was kind of useful. Helped a bit. Anyway the is no reason to destroy local stateam on any problem.

@chrisrlong

This comment has been minimized.

Copy link

chrisrlong commented Apr 24, 2017

I have had something very similar, but my backend was on s3. I had about 6 extra resources in AWS and the plan/apply diffs failed and crashed.

I have not set s3 version-ing to help against this.

@jbardin

This comment has been minimized.

Copy link
Contributor

jbardin commented Apr 26, 2017

Sorry the operation here isn't as clear as it could be. In order to lose updates to the state you usually have to force terraform to quit, and if that's done via multiple signals from ctrl+c, terraform will exit with the message:

Two interrupts received. Exiting immediately. Note that data
loss may have occurred.

The message is there to indicate that you may need to check the state and recover from the backup.

I just merged a change to the output in #13941 to help provide some forewarning, on the first interrupt signal, terraform will start with:

Interrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...

Hopefully that makes it a little more clear that some care needs to be taken in this situation, and that the state may not be complete.

@chrisrlong, In the next release of terraform, remote state backends should also leave a local backup file as another layer of protection, however the extra redundancy of the versioned s3 bucket is still recommended.

@jbardin jbardin closed this Apr 26, 2017

@aholbreich

This comment has been minimized.

Copy link

aholbreich commented Apr 27, 2017

@jbardin thank you. But honestly don't believe changing output resolves this issue. It's not about output it's about behaviour.

@dcosson

This comment has been minimized.

Copy link
Contributor

dcosson commented Oct 26, 2018

@jbardin Not sure if this is the right issue for this, but I find the new output still quite misleading

What is terraform doing when gradually shutting down? In most cases when I interrupt, it's because I'm waiting on one resource that takes a particularly long time to enter it's ready state.

When I hit control + c and it says it's gradually shutting down I would expect that to mean that terraform won't interrupt an active api call, i.e. the initial calls to actually create or configure the resource, but if it's just polling for a status to become "ready" it should stop doing that and just write to the state that the status is still creating or whatever it may be. What happens instead is the interrupt seems to have no effect, it prints the graceful shutdown message and then will happily keep polling for the ready state for 45 more minutes/until the global timeout.

@jbardin

This comment has been minimized.

Copy link
Contributor

jbardin commented Oct 26, 2018

Hi @dcosson,

When you interrupt an apply in terraform, any running providers are signaled to stop, but how they handle that signal is up to the individual provider, and may vary between resources. Once the blocking call returns, terraform will cease calling any other resource operations and continue shutting down.

Unfortunately it's the case that most providers have not implemented handling interrupts on many long-running resource operations. These changes would need to be implemented on the providers themselves. Once work starts on a new provider SDK, we will be able to re-evaluate the API and find a more convenient way for providers to handle this.

@apparentlymart apparentlymart added this to the v0.12.0 milestone Oct 29, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment