Remove stale locks/Ability to add timeout to locks

[abergmeier]

Terraform Version

0.12.28

Expected Behavior

There should be a mechanism to get rid of the Lock even if the Process dies suddenly.

Actual Behavior

Lock persists and nobody can use Terraform anymore. Developer needs to manually remove lock.
This is fine as long CI is stable - but gets really annoying if CI is terminating Terraform very frequently.

Steps to Reproduce

Kill -9 terraform when it it is planning.

Fix to this

The most elegant IMO would be to add a (optional) timeout option to backends. This way there could be a timeout set to 2 minutes. Terraform could then touch the lock every minute to ensure it does not time out.
If the Lock is present and older than 2 minutes, other Terraforms are allowed to remove it.

[danieldreier]

@abergmeier I think you're looking for the force-unlock command described in https://www.terraform.io/docs/commands/force-unlock.html - is that what you're looking for?

[abergmeier]

I think you're looking for the force-unlock command

One the contrary, I am looking for a mechanism to make force-unlock mostly unnecessary if the terraform machine is not that stable.

[aweingarten]

This is a very real concern on our project too. We worry about an unstable provision infrastructure acquiring a lock dying and never releasing the lock.

Since we are using terraform as part of an existing automation workflow force-unlock would represent a support escalation.

For now we are planning to write a bash script that will delete a lock that exceeds n minutes before running apply. We would like this to be a first class capability.

[Nadavpe]

We're facing similar issues, mostly when killing a github-action workflow, which happens every now and then.

The 2 minutes suggested above must be parametric, as some resources may have a longer response time, so it should be in the state's configuration, thus allowing admins to set the desired lock timeout.

[SathishKumarRamasamy]

+1
We would need this feature to avoid infinite locks when the process gets killed.