Destroying load balancers periodically fail and require manual intervention

[jhorwit2]

Terraform Version

$ terraform -v
Terraform v0.11.0
+ provider.dyn v1.1.0
+ provider.oci (unversioned)

Your version of Terraform is out of date! The latest version
is 0.11.1. You can update by downloading from www.terraform.io/downloads.html

OCI Provider Version

Execute the plugin directly to get the version:

$ ~/.terraform.d/plugins/terraform-provider-oci -v
2018/01/08 09:08:30 [INFO] terraform-provider-oci 2.0.5

Description:

I have a terraform plan that includes 1 load balancer with 3 listeners and 3 backendsets. Each backendset has 5 backends. Fairly often when destroying the load balancer i'll get the following error which requires manual intervention to be able to remove the load balancer.

Error: Error applying plan:

1 error(s) occurred:

* module.public-lb.oci_load_balancer_backend.authn[3] (destroy): 1 error(s) occurred:

* oci_load_balancer_backend.authn.3: unexpected state 'FAILED', wanted target 'SUCCEEDED_WORK_REQUEST, SUCCEEDED'. last error: %!s(<nil>)

The only way I can get around this is to remove the state for that resource from terraform.

[alexng-canuck]

Investigating this. @jhorwit2 could you share the sample TF config that reproduced this issue?

[alexng-canuck]

If the FAILED state is intermittently returned by the service, I would expect retry logic to kick in and eventually succeed down the road. Will investigate if that's even happening.

[alexng-canuck]

There appears to be a bug in the LoadBalancer backend resource. CRUD operations for this resource rely on async work requests. The reason we are erroring out on the "FAILED" state is because we don't check for this state. We should be able to fix this to give the retry logic a chance.

[sanketdjain]

+1
We have also encountered the same issue, would appreciate a fix in next version.

[kohashim]

A mitigation has been made and landed in the master branch: 66f1065, which removes at least the necessity of removing the failed resources from the state file when destroying. The next release will include this fix. Otherwise you can clone the master branch to try it out.

The ideal solution is still to retry on the failure of a work request in a load balancer resource, which is in our backlog.

[codycushing]

Fixed in https://github.com/oracle/terraform-provider-oci/releases/tag/2.0.7