Timing issue while destroying Key Protect resources

[ocofaigh]

We have been experiencing issue with the destroy of Key Protect Key Rings recently, and I am pretty sure its due to a timing issue (potentially backend slowness of key deletion). For example:

• the key deletion has been marked as successful:
  2023-09-30T14:49:01Z command.go:185: module.key_protect_all_inclusive.module.key_protect_keys["icd-pg.postgres-upg-tnjkeb-pg"].ibm_kms_key.key: Destruction complete after 2s

• terraform attempts to destroy key ring
  2023-09-30T14:49:01Z command.go:185: module.key_protect_all_inclusive.module.key_protect_key_rings["icd-pg"].ibm_kms_key_rings.key_ring: Destroying... [id=icd-pg:keyRing:crn:v1:bluemix:public:kms:eu-de:a/abac0df06b644a9cabc6e44f55b3880e:a510c309-84f6-47e8-b7b0-6e1aaa740455::]

• destroy fails with this error
  2023-09-30T14:49:56Z command.go:185: │ Error: failed to Destroy key ring with error: kp.Error: correlation_id='78823f65-e124-4339-9ca8-fb16160be0f9', msg='Conflict: Key ring could not be deleted: Please see reasons for more details (KEY_RING_NOT_EMPTY_ERR)', reasons='[KEY_RING_NOT_EMPTY_ERR: The specified key ring contains at least one key (in any state) - FOR_MORE_INFO_REFER: https://cloud.ibm.com/apidocs/key-protect]'

As you can see the successful delete message of the key and the attempt to delete the key ring have the exact same timestamp -> 2023-09-30T14:49:01Z
So my guess is the key is not actually fully deleted on the backend even though its being logged as deleted and hence the terraform dependency tree moves onto next resource to destroy.

Perhaps adding a retry to the ibm terraform provider code might prevent such a failure if there is some backend slowness?

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform IBM Provider Version

Affected Resource(s)

• ibm_kms_key_rings

Terraform Configuration Files

See https://github.com/terraform-ibm-modules/terraform-ibm-key-protect-all-inclusive/blob/main/main.tf

Debug Output

Panic Output

Expected Behavior

Key Ring deleted successfully

Actual Behavior

Key Ring deletion failed

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[BrunoHenriques]

Investigation is underway but this seems to be caused by an unintended consequence of a bug fix part of IBM Terraform Cloud Provider 1.58.0. Before that Key Rings were in fact not deleted, even if they had Keys, even when all Keys had state = 5 (destroyed). Most users did not notice that because the KP instance was deleted as part of Terraform destroy command, which deleted the KP instance, deleting the Key Ring.

Please use the force_delete = true attribute, as the example below. With that a Key Ring which has all Keys with state = 5 (destroyed) will in fact be deleted.

resource "ibm_kms_key_rings" "key_ring" {
  instance_id = ibm_resource_instance.kms_instance.guid
  key_ring_id = "key-ring-id"
  force_delete = true
}

[ocofaigh]

I see this issue was added to the 1.58.1 release notes, but it doesn't seem to be linked to any PR. Was there actually a fix added?

[hkantare]

#4841 - This pr