Terraform failing to destroy confluent resources with 401 Unauthorized error

[jantijink]

I am migrating our Confluent resources from one region to another. As we are still in the testing stage I don't need to pay attention to preserving messages, I can easily recreate them.

When migrating some of the resources I run into the following issues when I use terraform apply to destroy the current resources and recreate them. The cluster needs to be migrated, as well as some of the ACLs and kafka API keys. However, before the cluster can be moved, first some ACLs and kafka API keys need to be deleted. This is where I run into errors:

[ERROR] vertex "module.confluent-default.confluent_kafka_acl.name-read-topic["EXAMPLE_XYZ"] (destroy)" error: 401 Unauthorized:
[ERROR] vertex "module.confluent-default.confluent_api_key.name-kafka-api-key (destroy)" error: error deleting API Key "NOTREALKEYID": 401 Unauthorized: Unauthorized
[ERROR] vertex "module.confluent-default.confluent_kafka_acl.name-write-topic["EXAMPLE_XYZ"] (destroy)" error: 401 Unauthorized:
[ERROR] vertex "module.confluent-default.confluent_kafka_acl.read-consumer-group["EXAMPLE_XYZ"] (destroy)" error: 401 Unauthorized:
[ERROR] vertex "module.confluent-default.confluent_kafka_acl.name-write-topic["EXAMPLE_XYZ"] (destroy)" error: 401 Unauthorized:

The issue seems to be an authorization/rights issue. At the moment I'm using a confluent cloud API key that is linked to an OrganizationAdmin user. Therefore I would expect to be able to create/delete resources everywhere.

The only issue I can think of is that I also create a service account using this and this is configured as the owner as determined in the documentation (https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/resources/confluent_api_key) and the credentials of the service account for the ACL (https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/resources/confluent_kafka_acl). Since the service account is not linked with the Cloud API credentials used for Terraform this might provide the issue? Any help would be appreciated.

[linouk23]

@jantijink thanks for asking!

It'd be helpful to share your specific TF config with all sensitive values redacted.

One of the potential issues could be that role binding was deleted before deleting all listed resources.

For example, one of the fixes could look like this.

That said, it typically results in 403 and not 401 -- 401 is more for like wrong API Key ID & Secret pair.

[jantijink]

So I managed to identify the issue. The main reason this was happening was because we perform the terraform plan and terraform apply steps separately in the CI/CD pipeline for deployment. Within the terraform apply step the confluent cloud credentials were not any longer explicitly provided, as they should be included in the plan. However, when destroying the resources, it did require these credentials as environment variables. We have now added those as environment variables again in the apply step which ensures the destroy is performed succesfully.

In the end this was a mistake in the configuration on my end which we have now resolved. Thank you for your reply @linouk23