Can not create resource_api_key_v2 for clusters with private networking

[Jberlinsky]

When creating a cluster API key, the provider attempts to connect to the Kafka cluster to verify that the API key has successfully synced to the cluster before proceeding. For clusters that can be accessed from the location that Terraform is executing from, this works -- however, when creating a cluster without public connectivity, this can fail (i.e. if the path to the cluster traverses private IP addresses).

One possible remediation here is to add an input to the api_key_v2 resource e.g. wait_for_api_key_sync_to_cluster, which defaults to true, that governs whether this connection and check of the cluster state happens.

[muresan]

Adding some more information to the issue (an error message):

│ Error: error waiting for Kafka API Key "XXXXXXXXXXXX" to sync: error listing Kafka Topics using Kafka API Key "XXXXXXXXXXXX": Get "https://pkc-AAAAAA.us-east4.gcp.confluent.cloud:443/kafka/v3/clusters/lkc-AAAAAA/topics": GET https://pkc-AAAAAA.us-east4.gcp.confluent.cloud:443/kafka/v3/clusters/lkc-AAAAAA/topics giving up after 5 attempt(s): Get "https://pkc-AAAAAA.us-east4.gcp.confluent.cloud:443/kafka/v3/clusters/lkc-AAAAAA/topics": dial tcp 10.129.0.15:443: i/o timeout

Cluster network is peered with a GCP VPC and on the 10.129.0.0/16 network.

[linouk23]

Thanks for reporting an issue @Jberlinsky! This seems like a duplicate of #25.

The reason why we haven't implemented it yet is we didn't find a strong use case for it (yet) but it seems like we're getting there.

On a somewhat related note, could you share with us how you are going to create topic / ACLs using created Kafka API Key given the fact the Kafka cluster won't be reachable from the CI pipeline?

Could you describe your current TF setup in a little bit more detail where wait_for_api_key_sync_to_cluster would really help?

[Jberlinsky]

Hi @linouk23 -- we're working with a customer where application workloads are responsible for creating and managing topics/ACLs. Those workloads will have access to the cluster via private networking, but our Terraform execution environment is not guaranteed to execute from within that same environment. The workloads would still need to authenticate to the cluster to manipulate topics/ACLs.

I understand this may not be perceived as a strong use case -- would you be willing to accept a PR that implements it if we were able to provide one, or would it still not be prioritized for review?

[linouk23]

Well, it seems like there're 4 interested users at least (#25 + your message received 2 upvotes) and the change is simple enough so we'll try our best to include it in our next release, thanks for offering help though.

On a related note, we're always looking to improve UX of TF Provider so feel free to create an issue any time you think there's something we could improve on!

To add more details, I think the request is fairly reasonable for setups like kafka-ops-kafka-admin-product-team.

[maheshbhole]

@linouk23 when is next release planned with this fix ?
Thanks.

[linouk23]

@maheshbhole sure, early next week is our best estimate.

[maheshbhole]

is this fix part of 0.12 release ?

[linouk23]

@maheshbhole thanks for waiting, it's a part of 0.13.0 release.

@Jberlinsky @maheshbhole check out our latest 0.13.0 release where we

Added disable_wait_for_ready attribute to disable readiness check for confluent_api_key resource (#25, #51).