confluent_kafka_cluster Data Source is missing attribute 'http_endpoint'

[nikolai-fra]

Hello,

we've encountered the following problem:

The resource confluent_kafka_cluster is exposing an attribute 'http_endpoint'

The corresponding datasource confluent_kafka_cluster does not provide this attribute.

When using this attribute (it is defined the the schema and terraform validation is successful) it seems to be empty as the following error shows during "terrafom apply":
Error: Post "/kafka/v3/clusters/redacted/topics": POST /kafka/v3/clusters/redacted/topics giving up after 1 attempt(s): Post "/kafka/v3/clusters/redacted/topics": unsupported protocol scheme ""

I think this is a bug?

[linouk23]

@nikolai-fra could you share more details about your cluster? Is it Basic or Peering / Private Linked one?

upd: oh wait, you're saying it's missing completely?

[linouk23]

Just run the following for a Basic cluster:

terraform {
  required_providers {
    confluent = {
      source  = "confluentinc/confluent"
      version = "0.7.0"
    }
  }
}

data "confluent_kafka_cluster" "test-basic-cluster" {
  id = "lkc-b11bbb"
  environment {
    id = "env-a11a1"
  }
}

output "endpoint" {
  value = data.confluent_kafka_cluster.test-basic-cluster.http_endpoint
}

➜  demo git:(master) ✗ terraform apply --auto-approve
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
endpoint = "https://pkc-1a11a.us-central1.gcp.confluent.cloud:443"

so I suspect your cluster is either PL / Peering, could you share its ID / orgID to the support team / cflt-tf-access@confluent.io and we'll update your cluster such that http_endpoint attribute will be populated?

[nikolai-fra]

Thanks for your reply @linouk23
You're correct: Our cluster is a dedicated private link cluster on azure. We do have more problems as the http_endpoint isn't shown on the confluent cloud portal. We opened a support ticket too (number 104287).

So ok that explains why the value was empty on our cluster. So there is no general problem with the 'http_endpoint' attribute.

But: One issue still persists: The DataSource confluent_kafka_cluster does not list the attribute "http_endpoint". It's missing in the terraform docs -> https://github.com/confluentinc/terraform-provider-confluent/blob/master/docs/data-sources/confluent_kafka_cluster.md?plain=1#L52

Therefore I assumed a general problem with this attribute on the data source and opened this ticket.

[linouk23]

No worries at all!

Opened #15 to track doc update 👍 and will talk to our support team to update your Kafka cluster such that http_endpoint will be populated 👌

[nikolai-fra]

Thank you very much :)

[linouk23]

Let's keep it open for now for additional visibility until we get to a resolution.

[linouk23]

btw qq: is it a test cluster or something? The simplest solution for you could be to recreate a cluster (all new PL clusters should have http_endpoint populated by default) and it won't require intervention from our side, let me know if that's an option.

[linouk23]

@nikolai-fra ⬆️

I talked to some engineers internally and it seems like right now we can't fix it from our side for existing PL clusters (eta is a bit unknown but if I had to guess, we're blocked for like 2 weeks or something) so you might have to provision a new PL clusters for http_endpoint to work. Let me know if that helps.

[Marcus-James-Adams]

@linouk23 as I was blocked by #18,, as recommened I deleted the cluster, network service and private link. To ensure all all was fresh I even deleted my backend tfstatefile so there was no sign of any old configuration anywhere. When I provision new via terraform I hit the same issue the http_endpoint is missing.

Is the workaround to manually create the cluster and then import it into terraform in the short term?

[linouk23]

@Marcus-James-Adams are you saying you created a new cluster and http_endpoint was not populated still? That actually should have worked in most of the regions:

• AWS – eu-central-1, us-west-2, us-east-1, us-east-2, eu-west-2
• Azure – WestEurope, EastUS2, HK(EastAsia), Singapore (SouthEastAsia)

Our teams are working hard to enable all the regions for both existing and new clusters though.

[Marcus-James-Adams]

@linouk23 correct, our clusters are in Azure UKSouth if that makes a difference?

[linouk23]

@Marcus-James-Adams I see, unfortunately your region isn't enabled just yet so you may need to wait for a fix from our backend team 😕

[Marcus-James-Adams]

Is there an ETA for when each region will have it enabled by default? Or in the short term can it be manually set by support?

[linouk23]

@Marcus-James-Adams
The mentioned regions should work now, regarding UKSouth region (as well as existing clusters in all the regions) -- there's no official date set but we're hoping to enable them in June.

[Marcus-James-Adams]

Can this issue be fixed manually on a cluster by cluster basis by the support teams? We are restricted to UK South for regulatory reasons and have a project looking to go live before June.

[linouk23]

@Marcus-James-Adams they may be able to do a one-off "fix" for your cluster a little bit earlier but you might need to talk to them directly.

[Marcus-James-Adams]

@linouk23 This issue also effects the confluent_api_key resource

Error: error fetching Kafka Cluster "lkc-6kw7o2"'s "http_endpoint" attribute: http_endpoint is nil or empty for Kafka Cluster "lkc-6kw7o2"
with confluent_api_key.app-manager,
on confluent.cloud.users.svc_cluster_admin.tf line 15, in resource "confluent_api_key" "main":
15: resource "confluent_api_key" "main" {

This effectively stops terraform being useable in any region without the fix.
is there any way that we can get the issue or the fix escalated please?

[linouk23]

I understand the frustration and we'll do our best to enable other regions as soon as possible but regarding

This effectively stops terraform being useable in any region without the fix.

I'd like to highlight that TF works for new (or "reprovisioned") PL clusters in the following regions already:

• AWS – eu-central-1, us-west-2, us-east-1, us-east-2, eu-west-2
• Azure – WestEurope, EastUS2, HK(EastAsia), Singapore (SouthEastAsia)

[nikolai-fra]

Hi @linouk23 ,
as you suggested we recreated our cluster and everything works as expected.
Thanks for your help!

[Marcus-James-Adams]

Now that our region has had the fix applied this is working ok - thank you.
I think this ticket can be closed

[linouk23]

@Marcus-James-Adams that's great to hear!

We'll probably keep this ticket open until we receive the official confirmation that the enablement process is completed for all the clusters.

[komal-SkyNET]

Hey @linouk23 @Marcus-James-Adams, do you guys know if this has been fixed in AWS ap-southeast-2 region? Does it require a cluster recreation to resolve? We are pretty much blocked by this exact same problem and a recreation of the clusters would be a costly activity for us at this stage.

[linouk23]

do you guys know if this has been fixed in AWS ap-southeast-2 region?

@komal-SkyNET if the question is about the existing clusters, we're currently targeting end of June to enable http_endpoint for them.

[linouk23]

closing ⬆️ since our backend team rolled out a fix for

│ Error: error fetching Kafka Cluster "abc-cluster"'s "http_endpoint" attribute: http_endpoint is nil or empty for Kafka Cluster "abc-cluster"
│ 8: resource "confluent_api_key" "cluster-api-key" {

that should resolve the issue.

Also check out our latest 1.0.0 release

The Confluent Terraform Provider is now Generally Available and recommended for use in production workflows.