Using this provider with multiple AWS Kafka clusters in the same AWS account

[amplexus]

Hi,

We have multiple Amazon Kafka (MSK) clusters in our non-production AWS tenancy - one for each non-prod "environment" we manage - and would like to use this provider to provision topics into them.

Currently we have a backend config file for each environment - e.g. dev-backend.tfvars, test-backend.tfvars etc - and these use the AWS S3 backend to manage state. Then we have per-environment vars - e.g. dev.tfvars, test.tfvars etc - that contain the topics to provision.

So for example, I'd like to be able to specify our "dev" environment kafka bootstrap servers in the dev.tfvars file mentioned above.

However because the bootstrap servers are part of the provider configuration it appears we can't specify the bootstrap servers in per-environment tfvars files.

And it isn't clear to me how else I can possibly "parameterise" the provider configuration such that I can provision the topics in a single environment via a terraform apply -var-file dev.tfvars call.

Furthermore, if we wanted to create a terraform "module" to abstract away the topic creation, looking at the module provider docs because the bootstrap servers are part of the provider block, it seems to me that I would have to create a separate module for each "environment" - which kind of defeats the purpose of modules.

Is my understanding correct? If not can you point to what terraform mechanism can be used to achieve the above?

[LeoQuote]

Hi, I just discovered that you can achive your goal by using alias

// online
provider "kafka" {
  bootstrap_servers = ["cluster1:9092"]

}

// data
provider "kafka" {
  alias = "data"
  bootstrap_servers = ["cluster2:9092"]
}

resource "kafka_topic" "test" {
  name               = "test_logs"
  replication_factor = 1
  partitions         = 2

  config = {
    "segment.ms"     = "20000"
    "cleanup.policy" = "compact"
  }
}

resource "kafka_topic" "data_test" {
  provider = kafka.data
  name               = "test_logs"
  replication_factor = 1
  partitions         = 2

  config = {
    "segment.ms"     = "20000"
    "cleanup.policy" = "compact"
  }
}