Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using this provider with multiple AWS Kafka clusters in the same AWS account #400

Open
amplexus opened this issue Mar 28, 2024 · 1 comment

Comments

@amplexus
Copy link

Hi,

We have multiple Amazon Kafka (MSK) clusters in our non-production AWS tenancy - one for each non-prod "environment" we manage - and would like to use this provider to provision topics into them.

Currently we have a backend config file for each environment - e.g. dev-backend.tfvars, test-backend.tfvars etc - and these use the AWS S3 backend to manage state. Then we have per-environment vars - e.g. dev.tfvars, test.tfvars etc - that contain the topics to provision.

So for example, I'd like to be able to specify our "dev" environment kafka bootstrap servers in the dev.tfvars file mentioned above.

However because the bootstrap servers are part of the provider configuration it appears we can't specify the bootstrap servers in per-environment tfvars files.

And it isn't clear to me how else I can possibly "parameterise" the provider configuration such that I can provision the topics in a single environment via a terraform apply -var-file dev.tfvars call.

Furthermore, if we wanted to create a terraform "module" to abstract away the topic creation, looking at the module provider docs because the bootstrap servers are part of the provider block, it seems to me that I would have to create a separate module for each "environment" - which kind of defeats the purpose of modules.

Is my understanding correct? If not can you point to what terraform mechanism can be used to achieve the above?

@LeoQuote
Copy link

Hi, I just discovered that you can achive your goal by using alias

// online
provider "kafka" {
  bootstrap_servers = ["cluster1:9092"]

}

// data
provider "kafka" {
  alias = "data"
  bootstrap_servers = ["cluster2:9092"]
}

resource "kafka_topic" "test" {
  name               = "test_logs"
  replication_factor = 1
  partitions         = 2

  config = {
    "segment.ms"     = "20000"
    "cleanup.policy" = "compact"
  }
}

resource "kafka_topic" "data_test" {
  provider = kafka.data
  name               = "test_logs"
  replication_factor = 1
  partitions         = 2

  config = {
    "segment.ms"     = "20000"
    "cleanup.policy" = "compact"
  }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants