Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DM-19745] Single machine deployment of the DM-EFD using k3s ("kubes") #24

Merged
merged 4 commits into from May 22, 2019

Conversation

Projects
None yet
2 participants
@afausti
Copy link
Member

commented May 22, 2019

  • Specify storage class for kafka, influxdb and prometheus-operator
  • Specify size of influxdb disk
  • Control if telegraf client is deployed as daemonset (false if deployed to single node with k3s)
  • There was an assumption that the number of kafka loadbalancer services was the same as the number of brokers, and it was fixed to 3 in the terraform deployment. When doing single machine deployment we still can have n>1 brokers but only 1 loadbalancer. It is now possible to configure the number of kafka loadbalancer services.

afausti added some commits May 20, 2019

Configure the number of kafka lb services
- There was an assumption that the number of kafka loadbalancer services was the same as the number of brokers, and it was fixed to 3 in the terraform deployment. When doing single machine deployment we still can have n>1 brokers but only 1 loadbalancer.

@afausti afausti merged commit e0399c8 into master May 22, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
confluent_lb0_ip = "${lookup(data.kubernetes_service.lb0.load_balancer_ingress[0], "ip")}"
confluent_lb1_ip = "${lookup(data.kubernetes_service.lb1.load_balancer_ingress[0], "ip")}"
confluent_lb2_ip = "${lookup(data.kubernetes_service.lb2.load_balancer_ingress[0], "ip")}"
confluent_lb_ips = ["${data.kubernetes_service.lb.0.load_balancer_ingress.0.ip}"]

This comment has been minimized.

Copy link
@jhoblitt

jhoblitt May 22, 2019

Member

Was this tested with a count of > than 1? I believe a * should be required.

This comment has been minimized.

Copy link
@afausti

afausti May 23, 2019

Author Member

So I tried confluent_lb_ips = ["${data.kubernetes_service.lb.*.load_balancer_ingress.0.ip}"] as suggested here but it didn't work

Error: Error running plan: 1 error(s) occurred:

* module.efd.local.confluent_lb_ips: local.confluent_lb_ips: Resource 'data.kubernetes_service.lb' does not have attribute 'load_balancer_ingress.0.ip' for variable 'data.kubernetes_service.lb.*.load_balancer_ingress.0.ip'

using confluent_lb_ips = ["${data.kubernetes_service.lb.0.load_balancer_ingress.0.ip}"] seems to work.

Output of terraform plan:

 <= module.efd.data.kubernetes_service.lb[0]
      id:                          <computed>
      load_balancer_ingress.#:     <computed>
      metadata.#:                  "1"
      metadata.0.generation:       <computed>
      metadata.0.name:             "confluent-0-loadbalancer"
      metadata.0.namespace:        "kafka"
      metadata.0.resource_version: <computed>
      metadata.0.self_link:        <computed>
      metadata.0.uid:              <computed>
      spec.#:                      <computed>

 <= module.efd.data.kubernetes_service.lb[1]
      id:                          <computed>
      load_balancer_ingress.#:     <computed>
      metadata.#:                  "1"
      metadata.0.generation:       <computed>
      metadata.0.name:             "confluent-1-loadbalancer"
      metadata.0.namespace:        "kafka"
      metadata.0.resource_version: <computed>
      metadata.0.self_link:        <computed>
      metadata.0.uid:              <computed>
      spec.#:                      <computed>

 <= module.efd.data.kubernetes_service.nginx_ingress
      id:                          <computed>
      load_balancer_ingress.#:     <computed>
      metadata.#:                  "1"
      metadata.0.generation:       <computed>
      metadata.0.name:             "nginx-ingress-controller"
      metadata.0.namespace:        "nginx-ingress"
      metadata.0.resource_version: <computed>
      metadata.0.self_link:        <computed>
      metadata.0.uid:              <computed>
      spec.#:                      <computed>

  ~ module.efd.aws_route53_record.grafana
      records.#:                   "" => <computed>

  ~ module.efd.aws_route53_record.influxdb
      records.#:                   "" => <computed>

  ~ module.efd.aws_route53_record.kafka_lb[0]
      records.#:                   "" => <computed>

  + module.efd.aws_route53_record.kafka_lb[1]
      id:                          <computed>
      allow_overwrite:             "true"
      fqdn:                        <computed>
      name:                        "test-efd1.lsst.codes"
      records.#:                   <computed>
      ttl:                         "60"
      type:                        "A"
      zone_id:                     "Z3TH0HRSNU67AM"

however, I did not test this against gke yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.