Skip to content
This repository has been archived by the owner on Jul 27, 2023. It is now read-only.

kafka: create topics and cleanup config #1525

Merged
merged 6 commits into from Jun 21, 2016
Merged

Conversation

ryane
Copy link
Contributor

@ryane ryane commented Jun 9, 2016

  • Installs cleanly on a fresh build of most recent master branch
  • Upgrades cleanly from the most recent release
  • Updates documentation relevant to the changes

  • updates default broker config
  • auto-creates the specified topics
  • easier to understand config
  • sets advertised.host.name as the consul service name for the broker
  • optional installation of kafka manager: https://github.com/yahoo/kafka-manager
  • uses a new consul acl token for writing to consul during framework addon installs (requires re-running security-setup)

Testing:

  • install the kafka addon (ansible-playbook -e @security.yml addons/kafka.yml)
  • verify that the there are 3 brokers started after a few minutes
  • broker-1, broker-2, and broker-3 should be registered in consul
  • verify that the 4 topics were created. You can ask the kafka api by running a command like:

curl -sku admin:pw https://control-01/kafka/api/topic/list

  • optionally install kafka-manager (ansible-playbook -e @security.yml -e 'kafka_manager_install=yes' addons/kafka.yml)
    • to test the kafka manager ui, you can either add kafka_manager_load_balancer=external when installing to expose it via traefik (assuming you have dns) or find the host and port in marathon and then open firewall/security group accordingly

@ryane ryane added the addons label Jun 9, 2016
@ryane ryane added this to the 1.2 milestone Jun 9, 2016
@sehqlr
Copy link
Contributor

sehqlr commented Jun 16, 2016

We got green on tests. Is there anything else you need?

@ryane
Copy link
Contributor Author

ryane commented Jun 16, 2016

  • install the kafka addon
  • verify that the there are 3 brokers started after a few minutes
  • broker-1, broker-2, and broker-3 should be registered in consul
  • verify that the 4 topics were created. You can ask the kafka api by running a command like:

curl -sku admin:pw https://control-01/kafka/api/topic/list

@ryane
Copy link
Contributor Author

ryane commented Jun 16, 2016

actually, if you haven't already started, hold off on this. I'm going to update it with the changes suggested in #1468

@ryane
Copy link
Contributor Author

ryane commented Jun 17, 2016

aws build passed, other builds failed but it looks like for unrelated reasons. @sehqlr this is ready to test now

@ryane ryane mentioned this pull request Jun 17, 2016
@arichardet
Copy link

Tested when worked on another task. LGTM.

@langston-barrett langston-barrett merged commit 66d8966 into master Jun 21, 2016
@langston-barrett langston-barrett deleted the feature/kafka-topics branch June 21, 2016 13:52
crumley added a commit to waldophotos/mantl that referenced this pull request Jun 22, 2016
* upstream/release/1.2.0: (72 commits)
  docs: 1.2 upgrade instructions
  elk: increase default resources (mantl#1569)
  docs: 1.2 rc1 changelog update (mantl#1581)
  kafka: create topics and cleanup config (mantl#1525)
  Add playbook to force consul leader election (mantl#948)
  updated functional IP descriptions
  traefik: fixes traefik ui
  Add memory usage restrictions to Docker containers (mantl#1451)
  traefik: package version 1.0.0
  distrib-dnsmasq upgrade to 0.2.6
  use global yum repository instead of targeting an rpm file
  upgrade to calico 0.19.0
  use real vault 0.5.3 package
  use preliminary vault-0.5.3 package
  template the repos file
  Proper mantl version and repo destination
  Use versioned mantl bintray repositories
  remove test.mesos.yml. Fixes mantl#1522
  Replace CiscoCloud logo with Mantl logo in docs (mantl#1470)
  kibana: use consistent val for elasticsearch-http
  ...
crumley added a commit to waldophotos/mantl that referenced this pull request Jul 11, 2016
* upstream/release/1.2.0: (73 commits)
  update release date in changelog (mantl#1665)
  changelog update
  consul: add retries to release lock during restart
  Increase the time waiting for a consul leader to be elected
  1.2 changelog update
  Feature/kube consul (mantl#1630)
  1.1 -> 1.2 upgrade playbook (mantl#1631)
  docs: 1.2 upgrade instructions
  elk: increase default resources (mantl#1569)
  docs: 1.2 rc1 changelog update (mantl#1581)
  kafka: create topics and cleanup config (mantl#1525)
  Add playbook to force consul leader election (mantl#948)
  updated functional IP descriptions
  traefik: fixes traefik ui
  Add memory usage restrictions to Docker containers (mantl#1451)
  traefik: package version 1.0.0
  distrib-dnsmasq upgrade to 0.2.6
  use global yum repository instead of targeting an rpm file
  upgrade to calico 0.19.0
  use real vault 0.5.3 package
  ...
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants