kafka: create topics and cleanup config

[ryane]

• Installs cleanly on a fresh build of most recent master branch
• Upgrades cleanly from the most recent release
• Updates documentation relevant to the changes

---

• updates default broker config
• auto-creates the specified topics
• easier to understand config
• sets advertised.host.name as the consul service name for the broker
• optional installation of kafka manager: https://github.com/yahoo/kafka-manager
• uses a new consul acl token for writing to consul during framework addon installs (requires re-running security-setup)

---

Testing:

• install the kafka addon (ansible-playbook -e @security.yml addons/kafka.yml)
• verify that the there are 3 brokers started after a few minutes
• broker-1, broker-2, and broker-3 should be registered in consul
• verify that the 4 topics were created. You can ask the kafka api by running a command like:

curl -sku admin:pw https://control-01/kafka/api/topic/list

• optionally install kafka-manager (ansible-playbook -e @security.yml -e 'kafka_manager_install=yes' addons/kafka.yml)
  • to test the kafka manager ui, you can either add kafka_manager_load_balancer=external when installing to expose it via traefik (assuming you have dns) or find the host and port in marathon and then open firewall/security group accordingly

[sehqlr]

We got green on tests. Is there anything else you need?

[ryane]

• install the kafka addon
• verify that the there are 3 brokers started after a few minutes
• broker-1, broker-2, and broker-3 should be registered in consul
• verify that the 4 topics were created. You can ask the kafka api by running a command like:

curl -sku admin:pw https://control-01/kafka/api/topic/list

[ryane]

actually, if you haven't already started, hold off on this. I'm going to update it with the changes suggested in #1468

[ryane]

aws build passed, other builds failed but it looks like for unrelated reasons. @sehqlr this is ready to test now

[arichardet]

Tested when worked on another task. LGTM.