Skip to content
Permalink
master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

minotaur/labs/kafka/README.md

Go to file
 
 
Cannot retrieve contributors at this time
kafka-lab Usage
61 lines (43 sloc) 2.49 KB
Raw Blame
Open in GitHub Desktop

kafka-lab

This directory contains AWS, Chef and Vagrant scripts/recipes/templates to spin up Kafka-broker nodes.

Usage

usage: minotaur lab deploy kafka [-h] -e ENVIRONMENT -d DEPLOYMENT -r
                                 REGION -z AVAILABILITY_ZONE [-n NUM_NODES]
                                 [-i INSTANCE_TYPE] [-v ZK_VERSION]
                                 [-k KAFKA_URL]

optional arguments:
  -h, --help            show this help message and exit
  -e ENVIRONMENT, --environment ENVIRONMENT
                        CloudFormation environment to deploy to
  -d DEPLOYMENT, --deployment DEPLOYMENT
                        Unique name for the deployment
  -r REGION, --region REGION
                        Geographic area to deploy to
  -z AVAILABILITY_ZONE, --availability-zone AVAILABILITY_ZONE
                        Isolated location to deploy to
  -n NUM_NODES, --num-nodes NUM_NODES
                        Number of instances to deploy
  -i INSTANCE_TYPE, --instance-type INSTANCE_TYPE
                        AWS EC2 instance type to deploy
  -v ZK_VERSION, --zk-version ZK_VERSION
                        The Zookeeper version to deploy
  -k KAFKA_URL, --kafka-url KAFKA_URL
                        The Kafka URL

Mandatory arguments:

<environment> - name of the CloudFormation environment.

<deployment> - this term/option is used to logically separate groups of nodes within environment. Nodes that belong to different deployments won't interact with each other.

<region> - geographic area to deploy to.

<availability zone> - isolated location to deploy to.

Optional arguments:

[number of instances] defaults to 1

[instances flavor] defaults to m1.small

[source url] defaults to https://archive.apache.org/dist/kafka/0.8.2-beta/kafka_2.11-0.8.2-beta.tar.gz

Example:

./minotaur lab deploy kafka -e bdoss-dev -d testing -r us-east-1 -z us-east-1a -k http://example.com/kafka.tar.gz -n 3 -i m1.small - this will spin up 3 kafka-broker nodes in "testing" deployment.

NOTICE: If you're deploying a cluster - make sure that a separate Zookeeper node is running in the same environment and deployment.

During spin-up procedure, each Kafka node will try to reveal all Zookeeper nodes running in the same environment+deployment and will configure itself against them.

If no Zookeeper(s) found - each Kafka node will install Zookeeper locally and will rely on it.

If you want to remove your deployment - just delete a corresponding CloudFormation stack in AWS Web Console.