Skip to content

elodina/go-kafka-client-mesos

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

go_kafka_client Mesos Framework

Build Status

Description Installation

Typical operations

Navigating the CLI

Description

This framework aims to simplify running all sorts of software built on top of go_kafka_client on Mesos. Being actively developed right now.

Right now this framework supports running only the mirror maker.

Installation

Prerequisites

Install go 1.5 (or higher) http://golang.org/doc/install

Install glide:

  go get -u github.com/Masterminds/glide

Clone and build the project

# git clone https://github.com/elodina/go-kafka-client-mesos.git
# cd go-kafka-client-mesos
# export GO15VENDOREXPERIMENT=1
# glide install
# go build cli.go
# go build executor.go    

Scheduler configuration

# ./cli help scheduler
Usage: scheduler [options]

Options:
    --master: Mesos Master addresses.
    --api: API host:port for advertizing.
    --user: Mesos user. Defaults to current system user.
    --storage: Storage for cluster state. Examples: file:go_kafka_client_mesos.json; zk:master:2181/go-mesos.
    --log.level: Log level. trace|debug|info|warn|error|critical. Defaults to info.
    --framework.name: Framework name.
    --framework.role: Framework role.
    --framework.timeout: Framework failover timeout.

Quick start

In order not to pass the API url to each CLI call lets export the URL as follows:

# export GM_API=http://master:6666

First lets start 1 mirror maker task with the default settings. Further in the readme you can see how to change these from the defaults.

# ./cli add mirrormaker 0 --executor=executor
Added tasks 0

cluster:
  task:
    type: mirrormaker
    id: 0
    state: inactive
    cpu: 0.50
    mem: 512.00
    configs:
      num.producers: 1
      num.streams: 1
      queue.size: 10000

You now have a cluster with 1 task that is not started.

# ./cli status
cluster:
  task:
    type: mirrormaker
    id: 0
    state: inactive
    cpu: 0.50
    mem: 512.00
    configs:
      queue.size: 10000
      num.producers: 1
      num.streams: 1

Each mirror maker task requires some basic configuration.

# ./cli update 0 --producer.config producer.config --consumer.config consumer.config --whitelist "^mytopic$"
Configuration updated:

cluster:
  task:
    type: mirrormaker
    id: 0
    state: inactive
    cpu: 0.50
    mem: 512.00
    configs:
      num.producers: 1
      num.streams: 1
      queue.size: 10000
      consumer.config: consumer.config
      producer.config: producer.config
      whitelist: ^mytopic$

Now lets start the task. This call to CLI will block until the task is actually started but will wait no more than a configured timeout. Timeout can be passed via --timeout flag and defaults to 30s. If a timeout of 0ms is passed CLI won't wait for tasks to start at all and will reply with "Scheduled tasks ..." message.

# ./cli start 0
Started tasks 0

cluster:
  task:
    type: mirrormaker
    id: 0
    state: running
    cpu: 0.50
    mem: 512.00
    task id: mirrormaker-0-e5ab7581-1c31-d85d-ff14-77b5fb3de4d4
    slave id: 20150914-091943-84125888-5050-10717-S1
    executor id: mirrormaker-0
    attributes:
      hostname: slave0
    configs:
      consumer.config: consumer.config
      producer.config: producer.config
      whitelist: ^mytopic$
      num.producers: 1
      num.streams: 1
      queue.size: 10000

By now you should have a single mirror maker instance running. Here's how you stop it:

# ./cli stop 0
Stopped tasks 0

And remove:

# ./cli remove 0
Removed tasks 0

Typical operations


Shutting down framework

While the scheduler has a shutdown hook it doesn't actually finish the framework. To shutdown the framework completely (e.g. unregister it in Mesos) you may shoot a POST to /teardown specifying the framework id to shutdown:

# curl -d frameworkId=20150807-094500-84125888-5050-14187-0005 -X POST http://master:5050/teardown

Navigating the CLI

Requesting help

# ./cli help
Usage:
  help: show this message
  scheduler: start scheduler
  add: add task
  update: update configuration
  start: start task
  status: get current cluster status
  stop: stop task
  remove: remove task
Get detailed help from ./cli help <command>

Adding tasks to the cluster

# ./cli help add
Usage: add <type> <id-expr> [options]

Types:
    mirrormaker

Options:
    --executor: Executor binary name.
    --api: API host:port for advertizing. Optional if GM_API env is set.
    --cpu: CPUs per task. Defaults to 0.5.
    --mem: Mem per task. Defaults to 512.
    --constraints: Constraints (hostname=like:^master$,rack=like:^1.*$).

id-expr examples:
    0      - task 0
    0,1    - tasks 0,1
    0..2   - tasks 0,1,2
    0,1..2 - tasks 0,1,2
    *      - all tasks in cluster

constraint examples:
    like:slave0    - value equals 'slave0'
    unlike:slave0  - value is not equal to 'slave0'
    like:slave.*   - value starts with 'slave'
    unique         - all values are unique
    cluster        - all values are the same
    cluster:slave0 - value equals 'slave0'
    groupBy        - all values are the same
    groupBy:3      - all values are within 3 different groups

Configuring tasks in the cluster

All file-related configs should be set as file names located . (e.g. setting --producer.config producer.properties) OR http/https URLs (e.g. http://192.168.3.1:6666/producer.config). Please note that http URLs MUST end with file name to be downloaded as Mesos does not respect Content-Disposition header.

# ./cli help update
Usage: update <id-expr> [options]

Options:
    --api: API host:port for advertizing. Optional if GM_API env is set.
    --cpu: CPUs per task.
    --mem: Mem per task.
    --constraints: Constraints (hostname=like:^master$,rack=like:^1.*$).
    --whitelist: Regex pattern for whitelist. Providing both whitelist and blacklist is an error.
    --blacklist: Regex pattern for blacklist. Providing both whitelist and blacklist is an error.
    --producer.config: Producer config url or file name.
    --consumer.config: Consumer config url or file name.
    --num.producers: Number of producers.
    --num.streams: Number of consumption streams.
    --preserve.partitions: Preserve partition number. E.g. if message was read from partition 5 it'll be written to partition 5.
    --preserve.order: E.g. message sequence 1, 2, 3, 4, 5 will remain 1, 2, 3, 4, 5 in destination topic.
    --prefix: Destination topic prefix.")
    --queue.size: Maximum number of messages that are buffered between the consumer and producer.
    --options: Additional options for executor, separated by ';'.
    
id-expr examples:
    0      - task 0
    0,1    - tasks 0,1
    0..2   - tasks 0,1,2
    0,1..2 - tasks 0,1,2
    *      - all tasks in cluster
constraint examples:
    like:slave0    - value equals 'slave0'
    unlike:slave0  - value is not equal to 'slave0'
    like:slave.*   - value starts with 'slave'
    unique         - all values are unique
    cluster        - all values are the same
    cluster:slave0 - value equals 'slave0'
    groupBy        - all values are the same
    groupBy:3      - all values are within 3 different groups

Starting tasks in the cluster

# ./cli help start
Usage: start <id-expr> [options]

Options:
    --api: API host:port for advertizing. Optional if GM_API env is set.
    --timeout: Timeout in seconds to wait until the task receives Running status.
    
id-expr examples:
    0      - task 0
    0,1    - tasks 0,1
    0..2   - tasks 0,1,2
    0,1..2 - tasks 0,1,2
    *      - all tasks in cluster

Stopping tasks in the cluster

# ./cli help stop
Usage: stop <id-expr> [options]

Options:
    --api: API host:port for advertizing. Optional if GM_API env is set.
    
id-expr examples:
    0      - task 0
    0,1    - tasks 0,1
    0..2   - tasks 0,1,2
    0,1..2 - tasks 0,1,2
    *      - all tasks in cluster

Removing tasks from the cluster

# ./cli help remove
Usage: remove <id-expr> [options]

Options:
    --api: API host:port for advertizing. Optional if GM_API env is set.
    
id-expr examples:
    0      - task 0
    0,1    - tasks 0,1
    0..2   - tasks 0,1,2
    0,1..2 - tasks 0,1,2
    *      - all tasks in cluster

Cluster status

# ./cli help status
Usage: status [options]

Options:
    --api: API host:port for advertizing. Optional if GM_API env is set.

About

Go Kafka Client Mesos Framework

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published