Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
A simple Producer/Consumer client for Kafka (a distributed publish/subscribe messaging service)
branch: master


kafka-rb allows you to produce and consume messages to / from the Kafka distributed messaging service. This is an improved version of the original Ruby client written by Alejandro Crosa, and is used in production at wooga.


You need to have access to your Kafka instance and be able to connect through TCP. You can obtain a copy and instructions on how to setup kafka at

To make Snappy compression available, add

gem "snappy", "0.0.4", :git => "git://", :branch => "snappy-streaming"

to your Gemfile.


sudo gem install kafka-rb

(should work fine with JRuby, Ruby 1.8 and 1.9)


Sending a simple message

require 'kafka'
producer =
message ="some random message content")

Sending a sequence of messages

require 'kafka'
producer =
message1 ="some random message content")
message2 ="some more content")
producer.push([message1, message2])

Batching a bunch of messages using the block syntax

require 'kafka'
producer =
producer.batch do |messages|
    puts "Batching a push of multiple messages.."
    messages <<"first message to push")
    messages <<"second message to push")
  • they will be sent all at once, after the block execution

Consuming messages one by one

require 'kafka'
consumer =
messages = consumer.consume

Consuming messages using a block loop

require 'kafka'
consumer =
consumer.loop do |messages|
    puts "Received"
    puts messages

Using the cli

There is two cli programs to communicate with kafka from the command line interface mainly intended for debug. kafka-publish and kafka-consumer. You can configure the commands by command line arguments or by setting the environment variables: KAFKA_HOST, KAFKA_PORT, KAFKA_TOPIC, KAFKA_COMPRESSION.


$ kafka-publish --help
Usage: kafka-publish [options]

    -h, --host HOST                  Set the kafka hostname
    -p, --port PORT                  Set the kafka port
    -t, --topic TOPIC                Set the kafka topic
    -c, --compression no|gzip|snappy Set the compression method
    -m, --message MESSAGE            Message to send

If message is omitted, kafka-publish will read from STDIN, until EOF or SIG-INT.

NOTE: kafka-publish doesn't bach messages for the moment.

This could be quiet handy for piping directly to kafka:

$ tail -f /var/log/syslog | kafka-publish -t syslog


$ kafka-consumer --help
Usage: kafka-consumer [options]

    -h, --host HOST                  Set the kafka hostname
    -p, --port PORT                  Set the kafka port
    -t, --topic TOPIC                Set the kafka topic

Kafka consumer will loop and wait for messages until it is interrupted.

This could be nice for example to have a sample of messages.


alejandrocrosa at(@)

Something went wrong with that request. Please try again.