Skip to content
A simple Producer/Consumer client for Kafka (a distributed publish/subscribe messaging service)
Find file
New pull request
Fetching latest commit...
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


kafka-rb allows you to produce and consume messages to / from the Kafka distributed messaging service. This is an improved version of the original Ruby client written by Alejandro Crosa, and is used in production at wooga.


You need to have access to your Kafka instance and be able to connect through TCP. You can obtain a copy and instructions on how to setup kafka at

To make Snappy compression available, add

gem "snappy", "0.0.4", :git => "git://", :branch => "snappy-streaming"

to your Gemfile.


sudo gem install kafka-rb

(should work fine with JRuby, Ruby 1.8 and 1.9)


Sending a simple message

require 'kafka'
producer =
message ="some random message content")

Sending a sequence of messages

require 'kafka'
producer =
message1 ="some random message content")
message2 ="some more content")
producer.send([message1, message2])

Batching a bunch of messages using the block syntax

require 'kafka'
producer =
producer.batch do |messages|
    puts "Batching a send of multiple messages.."
    messages <<"first message to send")
    messages <<"second message to send")
  • they will be sent all at once, after the block execution

Consuming messages one by one

require 'kafka'
consumer =
messages = consumer.consume

Consuming messages using a block loop

require 'kafka'
consumer =
consumer.loop do |messages|
    puts "Received"
    puts messages

Using the cli

There is two cli programs to communicate with kafka from the command line interface mainly intended for debug. kafka-publish and kafka-consumer. You can configure the commands by command line arguments or by setting the environment variables: KAFKA_HOST, KAFKA_PORT, KAFKA_TOPIC, KAFKA_COMPRESSION.


$ kafka-publish --help
Usage: kafka-publish [options]

    -h, --host HOST                  Set the kafka hostname
    -p, --port PORT                  Set the kafka port
    -t, --topic TOPIC                Set the kafka topic
    -c, --compression no|gzip|snappy Set the compression method
    -m, --message MESSAGE            Message to send

If message is omitted, kafka-publish will read from STDIN, until EOF or SIG-INT.

NOTE: kafka-publish doesn't bach messages for the moment.

This could be quiet handy for piping directly to kafka:

$ tail -f /var/log/syslog | kafka-publish -t syslog


$ kafka-consumer --help
Usage: kafka-consumer [options]

    -h, --host HOST                  Set the kafka hostname
    -p, --port PORT                  Set the kafka port
    -t, --topic TOPIC                Set the kafka topic

Kafka consumer will loop and wait for messages until it is interrupted.

This could be nice for example to have a sample of messages.


alejandrocrosa at(@)

Something went wrong with that request. Please try again.