Skip to content
Modern and performant Kafka client library for Ruby based on librdkafka
Ruby
Branch: master
Clone or download
cosmo0920 and thijsc Refer CPPFLAGS and LDFLAGS when provided on macOS (#102)
Don't override CPPFLAGS and LDFLAGS if already set
Latest commit d0d71f6 Jan 6, 2020
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
ext
lib Improve consumer doc Nov 14, 2019
spec
.gitignore
.travis.yml
.yardopts
CHANGELOG.md Fix whitespace in changelog Oct 30, 2019
Gemfile Basic FFI setup Mar 23, 2017
LICENSE Update readme and add MIT license file Aug 28, 2017
README.md Remove support for Ruby 2.3 which is EOL Sep 9, 2019
Rakefile
docker-compose.yml Fix watermarks spec Feb 28, 2019
rdkafka.gemspec Allow use of Rake 13.x and up Nov 12, 2019

README.md

Rdkafka

Build Status Gem Version Maintainability Test Coverage

The rdkafka gem is a modern Kafka client library for Ruby based on librdkafka. It wraps the production-ready C client using the ffi gem and targets Kafka 1.0+ and Ruby 2.4+.

rdkafka was written because we needed a reliable Ruby client for Kafka that supports modern Kafka at AppSignal. We run it in production on very high traffic systems.

This gem only provides a high-level Kafka consumer. If you are running an older version of Kafka and/or need the legacy simple consumer we suggest using the Hermann gem.

The most important pieces of a Kafka client are implemented. We're working towards feature completeness, you can track that here: https://github.com/appsignal/rdkafka-ruby/milestone/1

Installation

This gem downloads and compiles librdkafka when it is installed. If you have any problems installing the gem please open an issue.

Usage

See the documentation for full details on how to use this gem. Two quick examples:

Consuming messages

Subscribe to a topic and get messages. Kafka will automatically spread the available partitions over consumers with the same group id.

config = {
  :"bootstrap.servers" => "localhost:9092",
  :"group.id" => "ruby-test"
}
consumer = Rdkafka::Config.new(config).consumer
consumer.subscribe("ruby-test-topic")

consumer.each do |message|
  puts "Message received: #{message}"
end

Producing messages

Produce a number of messages, put the delivery handles in an array and wait for them before exiting. This way the messages will be batched and sent to Kafka in an efficient way.

config = {:"bootstrap.servers" => "localhost:9092"}
producer = Rdkafka::Config.new(config).producer
delivery_handles = []

100.times do |i|
  puts "Producing message #{i}"
  delivery_handles << producer.produce(
      topic:   "ruby-test-topic",
      payload: "Payload #{i}",
      key:     "Key #{i}"
  )
end

delivery_handles.each(&:wait)

Development

A Docker Compose file is included to run Kafka and Zookeeper. To run that:

docker-compose up

Run bundle and cd ext && bundle exec rake && cd .. to download and compile librdkafka.

You can then run bundle exec rspec to run the tests. To see rdkafka debug output:

DEBUG_PRODUCER=true bundle exec rspec
DEBUG_CONSUMER=true bundle exec rspec

After running the tests you can bring the cluster down to start with a clean slate:

docker-compose down

Example

To see everything working run these in separate tabs:

bundle exec rake consume_messages
bundle exec rake produce_messages
You can’t perform that action at this time.