Note: This project is in early development stage.

This project is aimed to bring Confluent Platform's services, such as Kafka and Schema Registry, to Ruby world (MRI of course, since there's no problem using Java libraries from JRuby). Confluent has done a great job bringing us Kafka REST proxy, a project that helps working with Kafka consumers and producers from any non-JVM platform through REST API, but still: setting it up and using it correctly is not a trivial task. Hopefully, with kafka-rest-rb working with Kafka will be as easy as working with Sidekiq, Hutch or Resque.

Kafka-rest-rb consists of 4 main parts:

• Consumer and Producer DSLs: DSLs for defining consumers and producers within your app
• Producer: producing and sending messages.
• Client: Kafka REST proxy API client.
• Worker: a separate process that consumes and processes messages from Kafka REST proxy.

First, you need to get Zookeeper, Kafka, Kafka REST and Schema Registry (only if you want AVRO-serializing), all of which are included in Confluent Platform, which can be downloaded here.

Note, that theses inctructions are for development environments only. For production deployments refer to Confluent's documentation.

Then, start all services mentioned above in order:

cd confluent-platform-dir
./bin/zookeeper-server-start etc/kafka/zookeeper.properties
./bin/kafka-server-start etc/kafka/server.properties
./bin/schema-registry-start etc/schema-registry.properties # if you need AVRO serialization
./bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties

You might want to create some topics:

kafka-topics --zookeeper :2181 --create --topic my_topic --replication-factor 1 --partitions 1

Or just set auto.create.topics.enable to true in kafka/server.properties.

Run

gem install kafka-rest-rb -v 0.1.0.alpha6

Or just add this to your Gemfile:

gem 'kafka-rest', '0.1.0.alpha6'

Before using kafka-rest-rb you might want to get familiar with some Kafka concepts, such as topics, partitions, offsets, keys, consumer groups etc. You can read about it here.

Include KafkaRest::Producer module to some class in order to make a producer.

class MyProducer
  include KafkaRest::Producer
  
  topic :clicks
  format :json
  key :user_id
end

Include KafkaRest::Consumer module to some class in order to make a consumer from that class. Also, you must implement receive(msg) in consumer class to be able to process messages.

class MyConsumer
  include KafkaRest::Consumer
  
  topic :clicks
  format :json
  
  def receive(msg)
	Click.create(user_id: msg.value['user_id'], url: msg.value['url'])
  end
end

More on the way...