Atomic, bite-size, highly-focused interactive tutorials that teach you how to work with Kafka
Switch branches/tags
Nothing to show
Clone or download
Latest commit 70cee1b Nov 8, 2018
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci Remove .sh extension from all shell scripts Oct 15, 2018
ruby Merge pull request #51 from FundingCircle/update-kafka-clients Nov 8, 2018
LICENSE Fill in blanks in license Jan 5, 2018 Remove .sh extension from all shell scripts Oct 15, 2018

Learn You Some Kafka!

Atomic, bite-size, highly-focused hands-on exercises that teach how to work with Kafka using Ruby or Clojure.

Quick Start


Do these steps first, regardless of which programming system and/or runtime you wish to use:

  1. Ensure:
    1. Docker is running
    2. You’re connected to the Internet
    3. Your Internet connection is fairly fast
  2. Clone this repo


Coming soon!


(CRuby is the reference implementation of Ruby. (If you’re interested in the various names used for various implementations of Ruby, and why, see this thorough article.))

From the root of the repo, run:

cd ruby/cruby/unit_1 && ./start


(JRuby is the predominant implementation of Ruby for the JVM.)

From the root of the repo, run:

cd ruby/jruby/unit_1 && ./start


This is a work in progress!

This table shows which units are currently available for which programming system:

Unit Clojure CRuby JRuby
1: Producing and Consuming
2: Avro
3: Stream Processing
4: Kafka Connect  

For more detailed status, please see our projects.


Engineers who are already proficient with Ruby and/or Clojure but may have little or no hands-on experience with Kafka.


Provide hands-on exercises that help these engineers learn how to work with Kafka and streaming data using the practices and principles in use at Funding Circle.


  1. Introduce and explain Kafka’s basic model

The Plan


  1. A basic high-level understanding of Kafka’s basic model
  2. Having a recent and working installation of Docker and docker-compose
  3. Clojure
    1. Proficient with opening a clj file in an editor, connecting the editor to a REPL, and evaluating individual forms in the editor
  4. Ruby
    1. No particular prerequisites


  • Atomic, bite-size, highly-focused exercises
  • Each exercise gives you one new main piece of useful knowledge related to working with Kafka and streaming data
  • Published as a GitHub repository
  • The recommended approach will be for people to clone the repo and then work through the exercises one at a time, in order
  • When someone opens the main code file for an exercise, the specific process/experience/workflow they follow will vary by programming system:
    • Ruby:
      • They run the exercise in a terminal using ./start and they then enter into an interactive lesson which alternates between printing explanatory text and some code that will execute, and then running that code and displaying its results.
    • Clojure:
      • They start reading from the top of the comment form at the top of the file, and evaluate each form inside the comment, as they proceed.
      • As they proceed, they’ll fluidly read about an operation, then evaluate and see the results of that operation.

Units and Lessons

Each unit and lesson will be available for both Ruby and Clojure.

Unit One: Producing and Consuming

  1. Producing Strings
  2. Consuming Strings
  3. Consumer Groups
  4. Serialization and Deserialization

Unit Two: Avro

  1. Producing Avro
  2. Consuming Avro
  3. Producing and Consuming Avro with a Schema Registry

Unit Three: Stream Processing

  1. Linear Stream Processing with a Consumer and a Producer
  2. Linear Stream Processing with Kafka Streams
  3. Stream Processing with Joins
  4. Stream Processing with Windows

Unit Four: Kafka Connect

  1. What the hell is it even
  2. TBD

Guidance to Those Wishing to Contribute Units/Lessons

Coming soon!

Running the Tests


cd ruby/test

# Test all units:

# OR test a specific unit or units:
./test_units [path ...]

# For example:
./test_units ../jruby/unit_1/