Getting Started
Important
GitHub Wiki is just a mirror of our online documentation.
We highly recommend using our website docs due to Github Wiki limitations. Only some illustrations, links, screencasts, and code examples will work here, and the formatting may be broken.
Please use https://karafka.io/docs.
-
Make sure Apache Kafka is running. You can start it by following instructions from here.
-
Add Karafka to your Gemfile:
# Make sure to install Karafka 2.4 as Karafka 1.4 is no longer maintained
bundle add karafka --version ">= 2.4.0"
- Install Karafka (works for both Rails and standalone applications) by running:
bundle exec karafka install
the above command will create all the necessary files and directories to get you started:
-
karafka.rb
- main file where you configure Karafka and where you define which consumers should consume what topics. -
app/consumers/example_consumer.rb
- example consumer. -
app/consumers/application_consumer.rb
- base consumer from which all consumers should inherit.
- After that, you can run a development console to produce messages to this example topic:
# Works from any place in your code and is thread-safe
# You usually want to produce async but here it may raise exception if Kafka is not available, etc
Karafka.producer.produce_sync(topic: 'example', payload: { 'ping' => 'pong' }.to_json)
- Run the karafka server to start consuming messages:
bundle exec karafka server
# example outcome
[7616dc24-505a-417f-b87b-6bf8fc2d98c5] Polled 2 messages in 1000ms
[dcf3a8d8-0bd9-433a-8f63-b70a0cdb0732] Consume job for ExampleConsumer on example started
{"ping"=>"pong"}
{"ping"=>"pong"}
[dcf3a8d8-0bd9-433a-8f63-b70a0cdb0732] Consume job for ExampleConsumer on example finished in 0ms
Here's the demo of the installation process:
- (Optionally) Install and configure the Web UI by following this documentation section.
If you have any problems settings things up or want a ready application to play around with, then the best idea is just to clone our examples repository:
git clone https://github.com/karafka/example-apps ./example_apps
and follow the instructions from the example apps Wiki.
Karafka ships with a full integration test suite that illustrates various use-cases and edge-cases of working with Karafka and Kafka. Please visit this directory of the Karafka repository to see them.
Home | Getting started | Components | Producing Messages | Consuming messages | Concurrency and multithreading | Error handling and back off policy
This wiki is tracked by git and publicly editable. You are welcome to fix errors and typos. Any defacing or vandalism of content will result in your changes being reverted and you being blocked.