Skip to content

Standalone Karafka library for producing Kafka messages

License

Notifications You must be signed in to change notification settings

bruce-szalwinski-he/waterdrop

 
 

Repository files navigation

WaterDrop

Build Status Gem Version Join the chat at https://slack.karafka.io

WaterDrop is a standalone gem that sends messages to Kafka easily with an extra validation layer. It is a part of the Karafka ecosystem.

It:

  • Is thread-safe
  • Supports sync producing
  • Supports async producing
  • Supports transactions
  • Supports buffering
  • Supports producing messages to multiple clusters
  • Supports multiple delivery policies
  • Works with Kafka 1.0+ and Ruby 2.7+
  • Works with and without Karafka

Documentation

Karafka ecosystem components documentation, including WaterDrop, can be found here.

Getting Started

If you want to both produce and consume messages, please use Karafka. It integrates WaterDrop automatically.

To get started with WaterDrop:

  1. Add it to your Gemfile:
bundle add waterdrop
  1. Create and configure a producer:
producer = WaterDrop::Producer.new do |config|
  config.deliver = true
  config.kafka = {
    'bootstrap.servers': 'localhost:9092',
    'request.required.acks': 1
  }
end
  1. Use it as follows:
# sync producing
producer.produce_sync(topic: 'my-topic', payload: 'my message')

# or for async
producer.produce_async(topic: 'my-topic', payload: 'my message')

# or in sync batches
producer.produce_many_sync(
  [
    { topic: 'my-topic', payload: 'my message'},
    { topic: 'my-topic', payload: 'my message'}
  ]
)

# and async batches
producer.produce_many_async(
  [
    { topic: 'my-topic', payload: 'my message'},
    { topic: 'my-topic', payload: 'my message'}
  ]
)

# transactions
producer.transaction do
  producer.produce_async(topic: 'my-topic', payload: 'my message')
  producer.produce_async(topic: 'my-topic', payload: 'my message')
end

About

Standalone Karafka library for producing Kafka messages

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Ruby 100.0%