Support batching

[bedrock-adam]

Problem

Redis and Kafka support batching at the protocol / network level, but Outboxer does not take advantage of this.

Currently the interface is Outboxer::Publisher.publish_message { |message| } so this is not compatible.

Proposed solution

Support new interface

Outboxer::Publisher.publish_messages(batch_size: 100) do |messages|
  # TODO: publish messages in bulk here
end

Sidekiq

Outboxer::Publisher.publish_messages(batch_size: 100) |messages|
  Sidekiq::Client.push_bulk(
    'class' => EventCreatedJob,
    'args'  => messages.map { |message| [message[:messageable_id]] })
end

Bunny

Outboxer::Publisher.publish_messages(batch_size: 100) do |messages|
  messages.each do |message|
    bunny_channel.default_exchange.publish(
      JSON.dump(message), routing_key: "outbox.messages")
  end
end

Kafka

Outboxer::Publisher.publish_message(batch_size: 100) do |messages|
  messages.each do |message|
    kafka_producer.produce(
      JSON.dump(message), topic: "outbox")
  end

  kafka_producer.deliver_messages
end