Concurrency for consumers

[kapso]

We are trying to replace some of the Sidekiq use cases with Kafka and came across this gem, and it looks great given its similarity to Sidekiq for consuming messages.

The question I have is -

• is there an option to add concurrency to racecar consumers
• how does this gem compare to Sidekiq/Redis when it comes to performance

Thanks.

[dasch]

Concurrency is achieved by adding more consumer processes to the consumer group. There's currently no support for in-process concurrency (using e.g. threads), which is tricky because of the way Kafka checkpointing ("offset commits") work.

[dasch]

Performance is great. But note that Kafka consumers are message processors, not job runners – this means that data flows in a pipeline rather than being scheduled potentially out-of-order. I would not use Kafka for a job queue.

[Adithya-copart]

@dasch Would racecar be willing to support concurrent consumers now or in the future?
Something like #160.

Even though it makes sense to use separate processes, the startup and memory costs associated with spawning separate processes is higher in Ruby implementations like JRuby. The same could be true for applications that load multiple files in each racecar process with lighter kafka throughputs.

I agree that this does not add value for C-Ruby applications with significant kafka throughputs but do you see value in supporting use-cases like JRuby?