Skip to content

Clarifications for Kafka vs RabbitMQ

Luke Bakken edited this page Nov 13, 2019 · 2 revisions

I currently work as a member of Team RabbitMQ. We're responsible for all things RabbitMQ:

  • Core Engineering
  • Documentation
  • Customer Support
  • Free Support for OSS users

I subscribe to the rabbitmq tag on Stack Overflow and monitor the questions asked. The other day, I came across this question:

Message queue (like RabbitMQ) or Kafka for Microservices?

The main answer copied the contents of "Kafka vs RabbitMQ", published in 2018. This wiki page will address several incorrect statements made about RabbitMQ.

Both solutions are highly available, but Kafka took that a step further...

RabbitMQ has provided clustering and highly available queues for several major versions. Version 3.8.0 shipped with "Quorum Queues" which use the Raft consensus algorithm to provide data replication with higher performance than "classic" HA queues. Using a separate service like Zookeeper (known to be difficult to manage) is not an advantage, and the Kafka team recognizes this as they are working to remove Zookeeper as a dependency.

In RabbitMQ queues aren’t automatically replicable, this need to be configured.

Version 3.8.0 simplifies this. If your cluster has at least three nodes, all you must do is declare your queue as a Quorum Queue and replication is taken care of for you.

In RabbitMQ you can configure messages to be persistent, mark the queue as durable and messages as persistent, From the docs: The persistence guarantees aren’t strong

This is false. If you publish a persistent message to a durable queue, and correctly wait for the publisher confirmation, you can be 100% certain your message has been routed to a queue and written to disk. In cluster scenarios, you can be certain that all replicas have been written to disk.

Monitoring ... In RabbitMQ you have a built-in management UI (default <host_name>:15672)

This is only a small part of the monitoring options available. Please see the RabbitMQ Monitoring guide. Also note that version 3.8.0 supports external monitoring via Prometheus and Grafana.

"Let's Recap" table incorrect information:

  • "Distributed leader to manage cluster state" - this is not an advantage, as discussed above.
  • "Msg persistence - from the docs the persistence guarantees aren't strong" - I have no idea what docs are being referenced. Publish a persistent message to a durable queue and wait for the confirmation, and the guarantee is strong.
  • "Push/pull model" - I have no idea why this is marked as a red X in the RabbitMQ column. RabbitMQ consumers can either process an unbounded set of messages or they can use prefetch to specify a maximum number of unacknowledged messages to be sent.

"Use Kafka if you need..."

The following items from the Kafka list also apply to RabbitMQ:

  • "Many consumers for the same message" - Use a fanout exchange.
  • "High throughput" - this is obviously environment and workload specific, but RabbitMQ is capable of delivering tens of thousands of messages per second.
  • "Replicability" - both mirrored and quorum queues provide this feature in RabbitMQ.
  • "High availability" - again, RabbitMQ provides this.
  • "Message order" - this is confusing as the blog author should have called this "Message order across shards". RabbitMQ requires the use of plugins to provide this, if you need it.