Assume a queue of messages. Each message belongs to exactly one group. The goal is to execute messages of the same group in sequence. Messages of different groups are allowed to be executed in parallel.
All solutions rely on Apache Camel 2.23.x, a widely used integration framework.
[A] ActiveMQ - Message groups
mvn clean verify -U -Pa-activemq
- Documentation: http://activemq.apache.org/message-groups.html
- Implementation: A_ActiveMQ_IT.java
Note that this does not support safe dynamic rebalancing of consumers without external synchronization. We can reset a message group (by setting
-1), but we must first make sure that there are no pending messages for that particular group.
[B] RabbitMQ - Consistent hash exchange
mvn clean verify -U -Pb-rabbitmq
- Documentation: https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange
- Implementation: B_RabbitMQ_IT.java & definitions.json
Note that this does not support safe dynamic rebalancing of consumers without external synchronization. We can alter the routing topology by adding/removing a queue, but we must first make sure that there are no pending messages at all.
[C] RabbitMQ/ActiveMQ & sticky load balancer & in-memory SEDA queues
mvn clean verify -U -Pc1-rabbitmq&
mvn clean verify -U -Pc2-activemq
- Documentation: http://camel.apache.org/load-balancer.html & http://camel.apache.org/seda.html
- Implementation: C1_RabbitMQ_IT.java & definitions.json / C2_ActiveMQ_IT.java
Note that this does not support safe dynamic rebalancing of consumers.
Also note that messages could be lost after a service crash or restart! A message will be acknowledged once it has been added to the SEDA queue, as the route ends at that point. In theory, we could alter the implementation to manually acknowledge the message after successful asynchronous execution, but we currently face the following limitations in Apache Camel:
- Asynchronous processing is currently not supported.
- Manual acknowledgment is currently not supported.
- Asynchronous transactions are scheduled for Camel 3.0, as documented here.
- As an alternative to asynchronous transactions, we could use asynchronous processing (which can be enabled via the
asyncConsumer=trueproperty, as documented here) and the
INDIVIDUAL_MESSAGEacknowledgment mode, as documented here. Unfortunately, the
INDIVIDUAL_MESSAGEacknowledgment mode is currently not supported, as reported here.
This is not (yet) implemented here. This should be fairly trivial, as consumer groups are natively supported.
This should support safe dynamic rebalancing of consumers.
- Anything from https://jack-vanlightly.com. His blog and webinars are golden, highly recommended. In particular: