Pipeline

sustrik edited this page Sep 14, 2010 · 9 revisions
Clone this wiki locally

Pipeline pattern allows you to use RabbitMQ as an intermediate storage for messages as they are passed from one processing step in a parallelised pipeline to the next one:

Pipeline pattern distributes messages from its incoming (push) clients among its outgoing (pull) clients. It is the load-balancing that distinguishes it from PubSub.

The diagram below shows the corresponding wiring. The big box in the middle is RabbitMQ server process. Inside there’s a RabbitMQ broker (the top box) and r0mq plugin (the bottom box). AMQP clients are little boxes on the top. 0MQ applications are small boxes at the bottom of the diagram. The shared queue marked in red is the “rendez-vous point” for this messaging pattern. Its name corresponds to the name specified in r0mq configuration:

Following diagram shows various paths that messages traverse:

  1. Red: AMQP pusher to AMQP puller
  2. Magenta: 0MQ pusher to 0MQ puller (via RabbitMQ server)
  3. Green: AMQP pusher to 0MQ puller
  4. Blue: 0MQ pusher to AMQP puller