Reference implementation for handling a publish-one, subscribe-many pattern
Ruby
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
examples
img
lib
test
.gitignore
.travis.yml
Gemfile
Gemfile.lock
LICENSE
Makefile
README.md
Rakefile
amqp-subscribe-many.gemspec

README.md

Publish One, Subscribe Many

Build Status

Table of Contents

Introduction

This gem is a reference implementation for a Publish One, Subscribe Many pattern which is in use at SoundCloud.

The pattern assumes a single load balanced publish point and multiple direct consumption points as outlined in the follow diagram:

Publish One, Subscribe Many

In this scenario, a worker's single publish connection is load balanced to a broker. The workers have consumer connections to all possible brokers ensuring that no matter which broker a message ends up on the consumer can receive it.

Exchanges, queues, and bindings need to be declared upon connection, ensuring that all brokers will converge on the same set of routing information.

Some buzzwor{th,d}y advantages of this setup include:

  • Partitioned Load
  • Horizontal Scalability
  • High Availability (across the logical topology)

Drawbacks:

  • More complex client code
  • Another hop in the form of the load-balancer
  • A node going down, needs to be brought back up to access the messages

A common error is to conflate availability and durability. In this case, they are treated as separate concerns with the form of availability on offer refering to the ability of all connected clients to get a message from point to point under most conditions.

Durability requires messages to be published as persistent and manual intervention in the case of a node crash to bring the failed node (or disk) back into consumer awareness, to ensure the messages are flushed.

The code in this repository can be used either as a gem available on rubygems.org or as a guide to implement the above pattern using the ruby-amqp gem.

See ./examples/run for usage.

Testing

Run all the tests:

make test

Deprecations

Version 0.2.2:

  • Requiring this library via require 'messaging' is deprecated. Please use require 'amqp_subscribe_many' instead.

Contribute

For any problems, comments or feedback please create an issue here on GitHub.

Licence

amqp-subscribe-many is released under the Mozilla Public License Version 2.0