pubsub: generic type for asynchronous messaging

[zombiezen]

A common feature request is asynchronous messaging, like Pub/Sub. More research and design work is required to know exactly what this API should look like, but any design should work for (not necessarily saying it will be implemented):

• Amazon SQS
• Google Cloud Pub/Sub
• Azure Service Bus
• Open source standalone: RabbitMQ, Kafka, Redis

(Worth noting that the Contribute Bot proposed in #216 uses Pub/Sub as well: design doc.)

[asim]

We abstract pubsub as a Broker interface in go-micro which includes support for google pubsub, amazon sqs, kafka, redis, rabbitmq, nsq, nats, etc. It may help you understand how you want to start this https://github.com/micro/go-micro/blob/master/broker/broker.go

[ijt]

@zombiezen, nit: pubsub on AWS is SNS (simple notification service), sometimes combined with SQS (simple queuing service).

[ijt]

@asim, I'm not sure I understand how Broker in go-micro supports all the backends you listed. Can you say a bit more about how that works?

[asim]

See code in micro/go-plugins repo. Broker is an interface which provides a high level abstraction for pubsub. Go Plugins contains implementations of that interface for various messaging systems. These can be used anywhere a broker interface is accepted. You do lose some unique per message provider features but it also creates a simple pluggable messaging layer.

I imagine you guys will likely do something a bit different more similar to Blob and it's setup code.

[ijt]

Thanks @asim!

[ijt]

@asim, I'm surprised that SQS is used as a backend by itself since SQS is just a queuing service, not a pubsub service. I would have thought it would be a combination of SNS + SQS. How does this work?