Full Message Broker Support

[drn]

I'm currently in the process of evaluating message brokers for my company. We're specifically looking for a hosted solution, so we've settled on SNS/SQS over RabbitMQ / Kafka / etc. Looking at the current landscape of SNS/SQS pub-sub libraries for ruby, a few came up but none are very mature. https://github.com/iHiD/propono came up with great a publish API but unfortunately doesn't support an ActiveJob compatible message processing interface. That said, Shoryuken looks like a great option for message processing.

I am planning on building a lightweight pub-sub (message fan-out) library leveraging Shoryuken for message processing. I can either build it as an independent gem or integrate it directly with Shoryuken. Would your team consider merging something like this in or should I develop it separately? If it would be considered, my plan is to implement it without introducing any new dependencies and have it be an entirely optional feature.

Thoughts?

Semi-related issue - #367

[phstc]

Hi @drn

A few releases ago Shoryuken had support for basic SNS operations, but I removed it for keeping Shoryuken simpler, since not everybody use SNS, and most importantly because AWS SDK has a great support for SNS, it's not like SQS, which for consuming messages, it's most likely users want a process running, threads etc.

Do you have a specific need on SNS that the AWS SDK does not handle well?

[drn]

Hey @phstc - thanks for the quick response.

I was mainly looking for an equivalent for https://github.com/ruby-amqp/bunny but for AWS SNS/SQS. The AWS SDKs are great, but they require a good amount of boiler plate code which I don't want my team to have to write every time they want to publish a message or subscribe a queue to a topic.

Given I needed something ASAP, I threw together this gem that handles the use case I am looking to solve -
https://github.com/Thanx/aws-broker. I linked to this gem if you don't mind! This is the boilerplate logic I mentioned - https://github.com/Thanx/aws-broker/blob/master/lib/aws/broker/subscriber.rb#L12

Anyway, let me know if you'd to integrate some of that logic. I have some ideas. Otherwise, no worries - I'll maintain it separately.

Thanks for providing this great message processor!

[phstc]

@drn LGTM. How about CloudFormation for creating topics, subscriptions, policies etc? cfer has a nice DSL for CF. I would try to use CF as much as I could for provision resources in AWS. Change a policy in your code, then cfer converge and all ✨ magically gets updated.

linked to this gem if you don't mind!

Not at all. Thanks 🍻

Thanks for providing this great message processor!

❤️

[kookster]

@drn if this is useful, we made something similar to what you are describing:

https://github.com/PRX/announce/

[ryan-plated]

https://github.com/plated/maitredee For anybody interested in a bit more of a seamless integration with SNS