Dedicated async job support via opt in

[bessey]

(Opening this PR up to start discussion, it contains some internals that I can remove should it actually get up-streamed. Its unclear to me if this repo is still maintained so am avoiding committing too much time to a perfect PR for now, happy to revisit!)

Background

We have built out a reasonably high throughput internal domain events system using Wisper. Due to the nature of some of our events, its not uncommon for us to see multi-minute queue backlogs when a burst of events are fired. As such, its quite a problem for us to have all these events going into the same queue and wrapped in a single job.

The single queue means we can't prioritise them (we use Sidekiq, with queues of different priorities)

The single job means we cannot easily tell them apart in Datadog APM, so when the Wisper queue is backlogged, identifying the cause is harder than we'd like. Answering questions like "how much time is Sidekiq spending processing all of event X" is much harder than it needs to be too.

Solution

Our solution has been to provide a system where an event listener can opt in to being its own async wrapper job. I can explain further if necessary but I have updated the docs and specs, so hope its fairly self explanatory.

[krisleech]

thanks, how does this compare to #19

[bessey]

For our needs, #19 is not appropriate as we not only want to have jobs go to different queues, we also want different event subscribers to show up differently in observability tools like Datadog APM.

With our solution we can replace the default behaviour (top line in this screenshot) with clearly domain specific jobs (rest of the lines):

You might also argue its more flexible since it exposes ActiveJob's full API. E.g. you might use rescue_from or before_enqueue etc, on a per subscriber basis.