Single Client can monopolize server with poorly-defined job subject

[carlthuringer]

We had an issue today where a mistake caused blocking jobs to be queued onto the same workflow. When the size of this workflow grew, it caused a resource contention issue on the table, and on the JVM as GC began to intensify. In the end, the entire server was monopolized by repeated calls to ScheduleNextNode for the same workflow.

While we cannot prevent clients from behaving badly and creating massive backlogs of workflows, we can reduce the potential for one job to monopolize the worker pool.

A co-worker found: https://github.com/mhenrixon/sidekiq-unique-jobs. What do you think?

[courtneyb]

I like this idea. Would we want to make the type of locking configurable by backbeat clients? I think we should make sure to include client_id as a unique arg so multiple clients couldn't step on each other with workflows like: { name: "update", subject: {record_uuid: some_uuid } }.

Since we're turning backbeat server into a gem, do you think there are clients that would prefer to add and configure 'sidekiq-unique-jobs' independently of backbeat? Or would we always include it and pass all the configuration through backbeat's configs?

[carlthuringer]

The more I think about this, the more I think it's a step too far. If we have a performance issue around thousands of blocking jobs under one workflow, then we should address that performance issue rather than constructing arbitrary rules around who gets to use how much of the queue, dynamic partitioning, or other strategies for sharing a limited resource.

I'm considering waiting before implementing some heavy-handed resource protection. Sidekiq is really fast. Perhaps all we need is to enforce a timeout on client callbacks and address the performance issue with having thousands of blocking jobs on one workflow.