Limit workers

[hphilipps]

This PR is adding a config option MaxWorkerInstances to limit the number of concurrently executed tasks per worker. MaxWorkerInstances=0 (default) allows an unlimited number of concurrent task executions, so the default behavior of machinery stays the same. MaxWorkerInstances=1 essentially serializes task execution (which is a feature that I need).
Limiting is implemented for the Redis and the AMQP brokers.

All tests are passing with these changes. I would be glad if this could be merged. Let me know if there is anything else I can do to make that happen.

[RichardKnop]

@hphilipps Thanks for PR. I will take a look tomorrow and merge if all is good. Looking great at first glance though! 👍

[RichardKnop]

@hphilipps Sorry it took so long. I was quite busy. Merged! 👍

[codepushr]

@hphilipps @RichardKnop Does this change also work on a 'per Chain' level? I would like to set an option that a worker can only process X chains (with multiple tasks) at once. The motivation comes from chains that have long running tasks and I want a whole chain to complete before another random task gets processed.

I'm still new to Go but would collaborate if you could broadly describe what to do. :)

[RichardKnop]

I don't think this works on a chain level, just per task.

Current workaround would probably be to use RoutingKey to route tasks to specific workers. This would work with AMQP broker though. If you are using Redis not sure there is a straightforward way to achieve this.

[codepushr]

@RichardKnop Sorry to bother you again, but isn't MaxWorkerInstances a global config? If I understood correctly you would use a separate worker with MaxWorkerInstances=1 for the long running tasks in my chains, and a usual worker with MaxWorkerInstances=0 for all other tasks.