Multi dispatchers

[hughjfchen]

Problem statement

In a real deployment, the number of remote hosts which dispatcher manage may increase to hundreds or thousands. As the number keeps increasing, separate hosts by function/group may ease the management and scalability. However, current dispatcher implementation may cause some conflict when updating jobs status in a multi dispatchers environment.

Steps to reproduce

1. Run multi dispatchers
2. submit some jobs
3. job may run multi times or job status may not correct

Expected result

Job only run once and the status updated accordingly.

Actual result

Job may run multi times or the status not right.

batchd version

Github master

Please specify whether you are using installation from hackage, or building source from github.
Built from github master.