Perform some kind of heartbeating for workers

[mschubert]

If workers crash because they run out of memory (without ulimit protection) or call C code that is not caught by try(...), they just disappear and the master never completes.

Find an appropriate way to check if workers are up. Maybe combined this with PUSH/PULL sockets for work and REQ/REP for control (#30)

[mschubert]

Note: this could also be handled by setting TCP keepalive socket options (that are accessible via ZeroMQ).

The defaults are to disconnect after just over 2 hours, so this is too long. Customizing these options per socket is possible using

• TCP_KEEPCNT
• TCP_KEEPIDLE
• TCP_KEEPINTVL

on the OSs (source)

• Linux >= 2.4 (2001)
• OS X >= 10.7.0 ("Lion"; 2011)
• Windows >= 10.0.1709 (original 2015; release October 2017)

[mschubert]

Update: TCP keepalive does not work because disconnected peers will not raise a ZeroMQ message or exception. Also, killed (kill -9) workers still send a FIN packet to the master, which probably cleans up the low-level connection correctly, but we have no access to this via the ZeroMQ API.

Alternatively, application-level heartbeats could be used (ZeroMQ>=4), which would break e.g. Ubuntu support.

[mschubert]

I should also consider worker timeouts: If they miss heartbeats, also shut them down (irrespective of running computations) (ref: #131)

[mschubert]

Closing because #150 is a better approach