Multi-threaded server message endpoints

[Shillaker]

Currently our server message endpoints (FunctionCallServer, StateServer, and SnapshotServer) are backed by a single worker in a single thread, which makes them a potential bottleneck. This is particularly likely in a multi-tenant scenario where we might have a few applications running, all modifying state, chaining functions, or pushing snapshots simultaneously.

This PR converts both our "sync" and "async" server endpoints to being multi-threaded via a fan-in/ fan-out approach (i.e. all incoming requests still go to the same socket as before, but under the hood they're shared between several worker threads).

The flow of messages through sockets is:

1. Incoming remote requests (PUSH/ REQ from remote host).
2. Fan-in (all requests received on a single PULL/ ROUTER socket).
3. Fan-out (proxy between fan-in and a single PUSH/ DEALER connected to an inproc:// socket).
4. Workers (multiple PULL/ REPs connected to the inproc:// socket of the fan-out).

Changes:

• Use 0MQ ROUTER/ DEALER sockets for fanning out incoming REQ requests to multiple downstream REPs (example in docs).
• Use an interim PULL/ PUSH pair of sockets for fanning out incoming PUSH requests to multiple downstream PULLs (info here).
• Connect both fan-in/ fan-out sockets with a proxy_steerable. This allows us to stop the proxy programatically without a signal.
• Update server shutdown process to send a TERMINATE message to the proxy, followed by shutdown messages to each worker one-by-one.
• Make number of threads used in function call, snapshot and state servers configurable.

[csegarragonz]

One high-level observation is why are we using a thread-pool, and not assigning a different inproc address to each thread in our already existing thread pool.

I am concerned that ZeroMQ's scheduling is out of our control, and we rely on setting an arbitrarily adequate number of working threads, and we could always find a corner-case to break it (as it happened with MPI async layer).
As far as I can tell the current execution flow is (correct me if I am wrong):

• Local executor thread blocks due to a distributed coordination operation.
• Underlying transport thread (randomly assigned to this executor) blocks.
• Remote executor thread unlocks the local executor thread, thus sends a message to notify.
• A different transport thread logically unlocks the local transport thread that was locked.
  Main concern with this design is that it strongly depends on the availability of a free transport thread, and on ZeroMQ actually scheduling work to threads that are not blocked, which again is out of our control.

My alternative would be to give each executor in faabric an inproc socket, or use a PUB/SUB scheme over the fan-out in-proc fabric with topic being something like appid/<function/snapshot/state>/exec-id.

Then, implement the distributed coordination operations as transport primitives. This is, use ZeroMQ's blocking functionality to implement distributed barriers.

I have only drafted the idea here, happy to dismiss or discuss offline, as I am sure the reasoning needs a lot of polishing.