Latest commit daa6913
Python ZeroMQ Demos These are my initial attempts to build some proof-of-concept experiments using 0MQ's Python bindings. WARNING Due to the way CPython handles signals and the GIL, it is imperative that the main Python thread (the only thread that handles signals), is *not* used to run methods that may block for long periods (eg. thread.join(), mutex locks, etc.). Do these things in other threads, leaving the Main thread open to handle signals (eg. SIGINT). From: http://www.dabeaz.com/python/GIL.pdf, page 25. pool One key problem we need to solve is to allow any one server in a pool of services on a host, send a request to *all* such services -- both on its own host, plus possibly on a number of other hosts. The problems are: 1) We don't want every server connecting to one server; ideally, they would all elect one of themselves as a "master" on the host, and connect to that one. In turn, all the "master" servers on every host would connect, and distribute any requests out. All responses would make their way back to the issuer of the request. 2) Server failure is fine, but we need to know when all responses (from every non-failed server) have been collected. We can simply wait and hope that each server eventually either fails and disconnects, or returns a response -- but this would not account for "hung" servers. So, we need to implement an out-of-band "ping" with a response timeout to test for live-ness. Alternatievely, we could have timely "working" messages returned, followed eventually by a response (or abandonment of the failed server). 3) Detection and mitigation of a failed "master" node is more difficult; if it is hung, then all other servers on the host are cut off from communicating. Furthermore, they cannot re-elect a new master and re-bind -- because the interface:port remains bound by the disabled "master"... If a quorum of servers on a host decide that their "master" is hung, perhaps they should decide to kill it? assigner One of the libraries we use behind a webserver improperly conflates the ideas of a Hit and a Thread. Each API call used during processing of a Hit in the web-page code has (traditionally) been handled by the same thread, from start to finish. We wish to un-embed this library from within the web server (because the IIS webserver is broken/misdesigned in ways that prevent us from debugging our embedded native-code library). Using an RPC mechanism (WCF, Zero-C ICE, ...) is possible, but some of the API deals in (very) large binary object transfers, so 0MQ seemed in some ways like a better fit. This unfortunate Hit==Thread assumption is probably too difficult to pry out of the existing code, so we must ensure that when a Hit is started in the Web Server, a remote server is assigned to carry out all the requests for the Hit, using a single thread assigned for the complete duration of the Hit. Each server may be servicing a multitude of such Hits, simultaneously. RPC Thread Pools vs. Assigned Server This "Assigned Server Thread" model doesn't match well with the common "Thread Pool" designs of RPC libraries. Typically, they have a stable of threads awaiting the next incoming RPC request, and then carry out the local method invocation against the correct object. The correct serialized set of method invocations indeed occurs against the correct local object -- but a different Thread might be used to perform each one! Techniques like Zero-C ICE "AMD" (Asynchronous Method Dispatch) can be used to transfer responsibility for the incoming method invocation from the RPC thread(s) to the assigned Hit thread, but this adds yet another layer of complexity to an already complex solution. Using 0MQ, we could establish a persistent connection to 1 to N brokers by supplying all of their addresses to multiple connect calls, using the same socket. 0MQ transparently distributes individual messages (requests) to the servers, using Nagle's Fair Queueing algorithm (unfortunately, it doesn't perform weighted fair queueing, like TCP connects based on SRV records would.) Also, the connections are persistent, and are transparently established and maintained by 0MQ, behind the scenes. Any requests sent to crashed servers are lost, so end-to-end out-of-band "ping" or "i-am-alive" messages would be required (by the client end) to ascertain liveness. But most importantly -- each request is transparantly fair-queued to *any* available server; subsequent requests would almost never go to the same thread (or even the same server process, on the same box!) Therefore, we need to use a two step process: 1) Allocate a server thread using a 0MQ REQ/REP socket connected to every server broker. The response contains a routing path to the responding server thread. The path must contain the socket ID of the broker, and the socket ID of the server thread. 2a) Route messages from the client to the server via XREP/XREQ (NOTE reversal from normal usage!) the broker recognizes. or 2b) Create a PAIR socket, and connect directly to a specific server's public PAIR socket. If we use straight 0MQ REQ/REP to select a server. Problem is, the multitude of webserver processes must connect to the multitude of servers; we don't want to have that; nor do we want to manage a N:1 queue "forwarder" for each webserver (IIS App Pool) host -- an incoming Hit from one of the N IIS webservers must *directly* connect to exactly one of the M available servers. So, a 1:1 0MQ REQ/REP socket will be created and destroyed for each Hit. PROTOTYPE (incomplete, but operational) The resultant prototype uses 2 zmq.REQ sockets in the client; one for requesting the address of a worker, and one to send work to it. WARNING: This will *only* work with a single broker; we need to get the broker's address, as well as the server address, and use a different socket type to communicate to the server... assigner/aserver_scale1.py 'WAITING' --> waiting 'WAITING' <address> <-- waiting <address> 'Hello' --> broker 'World' <-- broker assigner/aclient_scale1.py client broker (server thread) ------ ---------------------- --------------- --> waiting --> | [idle] <-- server <address> <-- waiting <-- | <address> work --> broker --> server .... response <-- broker <-- server Run: $ python aserver_scale1.py & $ python aclient_scale1.py & # As many times as you want... You'll need to kill the server; it doesn't yet respond to keyboard interrupts and shut down cleanly.