Kestrel Workers
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.




Bullfinch is a worker queue that uses kestrel as a medium for storage. Bullfinch spins up worker threads that monitor and respond to messages in kestrel queues.

  1. Client connects to kestrel and puts a json-formatted message into a queue. In this message it includes the name of the queue where it will look for a response.

  2. A worker receives the message, does whatever work it needs to do, and puts one or more items into the queue specified by the client.

  3. The client pulls the items out and deletes the queue when it gets the EOF message.


Bullfinch uses kestrel-client and therefore only speaks Kestrel's text protocol.


We needed a way to bridge the gap between a new system and a legacy one. We have a future use-case for kestrel and wanted to get it into use. We also needed some way to interface with a legacy RDBMS so we wrote Bullinch and the JDBCWorker to solve that problem.


Bullfinch works like this:

                                                  /---| Worker A |--+
                                                 /    +-----------  |
                               +---------------+/     +----------+  |
                            /->| Request Queue |----->| Worker B |--+
+--------+      +---------+/   +---------------+      +----------+  |
| Client |----->| Kestrel |                                         /
+--------+      +---------+<   +----------------+                  /
                            \--| Response Queue |<----------------/

Crystal clear, right?

The Client puts an item into a queue (formatted as json) in the form of:

    "response_queue": "name_of_queue_you_will_read_for_response",
    "other": "stuff",
    "whatever": "you want"

The worker and it's thread(s) are configured to watch a particular queue (see CONFIGURATION). Each worker will receive an item from the queue (in some sort of fair, but random, order) by way of it's handle method. This method will return an Iterator of strings that will be fed into whatever "response_queue" was specified in the original request. (Each client should probably use a unique name, "response_" + pid or something) After your worker returns, bullfinch will confirm the item you handled. The client can then draw the items from the response_queue (whatever it is). How long it will be before the item(s) show up there depends on what your worker is doing. That's your problem.

The last record will be followed by an EOF like this:

{ "EOF":"EOF" }

Generally speaking, the best idea is for you to DELETE the response queue after you encounter the above record. Since it only exists to satisfy the request you specified.


Check out the docs directory.