✊ Handjob: Low-latency Realtime Job Broker

Hand job over to remote workers and get the result immediately via realtime streaming over websockets.

---

📖 Table of Contents

• Motivation
• Architecture
• Protocol

---

Motivation

I wanted to add OCR functionality to my app, but bundling the python interpreter + PyTorch + libraries would bloat the installer size to over 1 GB. This is a bad experience for end users. I thought about moving the OCR functionality to a cloud service, but renting a gpu instance is extremely expensive. Therefore I settled on offloading the processing work to my gaming PC at home. This job broker facilitates that system in a secure and efficient way.

---

🏗 Architecture

1. Client: The one that submits jobs
2. Handjob Broker
3. Worker: The one that processes jobs

Restrictions

Jobs are not multiplexed over a single websocket connection. This means that both client and worker can only have one active job at a time per websocket connection. If for example, the client is to dispatch up to 5 jobs concurrently, the client must make 5 websocket connections.

---

The Handjob Protocol

The handjob protocol layer sits on top of WebSockets.

The Handjob Packet

A websocket message is a handjob packet if it fits the defined structure. Packet structure is the same independent of data direction. Each handjob packet starts with either:

1. 0x7B ASCII for {, denoting that this packet is a JSON control packet.
2. 0xB00B Data packet. Does not contain job ID. Broker ensures data packets are routed to the correct parties.

All messages that do not have either of these prefixes are dropped by the client, broker, and worker.

Control Packets

JSON Structure	From	To	To	Description
{"type":"worker-register","accessToken":"2098nvbfd"}	Worker	Broker	-	Register worker to start accepting incoming jobs
{"type":"submit-job","accessToken":"fg89fsdgjnb"}	Client	Broker	Worker	Tells worker that job is available
{"type":"job-not-submitted","reason":"No worker available"}	Broker	Client	-	Sent when job couldn't be submitted
{"type":"job-submitted","jobId":123}	Worker	Broker	Client	Tells client it can start sending data
{"type":"data-end","jobId":123}	Client	Broker	Worker	Tells worker that client has finished sending job input data
{"type":"data-end","jobId":123}	Worker	Broker	Client	Tells client that worker has finished sending job output data

Data Packet

Both client and worker should skip over the first two bytes 0xB00B to read the actual data.

Control flow

The following depicts the data flow in a successful scenario.

1. Worker sends worker-register control packet to broker
2. Client send submit-job control packet to broker
3. Broker finds an available worker and forwards the submit-job to it.
4. Worker receives submit-job, creates a job ID (uuid), and replies with job-submitted that includes the job ID
5. Broker receives job-submitted, creates an entry in the map below, and forwards job-submitted to client

Map<
  jobId,
  {
    client: WebSocket;
    worker: WebSocket;
  }
>;

6. Bidirectional data flow begins. The client starts sending data required for the job. The broker uses the Map created previously to route data packets between the client and worker. The worker can wait for data-end before returning the result data, or it can start streaming the result data immediately. The Handjob protocol doesn't care.
7. Broker finishes processing the job and sends data-end to broker.
8. Broker receives data-end from worker and clears the job from the map. Broker forwards data-end to client.

Latency Analysis

Assumptions:

• Bandwidth is bottlenecked by client upload/download speed
• Worker does not start processing until input data is received entirely
• Worker does not send output data until processing is entirely finished.

Example numbers

• Round trip time: 80ms
• Client upload speed: 350 mbps
• Input data size: 150 kB
• Worker processing duration: 800ms
• Client download speed: 50 mbps
• Output data size: 3kB

Very rough estimate: (800ms processing time) + (320ms of network overhead) = ~1,120 ms