Distributed Task Queue

Task Queue in Go using Redis Streams

Relationship between task name and stream naming:

[task]:[stream]

# Example
upload:default
upload:result

No, cause we want to queue to have hetereogious task types.

Using Go Channels

1. The WorkerPool aggregates the job channel and result channel.
2. User enqueues jobs directly onto the job channel that is owned by worker pool.
3. The worker pool only calls a worker function to execute the job.
4. Put result into result channel

Using Redis Streams

1. A client enqueue a job onto the job stream.
2. Worker pool consumes messages from job stream and puts it on a local job channel
3. A set of workers then consumes jobs from this channel and processes the job function.
4. The worker that processed the job then puts the result on the result channel.
5. The worker pool then enqueues each result onto the redis result stream.
6. The client then listens to the result stream from the distributes data store.
7. The worker pool is responsible for cleaning/updating the stream after a success or failure.

Notes

• The worker pool write to taskStream while the worker function only reads from taskStream. This ensures confinement by keeping the scope small.
• The broker shares state between the client and the worker pool.

WTF

Why am I adding a client abstraction to aggregate the broker and worker pool. I can literally just communicate with the broker interface directly

Detach the workerpool and distributed queue by adding an interface we will call Broker