Skip to content

A Distributed Task Queue in Go using Redis Streams

Notifications You must be signed in to change notification settings

rubenvanstaden/tq

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Distributed Task Queue

Task Queue in Go using Redis Streams

Relationship between task name and stream naming:

[task]:[stream]

# Example
upload:default
upload:result

No, cause we want to queue to have hetereogious task types.

Using Go Channels

  1. The WorkerPool aggregates the job channel and result channel.
  2. User enqueues jobs directly onto the job channel that is owned by worker pool.
  3. The worker pool only calls a worker function to execute the job.
  4. Put result into result channel

Using Redis Streams

  1. A client enqueue a job onto the job stream.
  2. Worker pool consumes messages from job stream and puts it on a local job channel
  3. A set of workers then consumes jobs from this channel and processes the job function.
  4. The worker that processed the job then puts the result on the result channel.
  5. The worker pool then enqueues each result onto the redis result stream.
  6. The client then listens to the result stream from the distributes data store.
  7. The worker pool is responsible for cleaning/updating the stream after a success or failure.

Notes

  • The worker pool write to taskStream while the worker function only reads from taskStream. This ensures confinement by keeping the scope small.
  • The broker shares state between the client and the worker pool.

WTF

Why am I adding a client abstraction to aggregate the broker and worker pool. I can literally just communicate with the broker interface directly

Detach the workerpool and distributed queue by adding an interface we will call Broker

About

A Distributed Task Queue in Go using Redis Streams

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published