Skip to content
A worker pool implementation for Go
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitlab-ci.yml
.travis.yml
LICENSE
README.md
dispatcher.go
error.go
job.go
worker.go

README.md

WorkerGo

Build Status Godoc Go Report Card

WorkerGo is an MIT licensed worker pool implementation that can be used in any Go program to handle tasks with workers.

WorkerGo is heavily influenced by the Marcio Catilho's post here: http://marcio.io/2015/07/handling-1-million-requests-per-minute-with-golang/

I was trying to write a worker pool implementation that I could use in a program with different portions of it will require parallel processing. I found his post while researching, created a new package using portions of his samples in my project. Since the package I created can be used for calling any struct with a method, I thought it would be good to share, so, it can be used in any program that needs a worker pool implementation.

Installation

go get gopkg.in/yasarix/workergo.v2

Usage

First, a Dispatcher needs to be created.

maxWorkers := 5 // Maximum number of workers
queueBufferSize := 200 // Buffer size for job queue
d := workergo.New(maxWorkers, queueBufferSize)

// Start dispatcher
d.Start()

Workergo accepts any struct that implements Job interface that you can see below:

type Job interface {
    Run()
}

Then, a struct that satisfies Job interface can be created and sent to job queue:

type MyJob struct {
    id int
}

func (j *MyJob) Run() {
    fmt.Printf("This is job id: %d", id)
    time.Sleep(5 * time.Second)
}

...
...
func main() {
    d := workergo.New(5, 100)
    d.Start()
    for i := 0; i < 13; i++ {
        j := MyJob{id: i}
        d.Submit(&j)
    }

    // Wait for workers to finish executing `j.Run()` methods
    d.Wait()
}

Setting a rate limiter

You can also run the dispatcher with a rate limiter value. Simply, pass a time.Duration while creating the dispatcher:

d := workergo.New(5, 100, workergo.RateLimit(time.Millisecond * 500))

Now, each job that you have submitted will be dispatched with 0.5 seconds delay.

Stopping the dispatcher

Dispatcher can be stopped gracefully by calling Stop() method of Dispatcher. Dispatcher will wait until the actively running jobs to finish, and then stop dispatching rest of the jobs in the queue, and will shut down the workers.

Code Documentation

https://godoc.org/github.com/yasarix/workergo

You can’t perform that action at this time.