Sidekiq compatible background workers in golang
Go
Switch branches/tags
Nothing to show
Clone or download
Permalink
Failed to load latest commit information.
.travis.yml Update travis golang versions Nov 28, 2016
LICENSE.txt Add license file Aug 28, 2013
README.md add extended stats add more info to README.md (#66) Dec 2, 2016
all_specs_test.go Fix worker NPE when Start(), then Quit() in quick succession Feb 12, 2016
config.go Add test for setting poll_interval, and fix incorrect logic. May 3, 2015
config_test.go Add test for setting poll_interval, and fix incorrect logic. May 3, 2015
enqueue.go Fix durationToSecondsWithNanoPrecision and related tests May 12, 2015
enqueue_test.go Clean unit: store float64 seconds, with nanosecond precision in redis… May 6, 2015
fetch_test.go Fetcher shouldn't pull jobs if workers are busy. Sep 24, 2014
fetcher.go Make sure 'Fetcher' doesn't have a lingering job upon exit (#79) Jan 12, 2018
hooks.go Add a few few worker lifecycle hooks Feb 16, 2016
job.go Give full message object to job functions rather than just the Mar 13, 2014
manager.go Fix worker NPE when Start(), then Quit() in quick succession Feb 12, 2016
manager_test.go Make sure 'Fetcher' doesn't have a lingering job upon exit (#79) Jan 12, 2018
middleware.go Allow middleware to control acknowledgement of message. Mar 13, 2014
middleware_logging.go Update the logging (#63) Dec 2, 2016
middleware_retry.go Clean unit: store float64 seconds, with nanosecond precision in redis… May 6, 2015
middleware_retry_test.go Make configured namespace a public attribute. Sep 6, 2014
middleware_stats.go Make configured namespace a public attribute. Sep 6, 2014
middleware_stats_test.go Make configured namespace a public attribute. Sep 6, 2014
middleware_test.go Allow middleware to control acknowledgement of message. Mar 13, 2014
msg.go Count processing workers for each queue. Jul 20, 2013
msg_test.go Add message argments. Jul 19, 2013
scheduled.go Similar data race fix than #51 but in scheduled tasks routine Jun 13, 2015
scheduled_test.go Fix durationToSecondsWithNanoPrecision and related tests May 12, 2015
signals_posix.go Rename signal handling files. May 9, 2014
signals_windows.go Rename signal handling files. May 9, 2014
stats.go add extended stats add more info to README.md (#66) Dec 2, 2016
worker.go Add notification channel for finished work Feb 28, 2016
worker_test.go Allow middleware to control acknowledgement of message. Mar 13, 2014
workers.go Add a few few worker lifecycle hooks Feb 16, 2016
workers_logger.go Allowing for a custom logger to be assigned to workers.Logger Nov 3, 2014
workers_test.go Remove tricky test. Feb 16, 2016

README.md

Build Status GoDoc

Sidekiq compatible background workers in golang.

  • reliable queueing for all queues using brpoplpush
  • handles retries
  • support custom middleware
  • customize concurrency per queue
  • responds to Unix signals to safely wait for jobs to finish before exiting.
  • provides stats on what jobs are currently running
  • well tested

Example usage:

package main

import (
	"github.com/jrallison/go-workers"
)

func myJob(message *workers.Msg) {
  // do something with your message
  // message.Jid()
  // message.Args() is a wrapper around go-simplejson (http://godoc.org/github.com/bitly/go-simplejson)
}

type myMiddleware struct{}

func (r *myMiddleware) Call(queue string, message *workers.Msg, next func() bool) (acknowledge bool) {
  // do something before each message is processed
  acknowledge = next()
  // do something after each message is processed
  return
} 

func main() {
  workers.Configure(map[string]string{
    // location of redis instance
    "server":  "localhost:6379",
    // instance of the database
    "database":  "0",
    // number of connections to keep open with redis
    "pool":    "30",
    // unique process id for this instance of workers (for proper recovery of inprogress jobs on crash)
    "process": "1",
  })

  workers.Middleware.Append(&myMiddleware{})

  // pull messages from "myqueue" with concurrency of 10
  workers.Process("myqueue", myJob, 10)

  // pull messages from "myqueue2" with concurrency of 20
  workers.Process("myqueue2", myJob, 20)

  // Add a job to a queue
  workers.Enqueue("myqueue3", "Add", []int{1, 2})

  // Add a job to a queue with retry
  workers.EnqueueWithOptions("myqueue3", "Add", []int{1, 2}, workers.EnqueueOptions{Retry: true})

  // stats will be available at http://localhost:8080/stats
  go workers.StatsServer(8080)

  // Blocks until process is told to exit via unix signal
  workers.Run()
}

Initial development sponsored by Customer.io