parallel: a Go Parallel Processing Library

Concurrency is hard. This library doesn't aim to make it easy, but it will hopefully make it a little less painful.

Install

This library should be compatible with all recent and future versions of Go, and has no third party dependencies.

go get -u github.com/ryanskidmore/parallel

You can then import the library

import "github.com/ryanskidmore/parallel"

Testing

This library uses the standard Go testing tools, and doesn't use any third party testing libraries.

go test

Quick Start

package main

import (
    "log"
    "fmt"

    "github.com/ryanskidmore/parallel"
)

func main() {
    p := parallel.New() // Create a new instance of parallel
    worker, err := p.NewWorker("worker1", &parallel.WorkerConfig{Parallelism: 1}) // Create a new worker
    if err != nil {
        log.Fatalf("FATAL: Failed to create new worker: %v", err)
    }
    worker.SetExecution(func(wh *parallel.WorkerHelper, args interface{}) { // Set the worker execution function
        fmt.Println(args)
        wh.Done()
    })
    worker.Start(interface{}("Test String")) // Start with worker with specified args
    worker.Wait() // Wait for the worker to complete
}

Docs

The best source of reference is the GoDocs for this library. Noted below are parts of the library that may not be immediately obvious from the docs or otherwise.

WorkerHelper

The WorkerHelper struct gives you access to a WaitGroup via WorkerHelper.Done(). When you call Worker.Wait() this waits on the WorkGroup for that worker and will block until every instance of the worker has called WorkerHelper.Done().

The WorkerHelper can also be used to consume and publish data to/from other workers/goroutines and this is done using DataChannels.

DataChannels

DataChannels are intended to be a method of publishing and consuming data between different workers and goroutines.

Before using a DataChannel in an execution function, it must first be initialised using:

err := p.NewDataChannel("name")

Once the DataChannel has been initialised, it can be published to by calling:

err := WorkerHelper.PublishData("name", data)

This is an asynchronous operation, so will not block execution.

Data can be consumed from the DataChannel via either calling:

data, err := WorkerHelper.ConsumeData("name")

or

data, err := WorkerHelper.ConsumeDataInBatches("name", 20)

These functions will return an error when the DataChannel doesn't exist or when the channel is closed.

Examples

Examples of usage can be found in the examples directory.