Distributed long running work system in Go
Clone or download
Latest commit ec7240a Oct 4, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Documentation Revert "Replace github.com/coreos/etcd with github.com/etcd-io/etcd (… Oct 4, 2018
cmd/metaforactl Adding etcd coordinator changes Oct 2, 2014
embedded Fix balancer and tests; improve test logging Sep 16, 2015
examples Revert "Replace github.com/coreos/etcd with github.com/etcd-io/etcd (… Oct 4, 2018
httputil Marshal Task into httputil output Aug 4, 2015
metcdv3 Revert "Replace github.com/coreos/etcd with github.com/etcd-io/etcd (… Oct 4, 2018
resreporter Fix the worst logging in all of metafora Mar 3, 2015
scripts Small etcd script/test fixes Oct 9, 2014
statemachine fix comment Aug 11, 2016
.gitignore Switch task ID to task interface Jul 22, 2015
.travis.yml Revert "Replace github.com/coreos/etcd with github.com/etcd-io/etcd (… Oct 4, 2018
LICENSE Add Lytics copyright to license Oct 1, 2014
README.md Revert "Replace github.com/coreos/etcd with github.com/etcd-io/etcd (… Oct 4, 2018
balancer.go Switch task ID to task interface Jul 22, 2015
balancer_res.go Switch task ID to task interface Jul 22, 2015
balancer_res_test.go Switch to a global logger instead of contextualized ones Mar 3, 2015
balancer_sleep.go Minor doc cleanups Dec 22, 2014
balancer_test.go Switch task ID to task interface Jul 22, 2015
client.go Switch task ID to task interface Jul 22, 2015
command.go Fix double task release on shutdown Jan 5, 2015
command_test.go Rename Coordinator.String->Name for clarity May 5, 2015
coordinator.go Switch task ID to task interface Jul 22, 2015
doc.go Improve docs Oct 4, 2014
handler.go Switch task ID to task interface Jul 22, 2015
ignore.go only drop zero time tasks Aug 15, 2016
ignore_test.go Switch task ID to task interface Jul 22, 2015
logger.go Fix stack depth in new logger Mar 9, 2015
metafora.go Add etcdv3 client support for metafora Jun 6, 2018
metafora_test.go Switch task ID to task interface Jul 22, 2015
slowtask_test.go Switch task ID to task interface Jul 22, 2015
task.go Add more helper functions Jul 28, 2015
util_test.go Fix balancer and tests; improve test logging Sep 16, 2015

README.md

metafora

Join the chat at https://gitter.im/lytics/metafora Build Status GoDoc

Metafora is a Go library designed to run long-running (minutes to permanent) tasks in a cluster.

IRC: #lytics/metafora on irc.gitter.im

Features

  • Distributed - horizontally scalable
  • Elastic - online cluster resizing with automated rebalancing
  • Masterless - work stealing, not assigning, pluggable balancing
  • Fault tolerant - tasks are reassigned if nodes disappear
  • Simple - few states, no checkpointing, no configuration management
  • Extensible - well defined interfaces for implementing balancing and coordinating
  • Exactly-once - designed to enforce one-and-only-one instance of each submitted task is runningref

Metafora is a library for building distributed task work systems. You're responsible for creating a main() entrypoint for your application, writing a metafora.Handler and HandlerFunc to actually process tasks, and then starting Metafora's Consumer.

Metafora's task state machine is implemented as a Handler adapter. Simply implement your task processor as a StatefulHandler function, and create a metafora.Handler with statemachine.New.

Example

koalemosd is a sample consumer implementation that can be run as a daemon (it requires etcd). koalemosctl is a sample command line client for submitting tasks to koalemosd.

# Install etcd as per https://github.com/coreos/etcd#getting-etcd
# Run the following in one terminal:
go get -v -u github.com/lytics/metafora/examples/koalemosd
koalemosd

# Run the client in another
go get -v -u github.com/lytics/metafora/examples/koalemosctl
koalemosctl sleep 3 # where "sleep 3" is any command on your $PATH

Since koalemosd is a simple wrapper around OS processes, it does not use the state machine (statemachine.StatefulHandler).

Terms

BalancerGo interface consulted by Consumer for determining which tasks can be claimed and which should be released. See balancer.go.
Brokerexternal task and command store like etcd for the Coordinator to use.
Consumercore work runner. Integrates Balancer, Coordinator, and Handlers to get work done.
Coordinatorclient Go interface to Broker. See coordinator.go.
HandlerGo interface for executing tasks.
Taskunit of work. Executed by Handlers.

FAQ

Q. Is it ready for production use?

Yes. Metafora with the etcd coordinator has been the production work system at Lytics since January 2014 and runs thousands of tasks concurrently across a cluster of VMs.

Since Metafora is still under heavy development, you probably want to pin the dependencies to a commit hash or tag to keep the API stable. The master branch is automatically tested and is safe for use if you can tolerate API changes.

Q. Where is the metaforad daemon?

It doesn't exist. Metafora is library for you to import and use in a service you write. Metafora handles task management but leaves implementation details such as task implementation and daemonization up to the user.

FAQ continued in Documentation...