Metafora is a Go library designed to run long-running (minutes to permanent) tasks in a cluster.
#lytics/metafora on irc.gitter.im
- Distributed - horizontally scalable
- Elastic - online cluster resizing with automated rebalancing
- Masterless - work stealing, not assigning, pluggable balancing
- Fault tolerant - tasks are reassigned if nodes disappear
- Simple - few states, no checkpointing, no configuration management
- Extensible - well defined interfaces for implementing balancing and coordinating
- Exactly-once - designed to enforce one-and-only-one instance of each submitted task is runningref
Metafora is a library for building distributed task work systems. You're
responsible for creating a
main() entrypoint for your application, writing a
HandlerFunc to actually process tasks, and then
# Install etcd as per https://github.com/coreos/etcd#getting-etcd # Run the following in one terminal: go get -v -u github.com/lytics/metafora/examples/koalemosd koalemosd # Run the client in another go get -v -u github.com/lytics/metafora/examples/koalemosctl koalemosctl sleep 3 # where "sleep 3" is any command on your $PATH
Since koalemosd is a simple wrapper around OS processes, it does not use the
state machine (
|Balancer||Go interface consulted by Consumer for determining which tasks can be claimed and which should be released. See balancer.go.|
|Broker||external task and command store like etcd for the Coordinator to use.|
|Consumer||core work runner. Integrates Balancer, Coordinator, and Handlers to get work done.|
|Coordinator||client Go interface to Broker. See coordinator.go.|
|Handler||Go interface for executing tasks.|
|Task||unit of work. Executed by Handlers.|
Q. Is it ready for production use?
Yes. Metafora with the etcd coordinator has been the production work system at Lytics since January 2014 and runs thousands of tasks concurrently across a cluster of VMs.
Since Metafora is still under heavy development, you probably want to pin the
dependencies to a commit hash or
tag to keep the API stable. The
master branch is automatically tested and is safe for use if you can tolerate
Q. Where is the metaforad daemon?
It doesn't exist. Metafora is library for you to import and use in a service you write. Metafora handles task management but leaves implementation details such as task implementation and daemonization up to the user.