Consul-based rapid propagative task runner for large systems.
Table of Contents
- How to get started?
This program triggers to execute configured tasks on nodes in Consul
And tasks will be executed in propagative way.
Task propagation time takes O(log N) in theory; as N means node number in cluster.
So this tool can shorten the time of tasks which take O(N) time in system of
large number of nodes.
Typical usecase is software deployment.
The name fireap comes from "fire" and "reap".
Here is a benchmark result comparing fireap against GNU Parallel.
The job executed here is a directory sync operation by
rsync command which
contains a thousand of files up to total 12MB size, through 100 t2.micro instances
on AWS EC2.
Concurrency numbers of both GNU Parallel and fireap is 5 in this benchmark.
In fireap, the word "concurrency" means the maximum concurrent number that
one node can be "pulled" by other nodes.
You will grasp the concept by going under part of this document.
Take a look at how it works
Below is a demo of fireap task propagation at a 10-node Consul cluster.
On the top of the screen,
fireap monitor command is executed.
This continuously shows the application version and updated information of nodes
those are stored in Consul Kv.
On the bottom of the screen,
fireap fire command is executed which fires Consul
The triggered event is broadcasted to cluster members at once.
And it leads cluster members to execute
fireap reap command by Consul Watch
Eventually, configured tasks are executed on nodes in the cluster.
Task Propagation Procedure
The image below illustrates task propagation procedure by fireap in Consul cluster.
Publisher and Subscribers
One publisher node fires Consul events whose
And it is assumed to be the 1st node in the cluster which finishes the task.
All other nodes are subscribers who receive events and execute tasks.
Concept of publisher and subscriber is not related to role of server and client
Server or client in Consul cluster can be either publisher or subscriber.
- Publisher fires a Consul event.
- The event is broadcasted for appropriate subscribers at once.
- Subscribers execute the task in propagative way:
- All subscribers search for a finished node in the cluster to "pull" update
information or contents from the node.
In first stage, there is only publisher who finished the task. So they all tries to "pull" from publisher, but maximum number of who can "pull" from a particular node is limited by configuration. Then, only several subscribers succeed to "pull" and execute the task.
- In second stage, publisher and several subscribers now finished the task. Their update will be "pulled" by other subscribers.
- Stage goes on until all subscribers finish the task.
- All subscribers search for a finished node in the cluster to "pull" update information or contents from the node.
This propagation way looks like tree branching.
But it is rather robust because even if a subscriber happens to fail the task, the failure does not affect others.
Publisher and subscribers store information in Consul Kv about task completion and so on.
All keys of data related to this program begin with prefix
How to get started?
You can start fireap with the guide in Documentation.
The MIT License (MIT)
Copyright (c) 2016 YASUTAKE Kiyoshi