Copyright (c) 2013, Dean Wampler, Inc. All Rights Reserved.
Copyright (c) 2013, Think Big Analytics, Inc. All Rights Reserved.
Welcome to Stampede, the workflow tool that works as Cthulhu intended for *nix systems, using
make for dependency management and task seqeuencing,
bash for scripting, and
cron for scheduling. Stampede is Apache-licensed, open-source software.
Stampede originated as an alternative workflow tool for Hadoop, but it is not limited to Hadoop scenarios.
If you like Stampede, please consider joining the stampede-users Google group and following us on Twitter @StampedeWkFlow. Also, contributions in the form of patches are always welcome and appreciated.
First, clone this repo or expand the distribution archive somewhere useful, e.g.,
Since Stampede uses
bash as its weapon's of choice, run this
make command to test Stampede on your system and then install it:
make test install
tests target is not required, but we recommend it as a sanity check for your environment. The
install target will ask you for details like the target installation directory (the default is
If you don't have
syslog on your system, run this command instead, which will skip the
make test-core install
test target does not test the "extras" included with Stampede, currently limited to Hadoop-specific tools. To test these tools, first ensure that
$HADOOP_HOME is defined, then run this command:
make test-extras install
install target installs everything, whether you want to use
syslog and the "extras" or not. They are small and harmless, if left alone in a cold, dark room... ;^)
Next, assuming you installed in
/usr/local/stampede/, which we'll call
$STAMPEDE_HOME from now on, add
$STAMPEDE_HOME/bin to the
PATH for any user who plans to use Stampede. Also, the installation will include *nix
man pages, so add
$STAMPEDE_HOME/man to the
As part of the installation, the installer will ask you if you want a global
stampederc file installed in
/etc/sysconfig, or somewhere else. All statements in this file are commented out. If you want to make global changes to Stampede's environment variables, edit this file appropriately. Note these "rc" files won't contain all the possible variables you can define, see
$STAMPEDE_HOME/bin/env.sh for the complete list of variables, their default values, and comments that describe them.
Similarly, if you told the installer to copy
stampederc file to
$HOME/.stampederc, edit that file for your personal tasks.
Whenever you create a new Stampede project, it will also get its own
$PROJECT_HOME/.stampederc file, as we discuss next.
Building Java Components
As of this release, there is a small Hadoop application written in Java, in
src/hadop/mapreduce-configuration. It is used by the
bin/hadoop/mapreduce-prop command. For your convenience, a pre-built jar file is already provided. However, it is built with Java 1.6 (for maximum portability) and Hadoop v1.0.3. So, you may need to rebuild it if you use a different version of Hadoop or you want to use a newer version of Java. See
src/hadop/mapreduce-configuration/README.md for details.
An individual workflow definition is called a stampede.
To create a stampede, run the following command:
It will prompt you for properties such as the name of the stampede and the project's working directory.
makefile created in the project directory to define your workflow. See the
$STAMPEDE_HOME/examples for ideas. Note that
$STAMPEDE_HOME/bin contains helper scripts to ease the development of workflows. See also Make and Bash Notes in this directory for some tips.
Once a stampede has been created, you can invoke it using this command:
stampede -f /path/to/makefile [options] [make_targets]
For help on the
bashv3+ - Because OS X ships with an older bash version, all the scripts supplied are v3 compatible. You can use newer constructs if your version of bash supports them.
makev3.8+ - The
Makefilein this directory that's used to test and install Stampede requires GNU
makev3.8+, as do the
examples. However, you can adapt your project
Makefilesto use any version of
cron- If you plan to use
cronfor scheduling workflows. In fact, Stampede doesn't really do anything with
cronitself; we just recommend that you use it first, before adopting something more heavyweight and proprietary. Stampede projects will work fine with any scheduling tool that can invoke shell commands.
syslog- If you plan to use the *nix logging facility
syslog. See also the Installation section above.
- Hadoop - Stampede was originally designed as a flyweight option for Hadoop. However, it is not restricted to Hadoop scenarios. All of the Hadoop support consists of helper scripts in
$STAMPEDE_HOME/bin/hadoop(and corresponding tests and
Stampede is mostly agnostic to tool versions. For any particular tool, including its own scripts, Stampede relies on finding the tool in the user's
- Linux - All recent Linux distributions with
bashv3+. However, we have not tested all possible Linux variants!
- Mac OS X - All recent OS X versions.
cygwin and similar "Unix on Windows" toolkits are not supported, but only because we haven't tried them. We have tried to avoid any assumptions that would preclude this support. We welcome patches!
Note that as of this writing, support for running Hadoop in Windows environments was just recently announced.
The top-level directory contains the following files, in addition to directories that will be described next:
README.md- This file, as well as an HTML version of it.
LICENSE- The copyright and license (Apache 2.0).
FAQs.md- Frequently-asked Questions.
makefileused to test and install Stampede.
VERSION- The version number, used by the
Makefilefor building releases.
bin- See the following section.
src- The directory for tools implemented with Java. The corresponding jars are prebuilt and included in the distribution, but if you want to build them yourself, see the
README.mdfiles in the corresponding directories under
manpages for all the tools.
test- Tests for the tools.
Stampede supplies helper
bash scripts in the
bin directory and "extras" for specific applications (e.g., Hadoop) in subdirectories. All the scripts that end with
.sh are used internally by Stampede. The files without this extension are user-callable utilities for building workflows.
NOTE: All of these tools assume that
$STAMPEDE_HOME is defined. This is true when they are called in a stampede workflow, e.g., a
Briefly, here are the utilities in the
bin directory. All support a
--help option for more information:
stampede- The "stampede" (workflow) driver script. It can be invoked manually or by
cron. It has several options to configure behavior. Run
stampede --helpfor details.
install- Called by
make installto install Stampede.
abs-path- Return the absolute value for the specified paths.
create-project- Called by the
stampedescript to create new projects.
dates- Format dates and perform date arithmetic in a platform-portable way.
find-tool- Locate the command for the specified
curl) in the user's path, several possible system directories, or in user-specified directories.
to-log-level- Convert from a log-level string, e.g.,
DEBUGto the corresponding
syslog-compatible number and back again.
format-log-message- Format messages that are logged. If you want to customize the format beyond what's possible by editing the environment variables
env.sh), you can create your own version of this script and drop it in
$STAMPEDE_HOME/custom, which is on the
$STAMPEDE_HOME/bin. See the Custom section below for more details.
install- Install Stampede on your system.
log-file- Return the name of the log file used by stampede or
syslogis being used.
send-email- Use the *nux
split-string- Split a string on a delimiter and return an array or echo the elements to
stampede-log- Write your own messages to the log file (or
syslog) configured for Stampede.
success-or-failure- Return one of two strings depending on a "success" flag.
true-or-false- Return one of two strings depending on a whether a variable is empty or not.
to-seconds- Return the number of seconds specified for an input number of seconds, minutes, or hours.
to-seconds, but returns a nicely formatted string.
try-for- Repeated attempt an operation for a specified duration of time, until success or timeout.
try-for, but tries until a user-specified timestamp.
sleep(1)wrapper with logging for use in loops, like the one in the
ymd- Return the year, month, and day for the workflow's start time, which defaults to today's date.
yesterday-ymd- Return the year, month, and day for the day before the workflow's start time, i.e., yesterday's date, by default. For example, if you need to process yesterday's data, this is a convenient way to compute the correct date.
The following "helper" files are used by these scripts:
env.sh- Defines global shell variables. Start here to find variables you can set in
rcfiles to configure behavior.
common.sh- Many "common"
bashfunctions used in several scripts.
log.sh- Support functions for logging.
Hadoop-specific helper tools are in the
bin/hadoop directory. As for the
bin scripts, use
--help for more information on each tool.
mapreduce-prop- Return one or more property definitions for Hadoop MapReduce jobs.
hive-prop- Return one or more property definitions for Hive.
pig-prop- Return one or more property definitions for Pig.
Note: While the
*-prop utilities behave similarly and take similar arguments, there are some differences
in the results they produce that reflect differences in how they were implemented. See their
*-prop --help messages or
man pages for details.
Custom and Contrib Directories
If you want to override the behavior of any particular script, drop a new version in the
custom directory (or a subdirectory), which are added to the
We intend for
contrib to be a place where unsupported, community-contributed tools will go. This directory and any subdirectories will also be added to the path, after
example directory contains example stampedes that you can adapt for your purposes as well as a sample configuration file.
crontab- A sample
stampederc- A sample file that overrides environment variable definitions to customize the environment or a particular project. See
bin/env.shfor recommendations on where to install one or more of these
rcfiles and for the complete list of variables available.
hadoop- An example sequencing several Hadoop jobs into a typical ETL and analytics workflow.
Tests of Stampede itself are in the
test directory. The tests provide good examples of the individual tools in action. To execute the tests, run
make test. This
make target won't run the "extras" tests, e.g., for Hadoop. To run all tests, run
bin/send-emailscript requires the *nix mail service to be running on the server hosting the stampede.
- Supporting both Linux and Mac
datecommands added some complexity to the code. We might consider deprecating support for the Mac
datecommand (based on a BSD legacy) and instead require Mac users to install the Linux version of
dateusing Homebrew or MacPorts. Feedback welcome.