Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Building and installing
Clone this wiki locally
On the machine from which you want to run your Dumbo programs, do:
or, if you already have
$ wget -O ez_setup.py http://bit.ly/ezsetup $ python ez_setup.py -z dumbo
$ easy_install -z dumbo
-zoption is important since Dumbo only works when it’s installed as a self-contained, zipped egg.
Alternatively, you can also install Dumbo in a virtual Python environment:
$ wget -O virtualenv.py http://bit.ly/virtualenv $ python virtualenv.py env $ env/bin/easy_install -z dumbo
Once you have completed the steps above, you can move on to Running programs. The recommended Hadoop distribution to run your Dumbo programs on is Cloudera’s Hadoop distribution, which supports Dumbo out of the box from version 2 (CDH2) onwards.
Old Hadoop Versions
If you use an old version of Hadoop then you’ll first have to apply a few patches. More precisely, you then have to download the patches for HADOOP-1722, HADOOP-5450, and MAPREDUCE-764 and rebuild Hadoop after applying these patches (the order in which you apply the patches is important!):
If you want to use Dumbo’s convenient joining abstraction, you need to apply HADOOP-5528 as well.
$ cd /path/to/hadoop $ patch -p0 < /path/to/HADOOP-1722.patch $ patch -p0 < /path/to/HADOOP-5450.patch $ patch -p0 < /path/to/MAPREDUCE-764.patch $ ant package
Old Dumbo Versions
Dumbo 0.21 has been around for quite a while now and can be considered very stable, but if you still want to use 0.20 or lower for some reason then you’ll have to follow these instructions instead.
As part of Hadoop (mandatory)
To build Dumbo, you just have to add it to the src/contrib directory of Hadoop (version 0.18) and build Hadoop:
This should generate a Hadoop build in build/ that contains a contrib/dumbo directory:
$ wget http://github.com/klbostee/dumbo/tarball/release-0.20.28 -O dumbo.tar.gz $ tar zxvf dumbo.tar.gz $ mv klbostee-dumbo* $HADOOP_HOME/src/contrib/dumbo $ cd $HADOOP_HOME $ ant package
The shell script example in the subdirectory bin/ runs the wordcount.py example on Hadoop.
$ ls build/hadoop-*/contrib/dumbo bin examples lib
As a Python module (optional)
You can also install Dumbo as a Python module on your system:
This additional installation step is not required, but we do recommend it because it allows you to run programs locally using UNIX pipes, which can be very useful for debugging. The dumbo command that gets added to /usr/bin by this optional installation step can be used in the same way as $HADOOP_HOME/build/hadoop-*/contrib/dumbo/bin/dumbo. The only difference is that it requires an additional -hadoop <path_to_hadoop_dir> option. Hence, this same command can be used to run programs on different Hadoop clusters, and by omitting the -hadoop option you can run a Dumbo program locally using UNIX pipes.
$ cd $HADOOP_HOME/src/contrib/dumbo $ sudo ant install_pymod