Building and installing

Klaas Bosteels edited this page Sep 5, 2013 · 50 revisions

On the machine from which you want to run your Dumbo programs, do:

$ wget -O
$ python -z dumbo
or, if you already have easy_install installed, simply:
$ easy_install -z dumbo
The -z option is important since Dumbo only works when it’s installed as a self-contained, zipped egg.

Alternatively, you can also install Dumbo in a virtual Python environment:

$ wget -O
$ python env
$ env/bin/easy_install -z dumbo

Once you have completed the steps above, you can move on to Running programs. The recommended Hadoop distribution to run your Dumbo programs on is Cloudera’s Hadoop distribution, which supports Dumbo out of the box from version 2 (CDH2) onwards.

Old Hadoop Versions

If you use an old version of Hadoop then you’ll first have to apply a few patches. More precisely, you then have to download the patches for HADOOP-1722, HADOOP-5450, and MAPREDUCE-764 and rebuild Hadoop after applying these patches (the order in which you apply the patches is important!):

$ cd /path/to/hadoop
$ patch -p0 < /path/to/HADOOP-1722.patch
$ patch -p0 < /path/to/HADOOP-5450.patch
$ patch -p0 < /path/to/MAPREDUCE-764.patch
$ ant package
If you want to use Dumbo’s convenient joining abstraction, you need to apply HADOOP-5528 as well.

Old Dumbo Versions

Dumbo 0.21 has been around for quite a while now and can be considered very stable, but if you still want to use 0.20 or lower for some reason then you’ll have to follow these instructions instead.

As part of Hadoop (mandatory)

To build Dumbo, you just have to add it to the src/contrib directory of Hadoop (version 0.18) and build Hadoop:

$ wget -O dumbo.tar.gz
$ tar zxvf dumbo.tar.gz
$ mv klbostee-dumbo* $HADOOP_HOME/src/contrib/dumbo
$ ant package
This should generate a Hadoop build in build/ that contains a contrib/dumbo directory:
$ ls build/hadoop-*/contrib/dumbo
bin  examples  lib
The shell script example in the subdirectory bin/ runs the example on Hadoop.

As a Python module (optional)

You can also install Dumbo as a Python module on your system:

$ cd $HADOOP_HOME/src/contrib/dumbo
$ sudo ant install_pymod
This additional installation step is not required, but we do recommend it because it allows you to run programs locally using UNIX pipes, which can be very useful for debugging. The dumbo command that gets added to /usr/bin by this optional installation step can be used in the same way as $HADOOP_HOME/build/hadoop-*/contrib/dumbo/bin/dumbo. The only difference is that it requires an additional -hadoop <path_to_hadoop_dir> option. Hence, this same command can be used to run programs on different Hadoop clusters, and by omitting the -hadoop option you can run a Dumbo program locally using UNIX pipes.
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.