mrjob is a Python package that helps you write and run Hadoop Streaming jobs.
mrjob fully supports Amazon's Elastic MapReduce (EMR) service, which allows you to buy time on a Hadoop cluster on an hourly basis. It also works with your own Hadoop cluster.
Some important features:
Run jobs on EMR, your own Hadoop cluster, or locally (for testing).
Write multi-step jobs (one map-reduce step feeds into the next)
- Duplicate your production environment inside Hadoop
- Upload your source tree and put it in your job's
- Run make and other setup scripts
- Set environment variables (e.g.
- Easily install python packages from tarballs (EMR only)
- Setup handled transparently by
Automatically interpret error logs from EMR
SSH tunnel to hadoop job tracker on EMR
- Minimal setup
- To run on EMR, set
- To run on your Hadoop cluster, install
simplejsonand make sure
python setup.py install
Setting up EMR on Amazon
- create an Amazon Web Services account: <http://aws.amazon.com/>
- sign up for Elastic MapReduce: <http://aws.amazon.com/elasticmapreduce/>
- Get your access and secret keys (go to <http://aws.amazon.com/account/> and click on "Security Credentials")
- Set the environment variables
Try it out!
# locally python mrjob/examples/mr_word_freq_count.py README.md > counts # on EMR python mrjob/examples/mr_word_freq_count.py README.md -r emr > counts # on your Hadoop cluster python mrjob/examples/mr_word_freq_count.py README.md -r hadoop > counts
To run in other AWS regions, upload your source tree, run
make, and use
other advanced mrjob features, you'll need to set up
mrjob.conf. mrjob looks
for its conf file in:
mrjob.confanywhere in your
mrjob.conf.example for more information.
- source: <http://github.com/Yelp/mrjob>
- documentation: <http://packages.python.org/mrjob/>
- Hadoop MapReduce: <http://hadoop.apache.org/mapreduce/>
- Elastic MapReduce: <http://aws.amazon.com/documentation/elasticmapreduce/>