Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Run MapReduce jobs on Hadoop or Amazon Web Services

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 docs
Octocat-spinner-32 mrjob
Octocat-spinner-32 tests
Octocat-spinner-32 .gitignore
Octocat-spinner-32 CHANGES.txt
Octocat-spinner-32 LICENSE.txt
Octocat-spinner-32 MANIFEST.in
Octocat-spinner-32 README.rst
Octocat-spinner-32 examples
Octocat-spinner-32 mrjob.conf.example
Octocat-spinner-32 setup.cfg
Octocat-spinner-32 setup.py
README.rst

mrjob

mrjob is a Python package that helps you write and run Hadoop Streaming jobs.

mrjob fully supports Amazon's Elastic MapReduce (EMR) service, which allows you to buy time on a Hadoop cluster on an hourly basis. It also works with your own Hadoop cluster.

Some important features:

  • Run jobs on EMR, your own Hadoop cluster, or locally (for testing).

  • Write multi-step jobs (one map-reduce step feeds into the next)

  • Duplicate your production environment inside Hadoop
    • Upload your source tree and put it in your job's $PYTHONPATH
    • Run make and other setup scripts
    • Set environment variables (e.g. $TZ)
    • Easily install python packages from tarballs (EMR only)
    • Setup handled transparently by mrjob.conf config file
  • Automatically interpret error logs from EMR

  • SSH tunnel to hadoop job tracker on EMR

  • Minimal setup
    • To run on EMR, set $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY
    • To run on your Hadoop cluster, install simplejson and make sure $HADOOP_HOME is set.

Installation

python setup.py install

Setting up EMR on Amazon

Try it out!

# locally
python mrjob/examples/mr_word_freq_count.py README.rst > counts
# on EMR
python mrjob/examples/mr_word_freq_count.py README.rst -r emr > counts
# on your Hadoop cluster
python mrjob/examples/mr_word_freq_count.py README.rst -r hadoop > counts

Advanced Configuration

To run in other AWS regions, upload your source tree, run make, and use other advanced mrjob features, you'll need to set up mrjob.conf. mrjob looks for its conf file in:

  • ~/.mrjob
  • mrjob.conf anywhere in your $PYTHONPATH
  • /etc/mrjob.conf

See mrjob.conf.example for more information.

Links

Something went wrong with that request. Please try again.