Lightweight MapReduce in python
Python
Pull request Compare This branch is 77 commits ahead, 20 commits behind michaelfairley:master.
Latest commit 26cb51a Aug 14, 2013 @Yangqing Update README.md
Permalink
Failed to load latest commit information.
mincepie bugfix when map returns nothing May 6, 2013
.gitignore matlab.py bugfix Aug 27, 2012
README.md Update README.md Aug 13, 2013
setup.py setup.py Apr 16, 2013

README.md

mincepie: Lightweight MapReduce in Python

Mincepie is a lightweight mapreduce engine purely written in Python for some of my research code. It is meant for simple distribution of jobs that:

  • can be easily separated to map calls, such as extracting one feature for each image,
  • does not involve a lot of communication, i.e. the keys and values being transmitted are not huge,
  • the main computation time is spent inside the map() or reduce() function, not communication.

Also, the simplified system

  • holds everything in memory - the input, the keys, and the values in every stage of the mapreduce run.
  • does not handle server errors. If the server is down, you have to restart mapreduce.
  • partially tolerates client failure. If a client is down, its last map() call will simply be re-run on another client.

I have used it to perform feature extractions on ImageNet and it works pretty well for our research use. But it may or may not fit your use case.

To see more documentation read the wiki.