Skip to content
Go to file
This branch is 77 commits ahead, 20 commits behind michaelfairley:master.

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

mincepie: Lightweight MapReduce in Python

Mincepie is a lightweight mapreduce engine purely written in Python for some of my research code. It is meant for simple distribution of jobs that:

  • can be easily separated to map calls, such as extracting one feature for each image,
  • does not involve a lot of communication, i.e. the keys and values being transmitted are not huge,
  • the main computation time is spent inside the map() or reduce() function, not communication.

Also, the simplified system

  • holds everything in memory - the input, the keys, and the values in every stage of the mapreduce run.
  • does not handle server errors. If the server is down, you have to restart mapreduce.
  • partially tolerates client failure. If a client is down, its last map() call will simply be re-run on another client.

I have used it to perform feature extractions on ImageNet and it works pretty well for our research use. But it may or may not fit your use case.

To see more documentation read the wiki.


Lightweight MapReduce in python



No packages published


You can’t perform that action at this time.