unittest framework for dumbo mapreduce #21

Closed
klbostee opened this Issue Oct 6, 2010 · 1 comment

Projects

None yet

1 participant

@klbostee
Owner
klbostee commented Oct 6, 2010

Adam wrote a small module, inspired by cloudera's MRUnit, that lets one easily create unit-tests for dumbo mapreduce tasks.

The code is currently at:

http://github.com/adamhadani/dumbo/blob/master/dumbo/mapredtest.py

He also added a unittest for it that serves the double purpose of unit-testing the unit-testing itself
as well as serving as example of how to work with it:

http://github.com/adamhadani/dumbo/blob/master/tests/testmapredtest.py

The nice thing about it is that it takes care of some things behind the scenes, e.g deriving the mapper/reducer classes from mapredbase when needed, making sure input/output is iterable (allowing for arbitrarily large input/output test cases - need not fit in memory as seems to be the case with MRUnit), and so on.

@klbostee
Owner

Just merged this into my master branch.

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment