New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unittest framework for dumbo mapreduce #21

Closed
klbostee opened this Issue Oct 6, 2010 · 1 comment

Comments

Projects
None yet
1 participant
@klbostee
Owner

klbostee commented Oct 6, 2010

Adam wrote a small module, inspired by cloudera's MRUnit, that lets one easily create unit-tests for dumbo mapreduce tasks.

The code is currently at:

http://github.com/adamhadani/dumbo/blob/master/dumbo/mapredtest.py

He also added a unittest for it that serves the double purpose of unit-testing the unit-testing itself
as well as serving as example of how to work with it:

http://github.com/adamhadani/dumbo/blob/master/tests/testmapredtest.py

The nice thing about it is that it takes care of some things behind the scenes, e.g deriving the mapper/reducer classes from mapredbase when needed, making sure input/output is iterable (allowing for arbitrarily large input/output test cases - need not fit in memory as seems to be the case with MRUnit), and so on.

@klbostee

This comment has been minimized.

Show comment
Hide comment
@klbostee

klbostee Dec 10, 2010

Owner

Just merged this into my master branch.

Owner

klbostee commented Dec 10, 2010

Just merged this into my master branch.

This issue was closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment