MultiMapper fails with single-parameter mappers #70

Open
jkleint opened this Issue Mar 18, 2013 · 1 comment

Comments

Projects
None yet
2 participants
@jkleint
Contributor

jkleint commented Mar 18, 2013

If I write a map function with the alternative low-level single-parameter interface, then give it to MultiMapper:

import dumbo
from dumbo.lib import MultiMapper
from dumbo.decor import primary

@primary
def mapper(keysvals):
    for key, val in keysvals:
        yield key, val

if __name__ == "__main__":
    multimapper = MultiMapper()
    multimapper.add("primary", mapper)
    dumbo.run(multimapper)

It tries to call it with two parameters:

...
  File "build/bdist.linux-x86_64/egg/dumbo/core.py", line 322, in run
  File "build/bdist.linux-x86_64/egg/dumbo/util.py", line 49, in dumpcode
  File "build/bdist.linux-x86_64/egg/dumbo/core.py", line 314, in <genexpr>
  File "build/bdist.linux-x86_64/egg/dumbo/core.py", line 445, in mapfunc_iter
  File "build/bdist.linux-x86_64/egg/dumbo/lib/__init__.py", line 180, in __call__

TypeError: mapper() takes exactly 1 argument (2 given)

It looks like MultiMapper doesn't know about the alternative interface.

@a4tunado

This comment has been minimized.

Show comment Hide comment
@a4tunado

a4tunado Apr 23, 2013

In general cases mapper callable object should support 2-parameters interface for ex.

def mapper(key, value):
    yield key, value.split('\t')

Where value is usually a single value or a line but not a generator.

In general cases mapper callable object should support 2-parameters interface for ex.

def mapper(key, value):
    yield key, value.split('\t')

Where value is usually a single value or a line but not a generator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment