New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

why can not pickle lambda in spark but useful in pure python code. #2584

Open
143230 opened this Issue Aug 28, 2018 · 1 comment

Comments

Projects
None yet
2 participants
@143230

143230 commented Aug 28, 2018

i submit my spark job using pure python like below:
sc.textFile(dp)..map(lambda line: line.split(' '))
it is okay and perfect,
however, when using cython and change it into .so file, the spark raise exceptions and told
me that can not pickle lambda function.

help me, thanks

@robertwb

This comment has been minimized.

Show comment
Hide comment
@robertwb

robertwb Aug 28, 2018

Contributor

Spark pickles lambdas by capturing the bytecode/closure and re-creating it on the other end. There's no bytecode and the closure may be non-Python objects for Cython-defined lambdas. However, it is a valid feature request to make this possible by implementing their __reduce__ methods.

Contributor

robertwb commented Aug 28, 2018

Spark pickles lambdas by capturing the bytecode/closure and re-creating it on the other end. There's no bytecode and the closure may be non-Python objects for Cython-defined lambdas. However, it is a valid feature request to make this possible by implementing their __reduce__ methods.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment