Can't pickle lambda (while named functions are ok)

[facundobatista]

BPO	19272
Nosy	@facundobatista, @jcea, @amauryfa, @bitdancer, @ethanfurman
Files	issue19272.stoneleaf.01.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-10-18.07:57:47.349>
created_at = <Date 2013-10-16.16:56:36.277>
labels = ['invalid', 'type-bug', 'library']
title = "Can't pickle lambda (while named functions are ok)"
updated_at = <Date 2013-10-18.07:57:47.269>
user = 'https://github.com/facundobatista'

bugs.python.org fields:

activity = <Date 2013-10-18.07:57:47.269>
actor = 'ethan.furman'
assignee = 'docs@python'
closed = True
closed_date = <Date 2013-10-18.07:57:47.349>
closer = 'ethan.furman'
components = ['Library (Lib)']
creation = <Date 2013-10-16.16:56:36.277>
creator = 'facundobatista'
dependencies = []
files = ['32155']
hgrepos = []
issue_num = 19272
keywords = ['patch']
message_count = 14.0
messages = ['200062', '200063', '200064', '200066', '200067', '200068', '200069', '200070', '200074', '200090', '200099', '200101', '200192', '200227']
nosy_count = 7.0
nosy_names = ['facundobatista', 'jcea', 'amaury.forgeotdarc', 'r.david.murray', 'docs@python', 'ethan.furman', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19272'
versions = ['Python 3.4']

[facundobatista]

This is ok:

Python 3.4.0a3+ (default:86af5991c809, Oct 13 2013, 16:42:52) 
...
>>> import pickle
>>> def f():  
...   pass
... 
>>> pickle.dumps(f)
b'\x80\x03c__main__\nf\nq\x00.'


However, when trying to pickle a lambda, it fails:
>>>         
>>> pickle.dumps(lambda: None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup builtins.function failed

(Found this because I'm getting the same problem but when trying to use concurrent.futures.ProcessExecutor)

[amauryfa]

Functions are pickled by name, not by code.
Unpickling will only work if a function with the same name is present in in the same module (main in your example)

This is why pickling a lambda won't work: they have no individual names.

[jcea]

Would be interesting to be able to pickle "function.__code__".

Although, thinking about this, it would be not portable between Python releases, something that pickle guarantee.

Thoughs?

[ethanfurman]

According to the docs[1]:

12.1.4. What can be pickled and unpickled?

The following types can be pickled:

- None, True, and False
- integers, floating point numbers, complex numbers
- strings, bytes, bytearrays
- tuples, lists, sets, and dictionaries containing only picklable objects
- functions defined at the top level of a module
- built-in functions defined at the top level of a module
- classes that are defined at the top level of a module
- instances of such classes whose __dict__ or the result of calling
  __getstate__() is picklable 

Notice that lambda is not in that list.

The docs for concurrent.futures.ProcessPoolExecutor[2] state:

17.4.3. ProcessPoolExecutor

The ProcessPoolExecutor class is an Executor subclass that uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock but also means that only picklable objects can be executed and returned.

[1] http://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled
[2] http://docs.python.org/dev/libraryconcurrent.futures.html?highlight=concurrent#processpoolexecutor

[facundobatista]

Ethan, lambda functions are included in "functions defined at the top level of a module".

Probably we should note there something like "except lambdas, because function pickling is by name, not by code".

[facundobatista]

Jesús, Amaury:

What if pickle would assign a random unique name to the lambda (like, an UUID) so it can be pickled and unpickled?

[ethanfurman]

Yeah, that one line should say, "named functions defined at the top level of a module".

The following three paragraphs do, however, mention "named" several times.

Sounds like a doc issue.

[ethanfurman]

The problem with a randam unique name is making sure you get the same random unique name across different runs, different pythons, and different orders of execution.

[bitdancer]

So don't make it random, use a hash of the code object :)

(I'm not sure I'm serious, the thought just popped into my head...)

[jcea]

Lambdas are "anonymous" functions, by definition. And, usually, they are not top level functions, but defined inside others.

If at your "top level" (module) you do:

"""
a = lambda x: 2*x
"""

You don't have an anonymous function, but a function called "a". You can argue why "def a() : return 2*x" can be picked, but "a = lambda x: 2*x" can not. They look similar.

What the poster actually wanted (tell me if I am wrong), I guess, is to be able to serialize the function code and send it to other process to be executed there. Something like "mobile" code. As is, pickle doesn't allow it (that is the reason I was brainstorming about being able to pickle "function.__code__" objects), and it is good because pickle guarantees compatibilities between Python versions, but "__code__" objects are particular to a certain Python virtual machine and certain version of it.

That said, projects like PYRO provide "mobile code". I think that a recipe for that would be nice in the documentation, with a big warning saying "be sure you have the same implementation in both sides", and explicit versioning in the client and version checking in the server, in the recipe (to make it clear that same version is something you need).

[ethanfurman]

From the pickle docs:

=====================================================================
Note that functions (built-in and user-defined) are pickled by “fully
qualified” name reference, not by value. This means that only the
function name is pickled, along with the name of the module the
function is defined in. Neither the function’s code, nor any of its
function attributes are pickled. Thus the defining module must be
importable in the unpickling environment, and the module must contain
the named object, otherwise an exception will be raised.

Similarly, classes are pickled by named reference, ...
=====================================================================

There is no bug here; there /may/ be a doc clarification here.

However, if we're talking about enhancing pickle we can take 3.3 off the table, and unless somebody gets busy quick 3.4 as well.

Personally, I don't see the need. Just define your functions at the top level (with def, not lambda -- lambda functions are anonymous and don't have a useful __name__ attribute).

[ethanfurman]

Jesús Cea Avión added the comment:

If at your "top level" (module) you do:

"""
a = lambda x: 2*x
"""

You don't have an anonymous function, but a function called "a".

Actually, you do have an anonymous function, which happens to be bound to the name "a".

Compare:

--> def a():
...     pass
--> a.__name__
'a'

and

--> a = lambda: None
--> a.__name__
'<lambda>'

[python-dev]

New changeset d103ba56710e by Ethan Furman in branch 'default':
Issue bpo-19272: slight clarification of pickle docs with regard to lambda.
http://hg.python.org/cpython/rev/d103ba56710e

[ethanfurman]

Added some clarification to the docs to make it clearer that lambda functions cannot be pickled.

Facundo [1], if you want to pursue being able to pickle lambda functions please open an enhancement issue. Some of the questions that come to mind:

1. for a random name, where will the name be stored (I'm thinking a
   __pickle__ dict on the module)?

2. for pickling the code object itself, what are the ramifications
   (this is directly contrary to how pickle was designed)

These questions should be discussed on PyDev, and will probably require a PEP.

[1] or anyone else :)