Cannot pickle Ellipsis or NotImplemented

[JamesSanders]

BPO	13842
Nosy	@pitrou, @avassalotti, @ambv, @hynek
Files	pickling_ellipsis_notimplemented.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ambv'
closed_at = <Date 2012-03-12.19:01:04.997>
created_at = <Date 2012-01-23.19:24:18.518>
labels = ['type-feature']
title = 'Cannot pickle Ellipsis or NotImplemented'
updated_at = <Date 2012-03-12.21:59:55.192>
user = 'https://bugs.python.org/JamesSanders'

bugs.python.org fields:

activity = <Date 2012-03-12.21:59:55.192>
actor = 'python-dev'
assignee = 'lukasz.langa'
closed = True
closed_date = <Date 2012-03-12.19:01:04.997>
closer = 'lukasz.langa'
components = []
creation = <Date 2012-01-23.19:24:18.518>
creator = 'James.Sanders'
dependencies = []
files = ['24315']
hgrepos = []
issue_num = 13842
keywords = ['patch']
message_count = 10.0
messages = ['151838', '151843', '151844', '151852', '151911', '153708', '153709', '155460', '155461', '155492']
nosy_count = 6.0
nosy_names = ['pitrou', 'alexandre.vassalotti', 'lukasz.langa', 'python-dev', 'hynek', 'James.Sanders']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13842'
versions = ['Python 3.3']

[JamesSanders]

At present, the built-in constants Ellipsis (...) and NotImplemented cannot be pickled. Perhaps there is a good reason for this, but the only discussion I can find is at msg<108957>, where it is stated that these values (along with their types, and type(None)) cannot be pickled.

I ran across this in a class that keeps track of numpy-style slicing operations, and so sometimes stores references to Ellipsis. While this is easy to work around, it does seem a little surprising that ... cannot be pickled, when slice objects can be. I don't know if there is a likely use for pickling NotImplemented.

If this is not changed, perhaps it could be explicitly stated in the documentation that these objects cannot be pickled?

[pitrou]

I think it's a reasonable feature request. Now someone has to write a patch for it.

[avassalotti]

We will need to bump the protocol number to add support for None, Ellipsis, and NotImplemented. Antoine, can you add this to PEP-3154?

[pitrou]

We will need to bump the protocol number to add support for None,
Ellipsis, and NotImplemented. Antoine, can you add this to PEP-3154?

I don't think this needs a protocol bump. These are global singletons,
they can be pickled as such, and they will be unpickleable on any
Python.

[JamesSanders]

I've submitted a patch that just uses save_global to pickle Ellipsis and NotImplemented, so the resulting pickle should be unpicklable anywhere. I'm completely new to the C API so not sure if the way I am building python strings (to pass to save_global) is correct, but it seems to work.

[pitrou]

I haven't taken a look at the patch in detail, but it lacks some tests.
Pickling tests are generally in Lib/test/pickletester.py, they are invoked by test_pickle and test_pickletools.

[pitrou]

I'm completely new to the C API so not sure if the way I am building
python strings (to pass to save_global) is correct

It's correct but it's probably less efficient than calling a more specialized function, such as PyUnicode_FromString():
http://docs.python.org/dev/c-api/unicode.html#PyUnicode_FromString

You also have to check the return value (for non-NULL, NULL meaning an error occurred) before calling save_global with it.

[ambv]

There is indeed a 5% performance gain from using PyUnicode_FromString instead of the generic Py_BuildValue function.

[python-dev]

New changeset 5832d3622473 by Łukasz Langa in branch 'default':
Fixes bpo-13842: cannot pickle Ellipsis or NotImplemented.
http://hg.python.org/cpython/rev/5832d3622473

[python-dev]

New changeset 5353357382e2 by Łukasz Langa in branch 'default':
bpo-13842: check whether PyUnicode_FromString succeeded
http://hg.python.org/cpython/rev/5353357382e2