Pickle failure is raising AttributeError and not PicklingError

[MattDodge]

BPO	29187
Nosy	@avassalotti, @serhiy-storchaka, @iritkatriel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-01-06.23:06:54.718>
labels = ['extension-modules', 'type-bug', '3.8', '3.9', '3.10']
title = 'Pickle failure is raising AttributeError and not PicklingError'
updated_at = <Date 2021-02-17.19:39:09.059>
user = 'https://bugs.python.org/MattDodge'

bugs.python.org fields:

activity = <Date 2021-02-17.19:39:09.059>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2017-01-06.23:06:54.718>
creator = 'Matt.Dodge'
dependencies = []
files = []
hgrepos = []
issue_num = 29187
keywords = []
message_count = 3.0
messages = ['284869', '284896', '387185']
nosy_count = 4.0
nosy_names = ['alexandre.vassalotti', 'serhiy.storchaka', 'Matt.Dodge', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29187'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[MattDodge]

When failing to pickle something (like a locally scoped class) the documentation indicates that a PicklingError should be raised.
Doc links:

• https://docs.python.org/3/library/pickle.html?highlight=pickle#pickle-picklable
• https://docs.python.org/3.5/library/pickle.html#what-can-be-pickled-and-unpickled

However, instead I'm seeing AttributeError get raised instead, starting in Python 3.5. In Python 3.4 PicklingErrror was raised, as expected.

To reproduce, use the following file <pickletest.py>
def func():
class C: pass
return C
import pickle
pickle.dumps(func()())

In Python 3.4 you see:
Traceback (most recent call last):
  File "pickletest.py", line 5, in <module>
    pickle.dumps(func()())
_pickle.PicklingError: Can't pickle <class '__main__.func.<locals>.C'>: attribute lookup C on __main__ failed

But in 3.5/3.6 you see:
Traceback (most recent call last):
  File "pickletest.py", line 5, in <module>
    pickle.dumps(func()())
AttributeError: Can't pickle local object 'func.<locals>.C'

I don't necessarily mind that a different exception is being raised, but how should we be handling exceptions while pickling? Catch all exceptions? That doesn't feel right to me, but if we're trying to pickle data out of our control I'm not sure what else to do.

FYI, the UnpicklingError documentation (https://docs.python.org/3/library/pickle.html?highlight=pickle#pickle.UnpicklingError) indicates that other exceptions can possibly be raised during unpickling. I assume that is more related to the fact that the pickled data may not fit into the current class/module structure though, so I think it's unrelated.

[serhiy-storchaka]

Python implementation of pickle still raises PicklingError. Seems this was not intentional change.

>>> pickle._dumps(func()())
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 918, in save_global
    obj2, parent = _getattribute(module, name)
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 266, in _getattribute
    .format(name, obj))
AttributeError: Can't get local attribute 'func.<locals>.C' on <function func at 0xb7118d1c>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 1544, in _dumps
    _Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 409, in dump
    self.save(obj)
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 605, in save_reduce
    save(cls)
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 978, in save_type
    return self.save_global(obj)
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 922, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle <class '__main__.func.<locals>.C'>: it's not found as __main__.func.<locals>.C

[iritkatriel]

Still the same in 3.10:

Python 3.10.0a5+ (heads/bpo-43146-dirty:8f5cf4d381, Feb 17 2021, 14:51:27) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> try:
...     def func():
...         class C: pass
...         return C
...     import pickle
...     pickle.dumps(func()())
... except BaseException as e:
...   exc = e
...
>>> exc
AttributeError("Can't pickle local object 'func.<locals>.C'")
>>>