Fix pickling exceptions with multiple arguments

[slallum]

BPO	32696
Nosy	@jaraco, @avassalotti, @orivej, @cryvate, @slallum, @benoit-pierre, @iritkatriel, @Zefir-13000
PRs	bpo-32696 Fixed pickling exceptions with multiple arguments #30602

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-06.16:16:09.026>
created_at = <Date 2018-01-28.16:15:55.184>
labels = ['interpreter-core', 'type-bug', 'invalid', '3.11']
title = 'Fix pickling exceptions with multiple arguments'
updated_at = <Date 2022-01-15.07:35:30.412>
user = 'https://github.com/slallum'

bugs.python.org fields:

activity = <Date 2022-01-15.07:35:30.412>
actor = 'zseil'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-06.16:16:09.026>
closer = 'iritkatriel'
components = ['Interpreter Core']
creation = <Date 2018-01-28.16:15:55.184>
creator = 'slallum'
dependencies = []
files = []
hgrepos = []
issue_num = 32696
keywords = []
message_count = 7.0
messages = ['310963', '315225', '315230', '396035', '396040', '396601', '409867']
nosy_count = 11.0
nosy_names = ['jaraco', 'alexandre.vassalotti', 'orivej', 'sbt', 'cryvate', 'slallum', 'Kirill Matsaberydze', 'benoit-pierre', '4-launchpad-kalvdans-no-ip-org', 'iritkatriel', 'Zefir-13000']
pr_nums = ['30602']
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32696'
versions = ['Python 3.11']

[slallum]

Pickling exceptions fails when a custom exception class requires multiple arguments.

import pickle
class MultipleArgumentsError(Exception):
    def __init__(self, a, b):
        self.a = a
        self.b = b

pickle.loads(pickle.dumps(MultipleArgumentsError('a', 'b')))

this code produces the following error:

Traceback (most recent call last):
  File "/tmp/multiple_arguments_exception.py", line 8, in <module>
    pickle.loads(pickle.dumps(MultipleArgumentsError('a', 'b')))
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1388, in loads
    return Unpickler(file).load()
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1139, in load_reduce
    value = func(*args)
TypeError: __init__() takes exactly 3 arguments (2 given)

The same error occurs when using a built-in module like subprocess:
>>> import subprocess, pickle
>>> try:
...   subprocess.check_call(['python', '-c', 'raise SystemExit(1)'])
... except Exception as e:
...   pickle.loads(pickle.dumps(e))
... 

Related issue: https://bugs.python.org/issue1692335

[KirillMatsaberydze]

Hi, I encounter similar behavior in python 3.6.5 with following code:

import pickle
class CustomException(Exception):
    def __init__(self, arg1, arg2):
        msg = "Custom message {} {}".format(arg1, arg2)
        super().__init__(msg)


obj_dump = pickle.dumps(CustomException("arg1", "arg2"))
obj = pickle.loads(obj_dump)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __init__() missing 1 required positional argument: 'arg2'

So it looks like it not only python 2.7 problem

[jaraco]

Kirill, see https://bugs.python.org/issue1692335#msg310951 in the related issue for one possible way to work around the issue on Python 3.

[iritkatriel]

This is still the same in 3.11:

>>> import pickle
>>> class CustomException(Exception):
...     def __init__(self, arg1, arg2):
...         msg = "Custom message {} {}".format(arg1, arg2)
...         super().__init__(msg)
...
>>> obj_dump = pickle.dumps(CustomException("arg1", "arg2"))
>>> obj = pickle.loads(obj_dump)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: CustomException.__init__() missing 1 required positional argument: 'arg2'
>>>

[Cryvate]

It seems like the example in the OP now works (persist both arguments), but Irit's/Kirill's (create a msg) fails (I am running 3.9).

[iritkatriel]

See also bpo-43460, bpo-30005, bpo-29466

[iritkatriel]

This isn't really an issue with exceptions, it's just that __reduce__ goes out of sync with the object's constructor signature. Here's a simple example of the same:

class Base:
   def __init__(self, msg):
       self.msg = msg

   def __reduce__(self):
       return type(self), (self.msg,)

class Derived(Base):
   def __init__(self, a, b):
       super().__init__(f'{a}|{b}')

x = Derived('a', 'b')
assert x.msg == 'a|b'
y = pickle.dumps(x, -1)
z = pickle.loads(y)

---

Output:

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/zz.py", line 22, in <module>
    z = pickle.loads(y)
        ^^^^^^^^^^^^^^^
TypeError: Derived.__init__() missing 1 required positional argument: 'b'

If I define __reduce__ on Derived to return an arg tuple of the right length for its __init__, it works:

class Derived(Base):
   def __init__(self, a, b):
       super().__init__(f'{a}|{b}')

   def __reduce__(self):
       return type(self), tuple(self.msg.split('|'))

But note that this is not something we could make the base class do, it has no way of finding out what the semantics of the derived class constructor args are.

[youtux]

Should it be documented in the docs how to implement __reduce__ on custom exceptions?