UnicodeError becomes unpicklable if data is appended to args

[JooEiras]

BPO	39750
Nosy	@serhiy-storchaka
Files	test_unicode_error_unpickle.py: Test case

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-02-25.16:15:36.125>
labels = ['interpreter-core', 'type-bug', '3.8']
title = 'UnicodeError becomes unpicklable if data is appended to args'
updated_at = <Date 2020-03-01.17:55:50.780>
user = 'https://bugs.python.org/JooEiras'

bugs.python.org fields:

activity = <Date 2020-03-01.17:55:50.780>
actor = 'Jo\xc3\xa3o Eiras'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2020-02-25.16:15:36.125>
creator = 'Jo\xc3\xa3o Eiras'
dependencies = []
files = ['48914']
hgrepos = []
issue_num = 39750
keywords = []
message_count = 4.0
messages = ['362648', '362655', '362695', '363071']
nosy_count = 2.0
nosy_names = ['serhiy.storchaka', 'Jo\xc3\xa3o Eiras']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39750'
versions = ['Python 3.8']

[JooEiras]

Given some exception ex, you can append data like
ex.args += (value1, value2, ...)
and then re-raise.

This is something I do in my projects to sometime propagate context when errors are raised, e.g., stacktraces across process boundaries or blobs of text with pickling or unicode errors.

When this is done with UnicodeError, the exception becomes non-unpicklable:

TypeError: function takes exactly 5 arguments (6 given)

Example:
import pickle

    def test_unicode_error_unpickle():
        ex0 = UnicodeEncodeError('ascii','message', 1, 2, 'e')
        ex0.args += ("extra context",)
        ex1 = pickle.loads(pickle.dumps(ex0))
        assert type(ex0).args == type(ex1).args
        assert ex0.args == ex1.args

The issue seems to be UnicodeEncodeError_init() at https://github.com/python/cpython/blob/v3.8.1/Objects/exceptions.c#L1895 and also UnicodeDecodeError_init().

The BaseException is initialized, but then Unicode*Error_init() tries to reparse the arguments and does not tolerate extra values.

This because BaseException.__reduce__ return a tuple (class,args).

[serhiy-storchaka]

Yes, this is how exceptions are pickled. You save an information which which allows you to recreate the exception. If you fake args, you do not able to recreate it.

Just do not do this.

[JooEiras]

Hi.
It's perfectly fine for classes to have their public APIs and intended uses.

But then unpickling would be the worst place to complain, specially when running 40 parallel processes while an unhandled stacktrace appears between a couple hundred thousand lines of logging data :)

[JooEiras]

On a related note, after inspecting the UnicodeEror C code, the exception object keeps explicit references to 'encoding', 'object', 'start', 'end' and 'reason'. That means that if those properties are set (the C code does have setters) then the properties stored in UnicodeError go out of sync with the args tuple in BaseException. And so pickling and unpickling will restore the original values, and not those that were set after the exception being created, unless args is too modified.

My suggestion would be to just fetch all 5 properties from the args tuple inside the getters and setters, and in the setters, recreate the tuple with a modified value. The constructor could do argument validation but would not set any properties, because that would be delegated to BaseException.

[iritkatriel]

I agree with @serhiy-storchaka's comment. If you want to arbitrarily change the API of an object you need to reimplement its pickling functions accordingly.