Behaviour of unformattable ValueError on list.index has changed in Python 3.11 (change in PyErr_Format)

[da-woods]

Bug report

Bug description:

lst = []

class C:
    def __repr__(self):
        raise RuntimeError("Oh no!")

try:
    lst.index(C())
except ValueError as e:
    print("This is good")
    print(repr(e))

On Python 3.10 this prints

This is good
ValueError()

On Python 3.11 this prints

Traceback (most recent call last):
  File "<path>", line 8, in <module>
    lst.index(C())
  File "<path>", line 5, in __repr__
    raise RuntimeError("Oh no!")
RuntimeError: Oh no!

When PyErr_Format fails in current versions of Python, it doesn't set the exception and instead raises the new one:

cpython/Python/errors.c

Lines 1162 to 1167 in 11d88a1

	if (string != NULL) {
	_PyErr_SetObject(tstate, exception, string);
	Py_DECREF(string);
	}
	return NULL;
	}

In earlier versions of Python it used to raise the requested exception unconditionally

cpython/Python/errors.c

Lines 1059 to 1064 in b6535ea

	string = PyUnicode_FromFormatV(format, vargs);
	_PyErr_SetObject(tstate, exception, string);
	Py_XDECREF(string);
	return NULL;
	}

This looks to have been a deliberate change but it can affect the behaviour of real code (cython/cython#5894). Interestingly PyPy got there before you with this behaviour (https://foss.heptapod.net/pypy/pypy/-/issues/3978)

CPython versions tested on:

3.11

Operating systems tested on:

Linux

[da-woods]

Forgot to link where it was introduced: Issue #107915; PR: #107918

[Eclips4]

cc @serhiy-storchaka

[serhiy-storchaka]

Yes, it is a deliberate change. If Cython crashes because of this, it has bugs, and this change helped to expose them.

[da-woods]

If Cython crashes because of this, it has bugs, and this change helped to expose them.

I mostly agree with this, and have a fix for the Cython issue that will be applied whatever you decide to do with this.

If I were designing this I'd probably treat the first exception as the "main error", and any exception from generating the format string as "unraisable". Just because the error message attached to the exception feels like it could be treated as optional compared to the exception itself.

However, just wanted to flag that there's a behavioural change that isn't too hard to hit - I don't think it's too uncommon for __repr__ to be a bit fragile while a class is being initialized for example.

[terryjreedy]

Would it be possible to print a double traceback, as in somewhat similar situations? (The following is not exact.)

Traceback ...
... <traceback lines>
ValueError()

While processing this exception, another exception occurred.
Traceback...
... <traceback lines>
RuntimeError(...)

[serhiy-storchaka]

No, it is unreasonable. Another exception occurred before the intended exception was created, so the intended exception cannot be raised.

Look at the following example:

raise ValueError(1/0)

There is no way to get a ValueError instead or in addition to a ZeroDivisionError in a traceback. ValueError cannot be created, because its argument cannot be evaluated. Creation of exception with wrong argument is a bug, because the exception argument can matter. And an exception raised when calculate exception arguments (for example KeyboardInterrupt, MemoryError or RecursionError which can be raised by virtually any code) does matter and should not be silenced.