Assertion failure in Objects/unicodeobject.c `_PyUnicode_DecodeUnicodeEscapeInternal2: Assertion 'end - s <= writer.size - writer.pos' failed`

[YuanchengJiang]

Bug report

Bug description:

import codecs

decode = codecs.unicode_escape_decode
data = {b'\\x0': (b'\xe2\x98\x83', 0)}

def mutating(exc):
    key = exc.object[:exc.end]
    r = data.get(key)
    if r is not None:
        return ('Є', r[1])

codecs.register_error('test.mutating2', mutating)

input_obj = b'\\x0n\\z'
result, length = decode(input_obj, 'test.mutating2')

python: ../Objects/unicodeobject.c:6687: _PyUnicode_DecodeUnicodeEscapeInternal2: Assertion `end - s <= writer.size - writer.pos' failed.
Aborted (core dumped)

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-141336: Fix Unicode escape assertion failure in error handler #141344

[mohsinm-dev]

I think the issue comes from how the error handler return value is accounted for in the unicode-escape / raw-unicode-escape decoder path.

When the custom error handler returns a replacement string and also rewinds the input (so that more input remains to be processed), the decoder needs to ensure there is enough room for:

1. The entire replacement string, and
2. The newly increased "remaining input" that will be decoded next.

Currently, the helper only adds (replacement_length - 1) to the required buffer size. That works only if the caller already pre-reserved space for exactly one output character, which isn’t true here. So in cases where the replacement length is exactly 1 and the input is rewound, the buffer ends up short by 1 character → leading to the assertion failure.

The correct behavior is to always account for the full replacement length when updating writer.min_length, along with any delta in remaining input. That ensures the writer capacity stays consistent with the decoder’s assumptions across all callers.

[serhiy-storchaka]

I am not sure that this is the correct fix. It will cause an unnecessary overallocation in normal case.

Rewinding the position is already problematic, we were going to disable it, as it can lead to infinite loop. But at least there should be guard preventing crash. Something wrong with these guards. Or they were not added yet? I need to look at past issues.

[mohsinm-dev]

@serhiy-storchaka I think the core idea here is to avoid growing the buffer every time and instead only do so when it is actually needed. The goal is to keep the assertion safe when the error handler rewinds the position, without introducing unnecessary overallocation in the normal path.

The way I am thinking about it is:

required = writer->pos + replen + new_remaining;
if (required > writer->min_length) {
    writer->min_length = required;
}

So the buffer only grows when this lower bound is larger than the current minimum. In cases where replen == 1 and there is no rewind, it should not trigger any extra allocation. And when the error handler rewinds input, this should still ensure the writer has enough room so the assertion remains valid.

On the infinite loop concern with rewinding, I agree that is a broader behavioral question. Here I am only focusing on making sure the writer capacity stays consistent so the crash does not occur. I may be missing something.