Calling _PyBytes_Resize() on 1-byte bytes may raise error

[animalize]

BPO	43027
Nosy	@animalize

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-01-26.03:52:04.994>
labels = ['interpreter-core', 'type-bug', '3.8', '3.9', '3.10']
title = 'Calling _PyBytes_Resize() on 1-byte bytes may raise error'
updated_at = <Date 2021-01-26.03:52:04.994>
user = 'https://github.com/animalize'

bugs.python.org fields:

activity = <Date 2021-01-26.03:52:04.994>
actor = 'malin'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2021-01-26.03:52:04.994>
creator = 'malin'
dependencies = []
files = []
hgrepos = []
issue_num = 43027
keywords = []
message_count = 1.0
messages = ['385689']
nosy_count = 1.0
nosy_names = ['malin']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43027'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• gh-87193: Support bytes objects with refcount > 1 in _PyBytes_Resize() #117160

[animalize]

PyBytes_FromStringAndSize() uses a global cache for 1-byte bytes:
https://github.com/python/cpython/blob/v3.10.0a4/Objects/bytesobject.c#L147

    if (size == 1 && str != NULL) {
        struct _Py_bytes_state *state = get_bytes_state();
        op = state->characters[*str & UCHAR_MAX];
        if (op != NULL) {
            Py_INCREF(op);
            return (PyObject *)op;
        }
    }

_PyBytes_Resize() will raise an error when (refcount != 1):
https://github.com/python/cpython/blob/v3.10.0a4/Objects/bytesobject.c#L3029

Then this code will raise an exception:

    obj1 = PyBytes_FromStringAndSize("a", 1);
    obj2 = PyBytes_FromStringAndSize("a", 1);
    ret = _PyBytes_Resize(&obj2, 2);  // ret is -1

BTW, 0-byte bytes comes from a global singleton, but _PyBytes_Resize() processes it before checking refcount:
https://github.com/python/cpython/blob/v3.10.0a4/Objects/bytesobject.c#L3021

    if (Py_SIZE(v) == 0) {
        if (newsize == 0) {
            return 0;
        }
        *pv = _PyBytes_FromSize(newsize, 0);
        Py_DECREF(v);
        return (*pv == NULL) ? -1 : 0;
    }
    if (Py_REFCNT(v) != 1) {
        goto error;
    }

_PyBytes_Resize() doc:

Only use this to build up a brand new bytes object; don’t use this if the
bytes may already be known in other parts of the code. It is an error to
call this function if the refcount on the input bytes object is not one.

[encukou]

IMO, it's a docs issue: PyBytes_From* should say that it returns a new reference, not necessarily to a brand new object.

[serhiy-storchaka]

#117160 fixes this for cached 1-byte bytes objects, and also adds support of arbitrary bytes objects with refcount > 1. Note that resizing the empty bytes object was already supported even if it is always shared.

I am not sure whether this should be backported.

[encukou]

I'd say your fix is a new feature, and shouldn't be backported.