Provide a variant of PyDict_SetDefault that returns a new reference (instead of a borrowed reference)

[colesbury]

The PyDict_SetDefault(mp, key, defaultobj) function returns a borrowed reference to the value corresponding to key. This poses a thread-safety issue particularly for the case where key is already in the dict. In the --disable-gil builds, the returned value may no longer be valid if another thread concurrently modifies the dict.

Proposal (from Victor)

int PyDict_SetDefaultRef(PyObject *dict, PyObject *key, PyObject *default_value, PyObject **value);

The **value pointer is optional. If it is NULL, it is not used.

• If the key is present in the dict, set *value to a new reference to the current value (if value is not NULL), and return 1.
• If the key is missing from the dict, insert the key and default_value, and set *value to a new reference to default_value (if value is not NULL), and return 0.
• On error, set *value to NULL if value is not NULL, and return -1.

Ideally, this new function would be public and part of the stable ABI so that it could be used by all extensions, but even an internal-only function would unblock some of the nogil changes.

EDIT: Updated with @vstinner's proposal

Linked PRs

• gh-112066: Add PyDict_SetDefaultRef function. #112123
• gh-112066: Use PyDict_SetDefaultRef in place of PyDict_SetDefault. #112211

[colesbury]

cc @vstinner, @encukou

[vstinner]

Following previous API such as PyDict_GetItemRef() and PyDict_Pop(), I propose the following API:

int PyDict_SetDefaultRef(PyObject *dict, PyObject *key, PyObject *default_value, **value)

• If key is present in dict, set *value to a new reference to the current value if value is not NULL, and return 1.
• If key is missing in dict, set key to default_value in dict, set *value to a new reference to the default_value if value is not NULL, and return 0.
• On error, set *value to NULL if value is not NULL, and return -1.

So value can be NULL if the value is not used.

See also API evolution: Return value conventions.

cc @zooba @serhiy-storchaka @erlend-aasland

[colesbury]

@vstinner - sounds good. I updated the issue description with your proposal.

[vstinner]

Do you want to propose a PR?