[C API] Add PyObject_GetDict() function

[vstinner]

Feature or enhancement

Feature or enhancement

There is a private _PyObject_GetDictPtr() function which is used by 30 projects of the PyPI top 15,000 projects:

• accesscontrol (7.2)
• acquisition (6.1)
• backports_datetime_fromisoformat (2.0.3)
• casadi (3.7.2)
• catboost (1.2.8)
• cvxpy (1.7.2)
• cvxpy_base (1.7.2)
• cython (3.1.4)
• extensionclass (6.0)
• frozendict (2.4.6)
• gamsapi (51.1.0)
• gdal (3.11.4)
• gevent (25.9.1)
• guppy3 (3.1.5)
• htseq (2.0.9)
• hunter (3.9.0)
• ihm (2.7)
• jpype1 (1.6.0)
• m2crypto (0.45.1)
• mecab-ko (1.0.1)
• mecab_python3 (1.0.10)
• mupdf (1.21.1.20230112.1504)
• passagemath_standard_no_symbolics (10.6.26)
• persistence (5.1)
• persistent (6.1.1)
• pybind11 (3.0.1)
• pygdal (3.6.4.11)
• pygraphviz (1.14)
• python_crfsuite (0.9.11)
• sentencepiece (0.2.1)

I propose to make the function public with a different API to no longer ignore errors silently.

UPDATE: I now propose adding PyObject_GetDict() (PyObject **dict parameter), instead of PyObject_GetDictPtr() (PyObject ***dict_ptr parameter).

Linked PRs

• gh-139852: Add PyObject_GetDictPtr() function #139854
• gh-139852: Add PyObject_GetDict() function #139968

[markshannon]

I don't think this is a good idea.

There is reason this function is private. Using this function is potentially dangerous, the name is misleading and it is may be perpetuating a misunderstanding of how Python objects are implemented.

It is dangerous, because assigning directly to the pointer returned by this function could result in the object and its __dict__ getting out of sync.
Its name is misleading as it suggests that it is getting a pointer, without side effects, and that pointer can be written to.
It also perpetuates the idea that all objects have a dictionary (at the C level), and that accessing that dictionary is cheap.

Just because something is widely used does not mean we should make it public. We should be helping extension authors to write good code, not perpetuating their mistakes.

Why are C extensions using this method anyway? If they are using it just to get the object's dictionary for legitimate reasons, then we should add a PyObject_GetDict function for that purpose.

Also, we should probably change the docs for PyObject_GenericGetDict to make it clear that it is for implementing the __dict__ slot, not to be called as an API.

[kumaraditya303]

This function is also broken on free-threading because free-threading stores and reads dict pointer atomically, it is not safe to read the dict pointer directly as returned by this function.

[vstinner]

Why are C extensions using this method anyway? If they are using it just to get the object's dictionary for legitimate reasons, then we should add a PyObject_GetDict function for that purpose.

For various reasons, see for example #139854 (comment). Should we add a PyObject_SetDict() function as well?

[encukou]

Should we add a PyObject_SetDict() function as well?

Not with the current evidence. I'd be interested to know if SWIG_PYTHON_SLOW_GETSET_THIS is helpful nowadays -- ideally PyObject_SetAttr would be all they need.

[da-woods]

One of Cython's uses of this I think is duplicating PyType_Lookup so could probably go away if you provided that.

I think in general PyObject_GetDict would be more useful than GetDictPtr.

At least some of the time I'm using the dict rather than GetAttr/SetAttr because I can do PyDict_SetDefault and I know it'll be thread safe. That's very useful for lazy initialization, and is something you can't easily replicate without the dict.

[vstinner]

One of Cython's uses of this I think is duplicating PyType_Lookup so could probably go away if you provided that.

I created #139847 to add PyType_Lookup() function.