[Documentation] Clarify Reference Counting Semantics for Internal and Public API Functions

[wr-web]

Description

We have identified several functions where the reference counting semantics are not clearly documented.

Internal functions (PyAPI_FUNC) with undocumented semantics:

• Return new reference:
  • _PyXI_ApplyCapturedException
  • _PyXI_ApplyError
  • _Py_GetExecutor
  • _PyTime_AsLong
• Return borrowed reference:
  • _PyDict_GetItemStringWithError
  • _PySys_GetAttr

Public API functions lacking clear semantic tags:

• Return new reference:
  • PyAsyncGen_New
  • PyComplex_FromCComplex

Could these semantics be officially documented (e.g., in refcount.dat or the source code) to aid in analysis?

[skirpichev]

PyComplex_FromCComplex

What's wrong here? It's in Doc/data/refcounts.dat and stable docs shows:

Return value: New reference.

[wr-web]

What's wrong here? It's in Doc/data/refcounts.dat and stable docs shows:

You're right. I was wrong about PyComplex_FromCComplex; it is correctly documented as returning a new reference. Beside, the count changes in refcounts.dat (e.g., for PyCodec_ReplaceErrors) indicate the net refcount change, not necessarily a new reference.

[ZeroIntensity]

refcounts.dat generally isn't supposed to document anything in the private API, except for a few special cases.

[vstinner]

_PyDict_GetItemStringWithError() and _PySys_GetAttr(): you should use public functions instead, such as _PyDict_GetItemStringRef() and PySys_GetAttr().

_PyTime_AsLong() can be replaced with PyLong_FromInt64().

Could these semantics be officially documented (e.g., in refcount.dat or the source code) to aid in analysis?

Would you mind to elaborate why you use functions of internal C API? What do you mean by analysis?

[vstinner]

_PyTime_AsLong() can be replaced with PyLong_FromInt64()

I created #141053 to do exactly that in the Python code base :-)

[wr-web]

refcounts.dat generally isn't supposed to document anything in the private API, except for a few special cases.

I think it would also be helpful for developers if there were documentation for private APIs, such as recording whether there are reference count changes or documenting reference counting semantics.

[wr-web]

Would you mind to elaborate why you use functions of internal C API? What do you mean by analysis?

There is a paper that describes detecting reference counting defects through trace counting However, this static analysis approach (symbolic execution) relies on modeling functions, including both internal and external API functions. The modeling method in the paper depends on extracting reference counting semantics from documentation. Therefore, I believe having comprehensive documentation covering reference counting semantics for both internal and external API functions could be beneficial for such work.

[ZeroIntensity]

Unfortunately, our internal C API is massive. I wouldn't be surprised if it had over a thousand functions. Realistically, it would probably take us months or even years to get refcounts.dat to a point where it extensively documents every internal function in CPython. Even if we magically had the time and people to do it, we'd also need a new format, because refcounts.dat doesn't have any way to express things like immortality or reference stealing.

I don't mean to come across as dismissive or obtuse (and I really do appreciate your work on developing a reference count linter), but if your tool can't automatically deduce a function's reference count effect from other calls, I really don't think there's much we can do to help. Python is developed by volunteers, and we simply don't have the resources to take on a project like this. To put this into perspective: we currently struggle to document everything in our public, stable ABI.

If you continue to work on this, we're happy to answer questions about reference counting semantics on our help forum. Thanks!