Skip to content

Commit

Permalink
[3.12] pythongh-98154: Clarify Usage of "Reference Count" In the Docs (
Browse files Browse the repository at this point in the history
…pythongh-107552) (python#107752)

* pythongh-98154: Clarify Usage of "Reference Count" In the Docs (pythongh-107552)

PEP 683 (immortal objects) revealed some ways in which the Python documentation has been unnecessarily coupled to the implementation details of reference counts.  In the end users should focus on reference ownership, including taking references and releasing them, rather than on how many reference counts an object has.

This change updates the documentation to reflect that perspective.  It also updates the docs relative to immortal objects in a handful of places.
(cherry picked from commit 5dc825d)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>

* Fix a typo.

---------

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
  • Loading branch information
miss-islington and ericsnowcurrently committed Aug 8, 2023
1 parent e5582bd commit aa2ecef
Show file tree
Hide file tree
Showing 15 changed files with 162 additions and 90 deletions.
13 changes: 7 additions & 6 deletions Doc/c-api/allocation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,13 @@ Allocating Objects on the Heap
.. c:macro:: PyObject_New(TYPE, typeobj)
Allocate a new Python object using the C structure type *TYPE* and the
Python type object *typeobj* (``PyTypeObject*``).
Fields not defined by the Python object header
are not initialized; the object's reference count will be one. The size of
the memory allocation is determined from the :c:member:`~PyTypeObject.tp_basicsize` field of
the type object.
Allocate a new Python object using the C structure type *TYPE*
and the Python type object *typeobj* (``PyTypeObject*``).
Fields not defined by the Python object header are not initialized.
The caller will own the only reference to the object
(i.e. its reference count will be one).
The size of the memory allocation is determined from the
:c:member:`~PyTypeObject.tp_basicsize` field of the type object.
.. c:macro:: PyObject_NewVar(TYPE, typeobj, size)
Expand Down
17 changes: 11 additions & 6 deletions Doc/c-api/arg.rst
Original file line number Diff line number Diff line change
Expand Up @@ -293,8 +293,10 @@ Other objects

``O`` (object) [PyObject \*]
Store a Python object (without any conversion) in a C object pointer. The C
program thus receives the actual object that was passed. The object's reference
count is not increased. The pointer stored is not ``NULL``.
program thus receives the actual object that was passed. A new
:term:`strong reference` to the object is not created
(i.e. its reference count is not increased).
The pointer stored is not ``NULL``.

``O!`` (object) [*typeobject*, PyObject \*]
Store a Python object in a C object pointer. This is similar to ``O``, but
Expand Down Expand Up @@ -378,7 +380,8 @@ inside nested parentheses. They are:
mutually exclude each other.

Note that any Python object references which are provided to the caller are
*borrowed* references; do not decrement their reference count!
*borrowed* references; do not release them
(i.e. do not decrement their reference count)!

Additional arguments passed to these functions must be addresses of variables
whose type is determined by the format string; these are used to store values
Expand Down Expand Up @@ -613,8 +616,10 @@ Building values
Convert a C :c:type:`Py_complex` structure to a Python complex number.
``O`` (object) [PyObject \*]
Pass a Python object untouched (except for its reference count, which is
incremented by one). If the object passed in is a ``NULL`` pointer, it is assumed
Pass a Python object untouched but create a new
:term:`strong reference` to it
(i.e. its reference count is incremented by one).
If the object passed in is a ``NULL`` pointer, it is assumed
that this was caused because the call producing the argument found an error and
set an exception. Therefore, :c:func:`Py_BuildValue` will return ``NULL`` but won't
raise an exception. If no exception has been raised yet, :exc:`SystemError` is
Expand All @@ -624,7 +629,7 @@ Building values
Same as ``O``.
``N`` (object) [PyObject \*]
Same as ``O``, except it doesn't increment the reference count on the object.
Same as ``O``, except it doesn't create a new :term:`strong reference`.
Useful when the object is created by a call to an object constructor in the
argument list.
Expand Down
7 changes: 5 additions & 2 deletions Doc/c-api/buffer.rst
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,9 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
.. c:member:: PyObject *obj
A new reference to the exporting object. The reference is owned by
the consumer and automatically decremented and set to ``NULL`` by
the consumer and automatically released
(i.e. reference count decremented)
and set to ``NULL`` by
:c:func:`PyBuffer_Release`. The field is the equivalent of the return
value of any standard C-API function.

Expand Down Expand Up @@ -454,7 +456,8 @@ Buffer-related functions
.. c:function:: void PyBuffer_Release(Py_buffer *view)
Release the buffer *view* and decrement the reference count for
Release the buffer *view* and release the :term:`strong reference`
(i.e. decrement the reference count) to the view's supporting object,
``view->obj``. This function MUST be called when the buffer
is no longer being used, otherwise reference leaks may occur.
Expand Down
4 changes: 2 additions & 2 deletions Doc/c-api/bytes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -184,8 +184,8 @@ called with a non-bytes parameter.
.. c:function:: void PyBytes_ConcatAndDel(PyObject **bytes, PyObject *newpart)
Create a new bytes object in *\*bytes* containing the contents of *newpart*
appended to *bytes*. This version decrements the reference count of
*newpart*.
appended to *bytes*. This version releases the :term:`strong reference`
to *newpart* (i.e. decrements its reference count).
.. c:function:: int _PyBytes_Resize(PyObject **bytes, Py_ssize_t newsize)
Expand Down
3 changes: 2 additions & 1 deletion Doc/c-api/exceptions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,8 @@ For convenience, some of these functions will always return a
This is the most common way to set the error indicator. The first argument
specifies the exception type; it is normally one of the standard exceptions,
e.g. :c:data:`PyExc_RuntimeError`. You need not increment its reference count.
e.g. :c:data:`PyExc_RuntimeError`. You need not create a new
:term:`strong reference` to it (e.g. with :c:func:`Py_INCREF`).
The second argument is an error message; it is decoded from ``'utf-8'``.
Expand Down
2 changes: 1 addition & 1 deletion Doc/c-api/init_config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1109,7 +1109,7 @@ PyConfig
.. c:member:: int show_ref_count
Show total reference count at exit?
Show total reference count at exit (excluding immortal objects)?
Set to ``1`` by :option:`-X showrefcount <-X>` command line option.
Expand Down
59 changes: 33 additions & 26 deletions Doc/c-api/intro.rst
Original file line number Diff line number Diff line change
Expand Up @@ -287,60 +287,67 @@ true if (and only if) the object pointed to by *a* is a Python list.
Reference Counts
----------------

The reference count is important because today's computers have a finite (and
often severely limited) memory size; it counts how many different places there
are that have a reference to an object. Such a place could be another object,
or a global (or static) C variable, or a local variable in some C function.
When an object's reference count becomes zero, the object is deallocated. If
it contains references to other objects, their reference count is decremented.
Those other objects may be deallocated in turn, if this decrement makes their
reference count become zero, and so on. (There's an obvious problem with
objects that reference each other here; for now, the solution is "don't do
that.")
The reference count is important because today's computers have a finite
(and often severely limited) memory size; it counts how many different
places there are that have a :term:`strong reference` to an object.
Such a place could be another object, or a global (or static) C variable,
or a local variable in some C function.
When the last :term:`strong reference` to an object is released
(i.e. its reference count becomes zero), the object is deallocated.
If it contains references to other objects, those references are released.
Those other objects may be deallocated in turn, if there are no more
references to them, and so on. (There's an obvious problem with
objects that reference each other here; for now, the solution
is "don't do that.")

.. index::
single: Py_INCREF()
single: Py_DECREF()

Reference counts are always manipulated explicitly. The normal way is to use
the macro :c:func:`Py_INCREF` to increment an object's reference count by one,
and :c:func:`Py_DECREF` to decrement it by one. The :c:func:`Py_DECREF` macro
Reference counts are always manipulated explicitly. The normal way is
to use the macro :c:func:`Py_INCREF` to take a new reference to an
object (i.e. increment its reference count by one),
and :c:func:`Py_DECREF` to release that reference (i.e. decrement the
reference count by one). The :c:func:`Py_DECREF` macro
is considerably more complex than the incref one, since it must check whether
the reference count becomes zero and then cause the object's deallocator to be
called. The deallocator is a function pointer contained in the object's type
structure. The type-specific deallocator takes care of decrementing the
reference counts for other objects contained in the object if this is a compound
called. The deallocator is a function pointer contained in the object's type
structure. The type-specific deallocator takes care of releasing references
for other objects contained in the object if this is a compound
object type, such as a list, as well as performing any additional finalization
that's needed. There's no chance that the reference count can overflow; at
least as many bits are used to hold the reference count as there are distinct
memory locations in virtual memory (assuming ``sizeof(Py_ssize_t) >= sizeof(void*)``).
Thus, the reference count increment is a simple operation.

It is not necessary to increment an object's reference count for every local
variable that contains a pointer to an object. In theory, the object's
It is not necessary to hold a :term:`strong reference` (i.e. increment
the reference count) for every local variable that contains a pointer
to an object. In theory, the object's
reference count goes up by one when the variable is made to point to it and it
goes down by one when the variable goes out of scope. However, these two
cancel each other out, so at the end the reference count hasn't changed. The
only real reason to use the reference count is to prevent the object from being
deallocated as long as our variable is pointing to it. If we know that there
is at least one other reference to the object that lives at least as long as
our variable, there is no need to increment the reference count temporarily.
our variable, there is no need to take a new :term:`strong reference`
(i.e. increment the reference count) temporarily.
An important situation where this arises is in objects that are passed as
arguments to C functions in an extension module that are called from Python;
the call mechanism guarantees to hold a reference to every argument for the
duration of the call.

However, a common pitfall is to extract an object from a list and hold on to it
for a while without incrementing its reference count. Some other operation might
conceivably remove the object from the list, decrementing its reference count
for a while without taking a new reference. Some other operation might
conceivably remove the object from the list, releasing that reference,
and possibly deallocating it. The real danger is that innocent-looking
operations may invoke arbitrary Python code which could do this; there is a code
path which allows control to flow back to the user from a :c:func:`Py_DECREF`, so
almost any operation is potentially dangerous.

A safe approach is to always use the generic operations (functions whose name
begins with ``PyObject_``, ``PyNumber_``, ``PySequence_`` or ``PyMapping_``).
These operations always increment the reference count of the object they return.
These operations always create a new :term:`strong reference`
(i.e. increment the reference count) of the object they return.
This leaves the caller with the responsibility to call :c:func:`Py_DECREF` when
they are done with the result; this soon becomes second nature.

Expand All @@ -356,7 +363,7 @@ to objects (objects are not owned: they are always shared). "Owning a
reference" means being responsible for calling Py_DECREF on it when the
reference is no longer needed. Ownership can also be transferred, meaning that
the code that receives ownership of the reference then becomes responsible for
eventually decref'ing it by calling :c:func:`Py_DECREF` or :c:func:`Py_XDECREF`
eventually releasing it by calling :c:func:`Py_DECREF` or :c:func:`Py_XDECREF`
when it's no longer needed---or passing on this responsibility (usually to its
caller). When a function passes ownership of a reference on to its caller, the
caller is said to receive a *new* reference. When no ownership is transferred,
Expand Down Expand Up @@ -414,9 +421,9 @@ For example, the above two blocks of code could be replaced by the following

It is much more common to use :c:func:`PyObject_SetItem` and friends with items
whose references you are only borrowing, like arguments that were passed in to
the function you are writing. In that case, their behaviour regarding reference
counts is much saner, since you don't have to increment a reference count so you
can give a reference away ("have it be stolen"). For example, this function
the function you are writing. In that case, their behaviour regarding references
is much saner, since you don't have to take a new reference just so you
can give that reference away ("have it be stolen"). For example, this function
sets all items of a list (actually, any mutable sequence) to a given item::

int
Expand Down
2 changes: 1 addition & 1 deletion Doc/c-api/module.rst
Original file line number Diff line number Diff line change
Expand Up @@ -529,7 +529,7 @@ state:
.. note::
Unlike other functions that steal references, ``PyModule_AddObject()``
only decrements the reference count of *value* **on success**.
only releases the reference to *value* **on success**.
This means that its return value must be checked, and calling code must
:c:func:`Py_DECREF` *value* manually on error.
Expand Down
13 changes: 7 additions & 6 deletions Doc/c-api/object.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ Object Protocol
.. c:macro:: Py_RETURN_NOTIMPLEMENTED
Properly handle returning :c:data:`Py_NotImplemented` from within a C
function (that is, increment the reference count of NotImplemented and
return it).
function (that is, create a new :term:`strong reference`
to NotImplemented and return it).


.. c:function:: int PyObject_Print(PyObject *o, FILE *fp, int flags)
Expand Down Expand Up @@ -320,11 +320,12 @@ Object Protocol
When *o* is non-``NULL``, returns a type object corresponding to the object type
of object *o*. On failure, raises :exc:`SystemError` and returns ``NULL``. This
is equivalent to the Python expression ``type(o)``. This function increments the
reference count of the return value. There's really no reason to use this
is equivalent to the Python expression ``type(o)``.
This function creates a new :term:`strong reference` to the return value.
There's really no reason to use this
function instead of the :c:func:`Py_TYPE()` function, which returns a
pointer of type :c:expr:`PyTypeObject*`, except when the incremented reference
count is needed.
pointer of type :c:expr:`PyTypeObject*`, except when a new
:term:`strong reference` is needed.
.. c:function:: int PyObject_TypeCheck(PyObject *o, PyTypeObject *type)
Expand Down

0 comments on commit aa2ecef

Please sign in to comment.