Rationale

doc/index.rst

@@ -40,7 +40,6 @@ Pages
    runtimes
    old_c_api
    type_object
-   remove_functions
    optimization_ideas
    backward_compatibility
    os_vendors
@@ -76,7 +75,6 @@ Table of Contents
    runtimes
    old_c_api
    type_object
-   remove_functions
    optimization_ideas
    backward_compatibility
    os_vendors

doc/old_c_api.rst

@@ -4,11 +4,14 @@
 Old C API
 +++++++++
 
-The "Old C API" is the Python 3.7 API which "leaks" implementation details like
-``PyObject.ob_refcnt`` through :ref:`Py_INCREF() <incref>`. This API will
-remain available for CPython internals but also for specific use cases like
-:ref:`Cython <cython>` (for best performances) and :ref:`debugging tools
-<debug-tools>`.
+The "current" or "old" C API is the Python 3.7 API which "leaks" implementation
+details like ``PyObject.ob_refcnt`` through :ref:`Py_INCREF() <incref>` macro.
+
+With the new C API, the old C API will remain available thanks to the
+:ref:`regular runtime <regular-runtime>`, for CPython internals, for specific
+use cases like :ref:`Cython <cython>` (for best performances) and
+:ref:`debugging tools <debug-tools>`, but also for the long tail of C
+extensions on PyPI.
 
 See also :ref:`Calling conventensions <calling-conventions>`.
 

doc/rationale.rst

@@ -2,35 +2,47 @@
 Rationale to change the Python C API
 ++++++++++++++++++++++++++++++++++++
 
-Problem
-=======
+To be able to introduce backward incompatible changes to the C API without
+breaking too many C extensions, this project proposes two things:
 
-The current Python C API produces an ABI which is far from stable. The API
-leaks many implementation details into the ABI.
+* design a :ref:`helper layer <back-compat>` providing :ref:`removed functions
+  <remove-funcs>`;
+* a new :ref:`Python runtime <runtimes>` which is only usable with C extensions
+  compiled with the new stricter and smaller C API (and the new :ref:`stable
+  ABI <stable-abi>`) for Python 3.8 and newer, whereas the existing "regular
+  python" becomes the "regular runtime" which provides maximum backward
+  compatibility with Python 3.7 and older.
 
-For example, using the ``PyTuple_GET_ITEM()`` macro produces machine code which
-uses an **hardcoded** offset. If the PyObject structure changes, the offset
-changes, and so the **ABI** changes.
+The current C API has multiple issues:
 
-The Big Carrot
-==============
+* The Python lifecycle is shorter than the :ref:`lifecycle of some operating
+  systems <os-vendors>`: how to get the latest Python on an "old" but stable
+  operating system?
+* :ref:`Python debug build <debug-build>` is currently mostly unusable in
+  practice, making development of C extension harder, especially debugging.
 
-Changing the C API means that authors of C extensions have to do something. To
-justify these changes, we need a big carrot. Examples:
+PyPy cpyext is slow
+===================
 
-* faster Python if you pick new API? faster PyPy :ref:`cpyext <cpyext>`?
-* less bugs? no more surprising borrow references causing "funny" crashes
-* new features?
+The :ref:`PyPy cpyext <cpyext>` is slow because there is no efficient way to
+write a correct implementation of the :ref:`current C API <old-c-api>`.
+For example, :ref:`borrowed references <borrowed-ref>` is hard to optimize
+since the runtime cannot track the lifetime of a borrowed object.
 
-Performance problem: How can we make Python 2x faster?
-======================================================
 
-"Leaking" implementation details into the ABI prevents many optimizations
-opportunities.
+The stable ABI is not usable
+============================
 
+The current Python C API produces an ABI which is far from :ref:`stable
+<stable-abi>`. The API leaks many implementation details into the ABI.
 
-Issues with an unstable ABI
-===========================
+For example, using the ``PyTuple_GET_ITEM()`` macro produces machine code which
+uses an **hardcoded** offset. If the PyObject structure changes, the offset
+changes, and so the **ABI** changes.
+
+See :ref:`Relationship between the C API and the ABI <from-api-to-api>`.
+
+Issues with an unstable ABI:
 
 * **Packaging.** Developers have to publish one binary package per Python
   version (3.6, 3.7, 3.8, etc.). Upgrading Python 3 to a newer version
@@ -47,3 +59,32 @@ Issues with an unstable ABI
   debug builds would ease debugging since the debug builds add many debug
   checks which are too expensive (CPU/memory) to be enabled by default in a
   release build.
+
+
+Performance problem: How can we make Python 2x faster?
+======================================================
+
+"Leaking" implementation details into the ABI prevents many optimizations
+opportunities. See :ref:`Optimization ideas <optim-ideas>`.
+
+
+Keep backward compatibility
+===========================
+
+Existing C extensions will still be supported and will not have to be modified.
+The :ref:`old C API <old-c-api>` is not deprecated and there is no plan
+penalize users of the old C API.
+
+See :ref:`Backward compatibility <back-compat>`.
+
+
+The Big Carrot
+==============
+
+Changing the C API means that authors of C extensions have to do something. To
+justify these changes, we need a big carrot. Examples:
+
+* faster Python if you pick new API? faster PyPy :ref:`cpyext <cpyext>`?
+* less bugs? Bugs caused by borrow references are hard to debug.
+* new features?
+

doc/stable_abi.rst

@@ -4,6 +4,8 @@
 Python stable ABI?
 ++++++++++++++++++
 
+.. _from-api-to-api:
+
 Relationship between the C API and the ABI
 ==========================================