[ENH] Pythonise the documentation!

[scoder]

The current documentation stems mostly from a time when Pure Python Mode wasn't even being thought of and aims to present the special cdef syntax that Cython uses. The Pure Python Syntax is only presented on the pure Python tutorial page.

Today, more and more new users come from a Python background that uses type annotations. We should accommodate them by presenting both syntax variants hand in hand throughout the documentation.

If you want to help out, please do the following:

1. Grab a documentation page and duplicate the code examples into separate files, inlcuding the Pure Python variant before the cdef variant with literalinclude. See
   This currently excludes the pages that refer to the integration of external C or C++ code, since some of the required language features are not available in Pure Python code/mode (and this should be mentioned in those places).

For example, a file add1.pyx:

cdef int add1(int x) except -1:
    return x + 1

should be accompanied by a file add1.py:

@cython.exceptval(-1)
def add1(x: cython.int):
    return x + 1

2. Read the surrounding text to see if there is anything that refers to the specific syntax constructs. Make sure everything is well explained to the readers so that they can easily understand the two examples.

3. Put a paragraph near the top of the page to show a reference to import cython and briefly explain the difference between the two syntax variants:

.. include::
    ../two-syntax-variants-used

This includes the following text:

This page uses two different syntax variants: the Cython specific ``cdef`` syntax
and static Cython type declarations in :ref:`pure Python code <pep484_type_annotations>`,
following `PEP-484 <https://www.python.org/dev/peps/pep-0484/>`_ type hints
and `PEP 526 <https://www.python.org/dev/peps/pep-0526/>`_ variable annotations.
To make good use of the latter, including C data types etc., you need
the special ``cython`` module, which you can import with

.. code-block: python

      import cython

in the Python module that you want to compile.

Don't forget to link back to this issue in any PRs that you submit.

EDIT: Decisions/approaches taken further down:

• use different file extensions for the two variants: some_example.py and some_example.pyx
• present both in tabs to make it easy to switch between them
• present the Python syntax variant tab before (left of) the Cython syntax, as the first will eventually become preferable
• keep both source files aligned line by line to highlight the differences on tab switches
• run the .py variant also through Python for syntax/sanity checking (find a way to deal with pure Python cimports)
• automatically add import cython to the .py test files in runtests.py to avoid having to repeat it in all examples that need it
• explicitly write import cython in test examples that are long enough (and need it), especially if they use future imports or directive comments
• see [ENH] Pythonise the documentation! #4187 (comment) for further conventions

List of files to consider:

• Getting Started (Add pure python to Getting started #4283)
  • Building Cython code
  • Faster code via static typing
• Tutorials
  • Basic Tutorial (Pythonise the documentation according to #4187: Basic Tutorial (cython_tutorial.rst) #4226)
  • Calling C functions (Added Pure Python mode to Calling C functions tutorial #4247)
  • Using C libraries (Docs: Introduce pure python mode in Using C libraries tutorial #4294)
  • Extension types (aka. cdef classes) (Pythonise the docs #4187: "Extension types (aka. cdef classes)" (cdef_classes.rst) #4232)
  • pxd files (Pythonise docs: "pxd files" (pxd_files.rst) #4295)
  • Profiling (Docs: Migrate profiling tutorial to pure python #4311)
  • Unicode and passing strings (Pythonise Unicode and passing strings (strings.rst) #4389)
  • Memory Allocation (Pythonise Memory Allocation (memory_allocation.rst) #4316)
  • Embedding Cython modules in C/C++ applications
  • Pure Python Mode (Doc: Modernize pure mode tutorial (pure.rst) with dual syntax tabs #7286)
  • Working with NumPy
  • Working with Python arrays (Migrate array tutorial to pure python mode #4508) (Pythonise "Working with Python arrays" (array.rst) #4431)
• Users Guide
  • Language Basics (Introduce pure python mode in language basics documentation #4242)
  • Extension Types (Migrate extension_types.rst to pure python #4516)
  • Special Methods of Extension Types (Docs: Migrate special_methods.rst to pure python #4537)
  • Sharing Declarations Between Cython Modules (Docs: Migrate sharing_declarations.rst to pure python mode #4544)
  • Interfacing with External C Code
  • Source Files and Compilation
  • Early Binding for Speed
  • Using C++ in Cython
  • Fused Types (Templates)
  • Migrating from Cython 0.29 to 3.0
  • Differences between Cython and Pyrex
  • Typed Memoryviews ([Docs] Migrate memoryviews userguide to Pure python #5658)
  • Implementing the buffer protocol
  • Using Parallelism [Doc] Add pure python mode to parallelism.rst #4974
  • Cython for NumPy users ([Docs] Migrate Cython for NumPy users section to pure python #5913)

[da-woods]

And (where possible) use annotations over @cython.declare - right?

[scoder]

And (where possible) use annotations over @cython.declare - right?

Absolutely. The goal is to use the least intrusive and most readable way to get the same effect as with the original syntax.

[TeamSpen210]

Would it be worth it to maybe create a modified/custom directive for this, so that you could toggle between the two code samples - like a tabbed code block or a toggle at the top of the page?

[da-woods]

Tabbed code block seems like a very good idea to me - it lets us keep the examples short and to the point while still letting us show both syntaxes.

Obviously someone has to implement it...

[mattip]

There is a sphinx-tabs extension that supports code tabs extension that support syntax highlighting and seems to even synchronize the tabs on the page

[goldenrockefeller]

In some situations, a supplementary pxd file can be used to provide type information, with little-to-no modifications on the original py file. Given the pros and cons associated with this approach, how much should the documentation reflect this?

[scoder]

.pxd files use Cython syntax. They are a subset of the language, basically. I think it's enough to describe this possibility in one place (that's currently the pure page) and otherwise leave them out when describing the language. It doesn't add anything there.

[Bluenix2]

Put the (proposed) following paragraph near the top of the page to show a reference to import cython and briefly explain the difference between the two syntax variants:

I think that the paragraph above that you proposed should point to the document about Pure Python Mode.

[0dminnimda]

I think PR-s can cover one page or several. It is hardly worth expecting PR with the transformation of entire documentation.