Put hyperlink to API reference into readme and improve autogen of API mapping table.

.gitignore

@@ -1,6 +1,5 @@
 # HPy autogen
 hpy/tools/autogen/autogen_pypy.txt
-hpy/tools/autogen/api_mapping.txt
 
 # generated by setup.py:get_scm_config()
 hpy/devel/include/hpy/version.h

README.md

@@ -38,6 +38,9 @@ for more details on HPy motivation, goals, and features, for example:
 Do you want to see how HPy API looks in code? Check out
 our [quickstart example](https://docs.hpyproject.org/en/latest/quickstart.html).
 
+You may also be interested in HPy's
+[API reference](https://docs.hpyproject.org/en/latest/api-reference/index.html).
+
 This repository contains the API and ABI specification and implementation
 for the CPython interpreter. Other interpreters that support HPy natively: GraalPy
 and PyPy, provide their own builtin HPy implementations.

docs/api-reference/index.rst

@@ -3,14 +3,10 @@ API Reference
 
 HPy's public API consists of three parts:
 
-1. The **Core API** as defined in file ``public_api.h``
-2. **HPy Helper** functions 
+1. The **Core API** as defined in the :doc:`public-api`
+2. **HPy Helper** functions
 3. **Inline Helper** functions
 
-.. warning:: Generated API reference documentation is work in progress. Some
-    parts of the API are not included in this documentation yet.
-
-
 Core API
 --------
 
@@ -35,6 +31,7 @@ between the modes.
    hpy-err
    builder
    hpy-eval
+   public-api
 
 
 HPy Helper Functions

docs/api-reference/public-api.rst

@@ -0,0 +1,9 @@
+Public API Header
+=================
+
+The core API is defined in `public_api.h
+<https://github.com/hpyproject/hpy/blob/master/hpy/tools/autogen/public_api.h>`_:
+
+.. literalinclude:: ../../hpy/tools/autogen/public_api.h
+    :language: c
+    :linenos:

docs/api.rst

@@ -1,10 +1,6 @@
-HPy API introduction
+HPy API Introduction
 ====================
 
-.. warning::
-   HPy is still in the early stages of development and the API may change.
-
-
 Handles
 -------
 

docs/conf.py

@@ -17,11 +17,12 @@
 import sys
 import os
 import re
+import datetime
 
 # -- Project information -----------------------------------------------------
 
 project = "HPy"
-copyright = "2019-2020, HPy Collective"
+copyright = "2019-{}, HPy Collective".format(datetime.date.today().year)
 author = "HPy Collective"
 
 # The full version, including alpha/beta/rc tags

docs/misc/index.rst

@@ -1,4 +1,4 @@
-Misc notes
+Misc Notes
 ==========
 
 .. toctree::

docs/overview.rst

@@ -1,5 +1,5 @@
-HPy overview
-=============
+HPy Overview
+============
 
 Motivation and goals
 ---------------------
@@ -285,11 +285,12 @@ You have two choices:
 Current status and roadmap
 --------------------------
 
-HPy is still in the early stages of development, but many big pieces are
-already in place. As on April 2022, the following milestones have been reached:
+HPy left the early stages of development and already provides a noticeable set
+of features. As on April 2023, the following milestones have been reached:
 
-  - some real-world Python packages have been ported to HPy API.
-    The ports will be published soon.
+  - some prominent real-world Python packages have been ported to HPy API. There
+    is a list of HPy-compatible packages we know about on the HPy website
+    `hpyproject.org <https://hpyproject.org/>`_.
 
   - one can write extensions which expose module-level functions, with all
     the various kinds of calling conventions.
@@ -310,7 +311,10 @@ already in place. As on April 2022, the following milestones have been reached:
     recompiling. It can detect leaked handles or handles used after
     being closed.
 
-  - wheels can be build for HPy extensions with ``python setup.py bdist_wheel``
+  - trace mode has been implemented and can be activated just like the debug
+    mode. It helps analyzing the API usage (in particular wrt. performance).
+
+  - wheels can be built for HPy extensions with ``python setup.py bdist_wheel``
     and can be installed with ``pip install``.
 
   - it is possible to choose between the :term:`CPython ABI` and the
@@ -328,6 +332,10 @@ already in place. As on April 2022, the following milestones have been reached:
   - it is possible to load HPy Universal extensions on `GraalPy
     <https://github.com/graalvm/graalpython>`_.
 
+  - there is support for multi-phase module initialization.
+
+  - support for metaclasses has been added.
+
 
 However, there is still a long road before HPy is usable for the general
 public. In particular, the following features are on our roadmap but have not

docs/porting-guide.rst

@@ -149,15 +149,15 @@ mapping between C API and HPy API functions. This mapping is generated together
 with the code for the :term:`CPython ABI` mode, so it is guaranteed to be correct.
 
 
-..  _table-mapping:
-
+.. mark: BEGIN API MAPPING
+.. _table-mapping:
 .. table:: Safe API function mapping
     :widths: auto
 
     ================================================================================================================================== ================================================
     C API function                                                                                                                     HPY API function
     ================================================================================================================================== ================================================
-    `PyBool_FromLong <https://docs.python.org/3/c-api/bool.html#c.PyBool_FromLong>`_                                                   :c:func:`HPyBool_FromBool`
+    `PyBool_FromLong <https://docs.python.org/3/c-api/bool.html#c.PyBool_FromLong>`_                                                   :c:func:`HPyBool_FromLong`
     `PyBytes_AS_STRING <https://docs.python.org/3/c-api/bytes.html#c.PyBytes_AS_STRING>`_                                              :c:func:`HPyBytes_AS_STRING`
     `PyBytes_AsString <https://docs.python.org/3/c-api/bytes.html#c.PyBytes_AsString>`_                                                :c:func:`HPyBytes_AsString`
     `PyBytes_Check <https://docs.python.org/3/c-api/bytes.html#c.PyBytes_Check>`_                                                      :c:func:`HPyBytes_Check`
@@ -194,15 +194,21 @@ with the code for the :term:`CPython ABI` mode, so it is guaranteed to be correc
     `PyList_Check <https://docs.python.org/3/c-api/list.html#c.PyList_Check>`_                                                         :c:func:`HPyList_Check`
     `PyList_New <https://docs.python.org/3/c-api/list.html#c.PyList_New>`_                                                             :c:func:`HPyList_New`
     `PyLong_AsDouble <https://docs.python.org/3/c-api/long.html#c.PyLong_AsDouble>`_                                                   :c:func:`HPyLong_AsDouble`
+    `PyLong_AsLong <https://docs.python.org/3/c-api/long.html#c.PyLong_AsLong>`_                                                       :c:func:`HPyLong_AsLong`
+    `PyLong_AsLongLong <https://docs.python.org/3/c-api/long.html#c.PyLong_AsLongLong>`_                                               :c:func:`HPyLong_AsLongLong`
     `PyLong_AsSize_t <https://docs.python.org/3/c-api/long.html#c.PyLong_AsSize_t>`_                                                   :c:func:`HPyLong_AsSize_t`
     `PyLong_AsSsize_t <https://docs.python.org/3/c-api/long.html#c.PyLong_AsSsize_t>`_                                                 :c:func:`HPyLong_AsSsize_t`
+    `PyLong_AsUnsignedLong <https://docs.python.org/3/c-api/long.html#c.PyLong_AsUnsignedLong>`_                                       :c:func:`HPyLong_AsUnsignedLong`
+    `PyLong_AsUnsignedLongLong <https://docs.python.org/3/c-api/long.html#c.PyLong_AsUnsignedLongLong>`_                               :c:func:`HPyLong_AsUnsignedLongLong`
+    `PyLong_AsUnsignedLongLongMask <https://docs.python.org/3/c-api/long.html#c.PyLong_AsUnsignedLongLongMask>`_                       :c:func:`HPyLong_AsUnsignedLongLongMask`
+    `PyLong_AsUnsignedLongMask <https://docs.python.org/3/c-api/long.html#c.PyLong_AsUnsignedLongMask>`_                               :c:func:`HPyLong_AsUnsignedLongMask`
     `PyLong_AsVoidPtr <https://docs.python.org/3/c-api/long.html#c.PyLong_AsVoidPtr>`_                                                 :c:func:`HPyLong_AsVoidPtr`
-    `PyLong_FromLong <https://docs.python.org/3/c-api/long.html#c.PyLong_FromLong>`_                                                   :c:func:`HPyLong_FromSize_t`
-    `PyLong_FromLongLong <https://docs.python.org/3/c-api/long.html#c.PyLong_FromLongLong>`_                                           :c:func:`HPyLong_FromSize_t`
+    `PyLong_FromLong <https://docs.python.org/3/c-api/long.html#c.PyLong_FromLong>`_                                                   :c:func:`HPyLong_FromLong`
+    `PyLong_FromLongLong <https://docs.python.org/3/c-api/long.html#c.PyLong_FromLongLong>`_                                           :c:func:`HPyLong_FromLongLong`
     `PyLong_FromSize_t <https://docs.python.org/3/c-api/long.html#c.PyLong_FromSize_t>`_                                               :c:func:`HPyLong_FromSize_t`
     `PyLong_FromSsize_t <https://docs.python.org/3/c-api/long.html#c.PyLong_FromSsize_t>`_                                             :c:func:`HPyLong_FromSsize_t`
-    `PyLong_FromUnsignedLong <https://docs.python.org/3/c-api/long.html#c.PyLong_FromUnsignedLong>`_                                   :c:func:`HPyLong_FromSize_t`
-    `PyLong_FromUnsignedLongLong <https://docs.python.org/3/c-api/long.html#c.PyLong_FromUnsignedLongLong>`_                           :c:func:`HPyLong_FromSize_t`
+    `PyLong_FromUnsignedLong <https://docs.python.org/3/c-api/long.html#c.PyLong_FromUnsignedLong>`_                                   :c:func:`HPyLong_FromUnsignedLong`
+    `PyLong_FromUnsignedLongLong <https://docs.python.org/3/c-api/long.html#c.PyLong_FromUnsignedLongLong>`_                           :c:func:`HPyLong_FromUnsignedLongLong`
     `PyNumber_Absolute <https://docs.python.org/3/c-api/number.html#c.PyNumber_Absolute>`_                                             :c:func:`HPy_Absolute`
     `PyNumber_Add <https://docs.python.org/3/c-api/number.html#c.PyNumber_Add>`_                                                       :c:func:`HPy_Add`
     `PyNumber_And <https://docs.python.org/3/c-api/number.html#c.PyNumber_And>`_                                                       :c:func:`HPy_And`
@@ -281,7 +287,9 @@ with the code for the :term:`CPython ABI` mode, so it is guaranteed to be correc
     `PyUnicode_FromWideChar <https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_FromWideChar>`_                                  :c:func:`HPyUnicode_FromWideChar`
     `PyUnicode_ReadChar <https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_ReadChar>`_                                          :c:func:`HPyUnicode_ReadChar`
     `PyUnicode_Substring <https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Substring>`_                                        :c:func:`HPyUnicode_Substring`
+    `Py_FatalError <https://docs.python.org/3/c-api/sys.html#c.Py_FatalError>`_                                                        :c:func:`HPy_FatalError`
     ================================================================================================================================== ================================================
+.. mark: END API MAPPING
 
 
 .. note: There are, of course, also cases where it is not possible to map directly and safely from a C API function (or concept) to an HPy API function (or concept).

hpy/tools/autogen/conf.py

@@ -146,3 +146,58 @@
     'HPyType_IsSubtype': None,
     'HPy_SetCallFunction': None,
 }
+
+################################################################################
+#                    Configuration for auto-generating docs                    #
+################################################################################
+
+# A manual mapping of between CPython C API functions and HPy API functions.
+# Most of the mapping will be generated automatically from 'public_api.h' if an
+# HPy API function is not a special case (see 'conf.py'). However, in some
+# cases, it might be that we have inline helper functions or something similar
+# that map to a CPython C API function which cannot be determined automatically.
+# In those cases, the mapping can be manually specified here. Also, manual
+# mapping will always take precedence over automatically derived mappings.
+DOC_MANUAL_API_MAPPING = {
+    # key = C API function name
+    # value = HPy API function name
+    'Py_FatalError': 'HPy_FatalError',
+    'PyContextVar_Get': 'HPyContextVar_Get',
+    'PyLong_FromLong': 'HPyLong_FromLong',
+    'PyLong_FromLongLong': 'HPyLong_FromLongLong',
+    'PyLong_FromUnsignedLong': 'HPyLong_FromUnsignedLong',
+    'PyLong_FromUnsignedLongLong': 'HPyLong_FromUnsignedLongLong',
+    'PyLong_AsLong': 'HPyLong_AsLong',
+    'PyLong_AsLongLong': 'HPyLong_AsLongLong',
+    'PyLong_AsUnsignedLong': 'HPyLong_AsUnsignedLong',
+    'PyLong_AsUnsignedLongMask': 'HPyLong_AsUnsignedLongMask',
+    'PyLong_AsUnsignedLongLong': 'HPyLong_AsUnsignedLongLong',
+    'PyLong_AsUnsignedLongLongMask': 'HPyLong_AsUnsignedLongLongMask',
+    'PyBool_FromLong': 'HPyBool_FromLong',
+    'PyObject_TypeCheck': 'HPy_TypeCheck',
+    'PySlice_AdjustIndices': 'HPySlice_AdjustIndices',
+    'PyType_IsSubtype': 'HPyType_IsSubtype',
+    'PyObject_Call': 'HPy_CallTupleDict',
+    'PyObject_Vectorcall': 'HPy_Call',
+    'PyObject_VectorcallMethod': 'HPy_CallMethod',
+}
+
+# Some C API functions are documented in very different pages.
+DOC_C_API_PAGES_SPECIAL_CASES = {
+    'Py_FatalError': 'sys',
+    'PyEval_SaveThread': 'init',
+    'PyEval_RestoreThread': 'init',
+    'PyEval_EvalCode': 'veryhigh',
+    'PyObject_Call': 'call',
+    'PyObject_Vectorcall': 'call',
+    'PyObject_VectorcallMethod': 'call',
+}
+
+# We assume that, e.g., prefix 'PyLong_Something' belongs to 'longobject.c' and
+# its documentation is in '.../3/c-api/long.html'. In some cases, the prefix
+# maps to a different page and this can be specified here. E.g.
+# 'PyErr_Something' is documented in page '.../3/c-api/exceptions.html'
+DOC_PREFIX_TABLE = {
+    'err': 'exceptions',
+    'contextvar': 'contextvars'
+}

hpy/tools/autogen/doc.py

@@ -4,6 +4,10 @@
 from .autogenfile import AutoGenFile
 from .parse import toC
 from .ctx import autogen_ctx_h
+from .conf import (DOC_C_API_PAGES_SPECIAL_CASES,
+                   DOC_MANUAL_API_MAPPING,
+                   DOC_PREFIX_TABLE)
+
 
 CTX_NAME = '_HPyContext_s'
 
@@ -26,7 +30,6 @@ def __init__(self, api):
             self.DISCLAIMER = RST_DISCLAIMER
         self.api = api
 
-
 class autogen_function_index(AutoGenRstFile):
     PATH = 'docs/api-reference/function-index.rst'
     LANGUAGE = 'rst'
@@ -46,53 +49,75 @@ def generate(self):
         return '\n'.join(lines)
 
 
+class AutoGenFilePart:
+    PATH = None
+    BEGIN_MARKER = None
+    END_MARKER = None
+
+    def __init__(self, api):
+        self.api = api
+
+    def generate(self, old):
+        raise NotImplementedError
+
+    def write(self, root):
+        if not self.BEGIN_MARKER or not self.END_MARKER:
+            raise RuntimeError("missing BEGIN_MARKER or END_MARKER")
+        n_begin = len(self.BEGIN_MARKER)
+        with root.join(self.PATH).open('r') as f:
+            content = f.read()
+        start = content.find(self.BEGIN_MARKER)
+        if start < 0:
+            raise RuntimeError(f'begin marker "{self.BEGIN_MARKER}" not found'
+                               f'in file {self.PATH}')
+        end = content.find(self.END_MARKER, start + n_begin)
+        if end < 0:
+            raise RuntimeError(f'end marker "{self.END_MARKER}" not found in'
+                               f'file {self.PATH}')
+        new_content = self.generate(content[(start+n_begin):end])
+        with root.join(self.PATH).open('w') as f:
+            f.write(content[:start + n_begin] + new_content + content[end:])
+
+
 GROUP_PATTERN = re.compile('Py(\\w+)_.*')
 
-# Some C API functions are documented in very different pages.
-SPECIAL_CASES = {
-    'PyEval_SaveThread': 'init',
-    'PyEval_RestoreThread': 'init',
-    'PyEval_EvalCode': 'veryhigh',
-}
-
-# We assume that, e.g., prefix 'PyLong_Something' belongs to 'longobject.c' and
-# its documentation is in '.../3/c-api/long.html'. In some cases, the prefix
-# maps to a different page and this can be specified here. E.g.
-# 'PyErr_Something' is documented in page '.../3/c-api/exceptions.html'
-PREFIX_TABLE = {
-    'err': 'exceptions',
-    'contextvar': 'contextvars'
-}
-
-class autogen_doc_api_mapping(AutoGenFile):
-    PATH = 'hpy/tools/autogen/api_mapping.txt'
-    LANGUAGE = 'txt'
+class autogen_doc_api_mapping(AutoGenFilePart):
+    PATH = 'docs/porting-guide.rst'
+    BEGIN_MARKER = '.. mark: BEGIN API MAPPING\n'
+    END_MARKER = '.. mark: END API MAPPING\n'
 
     def _get_page(self, cpython_fun_name):
-        if cpython_fun_name in SPECIAL_CASES:
-            return SPECIAL_CASES[cpython_fun_name] + '.html'
+        if cpython_fun_name in DOC_C_API_PAGES_SPECIAL_CASES:
+            return DOC_C_API_PAGES_SPECIAL_CASES[cpython_fun_name] + '.html'
 
         first_underscore = cpython_fun_name.find('_')
         if cpython_fun_name.startswith('Py') and first_underscore != -1:
             prefix = cpython_fun_name[2:first_underscore].lower()
-            return PREFIX_TABLE.get(prefix, prefix) + '.html'
+            return DOC_PREFIX_TABLE.get(prefix, prefix) + '.html'
         else:
             return 'abstract.html'
 
-    def generate(self):
+    def generate(self, old_content):
+        table_directive = '.. _table-mapping:\n.. table:: Safe API function mapping\n'
+        assert old_content.strip().startswith(table_directive)
+
         lines = []
         w = lines.append
+        w(':widths: auto')
+        w('')
+        mapping = {x.cpython_name: x.name for x in self.api.functions if x.cpython_name}
+        mapping.update(DOC_MANUAL_API_MAPPING)
         max_width0 = 0
         max_width1 = 0
         rows = []
-        functions = [x for x in self.api.functions if x.cpython_name]
-        # sort the list of functions by 'func.cpython_name'
-        functions.sort(key=lambda x: x.cpython_name)
-        for func in functions:
-            assert func.cpython_name
-            page = self._get_page(func.cpython_name)
-            col0 = f'`{func.cpython_name} <https://docs.python.org/3/c-api/{page}#c.{func.cpython_name}>`_'
-            col1 = f':c:func:`{func.name}`'
+        cpy_functions = list(mapping.keys())
+        # sort the list of functions by 'cpython_name'
+        cpy_functions.sort()
+        for cpy_func in cpy_functions:
+            assert cpy_func
+            page = self._get_page(cpy_func)
+            col0 = f'`{cpy_func} <https://docs.python.org/3/c-api/{page}#c.{cpy_func}>`_'
+            col1 = f':c:func:`{mapping[cpy_func]}`'
             rows.append((col0, col1))
             max_width0 = max(max_width0, len(col0))
             max_width1 = max(max_width1, len(col1))
@@ -104,7 +129,8 @@ def generate(self):
         for row in rows:
             w(f'{row[0].ljust(max_width0)} {row[1]}')
         w(sep)
-        return textwrap.indent('\n'.join(lines), ' ' * 4)
+        w('')
+        return table_directive + textwrap.indent('\n'.join(lines), ' ' * 4)
 
 
 class autogen_hpy_ctx(AutoGenRstFile):