Issue #20287: Argument Clinic's output is now configurable, allowing

delaying its output or even redirecting it to a separate file.

Doc/howto/clinic.rst

@@ -23,6 +23,58 @@ Argument Clinic How-To
   version of Argument Clinic that ships with CPython 3.5 *could*
   be totally incompatible and break all your code.
 
+============================
+The Goals Of Argument Clinic
+============================
+
+Argument Clinic's primary goal
+is to take over responsibility for all argument parsing code
+inside CPython.  This means that, when you convert a function
+to work with Argument Clinic, that function should no longer
+do any of its own argument parsing--the code generated by
+Argument Clinic should be a "black box" to you, where CPython
+calls in at the top, and your code gets called at the bottom,
+with ``PyObject *args`` (and maybe ``PyObject *kwargs``)
+magically converted into the C variables and types you need.
+
+In order for Argument Clinic to accomplish its primary goal,
+it must be easy to use.  Currently, working with CPython's
+argument parsing library is a chore, requiring maintaining
+redundant information in a surprising number of places.
+When you use Argument Clinic, you don't have to repeat yourself.
+
+Obviously, no one would want to use Argument Clinic unless
+it's solving a their problem without creating problems of
+its own.
+So Argument Clinic should generate correct code, and its
+code should preferably be slower, and definitely should
+not introduce a major speed regression.  (Eventually Argument
+Clinic should enable a major speedup--we should be able
+to rewrite its code generator so it produces tailor-made
+parsing code, rather than using the general-purpose functions
+from the CPython code base, which would make for the fastest
+argument parsing possible.)
+
+Additionally, Argument Clinic must be flexible enough to
+work with any approach to argument parsing.  Python has
+some functions with some very strange parsing behaviors;
+Argument Clinic's goal is to support all of them.
+
+Finally, the original motivation for Argument Clinic was
+to provide introspection "signatures" for CPython builtins.
+It used to be, the introspection query functions would throw
+an exception if you passed in a builtin.  With Argument
+Clinic, that's a thing of the past!
+
+One idea you should keep in mind, as you work with
+Argument Clinic: the more information you give it, the
+better job it'll be able to do.
+Argument Clinic is admittedly relatively simple right
+now.  But as it evolves it will get more sophisticated,
+and it should be able to do many interesting and smart
+things with all the information you give it.
+
+
 ========================
 Basic Concepts And Usage
 ========================
@@ -84,7 +136,15 @@ Converting Your First Function
 ==============================
 
 The best way to get a sense of how Argument Clinic works is to
-convert a function to work with it.  Let's dive in!
+convert a function to work with it.  Here, then, are the bare
+minimum steps you'd need to follow to convert a function to
+work with Argument Clinic.  Note that for code you plan to
+check in to CPython, you really should take the conversion farther,
+using some of the advanced concepts you'll see later on in
+the document (like "return converters" and "self converters").
+But we'll keep it simple for this walkthrough so you can learn.
+
+Let's dive in!
 
 0. Make sure you're working with a freshly updated checkout
    of the CPython trunk.
@@ -1282,6 +1342,264 @@ available, the macro turns into nothing.  Perfect!
 (This is the preferred approach for optional functions; in the future,
 Argument Clinic may generate the entire ``PyMethodDef`` structure.)
 
+
+Changing and redirecting Clinic's output
+----------------------------------------
+
+It can be inconvenient to have Clinic's output interspersed with
+your conventional hand-edited C code.  Luckily, Clinic is configurable:
+you can buffer up its output for printing later (or earlier!), or write
+its output to a separate file.  You can also add a prefix or suffix to
+every line of Clinic's generated output.
+
+While changing Clinic's output in this manner can be a boon to readability,
+it may result in Clinic code using types before they are defined, or
+your code attempting to use Clinic-generated code befire it is defined.
+These problems can be easily solved by rearranging the declarations in your file,
+or moving where Clinic's generated code goes.  (This is why the default behavior
+of Clinic is to output everything into the current block; while many people
+consider this hampers readability, it will never require rearranging your
+code to fix definition-before-use problems.)
+
+Let's start with defining some terminology:
+
+*field*
+  A field, in this context, is a subsection of Clinic's output.
+  For example, the ``#define`` for the ``PyMethodDef`` structure
+  is a field, called ``methoddef_define``.  Clinic has seven
+  different fields it can output per function definition::
+
+      docstring_prototype
+      docstring_definition
+      methoddef_define
+      impl_prototype
+      parser_prototype
+      parser_definition
+      impl_definition
+
+  All the names are of the form ``"<a>_<b>"``,
+  where ``"<a>"`` is the semantic object represented (the parsing function,
+  the impl function, the docstring, or the methoddef structure) and ``"<b>"``
+  represents what kind of statement the field is.  Field names that end in
+  ``"_prototype"``
+  represent forward declarations of that thing, without the actual body/data
+  of the thing; field names that end in ``"_definition"`` represent the actual
+  definition of the thing, with the body/data of the thing.  (``"methoddef"``
+  is special, it's the only one that ends with ``"_define"``, representing that
+  it's a preprocessor #define.)
+
+*destination*
+  A destination is a place Clinic can write output to.  There are
+  five built-in destinations:
+
+  ``block``
+    The default destination: printed in the output section of
+    the current Clinic block.
+
+  ``buffer``
+    A text buffer where you can save text for later.  Text sent
+    here is appended to the end of any exsiting text.  It's an
+    error to have any text left in the buffer when Clinic finishes
+    processing a file.
+
+  ``file``
+    A separate "clinic file" that will be created automatically by Clinic.
+    The filename chosen for the file is ``{basename}.clinic{extension}``,
+    where ``basename`` and ``extension`` were assigned the output
+    from ``os.path.splitext()`` run on the current file.  (Example:
+    the ``file`` destination for ``_pickle.c`` would be written to
+    ``_pickle.clinic.c``.)
+
+    **Important: When using a** ``file`` **destination, you**
+    *must check in* **the generated file!**
+
+  ``two-pass``
+    A buffer like ``buffer``.  However, a two-pass buffer can only
+    be written once, and it prints out all text sent to it during
+    all of processing, even from Clinic blocks *after* the
+
+  ``suppress``
+    The text is suppressed--thrown away.
+
+
+Clinic defines five new directives that let you reconfigure its output.
+
+The first new directive is ``dump``::
+
+   dump <destination>
+
+This dumps the current contents of the named destination into the output of
+the current block, and empties it.  This only works with ``buffer`` and
+``two-pass`` destinations.
+
+The second new directive is ``output``.  The most basic form of ``output``
+is like this::
+
+    output <field> <destination>
+
+This tells Clinic to output *field* to *destination*.  ``output`` also
+supports a special meta-destination, called ``everything``, which tells
+Clinic to output *all* fields to that *destination*.
+
+``output`` has a number of other functions::
+
+    output push
+    output pop
+    output preset <preset>
+
+
+``output push`` and ``output pop`` allow you to push and pop
+configurations on an internal configuration stack, so that you
+can temporarily modify the output configuration, then easily restore
+the previous configuration.  Simply push before your change to save
+the current configuration, then pop when you wish to restore the
+previous configuration.
+
+``output preset`` sets Clinic's output to one of several built-in
+preset configurations, as follows:
+
+  ``original``
+    Clinic's starting configuration.
+
+    Suppress the ``parser_prototype``
+    and ``docstring_prototype``, write everything else to ``block``.
+
+  ``file``
+    Designed to write everything to the "clinic file" that it can.
+    You then ``#include`` this file near the top of your file.
+    You may need to rearrange your file to make this work, though
+    usually this just means creating forward declarations for various
+    ``typedef`` and ``PyTypeObject`` definitions.
+
+    Suppress the ``parser_prototype``
+    and ``docstring_prototype``, write the ``impl_definition`` to
+    ``block``, and write everything else to ``file``.
+
+  ``buffer``
+    Save up all most of the output from Clinic, to be written into
+    your file near the end.  For Python files implementing modules
+    or builtin types, it's recommended that you dump the buffer
+    just above the static structures for your module or
+    builtin type; these are normally very near the end.  Using
+    ``buffer`` may require even more editing than ``file``, if
+    your file has static ``PyMethodDef`` arrays defined in the
+    middle of the file.
+
+    Suppress the ``parser_prototype``, ``impl_prototype``,
+    and ``docstring_prototype``, write the ``impl_definition`` to
+    ``block``, and write everything else to ``file``.
+
+  ``two-pass``
+    Similar to the ``buffer`` preset, but writes forward declarations to
+    the ``two-pass`` buffer, and definitions to the ``buffer``.
+    This is similar to the ``buffer`` preset, but may require
+    less editing than ``buffer``.  Dump the ``two-pass`` buffer
+    near the top of your file, and dump the ``buffer`` near
+    the end just like you would when using the ``buffer`` preset.
+
+    Suppresses the ``impl_prototype``, write the ``impl_definition``
+    to ``block``, write ``docstring_prototype``, ``methoddef_define``,
+    and ``parser_prototype`` to ``two-pass``, write everything else
+    to ``buffer``.
+
+  ``partial-buffer``
+    Similar to the ``buffer`` preset, but writes more things to ``block``,
+    only writing the really big chunks of generated code to ``buffer``.
+    This avoids the definition-before-use problem of ``buffer`` completely,
+    at the small cost of having slightly more stuff in the block's output.
+    Dump the ``buffer`` near the end, just like you would when using
+    the ``buffer`` preset.
+
+    Suppresses the ``impl_prototype``, write the ``docstring_definition``
+    and ``parser_defintion`` to ``buffer``, write everything else to ``block``.
+
+The third new directive is ``destination``::
+
+    destination <name> <command> [...]
+
+This performs an operation on the destination named ``name``.
+
+There are two defined subcommands: ``new`` and ``clear``.
+
+The ``new`` subcommand works like this::
+
+    destination <name> new <type>
+
+This creates a new destination with name ``<name>`` and type ``<type>``.
+
+There are five destination types::
+
+    ``suppress``
+        Throws the text away.
+
+    ``block``
+        Writes the text to the current block.  This is what Clinic
+        originally did.
+
+    ``buffer``
+        A simple text buffer, like the "buffer" builtin destination above.
+
+    ``file``
+        A text file.  The file destination takes an extra argument,
+        a template to use for building the filename, like so:
+
+            destination <name> new <type> <file_template>
+
+        The template can use three strings internally that will be replaced
+        by bits of the filename:
+
+            {filename}
+                The full filename.
+            {basename}
+                Everything up to but not including the last '.'.
+            {extension}
+                The last '.' and everything after it.
+
+        If there are no periods in the filename, {basename} and {filename}
+        are the same, and {extension} is empty.  "{basename}{extension}"
+        is always exactly the same as "{filename}"."
+
+    ``two-pass``
+        A two-pass buffer, like the "two-pass" builtin destination above.
+
+
+The ``clear`` subcommand works like this::
+
+    destination <name> clear
+
+It removes all the accumulated text up to this point in the destination.
+(I don't know what you'd need this for, but I thought maybe it'd be
+useful while someone's experimenting.)
+
+The fourth new directive is ``set``::
+
+    set line_prefix "string"
+    set line_suffix "string"
+
+``set`` lets you set two internal variables in Clinic.
+``line_prefix`` is a string that will be prepended to every line of Clinic's output;
+``line_suffix`` is a string that will be appended to every line of Clinic's output.
+
+Both of these suport two format strings:
+
+  ``{block comment start}``
+    Turns into the string ``/*``, the start-comment text sequence for C files.
+
+  ``{block comment end}``
+    Turns into the string ``*/``, the end-comment text sequence for C files.
+
+The final new directive is one you shouldn't need to use directly,
+called ``preserve``::
+
+    preserve
+
+This tells Clinic that the current contents of the output should be kept, unmodifed.
+This is used internally by Clinic when dumping output into ``file`` files; wrapping
+it in a Clinic block lets Clinic use its existing checksum functionality to ensure
+the file was not modified by hand before it gets overwritten.
+
+
+Using Argument Clinic in Python files
 -------------------------------------
 
 It's actually possible to use Argument Clinic to preprocess Python files.

Misc/NEWS

@@ -88,6 +88,9 @@ Tests
 Tools/Demos
 -----------
 
+- Issue #20287: Argument Clinic's output is now configurable, allowing
+  delaying its output or even redirecting it to a separate file.
+
 - Issue #20226: Argument Clinic now permits simple expressions
   (e.g. "sys.maxsize - 1") as default values for parameters.