From 0a3100148c6b6b2006230ef89e30dc474f138f02 Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Sat, 4 Jan 2014 12:44:57 -0800 Subject: [PATCH 01/71] Issue #19659: Added documentation for Argument Clinic. --- Doc/howto/clinic.rst | 900 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 900 insertions(+) create mode 100644 Doc/howto/clinic.rst diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst new file mode 100644 index 000000000..afbeb417a --- /dev/null +++ b/Doc/howto/clinic.rst @@ -0,0 +1,900 @@ +====================== +Argument Clinic How-To +====================== + +:author: Larry Hastings + + +.. topic:: Abstract + + Argument Clinic is a preprocessor for CPython C files. + Its purpose is to automate all the boilerplate involved + with writing argument parsing code for "builtins". + This document shows you how to convert your first C + function to work with Argument Clinic, and then introduces + some advanced topics on Argument Clinic usage. + + Argument Clinic is currently considered an internal + tool for the CPython code tree. Its use is not supported + for files outside the CPython code tree, and no guarantees + are made regarding backwards compatibility for future + versions. In other words: if you maintain an external C + extension for CPython, you're welcome to experiment with + Argument Clinic in your own code. But the version of Argument + Clinic that ships with CPython 3.5 *could* be totally + incompatible and break all your code. + +======================== +Basic Concepts And Usage +======================== + +Argument Clinic ships with CPython. You can find it in ``Tools/clinic/clinic.py``. +If you run that script, specifying a C file as an argument:: + + % python3 Tools/clinic/clinic.py foo.c + +Argument Clinic will scan over the file looking for lines that +look exactly like this:: + + /*[clinic] + +When it finds one, it reads everything up to a line that looks +like this:: + + [clinic]*/ + +Everything in between these two lines is input for Argument Clinic. +All of these lines, including the beginning and ending comment +lines, are collectively called an Argument Clinic "input block", +or "block" for short. + +When Argument Clinic parses one of these blocks, it +generates output. This output is rewritten into the C file +immediately after the block, followed by a comment containing a checksum. +The resulting Argument Clinic block looks like this:: + + /*[clinic] + ... clinic input goes here ... + [clinic]*/ + ... clinic output goes here ... + /*[clinic checksum:...]*/ + +If you run Argument Clinic on the same file a second time, Argument Clinic +will discard the old output and write out the new output with a fresh checksum +line. However, if the input hasn't changed, the output won't change either. + +You should never modify the output portion of an Argument Clinic block. Instead, +change the input until it produces the output you want. (That's the purpose of the +checksum--to detect and warn you in case someone accidentally modifies the output.) + +For the sake of clarity, here's the terminology we'll use with Argument Clinic: + +* The first line of the comment (``/*[clinic]``) is the *start line*. +* The last line of the initial comment (``[clinic]*/``) is the *end line*. +* The last line (``/*[clinic checksum:...]*/``) is the *checksum line*. +* In between the start line and the end line is the *input*. +* In between the end line and the checksum line is the *output*. +* All the text collectively, from the start line to the checksum line inclusively, + is the *block*. (A block that hasn't been successfully processed by Argument + Clinic yet doesn't have output or a checksum line, but it's still considered + a block.) + + +============================== +Converting Your First Function +============================== + +The best way to get a sense of how Argument Clinic works is to +convert a function to work with it. Let's dive in! + +0. Make sure you're working with a freshly updated trunk. + +1. Find a Python builtin that calls either ``PyArg_ParseTuple()`` + or ``PyArg_ParseTupleAndKeywords()``, and hasn't been converted yet. + For my example I'm using ``pickle.Pickler.dump()``. + +2. If the call to the ``PyArg_Parse`` function uses any of the + following format units:: + + O& + O! + es + es# + et + et# + + or if it has multiple calls to ``PyArg_ParseTuple()``, + you should choose a different function. Argument Clinic *does* + support all of these scenarios. But these are advanced + topics--let's do something simpler for your first function. + +3. Add the following boilerplate above the function, creating our block:: + + /*[clinic] + [clinic]*/ + +4. Cut the docstring and paste it in between the ``[clinic]`` lines, + removing all the junk that makes it a properly quoted C string. + When you're done you should have just the text, based at the left + margin, with no line wider than 80 characters. + (Argument Clinic will preserve indents inside the docstring.) + + Sample:: + + /*[clinic] + Write a pickled representation of obj to the open file. + [clinic]*/ + +5. If your docstring doesn't have a "summary" line, Argument Clinic will + complain. So let's make sure it has one. The "summary" line should + be a paragraph consisting of a single 80-column line + at the beginning of the docstring. + + (Our docstring consists solely of the summary line, so the sample + code doesn't have to change for this step.) + +6. Above the docstring, enter the name of the function, followed + by a blank line. This should be the Python name of the function, + and should be the full dotted path + to the function--it should start with the name of the module, + include any sub-modules, and if the function is a method on + a class it should include the class name too. + + Sample:: + + /*[clinic] + pickle.Pickler.dump + + Write a pickled representation of obj to the open file. + [clinic]*/ + +7. If this is the first time that module or class has been used with Argument + Clinic in this C file, + you must declare the module and/or class. Proper Argument Clinic hygiene + prefers declaring these in a separate block somewhere near the + top of the C file, in the same way that include files and statics go at + the top. (In our sample code we'll just show the two blocks next to + each other.) + + Sample:: + + /*[clinic] + module pickle + class pickle.Pickler + [clinic]*/ + + /*[clinic] + pickle.Pickler.dump + + Write a pickled representation of obj to the open file. + [clinic]*/ + + +8. Declare each of the parameters to the function. Each parameter + should get its own line. All the parameter lines should be + indented from the function name and the docstring. + + The general form of these parameter lines is as follows:: + + name_of_parameter: converter + + If the parameter has a default value, add that after the + converter:: + + name_of_parameter: converter = default_value + + Add a blank line below the parameters. + + What's a "converter"? It establishes both the type + of the variable used in C, and the method to convert the Python + value into a C value at runtime. + For now you're going to use what's called a "legacy converter"--a + convenience syntax intended to make porting old code into Argument + Clinic easier. + + For each parameter, copy the "format unit" for that + parameter from the ``PyArg_Parse()`` format argument and + specify *that* as its converter, as a quoted + string. ("format unit" is the formal name for the one-to-three + character substring of the ``format`` parameter that tells + the argument parsing function what the type of the variable + is and how to convert it.) + + For multicharacter format units like ``z#``, use the + entire two-or-three character string. + + Sample:: + + /*[clinic] + module pickle + class pickle.Pickler + [clinic]*/ + + /*[clinic] + pickle.Pickler.dump + + obj: 'O' + + Write a pickled representation of obj to the open file. + [clinic]*/ + +9. If your function has ``|`` in the format string, meaning some + parameters have default values, you can ignore it. Argument + Clinic infers which parameters are optional based on whether + or not they have default values. + + If your function has ``$`` in the format string, meaning it + takes keyword-only arguments, specify ``*`` on a line by + itself before the first keyword-only argument, indented the + same as the parameter lines. + + (``pickle.Pickler.dump`` has neither, so our sample is unchanged.) + + +10. If the existing C function uses ``PyArg_ParseTuple()`` + (instead of ``PyArg_ParseTupleAndKeywords()``), then all its + arguments are positional-only. + + To mark all parameters as positional-only in Argument Clinic, + add a ``/`` on a line by itself after the last parameter, + indented the same as the parameter lines. + + Sample:: + + /*[clinic] + module pickle + class pickle.Pickler + [clinic]*/ + + /*[clinic] + pickle.Pickler.dump + + obj: 'O' + / + + Write a pickled representation of obj to the open file. + [clinic]*/ + +11. It's helpful to write a per-parameter docstring, indented + another level past the parameter declaration. But per-parameter + docstrings are optional; you can skip this step if you prefer. + + Here's how per-parameter docstrings work. The first line + of the per-parameter docstring must be indented further than the + parameter definition. This left margin establishes the left margin + for the whole per-parameter docstring; all the text you write will + be outdented by this amount. You can write as much as you like, + across multiple lines if you wish. + + Sample:: + + /*[clinic] + module pickle + class pickle.Pickler + [clinic]*/ + + /*[clinic] + pickle.Pickler.dump + + obj: 'O' + The object to be pickled. + / + + Write a pickled representation of obj to the open file. + [clinic]*/ + +12. Save and close the file, then run ``Tools/clinic/clinic.py`` on it. + With luck everything worked and your block now has output! Reopen + the file in your text editor to see:: + + /*[clinic] + module pickle + class pickle.Pickler + [clinic]*/ + /*[clinic checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ + + /*[clinic] + pickle.Pickler.dump + + obj: 'O' + The object to be pickled. + / + + Write a pickled representation of obj to the open file. + [clinic]*/ + + PyDoc_STRVAR(pickle_Pickler_dump__doc__, + "Write a pickled representation of obj to the open file.\n" + "\n" + ... + static PyObject * + pickle_Pickler_dump_impl(PyObject *self, PyObject *obj) + /*[clinic checksum: 3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ + +13. Double-check that the argument-parsing code Argument Clinic generated + looks basically the same as the existing code. + + First, ensure both places use the same argument-parsing function. + The existing code must call either + ``PyArg_ParseTuple()`` or ``PyArg_ParseTupleAndKeywords()``; + ensure that the code generated by Argument Clinic calls the + same function. + + Second, the format string passed in to ``PyArg_ParseTuple()`` or + ``PyArg_ParseTupleAndKeywords()`` should be *exactly* the same + as the hand-written one in the existing function. + + Well, there's one way that Argument Clinic's output is permitted + to be different. Argument Clinic always generates a format string + ending with ``:`` followed by the name of the function. If the + format string originally ended with ``;`` (to specify usage help), + this is harmless--don't worry about this difference. + + Apart from that, if either of these things differ in *any way*, + fix your input to Argument Clinic and rerun ``Tools/clinic/clinic.py`` + until they are the same. + + +14. Notice that the last line of its output is the declaration + of your "impl" function. This is where the builtin's implementation goes. + Delete the existing prototype of the function you're modifying, but leave + the opening curly brace. Now delete its argument parsing code and the + declarations of all the variables it dumps the arguments into. + Notice how the Python arguments are now arguments to this impl function; + if the implementation used different names for these variables, fix it. + The result should be a function that handles just the implementation + of the Python function without any argument-parsing code. + + Sample:: + + /*[clinic] + module pickle + class pickle.Pickler + [clinic]*/ + /*[clinic checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ + + /*[clinic] + pickle.Pickler.dump + + obj: 'O' + The object to be pickled. + / + + Write a pickled representation of obj to the open file. + [clinic]*/ + + PyDoc_STRVAR(pickle_Pickler_dump__doc__, + "Write a pickled representation of obj to the open file.\n" + "\n" + ... + static PyObject * + pickle_Pickler_dump_impl(PyObject *self, PyObject *obj) + /*[clinic checksum: 3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ + { + /* Check whether the Pickler was initialized correctly (issue3664). + Developers often forget to call __init__() in their subclasses, which + would trigger a segfault without this check. */ + if (self->write == NULL) { + PyErr_Format(PicklingError, + "Pickler.__init__() was not called by %s.__init__()", + Py_TYPE(self)->tp_name); + return NULL; + } + + if (_Pickler_ClearBuffer(self) < 0) + return NULL; + + ... + +15. Compile and run the relevant portions of the regression-test suite. + This change should not introduce any new compile-time warnings or errors, + and there should be no externally-visible change to Python's behavior. + + Well, except for one difference: ``inspect.signature()`` run on your function + should now provide a valid signature! + + Congratulations, you've ported your first function to work with Argument Clinic! + +=============== +Advanced Topics +=============== + + +Renaming the C functions generated by Argument Clinic +----------------------------------------------------- + +Argument Clinic automatically names the functions it generates for you. +Occasionally this may cause a problem, if the generated name collides with +the name of an existing C function. There's an easy solution: you can explicitly +specify the base name to use for the C functions. Just add the keyword ``"as"`` +to your function declaration line, followed by the function name you wish to use. +Argument Clinic will use the function name you use for the base (generated) function, +and then add ``"_impl"`` to the end for the name of the impl function. + +For example, if we wanted to rename the C function names generated for +``pickle.Pickler.dump``, it'd look like this:: + + /*[clinic] + pickle.Pickler.dump as pickler_dumper + + ... + +The base function would now be named ``pickler_dumper()``, +and the impl function would be named ``pickler_dumper_impl()``. + + +Optional Groups +--------------- + +Some legacy functions have a tricky approach to parsing their arguments: +they count the number of positional arguments, then use a ``switch`` statement +to call one of several different ``PyArg_ParseTuple()`` calls depending on +how many positional arguments there are. (These functions cannot accept +keyword-only arguments.) This approach was used to simulate optional +arguments back before ``PyArg_ParseTupleAndKeywords()`` was created. + +Functions using this approach can often be converted to +use ``PyArg_ParseTupleAndKeywords()``, optional arguments, and default values. +But it's not always possible, because some of these legacy functions have +behaviors ``PyArg_ParseTupleAndKeywords()`` can't directly support. +The most obvious example is the builtin function ``range()``, which has +an optional argument on the *left* side of its required argument! +Another example is ``curses.window.addch()``, which has a group of two +arguments that must always be specified together. (The arguments are +called ``x`` and ``y``; if you call the function passing in ``x``, +you must also pass in ``y``--and if you don't pass in ``x`` you may not +pass in ``y`` either.) + +For the sake of backwards compatibility, Argument Clinic supports this +alternate approach to parsing, using what are called *optional groups*. +Optional groups are groups of arguments that can only be specified together. +They can be to the left or the right of the required arguments. They +can *only* be used with positional-only parameters. + +To specify an optional group, add a ``[`` on a line by itself before +the parameters you wish to be +in a group together, and a ``]`` on a line by itself after the +parameters. As an example, here's how ``curses.window.addch`` +uses optional groups to make the first two parameters and the last +parameter optional:: + + /*[clinic] + + curses.window.addch + + [ + x: int + X-coordinate. + y: int + Y-coordinate. + ] + + ch: object + Character to add. + + [ + attr: long + Attributes for the character. + ] + / + + ... + + +Notes: + +* For every optional group, one additional parameter will be passed into the + impl function representing the group. The parameter will be an int, and it will + be named ``group_{direction}_{number}``, + where ``{direction}`` is either ``right`` or ``left`` depending on whether the group + is before or after the required parameters, and ``{number}`` is a monotonically + increasing number (starting at 1) indicating how far away the group is from + the required parameters. When the impl is called, this parameter will be set + to zero if this group was unused, and set to non-zero if this group was used. + (By used or unused, I mean whether or not the parameters received arguments + in this invocation.) + +* If there are no required arguments, the optional groups will behave + as if they are to the right of the required arguments. + +* In the case of ambiguity, the argument parsing code + favors parameters on the left (before the required parameters). + +* Optional groups are *only* intended for legacy code. Please do not + use optional groups for new code. + + +Using real Argument Clinic converters, instead of "legacy converters" +--------------------------------------------------------------------- + +To save time, and to minimize how much you need to learn +to achieve your first port to Argument Clinic, the walkthrough above tells +you to use the "legacy converters". "Legacy converters" are a convenience, +designed explicitly to make porting existing code to Argument Clinic +easier. And to be clear, their use is entirely acceptable when porting +code for Python 3.4. + +However, in the long term we probably want all our blocks to +use Argument Clinic's real syntax for converters. Why? A couple +reasons: + +* The proper converters are far easier to read and clearer in their intent. +* There are some format units that are unsupported as "legacy converters", + because they require arguments, and the legacy converter syntax doesn't + support specifying arguments. +* In the future we may have a new argument parsing library that isn't + restricted to what ``PyArg_ParseTuple()`` supports. + +So if you want +to go that extra effort, you should consider using normal +converters instead of the legacy converters. + +In a nutshell, the syntax for Argument Clinic (non-legacy) converters +looks like a Python function call. However, if there are no explicit +arguments to the function (all functions take their default values), +you may omit the parentheses. Thus ``bool`` and ``bool()`` are exactly +the same. All parameters to Argument Clinic converters are keyword-only. + +All Argument Clinic converters accept the following arguments: + +``doc_default`` + If the parameter takes a default value, normally this value is also + provided in the ``inspect.Signature`` metadata, and displayed in the + docstring. ``doc_default`` lets you override the value used in these + two places: pass in a string representing the Python value you wish + to use in these two contexts. + +``required`` + If a parameter takes a default value, Argument Clinic infers that the + parameter is optional. However, you may want a parameter to take a + default value in C, but not behave in Python as if the parameter is + optional. Passing in ``required=True`` to a converter tells Argument + Clinic that this parameter is not optional, even if it has a default + value. + +``annotation`` + The annotation value for this parameter. Not currently supported, + because PEP 8 mandates that the Python library may not use + annotations. + +Below is a table showing the mapping of legacy converters into real +Argument Clinic converters. On the left is the legacy converter, +on the right is the text you'd replace it with. + +========= ================================================================================= +``'B'`` ``byte(bitwise=True)`` +``'b'`` ``byte`` +``'c'`` ``char`` +``'C'`` ``int(types='str')`` +``'d'`` ``double`` +``'D'`` ``Py_complex`` +``'es#'`` ``str(encoding='name_of_encoding', length=True, zeroes=True)`` +``'es'`` ``str(encoding='name_of_encoding')`` +``'et#'`` ``str(encoding='name_of_encoding', types='bytes bytearray str', length=True)`` +``'et'`` ``str(encoding='name_of_encoding', types='bytes bytearray str')`` +``'f'`` ``float`` +``'h'`` ``short`` +``'H'`` ``unsigned_short`` +``'i'`` ``int`` +``'I'`` ``unsigned_int`` +``'K'`` ``unsigned_PY_LONG_LONG`` +``'L'`` ``PY_LONG_LONG`` +``'n'`` ``Py_ssize_t`` +``'O!'`` ``object(type='name_of_Python_type')`` +``'O&'`` ``object(converter='name_of_c_function')`` +``'O'`` ``object`` +``'p'`` ``bool`` +``'s#'`` ``str(length=True)`` +``'S'`` ``PyBytesObject`` +``'s'`` ``str`` +``'s*'`` ``Py_buffer(types='str bytes bytearray buffer')`` +``'u#'`` ``Py_UNICODE(length=True)`` +``'u'`` ``Py_UNICODE`` +``'U'`` ``unicode`` +``'w*'`` ``Py_buffer(types='bytearray rwbuffer')`` +``'y#'`` ``str(type='bytes', length=True)`` +``'Y'`` ``PyByteArrayObject`` +``'y'`` ``str(type='bytes')`` +``'y*'`` ``Py_buffer`` +``'Z#'`` ``Py_UNICODE(nullable=True, length=True)`` +``'z#'`` ``str(nullable=True, length=True)`` +``'Z'`` ``Py_UNICODE(nullable=True)`` +``'z'`` ``str(nullable=True)`` +``'z*'`` ``Py_buffer(types='str bytes bytearray buffer', nullable=True)`` +========= ================================================================================= + +As an example, here's our sample ``pickle.Pickler.dump`` using the proper +converter:: + + /*[clinic] + pickle.Pickler.dump + + obj: object + The object to be pickled. + / + + Write a pickled representation of obj to the open file. + [clinic]*/ + +Argument Clinic will show you all the converters it has +available. For each converter it'll show you all the parameters +it accepts, along with the default value for each parameter. +Just run ``Tools/clinic/clinic.py --converters`` to see the full list. + + +Advanced converters +------------------- + +Remeber those format units you skipped for your first +time because they were advanced? Here's how to handle those too. + +The trick is, all those format units take arguments--either +conversion functions, or types, or strings specifying an encoding. +(But "legacy converters" don't support arguments. That's why we +skipped them for your first function.) The argument you specified +to the format unit is now an argument to the converter; this +argument is either ``converter`` (for ``O&``), ``type`` (for ``O!``), +or ``encoding`` (for all the format units that start with ``e``). + +Note that ``object()`` must explicitly support each Python type you specify +for the ``type`` argument. Currently it only supports ``str``. It should be +easy to add more, just edit ``Tools/clinic/clinic.py``, search for ``O!`` in +the text, and add more entries to the dict mapping types to strings just above it. + +Note also that this approach takes away some possible flexibility for the format +units starting with ``e``. It used to be possible to decide at runtime what +encoding string to pass in to ``PyArg_ParseTuple()``. But now this string must +be hard-coded at compile-time. This limitation is deliberate; it made supporting +this format unit much easier, and may allow for future compile-time optimizations. +This restriction does not seem unreasonable; CPython itself always passes in static +hard-coded strings when using format units starting with ``e``. + + +Using a return converter +------------------------ + +By default the impl function Argument Clinic generates for you returns ``PyObject *``. +But your C function often computes some C type, then converts it into the ``PyObject *`` +at the last moment. Argument Clinic handles converting your inputs from Python types +into native C types--why not have it convert your return value from a native C type +into a Python type too? + +That's what a "return converter" does. It changes your impl function to return +some C type, then adds code to the generated (non-impl) function to handle converting +that value into the appropriate ``PyObject *``. + +The syntax for return converters is similar to that of parameter converters. +You specify the return converter like it was a return annotation on the +function itself. Return converters behave much the same as parameter converters; +they take arguments, the arguments are all keyword-only, and if you're not changing +any of the default arguments you can omit the parentheses. + +(If you use both ``"as"`` *and* a return converter for your function, +the ``"as"`` should come before the return converter.) + +There's one additional complication when using return converters: how do you +indicate an error has occured? Normally, a function returns a valid (non-``NULL``) +pointer for success, and ``NULL`` for failure. But if you use an integer return converter, +all integers are valid. How can Argument Clinic detect an error? Its solution: each return +converter implicitly looks for a special value that indicates an error. If you return +that value, and an error has been set (``PyErr_Occurred()`` returns a true +value), then the generated code will propogate the error. Otherwise it will +encode the value you return like normal. + +Currently Argument Clinic supports only a few return converters:: + + int + long + Py_ssize_t + DecodeFSDefault + +None of these take parameters. For the first three, return -1 to indicate +error. For ``DecodeFSDefault``, the return type is ``char *``; return a NULL +pointer to indicate an error. + +Calling Python code +------------------- + +The rest of the advanced topics require you to write Python code +which lives inside your C file and modifies Argument Clinic at +runtime. This is simple; you simply define a Python block. + +A Python block uses different delimiter lines than an Argument +Clinic function block. It looks like this:: + + /*[python] + # python code goes here + [python]*/ + +All the code inside the Python block is executed at the +time it's parsed. All text written to stdout inside the block +is redirected into the "output" after the block. + +As an example, here's a Python block that adds a static integer +variable to the C code:: + + /*[python] + print('static int __ignored_unused_variable__ = 0;') + [python]*/ + static int __ignored_unused_variable__ = 0; + /*[python checksum:...]*/ + + +Using a "self converter" +------------------------ + +Argument Clinic automatically adds a "self" parameter for you +using a default converter. However, you can override +Argument Clinic's converter and specify one yourself. +Just add your own ``self`` parameter as the first parameter in a +block, and ensure that its converter is an instance of +``self_converter`` or a subclass thereof. + +What's the point? This lets you automatically cast ``self`` +from ``PyObject *`` to a custom type. + +How do you specify the custom type you want to cast ``self`` to? +If you only have one or two functions with the same type for ``self``, +you can directly use Argument Clinic's existing ``self`` converter, +passing in the type you want to use as the ``type`` parameter:: + + /*[clinic] + + _pickle.Pickler.dump + + self: self(type="PicklerObject *") + obj: object + / + + Write a pickled representation of the given object to the open file. + [clinic]*/ + +On the other hand, if you have a lot of functions that will use the same +type for ``self``, it's best to create your own converter, subclassing +``self_converter`` but overwriting the ``type`` member:: + + /*[clinic] + class PicklerObject_converter(self_converter): + type = "PicklerObject *" + [clinic]*/ + + /*[clinic] + + _pickle.Pickler.dump + + self: PicklerObject + obj: object + / + + Write a pickled representation of the given object to the open file. + [clinic]*/ + + + +Writing a custom converter +-------------------------- + +As we hinted at in the previous section... you can write your own converters! +A converter is simply a Python class that inherits from ``CConverter``. +The main purpose of a custom converter is if you have a parameter using +the ``O&`` format unit--parsing this parameter means calling +a ``PyArg_ParseTuple()`` "converter function". + +Your converter class should be named ``*something*_converter``. +If the name follows this convention, then your converter class +will be automatically registered with Argument Clinic; its name +will be the name of your class with the ``_converter`` suffix +stripped off. (This is done automatically for you with a metaclass.) + +You shouldn't subclass ``CConverter.__init__``. Instead, you should +write a ``converter_init()`` function. ``converter_init()`` +always accepts a ``self`` parameter; after that, all additional +parameters *must* be keyword-only. Any arguments passed in to +the converter in Argument Clinic will be passed along to your +``converter_init()``. + +There are some additional members of ``CConverter`` you may wish +to specify in your subclass. Here's the current list: + +``type`` + The C type to use for this variable. + ``type`` should be a Python string specifying the type, e.g. ``int``. + If this is a pointer type, the type string should end with ``' *'``. + +``default`` + The Python default value for this parameter, as a Python value. + Or the magic value ``unspecified`` if there is no default. + +``doc_default`` + ``default`` as it should appear in the documentation, + as a string. + Or ``None`` if there is no default. + This string, when run through ``eval()``, should produce + a Python value. + +``py_default`` + ``default`` as it should appear in Python code, + as a string. + Or ``None`` if there is no default. + +``c_default`` + ``default`` as it should appear in C code, + as a string. + Or ``None`` if there is no default. + +``c_ignored_default`` + The default value used to initialize the C variable when + there is no default, but not specifying a default may + result in an "uninitialized variable" warning. This is + easily happen when using option groups--although + properly-written code won't actually use the variable, + the variable does get passed in to the _impl, and the + C compiler will complain about the "use" of the uninitialized + value. This value should be a string. + +``converter`` + The name of the C converter function, as a string. + +``impl_by_reference`` + A boolean value. If true, + Argument Clinic will add a ``&`` in front of the name of + the variable when passing it into the impl function. + +``parse_by_reference`` + A boolean value. If true, + Argument Clinic will add a ``&`` in front of the name of + the variable when passing it into ``PyArg_ParseTuple()``. + + +Here's the simplest example of a custom converter, from ``Modules/zlibmodule.c``:: + + /*[python] + + class uint_converter(CConverter): + type = 'unsigned int' + converter = 'uint_converter' + + [python]*/ + /*[python checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ + +This block adds a ``uint`` converter to Argument Clinic. Parameters +declared as ``uint`` will be declared as type ``unsigned int``, and will +be parsed by calling the ``uint_converter`` converter function in C. +``uint`` variables automatically support default values. + +More sophisticated custom converters can insert custom C code to +handle initialization and cleanup. +You can see more examples of custom converters in the CPython +source tree; grep the C files for the string ``CConverter``. + +Writing a custom return converter +--------------------------------- + +Writing a custom return converter is much like writing +a custom converter. Except it's much simpler, because return +converters are themselves much simpler. + +Return converters must subclass ``CReturnConverter``. +There are no examples yet of custom return converters, +because they are not widely used yet. If you wish to +write your own return converter, please read ``Tools/clinic/clinic.py``, +specifically the implementation of ``CReturnConverter`` and +all its subclasses. + + +Using Argument Clinic in Python files +------------------------------------- + +It's actually possible to use Argument Clinic to preprocess Python files. +There's no point to using Argument Clinic blocks, of course, as the output +wouldn't make any sense to the Python interpreter. But using Argument Clinic +to run Python blocks lets you use Python as a Python preprocessor! + +Since Python comments are different from C comments, Argument Clinic +blocks embedded in Python files look slightly different. They look like this:: + + #/*[python] + #print("def foo(): pass") + #[python]*/ + def foo(): pass + #/*[python checksum:...]*/ From 2aa5401c8ee745f9b8a6f3059ecd1cbeb5359920 Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Sun, 5 Jan 2014 02:50:45 -0800 Subject: [PATCH 02/71] Argument Clinic: fixed test suite, improved howto. --- Doc/howto/clinic.rst | 241 +++++++++++++++++++++++++++---------------- 1 file changed, 154 insertions(+), 87 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index afbeb417a..2d8a0533e 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -14,21 +14,20 @@ Argument Clinic How-To function to work with Argument Clinic, and then introduces some advanced topics on Argument Clinic usage. - Argument Clinic is currently considered an internal - tool for the CPython code tree. Its use is not supported - for files outside the CPython code tree, and no guarantees - are made regarding backwards compatibility for future - versions. In other words: if you maintain an external C - extension for CPython, you're welcome to experiment with - Argument Clinic in your own code. But the version of Argument - Clinic that ships with CPython 3.5 *could* be totally - incompatible and break all your code. + Currently Argument Clinic is considered internal-only + for CPython. Its use is not supported for files outside + CPython, and no guarantees are made regarding backwards + compatibility for future versions. In other words: if you + maintain an external C extension for CPython, you're welcome + to experiment with Argument Clinic in your own code. But the + version of Argument Clinic that ships with CPython 3.5 *could* + be totally incompatible and break all your code. ======================== Basic Concepts And Usage ======================== -Argument Clinic ships with CPython. You can find it in ``Tools/clinic/clinic.py``. +Argument Clinic ships with CPython; you'll find it in ``Tools/clinic/clinic.py``. If you run that script, specifying a C file as an argument:: % python3 Tools/clinic/clinic.py foo.c @@ -45,13 +44,12 @@ like this:: Everything in between these two lines is input for Argument Clinic. All of these lines, including the beginning and ending comment -lines, are collectively called an Argument Clinic "input block", -or "block" for short. +lines, are collectively called an Argument Clinic "block". When Argument Clinic parses one of these blocks, it generates output. This output is rewritten into the C file immediately after the block, followed by a comment containing a checksum. -The resulting Argument Clinic block looks like this:: +The Argument Clinic block now looks like this:: /*[clinic] ... clinic input goes here ... @@ -65,7 +63,8 @@ line. However, if the input hasn't changed, the output won't change either. You should never modify the output portion of an Argument Clinic block. Instead, change the input until it produces the output you want. (That's the purpose of the -checksum--to detect and warn you in case someone accidentally modifies the output.) +checksum--to detect if someone changed the output, as these edits would be lost +the next time Argument Clinic writes out fresh output.) For the sake of clarity, here's the terminology we'll use with Argument Clinic: @@ -87,10 +86,12 @@ Converting Your First Function The best way to get a sense of how Argument Clinic works is to convert a function to work with it. Let's dive in! -0. Make sure you're working with a freshly updated trunk. +0. Make sure you're working with a freshly updated checkout + of the CPython trunk. -1. Find a Python builtin that calls either ``PyArg_ParseTuple()`` - or ``PyArg_ParseTupleAndKeywords()``, and hasn't been converted yet. +1. Find a Python builtin that calls either :c:func:`PyArg_ParseTuple` + or :c:func:`PyArg_ParseTupleAndKeywords`, and hasn't been converted + to work with Argument Clinic yet. For my example I'm using ``pickle.Pickler.dump()``. 2. If the call to the ``PyArg_Parse`` function uses any of the @@ -103,7 +104,7 @@ convert a function to work with it. Let's dive in! et et# - or if it has multiple calls to ``PyArg_ParseTuple()``, + or if it has multiple calls to :c:func:`PyArg_ParseTuple`, you should choose a different function. Argument Clinic *does* support all of these scenarios. But these are advanced topics--let's do something simpler for your first function. @@ -130,7 +131,7 @@ convert a function to work with it. Let's dive in! be a paragraph consisting of a single 80-column line at the beginning of the docstring. - (Our docstring consists solely of the summary line, so the sample + (Our example docstring consists solely of a summary line, so the sample code doesn't have to change for this step.) 6. Above the docstring, enter the name of the function, followed @@ -198,7 +199,8 @@ convert a function to work with it. Let's dive in! string. ("format unit" is the formal name for the one-to-three character substring of the ``format`` parameter that tells the argument parsing function what the type of the variable - is and how to convert it.) + is and how to convert it. For more on format units please + see :ref:`arg-parsing`.) For multicharacter format units like ``z#``, use the entire two-or-three character string. @@ -231,14 +233,18 @@ convert a function to work with it. Let's dive in! (``pickle.Pickler.dump`` has neither, so our sample is unchanged.) -10. If the existing C function uses ``PyArg_ParseTuple()`` - (instead of ``PyArg_ParseTupleAndKeywords()``), then all its +10. If the existing C function calls :c:func:`PyArg_ParseTuple` + (as opposed to :c:func:`PyArg_ParseTupleAndKeywords`), then all its arguments are positional-only. To mark all parameters as positional-only in Argument Clinic, add a ``/`` on a line by itself after the last parameter, indented the same as the parameter lines. + Currently this is all-or-nothing; either all parameters are + positional-only, or none of them are. (In the future Argument + Clinic may relax this restriction.) + Sample:: /*[clinic] @@ -255,16 +261,16 @@ convert a function to work with it. Let's dive in! Write a pickled representation of obj to the open file. [clinic]*/ -11. It's helpful to write a per-parameter docstring, indented - another level past the parameter declaration. But per-parameter - docstrings are optional; you can skip this step if you prefer. +11. It's helpful to write a per-parameter docstring for each parameter. + But per-parameter docstrings are optional; you can skip this step + if you prefer. - Here's how per-parameter docstrings work. The first line + Here's how to add a per-parameter docstring. The first line of the per-parameter docstring must be indented further than the - parameter definition. This left margin establishes the left margin - for the whole per-parameter docstring; all the text you write will - be outdented by this amount. You can write as much as you like, - across multiple lines if you wish. + parameter definition. The left margin of this first line establishes + the left margin for the whole per-parameter docstring; all the text + you write will be outdented by this amount. You can write as much + text as you like, across multiple lines if you wish. Sample:: @@ -311,28 +317,47 @@ convert a function to work with it. Let's dive in! pickle_Pickler_dump_impl(PyObject *self, PyObject *obj) /*[clinic checksum: 3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ + Obviously, if Argument Clinic didn't produce any output, it's because + it found an error in your input. Keep fixing your errors and retrying + until Argument Clinic processes your file without complaint. + 13. Double-check that the argument-parsing code Argument Clinic generated looks basically the same as the existing code. First, ensure both places use the same argument-parsing function. The existing code must call either - ``PyArg_ParseTuple()`` or ``PyArg_ParseTupleAndKeywords()``; + :c:func:`PyArg_ParseTuple` or :c:func:`PyArg_ParseTupleAndKeywords`; ensure that the code generated by Argument Clinic calls the - same function. + *exact* same function. + + Second, the format string passed in to :c:func:`PyArg_ParseTuple` or + :c:func:`PyArg_ParseTupleAndKeywords` should be *exactly* the same + as the hand-written one in the existing function, up to the colon + or semi-colon. + + (Argument Clinic always generates its format strings + with a ``:`` followed by the name of the function. If the + existing code's format string ends with ``;``, to provide + usage help, this change is harmless--don't worry about it.) + + Third, for parameters whose format units require two arguments + (like a length variable, or an encoding string, or a pointer + to a conversion function), ensure that the second argument is + *exactly* the same between the two invocations. - Second, the format string passed in to ``PyArg_ParseTuple()`` or - ``PyArg_ParseTupleAndKeywords()`` should be *exactly* the same - as the hand-written one in the existing function. + Fourth, inside the output portion of the block you'll find a preprocessor + macro defining the appropriate static :c:type:`PyMethodDef` structure for + this builtin:: - Well, there's one way that Argument Clinic's output is permitted - to be different. Argument Clinic always generates a format string - ending with ``:`` followed by the name of the function. If the - format string originally ended with ``;`` (to specify usage help), - this is harmless--don't worry about this difference. + #define _PICKLE_PICKLER_DUMP_METHODDEF \ + {"dump", (PyCFunction)_pickle_Pickler_dump, METH_O, _pickle_Pickler_dump__doc__}, - Apart from that, if either of these things differ in *any way*, - fix your input to Argument Clinic and rerun ``Tools/clinic/clinic.py`` - until they are the same. + This static structure should be *exactly* the same as the existing static + :c:type:`PyMethodDef` structure for this builtin. + + If any of these items differ in *any way*, + adjust your Argument Clinic function specification and rerun + ``Tools/clinic/clinic.py`` until they *are* the same. 14. Notice that the last line of its output is the declaration @@ -342,8 +367,19 @@ convert a function to work with it. Let's dive in! declarations of all the variables it dumps the arguments into. Notice how the Python arguments are now arguments to this impl function; if the implementation used different names for these variables, fix it. - The result should be a function that handles just the implementation - of the Python function without any argument-parsing code. + + Let's reiterate, just because it's kind of weird. Your code should now + look like this:: + + static return_type + your_function_impl(...) + /*[clinic checksum: ...]*/ + { + ... + + Argument Clinic generated the checksum line and the function prototype just + above it. You should write the opening (and closing) curly braces for the + function, and the implementation inside. Sample:: @@ -386,7 +422,27 @@ convert a function to work with it. Let's dive in! ... -15. Compile and run the relevant portions of the regression-test suite. +15. Remember the macro with the :c:type:`PyMethodDef` structure for this + function? Find the existing :c:type:`PyMethodDef` structure for this + function and replace it with a reference to the macro. (If the builtin + is at module scope, this will probably be very near the end of the file; + if the builtin is a class method, this will probably be below but relatively + near to the implementation.) + + Note that the body of the macro contains a trailing comma. So when you + replace the existing static :c:type:`PyMethodDef` structure with the macro, + *don't* add a comma to the end. + + Sample:: + + static struct PyMethodDef Pickler_methods[] = { + _PICKLE_PICKLER_DUMP_METHODDEF + _PICKLE_PICKLER_CLEAR_MEMO_METHODDEF + {NULL, NULL} /* sentinel */ + }; + + +16. Compile, then run the relevant portions of the regression-test suite. This change should not introduce any new compile-time warnings or errors, and there should be no externally-visible change to Python's behavior. @@ -405,11 +461,11 @@ Renaming the C functions generated by Argument Clinic Argument Clinic automatically names the functions it generates for you. Occasionally this may cause a problem, if the generated name collides with -the name of an existing C function. There's an easy solution: you can explicitly -specify the base name to use for the C functions. Just add the keyword ``"as"`` +the name of an existing C function. There's an easy solution: override the names +used for the C functions. Just add the keyword ``"as"`` to your function declaration line, followed by the function name you wish to use. -Argument Clinic will use the function name you use for the base (generated) function, -and then add ``"_impl"`` to the end for the name of the impl function. +Argument Clinic will use that function name for the base (generated) function, +then add ``"_impl"`` to the end and use that for the name of the impl function. For example, if we wanted to rename the C function names generated for ``pickle.Pickler.dump``, it'd look like this:: @@ -420,7 +476,7 @@ For example, if we wanted to rename the C function names generated for ... The base function would now be named ``pickler_dumper()``, -and the impl function would be named ``pickler_dumper_impl()``. +and the impl function would now be named ``pickler_dumper_impl()``. Optional Groups @@ -428,15 +484,15 @@ Optional Groups Some legacy functions have a tricky approach to parsing their arguments: they count the number of positional arguments, then use a ``switch`` statement -to call one of several different ``PyArg_ParseTuple()`` calls depending on +to call one of several different :c:func:`PyArg_ParseTuple` calls depending on how many positional arguments there are. (These functions cannot accept keyword-only arguments.) This approach was used to simulate optional -arguments back before ``PyArg_ParseTupleAndKeywords()`` was created. +arguments back before :c:func:`PyArg_ParseTupleAndKeywords` was created. -Functions using this approach can often be converted to -use ``PyArg_ParseTupleAndKeywords()``, optional arguments, and default values. -But it's not always possible, because some of these legacy functions have -behaviors ``PyArg_ParseTupleAndKeywords()`` can't directly support. +While functions using this approach can often be converted to +use :c:func:`PyArg_ParseTupleAndKeywords`, optional arguments, and default values, +it's not always possible. Some of these legacy functions have +behaviors :c:func:`PyArg_ParseTupleAndKeywords` doesn't directly support. The most obvious example is the builtin function ``range()``, which has an optional argument on the *left* side of its required argument! Another example is ``curses.window.addch()``, which has a group of two @@ -445,16 +501,17 @@ called ``x`` and ``y``; if you call the function passing in ``x``, you must also pass in ``y``--and if you don't pass in ``x`` you may not pass in ``y`` either.) -For the sake of backwards compatibility, Argument Clinic supports this -alternate approach to parsing, using what are called *optional groups*. -Optional groups are groups of arguments that can only be specified together. +In any case, the goal of Argument Clinic is to support argument parsing +for all existing CPython builtins without changing their semantics. +Therefore Argument Clinic supports +this alternate approach to parsing, using what are called *optional groups*. +Optional groups are groups of arguments that must all be passed in together. They can be to the left or the right of the required arguments. They can *only* be used with positional-only parameters. To specify an optional group, add a ``[`` on a line by itself before -the parameters you wish to be -in a group together, and a ``]`` on a line by itself after the -parameters. As an example, here's how ``curses.window.addch`` +the parameters you wish to group together, and a ``]`` on a line by itself +after these parameters. As an example, here's how ``curses.window.addch`` uses optional groups to make the first two parameters and the last parameter optional:: @@ -484,8 +541,8 @@ parameter optional:: Notes: * For every optional group, one additional parameter will be passed into the - impl function representing the group. The parameter will be an int, and it will - be named ``group_{direction}_{number}``, + impl function representing the group. The parameter will be an int named + ``group_{direction}_{number}``, where ``{direction}`` is either ``right`` or ``left`` depending on whether the group is before or after the required parameters, and ``{number}`` is a monotonically increasing number (starting at 1) indicating how far away the group is from @@ -495,11 +552,13 @@ Notes: in this invocation.) * If there are no required arguments, the optional groups will behave - as if they are to the right of the required arguments. + as if they're to the right of the required arguments. * In the case of ambiguity, the argument parsing code favors parameters on the left (before the required parameters). +* Optional groups can only contain positional-only parameters. + * Optional groups are *only* intended for legacy code. Please do not use optional groups for new code. @@ -509,7 +568,7 @@ Using real Argument Clinic converters, instead of "legacy converters" To save time, and to minimize how much you need to learn to achieve your first port to Argument Clinic, the walkthrough above tells -you to use the "legacy converters". "Legacy converters" are a convenience, +you to use "legacy converters". "Legacy converters" are a convenience, designed explicitly to make porting existing code to Argument Clinic easier. And to be clear, their use is entirely acceptable when porting code for Python 3.4. @@ -523,18 +582,19 @@ reasons: because they require arguments, and the legacy converter syntax doesn't support specifying arguments. * In the future we may have a new argument parsing library that isn't - restricted to what ``PyArg_ParseTuple()`` supports. + restricted to what :c:func:`PyArg_ParseTuple` supports; this flexibility + won't be available to parameters using legacy converters. -So if you want -to go that extra effort, you should consider using normal -converters instead of the legacy converters. +Therefore, if you don't mind a little extra effort, you should consider +using normal converters instead of legacy converters. In a nutshell, the syntax for Argument Clinic (non-legacy) converters looks like a Python function call. However, if there are no explicit arguments to the function (all functions take their default values), you may omit the parentheses. Thus ``bool`` and ``bool()`` are exactly -the same. All parameters to Argument Clinic converters are keyword-only. +the same converters. +All arguments to Argument Clinic converters are keyword-only. All Argument Clinic converters accept the following arguments: ``doc_default`` @@ -643,11 +703,11 @@ the text, and add more entries to the dict mapping types to strings just above i Note also that this approach takes away some possible flexibility for the format units starting with ``e``. It used to be possible to decide at runtime what -encoding string to pass in to ``PyArg_ParseTuple()``. But now this string must +encoding string to pass in to :c:func:`PyArg_ParseTuple`. But now this string must be hard-coded at compile-time. This limitation is deliberate; it made supporting this format unit much easier, and may allow for future compile-time optimizations. This restriction does not seem unreasonable; CPython itself always passes in static -hard-coded strings when using format units starting with ``e``. +hard-coded encoding strings for parameters whose format units start with ``e``. Using a return converter @@ -692,12 +752,17 @@ None of these take parameters. For the first three, return -1 to indicate error. For ``DecodeFSDefault``, the return type is ``char *``; return a NULL pointer to indicate an error. +To see all the return converters Argument Clinic supports, along with +their parameters (if any), +just run ``Tools/clinic/clinic.py --converters`` for the full list. + + Calling Python code ------------------- The rest of the advanced topics require you to write Python code -which lives inside your C file and modifies Argument Clinic at -runtime. This is simple; you simply define a Python block. +which lives inside your C file and modifies Argument Clinic's +runtime state. This is simple: you simply define a Python block. A Python block uses different delimiter lines than an Argument Clinic function block. It looks like this:: @@ -778,13 +843,13 @@ As we hinted at in the previous section... you can write your own converters! A converter is simply a Python class that inherits from ``CConverter``. The main purpose of a custom converter is if you have a parameter using the ``O&`` format unit--parsing this parameter means calling -a ``PyArg_ParseTuple()`` "converter function". +a :c:func:`PyArg_ParseTuple` "converter function". Your converter class should be named ``*something*_converter``. If the name follows this convention, then your converter class will be automatically registered with Argument Clinic; its name will be the name of your class with the ``_converter`` suffix -stripped off. (This is done automatically for you with a metaclass.) +stripped off. (This is accomplished with a metaclass.) You shouldn't subclass ``CConverter.__init__``. Instead, you should write a ``converter_init()`` function. ``converter_init()`` @@ -825,12 +890,13 @@ to specify in your subclass. Here's the current list: ``c_ignored_default`` The default value used to initialize the C variable when there is no default, but not specifying a default may - result in an "uninitialized variable" warning. This is + result in an "uninitialized variable" warning. This can easily happen when using option groups--although - properly-written code won't actually use the variable, - the variable does get passed in to the _impl, and the - C compiler will complain about the "use" of the uninitialized - value. This value should be a string. + properly-written code will never actually use this value, + the variable does get passed in to the impl, and the + C compiler will complain about the "use" of the + uninitialized value. This value should always be a + non-empty string. ``converter`` The name of the C converter function, as a string. @@ -843,7 +909,7 @@ to specify in your subclass. Here's the current list: ``parse_by_reference`` A boolean value. If true, Argument Clinic will add a ``&`` in front of the name of - the variable when passing it into ``PyArg_ParseTuple()``. + the variable when passing it into :c:func:`PyArg_ParseTuple`. Here's the simplest example of a custom converter, from ``Modules/zlibmodule.c``:: @@ -857,9 +923,10 @@ Here's the simplest example of a custom converter, from ``Modules/zlibmodule.c`` [python]*/ /*[python checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ -This block adds a ``uint`` converter to Argument Clinic. Parameters +This block adds a converter to Argument Clinic named ``uint``. Parameters declared as ``uint`` will be declared as type ``unsigned int``, and will -be parsed by calling the ``uint_converter`` converter function in C. +be parsed by the ``'O&'`` format unit, which will call the ``uint_converter`` +converter function. ``uint`` variables automatically support default values. More sophisticated custom converters can insert custom C code to @@ -871,7 +938,7 @@ Writing a custom return converter --------------------------------- Writing a custom return converter is much like writing -a custom converter. Except it's much simpler, because return +a custom converter. Except it's somewhat simpler, because return converters are themselves much simpler. Return converters must subclass ``CReturnConverter``. From 5233c16bf3797d9245e46178a8bf102b6df20c5a Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Tue, 7 Jan 2014 12:13:13 -0800 Subject: [PATCH 03/71] Issue #20141: Improved Argument Clinic's support for the PyArg_Parse "O!" format unit. --- Doc/howto/clinic.rst | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 2d8a0533e..a22a11f22 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -640,7 +640,7 @@ on the right is the text you'd replace it with. ``'K'`` ``unsigned_PY_LONG_LONG`` ``'L'`` ``PY_LONG_LONG`` ``'n'`` ``Py_ssize_t`` -``'O!'`` ``object(type='name_of_Python_type')`` +``'O!'`` ``object(subclass_of='&PySomething_Type')`` ``'O&'`` ``object(converter='name_of_c_function')`` ``'O'`` ``object`` ``'p'`` ``bool`` @@ -693,20 +693,22 @@ conversion functions, or types, or strings specifying an encoding. (But "legacy converters" don't support arguments. That's why we skipped them for your first function.) The argument you specified to the format unit is now an argument to the converter; this -argument is either ``converter`` (for ``O&``), ``type`` (for ``O!``), +argument is either ``converter`` (for ``O&``), ``subclass_of`` (for ``O!``), or ``encoding`` (for all the format units that start with ``e``). -Note that ``object()`` must explicitly support each Python type you specify -for the ``type`` argument. Currently it only supports ``str``. It should be -easy to add more, just edit ``Tools/clinic/clinic.py``, search for ``O!`` in -the text, and add more entries to the dict mapping types to strings just above it. +When using ``subclass_of``, you may also want to use the other +custom argument for ``object()``: ``type``, which lets you set the type +actually used for the parameter. For example, if you want to ensure +that the object is a subclass of ``PyUnicode_Type``, you probably want +to use the converter ``object(type='PyUnicodeObject *', subclass_of='&PyUnicode_Type')``. -Note also that this approach takes away some possible flexibility for the format -units starting with ``e``. It used to be possible to decide at runtime what +One possible problem with using Argument Clinic: it takes away some possible +flexibility for the format units starting with ``e``. When writing a +``PyArg_Parse`` call by hand, you could theoretically decide at runtime what encoding string to pass in to :c:func:`PyArg_ParseTuple`. But now this string must -be hard-coded at compile-time. This limitation is deliberate; it made supporting -this format unit much easier, and may allow for future compile-time optimizations. -This restriction does not seem unreasonable; CPython itself always passes in static +be hard-coded at Argument-Clinic-preprocessing-time. This limitation is deliberate; +it made supporting this format unit much easier, and may allow for future optimizations. +This restriction doesn't seem unreasonable; CPython itself always passes in static hard-coded encoding strings for parameters whose format units start with ``e``. @@ -796,7 +798,8 @@ block, and ensure that its converter is an instance of ``self_converter`` or a subclass thereof. What's the point? This lets you automatically cast ``self`` -from ``PyObject *`` to a custom type. +from ``PyObject *`` to a custom type, just like ``object()`` +does with its ``type`` parameter. How do you specify the custom type you want to cast ``self`` to? If you only have one or two functions with the same type for ``self``, From dd8ecf3cde0e98200b491e6ebf2386cd4c810271 Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Tue, 7 Jan 2014 12:41:53 -0800 Subject: [PATCH 04/71] Issue #19273: The marker comments Argument Clinic uses have been changed to improve readability. --- Doc/howto/clinic.rst | 126 +++++++++++++++++++++---------------------- 1 file changed, 63 insertions(+), 63 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index a22a11f22..555fa68d4 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -35,12 +35,12 @@ If you run that script, specifying a C file as an argument:: Argument Clinic will scan over the file looking for lines that look exactly like this:: - /*[clinic] + /*[clinic input] When it finds one, it reads everything up to a line that looks -like this:: +exactly like this:: - [clinic]*/ + [clinic start generated code]*/ Everything in between these two lines is input for Argument Clinic. All of these lines, including the beginning and ending comment @@ -51,11 +51,11 @@ generates output. This output is rewritten into the C file immediately after the block, followed by a comment containing a checksum. The Argument Clinic block now looks like this:: - /*[clinic] + /*[clinic input] ... clinic input goes here ... - [clinic]*/ + [clinic start generated code]*/ ... clinic output goes here ... - /*[clinic checksum:...]*/ + /*[clinic end generated code: checksum=...]*/ If you run Argument Clinic on the same file a second time, Argument Clinic will discard the old output and write out the new output with a fresh checksum @@ -68,9 +68,9 @@ the next time Argument Clinic writes out fresh output.) For the sake of clarity, here's the terminology we'll use with Argument Clinic: -* The first line of the comment (``/*[clinic]``) is the *start line*. -* The last line of the initial comment (``[clinic]*/``) is the *end line*. -* The last line (``/*[clinic checksum:...]*/``) is the *checksum line*. +* The first line of the comment (``/*[clinic input]``) is the *start line*. +* The last line of the initial comment (``[clinic start generated code]*/``) is the *end line*. +* The last line (``/*[clinic end generated code: checksum=...]*/``) is the *checksum line*. * In between the start line and the end line is the *input*. * In between the end line and the checksum line is the *output*. * All the text collectively, from the start line to the checksum line inclusively, @@ -111,8 +111,8 @@ convert a function to work with it. Let's dive in! 3. Add the following boilerplate above the function, creating our block:: - /*[clinic] - [clinic]*/ + /*[clinic input] + [clinic start generated code]*/ 4. Cut the docstring and paste it in between the ``[clinic]`` lines, removing all the junk that makes it a properly quoted C string. @@ -122,9 +122,9 @@ convert a function to work with it. Let's dive in! Sample:: - /*[clinic] + /*[clinic input] Write a pickled representation of obj to the open file. - [clinic]*/ + [clinic start generated code]*/ 5. If your docstring doesn't have a "summary" line, Argument Clinic will complain. So let's make sure it has one. The "summary" line should @@ -143,11 +143,11 @@ convert a function to work with it. Let's dive in! Sample:: - /*[clinic] + /*[clinic input] pickle.Pickler.dump Write a pickled representation of obj to the open file. - [clinic]*/ + [clinic start generated code]*/ 7. If this is the first time that module or class has been used with Argument Clinic in this C file, @@ -159,16 +159,16 @@ convert a function to work with it. Let's dive in! Sample:: - /*[clinic] + /*[clinic input] module pickle class pickle.Pickler - [clinic]*/ + [clinic start generated code]*/ - /*[clinic] + /*[clinic input] pickle.Pickler.dump Write a pickled representation of obj to the open file. - [clinic]*/ + [clinic start generated code]*/ 8. Declare each of the parameters to the function. Each parameter @@ -207,18 +207,18 @@ convert a function to work with it. Let's dive in! Sample:: - /*[clinic] + /*[clinic input] module pickle class pickle.Pickler - [clinic]*/ + [clinic start generated code]*/ - /*[clinic] + /*[clinic input] pickle.Pickler.dump obj: 'O' Write a pickled representation of obj to the open file. - [clinic]*/ + [clinic start generated code]*/ 9. If your function has ``|`` in the format string, meaning some parameters have default values, you can ignore it. Argument @@ -247,19 +247,19 @@ convert a function to work with it. Let's dive in! Sample:: - /*[clinic] + /*[clinic input] module pickle class pickle.Pickler - [clinic]*/ + [clinic start generated code]*/ - /*[clinic] + /*[clinic input] pickle.Pickler.dump obj: 'O' / Write a pickled representation of obj to the open file. - [clinic]*/ + [clinic start generated code]*/ 11. It's helpful to write a per-parameter docstring for each parameter. But per-parameter docstrings are optional; you can skip this step @@ -274,12 +274,12 @@ convert a function to work with it. Let's dive in! Sample:: - /*[clinic] + /*[clinic input] module pickle class pickle.Pickler - [clinic]*/ + [clinic start generated code]*/ - /*[clinic] + /*[clinic input] pickle.Pickler.dump obj: 'O' @@ -287,19 +287,19 @@ convert a function to work with it. Let's dive in! / Write a pickled representation of obj to the open file. - [clinic]*/ + [clinic start generated code]*/ 12. Save and close the file, then run ``Tools/clinic/clinic.py`` on it. With luck everything worked and your block now has output! Reopen the file in your text editor to see:: - /*[clinic] + /*[clinic input] module pickle class pickle.Pickler - [clinic]*/ - /*[clinic checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ + [clinic start generated code]*/ + /*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ - /*[clinic] + /*[clinic input] pickle.Pickler.dump obj: 'O' @@ -307,7 +307,7 @@ convert a function to work with it. Let's dive in! / Write a pickled representation of obj to the open file. - [clinic]*/ + [clinic start generated code]*/ PyDoc_STRVAR(pickle_Pickler_dump__doc__, "Write a pickled representation of obj to the open file.\n" @@ -315,7 +315,7 @@ convert a function to work with it. Let's dive in! ... static PyObject * pickle_Pickler_dump_impl(PyObject *self, PyObject *obj) - /*[clinic checksum: 3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ + /*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ Obviously, if Argument Clinic didn't produce any output, it's because it found an error in your input. Keep fixing your errors and retrying @@ -373,7 +373,7 @@ convert a function to work with it. Let's dive in! static return_type your_function_impl(...) - /*[clinic checksum: ...]*/ + /*[clinic end generated code: checksum=...]*/ { ... @@ -383,13 +383,13 @@ convert a function to work with it. Let's dive in! Sample:: - /*[clinic] + /*[clinic input] module pickle class pickle.Pickler - [clinic]*/ - /*[clinic checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ + [clinic start generated code]*/ + /*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ - /*[clinic] + /*[clinic input] pickle.Pickler.dump obj: 'O' @@ -397,7 +397,7 @@ convert a function to work with it. Let's dive in! / Write a pickled representation of obj to the open file. - [clinic]*/ + [clinic start generated code]*/ PyDoc_STRVAR(pickle_Pickler_dump__doc__, "Write a pickled representation of obj to the open file.\n" @@ -405,7 +405,7 @@ convert a function to work with it. Let's dive in! ... static PyObject * pickle_Pickler_dump_impl(PyObject *self, PyObject *obj) - /*[clinic checksum: 3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ + /*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ { /* Check whether the Pickler was initialized correctly (issue3664). Developers often forget to call __init__() in their subclasses, which @@ -470,7 +470,7 @@ then add ``"_impl"`` to the end and use that for the name of the impl function. For example, if we wanted to rename the C function names generated for ``pickle.Pickler.dump``, it'd look like this:: - /*[clinic] + /*[clinic input] pickle.Pickler.dump as pickler_dumper ... @@ -515,7 +515,7 @@ after these parameters. As an example, here's how ``curses.window.addch`` uses optional groups to make the first two parameters and the last parameter optional:: - /*[clinic] + /*[clinic input] curses.window.addch @@ -666,7 +666,7 @@ on the right is the text you'd replace it with. As an example, here's our sample ``pickle.Pickler.dump`` using the proper converter:: - /*[clinic] + /*[clinic input] pickle.Pickler.dump obj: object @@ -674,7 +674,7 @@ converter:: / Write a pickled representation of obj to the open file. - [clinic]*/ + [clinic start generated code]*/ Argument Clinic will show you all the converters it has available. For each converter it'll show you all the parameters @@ -769,9 +769,9 @@ runtime state. This is simple: you simply define a Python block. A Python block uses different delimiter lines than an Argument Clinic function block. It looks like this:: - /*[python] + /*[python input] # python code goes here - [python]*/ + [python start generated code]*/ All the code inside the Python block is executed at the time it's parsed. All text written to stdout inside the block @@ -780,9 +780,9 @@ is redirected into the "output" after the block. As an example, here's a Python block that adds a static integer variable to the C code:: - /*[python] + /*[python input] print('static int __ignored_unused_variable__ = 0;') - [python]*/ + [python start generated code]*/ static int __ignored_unused_variable__ = 0; /*[python checksum:...]*/ @@ -806,7 +806,7 @@ If you only have one or two functions with the same type for ``self``, you can directly use Argument Clinic's existing ``self`` converter, passing in the type you want to use as the ``type`` parameter:: - /*[clinic] + /*[clinic input] _pickle.Pickler.dump @@ -815,18 +815,18 @@ passing in the type you want to use as the ``type`` parameter:: / Write a pickled representation of the given object to the open file. - [clinic]*/ + [clinic start generated code]*/ On the other hand, if you have a lot of functions that will use the same type for ``self``, it's best to create your own converter, subclassing ``self_converter`` but overwriting the ``type`` member:: - /*[clinic] + /*[clinic input] class PicklerObject_converter(self_converter): type = "PicklerObject *" - [clinic]*/ + [clinic start generated code]*/ - /*[clinic] + /*[clinic input] _pickle.Pickler.dump @@ -835,7 +835,7 @@ type for ``self``, it's best to create your own converter, subclassing / Write a pickled representation of the given object to the open file. - [clinic]*/ + [clinic start generated code]*/ @@ -917,14 +917,14 @@ to specify in your subclass. Here's the current list: Here's the simplest example of a custom converter, from ``Modules/zlibmodule.c``:: - /*[python] + /*[python input] class uint_converter(CConverter): type = 'unsigned int' converter = 'uint_converter' - [python]*/ - /*[python checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ + [python start generated code]*/ + /*[python end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ This block adds a converter to Argument Clinic named ``uint``. Parameters declared as ``uint`` will be declared as type ``unsigned int``, and will @@ -963,8 +963,8 @@ to run Python blocks lets you use Python as a Python preprocessor! Since Python comments are different from C comments, Argument Clinic blocks embedded in Python files look slightly different. They look like this:: - #/*[python] + #/*[python input] #print("def foo(): pass") - #[python]*/ + #[python start generated code]*/ def foo(): pass #/*[python checksum:...]*/ From 6d59917e5aad2bb5c6be9d341e466d942ec659bb Mon Sep 17 00:00:00 2001 From: Zachary Ware Date: Thu, 9 Jan 2014 21:41:23 -0600 Subject: [PATCH 05/71] Closes #20200: Argument Clinic custom converter example should be in a 'python' clinic block, not a 'clinic' clinic block. Patch written by Ryan Smith-Roberts. --- Doc/howto/clinic.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 555fa68d4..2324142c6 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -821,10 +821,10 @@ On the other hand, if you have a lot of functions that will use the same type for ``self``, it's best to create your own converter, subclassing ``self_converter`` but overwriting the ``type`` member:: - /*[clinic input] + /*[python input] class PicklerObject_converter(self_converter): type = "PicklerObject *" - [clinic start generated code]*/ + [python start generated code]*/ /*[clinic input] From 81492e6d7a243370634a5c76a7898dccf0d02630 Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Sun, 12 Jan 2014 11:09:57 -0800 Subject: [PATCH 06/71] Issue #20214: Fixed a number of small issues and documentation errors in Argument Clinic (see issue for details). --- Doc/howto/clinic.rst | 140 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 120 insertions(+), 20 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 2324142c6..e9e1377cb 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -109,6 +109,13 @@ convert a function to work with it. Let's dive in! support all of these scenarios. But these are advanced topics--let's do something simpler for your first function. + Also, if the function has multiple calls to :c:func:`PyArg_ParseTuple` + or :c:func:`PyArg_ParseTupleAndKeywords` where it supports different + types for the same argument, or if the function uses something besides + PyArg_Parse functions to parse its arguments, it probably + isn't suitable for conversion to Argument Clinic. Argument Clinic + doesn't support generic functions or polymorphic parameters. + 3. Add the following boilerplate above the function, creating our block:: /*[clinic input] @@ -170,6 +177,11 @@ convert a function to work with it. Let's dive in! Write a pickled representation of obj to the open file. [clinic start generated code]*/ + The name of the class and module should be the same as the one + seen by Python. Check the name defined in the :c:type:`PyModuleDef` + or :c:type:`PyTypeObject` as appropriate. + + 8. Declare each of the parameters to the function. Each parameter should get its own line. All the parameter lines should be @@ -455,6 +467,28 @@ convert a function to work with it. Let's dive in! Advanced Topics =============== +Now that you've had some experience working with Argument Clinic, it's time +for some advanced topics. + + +Symbolic default values +----------------------- + +The default value you provide for a parameter can't be any arbitrary +expression. Currently the following are explicitly supported: + +* Numeric constants (integer and float) +* String constants +* ``True``, ``False``, and ``None`` +* Simple symbolic constants like ``sys.maxsize``, which must + start with the name of the module + +In case you're curious, this is implemented in ``from_builtin()`` +in ``Lib/inspect.py``. + +(In the future, this may need to get even more elaborate, +to allow full expressions like ``CONSTANT - 1``.) + Renaming the C functions generated by Argument Clinic ----------------------------------------------------- @@ -479,6 +513,29 @@ The base function would now be named ``pickler_dumper()``, and the impl function would now be named ``pickler_dumper_impl()``. +The NULL default value +---------------------- + +For string and object parameters, you can set them to ``None`` to indicate +that there is no default. However, that means the C variable will be +initialized to ``Py_None``. For convenience's sakes, there's a special +value called ``NULL`` for just this case: from Python's perspective it +behaves like a default value of ``None``, but the C variable is initialized +with ``NULL``. + + +Converting functions using PyArg_UnpackTuple +-------------------------------------------- + +To convert a function parsing its arguments with :c:func:`PyArg_UnpackTuple`, +simply write out all the arguments, specifying each as an ``object``. You +may specify the ``type`` argument to cast the type as appropriate. All +arguments should be marked positional-only (add a ``/`` on a line by itself +after the last argument). + +Currently the generated code will use :c:func:`PyArg_ParseTuple`, but this +will change soon. + Optional Groups --------------- @@ -570,8 +627,8 @@ To save time, and to minimize how much you need to learn to achieve your first port to Argument Clinic, the walkthrough above tells you to use "legacy converters". "Legacy converters" are a convenience, designed explicitly to make porting existing code to Argument Clinic -easier. And to be clear, their use is entirely acceptable when porting -code for Python 3.4. +easier. And to be clear, their use is acceptable when porting code for +Python 3.4. However, in the long term we probably want all our blocks to use Argument Clinic's real syntax for converters. Why? A couple @@ -585,8 +642,8 @@ reasons: restricted to what :c:func:`PyArg_ParseTuple` supports; this flexibility won't be available to parameters using legacy converters. -Therefore, if you don't mind a little extra effort, you should consider -using normal converters instead of legacy converters. +Therefore, if you don't mind a little extra effort, please use the normal +converters instead of legacy converters. In a nutshell, the syntax for Argument Clinic (non-legacy) converters looks like a Python function call. However, if there are no explicit @@ -597,12 +654,19 @@ the same converters. All arguments to Argument Clinic converters are keyword-only. All Argument Clinic converters accept the following arguments: -``doc_default`` - If the parameter takes a default value, normally this value is also - provided in the ``inspect.Signature`` metadata, and displayed in the - docstring. ``doc_default`` lets you override the value used in these - two places: pass in a string representing the Python value you wish - to use in these two contexts. +``py_default`` + The default value for this parameter when defined in Python. + Specifically, the value that will be used in the ``inspect.Signature`` + string. + If a default value is specified for the parameter, defaults to + ``repr(default)``, else defaults to ``None``. + Specified as a string. + +``c_default`` + The default value for this parameter when defined in C. + Specifically, this will be the initializer for the variable declared + in the "parse function". + Specified as a string. ``required`` If a parameter takes a default value, Argument Clinic infers that the @@ -612,6 +676,9 @@ All Argument Clinic converters accept the following arguments: Clinic that this parameter is not optional, even if it has a default value. + (The need for ``required`` may be obviated by ``c_default``, which is + newer but arguably a better solution.) + ``annotation`` The annotation value for this parameter. Not currently supported, because PEP 8 mandates that the Python library may not use @@ -634,10 +701,11 @@ on the right is the text you'd replace it with. ``'et'`` ``str(encoding='name_of_encoding', types='bytes bytearray str')`` ``'f'`` ``float`` ``'h'`` ``short`` -``'H'`` ``unsigned_short`` +``'H'`` ``unsigned_short(bitwise=True)`` ``'i'`` ``int`` -``'I'`` ``unsigned_int`` -``'K'`` ``unsigned_PY_LONG_LONG`` +``'I'`` ``unsigned_int(bitwise=True)`` +``'k'`` ``unsigned_long(bitwise=True)`` +``'K'`` ``unsigned_PY_LONG_LONG(bitwise=True)`` ``'L'`` ``PY_LONG_LONG`` ``'n'`` ``Py_ssize_t`` ``'O!'`` ``object(subclass_of='&PySomething_Type')`` @@ -681,6 +749,14 @@ available. For each converter it'll show you all the parameters it accepts, along with the default value for each parameter. Just run ``Tools/clinic/clinic.py --converters`` to see the full list. +Py_buffer +--------- + +When using the ``Py_buffer`` converter +(or the ``'s*'``, ``'w*'``, ``'*y'``, or ``'z*'`` legacy converters) +note that the code Argument Clinic generates for you will automatically +call :c:func:`PyBuffer_Release` on the buffer for you. + Advanced converters ------------------- @@ -745,15 +821,26 @@ encode the value you return like normal. Currently Argument Clinic supports only a few return converters:: + bool int + unsigned int long + unsigned int + size_t Py_ssize_t + float + double DecodeFSDefault None of these take parameters. For the first three, return -1 to indicate error. For ``DecodeFSDefault``, the return type is ``char *``; return a NULL pointer to indicate an error. +(There's also an experimental ``NoneType`` converter, which lets you +return ``Py_None`` on success or ``NULL`` on failure, without having +to increment the reference count on ``Py_None``. I'm not sure it adds +enough clarity to be worth using.) + To see all the return converters Argument Clinic supports, along with their parameters (if any), just run ``Tools/clinic/clinic.py --converters`` for the full list. @@ -873,13 +960,6 @@ to specify in your subclass. Here's the current list: The Python default value for this parameter, as a Python value. Or the magic value ``unspecified`` if there is no default. -``doc_default`` - ``default`` as it should appear in the documentation, - as a string. - Or ``None`` if there is no default. - This string, when run through ``eval()``, should produce - a Python value. - ``py_default`` ``default`` as it should appear in Python code, as a string. @@ -951,6 +1031,26 @@ write your own return converter, please read ``Tools/clinic/clinic.py``, specifically the implementation of ``CReturnConverter`` and all its subclasses. +METH_O and METH_NOARGS +---------------------------------------------- + +To convert a function using ``METH_O``, make sure the function's +single argument is using the ``object`` converter, and mark the +arguments as positional-only:: + + /*[clinic input] + meth_o_sample + + argument: object + / + [clinic start generated code]*/ + + +To convert a function using ``METH_NOARGS``, just don't specify +any arguments. + +You can still use a self converter, a return converter, and specify +a ``type`` argument to the object converter for ``METH_O``. Using Argument Clinic in Python files ------------------------------------- From ade6a87b45b5523cab3d1cf1900a842dfc20135a Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Sun, 12 Jan 2014 13:57:36 -0800 Subject: [PATCH 07/71] Minor doc fix in Clinic howto. --- Doc/howto/clinic.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index e9e1377cb..9c558bc50 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -753,9 +753,10 @@ Py_buffer --------- When using the ``Py_buffer`` converter -(or the ``'s*'``, ``'w*'``, ``'*y'``, or ``'z*'`` legacy converters) -note that the code Argument Clinic generates for you will automatically -call :c:func:`PyBuffer_Release` on the buffer for you. +(or the ``'s*'``, ``'w*'``, ``'*y'``, or ``'z*'`` legacy converters), +you *must* not call :c:func:`PyBuffer_Release` on the provided buffer. +Argument Clinic generates code that does it for you (in the parsing function). + Advanced converters From 70f458e86e7c5ee67a1fb7158157edf87bd6a544 Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Tue, 14 Jan 2014 22:22:41 -0800 Subject: [PATCH 08/71] Issue #20268: Argument Clinic now supports cloning the parameters and return converter from existing functions. --- Doc/howto/clinic.rst | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 9c558bc50..0df8d445c 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -847,6 +847,49 @@ their parameters (if any), just run ``Tools/clinic/clinic.py --converters`` for the full list. +Cloning existing functions +-------------------------- + +If you have a number of functions that look similar, you may be able to +use Clinic's "clone" feature. When you clone an existing function, +you reuse: + +* its parameters, including + + * their names, + + * their converters, with all parameters, + + * their default values, + + * their per-parameter docstrings, + + * their *kind* (whether they're positional only, + positional or keyword, or keyword only), and + +* its return converter. + +The only thing not copied from the original function is its docstring; +the syntax allows you to specify a new docstring. + +Here's the syntax for cloning a function:: + + /*[clinic input] + module.class.new_function [as c_basename] = module.class.existing_function + + Docstring for new_function goes here. + [clinic start generated code]*/ + +(The functions can be in different modules or classes. I wrote +``module.class`` in the sample just to illustrate that you must +use the full path to *both* functions.) + +Sorry, there's no syntax for partially-cloning a function, or cloning a function +then modifying it. Cloning is an all-or nothing proposition. + +Also, the function you are cloning from must have been previously defined +in the current file. + Calling Python code ------------------- From c472248cddc580b76248e891ef3977e6898a6914 Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Thu, 16 Jan 2014 11:32:01 -0800 Subject: [PATCH 09/71] Issue #20226: Major improvements to Argument Clinic. * You may now specify an expression as the default value for a parameter! Example: "sys.maxsize - 1". This support is intentionally quite limited; you may only use values that can be represented as static C values. * Removed "doc_default", simplified support for "c_default" and "py_default". (I'm not sure we still even need "py_default", but I'm leaving it in for now in case a use presents itself.) * Parameter lines support a trailing '\\' as a line continuation character, allowing you to break up long lines. * The argument parsing code generated when supporting optional groups now uses PyTuple_GET_SIZE instead of PyTuple_GetSize, leading to a 850% speedup in parsing. (Just kidding, this is an unmeasurable difference.) * A bugfix for the recent regression where the generated prototype from pydoc for builtins would be littered with unreadable "="" default values for parameters that had no default value. * Converted some asserts into proper failure messages. * Many doc improvements and fixes. --- Doc/howto/clinic.rst | 269 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 227 insertions(+), 42 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 0df8d445c..96f84b01e 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -127,6 +127,12 @@ convert a function to work with it. Let's dive in! margin, with no line wider than 80 characters. (Argument Clinic will preserve indents inside the docstring.) + If the old docstring had a first line that looked like a function + signature, throw that line away. (The docstring doesn't need it + anymore--when you use ``help()`` on your builtin in the future, + the first line will be built automatically based on the function's + signature.) + Sample:: /*[clinic input] @@ -196,6 +202,10 @@ convert a function to work with it. Let's dive in! name_of_parameter: converter = default_value + Argument Clinic's support for "default values" is quite sophisticated; + please see :ref:`the section below on default values ` + for more information. + Add a blank line below the parameters. What's a "converter"? It establishes both the type @@ -513,16 +523,6 @@ The base function would now be named ``pickler_dumper()``, and the impl function would now be named ``pickler_dumper_impl()``. -The NULL default value ----------------------- - -For string and object parameters, you can set them to ``None`` to indicate -that there is no default. However, that means the C variable will be -initialized to ``Py_None``. For convenience's sakes, there's a special -value called ``NULL`` for just this case: from Python's perspective it -behaves like a default value of ``None``, but the C variable is initialized -with ``NULL``. - Converting functions using PyArg_UnpackTuple -------------------------------------------- @@ -654,35 +654,70 @@ the same converters. All arguments to Argument Clinic converters are keyword-only. All Argument Clinic converters accept the following arguments: -``py_default`` - The default value for this parameter when defined in Python. - Specifically, the value that will be used in the ``inspect.Signature`` - string. - If a default value is specified for the parameter, defaults to - ``repr(default)``, else defaults to ``None``. - Specified as a string. - -``c_default`` - The default value for this parameter when defined in C. - Specifically, this will be the initializer for the variable declared - in the "parse function". - Specified as a string. - -``required`` - If a parameter takes a default value, Argument Clinic infers that the - parameter is optional. However, you may want a parameter to take a - default value in C, but not behave in Python as if the parameter is - optional. Passing in ``required=True`` to a converter tells Argument - Clinic that this parameter is not optional, even if it has a default - value. - - (The need for ``required`` may be obviated by ``c_default``, which is - newer but arguably a better solution.) - -``annotation`` - The annotation value for this parameter. Not currently supported, - because PEP 8 mandates that the Python library may not use - annotations. + ``c_default`` + The default value for this parameter when defined in C. + Specifically, this will be the initializer for the variable declared + in the "parse function". See :ref:`the section on default values ` + for how to use this. + Specified as a string. + + ``annotation`` + The annotation value for this parameter. Not currently supported, + because PEP 8 mandates that the Python library may not use + annotations. + +In addition, some converters accept additional arguments. Here is a list +of these arguments, along with their meanings: + + ``bitwise`` + Only supported for unsigned integers. The native integer value of this + Python argument will be written to the parameter without any range checking, + even for negative values. + + ``converter`` + Only supported by the ``object`` converter. Specifies the name of a + :ref:`C "converter function" ` + to use to convert this object to a native type. + + ``encoding`` + Only supported for strings. Specifies the encoding to use when converting + this string from a Python str (Unicode) value into a C ``char *`` value. + + ``length`` + Only supported for strings. If true, requests that the length of the + string be passed in to the impl function, just after the string parameter, + in a parameter named ``_length``. + + ``nullable`` + Only supported for strings. If true, this parameter may also be set to + ``None``, in which case the C parameter will be set to ``NULL``. + + ``subclass_of`` + Only supported for the ``object`` converter. Requires that the Python + value be a subclass of a Python type, as expressed in C. + + ``types`` + Only supported for the ``object`` (and ``self``) converter. Specifies + the C type that will be used to declare the variable. Default value is + ``"PyObject *"``. + + ``types`` + A string containing a list of Python types (and possibly pseudo-types); + this restricts the allowable Python argument to values of these types. + (This is not a general-purpose facility; as a rule it only supports + specific lists of types as shown in the legacy converter table.) + + ``zeroes`` + Only supported for strings. If true, embedded NUL bytes (``'\\0'``) are + permitted inside the value. + +Please note, not every possible combination of arguments will work. +Often these arguments are implemented internally by specific ``PyArg_ParseTuple`` +*format units*, with specific behavior. For example, currently you cannot +call ``str`` and pass in ``zeroes=True`` without also specifying an ``encoding``; +although it's perfectly reasonable to think this would work, these semantics don't +map to any existing format unit. So Argument Clinic doesn't support it. (Or, at +least, not yet.) Below is a table showing the mapping of legacy converters into real Argument Clinic converters. On the left is the legacy converter, @@ -720,9 +755,9 @@ on the right is the text you'd replace it with. ``'u'`` ``Py_UNICODE`` ``'U'`` ``unicode`` ``'w*'`` ``Py_buffer(types='bytearray rwbuffer')`` -``'y#'`` ``str(type='bytes', length=True)`` +``'y#'`` ``str(types='bytes', length=True)`` ``'Y'`` ``PyByteArrayObject`` -``'y'`` ``str(type='bytes')`` +``'y'`` ``str(types='bytes')`` ``'y*'`` ``Py_buffer`` ``'Z#'`` ``Py_UNICODE(nullable=True, length=True)`` ``'z#'`` ``str(nullable=True, length=True)`` @@ -789,6 +824,90 @@ This restriction doesn't seem unreasonable; CPython itself always passes in stat hard-coded encoding strings for parameters whose format units start with ``e``. +.. _default_values: + +Parameter default values +------------------------ + +Default values for parameters can be any of a number of values. +At their simplest, they can be string, int, or float literals:: + + foo: str = "abc" + bar: int = 123 + bat: float = 45.6 + +They can also use any of Python's built-in constants:: + + yep: bool = True + nope: bool = False + nada: object = None + +There's also special support for a default value of ``NULL``, and +for simple expressions, documented in the following sections. + + +The ``NULL`` default value +-------------------------- + +For string and object parameters, you can set them to ``None`` to indicate +that there's no default. However, that means the C variable will be +initialized to ``Py_None``. For convenience's sakes, there's a special +value called ``NULL`` for just this reason: from Python's perspective it +behaves like a default value of ``None``, but the C variable is initialized +with ``NULL``. + +Expressions specified as default values +--------------------------------------- + +The default value for a parameter can be more than just a literal value. +It can be an entire expression, using math operators and looking up attributes +on objects. However, this support isn't exactly simple, because of some +non-obvious semantics. + +Consider the following example:: + + foo: Py_ssize_t = sys.maxsize - 1 + +``sys.maxsize`` can have different values on different platforms. Therefore +Argument Clinic can't simply evaluate that expression locally and hard-code it +in C. So it stores the default in such a way that it will get evaluated at +runtime, when the user asks for the function's signature. + +What namespace is available when the expression is evaluated? It's evaluated +in the context of the module the builtin came from. So, if your module has an +attribute called "``max_widgets``", you may simply use it:: + + foo: Py_ssize_t = max_widgets + +If the symbol isn't found in the current module, it fails over to looking in +``sys.modules``. That's how it can find ``sys.maxsize`` for example. (Since you +don't know in advance what modules the user will load into their interpreter, +it's best to restrict yourself to modules that are preloaded by Python itself.) + +Evaluating default values only at runtime means Argument Clinic can't compute +the correct equivalent C default value. So you need to tell it explicitly. +When you use an expression, you must also specify the equivalent expression +in C, using the ``c_default`` parameter to the converter:: + + foo: Py_ssize_t(c_default="PY_SSIZE_T_MAX - 1") = sys.maxsize - 1 + +Another complication: Argument Clinic can't know in advance whether or not the +expression you supply is valid. It parses it to make sure it looks legal, but +it can't *actually* know. You must be very careful when using expressions to +specify values that are guaranteed to be valid at runtime! + +Finally, because expressions must be representable as static C values, there +are many restrictions on legal expressions. Here's a list of Python features +you're not permitted to use: + +* Function calls. +* Inline if statements (``3 if foo else 5``). +* Automatic sequence unpacking (``*[1, 2, 3]``). +* List/set/dict comprehensions and generator expressions. +* Tuple/list/set/dict literals. + + + Using a return converter ------------------------ @@ -1096,7 +1215,73 @@ any arguments. You can still use a self converter, a return converter, and specify a ``type`` argument to the object converter for ``METH_O``. -Using Argument Clinic in Python files +The #ifdef trick +---------------------------------------------- + +If you're converting a function that isn't available on all platforms, +there's a trick you can use to make life a little easier. The existing +code probably looks like this:: + + #ifdef HAVE_FUNCTIONNAME + static module_functionname(...) + { + ... + } + #endif /* HAVE_FUNCTIONNAME */ + +And then in the ``PyMethodDef`` structure at the bottom you'll have:: + + #ifdef HAVE_FUNCTIONNAME + {'functionname', ... }, + #endif /* HAVE_FUNCTIONNAME */ + +In this scenario, you should change the code to look like the following:: + + #ifdef HAVE_FUNCTIONNAME + /*[clinic input] + module.functionname + ... + [clinic start generated code]*/ + static module_functionname(...) + { + ... + } + #endif /* HAVE_FUNCTIONNAME */ + +Run Argument Clinic on the code in this state, then refresh the file in +your editor. Now you'll have the generated code, including the #define +for the ``PyMethodDef``, like so:: + + #ifdef HAVE_FUNCTIONNAME + /*[clinic input] + ... + [clinic start generated code]*/ + ... + #define MODULE_FUNCTIONNAME \ + {'functionname', ... }, + ... + /*[clinic end generated code: checksum=...]*/ + static module_functionname(...) + { + ... + } + #endif /* HAVE_FUNCTIONNAME */ + +Change the #endif at the bottom as follows:: + + #else + #define MODULE_FUNCTIONNAME + #endif /* HAVE_FUNCTIONNAME */ + +Now you can remove the #ifdefs around the ``PyMethodDef`` structure +at the end, and replace those three lines with ``MODULE_FUNCTIONNAME``. +If the function is available, the macro turns into the ``PyMethodDef`` +static value, including the trailing comma; if the function isn't +available, the macro turns into nothing. Perfect! + +(This is the preferred approach for optional functions; in the future, +Argument Clinic may generate the entire ``PyMethodDef`` structure.) + ------------------------------------- It's actually possible to use Argument Clinic to preprocess Python files. From 5648fafbc078a99ff0673d21ae9fe3ba25ddf529 Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Fri, 17 Jan 2014 17:47:17 -0800 Subject: [PATCH 10/71] Issue #20287: Argument Clinic's output is now configurable, allowing delaying its output or even redirecting it to a separate file. --- Doc/howto/clinic.rst | 320 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 319 insertions(+), 1 deletion(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 96f84b01e..a61508e3f 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -23,6 +23,58 @@ Argument Clinic How-To version of Argument Clinic that ships with CPython 3.5 *could* be totally incompatible and break all your code. +============================ +The Goals Of Argument Clinic +============================ + +Argument Clinic's primary goal +is to take over responsibility for all argument parsing code +inside CPython. This means that, when you convert a function +to work with Argument Clinic, that function should no longer +do any of its own argument parsing--the code generated by +Argument Clinic should be a "black box" to you, where CPython +calls in at the top, and your code gets called at the bottom, +with ``PyObject *args`` (and maybe ``PyObject *kwargs``) +magically converted into the C variables and types you need. + +In order for Argument Clinic to accomplish its primary goal, +it must be easy to use. Currently, working with CPython's +argument parsing library is a chore, requiring maintaining +redundant information in a surprising number of places. +When you use Argument Clinic, you don't have to repeat yourself. + +Obviously, no one would want to use Argument Clinic unless +it's solving a their problem without creating problems of +its own. +So Argument Clinic should generate correct code, and its +code should preferably be slower, and definitely should +not introduce a major speed regression. (Eventually Argument +Clinic should enable a major speedup--we should be able +to rewrite its code generator so it produces tailor-made +parsing code, rather than using the general-purpose functions +from the CPython code base, which would make for the fastest +argument parsing possible.) + +Additionally, Argument Clinic must be flexible enough to +work with any approach to argument parsing. Python has +some functions with some very strange parsing behaviors; +Argument Clinic's goal is to support all of them. + +Finally, the original motivation for Argument Clinic was +to provide introspection "signatures" for CPython builtins. +It used to be, the introspection query functions would throw +an exception if you passed in a builtin. With Argument +Clinic, that's a thing of the past! + +One idea you should keep in mind, as you work with +Argument Clinic: the more information you give it, the +better job it'll be able to do. +Argument Clinic is admittedly relatively simple right +now. But as it evolves it will get more sophisticated, +and it should be able to do many interesting and smart +things with all the information you give it. + + ======================== Basic Concepts And Usage ======================== @@ -84,7 +136,15 @@ Converting Your First Function ============================== The best way to get a sense of how Argument Clinic works is to -convert a function to work with it. Let's dive in! +convert a function to work with it. Here, then, are the bare +minimum steps you'd need to follow to convert a function to +work with Argument Clinic. Note that for code you plan to +check in to CPython, you really should take the conversion farther, +using some of the advanced concepts you'll see later on in +the document (like "return converters" and "self converters"). +But we'll keep it simple for this walkthrough so you can learn. + +Let's dive in! 0. Make sure you're working with a freshly updated checkout of the CPython trunk. @@ -1282,6 +1342,264 @@ available, the macro turns into nothing. Perfect! (This is the preferred approach for optional functions; in the future, Argument Clinic may generate the entire ``PyMethodDef`` structure.) + +Changing and redirecting Clinic's output +---------------------------------------- + +It can be inconvenient to have Clinic's output interspersed with +your conventional hand-edited C code. Luckily, Clinic is configurable: +you can buffer up its output for printing later (or earlier!), or write +its output to a separate file. You can also add a prefix or suffix to +every line of Clinic's generated output. + +While changing Clinic's output in this manner can be a boon to readability, +it may result in Clinic code using types before they are defined, or +your code attempting to use Clinic-generated code befire it is defined. +These problems can be easily solved by rearranging the declarations in your file, +or moving where Clinic's generated code goes. (This is why the default behavior +of Clinic is to output everything into the current block; while many people +consider this hampers readability, it will never require rearranging your +code to fix definition-before-use problems.) + +Let's start with defining some terminology: + +*field* + A field, in this context, is a subsection of Clinic's output. + For example, the ``#define`` for the ``PyMethodDef`` structure + is a field, called ``methoddef_define``. Clinic has seven + different fields it can output per function definition:: + + docstring_prototype + docstring_definition + methoddef_define + impl_prototype + parser_prototype + parser_definition + impl_definition + + All the names are of the form ``"_"``, + where ``""`` is the semantic object represented (the parsing function, + the impl function, the docstring, or the methoddef structure) and ``""`` + represents what kind of statement the field is. Field names that end in + ``"_prototype"`` + represent forward declarations of that thing, without the actual body/data + of the thing; field names that end in ``"_definition"`` represent the actual + definition of the thing, with the body/data of the thing. (``"methoddef"`` + is special, it's the only one that ends with ``"_define"``, representing that + it's a preprocessor #define.) + +*destination* + A destination is a place Clinic can write output to. There are + five built-in destinations: + + ``block`` + The default destination: printed in the output section of + the current Clinic block. + + ``buffer`` + A text buffer where you can save text for later. Text sent + here is appended to the end of any exsiting text. It's an + error to have any text left in the buffer when Clinic finishes + processing a file. + + ``file`` + A separate "clinic file" that will be created automatically by Clinic. + The filename chosen for the file is ``{basename}.clinic{extension}``, + where ``basename`` and ``extension`` were assigned the output + from ``os.path.splitext()`` run on the current file. (Example: + the ``file`` destination for ``_pickle.c`` would be written to + ``_pickle.clinic.c``.) + + **Important: When using a** ``file`` **destination, you** + *must check in* **the generated file!** + + ``two-pass`` + A buffer like ``buffer``. However, a two-pass buffer can only + be written once, and it prints out all text sent to it during + all of processing, even from Clinic blocks *after* the + + ``suppress`` + The text is suppressed--thrown away. + + +Clinic defines five new directives that let you reconfigure its output. + +The first new directive is ``dump``:: + + dump + +This dumps the current contents of the named destination into the output of +the current block, and empties it. This only works with ``buffer`` and +``two-pass`` destinations. + +The second new directive is ``output``. The most basic form of ``output`` +is like this:: + + output + +This tells Clinic to output *field* to *destination*. ``output`` also +supports a special meta-destination, called ``everything``, which tells +Clinic to output *all* fields to that *destination*. + +``output`` has a number of other functions:: + + output push + output pop + output preset + + +``output push`` and ``output pop`` allow you to push and pop +configurations on an internal configuration stack, so that you +can temporarily modify the output configuration, then easily restore +the previous configuration. Simply push before your change to save +the current configuration, then pop when you wish to restore the +previous configuration. + +``output preset`` sets Clinic's output to one of several built-in +preset configurations, as follows: + + ``original`` + Clinic's starting configuration. + + Suppress the ``parser_prototype`` + and ``docstring_prototype``, write everything else to ``block``. + + ``file`` + Designed to write everything to the "clinic file" that it can. + You then ``#include`` this file near the top of your file. + You may need to rearrange your file to make this work, though + usually this just means creating forward declarations for various + ``typedef`` and ``PyTypeObject`` definitions. + + Suppress the ``parser_prototype`` + and ``docstring_prototype``, write the ``impl_definition`` to + ``block``, and write everything else to ``file``. + + ``buffer`` + Save up all most of the output from Clinic, to be written into + your file near the end. For Python files implementing modules + or builtin types, it's recommended that you dump the buffer + just above the static structures for your module or + builtin type; these are normally very near the end. Using + ``buffer`` may require even more editing than ``file``, if + your file has static ``PyMethodDef`` arrays defined in the + middle of the file. + + Suppress the ``parser_prototype``, ``impl_prototype``, + and ``docstring_prototype``, write the ``impl_definition`` to + ``block``, and write everything else to ``file``. + + ``two-pass`` + Similar to the ``buffer`` preset, but writes forward declarations to + the ``two-pass`` buffer, and definitions to the ``buffer``. + This is similar to the ``buffer`` preset, but may require + less editing than ``buffer``. Dump the ``two-pass`` buffer + near the top of your file, and dump the ``buffer`` near + the end just like you would when using the ``buffer`` preset. + + Suppresses the ``impl_prototype``, write the ``impl_definition`` + to ``block``, write ``docstring_prototype``, ``methoddef_define``, + and ``parser_prototype`` to ``two-pass``, write everything else + to ``buffer``. + + ``partial-buffer`` + Similar to the ``buffer`` preset, but writes more things to ``block``, + only writing the really big chunks of generated code to ``buffer``. + This avoids the definition-before-use problem of ``buffer`` completely, + at the small cost of having slightly more stuff in the block's output. + Dump the ``buffer`` near the end, just like you would when using + the ``buffer`` preset. + + Suppresses the ``impl_prototype``, write the ``docstring_definition`` + and ``parser_defintion`` to ``buffer``, write everything else to ``block``. + +The third new directive is ``destination``:: + + destination [...] + +This performs an operation on the destination named ``name``. + +There are two defined subcommands: ``new`` and ``clear``. + +The ``new`` subcommand works like this:: + + destination new + +This creates a new destination with name ```` and type ````. + +There are five destination types:: + + ``suppress`` + Throws the text away. + + ``block`` + Writes the text to the current block. This is what Clinic + originally did. + + ``buffer`` + A simple text buffer, like the "buffer" builtin destination above. + + ``file`` + A text file. The file destination takes an extra argument, + a template to use for building the filename, like so: + + destination new + + The template can use three strings internally that will be replaced + by bits of the filename: + + {filename} + The full filename. + {basename} + Everything up to but not including the last '.'. + {extension} + The last '.' and everything after it. + + If there are no periods in the filename, {basename} and {filename} + are the same, and {extension} is empty. "{basename}{extension}" + is always exactly the same as "{filename}"." + + ``two-pass`` + A two-pass buffer, like the "two-pass" builtin destination above. + + +The ``clear`` subcommand works like this:: + + destination clear + +It removes all the accumulated text up to this point in the destination. +(I don't know what you'd need this for, but I thought maybe it'd be +useful while someone's experimenting.) + +The fourth new directive is ``set``:: + + set line_prefix "string" + set line_suffix "string" + +``set`` lets you set two internal variables in Clinic. +``line_prefix`` is a string that will be prepended to every line of Clinic's output; +``line_suffix`` is a string that will be appended to every line of Clinic's output. + +Both of these suport two format strings: + + ``{block comment start}`` + Turns into the string ``/*``, the start-comment text sequence for C files. + + ``{block comment end}`` + Turns into the string ``*/``, the end-comment text sequence for C files. + +The final new directive is one you shouldn't need to use directly, +called ``preserve``:: + + preserve + +This tells Clinic that the current contents of the output should be kept, unmodifed. +This is used internally by Clinic when dumping output into ``file`` files; wrapping +it in a Clinic block lets Clinic use its existing checksum functionality to ensure +the file was not modified by hand before it gets overwritten. + + +Using Argument Clinic in Python files ------------------------------------- It's actually possible to use Argument Clinic to preprocess Python files. From 047b449876ea117a864e6f368db6cf1d50e7d572 Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Sat, 18 Jan 2014 01:08:50 -0800 Subject: [PATCH 11/71] Doc improvements for Clinic howto "Goals" section. --- Doc/howto/clinic.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index a61508e3f..1f96e8245 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -44,16 +44,16 @@ redundant information in a surprising number of places. When you use Argument Clinic, you don't have to repeat yourself. Obviously, no one would want to use Argument Clinic unless -it's solving a their problem without creating problems of +it's solving their problem--and without creating new problems of its own. -So Argument Clinic should generate correct code, and its -code should preferably be slower, and definitely should -not introduce a major speed regression. (Eventually Argument -Clinic should enable a major speedup--we should be able -to rewrite its code generator so it produces tailor-made -parsing code, rather than using the general-purpose functions -from the CPython code base, which would make for the fastest -argument parsing possible.) +So it's paramount that Argument Clinic generate correct code. +It'd be nice if the code was faster, too, but at the very least +it should not introduce a major speed regression. (Eventually Argument +Clinic *should* make a major speedup possible--we could +rewrite its code generator to produce tailor-made argument +parsing code, rather than calling the general-purpose CPython +argument parsing library. That would make for the fastest +argument parsing possible!) Additionally, Argument Clinic must be flexible enough to work with any approach to argument parsing. Python has From e17b6ffb59151c8ff50d057154c28baeab660d98 Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Sat, 18 Jan 2014 23:50:21 -0800 Subject: [PATCH 12/71] Issue #20294: Argument Clinic now supports argument parsing for __new__ and __init__ functions. --- Doc/howto/clinic.rst | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 1f96e8245..5ee6c3da1 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -784,8 +784,8 @@ Argument Clinic converters. On the left is the legacy converter, on the right is the text you'd replace it with. ========= ================================================================================= -``'B'`` ``byte(bitwise=True)`` -``'b'`` ``byte`` +``'B'`` ``unsigned_char(bitwise=True)`` +``'b'`` ``unsigned_char`` ``'c'`` ``char`` ``'C'`` ``int(types='str')`` ``'d'`` ``double`` @@ -1275,6 +1275,25 @@ any arguments. You can still use a self converter, a return converter, and specify a ``type`` argument to the object converter for ``METH_O``. +tp_new and tp_init functions +---------------------------------------------- + +You can convert ``tp_new`` and ``tp_init`` +functions. Just name them ``__new__`` or +``__init__`` as appropriate. Notes: + +* The function name generated for ``__new__`` doesn't end in ``__new__`` + like it would by default. It's just the name of the class, converted + into a valid C identifier. + +* No ``PyMethodDef`` ``#define`` is generated for these functions. + +* ``__init__`` functions return ``int``, not ``PyObject *``. + +* Argument Clinic supports any signature for these functions, even though + the parsing function is required to always take ``args`` and ``kwargs`` + objects. + The #ifdef trick ---------------------------------------------- From 2a0b318bc0a3c13476b624f2d781bfd32fa56aaf Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Wed, 22 Jan 2014 05:49:11 -0800 Subject: [PATCH 13/71] Doc fixes for Argument Clinic. --- Doc/howto/clinic.rst | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 5ee6c3da1..20477db99 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -626,6 +626,15 @@ Optional groups are groups of arguments that must all be passed in together. They can be to the left or the right of the required arguments. They can *only* be used with positional-only parameters. +.. note:: Optional groups are *only* intended for use when converting + functions that make multiple calls to :c:func:`PyArg_ParseTuple`! + Functions that use *any* other approach for parsing arguments + should *almost never* be converted to Argument Clinic using + optional groups. Functions using optional groups currently + cannot have accurate sigantures in Python, because Python just + doesn't understand the concept. Please avoid using optional + groups wherever possible. + To specify an optional group, add a ``[`` on a line by itself before the parameters you wish to group together, and a ``]`` on a line by itself after these parameters. As an example, here's how ``curses.window.addch`` @@ -1278,9 +1287,8 @@ a ``type`` argument to the object converter for ``METH_O``. tp_new and tp_init functions ---------------------------------------------- -You can convert ``tp_new`` and ``tp_init`` -functions. Just name them ``__new__`` or -``__init__`` as appropriate. Notes: +You can convert ``tp_new`` and ``tp_init`` functions. Just name +them ``__new__`` or ``__init__`` as appropriate. Notes: * The function name generated for ``__new__`` doesn't end in ``__new__`` like it would by default. It's just the name of the class, converted @@ -1290,9 +1298,13 @@ functions. Just name them ``__new__`` or * ``__init__`` functions return ``int``, not ``PyObject *``. -* Argument Clinic supports any signature for these functions, even though - the parsing function is required to always take ``args`` and ``kwargs`` - objects. +* Use the docstring as the class docstring. + +* Although ``__new__`` and ``__init__`` functions must always + accept both the ``args`` and ``kwargs`` objects, when converting + you may specify any signature for these functions that you like. + (If your function doesn't support keywords, the parsing function + generated will throw an exception if it receives any.) The #ifdef trick ---------------------------------------------- From f48045ac8aa7e8360d2c3be66390d343212c4b7a Mon Sep 17 00:00:00 2001 From: Ezio Melotti Date: Sat, 25 Jan 2014 17:27:46 +0200 Subject: [PATCH 14/71] #20348: fix headers markup in Argument Clinic howto. Patch by Moritz Neeb. --- Doc/howto/clinic.rst | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 20477db99..bfae20e7c 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1,6 +1,6 @@ -====================== +********************** Argument Clinic How-To -====================== +********************** :author: Larry Hastings @@ -23,7 +23,6 @@ Argument Clinic How-To version of Argument Clinic that ships with CPython 3.5 *could* be totally incompatible and break all your code. -============================ The Goals Of Argument Clinic ============================ @@ -75,7 +74,6 @@ and it should be able to do many interesting and smart things with all the information you give it. -======================== Basic Concepts And Usage ======================== @@ -131,7 +129,6 @@ For the sake of clarity, here's the terminology we'll use with Argument Clinic: a block.) -============================== Converting Your First Function ============================== @@ -533,7 +530,6 @@ Let's dive in! Congratulations, you've ported your first function to work with Argument Clinic! -=============== Advanced Topics =============== From 5f12ffc65227df2f01db4a98c32442a5ee71c95b Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Sun, 26 Jan 2014 00:42:02 -0800 Subject: [PATCH 15/71] Documentation fixes, including fixing "suspicious" problems. --- Doc/howto/clinic.rst | 114 ++++++++++++++++++++++++------------------- 1 file changed, 64 insertions(+), 50 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index bfae20e7c..8fcc2e82f 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -149,7 +149,7 @@ Let's dive in! 1. Find a Python builtin that calls either :c:func:`PyArg_ParseTuple` or :c:func:`PyArg_ParseTupleAndKeywords`, and hasn't been converted to work with Argument Clinic yet. - For my example I'm using ``pickle.Pickler.dump()``. + For my example I'm using ``_pickle.Pickler.dump()``. 2. If the call to the ``PyArg_Parse`` function uses any of the following format units:: @@ -214,7 +214,7 @@ Let's dive in! Sample:: /*[clinic input] - pickle.Pickler.dump + _pickle.Pickler.dump Write a pickled representation of obj to the open file. [clinic start generated code]*/ @@ -227,22 +227,27 @@ Let's dive in! the top. (In our sample code we'll just show the two blocks next to each other.) + The name of the class and module should be the same as the one + seen by Python. Check the name defined in the :c:type:`PyModuleDef` + or :c:type:`PyTypeObject` as appropriate. + + When you declare a class, you must also specify two aspects of its type + in C: the type declaration you'd use for a pointer to an instance of + this class, and a pointer to the :c:type:`PyTypeObject` for this class. + Sample:: - /*[clinic input] - module pickle - class pickle.Pickler - [clinic start generated code]*/ + /*[clinic input] + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" + [clinic start generated code]*/ - /*[clinic input] - pickle.Pickler.dump + /*[clinic input] + _pickle.Pickler.dump - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ - The name of the class and module should be the same as the one - seen by Python. Check the name defined in the :c:type:`PyModuleDef` - or :c:type:`PyTypeObject` as appropriate. @@ -286,13 +291,13 @@ Let's dive in! Sample:: - /*[clinic input] - module pickle - class pickle.Pickler - [clinic start generated code]*/ + /*[clinic input] + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" + [clinic start generated code]*/ - /*[clinic input] - pickle.Pickler.dump + /*[clinic input] + _pickle.Pickler.dump obj: 'O' @@ -309,7 +314,7 @@ Let's dive in! itself before the first keyword-only argument, indented the same as the parameter lines. - (``pickle.Pickler.dump`` has neither, so our sample is unchanged.) + (``_pickle.Pickler.dump`` has neither, so our sample is unchanged.) 10. If the existing C function calls :c:func:`PyArg_ParseTuple` @@ -327,12 +332,12 @@ Let's dive in! Sample:: /*[clinic input] - module pickle - class pickle.Pickler + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" [clinic start generated code]*/ /*[clinic input] - pickle.Pickler.dump + _pickle.Pickler.dump obj: 'O' / @@ -354,12 +359,12 @@ Let's dive in! Sample:: /*[clinic input] - module pickle - class pickle.Pickler + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" [clinic start generated code]*/ /*[clinic input] - pickle.Pickler.dump + _pickle.Pickler.dump obj: 'O' The object to be pickled. @@ -373,13 +378,13 @@ Let's dive in! the file in your text editor to see:: /*[clinic input] - module pickle - class pickle.Pickler + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" [clinic start generated code]*/ /*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ /*[clinic input] - pickle.Pickler.dump + _pickle.Pickler.dump obj: 'O' The object to be pickled. @@ -388,12 +393,12 @@ Let's dive in! Write a pickled representation of obj to the open file. [clinic start generated code]*/ - PyDoc_STRVAR(pickle_Pickler_dump__doc__, + PyDoc_STRVAR(_pickle_Pickler_dump__doc__, "Write a pickled representation of obj to the open file.\n" "\n" ... static PyObject * - pickle_Pickler_dump_impl(PyObject *self, PyObject *obj) + _pickle_Pickler_dump_impl(PicklerObject *self, PyObject *obj) /*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ Obviously, if Argument Clinic didn't produce any output, it's because @@ -428,8 +433,8 @@ Let's dive in! macro defining the appropriate static :c:type:`PyMethodDef` structure for this builtin:: - #define _PICKLE_PICKLER_DUMP_METHODDEF \ - {"dump", (PyCFunction)_pickle_Pickler_dump, METH_O, _pickle_Pickler_dump__doc__}, + #define __PICKLE_PICKLER_DUMP_METHODDEF \ + {"dump", (PyCFunction)__pickle_Pickler_dump, METH_O, __pickle_Pickler_dump__doc__}, This static structure should be *exactly* the same as the existing static :c:type:`PyMethodDef` structure for this builtin. @@ -463,13 +468,13 @@ Let's dive in! Sample:: /*[clinic input] - module pickle - class pickle.Pickler + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" [clinic start generated code]*/ /*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ /*[clinic input] - pickle.Pickler.dump + _pickle.Pickler.dump obj: 'O' The object to be pickled. @@ -478,12 +483,12 @@ Let's dive in! Write a pickled representation of obj to the open file. [clinic start generated code]*/ - PyDoc_STRVAR(pickle_Pickler_dump__doc__, + PyDoc_STRVAR(__pickle_Pickler_dump__doc__, "Write a pickled representation of obj to the open file.\n" "\n" ... static PyObject * - pickle_Pickler_dump_impl(PyObject *self, PyObject *obj) + _pickle_Pickler_dump_impl(PicklerObject *self, PyObject *obj) /*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ { /* Check whether the Pickler was initialized correctly (issue3664). @@ -515,8 +520,8 @@ Let's dive in! Sample:: static struct PyMethodDef Pickler_methods[] = { - _PICKLE_PICKLER_DUMP_METHODDEF - _PICKLE_PICKLER_CLEAR_MEMO_METHODDEF + __PICKLE_PICKLER_DUMP_METHODDEF + __PICKLE_PICKLER_CLEAR_MEMO_METHODDEF {NULL, NULL} /* sentinel */ }; @@ -1106,15 +1111,16 @@ Using a "self converter" ------------------------ Argument Clinic automatically adds a "self" parameter for you -using a default converter. However, you can override +using a default converter. It automatically sets the ``type`` +of this parameter to the "pointer to an instance" you specified +when you declared the type. However, you can override Argument Clinic's converter and specify one yourself. Just add your own ``self`` parameter as the first parameter in a block, and ensure that its converter is an instance of ``self_converter`` or a subclass thereof. -What's the point? This lets you automatically cast ``self`` -from ``PyObject *`` to a custom type, just like ``object()`` -does with its ``type`` parameter. +What's the point? This lets you override the type of ``self``, +or give it a different default name. How do you specify the custom type you want to cast ``self`` to? If you only have one or two functions with the same type for ``self``, @@ -1502,6 +1508,8 @@ preset configurations, as follows: and ``docstring_prototype``, write the ``impl_definition`` to ``block``, and write everything else to ``file``. + The default filename is ``"{dirname}/clinic/{basename}.h"``. + ``buffer`` Save up all most of the output from Clinic, to be written into your file near the end. For Python files implementing modules @@ -1554,7 +1562,7 @@ The ``new`` subcommand works like this:: This creates a new destination with name ```` and type ````. -There are five destination types:: +There are five destination types: ``suppress`` Throws the text away. @@ -1575,12 +1583,18 @@ There are five destination types:: The template can use three strings internally that will be replaced by bits of the filename: - {filename} - The full filename. + {path} + The full path to the file, including directory and full filename. + {dirname} + The name of the directory the file is in. {basename} - Everything up to but not including the last '.'. - {extension} - The last '.' and everything after it. + Just the name of the file, not including the directory. + {basename_root} + Basename with the extension clipped off + (everything up to but not including the last '.'). + {basename_extension} + The last '.' and everything after it. If the basename + does not contain a period, this will be the empty string. If there are no periods in the filename, {basename} and {filename} are the same, and {extension} is empty. "{basename}{extension}" From d270e9824ca5bacac5b44306d6ec8da44417421d Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Fri, 31 Jan 2014 22:03:12 -0800 Subject: [PATCH 16/71] #Issue 20456: Several improvements and bugfixes for Argument Clinic, including correctly generating code for Clinic blocks inside C preprocessor conditional blocks. --- Doc/howto/clinic.rst | 172 +++++++++++++++++++++++++------------------ 1 file changed, 100 insertions(+), 72 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 8fcc2e82f..750ddbe07 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -561,8 +561,8 @@ in ``Lib/inspect.py``. to allow full expressions like ``CONSTANT - 1``.) -Renaming the C functions generated by Argument Clinic ------------------------------------------------------ +Renaming the C functions and variables generated by Argument Clinic +------------------------------------------------------------------- Argument Clinic automatically names the functions it generates for you. Occasionally this may cause a problem, if the generated name collides with @@ -584,6 +584,25 @@ The base function would now be named ``pickler_dumper()``, and the impl function would now be named ``pickler_dumper_impl()``. +Similarly, you may have a problem where you want to give a parameter +a specific Python name, but that name may be inconvenient in C. Argument +Clinic allows you to give a parameter different names in Python and in C, +using the same ``"as"`` syntax:: + + /*[clinic input] + pickle.Pickler.dump + + obj: object + file as file_obj: object + protocol: object = NULL + * + fix_imports: bool = True + +Here, the name used in Python (in the signature and the ``keywords`` +array) would be ``file``, but the C variable would be named ``file_obj``. + +You can use this to rename the ``self`` parameter too! + Converting functions using PyArg_UnpackTuple -------------------------------------------- @@ -1308,74 +1327,6 @@ them ``__new__`` or ``__init__`` as appropriate. Notes: (If your function doesn't support keywords, the parsing function generated will throw an exception if it receives any.) -The #ifdef trick ----------------------------------------------- - -If you're converting a function that isn't available on all platforms, -there's a trick you can use to make life a little easier. The existing -code probably looks like this:: - - #ifdef HAVE_FUNCTIONNAME - static module_functionname(...) - { - ... - } - #endif /* HAVE_FUNCTIONNAME */ - -And then in the ``PyMethodDef`` structure at the bottom you'll have:: - - #ifdef HAVE_FUNCTIONNAME - {'functionname', ... }, - #endif /* HAVE_FUNCTIONNAME */ - -In this scenario, you should change the code to look like the following:: - - #ifdef HAVE_FUNCTIONNAME - /*[clinic input] - module.functionname - ... - [clinic start generated code]*/ - static module_functionname(...) - { - ... - } - #endif /* HAVE_FUNCTIONNAME */ - -Run Argument Clinic on the code in this state, then refresh the file in -your editor. Now you'll have the generated code, including the #define -for the ``PyMethodDef``, like so:: - - #ifdef HAVE_FUNCTIONNAME - /*[clinic input] - ... - [clinic start generated code]*/ - ... - #define MODULE_FUNCTIONNAME \ - {'functionname', ... }, - ... - /*[clinic end generated code: checksum=...]*/ - static module_functionname(...) - { - ... - } - #endif /* HAVE_FUNCTIONNAME */ - -Change the #endif at the bottom as follows:: - - #else - #define MODULE_FUNCTIONNAME - #endif /* HAVE_FUNCTIONNAME */ - -Now you can remove the #ifdefs around the ``PyMethodDef`` structure -at the end, and replace those three lines with ``MODULE_FUNCTIONNAME``. -If the function is available, the macro turns into the ``PyMethodDef`` -static value, including the trailing comma; if the function isn't -available, the macro turns into nothing. Perfect! - -(This is the preferred approach for optional functions; in the future, -Argument Clinic may generate the entire ``PyMethodDef`` structure.) - - Changing and redirecting Clinic's output ---------------------------------------- @@ -1491,8 +1442,9 @@ previous configuration. ``output preset`` sets Clinic's output to one of several built-in preset configurations, as follows: - ``original`` - Clinic's starting configuration. + ``block`` + Clinic's original starting configuration. Writes everything + immediately after the input block. Suppress the ``parser_prototype`` and ``docstring_prototype``, write everything else to ``block``. @@ -1640,6 +1592,82 @@ it in a Clinic block lets Clinic use its existing checksum functionality to ensu the file was not modified by hand before it gets overwritten. +The #ifdef trick +---------------------------------------------- + +If you're converting a function that isn't available on all platforms, +there's a trick you can use to make life a little easier. The existing +code probably looks like this:: + + #ifdef HAVE_FUNCTIONNAME + static module_functionname(...) + { + ... + } + #endif /* HAVE_FUNCTIONNAME */ + +And then in the ``PyMethodDef`` structure at the bottom the existing code +will have:: + + #ifdef HAVE_FUNCTIONNAME + {'functionname', ... }, + #endif /* HAVE_FUNCTIONNAME */ + +In this scenario, you should enclose the body of your impl function inside the ``#ifdef``, +like so:: + + #ifdef HAVE_FUNCTIONNAME + /*[clinic input] + module.functionname + ... + [clinic start generated code]*/ + static module_functionname(...) + { + ... + } + #endif /* HAVE_FUNCTIONNAME */ + +Then, remove those three lines from the ``PyMethodDef`` structure, +replacing them with the macro Argument Clinic generated:: + + MODULE_FUNCTIONNAME_METHODDEF + +(You can find the real name for this macro inside the generated code. +Or you can calculate it yourself: it's the name of your function as defined +on the first line of your block, but with periods changed to underscores, +uppercased, and ``"_METHODDEF"`` added to the end.) + +Perhaps you're wondering: what if ``HAVE_FUNCTIONNAME`` isn't defined? +The ``MODULE_FUNCTIONNAME_METHODDEF`` macro won't be defined either! + +Here's where Argument Clinic gets very clever. It actually detects that the +Argument Clinic block might be deactivated by the ``#ifdef``. When that +happens, it generates a little extra code that looks like this:: + + #ifndef MODULE_FUNCTIONNAME_METHODDEF + #define MODULE_FUNCTIONNAME_METHODDEF + #endif /* !defined(MODULE_FUNCTIONNAME_METHODDEF) */ + +That means the macro always works. If the function is defined, this turns +into the correct structure, including the trailing comma. If the function is +undefined, this turns into nothing. + +However, this causes one ticklish problem: where should Argument Clinic put this +extra code when using the "block" output preset? It can't go in the output block, +because that could be decativated by the ``#ifdef``. (That's the whole point!) + +In this situation, Argument Clinic writes the extra code to the "buffer" destination. +This may mean that you get a complaint from Argument Clinic:: + + Warning in file "Modules/posixmodule.c" on line 12357: + Destination buffer 'buffer' not empty at end of file, emptying. + +When this happens, just open your file, find the ``dump buffer`` block that +Argument Clinic added to your file (it'll be at the very bottom), then +move it above the ``PyMethodDef`` structure where that macro is used. + + + Using Argument Clinic in Python files ------------------------------------- From a0be8a338e03a24cb9424b99410c052796700ed3 Mon Sep 17 00:00:00 2001 From: Donald Stufft Date: Tue, 20 May 2014 12:58:38 -0400 Subject: [PATCH 17/71] Fix Issue #21528 - Fix documentation typos --- Doc/howto/clinic.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 750ddbe07..ca8e1cb87 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -886,7 +886,7 @@ Argument Clinic generates code that does it for you (in the parsing function). Advanced converters ------------------- -Remeber those format units you skipped for your first +Remember those format units you skipped for your first time because they were advanced? Here's how to handle those too. The trick is, all those format units take arguments--either @@ -1020,12 +1020,12 @@ any of the default arguments you can omit the parentheses. the ``"as"`` should come before the return converter.) There's one additional complication when using return converters: how do you -indicate an error has occured? Normally, a function returns a valid (non-``NULL``) +indicate an error has occurred? Normally, a function returns a valid (non-``NULL``) pointer for success, and ``NULL`` for failure. But if you use an integer return converter, all integers are valid. How can Argument Clinic detect an error? Its solution: each return converter implicitly looks for a special value that indicates an error. If you return that value, and an error has been set (``PyErr_Occurred()`` returns a true -value), then the generated code will propogate the error. Otherwise it will +value), then the generated code will propagate the error. Otherwise it will encode the value you return like normal. Currently Argument Clinic supports only a few return converters:: @@ -1573,7 +1573,7 @@ The fourth new directive is ``set``:: ``line_prefix`` is a string that will be prepended to every line of Clinic's output; ``line_suffix`` is a string that will be appended to every line of Clinic's output. -Both of these suport two format strings: +Both of these support two format strings: ``{block comment start}`` Turns into the string ``/*``, the start-comment text sequence for C files. From 64b53807a986b56178f3d9ae94b1c3c8e5874b70 Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Thu, 7 May 2015 23:30:09 -0700 Subject: [PATCH 18/71] Issue #24000: Improved Argument Clinic's mapping of converters to legacy "format units". Updated the documentation to match. --- Doc/howto/clinic.rst | 70 +++++++++++++++++++++----------------------- 1 file changed, 33 insertions(+), 37 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index ca8e1cb87..cb208d50b 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -758,6 +758,14 @@ All Argument Clinic converters accept the following arguments: In addition, some converters accept additional arguments. Here is a list of these arguments, along with their meanings: + ``accept`` + A set of Python types (and possibly pseudo-types); + this restricts the allowable Python argument to values of these types. + (This is not a general-purpose facility; as a rule it only supports + specific lists of types as shown in the legacy converter table.) + + To accept ``None``, add ``NoneType`` to this set. + ``bitwise`` Only supported for unsigned integers. The native integer value of this Python argument will be written to the parameter without any range checking, @@ -772,39 +780,27 @@ of these arguments, along with their meanings: Only supported for strings. Specifies the encoding to use when converting this string from a Python str (Unicode) value into a C ``char *`` value. - ``length`` - Only supported for strings. If true, requests that the length of the - string be passed in to the impl function, just after the string parameter, - in a parameter named ``_length``. - - ``nullable`` - Only supported for strings. If true, this parameter may also be set to - ``None``, in which case the C parameter will be set to ``NULL``. ``subclass_of`` Only supported for the ``object`` converter. Requires that the Python value be a subclass of a Python type, as expressed in C. - ``types`` - Only supported for the ``object`` (and ``self``) converter. Specifies + ``type`` + Only supported for the ``object`` and ``self`` converters. Specifies the C type that will be used to declare the variable. Default value is ``"PyObject *"``. - ``types`` - A string containing a list of Python types (and possibly pseudo-types); - this restricts the allowable Python argument to values of these types. - (This is not a general-purpose facility; as a rule it only supports - specific lists of types as shown in the legacy converter table.) - ``zeroes`` Only supported for strings. If true, embedded NUL bytes (``'\\0'``) are - permitted inside the value. + permitted inside the value. The length of the string will be passed in + to the impl function, just after the string parameter, as a parameter named + ``_length``. Please note, not every possible combination of arguments will work. -Often these arguments are implemented internally by specific ``PyArg_ParseTuple`` +Usually these arguments are implemented by specific ``PyArg_ParseTuple`` *format units*, with specific behavior. For example, currently you cannot -call ``str`` and pass in ``zeroes=True`` without also specifying an ``encoding``; -although it's perfectly reasonable to think this would work, these semantics don't +call ``unsigned_short`` without also specifying ``bitwise=True``. +Although it's perfectly reasonable to think this would work, these semantics don't map to any existing format unit. So Argument Clinic doesn't support it. (Or, at least, not yet.) @@ -816,13 +812,13 @@ on the right is the text you'd replace it with. ``'B'`` ``unsigned_char(bitwise=True)`` ``'b'`` ``unsigned_char`` ``'c'`` ``char`` -``'C'`` ``int(types='str')`` +``'C'`` ``int(accept={str})`` ``'d'`` ``double`` ``'D'`` ``Py_complex`` -``'es#'`` ``str(encoding='name_of_encoding', length=True, zeroes=True)`` ``'es'`` ``str(encoding='name_of_encoding')`` -``'et#'`` ``str(encoding='name_of_encoding', types='bytes bytearray str', length=True)`` -``'et'`` ``str(encoding='name_of_encoding', types='bytes bytearray str')`` +``'es#'`` ``str(encoding='name_of_encoding', zeroes=True)`` +``'et'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str})`` +``'et#'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str}, zeroes=True)`` ``'f'`` ``float`` ``'h'`` ``short`` ``'H'`` ``unsigned_short(bitwise=True)`` @@ -832,27 +828,27 @@ on the right is the text you'd replace it with. ``'K'`` ``unsigned_PY_LONG_LONG(bitwise=True)`` ``'L'`` ``PY_LONG_LONG`` ``'n'`` ``Py_ssize_t`` +``'O'`` ``object`` ``'O!'`` ``object(subclass_of='&PySomething_Type')`` ``'O&'`` ``object(converter='name_of_c_function')`` -``'O'`` ``object`` ``'p'`` ``bool`` -``'s#'`` ``str(length=True)`` ``'S'`` ``PyBytesObject`` ``'s'`` ``str`` -``'s*'`` ``Py_buffer(types='str bytes bytearray buffer')`` -``'u#'`` ``Py_UNICODE(length=True)`` -``'u'`` ``Py_UNICODE`` +``'s#'`` ``str(zeroes=True)`` +``'s*'`` ``Py_buffer(accept={buffer, str})`` ``'U'`` ``unicode`` -``'w*'`` ``Py_buffer(types='bytearray rwbuffer')`` -``'y#'`` ``str(types='bytes', length=True)`` +``'u'`` ``Py_UNICODE`` +``'u#'`` ``Py_UNICODE(zeroes=True)`` +``'w*'`` ``Py_buffer(accept={rwbuffer})`` ``'Y'`` ``PyByteArrayObject`` -``'y'`` ``str(types='bytes')`` +``'y'`` ``str(accept={bytes})`` +``'y#'`` ``str(accept={robuffer}, zeroes=True)`` ``'y*'`` ``Py_buffer`` -``'Z#'`` ``Py_UNICODE(nullable=True, length=True)`` -``'z#'`` ``str(nullable=True, length=True)`` -``'Z'`` ``Py_UNICODE(nullable=True)`` -``'z'`` ``str(nullable=True)`` -``'z*'`` ``Py_buffer(types='str bytes bytearray buffer', nullable=True)`` +``'Z'`` ``Py_UNICODE(accept={str, NoneType})`` +``'Z#'`` ``Py_UNICODE(accept={str, NoneType}, zeroes=True)`` +``'z'`` ``str(accept={str, NoneType})`` +``'z#'`` ``str(accept={str, NoneType}, zeroes=True)`` +``'z*'`` ``Py_buffer(accept={buffer, str, NoneType})`` ========= ================================================================================= As an example, here's our sample ``pickle.Pickler.dump`` using the proper From 66db86b8158616f3610d370281be0d57be864185 Mon Sep 17 00:00:00 2001 From: Tal Einat Date: Sat, 16 May 2015 14:12:15 +0300 Subject: [PATCH 19/71] Argument Clinic: added missing bit of info in howto --- Doc/howto/clinic.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index cb208d50b..4ff475fb3 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -826,6 +826,7 @@ on the right is the text you'd replace it with. ``'I'`` ``unsigned_int(bitwise=True)`` ``'k'`` ``unsigned_long(bitwise=True)`` ``'K'`` ``unsigned_PY_LONG_LONG(bitwise=True)`` +``'l'`` ``long`` ``'L'`` ``PY_LONG_LONG`` ``'n'`` ``Py_ssize_t`` ``'O'`` ``object`` From 051e5a29a9c2980e9990aef4caab8e6cc0d76d44 Mon Sep 17 00:00:00 2001 From: Berker Peksag Date: Tue, 19 May 2015 01:36:55 +0300 Subject: [PATCH 20/71] =?UTF-8?q?Issue=20#24232:=20Fix=20typos.=20Patch=20?= =?UTF-8?q?by=20Ville=20Skytt=C3=A4.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Doc/howto/clinic.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 750ddbe07..0b7cadad4 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1498,7 +1498,7 @@ preset configurations, as follows: the ``buffer`` preset. Suppresses the ``impl_prototype``, write the ``docstring_definition`` - and ``parser_defintion`` to ``buffer``, write everything else to ``block``. + and ``parser_definition`` to ``buffer``, write everything else to ``block``. The third new directive is ``destination``:: From aff9abe34cc50fe71da019252f00f070b892133c Mon Sep 17 00:00:00 2001 From: Senthil Kumaran Date: Sun, 14 Jun 2015 17:35:37 -0700 Subject: [PATCH 21/71] Back porting changeset db302b88fdb6 to 3.4 branch, which fixed multiple documentation typos. Related Issues: #issue21528 #issue24453 --- Doc/howto/clinic.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 0b7cadad4..e362631db 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -886,7 +886,7 @@ Argument Clinic generates code that does it for you (in the parsing function). Advanced converters ------------------- -Remeber those format units you skipped for your first +Remember those format units you skipped for your first time because they were advanced? Here's how to handle those too. The trick is, all those format units take arguments--either @@ -1020,12 +1020,12 @@ any of the default arguments you can omit the parentheses. the ``"as"`` should come before the return converter.) There's one additional complication when using return converters: how do you -indicate an error has occured? Normally, a function returns a valid (non-``NULL``) +indicate an error has occurred? Normally, a function returns a valid (non-``NULL``) pointer for success, and ``NULL`` for failure. But if you use an integer return converter, all integers are valid. How can Argument Clinic detect an error? Its solution: each return converter implicitly looks for a special value that indicates an error. If you return that value, and an error has been set (``PyErr_Occurred()`` returns a true -value), then the generated code will propogate the error. Otherwise it will +value), then the generated code will propagate the error. Otherwise it will encode the value you return like normal. Currently Argument Clinic supports only a few return converters:: @@ -1573,7 +1573,7 @@ The fourth new directive is ``set``:: ``line_prefix`` is a string that will be prepended to every line of Clinic's output; ``line_suffix`` is a string that will be appended to every line of Clinic's output. -Both of these suport two format strings: +Both of these support two format strings: ``{block comment start}`` Turns into the string ``/*``, the start-comment text sequence for C files. From cdc1d5b389232ff1890203f3a912833e1f734f92 Mon Sep 17 00:00:00 2001 From: Martin Panter Date: Fri, 20 Nov 2015 08:13:35 +0000 Subject: [PATCH 22/71] Issue #25626: Change zlib to accept Py_ssize_t and cap to UINT_MAX MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The underlying zlib library stores sizes in “unsigned int”. The corresponding Python parameters are all sizes of buffers filled in by zlib, so it is okay to reduce higher values to the UINT_MAX internal cap. OverflowError is still raised for sizes that do not fit in Py_ssize_t. Sizes are now limited to Py_ssize_t rather than unsigned long, because Python byte strings cannot be larger than Py_ssize_t. Previously this could result in a SystemError on 32-bit platforms. This resolves a regression in the gzip module when reading more than UINT_MAX or LONG_MAX bytes in one call, introduced by revision 62723172412c. --- Doc/howto/clinic.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 7524c4ad9..b04edea18 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1249,18 +1249,18 @@ Here's the simplest example of a custom converter, from ``Modules/zlibmodule.c`` /*[python input] - class uint_converter(CConverter): + class capped_uint_converter(CConverter): type = 'unsigned int' - converter = 'uint_converter' + converter = 'capped_uint_converter' [python start generated code]*/ - /*[python end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ + /*[python end generated code: output=da39a3ee5e6b4b0d input=35521e4e733823c7]*/ -This block adds a converter to Argument Clinic named ``uint``. Parameters -declared as ``uint`` will be declared as type ``unsigned int``, and will -be parsed by the ``'O&'`` format unit, which will call the ``uint_converter`` -converter function. -``uint`` variables automatically support default values. +This block adds a converter to Argument Clinic named ``capped_uint``. Parameters +declared as ``capped_uint`` will be declared as type ``unsigned int``, and will +be parsed by the ``'O&'`` format unit, which will call the +``capped_uint_converter`` converter function. ``capped_uint`` variables +automatically support default values. More sophisticated custom converters can insert custom C code to handle initialization and cleanup. From 5cbbd2f881655b3143aed6ec55403700078505a0 Mon Sep 17 00:00:00 2001 From: Martin Panter Date: Sat, 23 Jul 2016 03:02:07 +0000 Subject: [PATCH 23/71] =?UTF-8?q?Issue=20#27130:=20Fix=20handling=20of=20b?= =?UTF-8?q?uffers=20exceeding=20UINT=5FMAX=20in=20=E2=80=9Czlib=E2=80=9D?= =?UTF-8?q?=20module?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Patch by Xiang Zhang. --- Doc/howto/clinic.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index b04edea18..7f275a465 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1249,17 +1249,17 @@ Here's the simplest example of a custom converter, from ``Modules/zlibmodule.c`` /*[python input] - class capped_uint_converter(CConverter): - type = 'unsigned int' - converter = 'capped_uint_converter' + class ssize_t_converter(CConverter): + type = 'Py_ssize_t' + converter = 'ssize_t_converter' [python start generated code]*/ /*[python end generated code: output=da39a3ee5e6b4b0d input=35521e4e733823c7]*/ -This block adds a converter to Argument Clinic named ``capped_uint``. Parameters -declared as ``capped_uint`` will be declared as type ``unsigned int``, and will +This block adds a converter to Argument Clinic named ``ssize_t``. Parameters +declared as ``ssize_t`` will be declared as type ``Py_ssize_t``, and will be parsed by the ``'O&'`` format unit, which will call the -``capped_uint_converter`` converter function. ``capped_uint`` variables +``ssize_t_converter`` converter function. ``ssize_t`` variables automatically support default values. More sophisticated custom converters can insert custom C code to From 5a543c94e9eb8fd3f42e959e55b8acc0951fa1e8 Mon Sep 17 00:00:00 2001 From: Martin Panter Date: Thu, 28 Jul 2016 01:11:04 +0000 Subject: [PATCH 24/71] Issue #27626: Spelling fixes in docs, comments and internal names MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Based on patch by Ville Skyttä. --- Doc/howto/clinic.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 7f275a465..82eae01e7 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1583,7 +1583,7 @@ called ``preserve``:: preserve -This tells Clinic that the current contents of the output should be kept, unmodifed. +This tells Clinic that the current contents of the output should be kept, unmodified. This is used internally by Clinic when dumping output into ``file`` files; wrapping it in a Clinic block lets Clinic use its existing checksum functionality to ensure the file was not modified by hand before it gets overwritten. From a248952049701fc64bd028d72aec168c751d7b70 Mon Sep 17 00:00:00 2001 From: Martin Panter Date: Tue, 26 Jul 2016 11:18:21 +0200 Subject: [PATCH 25/71] Issue #26462: Doc: reduce literal_block warnings, fix syntax highlighting. Patch by Julien Palard. --- Doc/howto/clinic.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 82eae01e7..5a69ca8ab 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -152,7 +152,9 @@ Let's dive in! For my example I'm using ``_pickle.Pickler.dump()``. 2. If the call to the ``PyArg_Parse`` function uses any of the - following format units:: + following format units: + + .. code-block:: none O& O! From 1b726cb61e40909c4dc4ecc4575a27edc86195f5 Mon Sep 17 00:00:00 2001 From: Martin Panter Date: Fri, 12 Aug 2016 12:02:03 +0000 Subject: [PATCH 26/71] Issue #27745: Fix some typos in Argument Clinic howto, by Lele Gaifax --- Doc/howto/clinic.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 5a69ca8ab..caa497521 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -653,7 +653,7 @@ can *only* be used with positional-only parameters. Functions that use *any* other approach for parsing arguments should *almost never* be converted to Argument Clinic using optional groups. Functions using optional groups currently - cannot have accurate sigantures in Python, because Python just + cannot have accurate signatures in Python, because Python just doesn't understand the concept. Please avoid using optional groups wherever possible. @@ -1337,7 +1337,7 @@ every line of Clinic's generated output. While changing Clinic's output in this manner can be a boon to readability, it may result in Clinic code using types before they are defined, or -your code attempting to use Clinic-generated code befire it is defined. +your code attempting to use Clinic-generated code before it is defined. These problems can be easily solved by rearranging the declarations in your file, or moving where Clinic's generated code goes. (This is why the default behavior of Clinic is to output everything into the current block; while many people @@ -1381,7 +1381,7 @@ Let's start with defining some terminology: ``buffer`` A text buffer where you can save text for later. Text sent - here is appended to the end of any exsiting text. It's an + here is appended to the end of any existing text. It's an error to have any text left in the buffer when Clinic finishes processing a file. @@ -1653,7 +1653,7 @@ undefined, this turns into nothing. However, this causes one ticklish problem: where should Argument Clinic put this extra code when using the "block" output preset? It can't go in the output block, -because that could be decativated by the ``#ifdef``. (That's the whole point!) +because that could be deactivated by the ``#ifdef``. (That's the whole point!) In this situation, Argument Clinic writes the extra code to the "buffer" destination. This may mean that you get a complaint from Argument Clinic:: From 63abe55742c3c09b8e58b6c3b79ec7701932a777 Mon Sep 17 00:00:00 2001 From: Benjamin Peterson Date: Thu, 8 Sep 2016 09:29:11 -0700 Subject: [PATCH 27/71] clinic: PY_LONG_LONG -> long long --- Doc/howto/clinic.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index caa497521..f43588408 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -827,9 +827,9 @@ on the right is the text you'd replace it with. ``'i'`` ``int`` ``'I'`` ``unsigned_int(bitwise=True)`` ``'k'`` ``unsigned_long(bitwise=True)`` -``'K'`` ``unsigned_PY_LONG_LONG(bitwise=True)`` +``'K'`` ``unsigned_long_long(bitwise=True)`` ``'l'`` ``long`` -``'L'`` ``PY_LONG_LONG`` +``'L'`` ``long long`` ``'n'`` ``Py_ssize_t`` ``'O'`` ``object`` ``'O!'`` ``object(subclass_of='&PySomething_Type')`` From 9f1870c93ffbe810da48449580da089606e8dc72 Mon Sep 17 00:00:00 2001 From: Larry Hastings Date: Sun, 20 Nov 2016 11:30:36 -0800 Subject: [PATCH 28/71] Issue 28753: Argument Clinic howto docfix, courtesy Julien Palard. --- Doc/howto/clinic.rst | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index f43588408..0316901c9 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -375,15 +375,12 @@ Let's dive in! Write a pickled representation of obj to the open file. [clinic start generated code]*/ -12. Save and close the file, then run ``Tools/clinic/clinic.py`` on it. - With luck everything worked and your block now has output! Reopen - the file in your text editor to see:: +12. Save and close the file, then run ``Tools/clinic/clinic.py`` on + it. With luck everything worked--your block now has output, and + a ``.c.h`` file has been generated! Reopen the file in your + text editor to see: - /*[clinic input] - module _pickle - class _pickle.Pickler "PicklerObject *" "&Pickler_Type" - [clinic start generated code]*/ - /*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ + .. code-block:: c /*[clinic input] _pickle.Pickler.dump @@ -395,18 +392,22 @@ Let's dive in! Write a pickled representation of obj to the open file. [clinic start generated code]*/ - PyDoc_STRVAR(_pickle_Pickler_dump__doc__, - "Write a pickled representation of obj to the open file.\n" - "\n" - ... static PyObject * - _pickle_Pickler_dump_impl(PicklerObject *self, PyObject *obj) - /*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ + _pickle_Pickler_dump(PicklerObject *self, PyObject *obj) + /*[clinic end generated code: output=87ecad1261e02ac7 input=552eb1c0f52260d9]*/ Obviously, if Argument Clinic didn't produce any output, it's because it found an error in your input. Keep fixing your errors and retrying until Argument Clinic processes your file without complaint. + For readability, most of the glue code has been generated to a ``.c.h`` + file. You'll need to include that in your original ``.c`` file, + typically right after the clinic module block: + + .. code-block:: c + + #include "clinic/_pickle.c.h" + 13. Double-check that the argument-parsing code Argument Clinic generated looks basically the same as the existing code. From 3af3844cbf2714748030ac31401acfc57c7a4cce Mon Sep 17 00:00:00 2001 From: Martin Panter Date: Mon, 21 Nov 2016 00:15:20 +0000 Subject: [PATCH 29/71] Change double hyphens (en dashes) to em (longer) dashes --- Doc/howto/clinic.rst | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index caa497521..d4fd45ce8 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -30,7 +30,7 @@ Argument Clinic's primary goal is to take over responsibility for all argument parsing code inside CPython. This means that, when you convert a function to work with Argument Clinic, that function should no longer -do any of its own argument parsing--the code generated by +do any of its own argument parsing—the code generated by Argument Clinic should be a "black box" to you, where CPython calls in at the top, and your code gets called at the bottom, with ``PyObject *args`` (and maybe ``PyObject *kwargs``) @@ -43,12 +43,12 @@ redundant information in a surprising number of places. When you use Argument Clinic, you don't have to repeat yourself. Obviously, no one would want to use Argument Clinic unless -it's solving their problem--and without creating new problems of +it's solving their problem—and without creating new problems of its own. So it's paramount that Argument Clinic generate correct code. It'd be nice if the code was faster, too, but at the very least it should not introduce a major speed regression. (Eventually Argument -Clinic *should* make a major speedup possible--we could +Clinic *should* make a major speedup possible—we could rewrite its code generator to produce tailor-made argument parsing code, rather than calling the general-purpose CPython argument parsing library. That would make for the fastest @@ -113,7 +113,7 @@ line. However, if the input hasn't changed, the output won't change either. You should never modify the output portion of an Argument Clinic block. Instead, change the input until it produces the output you want. (That's the purpose of the -checksum--to detect if someone changed the output, as these edits would be lost +checksum—to detect if someone changed the output, as these edits would be lost the next time Argument Clinic writes out fresh output.) For the sake of clarity, here's the terminology we'll use with Argument Clinic: @@ -166,7 +166,7 @@ Let's dive in! or if it has multiple calls to :c:func:`PyArg_ParseTuple`, you should choose a different function. Argument Clinic *does* support all of these scenarios. But these are advanced - topics--let's do something simpler for your first function. + topics—let's do something simpler for your first function. Also, if the function has multiple calls to :c:func:`PyArg_ParseTuple` or :c:func:`PyArg_ParseTupleAndKeywords` where it supports different @@ -188,7 +188,7 @@ Let's dive in! If the old docstring had a first line that looked like a function signature, throw that line away. (The docstring doesn't need it - anymore--when you use ``help()`` on your builtin in the future, + anymore—when you use ``help()`` on your builtin in the future, the first line will be built automatically based on the function's signature.) @@ -209,7 +209,7 @@ Let's dive in! 6. Above the docstring, enter the name of the function, followed by a blank line. This should be the Python name of the function, and should be the full dotted path - to the function--it should start with the name of the module, + to the function—it should start with the name of the module, include any sub-modules, and if the function is a method on a class it should include the class name too. @@ -275,7 +275,7 @@ Let's dive in! What's a "converter"? It establishes both the type of the variable used in C, and the method to convert the Python value into a C value at runtime. - For now you're going to use what's called a "legacy converter"--a + For now you're going to use what's called a "legacy converter"—a convenience syntax intended to make porting old code into Argument Clinic easier. @@ -424,7 +424,7 @@ Let's dive in! (Argument Clinic always generates its format strings with a ``:`` followed by the name of the function. If the existing code's format string ends with ``;``, to provide - usage help, this change is harmless--don't worry about it.) + usage help, this change is harmless—don't worry about it.) Third, for parameters whose format units require two arguments (like a length variable, or an encoding string, or a pointer @@ -637,7 +637,7 @@ an optional argument on the *left* side of its required argument! Another example is ``curses.window.addch()``, which has a group of two arguments that must always be specified together. (The arguments are called ``x`` and ``y``; if you call the function passing in ``x``, -you must also pass in ``y``--and if you don't pass in ``x`` you may not +you must also pass in ``y``—and if you don't pass in ``x`` you may not pass in ``y`` either.) In any case, the goal of Argument Clinic is to support argument parsing @@ -888,7 +888,7 @@ Advanced converters Remember those format units you skipped for your first time because they were advanced? Here's how to handle those too. -The trick is, all those format units take arguments--either +The trick is, all those format units take arguments—either conversion functions, or types, or strings specifying an encoding. (But "legacy converters" don't support arguments. That's why we skipped them for your first function.) The argument you specified @@ -1002,7 +1002,7 @@ Using a return converter By default the impl function Argument Clinic generates for you returns ``PyObject *``. But your C function often computes some C type, then converts it into the ``PyObject *`` at the last moment. Argument Clinic handles converting your inputs from Python types -into native C types--why not have it convert your return value from a native C type +into native C types—why not have it convert your return value from a native C type into a Python type too? That's what a "return converter" does. It changes your impl function to return @@ -1184,7 +1184,7 @@ Writing a custom converter As we hinted at in the previous section... you can write your own converters! A converter is simply a Python class that inherits from ``CConverter``. The main purpose of a custom converter is if you have a parameter using -the ``O&`` format unit--parsing this parameter means calling +the ``O&`` format unit—parsing this parameter means calling a :c:func:`PyArg_ParseTuple` "converter function". Your converter class should be named ``*something*_converter``. @@ -1226,7 +1226,7 @@ to specify in your subclass. Here's the current list: The default value used to initialize the C variable when there is no default, but not specifying a default may result in an "uninitialized variable" warning. This can - easily happen when using option groups--although + easily happen when using option groups—although properly-written code will never actually use this value, the variable does get passed in to the impl, and the C compiler will complain about the "use" of the @@ -1402,7 +1402,7 @@ Let's start with defining some terminology: all of processing, even from Clinic blocks *after* the ``suppress`` - The text is suppressed--thrown away. + The text is suppressed—thrown away. Clinic defines five new directives that let you reconfigure its output. From aebb567c1e6d96b20df7e37a67ad022fd8473e18 Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Mon, 21 Nov 2016 13:40:45 +0200 Subject: [PATCH 30/71] Change an en-dash to an em-dash. --- Doc/howto/clinic.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 956120f3d..e586aa62a 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -376,7 +376,7 @@ Let's dive in! [clinic start generated code]*/ 12. Save and close the file, then run ``Tools/clinic/clinic.py`` on - it. With luck everything worked--your block now has output, and + it. With luck everything worked---your block now has output, and a ``.c.h`` file has been generated! Reopen the file in your text editor to see: From c3f61c04927f281d4a7e361030772d57640cd1b8 Mon Sep 17 00:00:00 2001 From: Martin Panter Date: Sat, 10 Dec 2016 03:49:12 +0000 Subject: [PATCH 31/71] Issue 28753: Argument Clinic howto docfix, courtesy Julien Palard. --- Doc/howto/clinic.rst | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index d4fd45ce8..8d6ab7f53 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -375,15 +375,12 @@ Let's dive in! Write a pickled representation of obj to the open file. [clinic start generated code]*/ -12. Save and close the file, then run ``Tools/clinic/clinic.py`` on it. - With luck everything worked and your block now has output! Reopen - the file in your text editor to see:: +12. Save and close the file, then run ``Tools/clinic/clinic.py`` on + it. With luck everything worked---your block now has output, and + a ``.c.h`` file has been generated! Reopen the file in your + text editor to see: - /*[clinic input] - module _pickle - class _pickle.Pickler "PicklerObject *" "&Pickler_Type" - [clinic start generated code]*/ - /*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ + .. code-block:: c /*[clinic input] _pickle.Pickler.dump @@ -395,18 +392,22 @@ Let's dive in! Write a pickled representation of obj to the open file. [clinic start generated code]*/ - PyDoc_STRVAR(_pickle_Pickler_dump__doc__, - "Write a pickled representation of obj to the open file.\n" - "\n" - ... static PyObject * - _pickle_Pickler_dump_impl(PicklerObject *self, PyObject *obj) - /*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ + _pickle_Pickler_dump(PicklerObject *self, PyObject *obj) + /*[clinic end generated code: output=87ecad1261e02ac7 input=552eb1c0f52260d9]*/ Obviously, if Argument Clinic didn't produce any output, it's because it found an error in your input. Keep fixing your errors and retrying until Argument Clinic processes your file without complaint. + For readability, most of the glue code has been generated to a ``.c.h`` + file. You'll need to include that in your original ``.c`` file, + typically right after the clinic module block: + + .. code-block:: c + + #include "clinic/_pickle.c.h" + 13. Double-check that the argument-parsing code Argument Clinic generated looks basically the same as the existing code. From 8089521f8e43d3424b3e0b257be55d3f2fafb396 Mon Sep 17 00:00:00 2001 From: Martin Panter Date: Sat, 10 Dec 2016 04:10:45 +0000 Subject: [PATCH 32/71] Issue #28755: Improve syntax highlighting in Arg Clinic howto --- Doc/howto/clinic.rst | 44 +++++++++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 15 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 8d6ab7f53..491c5a2ca 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1,3 +1,5 @@ +.. highlightlang:: c + ********************** Argument Clinic How-To ********************** @@ -78,17 +80,23 @@ Basic Concepts And Usage ======================== Argument Clinic ships with CPython; you'll find it in ``Tools/clinic/clinic.py``. -If you run that script, specifying a C file as an argument:: +If you run that script, specifying a C file as an argument: + +.. code-block:: shell-session - % python3 Tools/clinic/clinic.py foo.c + $ python3 Tools/clinic/clinic.py foo.c Argument Clinic will scan over the file looking for lines that -look exactly like this:: +look exactly like this: + +.. code-block:: none /*[clinic input] When it finds one, it reads everything up to a line that looks -exactly like this:: +exactly like this: + +.. code-block:: none [clinic start generated code]*/ @@ -99,7 +107,9 @@ lines, are collectively called an Argument Clinic "block". When Argument Clinic parses one of these blocks, it generates output. This output is rewritten into the C file immediately after the block, followed by a comment containing a checksum. -The Argument Clinic block now looks like this:: +The Argument Clinic block now looks like this: + +.. code-block:: none /*[clinic input] ... clinic input goes here ... @@ -378,9 +388,7 @@ Let's dive in! 12. Save and close the file, then run ``Tools/clinic/clinic.py`` on it. With luck everything worked---your block now has output, and a ``.c.h`` file has been generated! Reopen the file in your - text editor to see: - - .. code-block:: c + text editor to see:: /*[clinic input] _pickle.Pickler.dump @@ -402,9 +410,7 @@ Let's dive in! For readability, most of the glue code has been generated to a ``.c.h`` file. You'll need to include that in your original ``.c`` file, - typically right after the clinic module block: - - .. code-block:: c + typically right after the clinic module block:: #include "clinic/_pickle.c.h" @@ -1028,7 +1034,9 @@ that value, and an error has been set (``PyErr_Occurred()`` returns a true value), then the generated code will propagate the error. Otherwise it will encode the value you return like normal. -Currently Argument Clinic supports only a few return converters:: +Currently Argument Clinic supports only a few return converters: + +.. code-block:: none bool int @@ -1607,7 +1615,9 @@ code probably looks like this:: #endif /* HAVE_FUNCTIONNAME */ And then in the ``PyMethodDef`` structure at the bottom the existing code -will have:: +will have: + +.. code-block:: none #ifdef HAVE_FUNCTIONNAME {'functionname', ... }, @@ -1657,7 +1667,9 @@ extra code when using the "block" output preset? It can't go in the output bloc because that could be deactivated by the ``#ifdef``. (That's the whole point!) In this situation, Argument Clinic writes the extra code to the "buffer" destination. -This may mean that you get a complaint from Argument Clinic:: +This may mean that you get a complaint from Argument Clinic: + +.. code-block:: none Warning in file "Modules/posixmodule.c" on line 12357: Destination buffer 'buffer' not empty at end of file, emptying. @@ -1677,7 +1689,9 @@ wouldn't make any sense to the Python interpreter. But using Argument Clinic to run Python blocks lets you use Python as a Python preprocessor! Since Python comments are different from C comments, Argument Clinic -blocks embedded in Python files look slightly different. They look like this:: +blocks embedded in Python files look slightly different. They look like this: + +.. code-block:: python3 #/*[python input] #print("def foo(): pass") From 599e3d4c9ba262b04b6e5450bb267e543e26fcd2 Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Thu, 30 Mar 2017 10:01:03 +0300 Subject: [PATCH 33/71] bpo-29918: Add missed "const" modifiers in C API documentation. (#846) --- Doc/howto/clinic.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index eaab20ad9..b5c14d3af 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1050,7 +1050,7 @@ Currently Argument Clinic supports only a few return converters: DecodeFSDefault None of these take parameters. For the first three, return -1 to indicate -error. For ``DecodeFSDefault``, the return type is ``char *``; return a NULL +error. For ``DecodeFSDefault``, the return type is ``const char *``; return a NULL pointer to indicate an error. (There's also an experimental ``NoneType`` converter, which lets you From eeeceedc385d0033b6ccafa4e42b2e92f1ec010a Mon Sep 17 00:00:00 2001 From: gfyoung Date: Tue, 6 Jun 2017 15:23:52 -0400 Subject: [PATCH 34/71] bpo-29596: Improve clinic howto documentation (GH-1710) Clarify that `two-pass` buffer can only be dumped once, and it prints out all text sent to it during all processing, even from Clinic blocks *after* the dumping point. --- Doc/howto/clinic.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index b5c14d3af..d3c7d6689 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1407,8 +1407,8 @@ Let's start with defining some terminology: ``two-pass`` A buffer like ``buffer``. However, a two-pass buffer can only - be written once, and it prints out all text sent to it during - all of processing, even from Clinic blocks *after* the + be dumped once, and it prints out all text sent to it during + all processing, even from Clinic blocks *after* the dumping point. ``suppress`` The text is suppressed—thrown away. @@ -1471,7 +1471,7 @@ preset configurations, as follows: The default filename is ``"{dirname}/clinic/{basename}.h"``. ``buffer`` - Save up all most of the output from Clinic, to be written into + Save up most of the output from Clinic, to be written into your file near the end. For Python files implementing modules or builtin types, it's recommended that you dump the buffer just above the static structures for your module or From a0d67b7b4f382cc76ea3fd77d116b2c94d3ba404 Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Sun, 8 Apr 2018 19:18:04 +0300 Subject: [PATCH 35/71] Improve highlighting of some code blocks. (GH-6401) --- Doc/howto/clinic.rst | 68 +++++++++++++++++++++++++++++++++----------- 1 file changed, 51 insertions(+), 17 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index d3c7d6689..788a0eee2 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -267,12 +267,16 @@ Let's dive in! should get its own line. All the parameter lines should be indented from the function name and the docstring. - The general form of these parameter lines is as follows:: + The general form of these parameter lines is as follows: + + .. code-block:: none name_of_parameter: converter If the parameter has a default value, add that after the - converter:: + converter: + + .. code-block:: none name_of_parameter: converter = default_value @@ -925,13 +929,17 @@ Parameter default values ------------------------ Default values for parameters can be any of a number of values. -At their simplest, they can be string, int, or float literals:: +At their simplest, they can be string, int, or float literals: + +.. code-block:: none foo: str = "abc" bar: int = 123 bat: float = 45.6 -They can also use any of Python's built-in constants:: +They can also use any of Python's built-in constants: + +.. code-block:: none yep: bool = True nope: bool = False @@ -959,7 +967,9 @@ It can be an entire expression, using math operators and looking up attributes on objects. However, this support isn't exactly simple, because of some non-obvious semantics. -Consider the following example:: +Consider the following example: + +.. code-block:: none foo: Py_ssize_t = sys.maxsize - 1 @@ -970,7 +980,9 @@ runtime, when the user asks for the function's signature. What namespace is available when the expression is evaluated? It's evaluated in the context of the module the builtin came from. So, if your module has an -attribute called "``max_widgets``", you may simply use it:: +attribute called "``max_widgets``", you may simply use it: + +.. code-block:: none foo: Py_ssize_t = max_widgets @@ -982,7 +994,9 @@ it's best to restrict yourself to modules that are preloaded by Python itself.) Evaluating default values only at runtime means Argument Clinic can't compute the correct equivalent C default value. So you need to tell it explicitly. When you use an expression, you must also specify the equivalent expression -in C, using the ``c_default`` parameter to the converter:: +in C, using the ``c_default`` parameter to the converter: + +.. code-block:: none foo: Py_ssize_t(c_default="PY_SSIZE_T_MAX - 1") = sys.maxsize - 1 @@ -1359,7 +1373,9 @@ Let's start with defining some terminology: A field, in this context, is a subsection of Clinic's output. For example, the ``#define`` for the ``PyMethodDef`` structure is a field, called ``methoddef_define``. Clinic has seven - different fields it can output per function definition:: + different fields it can output per function definition: + + .. code-block:: none docstring_prototype docstring_definition @@ -1416,7 +1432,9 @@ Let's start with defining some terminology: Clinic defines five new directives that let you reconfigure its output. -The first new directive is ``dump``:: +The first new directive is ``dump``: + +.. code-block:: none dump @@ -1425,7 +1443,9 @@ the current block, and empties it. This only works with ``buffer`` and ``two-pass`` destinations. The second new directive is ``output``. The most basic form of ``output`` -is like this:: +is like this: + +.. code-block:: none output @@ -1433,7 +1453,9 @@ This tells Clinic to output *field* to *destination*. ``output`` also supports a special meta-destination, called ``everything``, which tells Clinic to output *all* fields to that *destination*. -``output`` has a number of other functions:: +``output`` has a number of other functions: + +.. code-block:: none output push output pop @@ -1508,7 +1530,9 @@ preset configurations, as follows: Suppresses the ``impl_prototype``, write the ``docstring_definition`` and ``parser_definition`` to ``buffer``, write everything else to ``block``. -The third new directive is ``destination``:: +The third new directive is ``destination``: + +.. code-block:: none destination [...] @@ -1516,7 +1540,9 @@ This performs an operation on the destination named ``name``. There are two defined subcommands: ``new`` and ``clear``. -The ``new`` subcommand works like this:: +The ``new`` subcommand works like this: + +.. code-block:: none destination new @@ -1564,7 +1590,9 @@ There are five destination types: A two-pass buffer, like the "two-pass" builtin destination above. -The ``clear`` subcommand works like this:: +The ``clear`` subcommand works like this: + +.. code-block:: none destination clear @@ -1572,7 +1600,9 @@ It removes all the accumulated text up to this point in the destination. (I don't know what you'd need this for, but I thought maybe it'd be useful while someone's experimenting.) -The fourth new directive is ``set``:: +The fourth new directive is ``set``: + +.. code-block:: none set line_prefix "string" set line_suffix "string" @@ -1590,7 +1620,9 @@ Both of these support two format strings: Turns into the string ``*/``, the end-comment text sequence for C files. The final new directive is one you shouldn't need to use directly, -called ``preserve``:: +called ``preserve``: + +.. code-block:: none preserve @@ -1638,7 +1670,9 @@ like so:: #endif /* HAVE_FUNCTIONNAME */ Then, remove those three lines from the ``PyMethodDef`` structure, -replacing them with the macro Argument Clinic generated:: +replacing them with the macro Argument Clinic generated: + +.. code-block:: none MODULE_FUNCTIONNAME_METHODDEF From 08cc339bdfcdde5e99c44ce083c21b2f461f3799 Mon Sep 17 00:00:00 2001 From: Eitan Adler Date: Sun, 20 May 2018 07:38:01 -0700 Subject: [PATCH 36/71] Docs: be less specific about python versions (GH-6985) CPython 3.5 is old now, and we don't bump this version often, so lets avoid using specific versions. --- Doc/howto/clinic.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 788a0eee2..695fbb1be 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -22,8 +22,8 @@ Argument Clinic How-To compatibility for future versions. In other words: if you maintain an external C extension for CPython, you're welcome to experiment with Argument Clinic in your own code. But the - version of Argument Clinic that ships with CPython 3.5 *could* - be totally incompatible and break all your code. + version of Argument Clinic that ships with the next version + of CPython *could* be totally incompatible and break all your code. The Goals Of Argument Clinic ============================ From 23e048861747fcdc7019832168f694123eabf6a9 Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Thu, 26 Jul 2018 13:22:16 +0300 Subject: [PATCH 37/71] bpo-20260: Implement non-bitwise unsigned int converters for Argument Clinic. (GH-8434) --- Doc/howto/clinic.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 695fbb1be..c75c31c98 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -878,6 +878,12 @@ converter:: Write a pickled representation of obj to the open file. [clinic start generated code]*/ +One advantage of real converters is that they're more flexible than legacy +converters. For example, the ``unsigned_int`` converter (and all the +``unsigned_`` converters) can be specified without ``bitwise=True``. Their +default behavior performs range checking on the value, and they won't accept +negative numbers. You just can't do that with a legacy converter! + Argument Clinic will show you all the converters it has available. For each converter it'll show you all the parameters it accepts, along with the default value for each parameter. From 52daf956fa7fab63b1f52d4296bdf003ee16237f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Wirtel?= Date: Sat, 27 Oct 2018 00:58:26 +0200 Subject: [PATCH 38/71] bpo-35042: Use the :pep: role where a PEP is specified (#10036) --- Doc/howto/clinic.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index c75c31c98..5b2457a16 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -765,7 +765,7 @@ All Argument Clinic converters accept the following arguments: ``annotation`` The annotation value for this parameter. Not currently supported, - because PEP 8 mandates that the Python library may not use + because :pep:`8` mandates that the Python library may not use annotations. In addition, some converters accept additional arguments. Here is a list From 1258db64af7c7f75e7580eaa26767c61b16c77d3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Wirtel?= Date: Fri, 17 May 2019 11:55:34 +0200 Subject: [PATCH 39/71] Doc: Replace the deprecated highlightlang directive by highlight. (#13377) highlightlang is deprecated since April 2018 in Sphinx. See https://github.com/sphinx-doc/sphinx/pull/4845 --- Doc/howto/clinic.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 5b2457a16..cfd9f2e40 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1,4 +1,4 @@ -.. highlightlang:: c +.. highlight:: c ********************** Argument Clinic How-To From 70be69577c4cd42addcae7f5838897b2c83c823e Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Wed, 30 Oct 2019 21:37:16 +0200 Subject: [PATCH 40/71] bpo-38600: NULL -> ``NULL``. (GH-17001) Also fix some other formatting. --- Doc/howto/clinic.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index cfd9f2e40..50041829b 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1070,7 +1070,7 @@ Currently Argument Clinic supports only a few return converters: DecodeFSDefault None of these take parameters. For the first three, return -1 to indicate -error. For ``DecodeFSDefault``, the return type is ``const char *``; return a NULL +error. For ``DecodeFSDefault``, the return type is ``const char *``; return a ``NULL`` pointer to indicate an error. (There's also an experimental ``NoneType`` converter, which lets you From 94a615ba02be1bbfc401ecdd5eb8579561053169 Mon Sep 17 00:00:00 2001 From: Erlend Egeberg Aasland Date: Wed, 20 Jan 2021 12:19:57 +0100 Subject: [PATCH 41/71] bpo-42048: Clinic Howto: Document AC's defining_class converter (GH-23978) --- Doc/howto/clinic.rst | 62 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 50041829b..3a3653a5e 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1206,6 +1206,68 @@ type for ``self``, it's best to create your own converter, subclassing [clinic start generated code]*/ +Using a "defining class" converter +---------------------------------- + +Argument Clinic facilitates gaining access to the defining class of a method. +This is useful for :ref:`heap type ` methods that need to fetch +module level state. Use :c:func:`PyType_FromModuleAndSpec` to associate a new +heap type with a module. You can now use :c:func:`PyType_GetModuleState` on +the defining class to fetch the module state, for example from a module method. + +Example from ``Modules/zlibmodule.c``. First, ``defining_class`` is added to +the clinic input:: + + /*[clinic input] + zlib.Compress.compress + + cls: defining_class + data: Py_buffer + Binary data to be compressed. + / + + +After running the Argument Clinic tool, the following function signature is +generated:: + + /*[clinic start generated code]*/ + static PyObject * + zlib_Compress_compress_impl(compobject *self, PyTypeObject *cls, + Py_buffer *data) + /*[clinic end generated code: output=6731b3f0ff357ca6 input=04d00f65ab01d260]*/ + + +The following code can now use ``PyType_GetModuleState(cls)`` to fetch the +module state:: + + zlibstate *state = PyType_GetModuleState(cls); + + +Each method may only have one argument using this converter, and it must appear +after ``self``, or, if ``self`` is not used, as the first argument. The argument +will be of type ``PyTypeObject *``. The argument will not appear in the +``__text_signature__``. + +The ``defining_class`` converter is not compatible with ``__init__`` and ``__new__`` +methods, which cannot use the ``METH_METHOD`` convention. + +It is not possible to use ``defining_class`` with slot methods. In order to +fetch the module state from such methods, use ``_PyType_GetModuleByDef`` to +look up the module and then :c:func:`PyModule_GetState` to fetch the module +state. Example from the ``setattro`` slot method in +``Modules/_threadmodule.c``:: + + static int + local_setattro(localobject *self, PyObject *name, PyObject *v) + { + PyObject *module = _PyType_GetModuleByDef(Py_TYPE(self), &thread_module); + thread_module_state *state = get_thread_state(module); + ... + } + + +See also :pep:`573`. + Writing a custom converter -------------------------- From fbe087ed18d6f6a0797c60e166c60e6f9d93dab2 Mon Sep 17 00:00:00 2001 From: Hugo van Kemenade Date: Wed, 20 Oct 2021 21:48:55 +0300 Subject: [PATCH 42/71] bpo-45320: Remove long-deprecated inspect methods (GH-28618) --- Doc/howto/clinic.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 3a3653a5e..a3c433029 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -567,9 +567,6 @@ expression. Currently the following are explicitly supported: * Simple symbolic constants like ``sys.maxsize``, which must start with the name of the module -In case you're curious, this is implemented in ``from_builtin()`` -in ``Lib/inspect.py``. - (In the future, this may need to get even more elaborate, to allow full expressions like ``CONSTANT - 1``.) From 9195d3dce0cd561ebe4504d23e055d63b3c32792 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Fri, 11 Feb 2022 17:22:11 +0100 Subject: [PATCH 43/71] bpo-46613: Add PyType_GetModuleByDef to the public API (GH-31081) * Make PyType_GetModuleByDef public (remove underscore) Co-authored-by: Victor Stinner --- Doc/howto/clinic.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index a3c433029..04b1a2cac 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1249,15 +1249,15 @@ The ``defining_class`` converter is not compatible with ``__init__`` and ``__new methods, which cannot use the ``METH_METHOD`` convention. It is not possible to use ``defining_class`` with slot methods. In order to -fetch the module state from such methods, use ``_PyType_GetModuleByDef`` to -look up the module and then :c:func:`PyModule_GetState` to fetch the module +fetch the module state from such methods, use :c:func:`PyType_GetModuleByDef` +to look up the module and then :c:func:`PyModule_GetState` to fetch the module state. Example from the ``setattro`` slot method in ``Modules/_threadmodule.c``:: static int local_setattro(localobject *self, PyObject *name, PyObject *v) { - PyObject *module = _PyType_GetModuleByDef(Py_TYPE(self), &thread_module); + PyObject *module = PyType_GetModuleByDef(Py_TYPE(self), &thread_module); thread_module_state *state = get_thread_state(module); ... } From c2e6013d02531c95be7a9d9fc0e0f241766bc760 Mon Sep 17 00:00:00 2001 From: Inada Naoki Date: Thu, 12 May 2022 14:48:38 +0900 Subject: [PATCH 44/71] gh-92536: PEP 623: Remove wstr and legacy APIs from Unicode (GH-92537) --- Doc/howto/clinic.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 04b1a2cac..989527b65 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -848,15 +848,15 @@ on the right is the text you'd replace it with. ``'s#'`` ``str(zeroes=True)`` ``'s*'`` ``Py_buffer(accept={buffer, str})`` ``'U'`` ``unicode`` -``'u'`` ``Py_UNICODE`` -``'u#'`` ``Py_UNICODE(zeroes=True)`` +``'u'`` ``wchar_t`` +``'u#'`` ``wchar_t(zeroes=True)`` ``'w*'`` ``Py_buffer(accept={rwbuffer})`` ``'Y'`` ``PyByteArrayObject`` ``'y'`` ``str(accept={bytes})`` ``'y#'`` ``str(accept={robuffer}, zeroes=True)`` ``'y*'`` ``Py_buffer`` -``'Z'`` ``Py_UNICODE(accept={str, NoneType})`` -``'Z#'`` ``Py_UNICODE(accept={str, NoneType}, zeroes=True)`` +``'Z'`` ``wchar_t(accept={str, NoneType})`` +``'Z#'`` ``wchar_t(accept={str, NoneType}, zeroes=True)`` ``'z'`` ``str(accept={str, NoneType})`` ``'z#'`` ``str(accept={str, NoneType}, zeroes=True)`` ``'z*'`` ``Py_buffer(accept={buffer, str, NoneType})`` From 26e30a156d976cc03c54620a1141243c4861807b Mon Sep 17 00:00:00 2001 From: Julien Palard Date: Fri, 13 May 2022 14:10:16 +0200 Subject: [PATCH 45/71] Document Py_ssize_t. (GH-92512) It fixes 252 errors from a Sphinx nitpicky run (sphinx-build -n). But there's 8182 errors left. Co-authored-by: Ezio Melotti --- Doc/howto/clinic.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 989527b65..7959bc3a5 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1347,7 +1347,7 @@ Here's the simplest example of a custom converter, from ``Modules/zlibmodule.c`` /*[python end generated code: output=da39a3ee5e6b4b0d input=35521e4e733823c7]*/ This block adds a converter to Argument Clinic named ``ssize_t``. Parameters -declared as ``ssize_t`` will be declared as type ``Py_ssize_t``, and will +declared as ``ssize_t`` will be declared as type :c:type:`Py_ssize_t`, and will be parsed by the ``'O&'`` format unit, which will call the ``ssize_t_converter`` converter function. ``ssize_t`` variables automatically support default values. From 0260219cf1ba8a697cf413ab4d1eb86923bfbb09 Mon Sep 17 00:00:00 2001 From: Ned Batchelder Date: Tue, 5 Jul 2022 05:16:10 -0400 Subject: [PATCH 46/71] Docs: remove redundant "adverb-adjective" hyphens from compound modifiers (GH-94551) Discussion: https://discuss.python.org/t/slight-grammar-fix-throughout-adverbs-dont-need-hyphen/17021 --- Doc/howto/clinic.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 7959bc3a5..2d368966b 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -541,7 +541,7 @@ Let's dive in! 16. Compile, then run the relevant portions of the regression-test suite. This change should not introduce any new compile-time warnings or errors, - and there should be no externally-visible change to Python's behavior. + and there should be no externally visible change to Python's behavior. Well, except for one difference: ``inspect.signature()`` run on your function should now provide a valid signature! @@ -1117,7 +1117,7 @@ Here's the syntax for cloning a function:: ``module.class`` in the sample just to illustrate that you must use the full path to *both* functions.) -Sorry, there's no syntax for partially-cloning a function, or cloning a function +Sorry, there's no syntax for partially cloning a function, or cloning a function then modifying it. Cloning is an all-or nothing proposition. Also, the function you are cloning from must have been previously defined @@ -1315,7 +1315,7 @@ to specify in your subclass. Here's the current list: there is no default, but not specifying a default may result in an "uninitialized variable" warning. This can easily happen when using option groups—although - properly-written code will never actually use this value, + properly written code will never actually use this value, the variable does get passed in to the impl, and the C compiler will complain about the "use" of the uninitialized value. This value should always be a From fb406e5e24ad508e980bfc4723be12dcae5ce768 Mon Sep 17 00:00:00 2001 From: Noam Cohen Date: Mon, 1 Aug 2022 16:07:35 +0300 Subject: [PATCH 47/71] gh-95007: Remove the NoneType return converter from Argument Clinic Doc (#95529) The converter was removed in 74b5e4ce80858ac5c7d03411cb8cce7e6865f181 --- Doc/howto/clinic.rst | 5 ----- 1 file changed, 5 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 2d368966b..d634c4b47 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1070,11 +1070,6 @@ None of these take parameters. For the first three, return -1 to indicate error. For ``DecodeFSDefault``, the return type is ``const char *``; return a ``NULL`` pointer to indicate an error. -(There's also an experimental ``NoneType`` converter, which lets you -return ``Py_None`` on success or ``NULL`` on failure, without having -to increment the reference count on ``Py_None``. I'm not sure it adds -enough clarity to be worth using.) - To see all the return converters Argument Clinic supports, along with their parameters (if any), just run ``Tools/clinic/clinic.py --converters`` for the full list. From 67dc719a6c7481a28c40ae39f9980d6aa4ee2eb8 Mon Sep 17 00:00:00 2001 From: Nikita Sobolev Date: Fri, 7 Oct 2022 21:54:45 +0300 Subject: [PATCH 48/71] gh-97956: Mention `generate_global_objects.py` in `AC How-To` (#97957) --- Doc/howto/clinic.rst | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index d634c4b47..b8afc7e6d 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -539,7 +539,15 @@ Let's dive in! }; -16. Compile, then run the relevant portions of the regression-test suite. +16. Argument Clinic may generate new instances of ``_Py_ID``. For example:: + + &_Py_ID(new_unique_py_id) + + If it does, you'll have to run ``Tools/scripts/generate_global_objects.py`` + to regenerate the list of precompiled identifiers at this point. + + +17. Compile, then run the relevant portions of the regression-test suite. This change should not introduce any new compile-time warnings or errors, and there should be no externally visible change to Python's behavior. From 7a1b039c94fec97597752dcb3a8387364b7a03a1 Mon Sep 17 00:00:00 2001 From: Stanley <46876382+slateny@users.noreply.github.com> Date: Tue, 25 Oct 2022 20:26:28 -0700 Subject: [PATCH 49/71] docs: Change links to label refs (#98454) Co-authored-by: C.A.M. Gerlach --- Doc/howto/clinic.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index b8afc7e6d..a97f1d23f 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1,5 +1,7 @@ .. highlight:: c +.. _howto-clinic: + ********************** Argument Clinic How-To ********************** From 89e8ca7a9b0613f119d2e58bc55a072e91f966dd Mon Sep 17 00:00:00 2001 From: Mariusz Felisiak Date: Wed, 11 Jan 2023 11:05:41 +0100 Subject: [PATCH 50/71] gh-98763: Prefer "python" over "python3" for command line examples in docs. (#98761) --- Doc/howto/clinic.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index a97f1d23f..8a10fe327 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -86,7 +86,7 @@ If you run that script, specifying a C file as an argument: .. code-block:: shell-session - $ python3 Tools/clinic/clinic.py foo.c + $ python Tools/clinic/clinic.py foo.c Argument Clinic will scan over the file looking for lines that look exactly like this: From e709d913c376e49d3b0206889c5bc2f1940cea86 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Fri, 5 May 2023 13:32:00 +0200 Subject: [PATCH 51/71] gh-64658: Expand Argument Clinic return converter docs (#104175) --- Doc/howto/clinic.rst | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 8a10fe327..6ebc2d9b0 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1033,19 +1033,36 @@ you're not permitted to use: Using a return converter ------------------------ -By default the impl function Argument Clinic generates for you returns ``PyObject *``. -But your C function often computes some C type, then converts it into the ``PyObject *`` +By default, the impl function Argument Clinic generates for you returns +:c:type:`PyObject * `. +But your C function often computes some C type, +then converts it into the :c:type:`!PyObject *` at the last moment. Argument Clinic handles converting your inputs from Python types into native C types—why not have it convert your return value from a native C type into a Python type too? That's what a "return converter" does. It changes your impl function to return some C type, then adds code to the generated (non-impl) function to handle converting -that value into the appropriate ``PyObject *``. +that value into the appropriate :c:type:`!PyObject *`. The syntax for return converters is similar to that of parameter converters. You specify the return converter like it was a return annotation on the -function itself. Return converters behave much the same as parameter converters; +function itself, using ``->`` notation. + +For example: + +.. code-block:: c + + /*[clinic input] + add -> int + + a: int + b: int + / + + [clinic start generated code]*/ + +Return converters behave much the same as parameter converters; they take arguments, the arguments are all keyword-only, and if you're not changing any of the default arguments you can omit the parentheses. @@ -1066,19 +1083,17 @@ Currently Argument Clinic supports only a few return converters: .. code-block:: none bool + double + float int - unsigned int long - unsigned int - size_t Py_ssize_t - float - double - DecodeFSDefault + size_t + unsigned int + unsigned long -None of these take parameters. For the first three, return -1 to indicate -error. For ``DecodeFSDefault``, the return type is ``const char *``; return a ``NULL`` -pointer to indicate an error. +None of these take parameters. +For all of these, return ``-1`` to indicate error. To see all the return converters Argument Clinic supports, along with their parameters (if any), From c11376955168ddaf8cb9b70c88082a57d269cbcd Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Fri, 12 May 2023 10:34:00 +0200 Subject: [PATCH 52/71] gh-104389: Add 'unused' keyword to Argument Clinic C converters (#104390) Use the unused keyword param in the converter to explicitly mark an argument as unused: /*[clinic input] SomeBaseClass.stubmethod flag: bool(unused=True) [clinic start generated code]*/ --- Doc/howto/clinic.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 6ebc2d9b0..4620b4617 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -775,6 +775,9 @@ All Argument Clinic converters accept the following arguments: because :pep:`8` mandates that the Python library may not use annotations. + ``unused`` + Wrap the argument with :c:macro:`Py_UNUSED` in the impl function signature. + In addition, some converters accept additional arguments. Here is a list of these arguments, along with their meanings: From 4f38aa3a9bb1c39c91174d7f85067c7fe1488cf2 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Sun, 16 Jul 2023 00:42:58 +0200 Subject: [PATCH 53/71] Docs: Normalize Argument Clinic How-To section capitalization (#106788) --- Doc/howto/clinic.rst | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 4620b4617..0f99cb649 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -27,7 +27,8 @@ Argument Clinic How-To version of Argument Clinic that ships with the next version of CPython *could* be totally incompatible and break all your code. -The Goals Of Argument Clinic + +The goals of Argument Clinic ============================ Argument Clinic's primary goal @@ -78,7 +79,7 @@ and it should be able to do many interesting and smart things with all the information you give it. -Basic Concepts And Usage +Basic concepts and usage ======================== Argument Clinic ships with CPython; you'll find it in ``Tools/clinic/clinic.py``. @@ -141,7 +142,7 @@ For the sake of clarity, here's the terminology we'll use with Argument Clinic: a block.) -Converting Your First Function +Converting your first function ============================== The best way to get a sense of how Argument Clinic works is to @@ -558,7 +559,8 @@ Let's dive in! Congratulations, you've ported your first function to work with Argument Clinic! -Advanced Topics + +Advanced topics =============== Now that you've had some experience working with Argument Clinic, it's time @@ -636,7 +638,8 @@ after the last argument). Currently the generated code will use :c:func:`PyArg_ParseTuple`, but this will change soon. -Optional Groups + +Optional groups --------------- Some legacy functions have a tricky approach to parsing their arguments: @@ -899,6 +902,7 @@ available. For each converter it'll show you all the parameters it accepts, along with the default value for each parameter. Just run ``Tools/clinic/clinic.py --converters`` to see the full list. + Py_buffer --------- @@ -908,7 +912,6 @@ you *must* not call :c:func:`PyBuffer_Release` on the provided buffer. Argument Clinic generates code that does it for you (in the parsing function). - Advanced converters ------------------- @@ -975,6 +978,7 @@ value called ``NULL`` for just this reason: from Python's perspective it behaves like a default value of ``None``, but the C variable is initialized with ``NULL``. + Expressions specified as default values --------------------------------------- @@ -1032,7 +1036,6 @@ you're not permitted to use: * Tuple/list/set/dict literals. - Using a return converter ------------------------ @@ -1146,6 +1149,7 @@ then modifying it. Cloning is an all-or nothing proposition. Also, the function you are cloning from must have been previously defined in the current file. + Calling Python code ------------------- @@ -1380,6 +1384,7 @@ handle initialization and cleanup. You can see more examples of custom converters in the CPython source tree; grep the C files for the string ``CConverter``. + Writing a custom return converter --------------------------------- @@ -1394,8 +1399,9 @@ write your own return converter, please read ``Tools/clinic/clinic.py``, specifically the implementation of ``CReturnConverter`` and all its subclasses. + METH_O and METH_NOARGS ----------------------------------------------- +---------------------- To convert a function using ``METH_O``, make sure the function's single argument is using the ``object`` converter, and mark the @@ -1415,8 +1421,9 @@ any arguments. You can still use a self converter, a return converter, and specify a ``type`` argument to the object converter for ``METH_O``. + tp_new and tp_init functions ----------------------------------------------- +---------------------------- You can convert ``tp_new`` and ``tp_init`` functions. Just name them ``__new__`` or ``__init__`` as appropriate. Notes: @@ -1437,6 +1444,7 @@ them ``__new__`` or ``__init__`` as appropriate. Notes: (If your function doesn't support keywords, the parsing function generated will throw an exception if it receives any.) + Changing and redirecting Clinic's output ---------------------------------------- @@ -1721,7 +1729,7 @@ the file was not modified by hand before it gets overwritten. The #ifdef trick ----------------------------------------------- +---------------- If you're converting a function that isn't available on all platforms, there's a trick you can use to make life a little easier. The existing @@ -1801,7 +1809,6 @@ Argument Clinic added to your file (it'll be at the very bottom), then move it above the ``PyMethodDef`` structure where that macro is used. - Using Argument Clinic in Python files ------------------------------------- From e67a548857fb1aeef16d22c88c6fd83cc013bea4 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Tue, 18 Jul 2023 10:50:17 +0200 Subject: [PATCH 54/71] Docs: Normalise Argument Clinic advanced topics headings (#106842) Co-authored-by: Ezio Melotti --- Doc/howto/clinic.rst | 95 +++++++++++++++++++++----------------------- 1 file changed, 46 insertions(+), 49 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 0f99cb649..12d7a77d4 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -560,15 +560,12 @@ Let's dive in! Congratulations, you've ported your first function to work with Argument Clinic! -Advanced topics -=============== +How-to guides +============= -Now that you've had some experience working with Argument Clinic, it's time -for some advanced topics. - -Symbolic default values ------------------------ +How to use symbolic default values +---------------------------------- The default value you provide for a parameter can't be any arbitrary expression. Currently the following are explicitly supported: @@ -583,8 +580,8 @@ expression. Currently the following are explicitly supported: to allow full expressions like ``CONSTANT - 1``.) -Renaming the C functions and variables generated by Argument Clinic -------------------------------------------------------------------- +How to to rename C functions and variables generated by Argument Clinic +----------------------------------------------------------------------- Argument Clinic automatically names the functions it generates for you. Occasionally this may cause a problem, if the generated name collides with @@ -626,8 +623,8 @@ array) would be ``file``, but the C variable would be named ``file_obj``. You can use this to rename the ``self`` parameter too! -Converting functions using PyArg_UnpackTuple --------------------------------------------- +How to convert functions using ``PyArg_UnpackTuple`` +---------------------------------------------------- To convert a function parsing its arguments with :c:func:`PyArg_UnpackTuple`, simply write out all the arguments, specifying each as an ``object``. You @@ -639,8 +636,8 @@ Currently the generated code will use :c:func:`PyArg_ParseTuple`, but this will change soon. -Optional groups ---------------- +How to use optional groups +-------------------------- Some legacy functions have a tricky approach to parsing their arguments: they count the number of positional arguments, then use a ``switch`` statement @@ -732,8 +729,8 @@ Notes: use optional groups for new code. -Using real Argument Clinic converters, instead of "legacy converters" ---------------------------------------------------------------------- +How to use real Argument Clinic converters, instead of "legacy converters" +-------------------------------------------------------------------------- To save time, and to minimize how much you need to learn to achieve your first port to Argument Clinic, the walkthrough above tells @@ -903,8 +900,8 @@ it accepts, along with the default value for each parameter. Just run ``Tools/clinic/clinic.py --converters`` to see the full list. -Py_buffer ---------- +How to use the ``Py_buffer`` converter +-------------------------------------- When using the ``Py_buffer`` converter (or the ``'s*'``, ``'w*'``, ``'*y'``, or ``'z*'`` legacy converters), @@ -912,8 +909,8 @@ you *must* not call :c:func:`PyBuffer_Release` on the provided buffer. Argument Clinic generates code that does it for you (in the parsing function). -Advanced converters -------------------- +How to use advanced converters +------------------------------ Remember those format units you skipped for your first time because they were advanced? Here's how to handle those too. @@ -944,8 +941,8 @@ hard-coded encoding strings for parameters whose format units start with ``e``. .. _default_values: -Parameter default values ------------------------- +How to assign default values to parameter +----------------------------------------- Default values for parameters can be any of a number of values. At their simplest, they can be string, int, or float literals: @@ -968,8 +965,8 @@ There's also special support for a default value of ``NULL``, and for simple expressions, documented in the following sections. -The ``NULL`` default value --------------------------- +How to use the ``NULL`` default value +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ For string and object parameters, you can set them to ``None`` to indicate that there's no default. However, that means the C variable will be @@ -979,8 +976,8 @@ behaves like a default value of ``None``, but the C variable is initialized with ``NULL``. -Expressions specified as default values ---------------------------------------- +How to use expressions as default values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The default value for a parameter can be more than just a literal value. It can be an entire expression, using math operators and looking up attributes @@ -1036,8 +1033,8 @@ you're not permitted to use: * Tuple/list/set/dict literals. -Using a return converter ------------------------- +How to use return converters +---------------------------- By default, the impl function Argument Clinic generates for you returns :c:type:`PyObject * `. @@ -1106,8 +1103,8 @@ their parameters (if any), just run ``Tools/clinic/clinic.py --converters`` for the full list. -Cloning existing functions --------------------------- +How to clone existing functions +------------------------------- If you have a number of functions that look similar, you may be able to use Clinic's "clone" feature. When you clone an existing function, @@ -1150,8 +1147,8 @@ Also, the function you are cloning from must have been previously defined in the current file. -Calling Python code -------------------- +How to call Python code +----------------------- The rest of the advanced topics require you to write Python code which lives inside your C file and modifies Argument Clinic's @@ -1178,8 +1175,8 @@ variable to the C code:: /*[python checksum:...]*/ -Using a "self converter" ------------------------- +How to use the "self converter" +------------------------------- Argument Clinic automatically adds a "self" parameter for you using a default converter. It automatically sets the ``type`` @@ -1230,8 +1227,8 @@ type for ``self``, it's best to create your own converter, subclassing [clinic start generated code]*/ -Using a "defining class" converter ----------------------------------- +How to use the "defining class" converter +----------------------------------------- Argument Clinic facilitates gaining access to the defining class of a method. This is useful for :ref:`heap type ` methods that need to fetch @@ -1293,8 +1290,8 @@ state. Example from the ``setattro`` slot method in See also :pep:`573`. -Writing a custom converter --------------------------- +How to write a custom converter +------------------------------- As we hinted at in the previous section... you can write your own converters! A converter is simply a Python class that inherits from ``CConverter``. @@ -1385,8 +1382,8 @@ You can see more examples of custom converters in the CPython source tree; grep the C files for the string ``CConverter``. -Writing a custom return converter ---------------------------------- +How to write a custom return converter +-------------------------------------- Writing a custom return converter is much like writing a custom converter. Except it's somewhat simpler, because return @@ -1400,8 +1397,8 @@ specifically the implementation of ``CReturnConverter`` and all its subclasses. -METH_O and METH_NOARGS ----------------------- +How to convert ``METH_O`` and ``METH_NOARGS`` functions +------------------------------------------------------- To convert a function using ``METH_O``, make sure the function's single argument is using the ``object`` converter, and mark the @@ -1422,8 +1419,8 @@ You can still use a self converter, a return converter, and specify a ``type`` argument to the object converter for ``METH_O``. -tp_new and tp_init functions ----------------------------- +How to convert ``tp_new`` and ``tp_init`` functions +--------------------------------------------------- You can convert ``tp_new`` and ``tp_init`` functions. Just name them ``__new__`` or ``__init__`` as appropriate. Notes: @@ -1445,8 +1442,8 @@ them ``__new__`` or ``__init__`` as appropriate. Notes: generated will throw an exception if it receives any.) -Changing and redirecting Clinic's output ----------------------------------------- +How to change and redirect Clinic's output +------------------------------------------ It can be inconvenient to have Clinic's output interspersed with your conventional hand-edited C code. Luckily, Clinic is configurable: @@ -1728,8 +1725,8 @@ it in a Clinic block lets Clinic use its existing checksum functionality to ensu the file was not modified by hand before it gets overwritten. -The #ifdef trick ----------------- +How to use the ``#ifdef`` trick +------------------------------- If you're converting a function that isn't available on all platforms, there's a trick you can use to make life a little easier. The existing @@ -1809,8 +1806,8 @@ Argument Clinic added to your file (it'll be at the very bottom), then move it above the ``PyMethodDef`` structure where that macro is used. -Using Argument Clinic in Python files -------------------------------------- +How to use Argument Clinic in Python files +------------------------------------------ It's actually possible to use Argument Clinic to preprocess Python files. There's no point to using Argument Clinic blocks, of course, as the output From d7a9d61cf2389793b3587580dd50a94ce0433aad Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Wed, 19 Jul 2023 00:46:50 +0200 Subject: [PATCH 55/71] Docs: Argument Clinic: Group guides about default values (#106872) Previous ToC layout (excerpt): - How to use symbolic default values ... - How to assign default values to parameter - How to use the ``NULL`` default value - How to use expressions as default values New layout: - How to assign default values to parameter - The ``NULL`` default value - Symbolic default values - Expressions as default values --- Doc/howto/clinic.rst | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 12d7a77d4..efeb22c61 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -564,22 +564,6 @@ How-to guides ============= -How to use symbolic default values ----------------------------------- - -The default value you provide for a parameter can't be any arbitrary -expression. Currently the following are explicitly supported: - -* Numeric constants (integer and float) -* String constants -* ``True``, ``False``, and ``None`` -* Simple symbolic constants like ``sys.maxsize``, which must - start with the name of the module - -(In the future, this may need to get even more elaborate, -to allow full expressions like ``CONSTANT - 1``.) - - How to to rename C functions and variables generated by Argument Clinic ----------------------------------------------------------------------- @@ -965,8 +949,8 @@ There's also special support for a default value of ``NULL``, and for simple expressions, documented in the following sections. -How to use the ``NULL`` default value -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The ``NULL`` default value +^^^^^^^^^^^^^^^^^^^^^^^^^^ For string and object parameters, you can set them to ``None`` to indicate that there's no default. However, that means the C variable will be @@ -976,8 +960,24 @@ behaves like a default value of ``None``, but the C variable is initialized with ``NULL``. -How to use expressions as default values -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Symbolic default values +^^^^^^^^^^^^^^^^^^^^^^^ + +The default value you provide for a parameter can't be any arbitrary +expression. Currently the following are explicitly supported: + +* Numeric constants (integer and float) +* String constants +* ``True``, ``False``, and ``None`` +* Simple symbolic constants like ``sys.maxsize``, which must + start with the name of the module + +(In the future, this may need to get even more elaborate, +to allow full expressions like ``CONSTANT - 1``.) + + +Expressions as default values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The default value for a parameter can be more than just a literal value. It can be an entire expression, using math operators and looking up attributes From c6bf58bc4dd43e869259ef7cf89da0b63e9eced2 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Fri, 21 Jul 2023 08:05:41 +0200 Subject: [PATCH 56/71] Docs: Argument Clinic: Add Background and Tutorial top-level sections (#106904) Add Background as a toplevel section with the following subsections: - Background - The goals of Argument Clinic - Basic concepts and usage Rename "Converting your first function" to Tutorial. Add anchors for Background, Tutorial, and How-to Guides: - :ref:`clinic-background` - :ref:`clinic-tutorial` - :ref:`clinic-howtos` Link to these from within the Abstract. Break the compatibility paragraph out of Abstract and make it a note. Co-authored-by: Ezio Melotti --- Doc/howto/clinic.rst | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index efeb22c61..f6bf1d223 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -13,12 +13,20 @@ Argument Clinic How-To Argument Clinic is a preprocessor for CPython C files. Its purpose is to automate all the boilerplate involved - with writing argument parsing code for "builtins". - This document shows you how to convert your first C - function to work with Argument Clinic, and then introduces - some advanced topics on Argument Clinic usage. + with writing argument parsing code for "builtins", + module level functions, and class methods. + This document is divided in three major sections: - Currently Argument Clinic is considered internal-only + * :ref:`clinic-background` talks about the basic concepts and goals of + Argument Clinic. + * :ref:`clinic-tutorial` guides you through all the steps required to + adapt an existing C function to Argument Clinic. + * :ref:`clinic-howtos` details how to handle specific tasks. + + +.. note:: + + Argument Clinic is considered internal-only for CPython. Its use is not supported for files outside CPython, and no guarantees are made regarding backwards compatibility for future versions. In other words: if you @@ -28,8 +36,14 @@ Argument Clinic How-To of CPython *could* be totally incompatible and break all your code. +.. _clinic-background: + +Background +========== + + The goals of Argument Clinic -============================ +---------------------------- Argument Clinic's primary goal is to take over responsibility for all argument parsing code @@ -80,7 +94,7 @@ things with all the information you give it. Basic concepts and usage -======================== +------------------------ Argument Clinic ships with CPython; you'll find it in ``Tools/clinic/clinic.py``. If you run that script, specifying a C file as an argument: @@ -142,8 +156,10 @@ For the sake of clarity, here's the terminology we'll use with Argument Clinic: a block.) -Converting your first function -============================== +.. _clinic-tutorial: + +Tutorial +======== The best way to get a sense of how Argument Clinic works is to convert a function to work with it. Here, then, are the bare @@ -560,6 +576,8 @@ Let's dive in! Congratulations, you've ported your first function to work with Argument Clinic! +.. _clinic-howtos: + How-to guides ============= From d7f9081d130549bd597d98c9b4124384b90c10ad Mon Sep 17 00:00:00 2001 From: Hakan Celik Date: Mon, 24 Jul 2023 14:54:39 +0300 Subject: [PATCH 57/71] Docs: Remove duplicate word in Argument Clinic howto heading (#107169) --- Doc/howto/clinic.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index f6bf1d223..933fecab9 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -582,8 +582,8 @@ How-to guides ============= -How to to rename C functions and variables generated by Argument Clinic ------------------------------------------------------------------------ +How to rename C functions and variables generated by Argument Clinic +-------------------------------------------------------------------- Argument Clinic automatically names the functions it generates for you. Occasionally this may cause a problem, if the generated name collides with From 6ffa5e8fd868d0499d5be4b1f2c44beddcc07a05 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Mon, 24 Jul 2023 17:22:18 +0200 Subject: [PATCH 58/71] Docs: Add missing markup to Argument Clinic docs (#106876) Co-authored-by: Ezio Melotti Co-authored-by: Serhiy Storchaka --- Doc/howto/clinic.rst | 269 +++++++++++++++++++++++-------------------- 1 file changed, 143 insertions(+), 126 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 933fecab9..98d3632ff 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -96,7 +96,8 @@ things with all the information you give it. Basic concepts and usage ------------------------ -Argument Clinic ships with CPython; you'll find it in ``Tools/clinic/clinic.py``. +Argument Clinic ships with CPython; you'll find it in +:source:`Tools/clinic/clinic.py`. If you run that script, specifying a C file as an argument: .. code-block:: shell-session @@ -178,9 +179,10 @@ Let's dive in! 1. Find a Python builtin that calls either :c:func:`PyArg_ParseTuple` or :c:func:`PyArg_ParseTupleAndKeywords`, and hasn't been converted to work with Argument Clinic yet. - For my example I'm using ``_pickle.Pickler.dump()``. + For my example I'm using + :py:meth:`_pickle.Pickler.dump `. -2. If the call to the ``PyArg_Parse`` function uses any of the +2. If the call to the :c:func:`!PyArg_Parse*` function uses any of the following format units: .. code-block:: none @@ -197,10 +199,10 @@ Let's dive in! support all of these scenarios. But these are advanced topics—let's do something simpler for your first function. - Also, if the function has multiple calls to :c:func:`PyArg_ParseTuple` + Also, if the function has multiple calls to :c:func:`!PyArg_ParseTuple` or :c:func:`PyArg_ParseTupleAndKeywords` where it supports different types for the same argument, or if the function uses something besides - PyArg_Parse functions to parse its arguments, it probably + :c:func:`!PyArg_Parse*` functions to parse its arguments, it probably isn't suitable for conversion to Argument Clinic. Argument Clinic doesn't support generic functions or polymorphic parameters. @@ -217,7 +219,7 @@ Let's dive in! If the old docstring had a first line that looked like a function signature, throw that line away. (The docstring doesn't need it - anymore—when you use ``help()`` on your builtin in the future, + anymore—when you use :py:func:`help` on your builtin in the future, the first line will be built automatically based on the function's signature.) @@ -264,7 +266,7 @@ Let's dive in! When you declare a class, you must also specify two aspects of its type in C: the type declaration you'd use for a pointer to an instance of - this class, and a pointer to the :c:type:`PyTypeObject` for this class. + this class, and a pointer to the :c:type:`!PyTypeObject` for this class. Sample:: @@ -313,10 +315,10 @@ Let's dive in! Clinic easier. For each parameter, copy the "format unit" for that - parameter from the ``PyArg_Parse()`` format argument and + parameter from the :c:func:`PyArg_Parse` format argument and specify *that* as its converter, as a quoted string. ("format unit" is the formal name for the one-to-three - character substring of the ``format`` parameter that tells + character substring of the *format* parameter that tells the argument parsing function what the type of the variable is and how to convert it. For more on format units please see :ref:`arg-parsing`.) @@ -349,7 +351,7 @@ Let's dive in! itself before the first keyword-only argument, indented the same as the parameter lines. - (``_pickle.Pickler.dump`` has neither, so our sample is unchanged.) + (:py:meth:`!_pickle.Pickler.dump` has neither, so our sample is unchanged.) 10. If the existing C function calls :c:func:`PyArg_ParseTuple` @@ -410,7 +412,7 @@ Let's dive in! 12. Save and close the file, then run ``Tools/clinic/clinic.py`` on it. With luck everything worked---your block now has output, and - a ``.c.h`` file has been generated! Reopen the file in your + a :file:`.c.h` file has been generated! Reopen the file in your text editor to see:: /*[clinic input] @@ -431,8 +433,8 @@ Let's dive in! it found an error in your input. Keep fixing your errors and retrying until Argument Clinic processes your file without complaint. - For readability, most of the glue code has been generated to a ``.c.h`` - file. You'll need to include that in your original ``.c`` file, + For readability, most of the glue code has been generated to a :file:`.c.h` + file. You'll need to include that in your original :file:`.c` file, typically right after the clinic module block:: #include "clinic/_pickle.c.h" @@ -446,8 +448,8 @@ Let's dive in! ensure that the code generated by Argument Clinic calls the *exact* same function. - Second, the format string passed in to :c:func:`PyArg_ParseTuple` or - :c:func:`PyArg_ParseTupleAndKeywords` should be *exactly* the same + Second, the format string passed in to :c:func:`!PyArg_ParseTuple` or + :c:func:`!PyArg_ParseTupleAndKeywords` should be *exactly* the same as the hand-written one in the existing function, up to the colon or semi-colon. @@ -469,7 +471,7 @@ Let's dive in! {"dump", (PyCFunction)__pickle_Pickler_dump, METH_O, __pickle_Pickler_dump__doc__}, This static structure should be *exactly* the same as the existing static - :c:type:`PyMethodDef` structure for this builtin. + :c:type:`!PyMethodDef` structure for this builtin. If any of these items differ in *any way*, adjust your Argument Clinic function specification and rerun @@ -539,14 +541,14 @@ Let's dive in! ... 15. Remember the macro with the :c:type:`PyMethodDef` structure for this - function? Find the existing :c:type:`PyMethodDef` structure for this + function? Find the existing :c:type:`!PyMethodDef` structure for this function and replace it with a reference to the macro. (If the builtin is at module scope, this will probably be very near the end of the file; if the builtin is a class method, this will probably be below but relatively near to the implementation.) Note that the body of the macro contains a trailing comma. So when you - replace the existing static :c:type:`PyMethodDef` structure with the macro, + replace the existing static :c:type:`!PyMethodDef` structure with the macro, *don't* add a comma to the end. Sample:: @@ -562,7 +564,7 @@ Let's dive in! &_Py_ID(new_unique_py_id) - If it does, you'll have to run ``Tools/scripts/generate_global_objects.py`` + If it does, you'll have to run ``make regen-global-objects`` to regenerate the list of precompiled identifiers at this point. @@ -570,7 +572,7 @@ Let's dive in! This change should not introduce any new compile-time warnings or errors, and there should be no externally visible change to Python's behavior. - Well, except for one difference: ``inspect.signature()`` run on your function + Well, except for one difference: :py:func:`inspect.signature` run on your function should now provide a valid signature! Congratulations, you've ported your first function to work with Argument Clinic! @@ -594,15 +596,15 @@ Argument Clinic will use that function name for the base (generated) function, then add ``"_impl"`` to the end and use that for the name of the impl function. For example, if we wanted to rename the C function names generated for -``pickle.Pickler.dump``, it'd look like this:: +:py:meth:`pickle.Pickler.dump`, it'd look like this:: /*[clinic input] pickle.Pickler.dump as pickler_dumper ... -The base function would now be named ``pickler_dumper()``, -and the impl function would now be named ``pickler_dumper_impl()``. +The base function would now be named :c:func:`!pickler_dumper`, +and the impl function would now be named :c:func:`!pickler_dumper_impl`. Similarly, you may have a problem where you want to give a parameter @@ -620,9 +622,9 @@ using the same ``"as"`` syntax:: fix_imports: bool = True Here, the name used in Python (in the signature and the ``keywords`` -array) would be ``file``, but the C variable would be named ``file_obj``. +array) would be *file*, but the C variable would be named ``file_obj``. -You can use this to rename the ``self`` parameter too! +You can use this to rename the *self* parameter too! How to convert functions using ``PyArg_UnpackTuple`` @@ -630,7 +632,7 @@ How to convert functions using ``PyArg_UnpackTuple`` To convert a function parsing its arguments with :c:func:`PyArg_UnpackTuple`, simply write out all the arguments, specifying each as an ``object``. You -may specify the ``type`` argument to cast the type as appropriate. All +may specify the *type* argument to cast the type as appropriate. All arguments should be marked positional-only (add a ``/`` on a line by itself after the last argument). @@ -649,16 +651,16 @@ keyword-only arguments.) This approach was used to simulate optional arguments back before :c:func:`PyArg_ParseTupleAndKeywords` was created. While functions using this approach can often be converted to -use :c:func:`PyArg_ParseTupleAndKeywords`, optional arguments, and default values, +use :c:func:`!PyArg_ParseTupleAndKeywords`, optional arguments, and default values, it's not always possible. Some of these legacy functions have -behaviors :c:func:`PyArg_ParseTupleAndKeywords` doesn't directly support. -The most obvious example is the builtin function ``range()``, which has +behaviors :c:func:`!PyArg_ParseTupleAndKeywords` doesn't directly support. +The most obvious example is the builtin function :py:func:`range`, which has an optional argument on the *left* side of its required argument! -Another example is ``curses.window.addch()``, which has a group of two +Another example is :py:meth:`curses.window.addch`, which has a group of two arguments that must always be specified together. (The arguments are -called ``x`` and ``y``; if you call the function passing in ``x``, -you must also pass in ``y``—and if you don't pass in ``x`` you may not -pass in ``y`` either.) +called *x* and *y*; if you call the function passing in *x*, +you must also pass in *y* — and if you don't pass in *x* you may not +pass in *y* either.) In any case, the goal of Argument Clinic is to support argument parsing for all existing CPython builtins without changing their semantics. @@ -679,7 +681,7 @@ can *only* be used with positional-only parameters. To specify an optional group, add a ``[`` on a line by itself before the parameters you wish to group together, and a ``]`` on a line by itself -after these parameters. As an example, here's how ``curses.window.addch`` +after these parameters. As an example, here's how :py:meth:`curses.window.addch` uses optional groups to make the first two parameters and the last parameter optional:: @@ -765,25 +767,25 @@ the same converters. All arguments to Argument Clinic converters are keyword-only. All Argument Clinic converters accept the following arguments: - ``c_default`` + *c_default* The default value for this parameter when defined in C. Specifically, this will be the initializer for the variable declared in the "parse function". See :ref:`the section on default values ` for how to use this. Specified as a string. - ``annotation`` + *annotation* The annotation value for this parameter. Not currently supported, because :pep:`8` mandates that the Python library may not use annotations. - ``unused`` + *unused* Wrap the argument with :c:macro:`Py_UNUSED` in the impl function signature. In addition, some converters accept additional arguments. Here is a list of these arguments, along with their meanings: - ``accept`` + *accept* A set of Python types (and possibly pseudo-types); this restricts the allowable Python argument to values of these types. (This is not a general-purpose facility; as a rule it only supports @@ -791,38 +793,38 @@ of these arguments, along with their meanings: To accept ``None``, add ``NoneType`` to this set. - ``bitwise`` + *bitwise* Only supported for unsigned integers. The native integer value of this Python argument will be written to the parameter without any range checking, even for negative values. - ``converter`` + *converter* Only supported by the ``object`` converter. Specifies the name of a :ref:`C "converter function" ` to use to convert this object to a native type. - ``encoding`` + *encoding* Only supported for strings. Specifies the encoding to use when converting this string from a Python str (Unicode) value into a C ``char *`` value. - ``subclass_of`` + *subclass_of* Only supported for the ``object`` converter. Requires that the Python value be a subclass of a Python type, as expressed in C. - ``type`` + *type* Only supported for the ``object`` and ``self`` converters. Specifies the C type that will be used to declare the variable. Default value is ``"PyObject *"``. - ``zeroes`` + *zeroes* Only supported for strings. If true, embedded NUL bytes (``'\\0'``) are permitted inside the value. The length of the string will be passed in to the impl function, just after the string parameter, as a parameter named ``_length``. Please note, not every possible combination of arguments will work. -Usually these arguments are implemented by specific ``PyArg_ParseTuple`` +Usually these arguments are implemented by specific :c:func:`PyArg_ParseTuple` *format units*, with specific behavior. For example, currently you cannot call ``unsigned_short`` without also specifying ``bitwise=True``. Although it's perfectly reasonable to think this would work, these semantics don't @@ -922,19 +924,19 @@ conversion functions, or types, or strings specifying an encoding. (But "legacy converters" don't support arguments. That's why we skipped them for your first function.) The argument you specified to the format unit is now an argument to the converter; this -argument is either ``converter`` (for ``O&``), ``subclass_of`` (for ``O!``), -or ``encoding`` (for all the format units that start with ``e``). +argument is either *converter* (for ``O&``), *subclass_of* (for ``O!``), +or *encoding* (for all the format units that start with ``e``). -When using ``subclass_of``, you may also want to use the other -custom argument for ``object()``: ``type``, which lets you set the type +When using *subclass_of*, you may also want to use the other +custom argument for ``object()``: *type*, which lets you set the type actually used for the parameter. For example, if you want to ensure -that the object is a subclass of ``PyUnicode_Type``, you probably want +that the object is a subclass of :c:var:`PyUnicode_Type`, you probably want to use the converter ``object(type='PyUnicodeObject *', subclass_of='&PyUnicode_Type')``. One possible problem with using Argument Clinic: it takes away some possible flexibility for the format units starting with ``e``. When writing a -``PyArg_Parse`` call by hand, you could theoretically decide at runtime what -encoding string to pass in to :c:func:`PyArg_ParseTuple`. But now this string must +:c:func:`!PyArg_Parse*` call by hand, you could theoretically decide at runtime what +encoding string to pass to that call. But now this string must be hard-coded at Argument-Clinic-preprocessing-time. This limitation is deliberate; it made supporting this format unit much easier, and may allow for future optimizations. This restriction doesn't seem unreasonable; CPython itself always passes in static @@ -987,7 +989,7 @@ expression. Currently the following are explicitly supported: * Numeric constants (integer and float) * String constants * ``True``, ``False``, and ``None`` -* Simple symbolic constants like ``sys.maxsize``, which must +* Simple symbolic constants like :py:data:`sys.maxsize`, which must start with the name of the module (In the future, this may need to get even more elaborate, @@ -1008,28 +1010,28 @@ Consider the following example: foo: Py_ssize_t = sys.maxsize - 1 -``sys.maxsize`` can have different values on different platforms. Therefore +:py:data:`sys.maxsize` can have different values on different platforms. Therefore Argument Clinic can't simply evaluate that expression locally and hard-code it in C. So it stores the default in such a way that it will get evaluated at runtime, when the user asks for the function's signature. What namespace is available when the expression is evaluated? It's evaluated in the context of the module the builtin came from. So, if your module has an -attribute called "``max_widgets``", you may simply use it: +attribute called :py:attr:`!max_widgets`, you may simply use it: .. code-block:: none foo: Py_ssize_t = max_widgets If the symbol isn't found in the current module, it fails over to looking in -``sys.modules``. That's how it can find ``sys.maxsize`` for example. (Since you -don't know in advance what modules the user will load into their interpreter, +:py:data:`sys.modules`. That's how it can find :py:data:`sys.maxsize` for example. +(Since you don't know in advance what modules the user will load into their interpreter, it's best to restrict yourself to modules that are preloaded by Python itself.) Evaluating default values only at runtime means Argument Clinic can't compute the correct equivalent C default value. So you need to tell it explicitly. When you use an expression, you must also specify the equivalent expression -in C, using the ``c_default`` parameter to the converter: +in C, using the *c_default* parameter to the converter: .. code-block:: none @@ -1095,7 +1097,7 @@ indicate an error has occurred? Normally, a function returns a valid (non-``NUL pointer for success, and ``NULL`` for failure. But if you use an integer return converter, all integers are valid. How can Argument Clinic detect an error? Its solution: each return converter implicitly looks for a special value that indicates an error. If you return -that value, and an error has been set (``PyErr_Occurred()`` returns a true +that value, and an error has been set (c:func:`PyErr_Occurred` returns a true value), then the generated code will propagate the error. Otherwise it will encode the value you return like normal. @@ -1201,9 +1203,9 @@ using a default converter. It automatically sets the ``type`` of this parameter to the "pointer to an instance" you specified when you declared the type. However, you can override Argument Clinic's converter and specify one yourself. -Just add your own ``self`` parameter as the first parameter in a +Just add your own *self* parameter as the first parameter in a block, and ensure that its converter is an instance of -``self_converter`` or a subclass thereof. +:class:`!self_converter` or a subclass thereof. What's the point? This lets you override the type of ``self``, or give it a different default name. @@ -1211,7 +1213,7 @@ or give it a different default name. How do you specify the custom type you want to cast ``self`` to? If you only have one or two functions with the same type for ``self``, you can directly use Argument Clinic's existing ``self`` converter, -passing in the type you want to use as the ``type`` parameter:: +passing in the type you want to use as the *type* parameter:: /*[clinic input] @@ -1226,7 +1228,7 @@ passing in the type you want to use as the ``type`` parameter:: On the other hand, if you have a lot of functions that will use the same type for ``self``, it's best to create your own converter, subclassing -``self_converter`` but overwriting the ``type`` member:: +:class:`!self_converter` but overwriting the :py:attr:`!type` member:: /*[python input] class PicklerObject_converter(self_converter): @@ -1254,8 +1256,8 @@ module level state. Use :c:func:`PyType_FromModuleAndSpec` to associate a new heap type with a module. You can now use :c:func:`PyType_GetModuleState` on the defining class to fetch the module state, for example from a module method. -Example from ``Modules/zlibmodule.c``. First, ``defining_class`` is added to -the clinic input:: +Example from :source:`Modules/zlibmodule.c`. +First, ``defining_class`` is added to the clinic input:: /*[clinic input] zlib.Compress.compress @@ -1285,16 +1287,17 @@ module state:: Each method may only have one argument using this converter, and it must appear after ``self``, or, if ``self`` is not used, as the first argument. The argument will be of type ``PyTypeObject *``. The argument will not appear in the -``__text_signature__``. +:py:attr:`!__text_signature__`. -The ``defining_class`` converter is not compatible with ``__init__`` and ``__new__`` -methods, which cannot use the ``METH_METHOD`` convention. +The ``defining_class`` converter is not compatible with :py:meth:`!__init__` +and :py:meth:`!__new__` methods, which cannot use the :c:macro:`METH_METHOD` +convention. It is not possible to use ``defining_class`` with slot methods. In order to fetch the module state from such methods, use :c:func:`PyType_GetModuleByDef` to look up the module and then :c:func:`PyModule_GetState` to fetch the module state. Example from the ``setattro`` slot method in -``Modules/_threadmodule.c``:: +:source:`Modules/_threadmodule.c`:: static int local_setattro(localobject *self, PyObject *name, PyObject *v) @@ -1312,7 +1315,7 @@ How to write a custom converter ------------------------------- As we hinted at in the previous section... you can write your own converters! -A converter is simply a Python class that inherits from ``CConverter``. +A converter is simply a Python class that inherits from :py:class:`!CConverter`. The main purpose of a custom converter is if you have a parameter using the ``O&`` format unit—parsing this parameter means calling a :c:func:`PyArg_ParseTuple` "converter function". @@ -1323,61 +1326,74 @@ will be automatically registered with Argument Clinic; its name will be the name of your class with the ``_converter`` suffix stripped off. (This is accomplished with a metaclass.) -You shouldn't subclass ``CConverter.__init__``. Instead, you should -write a ``converter_init()`` function. ``converter_init()`` -always accepts a ``self`` parameter; after that, all additional +You shouldn't subclass :py:meth:`!CConverter.__init__`. Instead, you should +write a :py:meth:`!converter_init` function. :py:meth:`!converter_init` +always accepts a *self* parameter; after that, all additional parameters *must* be keyword-only. Any arguments passed in to the converter in Argument Clinic will be passed along to your -``converter_init()``. +:py:meth:`!converter_init`. -There are some additional members of ``CConverter`` you may wish +There are some additional members of :py:class:`!CConverter` you may wish to specify in your subclass. Here's the current list: -``type`` - The C type to use for this variable. - ``type`` should be a Python string specifying the type, e.g. ``int``. - If this is a pointer type, the type string should end with ``' *'``. +.. module:: clinic -``default`` - The Python default value for this parameter, as a Python value. - Or the magic value ``unspecified`` if there is no default. +.. class:: CConverter -``py_default`` - ``default`` as it should appear in Python code, - as a string. - Or ``None`` if there is no default. + .. attribute:: type -``c_default`` - ``default`` as it should appear in C code, - as a string. - Or ``None`` if there is no default. + The C type to use for this variable. + :attr:`!type` should be a Python string specifying the type, + e.g. ``'int'``. + If this is a pointer type, the type string should end with ``' *'``. -``c_ignored_default`` - The default value used to initialize the C variable when - there is no default, but not specifying a default may - result in an "uninitialized variable" warning. This can - easily happen when using option groups—although - properly written code will never actually use this value, - the variable does get passed in to the impl, and the - C compiler will complain about the "use" of the - uninitialized value. This value should always be a - non-empty string. + .. attribute:: default -``converter`` - The name of the C converter function, as a string. + The Python default value for this parameter, as a Python value. + Or the magic value ``unspecified`` if there is no default. -``impl_by_reference`` - A boolean value. If true, - Argument Clinic will add a ``&`` in front of the name of - the variable when passing it into the impl function. + .. attribute:: py_default -``parse_by_reference`` - A boolean value. If true, - Argument Clinic will add a ``&`` in front of the name of - the variable when passing it into :c:func:`PyArg_ParseTuple`. + :attr:`!default` as it should appear in Python code, + as a string. + Or ``None`` if there is no default. + .. attribute:: c_default -Here's the simplest example of a custom converter, from ``Modules/zlibmodule.c``:: + :attr:`!default` as it should appear in C code, + as a string. + Or ``None`` if there is no default. + + .. attribute:: c_ignored_default + + The default value used to initialize the C variable when + there is no default, but not specifying a default may + result in an "uninitialized variable" warning. This can + easily happen when using option groups—although + properly written code will never actually use this value, + the variable does get passed in to the impl, and the + C compiler will complain about the "use" of the + uninitialized value. This value should always be a + non-empty string. + + .. attribute:: converter + + The name of the C converter function, as a string. + + .. attribute:: impl_by_reference + + A boolean value. If true, + Argument Clinic will add a ``&`` in front of the name of + the variable when passing it into the impl function. + + .. attribute:: parse_by_reference + + A boolean value. If true, + Argument Clinic will add a ``&`` in front of the name of + the variable when passing it into :c:func:`PyArg_ParseTuple`. + + +Here's the simplest example of a custom converter, from :source:`Modules/zlibmodule.c`:: /*[python input] @@ -1397,7 +1413,7 @@ automatically support default values. More sophisticated custom converters can insert custom C code to handle initialization and cleanup. You can see more examples of custom converters in the CPython -source tree; grep the C files for the string ``CConverter``. +source tree; grep the C files for the string :py:class:`!CConverter`. How to write a custom return converter @@ -1407,18 +1423,18 @@ Writing a custom return converter is much like writing a custom converter. Except it's somewhat simpler, because return converters are themselves much simpler. -Return converters must subclass ``CReturnConverter``. +Return converters must subclass :py:class:`!CReturnConverter`. There are no examples yet of custom return converters, because they are not widely used yet. If you wish to -write your own return converter, please read ``Tools/clinic/clinic.py``, -specifically the implementation of ``CReturnConverter`` and +write your own return converter, please read :source:`Tools/clinic/clinic.py`, +specifically the implementation of :py:class:`!CReturnConverter` and all its subclasses. How to convert ``METH_O`` and ``METH_NOARGS`` functions ------------------------------------------------------- -To convert a function using ``METH_O``, make sure the function's +To convert a function using :c:macro:`METH_O`, make sure the function's single argument is using the ``object`` converter, and mark the arguments as positional-only:: @@ -1430,24 +1446,25 @@ arguments as positional-only:: [clinic start generated code]*/ -To convert a function using ``METH_NOARGS``, just don't specify +To convert a function using :c:macro:`METH_NOARGS`, just don't specify any arguments. You can still use a self converter, a return converter, and specify -a ``type`` argument to the object converter for ``METH_O``. +a *type* argument to the object converter for :c:macro:`METH_O`. How to convert ``tp_new`` and ``tp_init`` functions --------------------------------------------------- -You can convert ``tp_new`` and ``tp_init`` functions. Just name -them ``__new__`` or ``__init__`` as appropriate. Notes: +You can convert :c:member:`~PyTypeObject.tp_new` and +:c:member:`~PyTypeObject.tp_init` functions. +Just name them ``__new__`` or ``__init__`` as appropriate. Notes: * The function name generated for ``__new__`` doesn't end in ``__new__`` like it would by default. It's just the name of the class, converted into a valid C identifier. -* No ``PyMethodDef`` ``#define`` is generated for these functions. +* No :c:type:`PyMethodDef` ``#define`` is generated for these functions. * ``__init__`` functions return ``int``, not ``PyObject *``. @@ -1482,7 +1499,7 @@ Let's start with defining some terminology: *field* A field, in this context, is a subsection of Clinic's output. - For example, the ``#define`` for the ``PyMethodDef`` structure + For example, the ``#define`` for the :c:type:`PyMethodDef` structure is a field, called ``methoddef_define``. Clinic has seven different fields it can output per function definition: @@ -1526,8 +1543,8 @@ Let's start with defining some terminology: The filename chosen for the file is ``{basename}.clinic{extension}``, where ``basename`` and ``extension`` were assigned the output from ``os.path.splitext()`` run on the current file. (Example: - the ``file`` destination for ``_pickle.c`` would be written to - ``_pickle.clinic.c``.) + the ``file`` destination for :file:`_pickle.c` would be written to + :file:`_pickle.clinic.c`.) **Important: When using a** ``file`` **destination, you** *must check in* **the generated file!** @@ -1780,7 +1797,7 @@ like so:: } #endif /* HAVE_FUNCTIONNAME */ -Then, remove those three lines from the ``PyMethodDef`` structure, +Then, remove those three lines from the :c:type:`PyMethodDef` structure, replacing them with the macro Argument Clinic generated: .. code-block:: none @@ -1821,7 +1838,7 @@ This may mean that you get a complaint from Argument Clinic: When this happens, just open your file, find the ``dump buffer`` block that Argument Clinic added to your file (it'll be at the very bottom), then -move it above the ``PyMethodDef`` structure where that macro is used. +move it above the :c:type:`PyMethodDef` structure where that macro is used. How to use Argument Clinic in Python files From 6dc096d10726de6a48ec043e08175eab4965b52d Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Wed, 26 Jul 2023 22:54:25 +0200 Subject: [PATCH 59/71] Docs: Remove the numbered steps from the Argument Clinic tutorial (#107203) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Instead, order the tutorial as one body of prose, making it easier to align the tutorial according to Diátaxis principles. --- Doc/howto/clinic.rst | 624 +++++++++++++++++++++---------------------- 1 file changed, 310 insertions(+), 314 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 98d3632ff..ea3b4537d 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -167,23 +167,23 @@ convert a function to work with it. Here, then, are the bare minimum steps you'd need to follow to convert a function to work with Argument Clinic. Note that for code you plan to check in to CPython, you really should take the conversion farther, -using some of the advanced concepts you'll see later on in -the document (like "return converters" and "self converters"). +using some of the :ref:`advanced concepts ` +you'll see later on in the document, +like :ref:`clinic-howto-return-converters` +and :ref:`clinic-howto-self-converter`. But we'll keep it simple for this walkthrough so you can learn. -Let's dive in! +First, make sure you're working with a freshly updated checkout +of the CPython trunk. -0. Make sure you're working with a freshly updated checkout - of the CPython trunk. +Next, find a Python builtin that calls either :c:func:`PyArg_ParseTuple` +or :c:func:`PyArg_ParseTupleAndKeywords`, and hasn't been converted +to work with Argument Clinic yet. +For this tutorial, we'll be using +:py:meth:`_pickle.Pickler.dump `. -1. Find a Python builtin that calls either :c:func:`PyArg_ParseTuple` - or :c:func:`PyArg_ParseTupleAndKeywords`, and hasn't been converted - to work with Argument Clinic yet. - For my example I'm using - :py:meth:`_pickle.Pickler.dump `. - -2. If the call to the :c:func:`!PyArg_Parse*` function uses any of the - following format units: +If the call to the :c:func:`!PyArg_Parse*` function uses any of the +following format units...: .. code-block:: none @@ -194,388 +194,377 @@ Let's dive in! et et# - or if it has multiple calls to :c:func:`PyArg_ParseTuple`, - you should choose a different function. Argument Clinic *does* - support all of these scenarios. But these are advanced - topics—let's do something simpler for your first function. - - Also, if the function has multiple calls to :c:func:`!PyArg_ParseTuple` - or :c:func:`PyArg_ParseTupleAndKeywords` where it supports different - types for the same argument, or if the function uses something besides - :c:func:`!PyArg_Parse*` functions to parse its arguments, it probably - isn't suitable for conversion to Argument Clinic. Argument Clinic - doesn't support generic functions or polymorphic parameters. - -3. Add the following boilerplate above the function, creating our block:: - - /*[clinic input] - [clinic start generated code]*/ - -4. Cut the docstring and paste it in between the ``[clinic]`` lines, - removing all the junk that makes it a properly quoted C string. - When you're done you should have just the text, based at the left - margin, with no line wider than 80 characters. - (Argument Clinic will preserve indents inside the docstring.) - - If the old docstring had a first line that looked like a function - signature, throw that line away. (The docstring doesn't need it - anymore—when you use :py:func:`help` on your builtin in the future, - the first line will be built automatically based on the function's - signature.) - - Sample:: +... or if it has multiple calls to :c:func:`PyArg_ParseTuple`, +you should choose a different function. +(See :ref:`clinic-howto-advanced-converters` for those scenarios.) - /*[clinic input] - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ - -5. If your docstring doesn't have a "summary" line, Argument Clinic will - complain. So let's make sure it has one. The "summary" line should - be a paragraph consisting of a single 80-column line - at the beginning of the docstring. - - (Our example docstring consists solely of a summary line, so the sample - code doesn't have to change for this step.) +Also, if the function has multiple calls to :c:func:`!PyArg_ParseTuple` +or :c:func:`PyArg_ParseTupleAndKeywords` where it supports different +types for the same argument, or if the function uses something besides +:c:func:`!PyArg_Parse*` functions to parse its arguments, it probably +isn't suitable for conversion to Argument Clinic. Argument Clinic +doesn't support generic functions or polymorphic parameters. -6. Above the docstring, enter the name of the function, followed - by a blank line. This should be the Python name of the function, - and should be the full dotted path - to the function—it should start with the name of the module, - include any sub-modules, and if the function is a method on - a class it should include the class name too. - - Sample:: +Next, add the following boilerplate above the function, +creating our input block:: /*[clinic input] - _pickle.Pickler.dump - - Write a pickled representation of obj to the open file. [clinic start generated code]*/ -7. If this is the first time that module or class has been used with Argument - Clinic in this C file, - you must declare the module and/or class. Proper Argument Clinic hygiene - prefers declaring these in a separate block somewhere near the - top of the C file, in the same way that include files and statics go at - the top. (In our sample code we'll just show the two blocks next to - each other.) +Cut the docstring and paste it in between the ``[clinic]`` lines, +removing all the junk that makes it a properly quoted C string. +When you're done you should have just the text, based at the left +margin, with no line wider than 80 characters. +Argument Clinic will preserve indents inside the docstring. - The name of the class and module should be the same as the one - seen by Python. Check the name defined in the :c:type:`PyModuleDef` - or :c:type:`PyTypeObject` as appropriate. +If the old docstring had a first line that looked like a function +signature, throw that line away; The docstring doesn't need it anymore --- +when you use :py:func:`help` on your builtin in the future, +the first line will be built automatically based on the function's signature. - When you declare a class, you must also specify two aspects of its type - in C: the type declaration you'd use for a pointer to an instance of - this class, and a pointer to the :c:type:`!PyTypeObject` for this class. +Example docstring summary line:: - Sample:: - - /*[clinic input] - module _pickle - class _pickle.Pickler "PicklerObject *" "&Pickler_Type" - [clinic start generated code]*/ - - /*[clinic input] - _pickle.Pickler.dump - - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ - - - - -8. Declare each of the parameters to the function. Each parameter - should get its own line. All the parameter lines should be - indented from the function name and the docstring. - - The general form of these parameter lines is as follows: - - .. code-block:: none - - name_of_parameter: converter - - If the parameter has a default value, add that after the - converter: - - .. code-block:: none + /*[clinic input] + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ - name_of_parameter: converter = default_value +If your docstring doesn't have a "summary" line, Argument Clinic will +complain, so let's make sure it has one. The "summary" line should +be a paragraph consisting of a single 80-column line +at the beginning of the docstring. +(See :pep:`257` regarding docstring conventions.) - Argument Clinic's support for "default values" is quite sophisticated; - please see :ref:`the section below on default values ` - for more information. +Our example docstring consists solely of a summary line, so the sample +code doesn't have to change for this step. - Add a blank line below the parameters. +Now, above the docstring, enter the name of the function, followed +by a blank line. This should be the Python name of the function, +and should be the full dotted path to the function --- +it should start with the name of the module, +include any sub-modules, and if the function is a method on +a class it should include the class name too. - What's a "converter"? It establishes both the type - of the variable used in C, and the method to convert the Python - value into a C value at runtime. - For now you're going to use what's called a "legacy converter"—a - convenience syntax intended to make porting old code into Argument - Clinic easier. +In our example, :mod:`!_pickle` is the module, :py:class:`!Pickler` is the class, +and :py:meth:`!dump` is the method, so the name becomes +:py:meth:`!_pickle.Pickler.dump`:: - For each parameter, copy the "format unit" for that - parameter from the :c:func:`PyArg_Parse` format argument and - specify *that* as its converter, as a quoted - string. ("format unit" is the formal name for the one-to-three - character substring of the *format* parameter that tells - the argument parsing function what the type of the variable - is and how to convert it. For more on format units please - see :ref:`arg-parsing`.) + /*[clinic input] + _pickle.Pickler.dump - For multicharacter format units like ``z#``, use the - entire two-or-three character string. + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ - Sample:: +If this is the first time that module or class has been used with Argument +Clinic in this C file, +you must declare the module and/or class. Proper Argument Clinic hygiene +prefers declaring these in a separate block somewhere near the +top of the C file, in the same way that include files and statics go at +the top. +In our sample code we'll just show the two blocks next to each other. - /*[clinic input] - module _pickle - class _pickle.Pickler "PicklerObject *" "&Pickler_Type" - [clinic start generated code]*/ +The name of the class and module should be the same as the one +seen by Python. Check the name defined in the :c:type:`PyModuleDef` +or :c:type:`PyTypeObject` as appropriate. - /*[clinic input] - _pickle.Pickler.dump +When you declare a class, you must also specify two aspects of its type +in C: the type declaration you'd use for a pointer to an instance of +this class, and a pointer to the :c:type:`!PyTypeObject` for this class:: - obj: 'O' + /*[clinic input] + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" + [clinic start generated code]*/ - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ + /*[clinic input] + _pickle.Pickler.dump -9. If your function has ``|`` in the format string, meaning some - parameters have default values, you can ignore it. Argument - Clinic infers which parameters are optional based on whether - or not they have default values. + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ - If your function has ``$`` in the format string, meaning it - takes keyword-only arguments, specify ``*`` on a line by - itself before the first keyword-only argument, indented the - same as the parameter lines. +Declare each of the parameters to the function. Each parameter +should get its own line. All the parameter lines should be +indented from the function name and the docstring. +The general form of these parameter lines is as follows: - (:py:meth:`!_pickle.Pickler.dump` has neither, so our sample is unchanged.) +.. code-block:: none + name_of_parameter: converter -10. If the existing C function calls :c:func:`PyArg_ParseTuple` - (as opposed to :c:func:`PyArg_ParseTupleAndKeywords`), then all its - arguments are positional-only. +If the parameter has a default value, add that after the +converter: - To mark all parameters as positional-only in Argument Clinic, - add a ``/`` on a line by itself after the last parameter, - indented the same as the parameter lines. +.. code-block:: none - Currently this is all-or-nothing; either all parameters are - positional-only, or none of them are. (In the future Argument - Clinic may relax this restriction.) + name_of_parameter: converter = default_value - Sample:: +Argument Clinic's support for "default values" is quite sophisticated; +see :ref:`clinic-howto-default-values` for more information. - /*[clinic input] - module _pickle - class _pickle.Pickler "PicklerObject *" "&Pickler_Type" - [clinic start generated code]*/ +Next, add a blank line below the parameters. - /*[clinic input] - _pickle.Pickler.dump +What's a "converter"? +It establishes both the type of the variable used in C, +and the method to convert the Python value into a C value at runtime. +For now you're going to use what's called a "legacy converter" --- +a convenience syntax intended to make porting old code into Argument +Clinic easier. - obj: 'O' - / +For each parameter, copy the "format unit" for that +parameter from the :c:func:`PyArg_Parse` format argument and +specify *that* as its converter, as a quoted string. +The "format unit" is the formal name for the one-to-three +character substring of the *format* parameter that tells +the argument parsing function what the type of the variable +is and how to convert it. +For more on format units please see :ref:`arg-parsing`. - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ +For multicharacter format units like ``z#``, +use the entire two-or-three character string. -11. It's helpful to write a per-parameter docstring for each parameter. - But per-parameter docstrings are optional; you can skip this step - if you prefer. +Sample:: - Here's how to add a per-parameter docstring. The first line - of the per-parameter docstring must be indented further than the - parameter definition. The left margin of this first line establishes - the left margin for the whole per-parameter docstring; all the text - you write will be outdented by this amount. You can write as much - text as you like, across multiple lines if you wish. + /*[clinic input] + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" + [clinic start generated code]*/ - Sample:: + /*[clinic input] + _pickle.Pickler.dump - /*[clinic input] - module _pickle - class _pickle.Pickler "PicklerObject *" "&Pickler_Type" - [clinic start generated code]*/ + obj: 'O' - /*[clinic input] - _pickle.Pickler.dump + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ - obj: 'O' - The object to be pickled. - / +If your function has ``|`` in the format string, +meaning some parameters have default values, you can ignore it. +Argument Clinic infers which parameters are optional +based on whether or not they have default values. - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ +If your function has ``$`` in the format string, +meaning it takes keyword-only arguments, +specify ``*`` on a line by itself before the first keyword-only argument, +indented the same as the parameter lines. -12. Save and close the file, then run ``Tools/clinic/clinic.py`` on - it. With luck everything worked---your block now has output, and - a :file:`.c.h` file has been generated! Reopen the file in your - text editor to see:: +:py:meth:`!_pickle.Pickler.dump` has neither, so our sample is unchanged. - /*[clinic input] - _pickle.Pickler.dump +Next, if the existing C function calls :c:func:`PyArg_ParseTuple` +(as opposed to :c:func:`PyArg_ParseTupleAndKeywords`), then all its +arguments are positional-only. - obj: 'O' - The object to be pickled. - / +To mark parameters as positional-only in Argument Clinic, +add a ``/`` on a line by itself after the last positional-only parameter, +indented the same as the parameter lines. - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ +Sample:: - static PyObject * - _pickle_Pickler_dump(PicklerObject *self, PyObject *obj) - /*[clinic end generated code: output=87ecad1261e02ac7 input=552eb1c0f52260d9]*/ + /*[clinic input] + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" + [clinic start generated code]*/ - Obviously, if Argument Clinic didn't produce any output, it's because - it found an error in your input. Keep fixing your errors and retrying - until Argument Clinic processes your file without complaint. + /*[clinic input] + _pickle.Pickler.dump - For readability, most of the glue code has been generated to a :file:`.c.h` - file. You'll need to include that in your original :file:`.c` file, - typically right after the clinic module block:: + obj: 'O' + / - #include "clinic/_pickle.c.h" + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ -13. Double-check that the argument-parsing code Argument Clinic generated - looks basically the same as the existing code. +It can be helpful to write a per-parameter docstring for each parameter. +Since per-parameter docstrings are optional, +you can skip this step if you prefer. - First, ensure both places use the same argument-parsing function. - The existing code must call either - :c:func:`PyArg_ParseTuple` or :c:func:`PyArg_ParseTupleAndKeywords`; - ensure that the code generated by Argument Clinic calls the - *exact* same function. +Nevertheless, here's how to add a per-parameter docstring. +The first line of the per-parameter docstring +must be indented further than the parameter definition. +The left margin of this first line establishes +the left margin for the whole per-parameter docstring; +all the text you write will be outdented by this amount. +You can write as much text as you like, across multiple lines if you wish. - Second, the format string passed in to :c:func:`!PyArg_ParseTuple` or - :c:func:`!PyArg_ParseTupleAndKeywords` should be *exactly* the same - as the hand-written one in the existing function, up to the colon - or semi-colon. +Sample:: - (Argument Clinic always generates its format strings - with a ``:`` followed by the name of the function. If the - existing code's format string ends with ``;``, to provide - usage help, this change is harmless—don't worry about it.) + /*[clinic input] + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" + [clinic start generated code]*/ - Third, for parameters whose format units require two arguments - (like a length variable, or an encoding string, or a pointer - to a conversion function), ensure that the second argument is - *exactly* the same between the two invocations. + /*[clinic input] + _pickle.Pickler.dump - Fourth, inside the output portion of the block you'll find a preprocessor - macro defining the appropriate static :c:type:`PyMethodDef` structure for - this builtin:: + obj: 'O' + The object to be pickled. + / - #define __PICKLE_PICKLER_DUMP_METHODDEF \ - {"dump", (PyCFunction)__pickle_Pickler_dump, METH_O, __pickle_Pickler_dump__doc__}, + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ - This static structure should be *exactly* the same as the existing static - :c:type:`!PyMethodDef` structure for this builtin. +Save and close the file, then run ``Tools/clinic/clinic.py`` on it. +With luck everything worked---your block now has output, +and a :file:`.c.h` file has been generated! +Reload the file in your text editor to see the generated code:: - If any of these items differ in *any way*, - adjust your Argument Clinic function specification and rerun - ``Tools/clinic/clinic.py`` until they *are* the same. + /*[clinic input] + _pickle.Pickler.dump + obj: 'O' + The object to be pickled. + / -14. Notice that the last line of its output is the declaration - of your "impl" function. This is where the builtin's implementation goes. - Delete the existing prototype of the function you're modifying, but leave - the opening curly brace. Now delete its argument parsing code and the - declarations of all the variables it dumps the arguments into. - Notice how the Python arguments are now arguments to this impl function; - if the implementation used different names for these variables, fix it. + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ - Let's reiterate, just because it's kind of weird. Your code should now - look like this:: + static PyObject * + _pickle_Pickler_dump(PicklerObject *self, PyObject *obj) + /*[clinic end generated code: output=87ecad1261e02ac7 input=552eb1c0f52260d9]*/ - static return_type - your_function_impl(...) - /*[clinic end generated code: checksum=...]*/ - { - ... +Obviously, if Argument Clinic didn't produce any output, +it's because it found an error in your input. +Keep fixing your errors and retrying until Argument Clinic processes your file +without complaint. - Argument Clinic generated the checksum line and the function prototype just - above it. You should write the opening (and closing) curly braces for the - function, and the implementation inside. +For readability, most of the glue code has been generated to a :file:`.c.h` +file. You'll need to include that in your original :file:`.c` file, +typically right after the clinic module block:: - Sample:: + #include "clinic/_pickle.c.h" - /*[clinic input] - module _pickle - class _pickle.Pickler "PicklerObject *" "&Pickler_Type" - [clinic start generated code]*/ - /*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ +Double-check that the argument-parsing code Argument Clinic generated +looks basically the same as the existing code. - /*[clinic input] - _pickle.Pickler.dump +First, ensure both places use the same argument-parsing function. +The existing code must call either +:c:func:`PyArg_ParseTuple` or :c:func:`PyArg_ParseTupleAndKeywords`; +ensure that the code generated by Argument Clinic calls the +*exact* same function. - obj: 'O' - The object to be pickled. - / +Second, the format string passed in to :c:func:`!PyArg_ParseTuple` or +:c:func:`!PyArg_ParseTupleAndKeywords` should be *exactly* the same +as the hand-written one in the existing function, +up to the colon or semi-colon. - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ +Argument Clinic always generates its format strings +with a ``:`` followed by the name of the function. +If the existing code's format string ends with ``;``, +to provide usage help, this change is harmless --- don't worry about it. - PyDoc_STRVAR(__pickle_Pickler_dump__doc__, - "Write a pickled representation of obj to the open file.\n" - "\n" - ... - static PyObject * - _pickle_Pickler_dump_impl(PicklerObject *self, PyObject *obj) - /*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ - { - /* Check whether the Pickler was initialized correctly (issue3664). - Developers often forget to call __init__() in their subclasses, which - would trigger a segfault without this check. */ - if (self->write == NULL) { - PyErr_Format(PicklingError, - "Pickler.__init__() was not called by %s.__init__()", - Py_TYPE(self)->tp_name); - return NULL; - } +Third, for parameters whose format units require two arguments, +like a length variable, an encoding string, or a pointer +to a conversion function, ensure that the second argument is +*exactly* the same between the two invocations. - if (_Pickler_ClearBuffer(self) < 0) - return NULL; +Fourth, inside the output portion of the block, +you'll find a preprocessor macro defining the appropriate static +:c:type:`PyMethodDef` structure for this builtin:: - ... + #define __PICKLE_PICKLER_DUMP_METHODDEF \ + {"dump", (PyCFunction)__pickle_Pickler_dump, METH_O, __pickle_Pickler_dump__doc__}, -15. Remember the macro with the :c:type:`PyMethodDef` structure for this - function? Find the existing :c:type:`!PyMethodDef` structure for this - function and replace it with a reference to the macro. (If the builtin - is at module scope, this will probably be very near the end of the file; - if the builtin is a class method, this will probably be below but relatively - near to the implementation.) +This static structure should be *exactly* the same as the existing static +:c:type:`!PyMethodDef` structure for this builtin. - Note that the body of the macro contains a trailing comma. So when you - replace the existing static :c:type:`!PyMethodDef` structure with the macro, - *don't* add a comma to the end. +If any of these items differ in *any way*, +adjust your Argument Clinic function specification and rerun +``Tools/clinic/clinic.py`` until they *are* the same. - Sample:: +Notice that the last line of its output is the declaration +of your "impl" function. This is where the builtin's implementation goes. +Delete the existing prototype of the function you're modifying, but leave +the opening curly brace. Now delete its argument parsing code and the +declarations of all the variables it dumps the arguments into. +Notice how the Python arguments are now arguments to this impl function; +if the implementation used different names for these variables, fix it. - static struct PyMethodDef Pickler_methods[] = { - __PICKLE_PICKLER_DUMP_METHODDEF - __PICKLE_PICKLER_CLEAR_MEMO_METHODDEF - {NULL, NULL} /* sentinel */ - }; +Let's reiterate, just because it's kind of weird. +Your code should now look like this:: + static return_type + your_function_impl(...) + /*[clinic end generated code: input=..., output=...]*/ + { + ... -16. Argument Clinic may generate new instances of ``_Py_ID``. For example:: +Argument Clinic generated the checksum line and the function prototype just +above it. You should write the opening and closing curly braces for the +function, and the implementation inside. - &_Py_ID(new_unique_py_id) +Sample:: - If it does, you'll have to run ``make regen-global-objects`` - to regenerate the list of precompiled identifiers at this point. + /*[clinic input] + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" + [clinic start generated code]*/ + /*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ + /*[clinic input] + _pickle.Pickler.dump -17. Compile, then run the relevant portions of the regression-test suite. - This change should not introduce any new compile-time warnings or errors, - and there should be no externally visible change to Python's behavior. + obj: 'O' + The object to be pickled. + / - Well, except for one difference: :py:func:`inspect.signature` run on your function - should now provide a valid signature! + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ - Congratulations, you've ported your first function to work with Argument Clinic! + PyDoc_STRVAR(__pickle_Pickler_dump__doc__, + "Write a pickled representation of obj to the open file.\n" + "\n" + ... + static PyObject * + _pickle_Pickler_dump_impl(PicklerObject *self, PyObject *obj) + /*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ + { + /* Check whether the Pickler was initialized correctly (issue3664). + Developers often forget to call __init__() in their subclasses, which + would trigger a segfault without this check. */ + if (self->write == NULL) { + PyErr_Format(PicklingError, + "Pickler.__init__() was not called by %s.__init__()", + Py_TYPE(self)->tp_name); + return NULL; + } + + if (_Pickler_ClearBuffer(self) < 0) { + return NULL; + } + + ... + +Remember the macro with the :c:type:`PyMethodDef` structure for this function? +Find the existing :c:type:`!PyMethodDef` structure for this +function and replace it with a reference to the macro. If the builtin +is at module scope, this will probably be very near the end of the file; +if the builtin is a class method, this will probably be below but relatively +near to the implementation. + +Note that the body of the macro contains a trailing comma; when you +replace the existing static :c:type:`!PyMethodDef` structure with the macro, +*don't* add a comma to the end. + +Sample:: + + static struct PyMethodDef Pickler_methods[] = { + __PICKLE_PICKLER_DUMP_METHODDEF + __PICKLE_PICKLER_CLEAR_MEMO_METHODDEF + {NULL, NULL} /* sentinel */ + }; + +Argument Clinic may generate new instances of ``_Py_ID``. For example:: + + &_Py_ID(new_unique_py_id) + +If it does, you'll have to run ``make regen-global-objects`` +to regenerate the list of precompiled identifiers at this point. + +Finally, compile, then run the relevant portions of the regression-test suite. +This change should not introduce any new compile-time warnings or errors, +and there should be no externally visible change to Python's behavior, +except for one difference: :py:func:`inspect.signature` run on your function +should now provide a valid signature! + +Congratulations, you've ported your first function to work with Argument Clinic! .. _clinic-howtos: @@ -913,6 +902,8 @@ you *must* not call :c:func:`PyBuffer_Release` on the provided buffer. Argument Clinic generates code that does it for you (in the parsing function). +.. _clinic-howto-advanced-converters: + How to use advanced converters ------------------------------ @@ -943,6 +934,7 @@ This restriction doesn't seem unreasonable; CPython itself always passes in stat hard-coded encoding strings for parameters whose format units start with ``e``. +.. _clinic-howto-default-values: .. _default_values: How to assign default values to parameter @@ -1053,6 +1045,8 @@ you're not permitted to use: * Tuple/list/set/dict literals. +.. _clinic-howto-return-converters: + How to use return converters ---------------------------- @@ -1195,6 +1189,8 @@ variable to the C code:: /*[python checksum:...]*/ +.. _clinic-howto-self-converter: + How to use the "self converter" ------------------------------- From 8c81360d40f0b17f90ef194cb7ea9d4fdacc6586 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Thu, 27 Jul 2023 00:08:43 +0200 Subject: [PATCH 60/71] Docs: Argument Clinic: Restructure "Basic concepts and usage" (#106981) Split "Basic concepts and usage" into: - Reference - Terminology - CLI reference - Background - Basic concepts Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Alex Waygood Co-authored-by: Ezio Melotti --- Doc/howto/clinic.rst | 172 ++++++++++++++++++++++++++++++++----------- 1 file changed, 129 insertions(+), 43 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index ea3b4537d..7aafd4871 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -8,6 +8,7 @@ Argument Clinic How-To :author: Larry Hastings +**Source code:** :source:`Tools/clinic/clinic.py`. .. topic:: Abstract @@ -15,10 +16,12 @@ Argument Clinic How-To Its purpose is to automate all the boilerplate involved with writing argument parsing code for "builtins", module level functions, and class methods. - This document is divided in three major sections: + This document is divided in four major sections: * :ref:`clinic-background` talks about the basic concepts and goals of Argument Clinic. + * :ref:`clinic-reference` describes the command-line interface and Argument + Clinic terminology. * :ref:`clinic-tutorial` guides you through all the steps required to adapt an existing C function to Argument Clinic. * :ref:`clinic-howtos` details how to handle specific tasks. @@ -93,39 +96,29 @@ and it should be able to do many interesting and smart things with all the information you give it. -Basic concepts and usage ------------------------- +Basic concepts +-------------- -Argument Clinic ships with CPython; you'll find it in -:source:`Tools/clinic/clinic.py`. -If you run that script, specifying a C file as an argument: - -.. code-block:: shell-session - - $ python Tools/clinic/clinic.py foo.c - -Argument Clinic will scan over the file looking for lines that -look exactly like this: +When Argument Clinic is run on a file, either via the :ref:`clinic-cli` +or via ``make clinic``, it will scan over the input files looking for +:term:`start lines `: .. code-block:: none /*[clinic input] -When it finds one, it reads everything up to a line that looks -exactly like this: +When it finds one, it reads everything up to the :term:`end line`: .. code-block:: none [clinic start generated code]*/ -Everything in between these two lines is input for Argument Clinic. -All of these lines, including the beginning and ending comment -lines, are collectively called an Argument Clinic "block". - -When Argument Clinic parses one of these blocks, it -generates output. This output is rewritten into the C file -immediately after the block, followed by a comment containing a checksum. -The Argument Clinic block now looks like this: +Everything in between these two lines is Argument Clinic :term:`input`. +When Argument Clinic parses input, it generates :term:`output`. +The output is rewritten into the C file immediately after the input, +followed by a :term:`checksum line`. +All of these lines, including the :term:`start line` and :term:`checksum line`, +are collectively called an Argument Clinic :term:`block`: .. code-block:: none @@ -133,28 +126,121 @@ The Argument Clinic block now looks like this: ... clinic input goes here ... [clinic start generated code]*/ ... clinic output goes here ... - /*[clinic end generated code: checksum=...]*/ + /*[clinic end generated code: ...]*/ If you run Argument Clinic on the same file a second time, Argument Clinic -will discard the old output and write out the new output with a fresh checksum -line. However, if the input hasn't changed, the output won't change either. - -You should never modify the output portion of an Argument Clinic block. Instead, -change the input until it produces the output you want. (That's the purpose of the -checksum—to detect if someone changed the output, as these edits would be lost -the next time Argument Clinic writes out fresh output.) - -For the sake of clarity, here's the terminology we'll use with Argument Clinic: - -* The first line of the comment (``/*[clinic input]``) is the *start line*. -* The last line of the initial comment (``[clinic start generated code]*/``) is the *end line*. -* The last line (``/*[clinic end generated code: checksum=...]*/``) is the *checksum line*. -* In between the start line and the end line is the *input*. -* In between the end line and the checksum line is the *output*. -* All the text collectively, from the start line to the checksum line inclusively, - is the *block*. (A block that hasn't been successfully processed by Argument - Clinic yet doesn't have output or a checksum line, but it's still considered - a block.) +will discard the old :term:`output` and write out the new output with a fresh +:term:`checksum line`. +If the :term:`input` hasn't changed, the output won't change either. + +.. note:: + + You should never modify the output of an Argument Clinic block, + as any change will be lost in future Argument Clinic runs; + Argument Clinic will detect an output checksum mismatch and regenerate the + correct output. + If you are not happy with the generated output, + you should instead change the input until it produces the output you want. + + +.. _clinic-reference: + +Reference +========= + + +.. _clinic-terminology: + +Terminology +----------- + +.. glossary:: + + start line + The line ``/*[clinic input]``. + This line marks the beginning of Argument Clinic input. + Note that the *start line* opens a C block comment. + + end line + The line ``[clinic start generated code]*/``. + The *end line* marks the _end_ of Argument Clinic :term:`input`, + but at the same time marks the _start_ of Argument Clinic :term:`output`, + thus the text *"clinic start start generated code"* + Note that the *end line* closes the C block comment opened + by the *start line*. + + checksum + A hash to distinguish unique :term:`inputs ` + and :term:`outputs `. + + checksum line + A line that looks like ``/*[clinic end generated code: ...]*/``. + The three dots will be replaced by a :term:`checksum` generated from the + :term:`input`, and a :term:`checksum` generated from the :term:`output`. + The checksum line marks the end of Argument Clinic generated code, + and is used by Argument Clinic to determine if it needs to regenerate + output. + + input + The text between the :term:`start line` and the :term:`end line`. + Note that the start and end lines open and close a C block comment; + the *input* is thus a part of that same C block comment. + + output + The text between the :term:`end line` and the :term:`checksum line`. + + block + All text from the :term:`start line` to the :term:`checksum line` inclusively. + + +.. _clinic-cli: + +Command-line interface +---------------------- + +The Argument Clinic :abbr:`CLI (Command-Line Interface)` is typically used to +process a single source file, like this: + +.. code-block:: shell-session + + $ python3 ./Tools/clinic/clinic.py foo.c + +The CLI supports the following options: + +.. program:: ./Tools/clinic/clinic.py [-h] [-f] [-o OUTPUT] [-v] \ + [--converters] [--make] [--srcdir SRCDIR] [FILE ...] + +.. option:: -h, --help + + Print CLI usage. + +.. option:: -f, --force + + Force output regeneration. + +.. option:: -o, --output OUTPUT + + Redirect file output to OUTPUT + +.. option:: -v, --verbose + + Enable verbose mode. + +.. option:: --converters + + Print a list of all supported converters and return converters. + +.. option:: --make + + Walk :option:`--srcdir` to run over all relevant files. + +.. option:: --srcdir SRCDIR + + The directory tree to walk in :option:`--make` mode. + +.. option:: FILE ... + + The list of files to process. .. _clinic-tutorial: From d9390c8aca5367290d237a05f2bcfaa4835fcde0 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Mon, 31 Jul 2023 23:33:13 +0200 Subject: [PATCH 61/71] gh-107507: Replace 'The goals of Argument Clinic' with a summary (#107508) Summarise the goals of Argument Clinic in a single sentence. Mention that Argument Clinic was introduced with PEP-436. --- Doc/howto/clinic.rst | 58 +++----------------------------------------- 1 file changed, 4 insertions(+), 54 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 7aafd4871..9c9a4f45d 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -13,8 +13,10 @@ Argument Clinic How-To .. topic:: Abstract Argument Clinic is a preprocessor for CPython C files. - Its purpose is to automate all the boilerplate involved - with writing argument parsing code for "builtins", + It was introduced in Python 3.4 with :pep:`436`, + in order to provide introspection signatures, + and to generate performant and tailor-made boilerplate code + for argument parsing in CPython builtins, module level functions, and class methods. This document is divided in four major sections: @@ -44,58 +46,6 @@ Argument Clinic How-To Background ========== - -The goals of Argument Clinic ----------------------------- - -Argument Clinic's primary goal -is to take over responsibility for all argument parsing code -inside CPython. This means that, when you convert a function -to work with Argument Clinic, that function should no longer -do any of its own argument parsing—the code generated by -Argument Clinic should be a "black box" to you, where CPython -calls in at the top, and your code gets called at the bottom, -with ``PyObject *args`` (and maybe ``PyObject *kwargs``) -magically converted into the C variables and types you need. - -In order for Argument Clinic to accomplish its primary goal, -it must be easy to use. Currently, working with CPython's -argument parsing library is a chore, requiring maintaining -redundant information in a surprising number of places. -When you use Argument Clinic, you don't have to repeat yourself. - -Obviously, no one would want to use Argument Clinic unless -it's solving their problem—and without creating new problems of -its own. -So it's paramount that Argument Clinic generate correct code. -It'd be nice if the code was faster, too, but at the very least -it should not introduce a major speed regression. (Eventually Argument -Clinic *should* make a major speedup possible—we could -rewrite its code generator to produce tailor-made argument -parsing code, rather than calling the general-purpose CPython -argument parsing library. That would make for the fastest -argument parsing possible!) - -Additionally, Argument Clinic must be flexible enough to -work with any approach to argument parsing. Python has -some functions with some very strange parsing behaviors; -Argument Clinic's goal is to support all of them. - -Finally, the original motivation for Argument Clinic was -to provide introspection "signatures" for CPython builtins. -It used to be, the introspection query functions would throw -an exception if you passed in a builtin. With Argument -Clinic, that's a thing of the past! - -One idea you should keep in mind, as you work with -Argument Clinic: the more information you give it, the -better job it'll be able to do. -Argument Clinic is admittedly relatively simple right -now. But as it evolves it will get more sophisticated, -and it should be able to do many interesting and smart -things with all the information you give it. - - Basic concepts -------------- From 7a96719c6ab3c76b1d294e34298ee8411137f892 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Sat, 5 Aug 2023 22:19:37 +0200 Subject: [PATCH 62/71] Docs: Argument Clinic: Improve 'How to write a custom converter' (#107328) - Omit unneccesary wording and sentences - Don't mention implementation details (no digression, explanation) Co-authored-by: Ezio Melotti --- Doc/howto/clinic.rst | 50 +++++++++++++++++++++++--------------------- 1 file changed, 26 insertions(+), 24 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 9c9a4f45d..dcede13a0 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1343,35 +1343,37 @@ state. Example from the ``setattro`` slot method in See also :pep:`573`. +.. _clinic-howto-custom-converter: + How to write a custom converter ------------------------------- -As we hinted at in the previous section... you can write your own converters! -A converter is simply a Python class that inherits from :py:class:`!CConverter`. -The main purpose of a custom converter is if you have a parameter using -the ``O&`` format unit—parsing this parameter means calling +A converter is a Python class that inherits from :py:class:`!CConverter`. +The main purpose of a custom converter, is for parameters parsed with +the ``O&`` format unit --- parsing such a parameter means calling a :c:func:`PyArg_ParseTuple` "converter function". -Your converter class should be named ``*something*_converter``. -If the name follows this convention, then your converter class -will be automatically registered with Argument Clinic; its name -will be the name of your class with the ``_converter`` suffix -stripped off. (This is accomplished with a metaclass.) - -You shouldn't subclass :py:meth:`!CConverter.__init__`. Instead, you should -write a :py:meth:`!converter_init` function. :py:meth:`!converter_init` -always accepts a *self* parameter; after that, all additional -parameters *must* be keyword-only. Any arguments passed in to -the converter in Argument Clinic will be passed along to your -:py:meth:`!converter_init`. +Your converter class should be named :samp:`{ConverterName}_converter`. +By following this convention, your converter class will be automatically +registered with Argument Clinic, with its *converter name* being the name of +your converter class with the ``_converter`` suffix stripped off. -There are some additional members of :py:class:`!CConverter` you may wish -to specify in your subclass. Here's the current list: +Instead of subclassing :py:meth:`!CConverter.__init__`, +write a :py:meth:`!converter_init` method. +Apart for the *self* parameter, all additional :py:meth:`!converter_init` +parameters **must** be keyword-only. +Any arguments passed to the converter in Argument Clinic +will be passed along to your :py:meth:`!converter_init` method. +See :py:class:`!CConverter` for a list of members you may wish to specify in +your subclass. .. module:: clinic .. class:: CConverter + The base class for all converters. + See :ref:`clinic-howto-custom-converter` for how to subclass this class. + .. attribute:: type The C type to use for this variable. @@ -1436,16 +1438,16 @@ Here's the simplest example of a custom converter, from :source:`Modules/zlibmod [python start generated code]*/ /*[python end generated code: output=da39a3ee5e6b4b0d input=35521e4e733823c7]*/ -This block adds a converter to Argument Clinic named ``ssize_t``. Parameters -declared as ``ssize_t`` will be declared as type :c:type:`Py_ssize_t`, and will -be parsed by the ``'O&'`` format unit, which will call the -``ssize_t_converter`` converter function. ``ssize_t`` variables -automatically support default values. +This block adds a converter named ``ssize_t`` to Argument Clinic. +Parameters declared as ``ssize_t`` will be declared with type :c:type:`Py_ssize_t`, +and will be parsed by the ``'O&'`` format unit, +which will call the :c:func:`!ssize_t_converter` converter C function. +``ssize_t`` variables automatically support default values. More sophisticated custom converters can insert custom C code to handle initialization and cleanup. You can see more examples of custom converters in the CPython -source tree; grep the C files for the string :py:class:`!CConverter`. +source tree; grep the C files for the string ``CConverter``. How to write a custom return converter From d2da97a19e3b6362af49aedb6b29579e827ec431 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Mon, 7 Aug 2023 02:08:34 +0200 Subject: [PATCH 63/71] Docs: Argument Clinic: Move the CConverter class to the reference (#107671) --- Doc/howto/clinic.rst | 133 ++++++++++++++++++++++--------------------- 1 file changed, 69 insertions(+), 64 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index dcede13a0..e8e6aace3 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -193,6 +193,71 @@ The CLI supports the following options: The list of files to process. +.. _clinic-classes: + +Classes for extending Argument Clinic +------------------------------------- + +.. module:: clinic + +.. class:: CConverter + + The base class for all converters. + See :ref:`clinic-howto-custom-converter` for how to subclass this class. + + .. attribute:: type + + The C type to use for this variable. + :attr:`!type` should be a Python string specifying the type, + e.g. ``'int'``. + If this is a pointer type, the type string should end with ``' *'``. + + .. attribute:: default + + The Python default value for this parameter, as a Python value. + Or the magic value ``unspecified`` if there is no default. + + .. attribute:: py_default + + :attr:`!default` as it should appear in Python code, + as a string. + Or ``None`` if there is no default. + + .. attribute:: c_default + + :attr:`!default` as it should appear in C code, + as a string. + Or ``None`` if there is no default. + + .. attribute:: c_ignored_default + + The default value used to initialize the C variable when + there is no default, but not specifying a default may + result in an "uninitialized variable" warning. This can + easily happen when using option groups—although + properly written code will never actually use this value, + the variable does get passed in to the impl, and the + C compiler will complain about the "use" of the + uninitialized value. This value should always be a + non-empty string. + + .. attribute:: converter + + The name of the C converter function, as a string. + + .. attribute:: impl_by_reference + + A boolean value. If true, + Argument Clinic will add a ``&`` in front of the name of + the variable when passing it into the impl function. + + .. attribute:: parse_by_reference + + A boolean value. If true, + Argument Clinic will add a ``&`` in front of the name of + the variable when passing it into :c:func:`PyArg_ParseTuple`. + + .. _clinic-tutorial: Tutorial @@ -1348,7 +1413,7 @@ See also :pep:`573`. How to write a custom converter ------------------------------- -A converter is a Python class that inherits from :py:class:`!CConverter`. +A converter is a Python class that inherits from :py:class:`CConverter`. The main purpose of a custom converter, is for parameters parsed with the ``O&`` format unit --- parsing such a parameter means calling a :c:func:`PyArg_ParseTuple` "converter function". @@ -1360,73 +1425,13 @@ your converter class with the ``_converter`` suffix stripped off. Instead of subclassing :py:meth:`!CConverter.__init__`, write a :py:meth:`!converter_init` method. -Apart for the *self* parameter, all additional :py:meth:`!converter_init` -parameters **must** be keyword-only. +:py:meth:`!converter_init` always accepts a *self* parameter. +After *self*, all additional parameters **must** be keyword-only. Any arguments passed to the converter in Argument Clinic will be passed along to your :py:meth:`!converter_init` method. -See :py:class:`!CConverter` for a list of members you may wish to specify in +See :py:class:`CConverter` for a list of members you may wish to specify in your subclass. -.. module:: clinic - -.. class:: CConverter - - The base class for all converters. - See :ref:`clinic-howto-custom-converter` for how to subclass this class. - - .. attribute:: type - - The C type to use for this variable. - :attr:`!type` should be a Python string specifying the type, - e.g. ``'int'``. - If this is a pointer type, the type string should end with ``' *'``. - - .. attribute:: default - - The Python default value for this parameter, as a Python value. - Or the magic value ``unspecified`` if there is no default. - - .. attribute:: py_default - - :attr:`!default` as it should appear in Python code, - as a string. - Or ``None`` if there is no default. - - .. attribute:: c_default - - :attr:`!default` as it should appear in C code, - as a string. - Or ``None`` if there is no default. - - .. attribute:: c_ignored_default - - The default value used to initialize the C variable when - there is no default, but not specifying a default may - result in an "uninitialized variable" warning. This can - easily happen when using option groups—although - properly written code will never actually use this value, - the variable does get passed in to the impl, and the - C compiler will complain about the "use" of the - uninitialized value. This value should always be a - non-empty string. - - .. attribute:: converter - - The name of the C converter function, as a string. - - .. attribute:: impl_by_reference - - A boolean value. If true, - Argument Clinic will add a ``&`` in front of the name of - the variable when passing it into the impl function. - - .. attribute:: parse_by_reference - - A boolean value. If true, - Argument Clinic will add a ``&`` in front of the name of - the variable when passing it into :c:func:`PyArg_ParseTuple`. - - Here's the simplest example of a custom converter, from :source:`Modules/zlibmodule.c`:: /*[python input] From d03a72aa9dbb46b7bce7e196a08c858d54b5c042 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Mon, 7 Aug 2023 13:28:08 +0200 Subject: [PATCH 64/71] gh-95065: Add Argument Clinic support for deprecating positional use of parameters (#95151) It is now possible to deprecate passing parameters positionally with Argument Clinic, using the new '* [from X.Y]' syntax. (To be read as "keyword-only from Python version X.Y") Co-authored-by: Alex Waygood Co-authored-by: Serhiy Storchaka --- Doc/howto/clinic.rst | 88 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 88 insertions(+) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index e8e6aace3..286623c24 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1898,3 +1898,91 @@ blocks embedded in Python files look slightly different. They look like this: #[python start generated code]*/ def foo(): pass #/*[python checksum:...]*/ + + +.. _clinic-howto-deprecate-positional: + +How to deprecate passing parameters positionally +------------------------------------------------ + +Argument Clinic provides syntax that makes it possible to generate code that +deprecates passing :term:`arguments ` positionally. +For example, say we've got a module-level function :py:func:`!foo.myfunc` +that has three :term:`parameters `: +positional-or-keyword parameters *a* and *b*, and a keyword-only parameter *c*:: + + /*[clinic input] + module foo + myfunc + a: int + b: int + * + c: int + [clinic start generated output]*/ + +We now want to make the *b* parameter keyword-only; +however, we'll have to wait two releases before making this change, +as mandated by Python's backwards-compatibility policy (see :pep:`387`). +For this example, imagine we're in the development phase for Python 3.12: +that means we'll be allowed to introduce deprecation warnings in Python 3.12 +whenever the *b* parameter is passed positionally, +and we'll be allowed to make it keyword-only in Python 3.14 at the earliest. + +We can use Argument Clinic to emit the desired deprecation warnings +using the ``* [from ...]``` syntax, +by adding the line ``* [from 3.14]`` right above the *b* parameter:: + + /*[clinic input] + module foo + myfunc + a: int + * [from 3.14] + b: int + * + c: int + [clinic start generated output]*/ + +Next, regenerate Argument Clinic code (``make clinic``), +and add unit tests for the new behaviour. + +The generated code will now emit a :exc:`DeprecationWarning` +when an :term:`argument` for the :term:`parameter` *b* is passed positionally. +C preprocessor directives are also generated for emitting +compiler warnings if the ``* [from ...]`` line has not been removed +from the Argument Clinic input when the deprecation period is over, +which means when the alpha phase of the specified Python version kicks in. + +Let's return to our example and skip ahead two years: +Python 3.14 development has now entered the alpha phase, +but we forgot all about updating the Argument Clinic code +for :py:func:`!myfunc`! +Luckily for us, compiler warnings are now generated: + +.. code-block:: none + + In file included from Modules/foomodule.c:139: + Modules/clinic/foomodule.c.h:83:8: warning: Update 'b' in 'myfunc' in 'foomodule.c' to be keyword-only. [-W#warnings] + # warning "Update 'b' in 'myfunc' in 'foomodule.c' to be keyword-only." + ^ + +We now close the deprecation phase by making *b* keyword-only; +replace the ``* [from ...]``` line above *b* +with the ``*`` from the line above *c*:: + + /*[clinic input] + module foo + myfunc + a: int + * + b: int + c: int + [clinic start generated output]*/ + +Finally, run ``make clinic`` to regenerate the Argument Clinic code, +and update your unit tests to reflect the new behaviour. + +.. note:: + + If you forget to update your input block during the alpha and beta phases, + the compiler warning will turn into a compiler error when the + release candidate phase begins. From bfaa3552f03a560227104b2555640ac46454c404 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Tue, 8 Aug 2023 08:42:08 +0200 Subject: [PATCH 65/71] gh-86457: Add docs for Argument Clinic @text_signature directive (#107747) Co-authored-by: Alex Waygood --- Doc/howto/clinic.rst | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 286623c24..743c7c9cb 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1900,6 +1900,40 @@ blocks embedded in Python files look slightly different. They look like this: #/*[python checksum:...]*/ +.. _clinic-howto-override-signature: + +How to override the generated signature +--------------------------------------- + +You can use the ``@text_signature`` directive to override the default generated +signature in the docstring. +This can be useful for complex signatures that Argument Clinic cannot handle. +The ``@text_signature`` directive takes one argument: +the custom signature as a string. +The provided signature is copied verbatim to the generated docstring. + +Example from :source:`Objects/codeobject.c`:: + + /*[clinic input] + @text_signature "($self, /, **changes)" + code.replace + * + co_argcount: int(c_default="self->co_argcount") = unchanged + co_posonlyargcount: int(c_default="self->co_posonlyargcount") = unchanged + # etc ... + + Return a copy of the code object with new values for the specified fields. + [clinic start generated output]*/ + +The generated docstring ends up looking like this:: + + Doc_STRVAR(code_replace__doc__, + "replace($self, /, **changes)\n" + "--\n" + "\n" + "Return a copy of the code object with new values for the specified fields."); + + .. _clinic-howto-deprecate-positional: How to deprecate passing parameters positionally From fbef1afd8065d020c31b59f339e8a47ba11aa97b Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Tue, 8 Aug 2023 22:50:54 +0200 Subject: [PATCH 66/71] gh-104683: Add --exclude option to Argument Clinic CLI (#107770) Co-authored-by: Serhiy Storchaka --- Doc/howto/clinic.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 743c7c9cb..2d89ccc20 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -188,6 +188,11 @@ The CLI supports the following options: The directory tree to walk in :option:`--make` mode. +.. option:: --exclude EXCLUDE + + A file to exclude in :option:`--make` mode. + This option can be given multiple times. + .. option:: FILE ... The list of files to process. From f8d0c6702ed098ee5384ac6d1951456c596341c1 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Wed, 9 Aug 2023 09:43:02 +0200 Subject: [PATCH 67/71] Docs: clean up Argument Clinic howto's (#107797) - fix formatting in @text_signature howto and NEWS entry - fix formatting and example text in deprecate-positionals howto --- Doc/howto/clinic.rst | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 2d89ccc20..463a938fa 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -1930,13 +1930,14 @@ Example from :source:`Objects/codeobject.c`:: Return a copy of the code object with new values for the specified fields. [clinic start generated output]*/ -The generated docstring ends up looking like this:: +The generated docstring ends up looking like this: - Doc_STRVAR(code_replace__doc__, - "replace($self, /, **changes)\n" - "--\n" - "\n" - "Return a copy of the code object with new values for the specified fields."); +.. code-block:: none + + replace($self, /, **changes) + -- + + Return a copy of the code object with new values for the specified fields. .. _clinic-howto-deprecate-positional: @@ -1968,7 +1969,7 @@ whenever the *b* parameter is passed positionally, and we'll be allowed to make it keyword-only in Python 3.14 at the earliest. We can use Argument Clinic to emit the desired deprecation warnings -using the ``* [from ...]``` syntax, +using the ``* [from ...]`` syntax, by adding the line ``* [from 3.14]`` right above the *b* parameter:: /*[clinic input] @@ -2000,12 +2001,12 @@ Luckily for us, compiler warnings are now generated: .. code-block:: none In file included from Modules/foomodule.c:139: - Modules/clinic/foomodule.c.h:83:8: warning: Update 'b' in 'myfunc' in 'foomodule.c' to be keyword-only. [-W#warnings] - # warning "Update 'b' in 'myfunc' in 'foomodule.c' to be keyword-only." + Modules/clinic/foomodule.c.h:139:8: warning: In 'foomodule.c', update parameter(s) 'a' and 'b' in the clinic input of 'mymod.myfunc' to be keyword-only. [-W#warnings] + # warning "In 'foomodule.c', update parameter(s) 'a' and 'b' in the clinic input of 'mymod.myfunc' to be keyword-only. [-W#warnings]" ^ We now close the deprecation phase by making *b* keyword-only; -replace the ``* [from ...]``` line above *b* +replace the ``* [from ...]`` line above *b* with the ``*`` from the line above *c*:: /*[clinic input] From d3b2f416e26b5e53012e636d2a6467551e49588d Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Wed, 9 Aug 2023 15:51:35 +0200 Subject: [PATCH 68/71] Add clinic.rst with history --- development-tools/clinic.rst | 2028 ++++++++++++++++++++++++++++++++++ 1 file changed, 2028 insertions(+) create mode 100644 development-tools/clinic.rst diff --git a/development-tools/clinic.rst b/development-tools/clinic.rst new file mode 100644 index 000000000..463a938fa --- /dev/null +++ b/development-tools/clinic.rst @@ -0,0 +1,2028 @@ +.. highlight:: c + +.. _howto-clinic: + +********************** +Argument Clinic How-To +********************** + +:author: Larry Hastings + +**Source code:** :source:`Tools/clinic/clinic.py`. + +.. topic:: Abstract + + Argument Clinic is a preprocessor for CPython C files. + It was introduced in Python 3.4 with :pep:`436`, + in order to provide introspection signatures, + and to generate performant and tailor-made boilerplate code + for argument parsing in CPython builtins, + module level functions, and class methods. + This document is divided in four major sections: + + * :ref:`clinic-background` talks about the basic concepts and goals of + Argument Clinic. + * :ref:`clinic-reference` describes the command-line interface and Argument + Clinic terminology. + * :ref:`clinic-tutorial` guides you through all the steps required to + adapt an existing C function to Argument Clinic. + * :ref:`clinic-howtos` details how to handle specific tasks. + + +.. note:: + + Argument Clinic is considered internal-only + for CPython. Its use is not supported for files outside + CPython, and no guarantees are made regarding backwards + compatibility for future versions. In other words: if you + maintain an external C extension for CPython, you're welcome + to experiment with Argument Clinic in your own code. But the + version of Argument Clinic that ships with the next version + of CPython *could* be totally incompatible and break all your code. + + +.. _clinic-background: + +Background +========== + +Basic concepts +-------------- + +When Argument Clinic is run on a file, either via the :ref:`clinic-cli` +or via ``make clinic``, it will scan over the input files looking for +:term:`start lines `: + +.. code-block:: none + + /*[clinic input] + +When it finds one, it reads everything up to the :term:`end line`: + +.. code-block:: none + + [clinic start generated code]*/ + +Everything in between these two lines is Argument Clinic :term:`input`. +When Argument Clinic parses input, it generates :term:`output`. +The output is rewritten into the C file immediately after the input, +followed by a :term:`checksum line`. +All of these lines, including the :term:`start line` and :term:`checksum line`, +are collectively called an Argument Clinic :term:`block`: + +.. code-block:: none + + /*[clinic input] + ... clinic input goes here ... + [clinic start generated code]*/ + ... clinic output goes here ... + /*[clinic end generated code: ...]*/ + +If you run Argument Clinic on the same file a second time, Argument Clinic +will discard the old :term:`output` and write out the new output with a fresh +:term:`checksum line`. +If the :term:`input` hasn't changed, the output won't change either. + +.. note:: + + You should never modify the output of an Argument Clinic block, + as any change will be lost in future Argument Clinic runs; + Argument Clinic will detect an output checksum mismatch and regenerate the + correct output. + If you are not happy with the generated output, + you should instead change the input until it produces the output you want. + + +.. _clinic-reference: + +Reference +========= + + +.. _clinic-terminology: + +Terminology +----------- + +.. glossary:: + + start line + The line ``/*[clinic input]``. + This line marks the beginning of Argument Clinic input. + Note that the *start line* opens a C block comment. + + end line + The line ``[clinic start generated code]*/``. + The *end line* marks the _end_ of Argument Clinic :term:`input`, + but at the same time marks the _start_ of Argument Clinic :term:`output`, + thus the text *"clinic start start generated code"* + Note that the *end line* closes the C block comment opened + by the *start line*. + + checksum + A hash to distinguish unique :term:`inputs ` + and :term:`outputs `. + + checksum line + A line that looks like ``/*[clinic end generated code: ...]*/``. + The three dots will be replaced by a :term:`checksum` generated from the + :term:`input`, and a :term:`checksum` generated from the :term:`output`. + The checksum line marks the end of Argument Clinic generated code, + and is used by Argument Clinic to determine if it needs to regenerate + output. + + input + The text between the :term:`start line` and the :term:`end line`. + Note that the start and end lines open and close a C block comment; + the *input* is thus a part of that same C block comment. + + output + The text between the :term:`end line` and the :term:`checksum line`. + + block + All text from the :term:`start line` to the :term:`checksum line` inclusively. + + +.. _clinic-cli: + +Command-line interface +---------------------- + +The Argument Clinic :abbr:`CLI (Command-Line Interface)` is typically used to +process a single source file, like this: + +.. code-block:: shell-session + + $ python3 ./Tools/clinic/clinic.py foo.c + +The CLI supports the following options: + +.. program:: ./Tools/clinic/clinic.py [-h] [-f] [-o OUTPUT] [-v] \ + [--converters] [--make] [--srcdir SRCDIR] [FILE ...] + +.. option:: -h, --help + + Print CLI usage. + +.. option:: -f, --force + + Force output regeneration. + +.. option:: -o, --output OUTPUT + + Redirect file output to OUTPUT + +.. option:: -v, --verbose + + Enable verbose mode. + +.. option:: --converters + + Print a list of all supported converters and return converters. + +.. option:: --make + + Walk :option:`--srcdir` to run over all relevant files. + +.. option:: --srcdir SRCDIR + + The directory tree to walk in :option:`--make` mode. + +.. option:: --exclude EXCLUDE + + A file to exclude in :option:`--make` mode. + This option can be given multiple times. + +.. option:: FILE ... + + The list of files to process. + + +.. _clinic-classes: + +Classes for extending Argument Clinic +------------------------------------- + +.. module:: clinic + +.. class:: CConverter + + The base class for all converters. + See :ref:`clinic-howto-custom-converter` for how to subclass this class. + + .. attribute:: type + + The C type to use for this variable. + :attr:`!type` should be a Python string specifying the type, + e.g. ``'int'``. + If this is a pointer type, the type string should end with ``' *'``. + + .. attribute:: default + + The Python default value for this parameter, as a Python value. + Or the magic value ``unspecified`` if there is no default. + + .. attribute:: py_default + + :attr:`!default` as it should appear in Python code, + as a string. + Or ``None`` if there is no default. + + .. attribute:: c_default + + :attr:`!default` as it should appear in C code, + as a string. + Or ``None`` if there is no default. + + .. attribute:: c_ignored_default + + The default value used to initialize the C variable when + there is no default, but not specifying a default may + result in an "uninitialized variable" warning. This can + easily happen when using option groups—although + properly written code will never actually use this value, + the variable does get passed in to the impl, and the + C compiler will complain about the "use" of the + uninitialized value. This value should always be a + non-empty string. + + .. attribute:: converter + + The name of the C converter function, as a string. + + .. attribute:: impl_by_reference + + A boolean value. If true, + Argument Clinic will add a ``&`` in front of the name of + the variable when passing it into the impl function. + + .. attribute:: parse_by_reference + + A boolean value. If true, + Argument Clinic will add a ``&`` in front of the name of + the variable when passing it into :c:func:`PyArg_ParseTuple`. + + +.. _clinic-tutorial: + +Tutorial +======== + +The best way to get a sense of how Argument Clinic works is to +convert a function to work with it. Here, then, are the bare +minimum steps you'd need to follow to convert a function to +work with Argument Clinic. Note that for code you plan to +check in to CPython, you really should take the conversion farther, +using some of the :ref:`advanced concepts ` +you'll see later on in the document, +like :ref:`clinic-howto-return-converters` +and :ref:`clinic-howto-self-converter`. +But we'll keep it simple for this walkthrough so you can learn. + +First, make sure you're working with a freshly updated checkout +of the CPython trunk. + +Next, find a Python builtin that calls either :c:func:`PyArg_ParseTuple` +or :c:func:`PyArg_ParseTupleAndKeywords`, and hasn't been converted +to work with Argument Clinic yet. +For this tutorial, we'll be using +:py:meth:`_pickle.Pickler.dump `. + +If the call to the :c:func:`!PyArg_Parse*` function uses any of the +following format units...: + + .. code-block:: none + + O& + O! + es + es# + et + et# + +... or if it has multiple calls to :c:func:`PyArg_ParseTuple`, +you should choose a different function. +(See :ref:`clinic-howto-advanced-converters` for those scenarios.) + +Also, if the function has multiple calls to :c:func:`!PyArg_ParseTuple` +or :c:func:`PyArg_ParseTupleAndKeywords` where it supports different +types for the same argument, or if the function uses something besides +:c:func:`!PyArg_Parse*` functions to parse its arguments, it probably +isn't suitable for conversion to Argument Clinic. Argument Clinic +doesn't support generic functions or polymorphic parameters. + +Next, add the following boilerplate above the function, +creating our input block:: + + /*[clinic input] + [clinic start generated code]*/ + +Cut the docstring and paste it in between the ``[clinic]`` lines, +removing all the junk that makes it a properly quoted C string. +When you're done you should have just the text, based at the left +margin, with no line wider than 80 characters. +Argument Clinic will preserve indents inside the docstring. + +If the old docstring had a first line that looked like a function +signature, throw that line away; The docstring doesn't need it anymore --- +when you use :py:func:`help` on your builtin in the future, +the first line will be built automatically based on the function's signature. + +Example docstring summary line:: + + /*[clinic input] + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ + +If your docstring doesn't have a "summary" line, Argument Clinic will +complain, so let's make sure it has one. The "summary" line should +be a paragraph consisting of a single 80-column line +at the beginning of the docstring. +(See :pep:`257` regarding docstring conventions.) + +Our example docstring consists solely of a summary line, so the sample +code doesn't have to change for this step. + +Now, above the docstring, enter the name of the function, followed +by a blank line. This should be the Python name of the function, +and should be the full dotted path to the function --- +it should start with the name of the module, +include any sub-modules, and if the function is a method on +a class it should include the class name too. + +In our example, :mod:`!_pickle` is the module, :py:class:`!Pickler` is the class, +and :py:meth:`!dump` is the method, so the name becomes +:py:meth:`!_pickle.Pickler.dump`:: + + /*[clinic input] + _pickle.Pickler.dump + + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ + +If this is the first time that module or class has been used with Argument +Clinic in this C file, +you must declare the module and/or class. Proper Argument Clinic hygiene +prefers declaring these in a separate block somewhere near the +top of the C file, in the same way that include files and statics go at +the top. +In our sample code we'll just show the two blocks next to each other. + +The name of the class and module should be the same as the one +seen by Python. Check the name defined in the :c:type:`PyModuleDef` +or :c:type:`PyTypeObject` as appropriate. + +When you declare a class, you must also specify two aspects of its type +in C: the type declaration you'd use for a pointer to an instance of +this class, and a pointer to the :c:type:`!PyTypeObject` for this class:: + + /*[clinic input] + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" + [clinic start generated code]*/ + + /*[clinic input] + _pickle.Pickler.dump + + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ + +Declare each of the parameters to the function. Each parameter +should get its own line. All the parameter lines should be +indented from the function name and the docstring. +The general form of these parameter lines is as follows: + +.. code-block:: none + + name_of_parameter: converter + +If the parameter has a default value, add that after the +converter: + +.. code-block:: none + + name_of_parameter: converter = default_value + +Argument Clinic's support for "default values" is quite sophisticated; +see :ref:`clinic-howto-default-values` for more information. + +Next, add a blank line below the parameters. + +What's a "converter"? +It establishes both the type of the variable used in C, +and the method to convert the Python value into a C value at runtime. +For now you're going to use what's called a "legacy converter" --- +a convenience syntax intended to make porting old code into Argument +Clinic easier. + +For each parameter, copy the "format unit" for that +parameter from the :c:func:`PyArg_Parse` format argument and +specify *that* as its converter, as a quoted string. +The "format unit" is the formal name for the one-to-three +character substring of the *format* parameter that tells +the argument parsing function what the type of the variable +is and how to convert it. +For more on format units please see :ref:`arg-parsing`. + +For multicharacter format units like ``z#``, +use the entire two-or-three character string. + +Sample:: + + /*[clinic input] + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" + [clinic start generated code]*/ + + /*[clinic input] + _pickle.Pickler.dump + + obj: 'O' + + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ + +If your function has ``|`` in the format string, +meaning some parameters have default values, you can ignore it. +Argument Clinic infers which parameters are optional +based on whether or not they have default values. + +If your function has ``$`` in the format string, +meaning it takes keyword-only arguments, +specify ``*`` on a line by itself before the first keyword-only argument, +indented the same as the parameter lines. + +:py:meth:`!_pickle.Pickler.dump` has neither, so our sample is unchanged. + +Next, if the existing C function calls :c:func:`PyArg_ParseTuple` +(as opposed to :c:func:`PyArg_ParseTupleAndKeywords`), then all its +arguments are positional-only. + +To mark parameters as positional-only in Argument Clinic, +add a ``/`` on a line by itself after the last positional-only parameter, +indented the same as the parameter lines. + +Sample:: + + /*[clinic input] + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" + [clinic start generated code]*/ + + /*[clinic input] + _pickle.Pickler.dump + + obj: 'O' + / + + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ + +It can be helpful to write a per-parameter docstring for each parameter. +Since per-parameter docstrings are optional, +you can skip this step if you prefer. + +Nevertheless, here's how to add a per-parameter docstring. +The first line of the per-parameter docstring +must be indented further than the parameter definition. +The left margin of this first line establishes +the left margin for the whole per-parameter docstring; +all the text you write will be outdented by this amount. +You can write as much text as you like, across multiple lines if you wish. + +Sample:: + + /*[clinic input] + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" + [clinic start generated code]*/ + + /*[clinic input] + _pickle.Pickler.dump + + obj: 'O' + The object to be pickled. + / + + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ + +Save and close the file, then run ``Tools/clinic/clinic.py`` on it. +With luck everything worked---your block now has output, +and a :file:`.c.h` file has been generated! +Reload the file in your text editor to see the generated code:: + + /*[clinic input] + _pickle.Pickler.dump + + obj: 'O' + The object to be pickled. + / + + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ + + static PyObject * + _pickle_Pickler_dump(PicklerObject *self, PyObject *obj) + /*[clinic end generated code: output=87ecad1261e02ac7 input=552eb1c0f52260d9]*/ + +Obviously, if Argument Clinic didn't produce any output, +it's because it found an error in your input. +Keep fixing your errors and retrying until Argument Clinic processes your file +without complaint. + +For readability, most of the glue code has been generated to a :file:`.c.h` +file. You'll need to include that in your original :file:`.c` file, +typically right after the clinic module block:: + + #include "clinic/_pickle.c.h" + +Double-check that the argument-parsing code Argument Clinic generated +looks basically the same as the existing code. + +First, ensure both places use the same argument-parsing function. +The existing code must call either +:c:func:`PyArg_ParseTuple` or :c:func:`PyArg_ParseTupleAndKeywords`; +ensure that the code generated by Argument Clinic calls the +*exact* same function. + +Second, the format string passed in to :c:func:`!PyArg_ParseTuple` or +:c:func:`!PyArg_ParseTupleAndKeywords` should be *exactly* the same +as the hand-written one in the existing function, +up to the colon or semi-colon. + +Argument Clinic always generates its format strings +with a ``:`` followed by the name of the function. +If the existing code's format string ends with ``;``, +to provide usage help, this change is harmless --- don't worry about it. + +Third, for parameters whose format units require two arguments, +like a length variable, an encoding string, or a pointer +to a conversion function, ensure that the second argument is +*exactly* the same between the two invocations. + +Fourth, inside the output portion of the block, +you'll find a preprocessor macro defining the appropriate static +:c:type:`PyMethodDef` structure for this builtin:: + + #define __PICKLE_PICKLER_DUMP_METHODDEF \ + {"dump", (PyCFunction)__pickle_Pickler_dump, METH_O, __pickle_Pickler_dump__doc__}, + +This static structure should be *exactly* the same as the existing static +:c:type:`!PyMethodDef` structure for this builtin. + +If any of these items differ in *any way*, +adjust your Argument Clinic function specification and rerun +``Tools/clinic/clinic.py`` until they *are* the same. + +Notice that the last line of its output is the declaration +of your "impl" function. This is where the builtin's implementation goes. +Delete the existing prototype of the function you're modifying, but leave +the opening curly brace. Now delete its argument parsing code and the +declarations of all the variables it dumps the arguments into. +Notice how the Python arguments are now arguments to this impl function; +if the implementation used different names for these variables, fix it. + +Let's reiterate, just because it's kind of weird. +Your code should now look like this:: + + static return_type + your_function_impl(...) + /*[clinic end generated code: input=..., output=...]*/ + { + ... + +Argument Clinic generated the checksum line and the function prototype just +above it. You should write the opening and closing curly braces for the +function, and the implementation inside. + +Sample:: + + /*[clinic input] + module _pickle + class _pickle.Pickler "PicklerObject *" "&Pickler_Type" + [clinic start generated code]*/ + /*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ + + /*[clinic input] + _pickle.Pickler.dump + + obj: 'O' + The object to be pickled. + / + + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ + + PyDoc_STRVAR(__pickle_Pickler_dump__doc__, + "Write a pickled representation of obj to the open file.\n" + "\n" + ... + static PyObject * + _pickle_Pickler_dump_impl(PicklerObject *self, PyObject *obj) + /*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ + { + /* Check whether the Pickler was initialized correctly (issue3664). + Developers often forget to call __init__() in their subclasses, which + would trigger a segfault without this check. */ + if (self->write == NULL) { + PyErr_Format(PicklingError, + "Pickler.__init__() was not called by %s.__init__()", + Py_TYPE(self)->tp_name); + return NULL; + } + + if (_Pickler_ClearBuffer(self) < 0) { + return NULL; + } + + ... + +Remember the macro with the :c:type:`PyMethodDef` structure for this function? +Find the existing :c:type:`!PyMethodDef` structure for this +function and replace it with a reference to the macro. If the builtin +is at module scope, this will probably be very near the end of the file; +if the builtin is a class method, this will probably be below but relatively +near to the implementation. + +Note that the body of the macro contains a trailing comma; when you +replace the existing static :c:type:`!PyMethodDef` structure with the macro, +*don't* add a comma to the end. + +Sample:: + + static struct PyMethodDef Pickler_methods[] = { + __PICKLE_PICKLER_DUMP_METHODDEF + __PICKLE_PICKLER_CLEAR_MEMO_METHODDEF + {NULL, NULL} /* sentinel */ + }; + +Argument Clinic may generate new instances of ``_Py_ID``. For example:: + + &_Py_ID(new_unique_py_id) + +If it does, you'll have to run ``make regen-global-objects`` +to regenerate the list of precompiled identifiers at this point. + +Finally, compile, then run the relevant portions of the regression-test suite. +This change should not introduce any new compile-time warnings or errors, +and there should be no externally visible change to Python's behavior, +except for one difference: :py:func:`inspect.signature` run on your function +should now provide a valid signature! + +Congratulations, you've ported your first function to work with Argument Clinic! + + +.. _clinic-howtos: + +How-to guides +============= + + +How to rename C functions and variables generated by Argument Clinic +-------------------------------------------------------------------- + +Argument Clinic automatically names the functions it generates for you. +Occasionally this may cause a problem, if the generated name collides with +the name of an existing C function. There's an easy solution: override the names +used for the C functions. Just add the keyword ``"as"`` +to your function declaration line, followed by the function name you wish to use. +Argument Clinic will use that function name for the base (generated) function, +then add ``"_impl"`` to the end and use that for the name of the impl function. + +For example, if we wanted to rename the C function names generated for +:py:meth:`pickle.Pickler.dump`, it'd look like this:: + + /*[clinic input] + pickle.Pickler.dump as pickler_dumper + + ... + +The base function would now be named :c:func:`!pickler_dumper`, +and the impl function would now be named :c:func:`!pickler_dumper_impl`. + + +Similarly, you may have a problem where you want to give a parameter +a specific Python name, but that name may be inconvenient in C. Argument +Clinic allows you to give a parameter different names in Python and in C, +using the same ``"as"`` syntax:: + + /*[clinic input] + pickle.Pickler.dump + + obj: object + file as file_obj: object + protocol: object = NULL + * + fix_imports: bool = True + +Here, the name used in Python (in the signature and the ``keywords`` +array) would be *file*, but the C variable would be named ``file_obj``. + +You can use this to rename the *self* parameter too! + + +How to convert functions using ``PyArg_UnpackTuple`` +---------------------------------------------------- + +To convert a function parsing its arguments with :c:func:`PyArg_UnpackTuple`, +simply write out all the arguments, specifying each as an ``object``. You +may specify the *type* argument to cast the type as appropriate. All +arguments should be marked positional-only (add a ``/`` on a line by itself +after the last argument). + +Currently the generated code will use :c:func:`PyArg_ParseTuple`, but this +will change soon. + + +How to use optional groups +-------------------------- + +Some legacy functions have a tricky approach to parsing their arguments: +they count the number of positional arguments, then use a ``switch`` statement +to call one of several different :c:func:`PyArg_ParseTuple` calls depending on +how many positional arguments there are. (These functions cannot accept +keyword-only arguments.) This approach was used to simulate optional +arguments back before :c:func:`PyArg_ParseTupleAndKeywords` was created. + +While functions using this approach can often be converted to +use :c:func:`!PyArg_ParseTupleAndKeywords`, optional arguments, and default values, +it's not always possible. Some of these legacy functions have +behaviors :c:func:`!PyArg_ParseTupleAndKeywords` doesn't directly support. +The most obvious example is the builtin function :py:func:`range`, which has +an optional argument on the *left* side of its required argument! +Another example is :py:meth:`curses.window.addch`, which has a group of two +arguments that must always be specified together. (The arguments are +called *x* and *y*; if you call the function passing in *x*, +you must also pass in *y* — and if you don't pass in *x* you may not +pass in *y* either.) + +In any case, the goal of Argument Clinic is to support argument parsing +for all existing CPython builtins without changing their semantics. +Therefore Argument Clinic supports +this alternate approach to parsing, using what are called *optional groups*. +Optional groups are groups of arguments that must all be passed in together. +They can be to the left or the right of the required arguments. They +can *only* be used with positional-only parameters. + +.. note:: Optional groups are *only* intended for use when converting + functions that make multiple calls to :c:func:`PyArg_ParseTuple`! + Functions that use *any* other approach for parsing arguments + should *almost never* be converted to Argument Clinic using + optional groups. Functions using optional groups currently + cannot have accurate signatures in Python, because Python just + doesn't understand the concept. Please avoid using optional + groups wherever possible. + +To specify an optional group, add a ``[`` on a line by itself before +the parameters you wish to group together, and a ``]`` on a line by itself +after these parameters. As an example, here's how :py:meth:`curses.window.addch` +uses optional groups to make the first two parameters and the last +parameter optional:: + + /*[clinic input] + + curses.window.addch + + [ + x: int + X-coordinate. + y: int + Y-coordinate. + ] + + ch: object + Character to add. + + [ + attr: long + Attributes for the character. + ] + / + + ... + + +Notes: + +* For every optional group, one additional parameter will be passed into the + impl function representing the group. The parameter will be an int named + ``group_{direction}_{number}``, + where ``{direction}`` is either ``right`` or ``left`` depending on whether the group + is before or after the required parameters, and ``{number}`` is a monotonically + increasing number (starting at 1) indicating how far away the group is from + the required parameters. When the impl is called, this parameter will be set + to zero if this group was unused, and set to non-zero if this group was used. + (By used or unused, I mean whether or not the parameters received arguments + in this invocation.) + +* If there are no required arguments, the optional groups will behave + as if they're to the right of the required arguments. + +* In the case of ambiguity, the argument parsing code + favors parameters on the left (before the required parameters). + +* Optional groups can only contain positional-only parameters. + +* Optional groups are *only* intended for legacy code. Please do not + use optional groups for new code. + + +How to use real Argument Clinic converters, instead of "legacy converters" +-------------------------------------------------------------------------- + +To save time, and to minimize how much you need to learn +to achieve your first port to Argument Clinic, the walkthrough above tells +you to use "legacy converters". "Legacy converters" are a convenience, +designed explicitly to make porting existing code to Argument Clinic +easier. And to be clear, their use is acceptable when porting code for +Python 3.4. + +However, in the long term we probably want all our blocks to +use Argument Clinic's real syntax for converters. Why? A couple +reasons: + +* The proper converters are far easier to read and clearer in their intent. +* There are some format units that are unsupported as "legacy converters", + because they require arguments, and the legacy converter syntax doesn't + support specifying arguments. +* In the future we may have a new argument parsing library that isn't + restricted to what :c:func:`PyArg_ParseTuple` supports; this flexibility + won't be available to parameters using legacy converters. + +Therefore, if you don't mind a little extra effort, please use the normal +converters instead of legacy converters. + +In a nutshell, the syntax for Argument Clinic (non-legacy) converters +looks like a Python function call. However, if there are no explicit +arguments to the function (all functions take their default values), +you may omit the parentheses. Thus ``bool`` and ``bool()`` are exactly +the same converters. + +All arguments to Argument Clinic converters are keyword-only. +All Argument Clinic converters accept the following arguments: + + *c_default* + The default value for this parameter when defined in C. + Specifically, this will be the initializer for the variable declared + in the "parse function". See :ref:`the section on default values ` + for how to use this. + Specified as a string. + + *annotation* + The annotation value for this parameter. Not currently supported, + because :pep:`8` mandates that the Python library may not use + annotations. + + *unused* + Wrap the argument with :c:macro:`Py_UNUSED` in the impl function signature. + +In addition, some converters accept additional arguments. Here is a list +of these arguments, along with their meanings: + + *accept* + A set of Python types (and possibly pseudo-types); + this restricts the allowable Python argument to values of these types. + (This is not a general-purpose facility; as a rule it only supports + specific lists of types as shown in the legacy converter table.) + + To accept ``None``, add ``NoneType`` to this set. + + *bitwise* + Only supported for unsigned integers. The native integer value of this + Python argument will be written to the parameter without any range checking, + even for negative values. + + *converter* + Only supported by the ``object`` converter. Specifies the name of a + :ref:`C "converter function" ` + to use to convert this object to a native type. + + *encoding* + Only supported for strings. Specifies the encoding to use when converting + this string from a Python str (Unicode) value into a C ``char *`` value. + + + *subclass_of* + Only supported for the ``object`` converter. Requires that the Python + value be a subclass of a Python type, as expressed in C. + + *type* + Only supported for the ``object`` and ``self`` converters. Specifies + the C type that will be used to declare the variable. Default value is + ``"PyObject *"``. + + *zeroes* + Only supported for strings. If true, embedded NUL bytes (``'\\0'``) are + permitted inside the value. The length of the string will be passed in + to the impl function, just after the string parameter, as a parameter named + ``_length``. + +Please note, not every possible combination of arguments will work. +Usually these arguments are implemented by specific :c:func:`PyArg_ParseTuple` +*format units*, with specific behavior. For example, currently you cannot +call ``unsigned_short`` without also specifying ``bitwise=True``. +Although it's perfectly reasonable to think this would work, these semantics don't +map to any existing format unit. So Argument Clinic doesn't support it. (Or, at +least, not yet.) + +Below is a table showing the mapping of legacy converters into real +Argument Clinic converters. On the left is the legacy converter, +on the right is the text you'd replace it with. + +========= ================================================================================= +``'B'`` ``unsigned_char(bitwise=True)`` +``'b'`` ``unsigned_char`` +``'c'`` ``char`` +``'C'`` ``int(accept={str})`` +``'d'`` ``double`` +``'D'`` ``Py_complex`` +``'es'`` ``str(encoding='name_of_encoding')`` +``'es#'`` ``str(encoding='name_of_encoding', zeroes=True)`` +``'et'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str})`` +``'et#'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str}, zeroes=True)`` +``'f'`` ``float`` +``'h'`` ``short`` +``'H'`` ``unsigned_short(bitwise=True)`` +``'i'`` ``int`` +``'I'`` ``unsigned_int(bitwise=True)`` +``'k'`` ``unsigned_long(bitwise=True)`` +``'K'`` ``unsigned_long_long(bitwise=True)`` +``'l'`` ``long`` +``'L'`` ``long long`` +``'n'`` ``Py_ssize_t`` +``'O'`` ``object`` +``'O!'`` ``object(subclass_of='&PySomething_Type')`` +``'O&'`` ``object(converter='name_of_c_function')`` +``'p'`` ``bool`` +``'S'`` ``PyBytesObject`` +``'s'`` ``str`` +``'s#'`` ``str(zeroes=True)`` +``'s*'`` ``Py_buffer(accept={buffer, str})`` +``'U'`` ``unicode`` +``'u'`` ``wchar_t`` +``'u#'`` ``wchar_t(zeroes=True)`` +``'w*'`` ``Py_buffer(accept={rwbuffer})`` +``'Y'`` ``PyByteArrayObject`` +``'y'`` ``str(accept={bytes})`` +``'y#'`` ``str(accept={robuffer}, zeroes=True)`` +``'y*'`` ``Py_buffer`` +``'Z'`` ``wchar_t(accept={str, NoneType})`` +``'Z#'`` ``wchar_t(accept={str, NoneType}, zeroes=True)`` +``'z'`` ``str(accept={str, NoneType})`` +``'z#'`` ``str(accept={str, NoneType}, zeroes=True)`` +``'z*'`` ``Py_buffer(accept={buffer, str, NoneType})`` +========= ================================================================================= + +As an example, here's our sample ``pickle.Pickler.dump`` using the proper +converter:: + + /*[clinic input] + pickle.Pickler.dump + + obj: object + The object to be pickled. + / + + Write a pickled representation of obj to the open file. + [clinic start generated code]*/ + +One advantage of real converters is that they're more flexible than legacy +converters. For example, the ``unsigned_int`` converter (and all the +``unsigned_`` converters) can be specified without ``bitwise=True``. Their +default behavior performs range checking on the value, and they won't accept +negative numbers. You just can't do that with a legacy converter! + +Argument Clinic will show you all the converters it has +available. For each converter it'll show you all the parameters +it accepts, along with the default value for each parameter. +Just run ``Tools/clinic/clinic.py --converters`` to see the full list. + + +How to use the ``Py_buffer`` converter +-------------------------------------- + +When using the ``Py_buffer`` converter +(or the ``'s*'``, ``'w*'``, ``'*y'``, or ``'z*'`` legacy converters), +you *must* not call :c:func:`PyBuffer_Release` on the provided buffer. +Argument Clinic generates code that does it for you (in the parsing function). + + +.. _clinic-howto-advanced-converters: + +How to use advanced converters +------------------------------ + +Remember those format units you skipped for your first +time because they were advanced? Here's how to handle those too. + +The trick is, all those format units take arguments—either +conversion functions, or types, or strings specifying an encoding. +(But "legacy converters" don't support arguments. That's why we +skipped them for your first function.) The argument you specified +to the format unit is now an argument to the converter; this +argument is either *converter* (for ``O&``), *subclass_of* (for ``O!``), +or *encoding* (for all the format units that start with ``e``). + +When using *subclass_of*, you may also want to use the other +custom argument for ``object()``: *type*, which lets you set the type +actually used for the parameter. For example, if you want to ensure +that the object is a subclass of :c:var:`PyUnicode_Type`, you probably want +to use the converter ``object(type='PyUnicodeObject *', subclass_of='&PyUnicode_Type')``. + +One possible problem with using Argument Clinic: it takes away some possible +flexibility for the format units starting with ``e``. When writing a +:c:func:`!PyArg_Parse*` call by hand, you could theoretically decide at runtime what +encoding string to pass to that call. But now this string must +be hard-coded at Argument-Clinic-preprocessing-time. This limitation is deliberate; +it made supporting this format unit much easier, and may allow for future optimizations. +This restriction doesn't seem unreasonable; CPython itself always passes in static +hard-coded encoding strings for parameters whose format units start with ``e``. + + +.. _clinic-howto-default-values: +.. _default_values: + +How to assign default values to parameter +----------------------------------------- + +Default values for parameters can be any of a number of values. +At their simplest, they can be string, int, or float literals: + +.. code-block:: none + + foo: str = "abc" + bar: int = 123 + bat: float = 45.6 + +They can also use any of Python's built-in constants: + +.. code-block:: none + + yep: bool = True + nope: bool = False + nada: object = None + +There's also special support for a default value of ``NULL``, and +for simple expressions, documented in the following sections. + + +The ``NULL`` default value +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For string and object parameters, you can set them to ``None`` to indicate +that there's no default. However, that means the C variable will be +initialized to ``Py_None``. For convenience's sakes, there's a special +value called ``NULL`` for just this reason: from Python's perspective it +behaves like a default value of ``None``, but the C variable is initialized +with ``NULL``. + + +Symbolic default values +^^^^^^^^^^^^^^^^^^^^^^^ + +The default value you provide for a parameter can't be any arbitrary +expression. Currently the following are explicitly supported: + +* Numeric constants (integer and float) +* String constants +* ``True``, ``False``, and ``None`` +* Simple symbolic constants like :py:data:`sys.maxsize`, which must + start with the name of the module + +(In the future, this may need to get even more elaborate, +to allow full expressions like ``CONSTANT - 1``.) + + +Expressions as default values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default value for a parameter can be more than just a literal value. +It can be an entire expression, using math operators and looking up attributes +on objects. However, this support isn't exactly simple, because of some +non-obvious semantics. + +Consider the following example: + +.. code-block:: none + + foo: Py_ssize_t = sys.maxsize - 1 + +:py:data:`sys.maxsize` can have different values on different platforms. Therefore +Argument Clinic can't simply evaluate that expression locally and hard-code it +in C. So it stores the default in such a way that it will get evaluated at +runtime, when the user asks for the function's signature. + +What namespace is available when the expression is evaluated? It's evaluated +in the context of the module the builtin came from. So, if your module has an +attribute called :py:attr:`!max_widgets`, you may simply use it: + +.. code-block:: none + + foo: Py_ssize_t = max_widgets + +If the symbol isn't found in the current module, it fails over to looking in +:py:data:`sys.modules`. That's how it can find :py:data:`sys.maxsize` for example. +(Since you don't know in advance what modules the user will load into their interpreter, +it's best to restrict yourself to modules that are preloaded by Python itself.) + +Evaluating default values only at runtime means Argument Clinic can't compute +the correct equivalent C default value. So you need to tell it explicitly. +When you use an expression, you must also specify the equivalent expression +in C, using the *c_default* parameter to the converter: + +.. code-block:: none + + foo: Py_ssize_t(c_default="PY_SSIZE_T_MAX - 1") = sys.maxsize - 1 + +Another complication: Argument Clinic can't know in advance whether or not the +expression you supply is valid. It parses it to make sure it looks legal, but +it can't *actually* know. You must be very careful when using expressions to +specify values that are guaranteed to be valid at runtime! + +Finally, because expressions must be representable as static C values, there +are many restrictions on legal expressions. Here's a list of Python features +you're not permitted to use: + +* Function calls. +* Inline if statements (``3 if foo else 5``). +* Automatic sequence unpacking (``*[1, 2, 3]``). +* List/set/dict comprehensions and generator expressions. +* Tuple/list/set/dict literals. + + +.. _clinic-howto-return-converters: + +How to use return converters +---------------------------- + +By default, the impl function Argument Clinic generates for you returns +:c:type:`PyObject * `. +But your C function often computes some C type, +then converts it into the :c:type:`!PyObject *` +at the last moment. Argument Clinic handles converting your inputs from Python types +into native C types—why not have it convert your return value from a native C type +into a Python type too? + +That's what a "return converter" does. It changes your impl function to return +some C type, then adds code to the generated (non-impl) function to handle converting +that value into the appropriate :c:type:`!PyObject *`. + +The syntax for return converters is similar to that of parameter converters. +You specify the return converter like it was a return annotation on the +function itself, using ``->`` notation. + +For example: + +.. code-block:: c + + /*[clinic input] + add -> int + + a: int + b: int + / + + [clinic start generated code]*/ + +Return converters behave much the same as parameter converters; +they take arguments, the arguments are all keyword-only, and if you're not changing +any of the default arguments you can omit the parentheses. + +(If you use both ``"as"`` *and* a return converter for your function, +the ``"as"`` should come before the return converter.) + +There's one additional complication when using return converters: how do you +indicate an error has occurred? Normally, a function returns a valid (non-``NULL``) +pointer for success, and ``NULL`` for failure. But if you use an integer return converter, +all integers are valid. How can Argument Clinic detect an error? Its solution: each return +converter implicitly looks for a special value that indicates an error. If you return +that value, and an error has been set (c:func:`PyErr_Occurred` returns a true +value), then the generated code will propagate the error. Otherwise it will +encode the value you return like normal. + +Currently Argument Clinic supports only a few return converters: + +.. code-block:: none + + bool + double + float + int + long + Py_ssize_t + size_t + unsigned int + unsigned long + +None of these take parameters. +For all of these, return ``-1`` to indicate error. + +To see all the return converters Argument Clinic supports, along with +their parameters (if any), +just run ``Tools/clinic/clinic.py --converters`` for the full list. + + +How to clone existing functions +------------------------------- + +If you have a number of functions that look similar, you may be able to +use Clinic's "clone" feature. When you clone an existing function, +you reuse: + +* its parameters, including + + * their names, + + * their converters, with all parameters, + + * their default values, + + * their per-parameter docstrings, + + * their *kind* (whether they're positional only, + positional or keyword, or keyword only), and + +* its return converter. + +The only thing not copied from the original function is its docstring; +the syntax allows you to specify a new docstring. + +Here's the syntax for cloning a function:: + + /*[clinic input] + module.class.new_function [as c_basename] = module.class.existing_function + + Docstring for new_function goes here. + [clinic start generated code]*/ + +(The functions can be in different modules or classes. I wrote +``module.class`` in the sample just to illustrate that you must +use the full path to *both* functions.) + +Sorry, there's no syntax for partially cloning a function, or cloning a function +then modifying it. Cloning is an all-or nothing proposition. + +Also, the function you are cloning from must have been previously defined +in the current file. + + +How to call Python code +----------------------- + +The rest of the advanced topics require you to write Python code +which lives inside your C file and modifies Argument Clinic's +runtime state. This is simple: you simply define a Python block. + +A Python block uses different delimiter lines than an Argument +Clinic function block. It looks like this:: + + /*[python input] + # python code goes here + [python start generated code]*/ + +All the code inside the Python block is executed at the +time it's parsed. All text written to stdout inside the block +is redirected into the "output" after the block. + +As an example, here's a Python block that adds a static integer +variable to the C code:: + + /*[python input] + print('static int __ignored_unused_variable__ = 0;') + [python start generated code]*/ + static int __ignored_unused_variable__ = 0; + /*[python checksum:...]*/ + + +.. _clinic-howto-self-converter: + +How to use the "self converter" +------------------------------- + +Argument Clinic automatically adds a "self" parameter for you +using a default converter. It automatically sets the ``type`` +of this parameter to the "pointer to an instance" you specified +when you declared the type. However, you can override +Argument Clinic's converter and specify one yourself. +Just add your own *self* parameter as the first parameter in a +block, and ensure that its converter is an instance of +:class:`!self_converter` or a subclass thereof. + +What's the point? This lets you override the type of ``self``, +or give it a different default name. + +How do you specify the custom type you want to cast ``self`` to? +If you only have one or two functions with the same type for ``self``, +you can directly use Argument Clinic's existing ``self`` converter, +passing in the type you want to use as the *type* parameter:: + + /*[clinic input] + + _pickle.Pickler.dump + + self: self(type="PicklerObject *") + obj: object + / + + Write a pickled representation of the given object to the open file. + [clinic start generated code]*/ + +On the other hand, if you have a lot of functions that will use the same +type for ``self``, it's best to create your own converter, subclassing +:class:`!self_converter` but overwriting the :py:attr:`!type` member:: + + /*[python input] + class PicklerObject_converter(self_converter): + type = "PicklerObject *" + [python start generated code]*/ + + /*[clinic input] + + _pickle.Pickler.dump + + self: PicklerObject + obj: object + / + + Write a pickled representation of the given object to the open file. + [clinic start generated code]*/ + + +How to use the "defining class" converter +----------------------------------------- + +Argument Clinic facilitates gaining access to the defining class of a method. +This is useful for :ref:`heap type ` methods that need to fetch +module level state. Use :c:func:`PyType_FromModuleAndSpec` to associate a new +heap type with a module. You can now use :c:func:`PyType_GetModuleState` on +the defining class to fetch the module state, for example from a module method. + +Example from :source:`Modules/zlibmodule.c`. +First, ``defining_class`` is added to the clinic input:: + + /*[clinic input] + zlib.Compress.compress + + cls: defining_class + data: Py_buffer + Binary data to be compressed. + / + + +After running the Argument Clinic tool, the following function signature is +generated:: + + /*[clinic start generated code]*/ + static PyObject * + zlib_Compress_compress_impl(compobject *self, PyTypeObject *cls, + Py_buffer *data) + /*[clinic end generated code: output=6731b3f0ff357ca6 input=04d00f65ab01d260]*/ + + +The following code can now use ``PyType_GetModuleState(cls)`` to fetch the +module state:: + + zlibstate *state = PyType_GetModuleState(cls); + + +Each method may only have one argument using this converter, and it must appear +after ``self``, or, if ``self`` is not used, as the first argument. The argument +will be of type ``PyTypeObject *``. The argument will not appear in the +:py:attr:`!__text_signature__`. + +The ``defining_class`` converter is not compatible with :py:meth:`!__init__` +and :py:meth:`!__new__` methods, which cannot use the :c:macro:`METH_METHOD` +convention. + +It is not possible to use ``defining_class`` with slot methods. In order to +fetch the module state from such methods, use :c:func:`PyType_GetModuleByDef` +to look up the module and then :c:func:`PyModule_GetState` to fetch the module +state. Example from the ``setattro`` slot method in +:source:`Modules/_threadmodule.c`:: + + static int + local_setattro(localobject *self, PyObject *name, PyObject *v) + { + PyObject *module = PyType_GetModuleByDef(Py_TYPE(self), &thread_module); + thread_module_state *state = get_thread_state(module); + ... + } + + +See also :pep:`573`. + + +.. _clinic-howto-custom-converter: + +How to write a custom converter +------------------------------- + +A converter is a Python class that inherits from :py:class:`CConverter`. +The main purpose of a custom converter, is for parameters parsed with +the ``O&`` format unit --- parsing such a parameter means calling +a :c:func:`PyArg_ParseTuple` "converter function". + +Your converter class should be named :samp:`{ConverterName}_converter`. +By following this convention, your converter class will be automatically +registered with Argument Clinic, with its *converter name* being the name of +your converter class with the ``_converter`` suffix stripped off. + +Instead of subclassing :py:meth:`!CConverter.__init__`, +write a :py:meth:`!converter_init` method. +:py:meth:`!converter_init` always accepts a *self* parameter. +After *self*, all additional parameters **must** be keyword-only. +Any arguments passed to the converter in Argument Clinic +will be passed along to your :py:meth:`!converter_init` method. +See :py:class:`CConverter` for a list of members you may wish to specify in +your subclass. + +Here's the simplest example of a custom converter, from :source:`Modules/zlibmodule.c`:: + + /*[python input] + + class ssize_t_converter(CConverter): + type = 'Py_ssize_t' + converter = 'ssize_t_converter' + + [python start generated code]*/ + /*[python end generated code: output=da39a3ee5e6b4b0d input=35521e4e733823c7]*/ + +This block adds a converter named ``ssize_t`` to Argument Clinic. +Parameters declared as ``ssize_t`` will be declared with type :c:type:`Py_ssize_t`, +and will be parsed by the ``'O&'`` format unit, +which will call the :c:func:`!ssize_t_converter` converter C function. +``ssize_t`` variables automatically support default values. + +More sophisticated custom converters can insert custom C code to +handle initialization and cleanup. +You can see more examples of custom converters in the CPython +source tree; grep the C files for the string ``CConverter``. + + +How to write a custom return converter +-------------------------------------- + +Writing a custom return converter is much like writing +a custom converter. Except it's somewhat simpler, because return +converters are themselves much simpler. + +Return converters must subclass :py:class:`!CReturnConverter`. +There are no examples yet of custom return converters, +because they are not widely used yet. If you wish to +write your own return converter, please read :source:`Tools/clinic/clinic.py`, +specifically the implementation of :py:class:`!CReturnConverter` and +all its subclasses. + + +How to convert ``METH_O`` and ``METH_NOARGS`` functions +------------------------------------------------------- + +To convert a function using :c:macro:`METH_O`, make sure the function's +single argument is using the ``object`` converter, and mark the +arguments as positional-only:: + + /*[clinic input] + meth_o_sample + + argument: object + / + [clinic start generated code]*/ + + +To convert a function using :c:macro:`METH_NOARGS`, just don't specify +any arguments. + +You can still use a self converter, a return converter, and specify +a *type* argument to the object converter for :c:macro:`METH_O`. + + +How to convert ``tp_new`` and ``tp_init`` functions +--------------------------------------------------- + +You can convert :c:member:`~PyTypeObject.tp_new` and +:c:member:`~PyTypeObject.tp_init` functions. +Just name them ``__new__`` or ``__init__`` as appropriate. Notes: + +* The function name generated for ``__new__`` doesn't end in ``__new__`` + like it would by default. It's just the name of the class, converted + into a valid C identifier. + +* No :c:type:`PyMethodDef` ``#define`` is generated for these functions. + +* ``__init__`` functions return ``int``, not ``PyObject *``. + +* Use the docstring as the class docstring. + +* Although ``__new__`` and ``__init__`` functions must always + accept both the ``args`` and ``kwargs`` objects, when converting + you may specify any signature for these functions that you like. + (If your function doesn't support keywords, the parsing function + generated will throw an exception if it receives any.) + + +How to change and redirect Clinic's output +------------------------------------------ + +It can be inconvenient to have Clinic's output interspersed with +your conventional hand-edited C code. Luckily, Clinic is configurable: +you can buffer up its output for printing later (or earlier!), or write +its output to a separate file. You can also add a prefix or suffix to +every line of Clinic's generated output. + +While changing Clinic's output in this manner can be a boon to readability, +it may result in Clinic code using types before they are defined, or +your code attempting to use Clinic-generated code before it is defined. +These problems can be easily solved by rearranging the declarations in your file, +or moving where Clinic's generated code goes. (This is why the default behavior +of Clinic is to output everything into the current block; while many people +consider this hampers readability, it will never require rearranging your +code to fix definition-before-use problems.) + +Let's start with defining some terminology: + +*field* + A field, in this context, is a subsection of Clinic's output. + For example, the ``#define`` for the :c:type:`PyMethodDef` structure + is a field, called ``methoddef_define``. Clinic has seven + different fields it can output per function definition: + + .. code-block:: none + + docstring_prototype + docstring_definition + methoddef_define + impl_prototype + parser_prototype + parser_definition + impl_definition + + All the names are of the form ``"_"``, + where ``""`` is the semantic object represented (the parsing function, + the impl function, the docstring, or the methoddef structure) and ``""`` + represents what kind of statement the field is. Field names that end in + ``"_prototype"`` + represent forward declarations of that thing, without the actual body/data + of the thing; field names that end in ``"_definition"`` represent the actual + definition of the thing, with the body/data of the thing. (``"methoddef"`` + is special, it's the only one that ends with ``"_define"``, representing that + it's a preprocessor #define.) + +*destination* + A destination is a place Clinic can write output to. There are + five built-in destinations: + + ``block`` + The default destination: printed in the output section of + the current Clinic block. + + ``buffer`` + A text buffer where you can save text for later. Text sent + here is appended to the end of any existing text. It's an + error to have any text left in the buffer when Clinic finishes + processing a file. + + ``file`` + A separate "clinic file" that will be created automatically by Clinic. + The filename chosen for the file is ``{basename}.clinic{extension}``, + where ``basename`` and ``extension`` were assigned the output + from ``os.path.splitext()`` run on the current file. (Example: + the ``file`` destination for :file:`_pickle.c` would be written to + :file:`_pickle.clinic.c`.) + + **Important: When using a** ``file`` **destination, you** + *must check in* **the generated file!** + + ``two-pass`` + A buffer like ``buffer``. However, a two-pass buffer can only + be dumped once, and it prints out all text sent to it during + all processing, even from Clinic blocks *after* the dumping point. + + ``suppress`` + The text is suppressed—thrown away. + + +Clinic defines five new directives that let you reconfigure its output. + +The first new directive is ``dump``: + +.. code-block:: none + + dump + +This dumps the current contents of the named destination into the output of +the current block, and empties it. This only works with ``buffer`` and +``two-pass`` destinations. + +The second new directive is ``output``. The most basic form of ``output`` +is like this: + +.. code-block:: none + + output + +This tells Clinic to output *field* to *destination*. ``output`` also +supports a special meta-destination, called ``everything``, which tells +Clinic to output *all* fields to that *destination*. + +``output`` has a number of other functions: + +.. code-block:: none + + output push + output pop + output preset + + +``output push`` and ``output pop`` allow you to push and pop +configurations on an internal configuration stack, so that you +can temporarily modify the output configuration, then easily restore +the previous configuration. Simply push before your change to save +the current configuration, then pop when you wish to restore the +previous configuration. + +``output preset`` sets Clinic's output to one of several built-in +preset configurations, as follows: + + ``block`` + Clinic's original starting configuration. Writes everything + immediately after the input block. + + Suppress the ``parser_prototype`` + and ``docstring_prototype``, write everything else to ``block``. + + ``file`` + Designed to write everything to the "clinic file" that it can. + You then ``#include`` this file near the top of your file. + You may need to rearrange your file to make this work, though + usually this just means creating forward declarations for various + ``typedef`` and ``PyTypeObject`` definitions. + + Suppress the ``parser_prototype`` + and ``docstring_prototype``, write the ``impl_definition`` to + ``block``, and write everything else to ``file``. + + The default filename is ``"{dirname}/clinic/{basename}.h"``. + + ``buffer`` + Save up most of the output from Clinic, to be written into + your file near the end. For Python files implementing modules + or builtin types, it's recommended that you dump the buffer + just above the static structures for your module or + builtin type; these are normally very near the end. Using + ``buffer`` may require even more editing than ``file``, if + your file has static ``PyMethodDef`` arrays defined in the + middle of the file. + + Suppress the ``parser_prototype``, ``impl_prototype``, + and ``docstring_prototype``, write the ``impl_definition`` to + ``block``, and write everything else to ``file``. + + ``two-pass`` + Similar to the ``buffer`` preset, but writes forward declarations to + the ``two-pass`` buffer, and definitions to the ``buffer``. + This is similar to the ``buffer`` preset, but may require + less editing than ``buffer``. Dump the ``two-pass`` buffer + near the top of your file, and dump the ``buffer`` near + the end just like you would when using the ``buffer`` preset. + + Suppresses the ``impl_prototype``, write the ``impl_definition`` + to ``block``, write ``docstring_prototype``, ``methoddef_define``, + and ``parser_prototype`` to ``two-pass``, write everything else + to ``buffer``. + + ``partial-buffer`` + Similar to the ``buffer`` preset, but writes more things to ``block``, + only writing the really big chunks of generated code to ``buffer``. + This avoids the definition-before-use problem of ``buffer`` completely, + at the small cost of having slightly more stuff in the block's output. + Dump the ``buffer`` near the end, just like you would when using + the ``buffer`` preset. + + Suppresses the ``impl_prototype``, write the ``docstring_definition`` + and ``parser_definition`` to ``buffer``, write everything else to ``block``. + +The third new directive is ``destination``: + +.. code-block:: none + + destination [...] + +This performs an operation on the destination named ``name``. + +There are two defined subcommands: ``new`` and ``clear``. + +The ``new`` subcommand works like this: + +.. code-block:: none + + destination new + +This creates a new destination with name ```` and type ````. + +There are five destination types: + + ``suppress`` + Throws the text away. + + ``block`` + Writes the text to the current block. This is what Clinic + originally did. + + ``buffer`` + A simple text buffer, like the "buffer" builtin destination above. + + ``file`` + A text file. The file destination takes an extra argument, + a template to use for building the filename, like so: + + destination new + + The template can use three strings internally that will be replaced + by bits of the filename: + + {path} + The full path to the file, including directory and full filename. + {dirname} + The name of the directory the file is in. + {basename} + Just the name of the file, not including the directory. + {basename_root} + Basename with the extension clipped off + (everything up to but not including the last '.'). + {basename_extension} + The last '.' and everything after it. If the basename + does not contain a period, this will be the empty string. + + If there are no periods in the filename, {basename} and {filename} + are the same, and {extension} is empty. "{basename}{extension}" + is always exactly the same as "{filename}"." + + ``two-pass`` + A two-pass buffer, like the "two-pass" builtin destination above. + + +The ``clear`` subcommand works like this: + +.. code-block:: none + + destination clear + +It removes all the accumulated text up to this point in the destination. +(I don't know what you'd need this for, but I thought maybe it'd be +useful while someone's experimenting.) + +The fourth new directive is ``set``: + +.. code-block:: none + + set line_prefix "string" + set line_suffix "string" + +``set`` lets you set two internal variables in Clinic. +``line_prefix`` is a string that will be prepended to every line of Clinic's output; +``line_suffix`` is a string that will be appended to every line of Clinic's output. + +Both of these support two format strings: + + ``{block comment start}`` + Turns into the string ``/*``, the start-comment text sequence for C files. + + ``{block comment end}`` + Turns into the string ``*/``, the end-comment text sequence for C files. + +The final new directive is one you shouldn't need to use directly, +called ``preserve``: + +.. code-block:: none + + preserve + +This tells Clinic that the current contents of the output should be kept, unmodified. +This is used internally by Clinic when dumping output into ``file`` files; wrapping +it in a Clinic block lets Clinic use its existing checksum functionality to ensure +the file was not modified by hand before it gets overwritten. + + +How to use the ``#ifdef`` trick +------------------------------- + +If you're converting a function that isn't available on all platforms, +there's a trick you can use to make life a little easier. The existing +code probably looks like this:: + + #ifdef HAVE_FUNCTIONNAME + static module_functionname(...) + { + ... + } + #endif /* HAVE_FUNCTIONNAME */ + +And then in the ``PyMethodDef`` structure at the bottom the existing code +will have: + +.. code-block:: none + + #ifdef HAVE_FUNCTIONNAME + {'functionname', ... }, + #endif /* HAVE_FUNCTIONNAME */ + +In this scenario, you should enclose the body of your impl function inside the ``#ifdef``, +like so:: + + #ifdef HAVE_FUNCTIONNAME + /*[clinic input] + module.functionname + ... + [clinic start generated code]*/ + static module_functionname(...) + { + ... + } + #endif /* HAVE_FUNCTIONNAME */ + +Then, remove those three lines from the :c:type:`PyMethodDef` structure, +replacing them with the macro Argument Clinic generated: + +.. code-block:: none + + MODULE_FUNCTIONNAME_METHODDEF + +(You can find the real name for this macro inside the generated code. +Or you can calculate it yourself: it's the name of your function as defined +on the first line of your block, but with periods changed to underscores, +uppercased, and ``"_METHODDEF"`` added to the end.) + +Perhaps you're wondering: what if ``HAVE_FUNCTIONNAME`` isn't defined? +The ``MODULE_FUNCTIONNAME_METHODDEF`` macro won't be defined either! + +Here's where Argument Clinic gets very clever. It actually detects that the +Argument Clinic block might be deactivated by the ``#ifdef``. When that +happens, it generates a little extra code that looks like this:: + + #ifndef MODULE_FUNCTIONNAME_METHODDEF + #define MODULE_FUNCTIONNAME_METHODDEF + #endif /* !defined(MODULE_FUNCTIONNAME_METHODDEF) */ + +That means the macro always works. If the function is defined, this turns +into the correct structure, including the trailing comma. If the function is +undefined, this turns into nothing. + +However, this causes one ticklish problem: where should Argument Clinic put this +extra code when using the "block" output preset? It can't go in the output block, +because that could be deactivated by the ``#ifdef``. (That's the whole point!) + +In this situation, Argument Clinic writes the extra code to the "buffer" destination. +This may mean that you get a complaint from Argument Clinic: + +.. code-block:: none + + Warning in file "Modules/posixmodule.c" on line 12357: + Destination buffer 'buffer' not empty at end of file, emptying. + +When this happens, just open your file, find the ``dump buffer`` block that +Argument Clinic added to your file (it'll be at the very bottom), then +move it above the :c:type:`PyMethodDef` structure where that macro is used. + + +How to use Argument Clinic in Python files +------------------------------------------ + +It's actually possible to use Argument Clinic to preprocess Python files. +There's no point to using Argument Clinic blocks, of course, as the output +wouldn't make any sense to the Python interpreter. But using Argument Clinic +to run Python blocks lets you use Python as a Python preprocessor! + +Since Python comments are different from C comments, Argument Clinic +blocks embedded in Python files look slightly different. They look like this: + +.. code-block:: python3 + + #/*[python input] + #print("def foo(): pass") + #[python start generated code]*/ + def foo(): pass + #/*[python checksum:...]*/ + + +.. _clinic-howto-override-signature: + +How to override the generated signature +--------------------------------------- + +You can use the ``@text_signature`` directive to override the default generated +signature in the docstring. +This can be useful for complex signatures that Argument Clinic cannot handle. +The ``@text_signature`` directive takes one argument: +the custom signature as a string. +The provided signature is copied verbatim to the generated docstring. + +Example from :source:`Objects/codeobject.c`:: + + /*[clinic input] + @text_signature "($self, /, **changes)" + code.replace + * + co_argcount: int(c_default="self->co_argcount") = unchanged + co_posonlyargcount: int(c_default="self->co_posonlyargcount") = unchanged + # etc ... + + Return a copy of the code object with new values for the specified fields. + [clinic start generated output]*/ + +The generated docstring ends up looking like this: + +.. code-block:: none + + replace($self, /, **changes) + -- + + Return a copy of the code object with new values for the specified fields. + + +.. _clinic-howto-deprecate-positional: + +How to deprecate passing parameters positionally +------------------------------------------------ + +Argument Clinic provides syntax that makes it possible to generate code that +deprecates passing :term:`arguments ` positionally. +For example, say we've got a module-level function :py:func:`!foo.myfunc` +that has three :term:`parameters `: +positional-or-keyword parameters *a* and *b*, and a keyword-only parameter *c*:: + + /*[clinic input] + module foo + myfunc + a: int + b: int + * + c: int + [clinic start generated output]*/ + +We now want to make the *b* parameter keyword-only; +however, we'll have to wait two releases before making this change, +as mandated by Python's backwards-compatibility policy (see :pep:`387`). +For this example, imagine we're in the development phase for Python 3.12: +that means we'll be allowed to introduce deprecation warnings in Python 3.12 +whenever the *b* parameter is passed positionally, +and we'll be allowed to make it keyword-only in Python 3.14 at the earliest. + +We can use Argument Clinic to emit the desired deprecation warnings +using the ``* [from ...]`` syntax, +by adding the line ``* [from 3.14]`` right above the *b* parameter:: + + /*[clinic input] + module foo + myfunc + a: int + * [from 3.14] + b: int + * + c: int + [clinic start generated output]*/ + +Next, regenerate Argument Clinic code (``make clinic``), +and add unit tests for the new behaviour. + +The generated code will now emit a :exc:`DeprecationWarning` +when an :term:`argument` for the :term:`parameter` *b* is passed positionally. +C preprocessor directives are also generated for emitting +compiler warnings if the ``* [from ...]`` line has not been removed +from the Argument Clinic input when the deprecation period is over, +which means when the alpha phase of the specified Python version kicks in. + +Let's return to our example and skip ahead two years: +Python 3.14 development has now entered the alpha phase, +but we forgot all about updating the Argument Clinic code +for :py:func:`!myfunc`! +Luckily for us, compiler warnings are now generated: + +.. code-block:: none + + In file included from Modules/foomodule.c:139: + Modules/clinic/foomodule.c.h:139:8: warning: In 'foomodule.c', update parameter(s) 'a' and 'b' in the clinic input of 'mymod.myfunc' to be keyword-only. [-W#warnings] + # warning "In 'foomodule.c', update parameter(s) 'a' and 'b' in the clinic input of 'mymod.myfunc' to be keyword-only. [-W#warnings]" + ^ + +We now close the deprecation phase by making *b* keyword-only; +replace the ``* [from ...]`` line above *b* +with the ``*`` from the line above *c*:: + + /*[clinic input] + module foo + myfunc + a: int + * + b: int + c: int + [clinic start generated output]*/ + +Finally, run ``make clinic`` to regenerate the Argument Clinic code, +and update your unit tests to reflect the new behaviour. + +.. note:: + + If you forget to update your input block during the alpha and beta phases, + the compiler warning will turn into a compiler error when the + release candidate phase begins. From 62ffd34a07b6c3b54b3c4878a8d8df6977c53f07 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Wed, 9 Aug 2023 15:52:27 +0200 Subject: [PATCH 69/71] Remove migrated clinic.rst path (Doc/howto/clinic.rst) --- Doc/howto/clinic.rst | 2028 ------------------------------------------ 1 file changed, 2028 deletions(-) delete mode 100644 Doc/howto/clinic.rst diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst deleted file mode 100644 index 463a938fa..000000000 --- a/Doc/howto/clinic.rst +++ /dev/null @@ -1,2028 +0,0 @@ -.. highlight:: c - -.. _howto-clinic: - -********************** -Argument Clinic How-To -********************** - -:author: Larry Hastings - -**Source code:** :source:`Tools/clinic/clinic.py`. - -.. topic:: Abstract - - Argument Clinic is a preprocessor for CPython C files. - It was introduced in Python 3.4 with :pep:`436`, - in order to provide introspection signatures, - and to generate performant and tailor-made boilerplate code - for argument parsing in CPython builtins, - module level functions, and class methods. - This document is divided in four major sections: - - * :ref:`clinic-background` talks about the basic concepts and goals of - Argument Clinic. - * :ref:`clinic-reference` describes the command-line interface and Argument - Clinic terminology. - * :ref:`clinic-tutorial` guides you through all the steps required to - adapt an existing C function to Argument Clinic. - * :ref:`clinic-howtos` details how to handle specific tasks. - - -.. note:: - - Argument Clinic is considered internal-only - for CPython. Its use is not supported for files outside - CPython, and no guarantees are made regarding backwards - compatibility for future versions. In other words: if you - maintain an external C extension for CPython, you're welcome - to experiment with Argument Clinic in your own code. But the - version of Argument Clinic that ships with the next version - of CPython *could* be totally incompatible and break all your code. - - -.. _clinic-background: - -Background -========== - -Basic concepts --------------- - -When Argument Clinic is run on a file, either via the :ref:`clinic-cli` -or via ``make clinic``, it will scan over the input files looking for -:term:`start lines `: - -.. code-block:: none - - /*[clinic input] - -When it finds one, it reads everything up to the :term:`end line`: - -.. code-block:: none - - [clinic start generated code]*/ - -Everything in between these two lines is Argument Clinic :term:`input`. -When Argument Clinic parses input, it generates :term:`output`. -The output is rewritten into the C file immediately after the input, -followed by a :term:`checksum line`. -All of these lines, including the :term:`start line` and :term:`checksum line`, -are collectively called an Argument Clinic :term:`block`: - -.. code-block:: none - - /*[clinic input] - ... clinic input goes here ... - [clinic start generated code]*/ - ... clinic output goes here ... - /*[clinic end generated code: ...]*/ - -If you run Argument Clinic on the same file a second time, Argument Clinic -will discard the old :term:`output` and write out the new output with a fresh -:term:`checksum line`. -If the :term:`input` hasn't changed, the output won't change either. - -.. note:: - - You should never modify the output of an Argument Clinic block, - as any change will be lost in future Argument Clinic runs; - Argument Clinic will detect an output checksum mismatch and regenerate the - correct output. - If you are not happy with the generated output, - you should instead change the input until it produces the output you want. - - -.. _clinic-reference: - -Reference -========= - - -.. _clinic-terminology: - -Terminology ------------ - -.. glossary:: - - start line - The line ``/*[clinic input]``. - This line marks the beginning of Argument Clinic input. - Note that the *start line* opens a C block comment. - - end line - The line ``[clinic start generated code]*/``. - The *end line* marks the _end_ of Argument Clinic :term:`input`, - but at the same time marks the _start_ of Argument Clinic :term:`output`, - thus the text *"clinic start start generated code"* - Note that the *end line* closes the C block comment opened - by the *start line*. - - checksum - A hash to distinguish unique :term:`inputs ` - and :term:`outputs `. - - checksum line - A line that looks like ``/*[clinic end generated code: ...]*/``. - The three dots will be replaced by a :term:`checksum` generated from the - :term:`input`, and a :term:`checksum` generated from the :term:`output`. - The checksum line marks the end of Argument Clinic generated code, - and is used by Argument Clinic to determine if it needs to regenerate - output. - - input - The text between the :term:`start line` and the :term:`end line`. - Note that the start and end lines open and close a C block comment; - the *input* is thus a part of that same C block comment. - - output - The text between the :term:`end line` and the :term:`checksum line`. - - block - All text from the :term:`start line` to the :term:`checksum line` inclusively. - - -.. _clinic-cli: - -Command-line interface ----------------------- - -The Argument Clinic :abbr:`CLI (Command-Line Interface)` is typically used to -process a single source file, like this: - -.. code-block:: shell-session - - $ python3 ./Tools/clinic/clinic.py foo.c - -The CLI supports the following options: - -.. program:: ./Tools/clinic/clinic.py [-h] [-f] [-o OUTPUT] [-v] \ - [--converters] [--make] [--srcdir SRCDIR] [FILE ...] - -.. option:: -h, --help - - Print CLI usage. - -.. option:: -f, --force - - Force output regeneration. - -.. option:: -o, --output OUTPUT - - Redirect file output to OUTPUT - -.. option:: -v, --verbose - - Enable verbose mode. - -.. option:: --converters - - Print a list of all supported converters and return converters. - -.. option:: --make - - Walk :option:`--srcdir` to run over all relevant files. - -.. option:: --srcdir SRCDIR - - The directory tree to walk in :option:`--make` mode. - -.. option:: --exclude EXCLUDE - - A file to exclude in :option:`--make` mode. - This option can be given multiple times. - -.. option:: FILE ... - - The list of files to process. - - -.. _clinic-classes: - -Classes for extending Argument Clinic -------------------------------------- - -.. module:: clinic - -.. class:: CConverter - - The base class for all converters. - See :ref:`clinic-howto-custom-converter` for how to subclass this class. - - .. attribute:: type - - The C type to use for this variable. - :attr:`!type` should be a Python string specifying the type, - e.g. ``'int'``. - If this is a pointer type, the type string should end with ``' *'``. - - .. attribute:: default - - The Python default value for this parameter, as a Python value. - Or the magic value ``unspecified`` if there is no default. - - .. attribute:: py_default - - :attr:`!default` as it should appear in Python code, - as a string. - Or ``None`` if there is no default. - - .. attribute:: c_default - - :attr:`!default` as it should appear in C code, - as a string. - Or ``None`` if there is no default. - - .. attribute:: c_ignored_default - - The default value used to initialize the C variable when - there is no default, but not specifying a default may - result in an "uninitialized variable" warning. This can - easily happen when using option groups—although - properly written code will never actually use this value, - the variable does get passed in to the impl, and the - C compiler will complain about the "use" of the - uninitialized value. This value should always be a - non-empty string. - - .. attribute:: converter - - The name of the C converter function, as a string. - - .. attribute:: impl_by_reference - - A boolean value. If true, - Argument Clinic will add a ``&`` in front of the name of - the variable when passing it into the impl function. - - .. attribute:: parse_by_reference - - A boolean value. If true, - Argument Clinic will add a ``&`` in front of the name of - the variable when passing it into :c:func:`PyArg_ParseTuple`. - - -.. _clinic-tutorial: - -Tutorial -======== - -The best way to get a sense of how Argument Clinic works is to -convert a function to work with it. Here, then, are the bare -minimum steps you'd need to follow to convert a function to -work with Argument Clinic. Note that for code you plan to -check in to CPython, you really should take the conversion farther, -using some of the :ref:`advanced concepts ` -you'll see later on in the document, -like :ref:`clinic-howto-return-converters` -and :ref:`clinic-howto-self-converter`. -But we'll keep it simple for this walkthrough so you can learn. - -First, make sure you're working with a freshly updated checkout -of the CPython trunk. - -Next, find a Python builtin that calls either :c:func:`PyArg_ParseTuple` -or :c:func:`PyArg_ParseTupleAndKeywords`, and hasn't been converted -to work with Argument Clinic yet. -For this tutorial, we'll be using -:py:meth:`_pickle.Pickler.dump `. - -If the call to the :c:func:`!PyArg_Parse*` function uses any of the -following format units...: - - .. code-block:: none - - O& - O! - es - es# - et - et# - -... or if it has multiple calls to :c:func:`PyArg_ParseTuple`, -you should choose a different function. -(See :ref:`clinic-howto-advanced-converters` for those scenarios.) - -Also, if the function has multiple calls to :c:func:`!PyArg_ParseTuple` -or :c:func:`PyArg_ParseTupleAndKeywords` where it supports different -types for the same argument, or if the function uses something besides -:c:func:`!PyArg_Parse*` functions to parse its arguments, it probably -isn't suitable for conversion to Argument Clinic. Argument Clinic -doesn't support generic functions or polymorphic parameters. - -Next, add the following boilerplate above the function, -creating our input block:: - - /*[clinic input] - [clinic start generated code]*/ - -Cut the docstring and paste it in between the ``[clinic]`` lines, -removing all the junk that makes it a properly quoted C string. -When you're done you should have just the text, based at the left -margin, with no line wider than 80 characters. -Argument Clinic will preserve indents inside the docstring. - -If the old docstring had a first line that looked like a function -signature, throw that line away; The docstring doesn't need it anymore --- -when you use :py:func:`help` on your builtin in the future, -the first line will be built automatically based on the function's signature. - -Example docstring summary line:: - - /*[clinic input] - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ - -If your docstring doesn't have a "summary" line, Argument Clinic will -complain, so let's make sure it has one. The "summary" line should -be a paragraph consisting of a single 80-column line -at the beginning of the docstring. -(See :pep:`257` regarding docstring conventions.) - -Our example docstring consists solely of a summary line, so the sample -code doesn't have to change for this step. - -Now, above the docstring, enter the name of the function, followed -by a blank line. This should be the Python name of the function, -and should be the full dotted path to the function --- -it should start with the name of the module, -include any sub-modules, and if the function is a method on -a class it should include the class name too. - -In our example, :mod:`!_pickle` is the module, :py:class:`!Pickler` is the class, -and :py:meth:`!dump` is the method, so the name becomes -:py:meth:`!_pickle.Pickler.dump`:: - - /*[clinic input] - _pickle.Pickler.dump - - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ - -If this is the first time that module or class has been used with Argument -Clinic in this C file, -you must declare the module and/or class. Proper Argument Clinic hygiene -prefers declaring these in a separate block somewhere near the -top of the C file, in the same way that include files and statics go at -the top. -In our sample code we'll just show the two blocks next to each other. - -The name of the class and module should be the same as the one -seen by Python. Check the name defined in the :c:type:`PyModuleDef` -or :c:type:`PyTypeObject` as appropriate. - -When you declare a class, you must also specify two aspects of its type -in C: the type declaration you'd use for a pointer to an instance of -this class, and a pointer to the :c:type:`!PyTypeObject` for this class:: - - /*[clinic input] - module _pickle - class _pickle.Pickler "PicklerObject *" "&Pickler_Type" - [clinic start generated code]*/ - - /*[clinic input] - _pickle.Pickler.dump - - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ - -Declare each of the parameters to the function. Each parameter -should get its own line. All the parameter lines should be -indented from the function name and the docstring. -The general form of these parameter lines is as follows: - -.. code-block:: none - - name_of_parameter: converter - -If the parameter has a default value, add that after the -converter: - -.. code-block:: none - - name_of_parameter: converter = default_value - -Argument Clinic's support for "default values" is quite sophisticated; -see :ref:`clinic-howto-default-values` for more information. - -Next, add a blank line below the parameters. - -What's a "converter"? -It establishes both the type of the variable used in C, -and the method to convert the Python value into a C value at runtime. -For now you're going to use what's called a "legacy converter" --- -a convenience syntax intended to make porting old code into Argument -Clinic easier. - -For each parameter, copy the "format unit" for that -parameter from the :c:func:`PyArg_Parse` format argument and -specify *that* as its converter, as a quoted string. -The "format unit" is the formal name for the one-to-three -character substring of the *format* parameter that tells -the argument parsing function what the type of the variable -is and how to convert it. -For more on format units please see :ref:`arg-parsing`. - -For multicharacter format units like ``z#``, -use the entire two-or-three character string. - -Sample:: - - /*[clinic input] - module _pickle - class _pickle.Pickler "PicklerObject *" "&Pickler_Type" - [clinic start generated code]*/ - - /*[clinic input] - _pickle.Pickler.dump - - obj: 'O' - - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ - -If your function has ``|`` in the format string, -meaning some parameters have default values, you can ignore it. -Argument Clinic infers which parameters are optional -based on whether or not they have default values. - -If your function has ``$`` in the format string, -meaning it takes keyword-only arguments, -specify ``*`` on a line by itself before the first keyword-only argument, -indented the same as the parameter lines. - -:py:meth:`!_pickle.Pickler.dump` has neither, so our sample is unchanged. - -Next, if the existing C function calls :c:func:`PyArg_ParseTuple` -(as opposed to :c:func:`PyArg_ParseTupleAndKeywords`), then all its -arguments are positional-only. - -To mark parameters as positional-only in Argument Clinic, -add a ``/`` on a line by itself after the last positional-only parameter, -indented the same as the parameter lines. - -Sample:: - - /*[clinic input] - module _pickle - class _pickle.Pickler "PicklerObject *" "&Pickler_Type" - [clinic start generated code]*/ - - /*[clinic input] - _pickle.Pickler.dump - - obj: 'O' - / - - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ - -It can be helpful to write a per-parameter docstring for each parameter. -Since per-parameter docstrings are optional, -you can skip this step if you prefer. - -Nevertheless, here's how to add a per-parameter docstring. -The first line of the per-parameter docstring -must be indented further than the parameter definition. -The left margin of this first line establishes -the left margin for the whole per-parameter docstring; -all the text you write will be outdented by this amount. -You can write as much text as you like, across multiple lines if you wish. - -Sample:: - - /*[clinic input] - module _pickle - class _pickle.Pickler "PicklerObject *" "&Pickler_Type" - [clinic start generated code]*/ - - /*[clinic input] - _pickle.Pickler.dump - - obj: 'O' - The object to be pickled. - / - - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ - -Save and close the file, then run ``Tools/clinic/clinic.py`` on it. -With luck everything worked---your block now has output, -and a :file:`.c.h` file has been generated! -Reload the file in your text editor to see the generated code:: - - /*[clinic input] - _pickle.Pickler.dump - - obj: 'O' - The object to be pickled. - / - - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ - - static PyObject * - _pickle_Pickler_dump(PicklerObject *self, PyObject *obj) - /*[clinic end generated code: output=87ecad1261e02ac7 input=552eb1c0f52260d9]*/ - -Obviously, if Argument Clinic didn't produce any output, -it's because it found an error in your input. -Keep fixing your errors and retrying until Argument Clinic processes your file -without complaint. - -For readability, most of the glue code has been generated to a :file:`.c.h` -file. You'll need to include that in your original :file:`.c` file, -typically right after the clinic module block:: - - #include "clinic/_pickle.c.h" - -Double-check that the argument-parsing code Argument Clinic generated -looks basically the same as the existing code. - -First, ensure both places use the same argument-parsing function. -The existing code must call either -:c:func:`PyArg_ParseTuple` or :c:func:`PyArg_ParseTupleAndKeywords`; -ensure that the code generated by Argument Clinic calls the -*exact* same function. - -Second, the format string passed in to :c:func:`!PyArg_ParseTuple` or -:c:func:`!PyArg_ParseTupleAndKeywords` should be *exactly* the same -as the hand-written one in the existing function, -up to the colon or semi-colon. - -Argument Clinic always generates its format strings -with a ``:`` followed by the name of the function. -If the existing code's format string ends with ``;``, -to provide usage help, this change is harmless --- don't worry about it. - -Third, for parameters whose format units require two arguments, -like a length variable, an encoding string, or a pointer -to a conversion function, ensure that the second argument is -*exactly* the same between the two invocations. - -Fourth, inside the output portion of the block, -you'll find a preprocessor macro defining the appropriate static -:c:type:`PyMethodDef` structure for this builtin:: - - #define __PICKLE_PICKLER_DUMP_METHODDEF \ - {"dump", (PyCFunction)__pickle_Pickler_dump, METH_O, __pickle_Pickler_dump__doc__}, - -This static structure should be *exactly* the same as the existing static -:c:type:`!PyMethodDef` structure for this builtin. - -If any of these items differ in *any way*, -adjust your Argument Clinic function specification and rerun -``Tools/clinic/clinic.py`` until they *are* the same. - -Notice that the last line of its output is the declaration -of your "impl" function. This is where the builtin's implementation goes. -Delete the existing prototype of the function you're modifying, but leave -the opening curly brace. Now delete its argument parsing code and the -declarations of all the variables it dumps the arguments into. -Notice how the Python arguments are now arguments to this impl function; -if the implementation used different names for these variables, fix it. - -Let's reiterate, just because it's kind of weird. -Your code should now look like this:: - - static return_type - your_function_impl(...) - /*[clinic end generated code: input=..., output=...]*/ - { - ... - -Argument Clinic generated the checksum line and the function prototype just -above it. You should write the opening and closing curly braces for the -function, and the implementation inside. - -Sample:: - - /*[clinic input] - module _pickle - class _pickle.Pickler "PicklerObject *" "&Pickler_Type" - [clinic start generated code]*/ - /*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ - - /*[clinic input] - _pickle.Pickler.dump - - obj: 'O' - The object to be pickled. - / - - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ - - PyDoc_STRVAR(__pickle_Pickler_dump__doc__, - "Write a pickled representation of obj to the open file.\n" - "\n" - ... - static PyObject * - _pickle_Pickler_dump_impl(PicklerObject *self, PyObject *obj) - /*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/ - { - /* Check whether the Pickler was initialized correctly (issue3664). - Developers often forget to call __init__() in their subclasses, which - would trigger a segfault without this check. */ - if (self->write == NULL) { - PyErr_Format(PicklingError, - "Pickler.__init__() was not called by %s.__init__()", - Py_TYPE(self)->tp_name); - return NULL; - } - - if (_Pickler_ClearBuffer(self) < 0) { - return NULL; - } - - ... - -Remember the macro with the :c:type:`PyMethodDef` structure for this function? -Find the existing :c:type:`!PyMethodDef` structure for this -function and replace it with a reference to the macro. If the builtin -is at module scope, this will probably be very near the end of the file; -if the builtin is a class method, this will probably be below but relatively -near to the implementation. - -Note that the body of the macro contains a trailing comma; when you -replace the existing static :c:type:`!PyMethodDef` structure with the macro, -*don't* add a comma to the end. - -Sample:: - - static struct PyMethodDef Pickler_methods[] = { - __PICKLE_PICKLER_DUMP_METHODDEF - __PICKLE_PICKLER_CLEAR_MEMO_METHODDEF - {NULL, NULL} /* sentinel */ - }; - -Argument Clinic may generate new instances of ``_Py_ID``. For example:: - - &_Py_ID(new_unique_py_id) - -If it does, you'll have to run ``make regen-global-objects`` -to regenerate the list of precompiled identifiers at this point. - -Finally, compile, then run the relevant portions of the regression-test suite. -This change should not introduce any new compile-time warnings or errors, -and there should be no externally visible change to Python's behavior, -except for one difference: :py:func:`inspect.signature` run on your function -should now provide a valid signature! - -Congratulations, you've ported your first function to work with Argument Clinic! - - -.. _clinic-howtos: - -How-to guides -============= - - -How to rename C functions and variables generated by Argument Clinic --------------------------------------------------------------------- - -Argument Clinic automatically names the functions it generates for you. -Occasionally this may cause a problem, if the generated name collides with -the name of an existing C function. There's an easy solution: override the names -used for the C functions. Just add the keyword ``"as"`` -to your function declaration line, followed by the function name you wish to use. -Argument Clinic will use that function name for the base (generated) function, -then add ``"_impl"`` to the end and use that for the name of the impl function. - -For example, if we wanted to rename the C function names generated for -:py:meth:`pickle.Pickler.dump`, it'd look like this:: - - /*[clinic input] - pickle.Pickler.dump as pickler_dumper - - ... - -The base function would now be named :c:func:`!pickler_dumper`, -and the impl function would now be named :c:func:`!pickler_dumper_impl`. - - -Similarly, you may have a problem where you want to give a parameter -a specific Python name, but that name may be inconvenient in C. Argument -Clinic allows you to give a parameter different names in Python and in C, -using the same ``"as"`` syntax:: - - /*[clinic input] - pickle.Pickler.dump - - obj: object - file as file_obj: object - protocol: object = NULL - * - fix_imports: bool = True - -Here, the name used in Python (in the signature and the ``keywords`` -array) would be *file*, but the C variable would be named ``file_obj``. - -You can use this to rename the *self* parameter too! - - -How to convert functions using ``PyArg_UnpackTuple`` ----------------------------------------------------- - -To convert a function parsing its arguments with :c:func:`PyArg_UnpackTuple`, -simply write out all the arguments, specifying each as an ``object``. You -may specify the *type* argument to cast the type as appropriate. All -arguments should be marked positional-only (add a ``/`` on a line by itself -after the last argument). - -Currently the generated code will use :c:func:`PyArg_ParseTuple`, but this -will change soon. - - -How to use optional groups --------------------------- - -Some legacy functions have a tricky approach to parsing their arguments: -they count the number of positional arguments, then use a ``switch`` statement -to call one of several different :c:func:`PyArg_ParseTuple` calls depending on -how many positional arguments there are. (These functions cannot accept -keyword-only arguments.) This approach was used to simulate optional -arguments back before :c:func:`PyArg_ParseTupleAndKeywords` was created. - -While functions using this approach can often be converted to -use :c:func:`!PyArg_ParseTupleAndKeywords`, optional arguments, and default values, -it's not always possible. Some of these legacy functions have -behaviors :c:func:`!PyArg_ParseTupleAndKeywords` doesn't directly support. -The most obvious example is the builtin function :py:func:`range`, which has -an optional argument on the *left* side of its required argument! -Another example is :py:meth:`curses.window.addch`, which has a group of two -arguments that must always be specified together. (The arguments are -called *x* and *y*; if you call the function passing in *x*, -you must also pass in *y* — and if you don't pass in *x* you may not -pass in *y* either.) - -In any case, the goal of Argument Clinic is to support argument parsing -for all existing CPython builtins without changing their semantics. -Therefore Argument Clinic supports -this alternate approach to parsing, using what are called *optional groups*. -Optional groups are groups of arguments that must all be passed in together. -They can be to the left or the right of the required arguments. They -can *only* be used with positional-only parameters. - -.. note:: Optional groups are *only* intended for use when converting - functions that make multiple calls to :c:func:`PyArg_ParseTuple`! - Functions that use *any* other approach for parsing arguments - should *almost never* be converted to Argument Clinic using - optional groups. Functions using optional groups currently - cannot have accurate signatures in Python, because Python just - doesn't understand the concept. Please avoid using optional - groups wherever possible. - -To specify an optional group, add a ``[`` on a line by itself before -the parameters you wish to group together, and a ``]`` on a line by itself -after these parameters. As an example, here's how :py:meth:`curses.window.addch` -uses optional groups to make the first two parameters and the last -parameter optional:: - - /*[clinic input] - - curses.window.addch - - [ - x: int - X-coordinate. - y: int - Y-coordinate. - ] - - ch: object - Character to add. - - [ - attr: long - Attributes for the character. - ] - / - - ... - - -Notes: - -* For every optional group, one additional parameter will be passed into the - impl function representing the group. The parameter will be an int named - ``group_{direction}_{number}``, - where ``{direction}`` is either ``right`` or ``left`` depending on whether the group - is before or after the required parameters, and ``{number}`` is a monotonically - increasing number (starting at 1) indicating how far away the group is from - the required parameters. When the impl is called, this parameter will be set - to zero if this group was unused, and set to non-zero if this group was used. - (By used or unused, I mean whether or not the parameters received arguments - in this invocation.) - -* If there are no required arguments, the optional groups will behave - as if they're to the right of the required arguments. - -* In the case of ambiguity, the argument parsing code - favors parameters on the left (before the required parameters). - -* Optional groups can only contain positional-only parameters. - -* Optional groups are *only* intended for legacy code. Please do not - use optional groups for new code. - - -How to use real Argument Clinic converters, instead of "legacy converters" --------------------------------------------------------------------------- - -To save time, and to minimize how much you need to learn -to achieve your first port to Argument Clinic, the walkthrough above tells -you to use "legacy converters". "Legacy converters" are a convenience, -designed explicitly to make porting existing code to Argument Clinic -easier. And to be clear, their use is acceptable when porting code for -Python 3.4. - -However, in the long term we probably want all our blocks to -use Argument Clinic's real syntax for converters. Why? A couple -reasons: - -* The proper converters are far easier to read and clearer in their intent. -* There are some format units that are unsupported as "legacy converters", - because they require arguments, and the legacy converter syntax doesn't - support specifying arguments. -* In the future we may have a new argument parsing library that isn't - restricted to what :c:func:`PyArg_ParseTuple` supports; this flexibility - won't be available to parameters using legacy converters. - -Therefore, if you don't mind a little extra effort, please use the normal -converters instead of legacy converters. - -In a nutshell, the syntax for Argument Clinic (non-legacy) converters -looks like a Python function call. However, if there are no explicit -arguments to the function (all functions take their default values), -you may omit the parentheses. Thus ``bool`` and ``bool()`` are exactly -the same converters. - -All arguments to Argument Clinic converters are keyword-only. -All Argument Clinic converters accept the following arguments: - - *c_default* - The default value for this parameter when defined in C. - Specifically, this will be the initializer for the variable declared - in the "parse function". See :ref:`the section on default values ` - for how to use this. - Specified as a string. - - *annotation* - The annotation value for this parameter. Not currently supported, - because :pep:`8` mandates that the Python library may not use - annotations. - - *unused* - Wrap the argument with :c:macro:`Py_UNUSED` in the impl function signature. - -In addition, some converters accept additional arguments. Here is a list -of these arguments, along with their meanings: - - *accept* - A set of Python types (and possibly pseudo-types); - this restricts the allowable Python argument to values of these types. - (This is not a general-purpose facility; as a rule it only supports - specific lists of types as shown in the legacy converter table.) - - To accept ``None``, add ``NoneType`` to this set. - - *bitwise* - Only supported for unsigned integers. The native integer value of this - Python argument will be written to the parameter without any range checking, - even for negative values. - - *converter* - Only supported by the ``object`` converter. Specifies the name of a - :ref:`C "converter function" ` - to use to convert this object to a native type. - - *encoding* - Only supported for strings. Specifies the encoding to use when converting - this string from a Python str (Unicode) value into a C ``char *`` value. - - - *subclass_of* - Only supported for the ``object`` converter. Requires that the Python - value be a subclass of a Python type, as expressed in C. - - *type* - Only supported for the ``object`` and ``self`` converters. Specifies - the C type that will be used to declare the variable. Default value is - ``"PyObject *"``. - - *zeroes* - Only supported for strings. If true, embedded NUL bytes (``'\\0'``) are - permitted inside the value. The length of the string will be passed in - to the impl function, just after the string parameter, as a parameter named - ``_length``. - -Please note, not every possible combination of arguments will work. -Usually these arguments are implemented by specific :c:func:`PyArg_ParseTuple` -*format units*, with specific behavior. For example, currently you cannot -call ``unsigned_short`` without also specifying ``bitwise=True``. -Although it's perfectly reasonable to think this would work, these semantics don't -map to any existing format unit. So Argument Clinic doesn't support it. (Or, at -least, not yet.) - -Below is a table showing the mapping of legacy converters into real -Argument Clinic converters. On the left is the legacy converter, -on the right is the text you'd replace it with. - -========= ================================================================================= -``'B'`` ``unsigned_char(bitwise=True)`` -``'b'`` ``unsigned_char`` -``'c'`` ``char`` -``'C'`` ``int(accept={str})`` -``'d'`` ``double`` -``'D'`` ``Py_complex`` -``'es'`` ``str(encoding='name_of_encoding')`` -``'es#'`` ``str(encoding='name_of_encoding', zeroes=True)`` -``'et'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str})`` -``'et#'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str}, zeroes=True)`` -``'f'`` ``float`` -``'h'`` ``short`` -``'H'`` ``unsigned_short(bitwise=True)`` -``'i'`` ``int`` -``'I'`` ``unsigned_int(bitwise=True)`` -``'k'`` ``unsigned_long(bitwise=True)`` -``'K'`` ``unsigned_long_long(bitwise=True)`` -``'l'`` ``long`` -``'L'`` ``long long`` -``'n'`` ``Py_ssize_t`` -``'O'`` ``object`` -``'O!'`` ``object(subclass_of='&PySomething_Type')`` -``'O&'`` ``object(converter='name_of_c_function')`` -``'p'`` ``bool`` -``'S'`` ``PyBytesObject`` -``'s'`` ``str`` -``'s#'`` ``str(zeroes=True)`` -``'s*'`` ``Py_buffer(accept={buffer, str})`` -``'U'`` ``unicode`` -``'u'`` ``wchar_t`` -``'u#'`` ``wchar_t(zeroes=True)`` -``'w*'`` ``Py_buffer(accept={rwbuffer})`` -``'Y'`` ``PyByteArrayObject`` -``'y'`` ``str(accept={bytes})`` -``'y#'`` ``str(accept={robuffer}, zeroes=True)`` -``'y*'`` ``Py_buffer`` -``'Z'`` ``wchar_t(accept={str, NoneType})`` -``'Z#'`` ``wchar_t(accept={str, NoneType}, zeroes=True)`` -``'z'`` ``str(accept={str, NoneType})`` -``'z#'`` ``str(accept={str, NoneType}, zeroes=True)`` -``'z*'`` ``Py_buffer(accept={buffer, str, NoneType})`` -========= ================================================================================= - -As an example, here's our sample ``pickle.Pickler.dump`` using the proper -converter:: - - /*[clinic input] - pickle.Pickler.dump - - obj: object - The object to be pickled. - / - - Write a pickled representation of obj to the open file. - [clinic start generated code]*/ - -One advantage of real converters is that they're more flexible than legacy -converters. For example, the ``unsigned_int`` converter (and all the -``unsigned_`` converters) can be specified without ``bitwise=True``. Their -default behavior performs range checking on the value, and they won't accept -negative numbers. You just can't do that with a legacy converter! - -Argument Clinic will show you all the converters it has -available. For each converter it'll show you all the parameters -it accepts, along with the default value for each parameter. -Just run ``Tools/clinic/clinic.py --converters`` to see the full list. - - -How to use the ``Py_buffer`` converter --------------------------------------- - -When using the ``Py_buffer`` converter -(or the ``'s*'``, ``'w*'``, ``'*y'``, or ``'z*'`` legacy converters), -you *must* not call :c:func:`PyBuffer_Release` on the provided buffer. -Argument Clinic generates code that does it for you (in the parsing function). - - -.. _clinic-howto-advanced-converters: - -How to use advanced converters ------------------------------- - -Remember those format units you skipped for your first -time because they were advanced? Here's how to handle those too. - -The trick is, all those format units take arguments—either -conversion functions, or types, or strings specifying an encoding. -(But "legacy converters" don't support arguments. That's why we -skipped them for your first function.) The argument you specified -to the format unit is now an argument to the converter; this -argument is either *converter* (for ``O&``), *subclass_of* (for ``O!``), -or *encoding* (for all the format units that start with ``e``). - -When using *subclass_of*, you may also want to use the other -custom argument for ``object()``: *type*, which lets you set the type -actually used for the parameter. For example, if you want to ensure -that the object is a subclass of :c:var:`PyUnicode_Type`, you probably want -to use the converter ``object(type='PyUnicodeObject *', subclass_of='&PyUnicode_Type')``. - -One possible problem with using Argument Clinic: it takes away some possible -flexibility for the format units starting with ``e``. When writing a -:c:func:`!PyArg_Parse*` call by hand, you could theoretically decide at runtime what -encoding string to pass to that call. But now this string must -be hard-coded at Argument-Clinic-preprocessing-time. This limitation is deliberate; -it made supporting this format unit much easier, and may allow for future optimizations. -This restriction doesn't seem unreasonable; CPython itself always passes in static -hard-coded encoding strings for parameters whose format units start with ``e``. - - -.. _clinic-howto-default-values: -.. _default_values: - -How to assign default values to parameter ------------------------------------------ - -Default values for parameters can be any of a number of values. -At their simplest, they can be string, int, or float literals: - -.. code-block:: none - - foo: str = "abc" - bar: int = 123 - bat: float = 45.6 - -They can also use any of Python's built-in constants: - -.. code-block:: none - - yep: bool = True - nope: bool = False - nada: object = None - -There's also special support for a default value of ``NULL``, and -for simple expressions, documented in the following sections. - - -The ``NULL`` default value -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -For string and object parameters, you can set them to ``None`` to indicate -that there's no default. However, that means the C variable will be -initialized to ``Py_None``. For convenience's sakes, there's a special -value called ``NULL`` for just this reason: from Python's perspective it -behaves like a default value of ``None``, but the C variable is initialized -with ``NULL``. - - -Symbolic default values -^^^^^^^^^^^^^^^^^^^^^^^ - -The default value you provide for a parameter can't be any arbitrary -expression. Currently the following are explicitly supported: - -* Numeric constants (integer and float) -* String constants -* ``True``, ``False``, and ``None`` -* Simple symbolic constants like :py:data:`sys.maxsize`, which must - start with the name of the module - -(In the future, this may need to get even more elaborate, -to allow full expressions like ``CONSTANT - 1``.) - - -Expressions as default values -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The default value for a parameter can be more than just a literal value. -It can be an entire expression, using math operators and looking up attributes -on objects. However, this support isn't exactly simple, because of some -non-obvious semantics. - -Consider the following example: - -.. code-block:: none - - foo: Py_ssize_t = sys.maxsize - 1 - -:py:data:`sys.maxsize` can have different values on different platforms. Therefore -Argument Clinic can't simply evaluate that expression locally and hard-code it -in C. So it stores the default in such a way that it will get evaluated at -runtime, when the user asks for the function's signature. - -What namespace is available when the expression is evaluated? It's evaluated -in the context of the module the builtin came from. So, if your module has an -attribute called :py:attr:`!max_widgets`, you may simply use it: - -.. code-block:: none - - foo: Py_ssize_t = max_widgets - -If the symbol isn't found in the current module, it fails over to looking in -:py:data:`sys.modules`. That's how it can find :py:data:`sys.maxsize` for example. -(Since you don't know in advance what modules the user will load into their interpreter, -it's best to restrict yourself to modules that are preloaded by Python itself.) - -Evaluating default values only at runtime means Argument Clinic can't compute -the correct equivalent C default value. So you need to tell it explicitly. -When you use an expression, you must also specify the equivalent expression -in C, using the *c_default* parameter to the converter: - -.. code-block:: none - - foo: Py_ssize_t(c_default="PY_SSIZE_T_MAX - 1") = sys.maxsize - 1 - -Another complication: Argument Clinic can't know in advance whether or not the -expression you supply is valid. It parses it to make sure it looks legal, but -it can't *actually* know. You must be very careful when using expressions to -specify values that are guaranteed to be valid at runtime! - -Finally, because expressions must be representable as static C values, there -are many restrictions on legal expressions. Here's a list of Python features -you're not permitted to use: - -* Function calls. -* Inline if statements (``3 if foo else 5``). -* Automatic sequence unpacking (``*[1, 2, 3]``). -* List/set/dict comprehensions and generator expressions. -* Tuple/list/set/dict literals. - - -.. _clinic-howto-return-converters: - -How to use return converters ----------------------------- - -By default, the impl function Argument Clinic generates for you returns -:c:type:`PyObject * `. -But your C function often computes some C type, -then converts it into the :c:type:`!PyObject *` -at the last moment. Argument Clinic handles converting your inputs from Python types -into native C types—why not have it convert your return value from a native C type -into a Python type too? - -That's what a "return converter" does. It changes your impl function to return -some C type, then adds code to the generated (non-impl) function to handle converting -that value into the appropriate :c:type:`!PyObject *`. - -The syntax for return converters is similar to that of parameter converters. -You specify the return converter like it was a return annotation on the -function itself, using ``->`` notation. - -For example: - -.. code-block:: c - - /*[clinic input] - add -> int - - a: int - b: int - / - - [clinic start generated code]*/ - -Return converters behave much the same as parameter converters; -they take arguments, the arguments are all keyword-only, and if you're not changing -any of the default arguments you can omit the parentheses. - -(If you use both ``"as"`` *and* a return converter for your function, -the ``"as"`` should come before the return converter.) - -There's one additional complication when using return converters: how do you -indicate an error has occurred? Normally, a function returns a valid (non-``NULL``) -pointer for success, and ``NULL`` for failure. But if you use an integer return converter, -all integers are valid. How can Argument Clinic detect an error? Its solution: each return -converter implicitly looks for a special value that indicates an error. If you return -that value, and an error has been set (c:func:`PyErr_Occurred` returns a true -value), then the generated code will propagate the error. Otherwise it will -encode the value you return like normal. - -Currently Argument Clinic supports only a few return converters: - -.. code-block:: none - - bool - double - float - int - long - Py_ssize_t - size_t - unsigned int - unsigned long - -None of these take parameters. -For all of these, return ``-1`` to indicate error. - -To see all the return converters Argument Clinic supports, along with -their parameters (if any), -just run ``Tools/clinic/clinic.py --converters`` for the full list. - - -How to clone existing functions -------------------------------- - -If you have a number of functions that look similar, you may be able to -use Clinic's "clone" feature. When you clone an existing function, -you reuse: - -* its parameters, including - - * their names, - - * their converters, with all parameters, - - * their default values, - - * their per-parameter docstrings, - - * their *kind* (whether they're positional only, - positional or keyword, or keyword only), and - -* its return converter. - -The only thing not copied from the original function is its docstring; -the syntax allows you to specify a new docstring. - -Here's the syntax for cloning a function:: - - /*[clinic input] - module.class.new_function [as c_basename] = module.class.existing_function - - Docstring for new_function goes here. - [clinic start generated code]*/ - -(The functions can be in different modules or classes. I wrote -``module.class`` in the sample just to illustrate that you must -use the full path to *both* functions.) - -Sorry, there's no syntax for partially cloning a function, or cloning a function -then modifying it. Cloning is an all-or nothing proposition. - -Also, the function you are cloning from must have been previously defined -in the current file. - - -How to call Python code ------------------------ - -The rest of the advanced topics require you to write Python code -which lives inside your C file and modifies Argument Clinic's -runtime state. This is simple: you simply define a Python block. - -A Python block uses different delimiter lines than an Argument -Clinic function block. It looks like this:: - - /*[python input] - # python code goes here - [python start generated code]*/ - -All the code inside the Python block is executed at the -time it's parsed. All text written to stdout inside the block -is redirected into the "output" after the block. - -As an example, here's a Python block that adds a static integer -variable to the C code:: - - /*[python input] - print('static int __ignored_unused_variable__ = 0;') - [python start generated code]*/ - static int __ignored_unused_variable__ = 0; - /*[python checksum:...]*/ - - -.. _clinic-howto-self-converter: - -How to use the "self converter" -------------------------------- - -Argument Clinic automatically adds a "self" parameter for you -using a default converter. It automatically sets the ``type`` -of this parameter to the "pointer to an instance" you specified -when you declared the type. However, you can override -Argument Clinic's converter and specify one yourself. -Just add your own *self* parameter as the first parameter in a -block, and ensure that its converter is an instance of -:class:`!self_converter` or a subclass thereof. - -What's the point? This lets you override the type of ``self``, -or give it a different default name. - -How do you specify the custom type you want to cast ``self`` to? -If you only have one or two functions with the same type for ``self``, -you can directly use Argument Clinic's existing ``self`` converter, -passing in the type you want to use as the *type* parameter:: - - /*[clinic input] - - _pickle.Pickler.dump - - self: self(type="PicklerObject *") - obj: object - / - - Write a pickled representation of the given object to the open file. - [clinic start generated code]*/ - -On the other hand, if you have a lot of functions that will use the same -type for ``self``, it's best to create your own converter, subclassing -:class:`!self_converter` but overwriting the :py:attr:`!type` member:: - - /*[python input] - class PicklerObject_converter(self_converter): - type = "PicklerObject *" - [python start generated code]*/ - - /*[clinic input] - - _pickle.Pickler.dump - - self: PicklerObject - obj: object - / - - Write a pickled representation of the given object to the open file. - [clinic start generated code]*/ - - -How to use the "defining class" converter ------------------------------------------ - -Argument Clinic facilitates gaining access to the defining class of a method. -This is useful for :ref:`heap type ` methods that need to fetch -module level state. Use :c:func:`PyType_FromModuleAndSpec` to associate a new -heap type with a module. You can now use :c:func:`PyType_GetModuleState` on -the defining class to fetch the module state, for example from a module method. - -Example from :source:`Modules/zlibmodule.c`. -First, ``defining_class`` is added to the clinic input:: - - /*[clinic input] - zlib.Compress.compress - - cls: defining_class - data: Py_buffer - Binary data to be compressed. - / - - -After running the Argument Clinic tool, the following function signature is -generated:: - - /*[clinic start generated code]*/ - static PyObject * - zlib_Compress_compress_impl(compobject *self, PyTypeObject *cls, - Py_buffer *data) - /*[clinic end generated code: output=6731b3f0ff357ca6 input=04d00f65ab01d260]*/ - - -The following code can now use ``PyType_GetModuleState(cls)`` to fetch the -module state:: - - zlibstate *state = PyType_GetModuleState(cls); - - -Each method may only have one argument using this converter, and it must appear -after ``self``, or, if ``self`` is not used, as the first argument. The argument -will be of type ``PyTypeObject *``. The argument will not appear in the -:py:attr:`!__text_signature__`. - -The ``defining_class`` converter is not compatible with :py:meth:`!__init__` -and :py:meth:`!__new__` methods, which cannot use the :c:macro:`METH_METHOD` -convention. - -It is not possible to use ``defining_class`` with slot methods. In order to -fetch the module state from such methods, use :c:func:`PyType_GetModuleByDef` -to look up the module and then :c:func:`PyModule_GetState` to fetch the module -state. Example from the ``setattro`` slot method in -:source:`Modules/_threadmodule.c`:: - - static int - local_setattro(localobject *self, PyObject *name, PyObject *v) - { - PyObject *module = PyType_GetModuleByDef(Py_TYPE(self), &thread_module); - thread_module_state *state = get_thread_state(module); - ... - } - - -See also :pep:`573`. - - -.. _clinic-howto-custom-converter: - -How to write a custom converter -------------------------------- - -A converter is a Python class that inherits from :py:class:`CConverter`. -The main purpose of a custom converter, is for parameters parsed with -the ``O&`` format unit --- parsing such a parameter means calling -a :c:func:`PyArg_ParseTuple` "converter function". - -Your converter class should be named :samp:`{ConverterName}_converter`. -By following this convention, your converter class will be automatically -registered with Argument Clinic, with its *converter name* being the name of -your converter class with the ``_converter`` suffix stripped off. - -Instead of subclassing :py:meth:`!CConverter.__init__`, -write a :py:meth:`!converter_init` method. -:py:meth:`!converter_init` always accepts a *self* parameter. -After *self*, all additional parameters **must** be keyword-only. -Any arguments passed to the converter in Argument Clinic -will be passed along to your :py:meth:`!converter_init` method. -See :py:class:`CConverter` for a list of members you may wish to specify in -your subclass. - -Here's the simplest example of a custom converter, from :source:`Modules/zlibmodule.c`:: - - /*[python input] - - class ssize_t_converter(CConverter): - type = 'Py_ssize_t' - converter = 'ssize_t_converter' - - [python start generated code]*/ - /*[python end generated code: output=da39a3ee5e6b4b0d input=35521e4e733823c7]*/ - -This block adds a converter named ``ssize_t`` to Argument Clinic. -Parameters declared as ``ssize_t`` will be declared with type :c:type:`Py_ssize_t`, -and will be parsed by the ``'O&'`` format unit, -which will call the :c:func:`!ssize_t_converter` converter C function. -``ssize_t`` variables automatically support default values. - -More sophisticated custom converters can insert custom C code to -handle initialization and cleanup. -You can see more examples of custom converters in the CPython -source tree; grep the C files for the string ``CConverter``. - - -How to write a custom return converter --------------------------------------- - -Writing a custom return converter is much like writing -a custom converter. Except it's somewhat simpler, because return -converters are themselves much simpler. - -Return converters must subclass :py:class:`!CReturnConverter`. -There are no examples yet of custom return converters, -because they are not widely used yet. If you wish to -write your own return converter, please read :source:`Tools/clinic/clinic.py`, -specifically the implementation of :py:class:`!CReturnConverter` and -all its subclasses. - - -How to convert ``METH_O`` and ``METH_NOARGS`` functions -------------------------------------------------------- - -To convert a function using :c:macro:`METH_O`, make sure the function's -single argument is using the ``object`` converter, and mark the -arguments as positional-only:: - - /*[clinic input] - meth_o_sample - - argument: object - / - [clinic start generated code]*/ - - -To convert a function using :c:macro:`METH_NOARGS`, just don't specify -any arguments. - -You can still use a self converter, a return converter, and specify -a *type* argument to the object converter for :c:macro:`METH_O`. - - -How to convert ``tp_new`` and ``tp_init`` functions ---------------------------------------------------- - -You can convert :c:member:`~PyTypeObject.tp_new` and -:c:member:`~PyTypeObject.tp_init` functions. -Just name them ``__new__`` or ``__init__`` as appropriate. Notes: - -* The function name generated for ``__new__`` doesn't end in ``__new__`` - like it would by default. It's just the name of the class, converted - into a valid C identifier. - -* No :c:type:`PyMethodDef` ``#define`` is generated for these functions. - -* ``__init__`` functions return ``int``, not ``PyObject *``. - -* Use the docstring as the class docstring. - -* Although ``__new__`` and ``__init__`` functions must always - accept both the ``args`` and ``kwargs`` objects, when converting - you may specify any signature for these functions that you like. - (If your function doesn't support keywords, the parsing function - generated will throw an exception if it receives any.) - - -How to change and redirect Clinic's output ------------------------------------------- - -It can be inconvenient to have Clinic's output interspersed with -your conventional hand-edited C code. Luckily, Clinic is configurable: -you can buffer up its output for printing later (or earlier!), or write -its output to a separate file. You can also add a prefix or suffix to -every line of Clinic's generated output. - -While changing Clinic's output in this manner can be a boon to readability, -it may result in Clinic code using types before they are defined, or -your code attempting to use Clinic-generated code before it is defined. -These problems can be easily solved by rearranging the declarations in your file, -or moving where Clinic's generated code goes. (This is why the default behavior -of Clinic is to output everything into the current block; while many people -consider this hampers readability, it will never require rearranging your -code to fix definition-before-use problems.) - -Let's start with defining some terminology: - -*field* - A field, in this context, is a subsection of Clinic's output. - For example, the ``#define`` for the :c:type:`PyMethodDef` structure - is a field, called ``methoddef_define``. Clinic has seven - different fields it can output per function definition: - - .. code-block:: none - - docstring_prototype - docstring_definition - methoddef_define - impl_prototype - parser_prototype - parser_definition - impl_definition - - All the names are of the form ``"_"``, - where ``""`` is the semantic object represented (the parsing function, - the impl function, the docstring, or the methoddef structure) and ``""`` - represents what kind of statement the field is. Field names that end in - ``"_prototype"`` - represent forward declarations of that thing, without the actual body/data - of the thing; field names that end in ``"_definition"`` represent the actual - definition of the thing, with the body/data of the thing. (``"methoddef"`` - is special, it's the only one that ends with ``"_define"``, representing that - it's a preprocessor #define.) - -*destination* - A destination is a place Clinic can write output to. There are - five built-in destinations: - - ``block`` - The default destination: printed in the output section of - the current Clinic block. - - ``buffer`` - A text buffer where you can save text for later. Text sent - here is appended to the end of any existing text. It's an - error to have any text left in the buffer when Clinic finishes - processing a file. - - ``file`` - A separate "clinic file" that will be created automatically by Clinic. - The filename chosen for the file is ``{basename}.clinic{extension}``, - where ``basename`` and ``extension`` were assigned the output - from ``os.path.splitext()`` run on the current file. (Example: - the ``file`` destination for :file:`_pickle.c` would be written to - :file:`_pickle.clinic.c`.) - - **Important: When using a** ``file`` **destination, you** - *must check in* **the generated file!** - - ``two-pass`` - A buffer like ``buffer``. However, a two-pass buffer can only - be dumped once, and it prints out all text sent to it during - all processing, even from Clinic blocks *after* the dumping point. - - ``suppress`` - The text is suppressed—thrown away. - - -Clinic defines five new directives that let you reconfigure its output. - -The first new directive is ``dump``: - -.. code-block:: none - - dump - -This dumps the current contents of the named destination into the output of -the current block, and empties it. This only works with ``buffer`` and -``two-pass`` destinations. - -The second new directive is ``output``. The most basic form of ``output`` -is like this: - -.. code-block:: none - - output - -This tells Clinic to output *field* to *destination*. ``output`` also -supports a special meta-destination, called ``everything``, which tells -Clinic to output *all* fields to that *destination*. - -``output`` has a number of other functions: - -.. code-block:: none - - output push - output pop - output preset - - -``output push`` and ``output pop`` allow you to push and pop -configurations on an internal configuration stack, so that you -can temporarily modify the output configuration, then easily restore -the previous configuration. Simply push before your change to save -the current configuration, then pop when you wish to restore the -previous configuration. - -``output preset`` sets Clinic's output to one of several built-in -preset configurations, as follows: - - ``block`` - Clinic's original starting configuration. Writes everything - immediately after the input block. - - Suppress the ``parser_prototype`` - and ``docstring_prototype``, write everything else to ``block``. - - ``file`` - Designed to write everything to the "clinic file" that it can. - You then ``#include`` this file near the top of your file. - You may need to rearrange your file to make this work, though - usually this just means creating forward declarations for various - ``typedef`` and ``PyTypeObject`` definitions. - - Suppress the ``parser_prototype`` - and ``docstring_prototype``, write the ``impl_definition`` to - ``block``, and write everything else to ``file``. - - The default filename is ``"{dirname}/clinic/{basename}.h"``. - - ``buffer`` - Save up most of the output from Clinic, to be written into - your file near the end. For Python files implementing modules - or builtin types, it's recommended that you dump the buffer - just above the static structures for your module or - builtin type; these are normally very near the end. Using - ``buffer`` may require even more editing than ``file``, if - your file has static ``PyMethodDef`` arrays defined in the - middle of the file. - - Suppress the ``parser_prototype``, ``impl_prototype``, - and ``docstring_prototype``, write the ``impl_definition`` to - ``block``, and write everything else to ``file``. - - ``two-pass`` - Similar to the ``buffer`` preset, but writes forward declarations to - the ``two-pass`` buffer, and definitions to the ``buffer``. - This is similar to the ``buffer`` preset, but may require - less editing than ``buffer``. Dump the ``two-pass`` buffer - near the top of your file, and dump the ``buffer`` near - the end just like you would when using the ``buffer`` preset. - - Suppresses the ``impl_prototype``, write the ``impl_definition`` - to ``block``, write ``docstring_prototype``, ``methoddef_define``, - and ``parser_prototype`` to ``two-pass``, write everything else - to ``buffer``. - - ``partial-buffer`` - Similar to the ``buffer`` preset, but writes more things to ``block``, - only writing the really big chunks of generated code to ``buffer``. - This avoids the definition-before-use problem of ``buffer`` completely, - at the small cost of having slightly more stuff in the block's output. - Dump the ``buffer`` near the end, just like you would when using - the ``buffer`` preset. - - Suppresses the ``impl_prototype``, write the ``docstring_definition`` - and ``parser_definition`` to ``buffer``, write everything else to ``block``. - -The third new directive is ``destination``: - -.. code-block:: none - - destination [...] - -This performs an operation on the destination named ``name``. - -There are two defined subcommands: ``new`` and ``clear``. - -The ``new`` subcommand works like this: - -.. code-block:: none - - destination new - -This creates a new destination with name ```` and type ````. - -There are five destination types: - - ``suppress`` - Throws the text away. - - ``block`` - Writes the text to the current block. This is what Clinic - originally did. - - ``buffer`` - A simple text buffer, like the "buffer" builtin destination above. - - ``file`` - A text file. The file destination takes an extra argument, - a template to use for building the filename, like so: - - destination new - - The template can use three strings internally that will be replaced - by bits of the filename: - - {path} - The full path to the file, including directory and full filename. - {dirname} - The name of the directory the file is in. - {basename} - Just the name of the file, not including the directory. - {basename_root} - Basename with the extension clipped off - (everything up to but not including the last '.'). - {basename_extension} - The last '.' and everything after it. If the basename - does not contain a period, this will be the empty string. - - If there are no periods in the filename, {basename} and {filename} - are the same, and {extension} is empty. "{basename}{extension}" - is always exactly the same as "{filename}"." - - ``two-pass`` - A two-pass buffer, like the "two-pass" builtin destination above. - - -The ``clear`` subcommand works like this: - -.. code-block:: none - - destination clear - -It removes all the accumulated text up to this point in the destination. -(I don't know what you'd need this for, but I thought maybe it'd be -useful while someone's experimenting.) - -The fourth new directive is ``set``: - -.. code-block:: none - - set line_prefix "string" - set line_suffix "string" - -``set`` lets you set two internal variables in Clinic. -``line_prefix`` is a string that will be prepended to every line of Clinic's output; -``line_suffix`` is a string that will be appended to every line of Clinic's output. - -Both of these support two format strings: - - ``{block comment start}`` - Turns into the string ``/*``, the start-comment text sequence for C files. - - ``{block comment end}`` - Turns into the string ``*/``, the end-comment text sequence for C files. - -The final new directive is one you shouldn't need to use directly, -called ``preserve``: - -.. code-block:: none - - preserve - -This tells Clinic that the current contents of the output should be kept, unmodified. -This is used internally by Clinic when dumping output into ``file`` files; wrapping -it in a Clinic block lets Clinic use its existing checksum functionality to ensure -the file was not modified by hand before it gets overwritten. - - -How to use the ``#ifdef`` trick -------------------------------- - -If you're converting a function that isn't available on all platforms, -there's a trick you can use to make life a little easier. The existing -code probably looks like this:: - - #ifdef HAVE_FUNCTIONNAME - static module_functionname(...) - { - ... - } - #endif /* HAVE_FUNCTIONNAME */ - -And then in the ``PyMethodDef`` structure at the bottom the existing code -will have: - -.. code-block:: none - - #ifdef HAVE_FUNCTIONNAME - {'functionname', ... }, - #endif /* HAVE_FUNCTIONNAME */ - -In this scenario, you should enclose the body of your impl function inside the ``#ifdef``, -like so:: - - #ifdef HAVE_FUNCTIONNAME - /*[clinic input] - module.functionname - ... - [clinic start generated code]*/ - static module_functionname(...) - { - ... - } - #endif /* HAVE_FUNCTIONNAME */ - -Then, remove those three lines from the :c:type:`PyMethodDef` structure, -replacing them with the macro Argument Clinic generated: - -.. code-block:: none - - MODULE_FUNCTIONNAME_METHODDEF - -(You can find the real name for this macro inside the generated code. -Or you can calculate it yourself: it's the name of your function as defined -on the first line of your block, but with periods changed to underscores, -uppercased, and ``"_METHODDEF"`` added to the end.) - -Perhaps you're wondering: what if ``HAVE_FUNCTIONNAME`` isn't defined? -The ``MODULE_FUNCTIONNAME_METHODDEF`` macro won't be defined either! - -Here's where Argument Clinic gets very clever. It actually detects that the -Argument Clinic block might be deactivated by the ``#ifdef``. When that -happens, it generates a little extra code that looks like this:: - - #ifndef MODULE_FUNCTIONNAME_METHODDEF - #define MODULE_FUNCTIONNAME_METHODDEF - #endif /* !defined(MODULE_FUNCTIONNAME_METHODDEF) */ - -That means the macro always works. If the function is defined, this turns -into the correct structure, including the trailing comma. If the function is -undefined, this turns into nothing. - -However, this causes one ticklish problem: where should Argument Clinic put this -extra code when using the "block" output preset? It can't go in the output block, -because that could be deactivated by the ``#ifdef``. (That's the whole point!) - -In this situation, Argument Clinic writes the extra code to the "buffer" destination. -This may mean that you get a complaint from Argument Clinic: - -.. code-block:: none - - Warning in file "Modules/posixmodule.c" on line 12357: - Destination buffer 'buffer' not empty at end of file, emptying. - -When this happens, just open your file, find the ``dump buffer`` block that -Argument Clinic added to your file (it'll be at the very bottom), then -move it above the :c:type:`PyMethodDef` structure where that macro is used. - - -How to use Argument Clinic in Python files ------------------------------------------- - -It's actually possible to use Argument Clinic to preprocess Python files. -There's no point to using Argument Clinic blocks, of course, as the output -wouldn't make any sense to the Python interpreter. But using Argument Clinic -to run Python blocks lets you use Python as a Python preprocessor! - -Since Python comments are different from C comments, Argument Clinic -blocks embedded in Python files look slightly different. They look like this: - -.. code-block:: python3 - - #/*[python input] - #print("def foo(): pass") - #[python start generated code]*/ - def foo(): pass - #/*[python checksum:...]*/ - - -.. _clinic-howto-override-signature: - -How to override the generated signature ---------------------------------------- - -You can use the ``@text_signature`` directive to override the default generated -signature in the docstring. -This can be useful for complex signatures that Argument Clinic cannot handle. -The ``@text_signature`` directive takes one argument: -the custom signature as a string. -The provided signature is copied verbatim to the generated docstring. - -Example from :source:`Objects/codeobject.c`:: - - /*[clinic input] - @text_signature "($self, /, **changes)" - code.replace - * - co_argcount: int(c_default="self->co_argcount") = unchanged - co_posonlyargcount: int(c_default="self->co_posonlyargcount") = unchanged - # etc ... - - Return a copy of the code object with new values for the specified fields. - [clinic start generated output]*/ - -The generated docstring ends up looking like this: - -.. code-block:: none - - replace($self, /, **changes) - -- - - Return a copy of the code object with new values for the specified fields. - - -.. _clinic-howto-deprecate-positional: - -How to deprecate passing parameters positionally ------------------------------------------------- - -Argument Clinic provides syntax that makes it possible to generate code that -deprecates passing :term:`arguments ` positionally. -For example, say we've got a module-level function :py:func:`!foo.myfunc` -that has three :term:`parameters `: -positional-or-keyword parameters *a* and *b*, and a keyword-only parameter *c*:: - - /*[clinic input] - module foo - myfunc - a: int - b: int - * - c: int - [clinic start generated output]*/ - -We now want to make the *b* parameter keyword-only; -however, we'll have to wait two releases before making this change, -as mandated by Python's backwards-compatibility policy (see :pep:`387`). -For this example, imagine we're in the development phase for Python 3.12: -that means we'll be allowed to introduce deprecation warnings in Python 3.12 -whenever the *b* parameter is passed positionally, -and we'll be allowed to make it keyword-only in Python 3.14 at the earliest. - -We can use Argument Clinic to emit the desired deprecation warnings -using the ``* [from ...]`` syntax, -by adding the line ``* [from 3.14]`` right above the *b* parameter:: - - /*[clinic input] - module foo - myfunc - a: int - * [from 3.14] - b: int - * - c: int - [clinic start generated output]*/ - -Next, regenerate Argument Clinic code (``make clinic``), -and add unit tests for the new behaviour. - -The generated code will now emit a :exc:`DeprecationWarning` -when an :term:`argument` for the :term:`parameter` *b* is passed positionally. -C preprocessor directives are also generated for emitting -compiler warnings if the ``* [from ...]`` line has not been removed -from the Argument Clinic input when the deprecation period is over, -which means when the alpha phase of the specified Python version kicks in. - -Let's return to our example and skip ahead two years: -Python 3.14 development has now entered the alpha phase, -but we forgot all about updating the Argument Clinic code -for :py:func:`!myfunc`! -Luckily for us, compiler warnings are now generated: - -.. code-block:: none - - In file included from Modules/foomodule.c:139: - Modules/clinic/foomodule.c.h:139:8: warning: In 'foomodule.c', update parameter(s) 'a' and 'b' in the clinic input of 'mymod.myfunc' to be keyword-only. [-W#warnings] - # warning "In 'foomodule.c', update parameter(s) 'a' and 'b' in the clinic input of 'mymod.myfunc' to be keyword-only. [-W#warnings]" - ^ - -We now close the deprecation phase by making *b* keyword-only; -replace the ``* [from ...]`` line above *b* -with the ``*`` from the line above *c*:: - - /*[clinic input] - module foo - myfunc - a: int - * - b: int - c: int - [clinic start generated output]*/ - -Finally, run ``make clinic`` to regenerate the Argument Clinic code, -and update your unit tests to reflect the new behaviour. - -.. note:: - - If you forget to update your input block during the alpha and beta phases, - the compiler warning will turn into a compiler error when the - release candidate phase begins. From 5d3ca1a005491c6f42f3c303cd04b5d6c94415a2 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Wed, 9 Aug 2023 15:52:52 +0200 Subject: [PATCH 70/71] Add clinic.rst to Development Tools and conf.py --- .github/CODEOWNERS | 1 + conf.py | 1 + development-tools/index.rst | 1 + 3 files changed, 3 insertions(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 668bb5ace..50163563b 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -6,3 +6,4 @@ garbage_collector.rst @pablogsal +clinic.rst @erlend-aasland diff --git a/conf.py b/conf.py index a9b0ec1a7..4a84383eb 100644 --- a/conf.py +++ b/conf.py @@ -110,6 +110,7 @@ def _asset_hash(path: os.PathLike[str]) -> str: rediraffe_redirects = { # Development Tools "clang.rst": "development-tools/clang.rst", + "clinic.rst": "development-tools/clinic.rst", "coverity.rst": "development-tools/coverity.rst", "gdb.rst": "development-tools/gdb.rst", # Advanced Tools was renamed Development Tools in gh-1149 diff --git a/development-tools/index.rst b/development-tools/index.rst index 6a3e7466e..d555fdc1b 100644 --- a/development-tools/index.rst +++ b/development-tools/index.rst @@ -5,6 +5,7 @@ Development Tools .. toctree:: :maxdepth: 5 + clinic gdb clang coverity From 97fe521410b82d7183e5a910d03f71ffdadb8514 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Wed, 9 Aug 2023 16:05:32 +0200 Subject: [PATCH 71/71] Adapt clinic.rst - use cpy-file iso. source - fix Sphinx complaint about 'range' - mark up abstract similar to other devguide pages --- development-tools/clinic.rst | 48 ++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 27 deletions(-) diff --git a/development-tools/clinic.rst b/development-tools/clinic.rst index 463a938fa..c03f4e9b3 100644 --- a/development-tools/clinic.rst +++ b/development-tools/clinic.rst @@ -1,32 +1,26 @@ .. highlight:: c -.. _howto-clinic: +.. _clinic: -********************** -Argument Clinic How-To -********************** +*************** +Argument Clinic +*************** :author: Larry Hastings -**Source code:** :source:`Tools/clinic/clinic.py`. +**Source code:** :cpy-file:`Tools/clinic/clinic.py`. -.. topic:: Abstract +Argument Clinic is a preprocessor for CPython C files. +It was introduced in Python 3.4 with :pep:`436`, +in order to provide introspection signatures, +and to generate performant and tailor-made boilerplate code +for argument parsing in CPython builtins, module level functions, and class methods. +This document is divided in four major sections: - Argument Clinic is a preprocessor for CPython C files. - It was introduced in Python 3.4 with :pep:`436`, - in order to provide introspection signatures, - and to generate performant and tailor-made boilerplate code - for argument parsing in CPython builtins, - module level functions, and class methods. - This document is divided in four major sections: - - * :ref:`clinic-background` talks about the basic concepts and goals of - Argument Clinic. - * :ref:`clinic-reference` describes the command-line interface and Argument - Clinic terminology. - * :ref:`clinic-tutorial` guides you through all the steps required to - adapt an existing C function to Argument Clinic. - * :ref:`clinic-howtos` details how to handle specific tasks. +* :ref:`clinic-background` talks about the basic concepts and goals of Argument Clinic. +* :ref:`clinic-reference` describes the command-line interface and Argument Clinic terminology. +* :ref:`clinic-tutorial` guides you through all the steps required to adapt an existing C function to Argument Clinic. +* :ref:`clinic-howtos` details how to handle specific tasks. .. note:: @@ -749,7 +743,7 @@ While functions using this approach can often be converted to use :c:func:`!PyArg_ParseTupleAndKeywords`, optional arguments, and default values, it's not always possible. Some of these legacy functions have behaviors :c:func:`!PyArg_ParseTupleAndKeywords` doesn't directly support. -The most obvious example is the builtin function :py:func:`range`, which has +The most obvious example is the builtin function :py:func:`!range`, which has an optional argument on the *left* side of its required argument! Another example is :py:meth:`curses.window.addch`, which has a group of two arguments that must always be specified together. (The arguments are @@ -1358,7 +1352,7 @@ module level state. Use :c:func:`PyType_FromModuleAndSpec` to associate a new heap type with a module. You can now use :c:func:`PyType_GetModuleState` on the defining class to fetch the module state, for example from a module method. -Example from :source:`Modules/zlibmodule.c`. +Example from :cpy-file:`Modules/zlibmodule.c`. First, ``defining_class`` is added to the clinic input:: /*[clinic input] @@ -1399,7 +1393,7 @@ It is not possible to use ``defining_class`` with slot methods. In order to fetch the module state from such methods, use :c:func:`PyType_GetModuleByDef` to look up the module and then :c:func:`PyModule_GetState` to fetch the module state. Example from the ``setattro`` slot method in -:source:`Modules/_threadmodule.c`:: +:cpy-file:`Modules/_threadmodule.c`:: static int local_setattro(localobject *self, PyObject *name, PyObject *v) @@ -1437,7 +1431,7 @@ will be passed along to your :py:meth:`!converter_init` method. See :py:class:`CConverter` for a list of members you may wish to specify in your subclass. -Here's the simplest example of a custom converter, from :source:`Modules/zlibmodule.c`:: +Here's the simplest example of a custom converter, from :cpy-file:`Modules/zlibmodule.c`:: /*[python input] @@ -1470,7 +1464,7 @@ converters are themselves much simpler. Return converters must subclass :py:class:`!CReturnConverter`. There are no examples yet of custom return converters, because they are not widely used yet. If you wish to -write your own return converter, please read :source:`Tools/clinic/clinic.py`, +write your own return converter, please read :cpy-file:`Tools/clinic/clinic.py`, specifically the implementation of :py:class:`!CReturnConverter` and all its subclasses. @@ -1917,7 +1911,7 @@ The ``@text_signature`` directive takes one argument: the custom signature as a string. The provided signature is copied verbatim to the generated docstring. -Example from :source:`Objects/codeobject.c`:: +Example from :cpy-file:`Objects/codeobject.c`:: /*[clinic input] @text_signature "($self, /, **changes)"