diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 1db5c505149397..dc91c0dbcb027b 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -31,7 +31,6 @@ jobs: runs-on: ubuntu-latest outputs: run_tests: ${{ steps.check.outputs.run_tests }} - run_ssl_tests: ${{ steps.check.outputs.run_ssl_tests }} steps: - uses: actions/checkout@v3 - name: Check for source changes @@ -39,7 +38,6 @@ jobs: run: | if [ -z "$GITHUB_BASE_REF" ]; then echo '::set-output name=run_tests::true' - echo '::set-output name=run_ssl_tests::true' else git fetch origin $GITHUB_BASE_REF --depth=1 # git diff "origin/$GITHUB_BASE_REF..." (3 dots) may be more @@ -56,7 +54,6 @@ jobs: # # https://github.com/python/core-workflow/issues/373 git diff --name-only origin/$GITHUB_BASE_REF.. | grep -qvE '(\.rst$|^Doc|^Misc)' && echo '::set-output name=run_tests::true' || true - git diff --name-only origin/$GITHUB_BASE_REF.. | grep -qE '(ssl|hashlib|hmac|^.github)' && echo '::set-output name=run_ssl_tests::true' || true fi check_generated_files: @@ -230,7 +227,7 @@ jobs: name: 'Ubuntu SSL tests with OpenSSL' runs-on: ubuntu-20.04 needs: check_source - if: needs.check_source.outputs.run_tests == 'true' && needs.check_source.outputs.run_ssl_tests == 'true' + if: needs.check_source.outputs.run_tests == 'true' strategy: fail-fast: false matrix: diff --git a/Doc/c-api/conversion.rst b/Doc/c-api/conversion.rst index 9b9c4ffa4d0343..fdb321fe7ab3f2 100644 --- a/Doc/c-api/conversion.rst +++ b/Doc/c-api/conversion.rst @@ -28,7 +28,8 @@ not. The wrappers ensure that ``str[size-1]`` is always ``'\0'`` upon return. They never write more than *size* bytes (including the trailing ``'\0'``) into str. Both functions require that ``str != NULL``, ``size > 0``, ``format != NULL`` -and ``size < INT_MAX``. +and ``size < INT_MAX``. Note that this means there is no equivalent to the C99 +``n = snprintf(NULL, 0, ...)`` which would determine the necessary buffer size. The return value (*rv*) for these functions should be interpreted as follows: diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst index ccdc9bf02b1c7b..11d01374dc1e79 100644 --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -129,7 +129,7 @@ reference or call the method from a particular class. In C++, if you want to use a method from a base class which is overridden in a derived class, you have to use the ``::`` operator -- in Python you can write ``baseclass.methodname(self, )``. This is particularly useful -for :meth:`__init__` methods, and in general in cases where a derived class +for :meth:`~object.__init__` methods, and in general in cases where a derived class method wants to extend the base class method of the same name and thus has to call the base class method somehow. @@ -232,7 +232,8 @@ Similar methods exist for bytes and bytearray objects. How fast are exceptions? ------------------------ -A try/except block is extremely efficient if no exceptions are raised. Actually +A :keyword:`try`/:keyword:`except` block is extremely efficient if no exceptions +are raised. Actually catching an exception is expensive. In versions of Python prior to 2.0 it was common to use this idiom:: @@ -352,7 +353,7 @@ will probably run out of file descriptors:: c = f.read(1) Indeed, using CPython's reference counting and destructor scheme, each new -assignment to *f* closes the previous file. With a traditional GC, however, +assignment to ``f`` closes the previous file. With a traditional GC, however, those file objects will only get collected (and closed) at varying and possibly long intervals. @@ -376,10 +377,10 @@ Python to work with it.) Traditional GC also becomes a problem when Python is embedded into other applications. While in a standalone Python it's fine to replace the standard -malloc() and free() with versions provided by the GC library, an application -embedding Python may want to have its *own* substitute for malloc() and free(), +``malloc()`` and ``free()`` with versions provided by the GC library, an application +embedding Python may want to have its *own* substitute for ``malloc()`` and ``free()``, and may not want Python's. Right now, CPython works with anything that -implements malloc() and free() properly. +implements ``malloc()`` and ``free()`` properly. Why isn't all memory freed when CPython exits? @@ -401,14 +402,15 @@ Why are there separate tuple and list data types? Lists and tuples, while similar in many respects, are generally used in fundamentally different ways. Tuples can be thought of as being similar to -Pascal records or C structs; they're small collections of related data which may +Pascal ``records`` or C ``structs``; they're small collections of related data which may be of different types which are operated on as a group. For example, a Cartesian coordinate is appropriately represented as a tuple of two or three numbers. Lists, on the other hand, are more like arrays in other languages. They tend to hold a varying number of objects all of which have the same type and which are -operated on one-by-one. For example, ``os.listdir('.')`` returns a list of +operated on one-by-one. For example, :func:`os.listdir('.') ` +returns a list of strings representing the files in the current directory. Functions which operate on this output would generally not break if you added another file or two to the directory. @@ -444,9 +446,9 @@ far) under most circumstances, and the implementation is simpler. Dictionaries work by computing a hash code for each key stored in the dictionary using the :func:`hash` built-in function. The hash code varies widely depending -on the key and a per-process seed; for example, "Python" could hash to --539294296 while "python", a string that differs by a single bit, could hash -to 1142331976. The hash code is then used to calculate a location in an +on the key and a per-process seed; for example, ``'Python'`` could hash to +``-539294296`` while ``'python'``, a string that differs by a single bit, could hash +to ``1142331976``. The hash code is then used to calculate a location in an internal array where the value will be stored. Assuming that you're storing keys that all have different hash values, this means that dictionaries take constant time -- O(1), in Big-O notation -- to retrieve a key. @@ -497,7 +499,8 @@ Some unacceptable solutions that have been proposed: There is a trick to get around this if you need to, but use it at your own risk: You can wrap a mutable structure inside a class instance which has both a -:meth:`__eq__` and a :meth:`__hash__` method. You must then make sure that the +:meth:`~object.__eq__` and a :meth:`~object.__hash__` method. +You must then make sure that the hash value for all such wrapper objects that reside in a dictionary (or other hash based structure), remain fixed while the object is in the dictionary (or other structure). :: @@ -528,7 +531,7 @@ is True``) then ``hash(o1) == hash(o2)`` (ie, ``o1.__hash__() == o2.__hash__()`` regardless of whether the object is in a dictionary or not. If you fail to meet these restrictions dictionaries and other hash based structures will misbehave. -In the case of ListWrapper, whenever the wrapper object is in a dictionary the +In the case of :class:`!ListWrapper`, whenever the wrapper object is in a dictionary the wrapped list must not change to avoid anomalies. Don't do this unless you are prepared to think hard about the requirements and the consequences of not meeting them correctly. Consider yourself warned. @@ -581,9 +584,9 @@ exhaustive test suites that exercise every line of code in a module. An appropriate testing discipline can help build large complex applications in Python as well as having interface specifications would. In fact, it can be better because an interface specification cannot test certain properties of a -program. For example, the :meth:`append` method is expected to add new elements +program. For example, the :meth:`list.append` method is expected to add new elements to the end of some internal list; an interface specification cannot test that -your :meth:`append` implementation will actually do this correctly, but it's +your :meth:`list.append` implementation will actually do this correctly, but it's trivial to check this property in a test suite. Writing test suites is very helpful, and you might want to design your code to @@ -599,14 +602,14 @@ Why is there no goto? In the 1970s people realized that unrestricted goto could lead to messy "spaghetti" code that was hard to understand and revise. In a high-level language, it is also unneeded as long as there -are ways to branch (in Python, with ``if`` statements and ``or``, -``and``, and ``if-else`` expressions) and loop (with ``while`` -and ``for`` statements, possibly containing ``continue`` and ``break``). +are ways to branch (in Python, with :keyword:`if` statements and :keyword:`or`, +:keyword:`and`, and :keyword:`if`/:keyword:`else` expressions) and loop (with :keyword:`while` +and :keyword:`for` statements, possibly containing :keyword:`continue` and :keyword:`break`). One can also use exceptions to provide a "structured goto" that works even across function calls. Many feel that exceptions can conveniently emulate all -reasonable uses of the "go" or "goto" constructs of C, Fortran, and other +reasonable uses of the ``go`` or ``goto`` constructs of C, Fortran, and other languages. For example:: class label(Exception): pass # declare a label @@ -620,7 +623,7 @@ languages. For example:: ... This doesn't allow you to jump into the middle of a loop, but that's usually -considered an abuse of goto anyway. Use sparingly. +considered an abuse of ``goto`` anyway. Use sparingly. Why can't raw strings (r-strings) end with a backslash? @@ -652,7 +655,7 @@ If you're trying to build a pathname for a DOS command, try e.g. one of :: Why doesn't Python have a "with" statement for attribute assignments? --------------------------------------------------------------------- -Python has a 'with' statement that wraps the execution of a block, calling code +Python has a :keyword:`with` statement that wraps the execution of a block, calling code on the entrance and exit from the block. Some languages have a construct that looks like this:: @@ -679,13 +682,13 @@ For instance, take the following incomplete snippet:: with a: print(x) -The snippet assumes that "a" must have a member attribute called "x". However, +The snippet assumes that ``a`` must have a member attribute called ``x``. However, there is nothing in Python that tells the interpreter this. What should happen -if "a" is, let us say, an integer? If there is a global variable named "x", -will it be used inside the with block? As you see, the dynamic nature of Python +if ``a`` is, let us say, an integer? If there is a global variable named ``x``, +will it be used inside the :keyword:`with` block? As you see, the dynamic nature of Python makes such choices much harder. -The primary benefit of "with" and similar language features (reduction of code +The primary benefit of :keyword:`with` and similar language features (reduction of code volume) can, however, easily be achieved in Python by assignment. Instead of:: function(args).mydict[index][index].a = 21 @@ -714,7 +717,8 @@ Why don't generators support the with statement? For technical reasons, a generator used directly as a context manager would not work correctly. When, as is most common, a generator is used as an iterator run to completion, no closing is needed. When it is, wrap -it as "contextlib.closing(generator)" in the 'with' statement. +it as :func:`contextlib.closing(generator) ` +in the :keyword:`with` statement. Why are colons required for the if/while/def/class statements? diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst index f79cf485712274..a9cde456575020 100644 --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -609,7 +609,7 @@ use ``p.read(n)``. substituted for standard input and output. You will have to use pseudo ttys ("ptys") instead of pipes. Or you can use a Python interface to Don Libes' "expect" library. A Python extension that interfaces to expect is called - "expy" and available from http://expectpy.sourceforge.net. A pure Python + "expy" and available from https://expectpy.sourceforge.net. A pure Python solution that works like expect is `pexpect `_. diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index d634c4b47db90e..b8afc7e6d762aa 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -539,7 +539,15 @@ Let's dive in! }; -16. Compile, then run the relevant portions of the regression-test suite. +16. Argument Clinic may generate new instances of ``_Py_ID``. For example:: + + &_Py_ID(new_unique_py_id) + + If it does, you'll have to run ``Tools/scripts/generate_global_objects.py`` + to regenerate the list of precompiled identifiers at this point. + + +17. Compile, then run the relevant portions of the regression-test suite. This change should not introduce any new compile-time warnings or errors, and there should be no externally visible change to Python's behavior. diff --git a/Tools/scripts/diff.py b/Doc/includes/diff.py old mode 100755 new mode 100644 similarity index 98% rename from Tools/scripts/diff.py rename to Doc/includes/diff.py index 96199b85116d7d..001619f5f83fc0 --- a/Tools/scripts/diff.py +++ b/Doc/includes/diff.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """ Command line interface to difflib.py providing diffs in four formats: * ndiff: lists every line and highlights interline changes. diff --git a/Tools/scripts/ndiff.py b/Doc/includes/ndiff.py old mode 100755 new mode 100644 similarity index 80% rename from Tools/scripts/ndiff.py rename to Doc/includes/ndiff.py index c6d09b8f242fef..32c251bce9120b --- a/Tools/scripts/ndiff.py +++ b/Doc/includes/ndiff.py @@ -1,16 +1,3 @@ -#! /usr/bin/env python3 - -# Module ndiff version 1.7.0 -# Released to the public domain 08-Dec-2000, -# by Tim Peters (tim.one@home.com). - -# Provided as-is; use at your own risk; no warranty; no promises; enjoy! - -# ndiff.py is now simply a front-end to the difflib.ndiff() function. -# Originally, it contained the difflib.SequenceMatcher class as well. -# This completes the raiding of reusable code from this formerly -# self-contained script. - """ndiff [-q] file1 file2 or ndiff (-r1 | -r2) < ndiff_output > file1_or_file2 @@ -121,13 +108,4 @@ def restore(which): sys.stdout.writelines(restored) if __name__ == '__main__': - args = sys.argv[1:] - if "-profile" in args: - import profile, pstats - args.remove("-profile") - statf = "ndiff.pro" - profile.run("main(args)", statf) - stats = pstats.Stats(statf) - stats.strip_dirs().sort_stats('time').print_stats() - else: - main(args) + main(sys.argv[1:]) diff --git a/Doc/includes/tzinfo_examples.py b/Doc/includes/tzinfo_examples.py index 9b9e32a553e7d8..1fa6e615e46a76 100644 --- a/Doc/includes/tzinfo_examples.py +++ b/Doc/includes/tzinfo_examples.py @@ -71,7 +71,7 @@ def first_sunday_on_or_after(dt): # DST start and end times. For a complete and up-to-date set of DST rules # and timezone definitions, visit the Olson Database (or try pytz): # http://www.twinsun.com/tz/tz-link.htm -# http://sourceforge.net/projects/pytz/ (might not be up-to-date) +# https://sourceforge.net/projects/pytz/ (might not be up-to-date) # # In the US, since 2007, DST starts at 2am (standard time) on the second # Sunday in March, which is the first Sunday on or after Mar 8. diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index 0cab95e1500a66..41f11505aa1030 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -420,7 +420,7 @@ Dialects support the following attributes: .. attribute:: Dialect.skipinitialspace - When :const:`True`, whitespace immediately following the *delimiter* is ignored. + When :const:`True`, spaces immediately following the *delimiter* are ignored. The default is :const:`False`. diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index c5a279688a4491..5ee1f4a02c6816 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -145,8 +145,6 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. The arguments for this method are the same as those for the :meth:`make_file` method. - :file:`Tools/scripts/diff.py` is a command-line front-end to this class and - contains a good example of its use. .. function:: context_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\n') @@ -240,8 +238,6 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. function :func:`IS_CHARACTER_JUNK`, which filters out whitespace characters (a blank or tab; it's a bad idea to include newline in this!). - :file:`Tools/scripts/ndiff.py` is a command-line front-end to this function. - >>> diff = ndiff('one\ntwo\nthree\n'.splitlines(keepends=True), ... 'ore\ntree\nemu\n'.splitlines(keepends=True)) >>> print(''.join(diff), end="") @@ -759,7 +755,12 @@ A command-line interface to difflib ----------------------------------- This example shows how to use difflib to create a ``diff``-like utility. -It is also contained in the Python source distribution, as -:file:`Tools/scripts/diff.py`. -.. literalinclude:: ../../Tools/scripts/diff.py +.. literalinclude:: ../includes/diff.py + +ndiff example +------------- + +This example shows how to use :func:`difflib.ndiff`. + +.. literalinclude:: ../includes/ndiff.py diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 30a336197c397d..33df7be4363423 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -30,6 +30,10 @@ interpreter. Use 2 bytes for each instruction. Previously the number of bytes varied by instruction. + .. versionchanged:: 3.10 + The argument of jump, exception handling and loop instructions is now + the instruction offset rather than the byte offset. + .. versionchanged:: 3.11 Some instructions are accompanied by one or more inline cache entries, which take the form of :opcode:`CACHE` instructions. These instructions diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 26ee302d2eabc3..93c9f4ad2bc23a 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1254,8 +1254,8 @@ are always available. They are listed here in alphabetical order. .. _open-newline-parameter: - *newline* controls how :term:`universal newlines` mode works (it only - applies to text mode). It can be ``None``, ``''``, ``'\n'``, ``'\r'``, and + *newline* determines how to parse newline characters from the stream. + It can be ``None``, ``''``, ``'\n'``, ``'\r'``, and ``'\r\n'``. It works as follows: * When reading input from the stream, if *newline* is ``None``, universal diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst index 943a05c39d6840..2f0a9bd8be8815 100644 --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -150,7 +150,7 @@ The :mod:`functools` module defines the following functions: arguments to the function must be hashable. Distinct argument patterns may be considered to be distinct calls with - separate cache entries. For example, `f(a=1, b=2)` and `f(b=2, a=1)` + separate cache entries. For example, ``f(a=1, b=2)`` and ``f(b=2, a=1)`` differ in their keyword argument order and may have two separate cache entries. @@ -197,7 +197,7 @@ The :mod:`functools` module defines the following functions: The cache keeps references to the arguments and return values until they age out of the cache or until the cache is cleared. - If a method is cached, the `self` instance argument is included in the + If a method is cached, the ``self`` instance argument is included in the cache. See :ref:`faq-cache-method-calls` An `LRU (least recently used) cache diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst index eb31315438f66a..87ef156a0bed57 100644 --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -61,7 +61,7 @@ The following classes are provided: responsible for storing and retrieving cookies from a file or database. -.. class:: FileCookieJar(filename, delayload=None, policy=None) +.. class:: FileCookieJar(filename=None, delayload=None, policy=None) *policy* is an object implementing the :class:`CookiePolicy` interface. For the other arguments, see the documentation for the corresponding attributes. @@ -71,6 +71,8 @@ The following classes are provided: :meth:`load` or :meth:`revert` method is called. Subclasses of this class are documented in section :ref:`file-cookie-jar-classes`. + This should not be initialized directly – use its subclasses below instead. + .. versionchanged:: 3.8 The filename parameter supports a :term:`path-like object`. @@ -317,7 +319,7 @@ FileCookieJar subclasses and co-operation with web browsers The following :class:`CookieJar` subclasses are provided for reading and writing. -.. class:: MozillaCookieJar(filename, delayload=None, policy=None) +.. class:: MozillaCookieJar(filename=None, delayload=None, policy=None) A :class:`FileCookieJar` that can load from and save cookies to disk in the Mozilla ``cookies.txt`` file format (which is also used by curl and the Lynx @@ -338,7 +340,7 @@ writing. Mozilla. -.. class:: LWPCookieJar(filename, delayload=None, policy=None) +.. class:: LWPCookieJar(filename=None, delayload=None, policy=None) A :class:`FileCookieJar` that can load from and save cookies to disk in format compatible with the libwww-perl library's ``Set-Cookie3`` file format. This is diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index b83163560f4d9a..6571114ef311f9 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -775,10 +775,7 @@ which incur interpreter overhead. return sum(map(pred, iterable)) def pad_none(iterable): - """Returns the sequence elements and then returns None indefinitely. - - Useful for emulating the behavior of the built-in map() function. - """ + "Returns the sequence elements and then returns None indefinitely." return chain(iterable, repeat(None)) def ncycles(iterable, n): @@ -850,6 +847,13 @@ which incur interpreter overhead. else: raise ValueError('Expected fill, strict, or ignore') + def batched(iterable, n): + "Batch data into lists of length n. The last batch may be shorter." + # batched('ABCDEFG', 3) --> ABC DEF G + it = iter(iterable) + while (batch := list(islice(it, n))): + yield batch + def triplewise(iterable): "Return overlapping triplets from an iterable" # triplewise('ABCDEFG') --> ABC BCD CDE DEF EFG @@ -1168,8 +1172,8 @@ which incur interpreter overhead. >>> list(sieve(30)) [2, 3, 5, 7, 11, 13, 17, 19, 23, 29] - >>> small_primes = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59] - >>> all(list(sieve(n)) == [p for p in small_primes if p < n] for n in range(60)) + >>> small_primes = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97] + >>> all(list(sieve(n)) == [p for p in small_primes if p < n] for n in range(101)) True >>> len(list(sieve(100))) 25 @@ -1212,6 +1216,36 @@ which incur interpreter overhead. >>> list(grouper('abcdefg', n=3, incomplete='ignore')) [('a', 'b', 'c'), ('d', 'e', 'f')] + >>> list(batched('ABCDEFG', 3)) + [['A', 'B', 'C'], ['D', 'E', 'F'], ['G']] + >>> list(batched('ABCDEF', 3)) + [['A', 'B', 'C'], ['D', 'E', 'F']] + >>> list(batched('ABCDE', 3)) + [['A', 'B', 'C'], ['D', 'E']] + >>> list(batched('ABCD', 3)) + [['A', 'B', 'C'], ['D']] + >>> list(batched('ABC', 3)) + [['A', 'B', 'C']] + >>> list(batched('AB', 3)) + [['A', 'B']] + >>> list(batched('A', 3)) + [['A']] + >>> list(batched('', 3)) + [] + >>> list(batched('ABCDEFG', 2)) + [['A', 'B'], ['C', 'D'], ['E', 'F'], ['G']] + >>> list(batched('ABCDEFG', 1)) + [['A'], ['B'], ['C'], ['D'], ['E'], ['F'], ['G']] + >>> list(batched('ABCDEFG', 0)) + [] + >>> list(batched('ABCDEFG', -1)) + Traceback (most recent call last): + ... + ValueError: Stop argument for islice() must be None or an integer: 0 <= x <= sys.maxsize. + >>> s = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' + >>> all(list(flatten(batched(s[:n], 5))) == list(s[:n]) for n in range(len(s))) + True + >>> list(triplewise('ABCDEFG')) [('A', 'B', 'C'), ('B', 'C', 'D'), ('C', 'D', 'E'), ('D', 'E', 'F'), ('E', 'F', 'G')] diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 23b014b0b65924..8727f811def1c0 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -3222,7 +3222,8 @@ features: filenames)``. *dirpath* is a string, the path to the directory. *dirnames* is a list of the - names of the subdirectories in *dirpath* (excluding ``'.'`` and ``'..'``). + names of the subdirectories in *dirpath* (including symlinks to directories, + and excluding ``'.'`` and ``'..'``). *filenames* is a list of the names of the non-directory files in *dirpath*. Note that the names in the lists contain no path components. To get a full path (which begins with *top*) to a file or directory in *dirpath*, do diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst index 1478b34bc95514..96b396510794b4 100644 --- a/Doc/library/urllib.parse.rst +++ b/Doc/library/urllib.parse.rst @@ -113,7 +113,8 @@ or on combining URL components into a URL string. +------------------+-------+-------------------------+------------------------+ | :attr:`path` | 2 | Hierarchical path | empty string | +------------------+-------+-------------------------+------------------------+ - | :attr:`params` | 3 | No longer used | always an empty string | + | :attr:`params` | 3 | Parameters for last | empty string | + | | | path element | | +------------------+-------+-------------------------+------------------------+ | :attr:`query` | 4 | Query component | empty string | +------------------+-------+-------------------------+------------------------+ diff --git a/Doc/requirements.txt b/Doc/requirements.txt index 7f82dc32113a9d..32c53e95917ee4 100644 --- a/Doc/requirements.txt +++ b/Doc/requirements.txt @@ -10,7 +10,7 @@ blurb # sphinx-lint 0.6.2 yields many default role errors due to the new regular # expression used for default role detection, so we don't use the version # until the errors are fixed. -sphinx-lint==0.6.4 +sphinx-lint==0.6.5 # The theme used by the documentation is stored separately, so we need # to install that as well. diff --git a/Doc/tutorial/classes.rst b/Doc/tutorial/classes.rst index f27abe48b2d4ed..9ecbf8b87efbf1 100644 --- a/Doc/tutorial/classes.rst +++ b/Doc/tutorial/classes.rst @@ -581,7 +581,8 @@ this:: . -The name :class:`BaseClassName` must be defined in a scope containing the +The name :class:`BaseClassName` must be defined in a +namespace accessible from the scope containing the derived class definition. In place of a base class name, other arbitrary expressions are also allowed. This can be useful, for example, when the base class is defined in another module:: diff --git a/Include/internal/pycore_global_strings.h b/Include/internal/pycore_global_strings.h index 1523eef73931c9..2966b60e0cd830 100644 --- a/Include/internal/pycore_global_strings.h +++ b/Include/internal/pycore_global_strings.h @@ -468,6 +468,7 @@ struct _Py_global_strings { STRUCT_FOR_ID(modules) STRUCT_FOR_ID(mro) STRUCT_FOR_ID(msg) + STRUCT_FOR_ID(mycmp) STRUCT_FOR_ID(n) STRUCT_FOR_ID(n_arg) STRUCT_FOR_ID(n_fields) @@ -493,6 +494,7 @@ struct _Py_global_strings { STRUCT_FOR_ID(offset_src) STRUCT_FOR_ID(on_type_read) STRUCT_FOR_ID(onceregistry) + STRUCT_FOR_ID(only_keys) STRUCT_FOR_ID(oparg) STRUCT_FOR_ID(opcode) STRUCT_FOR_ID(open) diff --git a/Include/internal/pycore_runtime_init_generated.h b/Include/internal/pycore_runtime_init_generated.h index 32ff57b731e1a0..617582f96e33f5 100644 --- a/Include/internal/pycore_runtime_init_generated.h +++ b/Include/internal/pycore_runtime_init_generated.h @@ -977,6 +977,7 @@ extern "C" { INIT_ID(modules), \ INIT_ID(mro), \ INIT_ID(msg), \ + INIT_ID(mycmp), \ INIT_ID(n), \ INIT_ID(n_arg), \ INIT_ID(n_fields), \ @@ -1002,6 +1003,7 @@ extern "C" { INIT_ID(offset_src), \ INIT_ID(on_type_read), \ INIT_ID(onceregistry), \ + INIT_ID(only_keys), \ INIT_ID(oparg), \ INIT_ID(opcode), \ INIT_ID(open), \ @@ -2260,6 +2262,8 @@ _PyUnicode_InitStaticStrings(void) { PyUnicode_InternInPlace(&string); string = &_Py_ID(msg); PyUnicode_InternInPlace(&string); + string = &_Py_ID(mycmp); + PyUnicode_InternInPlace(&string); string = &_Py_ID(n); PyUnicode_InternInPlace(&string); string = &_Py_ID(n_arg); @@ -2310,6 +2314,8 @@ _PyUnicode_InitStaticStrings(void) { PyUnicode_InternInPlace(&string); string = &_Py_ID(onceregistry); PyUnicode_InternInPlace(&string); + string = &_Py_ID(only_keys); + PyUnicode_InternInPlace(&string); string = &_Py_ID(oparg); PyUnicode_InternInPlace(&string); string = &_Py_ID(opcode); @@ -6447,6 +6453,10 @@ _PyStaticObjects_CheckRefcnt(void) { _PyObject_Dump((PyObject *)&_Py_ID(msg)); Py_FatalError("immortal object has less refcnt than expected _PyObject_IMMORTAL_REFCNT"); }; + if (Py_REFCNT((PyObject *)&_Py_ID(mycmp)) < _PyObject_IMMORTAL_REFCNT) { + _PyObject_Dump((PyObject *)&_Py_ID(mycmp)); + Py_FatalError("immortal object has less refcnt than expected _PyObject_IMMORTAL_REFCNT"); + }; if (Py_REFCNT((PyObject *)&_Py_ID(n)) < _PyObject_IMMORTAL_REFCNT) { _PyObject_Dump((PyObject *)&_Py_ID(n)); Py_FatalError("immortal object has less refcnt than expected _PyObject_IMMORTAL_REFCNT"); @@ -6547,6 +6557,10 @@ _PyStaticObjects_CheckRefcnt(void) { _PyObject_Dump((PyObject *)&_Py_ID(onceregistry)); Py_FatalError("immortal object has less refcnt than expected _PyObject_IMMORTAL_REFCNT"); }; + if (Py_REFCNT((PyObject *)&_Py_ID(only_keys)) < _PyObject_IMMORTAL_REFCNT) { + _PyObject_Dump((PyObject *)&_Py_ID(only_keys)); + Py_FatalError("immortal object has less refcnt than expected _PyObject_IMMORTAL_REFCNT"); + }; if (Py_REFCNT((PyObject *)&_Py_ID(oparg)) < _PyObject_IMMORTAL_REFCNT) { _PyObject_Dump((PyObject *)&_Py_ID(oparg)); Py_FatalError("immortal object has less refcnt than expected _PyObject_IMMORTAL_REFCNT"); diff --git a/Include/internal/pycore_symtable.h b/Include/internal/pycore_symtable.h index 2d64aba22ff905..8532646ce7d95c 100644 --- a/Include/internal/pycore_symtable.h +++ b/Include/internal/pycore_symtable.h @@ -49,7 +49,7 @@ typedef struct _symtable_entry { PyObject *ste_varnames; /* list of function parameters */ PyObject *ste_children; /* list of child blocks */ PyObject *ste_directives;/* locations of global and nonlocal statements */ - _Py_block_ty ste_type; /* module, class or function */ + _Py_block_ty ste_type; /* module, class, function or annotation */ int ste_nested; /* true if block is nested */ unsigned ste_free : 1; /* true if block has free variables */ unsigned ste_child_free : 1; /* true if a child block has free vars, diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py index c9bbe2ac014351..d2ee49dd88f8cf 100644 --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -58,6 +58,7 @@ def __init__(self, selector=None): def _make_socket_transport(self, sock, protocol, waiter=None, *, extra=None, server=None): + self._ensure_fd_no_transport(sock) return _SelectorSocketTransport(self, sock, protocol, waiter, extra, server) @@ -68,6 +69,7 @@ def _make_ssl_transport( ssl_handshake_timeout=constants.SSL_HANDSHAKE_TIMEOUT, ssl_shutdown_timeout=constants.SSL_SHUTDOWN_TIMEOUT, ): + self._ensure_fd_no_transport(rawsock) ssl_protocol = sslproto.SSLProtocol( self, protocol, sslcontext, waiter, server_side, server_hostname, @@ -80,6 +82,7 @@ def _make_ssl_transport( def _make_datagram_transport(self, sock, protocol, address=None, waiter=None, extra=None): + self._ensure_fd_no_transport(sock) return _SelectorDatagramTransport(self, sock, protocol, address, waiter, extra) diff --git a/Lib/inspect.py b/Lib/inspect.py index 8a107a894909eb..f6750c3b211fbd 100644 --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -1433,7 +1433,10 @@ def getargvalues(frame): def formatannotation(annotation, base_module=None): if getattr(annotation, '__module__', None) == 'typing': - return repr(annotation).replace('typing.', '') + def repl(match): + text = match.group() + return text.removeprefix('typing.') + return re.sub(r'[\w\.]+', repl, repr(annotation)) if isinstance(annotation, types.GenericAlias): return str(annotation) if isinstance(annotation, type): @@ -3099,8 +3102,12 @@ def _bind(self, args, kwargs, *, partial=False): parameters_ex = (param,) break else: - msg = 'missing a required argument: {arg!r}' - msg = msg.format(arg=param.name) + if param.kind == _KEYWORD_ONLY: + argtype = ' keyword-only' + else: + argtype = '' + msg = 'missing a required{argtype} argument: {arg!r}' + msg = msg.format(arg=param.name, argtype=argtype) raise TypeError(msg) from None else: # We have a positional argument to process diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py index f6c43b3b92bc50..3224363a3f2bfb 100644 --- a/Lib/mimetypes.py +++ b/Lib/mimetypes.py @@ -427,8 +427,8 @@ def _default_mime_types(): # Make sure the entry with the preferred file extension for a particular mime type # appears before any others of the same mimetype. types_map = _types_map_default = { - '.js' : 'application/javascript', - '.mjs' : 'application/javascript', + '.js' : 'text/javascript', + '.mjs' : 'text/javascript', '.json' : 'application/json', '.webmanifest': 'application/manifest+json', '.doc' : 'application/msword', diff --git a/Lib/os.py b/Lib/os.py index 648188e0f13490..fd1e774fdcbcfa 100644 --- a/Lib/os.py +++ b/Lib/os.py @@ -288,7 +288,8 @@ def walk(top, topdown=True, onerror=None, followlinks=False): dirpath, dirnames, filenames dirpath is a string, the path to the directory. dirnames is a list of - the names of the subdirectories in dirpath (excluding '.' and '..'). + the names of the subdirectories in dirpath (including symlinks to directories, + and excluding '.' and '..'). filenames is a list of the names of the non-directory files in dirpath. Note that the names in the lists are just names, with no path components. To get a full path (which begins with top) to a file or directory in diff --git a/Lib/re/__init__.py b/Lib/re/__init__.py index d58c2117ef3e14..8d6a4ef3880f0c 100644 --- a/Lib/re/__init__.py +++ b/Lib/re/__init__.py @@ -229,6 +229,7 @@ def compile(pattern, flags=0): def purge(): "Clear the regular expression caches" _cache.clear() + _cache2.clear() _compile_repl.cache_clear() def template(pattern, flags=0): @@ -266,40 +267,64 @@ def escape(pattern): # -------------------------------------------------------------------- # internals -_cache = {} # ordered! - +# Use the fact that dict keeps the insertion order. +# _cache2 uses the simple FIFO policy which has better latency. +# _cache uses the LRU policy which has better hit rate. +_cache = {} # LRU +_cache2 = {} # FIFO _MAXCACHE = 512 +_MAXCACHE2 = 256 +assert _MAXCACHE2 < _MAXCACHE + def _compile(pattern, flags): # internal: compile pattern if isinstance(flags, RegexFlag): flags = flags.value try: - return _cache[type(pattern), pattern, flags] + return _cache2[type(pattern), pattern, flags] except KeyError: pass - if isinstance(pattern, Pattern): - if flags: - raise ValueError( - "cannot process flags argument with a compiled pattern") - return pattern - if not _compiler.isstring(pattern): - raise TypeError("first argument must be string or compiled pattern") - if flags & T: - import warnings - warnings.warn("The re.TEMPLATE/re.T flag is deprecated " - "as it is an undocumented flag " - "without an obvious purpose. " - "Don't use it.", - DeprecationWarning) - p = _compiler.compile(pattern, flags) - if not (flags & DEBUG): + + key = (type(pattern), pattern, flags) + # Item in _cache should be moved to the end if found. + p = _cache.pop(key, None) + if p is None: + if isinstance(pattern, Pattern): + if flags: + raise ValueError( + "cannot process flags argument with a compiled pattern") + return pattern + if not _compiler.isstring(pattern): + raise TypeError("first argument must be string or compiled pattern") + if flags & T: + import warnings + warnings.warn("The re.TEMPLATE/re.T flag is deprecated " + "as it is an undocumented flag " + "without an obvious purpose. " + "Don't use it.", + DeprecationWarning) + p = _compiler.compile(pattern, flags) + if flags & DEBUG: + return p if len(_cache) >= _MAXCACHE: - # Drop the oldest item + # Drop the least recently used item. + # next(iter(_cache)) is known to have linear amortized time, + # but it is used here to avoid a dependency from using OrderedDict. + # For the small _MAXCACHE value it doesn't make much of a difference. try: del _cache[next(iter(_cache))] except (StopIteration, RuntimeError, KeyError): pass - _cache[type(pattern), pattern, flags] = p + # Append to the end. + _cache[key] = p + + if len(_cache2) >= _MAXCACHE2: + # Drop the oldest item. + try: + del _cache2[next(iter(_cache2))] + except (StopIteration, RuntimeError, KeyError): + pass + _cache2[key] = p return p @functools.lru_cache(_MAXCACHE) diff --git a/Lib/subprocess.py b/Lib/subprocess.py index 760b93b47ebba6..9cadd1bf8e622c 100644 --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -509,7 +509,8 @@ def run(*popenargs, The returned instance will have attributes args, returncode, stdout and stderr. By default, stdout and stderr are not captured, and those attributes - will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them. + will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them, + or pass capture_output=True to capture both. If check is True and the exit code was non-zero, it raises a CalledProcessError. The CalledProcessError object will have the return code diff --git a/Lib/test/audit-tests.py b/Lib/test/audit-tests.py index 66c08f7f2ff8d5..4abf33d7cad1b6 100644 --- a/Lib/test/audit-tests.py +++ b/Lib/test/audit-tests.py @@ -429,6 +429,26 @@ def hook(event, args): sys.addaudithook(hook) _wmi.exec_query("SELECT * FROM Win32_OperatingSystem") +def test_syslog(): + import syslog + + def hook(event, args): + if event.startswith("syslog."): + print(event, *args) + + sys.addaudithook(hook) + syslog.openlog('python') + syslog.syslog('test') + syslog.setlogmask(syslog.LOG_DEBUG) + syslog.closelog() + # implicit open + syslog.syslog('test2') + # open with default ident + syslog.openlog(logoption=syslog.LOG_NDELAY, facility=syslog.LOG_LOCAL0) + sys.argv = None + syslog.openlog() + syslog.closelog() + if __name__ == "__main__": from test.support import suppress_msvcrt_asserts diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py index 22dcfb23083522..796037bcf59c49 100644 --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -61,8 +61,10 @@ def setUp(self): def test_make_socket_transport(self): m = mock.Mock() self.loop.add_reader = mock.Mock() + self.loop._ensure_fd_no_transport = mock.Mock() transport = self.loop._make_socket_transport(m, asyncio.Protocol()) self.assertIsInstance(transport, _SelectorSocketTransport) + self.assertEqual(self.loop._ensure_fd_no_transport.call_count, 1) # Calling repr() must not fail when the event loop is closed self.loop.close() @@ -78,8 +80,10 @@ def test_make_ssl_transport_without_ssl_error(self): self.loop.add_writer = mock.Mock() self.loop.remove_reader = mock.Mock() self.loop.remove_writer = mock.Mock() + self.loop._ensure_fd_no_transport = mock.Mock() with self.assertRaises(RuntimeError): self.loop._make_ssl_transport(m, m, m, m) + self.assertEqual(self.loop._ensure_fd_no_transport.call_count, 1) def test_close(self): class EventLoop(BaseSelectorEventLoop): diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py index 61d5e984dfbfbb..0c49099bc499a5 100644 --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -566,10 +566,46 @@ def test_exception_cancel(self): test_utils.run_briefly(self.loop) self.assertIs(stream._waiter, None) - -class NewStreamTests(unittest.IsolatedAsyncioTestCase): - - async def test_start_server(self): + def test_start_server(self): + + class MyServer: + + def __init__(self, loop): + self.server = None + self.loop = loop + + async def handle_client(self, client_reader, client_writer): + data = await client_reader.readline() + client_writer.write(data) + await client_writer.drain() + client_writer.close() + await client_writer.wait_closed() + + def start(self): + sock = socket.create_server(('127.0.0.1', 0)) + self.server = self.loop.run_until_complete( + asyncio.start_server(self.handle_client, + sock=sock)) + return sock.getsockname() + + def handle_client_callback(self, client_reader, client_writer): + self.loop.create_task(self.handle_client(client_reader, + client_writer)) + + def start_callback(self): + sock = socket.create_server(('127.0.0.1', 0)) + addr = sock.getsockname() + sock.close() + self.server = self.loop.run_until_complete( + asyncio.start_server(self.handle_client_callback, + host=addr[0], port=addr[1])) + return addr + + def stop(self): + if self.server is not None: + self.server.close() + self.loop.run_until_complete(self.server.wait_closed()) + self.server = None async def client(addr): reader, writer = await asyncio.open_connection(*addr) @@ -581,43 +617,61 @@ async def client(addr): await writer.wait_closed() return msgback - async def handle_client(client_reader, client_writer): - data = await client_reader.readline() - client_writer.write(data) - await client_writer.drain() - client_writer.close() - await client_writer.wait_closed() - - with self.subTest(msg="coroutine"): - server = await asyncio.start_server( - handle_client, - host=socket_helper.HOSTv4 - ) - addr = server.sockets[0].getsockname() - msg = await client(addr) - server.close() - await server.wait_closed() - self.assertEqual(msg, b"hello world!\n") + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) - with self.subTest(msg="callback"): - async def handle_client_callback(client_reader, client_writer): - asyncio.get_running_loop().create_task( - handle_client(client_reader, client_writer) - ) + # test the server variant with a coroutine as client handler + server = MyServer(self.loop) + addr = server.start() + msg = self.loop.run_until_complete(self.loop.create_task(client(addr))) + server.stop() + self.assertEqual(msg, b"hello world!\n") - server = await asyncio.start_server( - handle_client_callback, - host=socket_helper.HOSTv4 - ) - addr = server.sockets[0].getsockname() - reader, writer = await asyncio.open_connection(*addr) - msg = await client(addr) - server.close() - await server.wait_closed() - self.assertEqual(msg, b"hello world!\n") + # test the server variant with a callback as client handler + server = MyServer(self.loop) + addr = server.start_callback() + msg = self.loop.run_until_complete(self.loop.create_task(client(addr))) + server.stop() + self.assertEqual(msg, b"hello world!\n") + + self.assertEqual(messages, []) @socket_helper.skip_unless_bind_unix_socket - async def test_start_unix_server(self): + def test_start_unix_server(self): + + class MyServer: + + def __init__(self, loop, path): + self.server = None + self.loop = loop + self.path = path + + async def handle_client(self, client_reader, client_writer): + data = await client_reader.readline() + client_writer.write(data) + await client_writer.drain() + client_writer.close() + await client_writer.wait_closed() + + def start(self): + self.server = self.loop.run_until_complete( + asyncio.start_unix_server(self.handle_client, + path=self.path)) + + def handle_client_callback(self, client_reader, client_writer): + self.loop.create_task(self.handle_client(client_reader, + client_writer)) + + def start_callback(self): + start = asyncio.start_unix_server(self.handle_client_callback, + path=self.path) + self.server = self.loop.run_until_complete(start) + + def stop(self): + if self.server is not None: + self.server.close() + self.loop.run_until_complete(self.server.wait_closed()) + self.server = None async def client(path): reader, writer = await asyncio.open_unix_connection(path) @@ -629,42 +683,64 @@ async def client(path): await writer.wait_closed() return msgback - async def handle_client(client_reader, client_writer): - data = await client_reader.readline() - client_writer.write(data) - await client_writer.drain() - client_writer.close() - await client_writer.wait_closed() - - with self.subTest(msg="coroutine"): - with test_utils.unix_socket_path() as path: - server = await asyncio.start_unix_server( - handle_client, - path=path - ) - msg = await client(path) - server.close() - await server.wait_closed() - self.assertEqual(msg, b"hello world!\n") - - with self.subTest(msg="callback"): - async def handle_client_callback(client_reader, client_writer): - asyncio.get_running_loop().create_task( - handle_client(client_reader, client_writer) - ) - - with test_utils.unix_socket_path() as path: - server = await asyncio.start_unix_server( - handle_client_callback, - path=path - ) - msg = await client(path) - server.close() - await server.wait_closed() - self.assertEqual(msg, b"hello world!\n") + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + + # test the server variant with a coroutine as client handler + with test_utils.unix_socket_path() as path: + server = MyServer(self.loop, path) + server.start() + msg = self.loop.run_until_complete( + self.loop.create_task(client(path))) + server.stop() + self.assertEqual(msg, b"hello world!\n") + + # test the server variant with a callback as client handler + with test_utils.unix_socket_path() as path: + server = MyServer(self.loop, path) + server.start_callback() + msg = self.loop.run_until_complete( + self.loop.create_task(client(path))) + server.stop() + self.assertEqual(msg, b"hello world!\n") + + self.assertEqual(messages, []) @unittest.skipIf(ssl is None, 'No ssl module') - async def test_start_tls(self): + def test_start_tls(self): + + class MyServer: + + def __init__(self, loop): + self.server = None + self.loop = loop + + async def handle_client(self, client_reader, client_writer): + data1 = await client_reader.readline() + client_writer.write(data1) + await client_writer.drain() + assert client_writer.get_extra_info('sslcontext') is None + await client_writer.start_tls( + test_utils.simple_server_sslcontext()) + assert client_writer.get_extra_info('sslcontext') is not None + data2 = await client_reader.readline() + client_writer.write(data2) + await client_writer.drain() + client_writer.close() + await client_writer.wait_closed() + + def start(self): + sock = socket.create_server(('127.0.0.1', 0)) + self.server = self.loop.run_until_complete( + asyncio.start_server(self.handle_client, + sock=sock)) + return sock.getsockname() + + def stop(self): + if self.server is not None: + self.server.close() + self.loop.run_until_complete(self.server.wait_closed()) + self.server = None async def client(addr): reader, writer = await asyncio.open_connection(*addr) @@ -681,48 +757,17 @@ async def client(addr): await writer.wait_closed() return msgback1, msgback2 - async def handle_client(client_reader, client_writer): - data1 = await client_reader.readline() - client_writer.write(data1) - await client_writer.drain() - assert client_writer.get_extra_info('sslcontext') is None - await client_writer.start_tls( - test_utils.simple_server_sslcontext()) - assert client_writer.get_extra_info('sslcontext') is not None - - data2 = await client_reader.readline() - client_writer.write(data2) - await client_writer.drain() - client_writer.close() - await client_writer.wait_closed() - - server = await asyncio.start_server( - handle_client, - host=socket_helper.HOSTv4 - ) - addr = server.sockets[0].getsockname() - - msg1, msg2 = await client(addr) - server.close() - await server.wait_closed() - self.assertEqual(msg1, b"hello world 1!\n") - self.assertEqual(msg2, b"hello world 2!\n") - - -class StreamTests2(test_utils.TestCase): - - def setUp(self): - super().setUp() - self.loop = asyncio.new_event_loop() - self.set_event_loop(self.loop) + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) - def tearDown(self): - # just in case if we have transport close callbacks - test_utils.run_briefly(self.loop) + server = MyServer(self.loop) + addr = server.start() + msg1, msg2 = self.loop.run_until_complete(client(addr)) + server.stop() - self.loop.close() - gc.collect() - super().tearDown() + self.assertEqual(messages, []) + self.assertEqual(msg1, b"hello world 1!\n") + self.assertEqual(msg2, b"hello world 2!\n") @unittest.skipIf(sys.platform == 'win32', "Don't have pipes") def test_read_all_from_pipe_reader(self): @@ -941,32 +986,36 @@ def test_LimitOverrunError_pickleable(self): self.assertEqual(str(e), str(e2)) self.assertEqual(e.consumed, e2.consumed) -class NewStreamTests2(unittest.IsolatedAsyncioTestCase): - async def test_wait_closed_on_close(self): + def test_wait_closed_on_close(self): with test_utils.run_test_server() as httpd: - rd, wr = await asyncio.open_connection(*httpd.address) + rd, wr = self.loop.run_until_complete( + asyncio.open_connection(*httpd.address)) wr.write(b'GET / HTTP/1.0\r\n\r\n') - data = await rd.readline() + f = rd.readline() + data = self.loop.run_until_complete(f) self.assertEqual(data, b'HTTP/1.0 200 OK\r\n') - data = await rd.read() + f = rd.read() + data = self.loop.run_until_complete(f) self.assertTrue(data.endswith(b'\r\n\r\nTest message')) self.assertFalse(wr.is_closing()) wr.close() self.assertTrue(wr.is_closing()) - await wr.wait_closed() + self.loop.run_until_complete(wr.wait_closed()) - async def test_wait_closed_on_close_with_unread_data(self): + def test_wait_closed_on_close_with_unread_data(self): with test_utils.run_test_server() as httpd: - rd, wr = await asyncio.open_connection(*httpd.address) + rd, wr = self.loop.run_until_complete( + asyncio.open_connection(*httpd.address)) wr.write(b'GET / HTTP/1.0\r\n\r\n') - data = await rd.readline() + f = rd.readline() + data = self.loop.run_until_complete(f) self.assertEqual(data, b'HTTP/1.0 200 OK\r\n') wr.close() - await wr.wait_closed() + self.loop.run_until_complete(wr.wait_closed()) - async def test_async_writer_api(self): + def test_async_writer_api(self): async def inner(httpd): rd, wr = await asyncio.open_connection(*httpd.address) @@ -978,10 +1027,15 @@ async def inner(httpd): wr.close() await wr.wait_closed() + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + with test_utils.run_test_server() as httpd: - await inner(httpd) + self.loop.run_until_complete(inner(httpd)) - async def test_async_writer_api_exception_after_close(self): + self.assertEqual(messages, []) + + def test_async_writer_api_exception_after_close(self): async def inner(httpd): rd, wr = await asyncio.open_connection(*httpd.address) @@ -995,19 +1049,33 @@ async def inner(httpd): wr.write(b'data') await wr.drain() + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + with test_utils.run_test_server() as httpd: - await inner(httpd) + self.loop.run_until_complete(inner(httpd)) - async def test_eof_feed_when_closing_writer(self): + self.assertEqual(messages, []) + + def test_eof_feed_when_closing_writer(self): # See http://bugs.python.org/issue35065 + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + with test_utils.run_test_server() as httpd: - rd, wr = await asyncio.open_connection(*httpd.address) + rd, wr = self.loop.run_until_complete( + asyncio.open_connection(*httpd.address)) + wr.close() - await wr.wait_closed() + f = wr.wait_closed() + self.loop.run_until_complete(f) self.assertTrue(rd.at_eof()) - data = await rd.read() + f = rd.read() + data = self.loop.run_until_complete(f) self.assertEqual(data, b'') + self.assertEqual(messages, []) + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_audit.py b/Lib/test/test_audit.py index 09b3333afe184f..50f78f2a6b8a46 100644 --- a/Lib/test/test_audit.py +++ b/Lib/test/test_audit.py @@ -16,6 +16,7 @@ class AuditTest(unittest.TestCase): + maxDiff = None @support.requires_subprocess() def do_test(self, *args): @@ -185,7 +186,6 @@ def test_sys_getframe(self): self.assertEqual(actual, expected) - def test_wmi_exec_query(self): import_helper.import_module("_wmi") returncode, events, stderr = self.run_python("test_wmi_exec_query") @@ -199,6 +199,29 @@ def test_wmi_exec_query(self): self.assertEqual(actual, expected) + def test_syslog(self): + syslog = import_helper.import_module("syslog") + + returncode, events, stderr = self.run_python("test_syslog") + if returncode: + self.fail(stderr) + + if support.verbose: + print('Events:', *events, sep='\n ') + + self.assertSequenceEqual( + events, + [('syslog.openlog', ' ', f'python 0 {syslog.LOG_USER}'), + ('syslog.syslog', ' ', f'{syslog.LOG_INFO} test'), + ('syslog.setlogmask', ' ', f'{syslog.LOG_DEBUG}'), + ('syslog.closelog', '', ''), + ('syslog.syslog', ' ', f'{syslog.LOG_INFO} test2'), + ('syslog.openlog', ' ', f'audit-tests.py 0 {syslog.LOG_USER}'), + ('syslog.openlog', ' ', f'audit-tests.py {syslog.LOG_NDELAY} {syslog.LOG_LOCAL0}'), + ('syslog.openlog', ' ', f'None 0 {syslog.LOG_USER}'), + ('syslog.closelog', '', '')] + ) + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index cb90d55941cae7..19367dfcc1ccbb 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -1495,6 +1495,9 @@ def unraisable_hook(unraisable): unraisable = unraisables[0] self.assertIs(unraisable.object, d) self.assertEqual(str(unraisable.exc_value), "boom!") + # avoid leaking reference cycles + del unraisable + del unraisables def test_two_watchers(self): d1 = {} diff --git a/Lib/test/test_clinic.py b/Lib/test/test_clinic.py index 4aa9691a4829d1..8ab40c694f711d 100644 --- a/Lib/test/test_clinic.py +++ b/Lib/test/test_clinic.py @@ -153,7 +153,7 @@ def test_right_only(self): def test_have_left_options_but_required_is_empty(self): def fn(): clinic.permute_optional_groups(['a'], [], []) - self.assertRaises(AssertionError, fn) + self.assertRaises(ValueError, fn) class ClinicLinearFormatTest(TestCase): diff --git a/Lib/test/test_coroutines.py b/Lib/test/test_coroutines.py index 9a2279d353ffa4..f91c9cc47741b5 100644 --- a/Lib/test/test_coroutines.py +++ b/Lib/test/test_coroutines.py @@ -1287,7 +1287,7 @@ async def __aexit__(self, *exc): async def func(): async with CM(): - assert (1, ) == 1 + self.assertEqual((1, ), 1) with self.assertRaises(AssertionError): run_async(func()) diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py index a2b00430c7dd60..d64bff13a44e87 100644 --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -362,6 +362,11 @@ def test_read_quoting(self): self._read_test(['1,@,3,@,5'], [['1', ',3,', '5']], quotechar='@') self._read_test(['1,\0,3,\0,5'], [['1', ',3,', '5']], quotechar='\0') + def test_read_skipinitialspace(self): + self._read_test(['no space, space, spaces,\ttab'], + [['no space', 'space', 'spaces', '\ttab']], + skipinitialspace=True) + def test_read_bigfield(self): # This exercises the buffer realloc functionality and field size # limits. diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py index fc2862c61baadb..667b267d27a06a 100644 --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -1212,7 +1212,7 @@ def get_disassembly(self, func, lasti=-1, wrapper=True, **kwargs): return output.getvalue() -if sys.flags.optimize: +if dis.code_info.__doc__ is None: code_info_consts = "0: None" else: code_info_consts = "0: 'Formatted details of methods, functions, or code.'" diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py index a4b098a2a5b8ed..730ab1f595f22c 100644 --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -17,6 +17,7 @@ import gc from weakref import proxy import contextlib +from inspect import Signature from test.support import import_helper from test.support import threading_helper @@ -941,6 +942,10 @@ def mycmp(x, y): self.assertRaises(TypeError, hash, k) self.assertNotIsInstance(k, collections.abc.Hashable) + def test_cmp_to_signature(self): + self.assertEqual(str(Signature.from_callable(self.cmp_to_key)), + '(mycmp)') + @unittest.skipUnless(c_functools, 'requires the C _functools module') class TestCmpToKeyC(TestCmpToKey, unittest.TestCase): @@ -1853,6 +1858,13 @@ def test_staticmethod(x): for ref in refs: self.assertIsNone(ref()) + def test_common_signatures(self): + def orig(): ... + lru = self.module.lru_cache(1)(orig) + + self.assertEqual(str(Signature.from_callable(lru.cache_info)), '()') + self.assertEqual(str(Signature.from_callable(lru.cache_clear)), '()') + @py_functools.lru_cache() def py_cached_func(x, y): diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py index 710b609ce550d6..cfc6e411ea680d 100644 --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -1421,6 +1421,13 @@ def wrapper(a, b): self.assertEqual(inspect.get_annotations(isa.MyClassWithLocalAnnotations, eval_str=True), {'x': int}) +class TestFormatAnnotation(unittest.TestCase): + def test_typing_replacement(self): + from test.typinganndata.ann_module9 import ann, ann1 + self.assertEqual(inspect.formatannotation(ann), 'Union[List[str], int]') + self.assertEqual(inspect.formatannotation(ann1), 'Union[List[testModule.typing.A], int]') + + class TestIsDataDescriptor(unittest.TestCase): def test_custom_descriptors(self): @@ -3891,7 +3898,8 @@ def test(foo, *, bar): self.call(test, 1, bar=2, spam='ham') with self.assertRaisesRegex(TypeError, - "missing a required argument: 'bar'"): + "missing a required keyword-only " + "argument: 'bar'"): self.call(test, 1) def test(foo, *, bar, **bin): diff --git a/Lib/test/test_syslog.py b/Lib/test/test_syslog.py index fe09bd39f8b7fd..2125ec58d87e03 100644 --- a/Lib/test/test_syslog.py +++ b/Lib/test/test_syslog.py @@ -1,5 +1,9 @@ -from test.support import import_helper +from test.support import import_helper, threading_helper syslog = import_helper.import_module("syslog") #skip if not supported +from test import support +import sys +import threading +import time import unittest # XXX(nnorwitz): This test sucks. I don't know of a platform independent way @@ -8,6 +12,9 @@ class Test(unittest.TestCase): + def tearDown(self): + syslog.closelog() + def test_openlog(self): syslog.openlog('python') # Issue #6697. @@ -18,22 +25,59 @@ def test_syslog(self): syslog.syslog('test message from python test_syslog') syslog.syslog(syslog.LOG_ERR, 'test error from python test_syslog') + def test_syslog_implicit_open(self): + syslog.closelog() # Make sure log is closed + syslog.syslog('test message from python test_syslog') + syslog.syslog(syslog.LOG_ERR, 'test error from python test_syslog') + def test_closelog(self): syslog.openlog('python') syslog.closelog() + syslog.closelog() # idempotent operation def test_setlogmask(self): - syslog.setlogmask(syslog.LOG_DEBUG) + mask = syslog.LOG_UPTO(syslog.LOG_WARNING) + oldmask = syslog.setlogmask(mask) + self.assertEqual(syslog.setlogmask(0), mask) + self.assertEqual(syslog.setlogmask(oldmask), mask) def test_log_mask(self): - syslog.LOG_MASK(syslog.LOG_INFO) - - def test_log_upto(self): - syslog.LOG_UPTO(syslog.LOG_INFO) + mask = syslog.LOG_UPTO(syslog.LOG_WARNING) + self.assertTrue(mask & syslog.LOG_MASK(syslog.LOG_WARNING)) + self.assertTrue(mask & syslog.LOG_MASK(syslog.LOG_ERR)) + self.assertFalse(mask & syslog.LOG_MASK(syslog.LOG_INFO)) def test_openlog_noargs(self): syslog.openlog() syslog.syslog('test message from python test_syslog') + @threading_helper.requires_working_threading() + def test_syslog_threaded(self): + start = threading.Event() + stop = False + def opener(): + start.wait(10) + i = 1 + while not stop: + syslog.openlog(f'python-test-{i}') # new string object + i += 1 + def logger(): + start.wait(10) + while not stop: + syslog.syslog('test message from python test_syslog') + + orig_si = sys.getswitchinterval() + support.setswitchinterval(1e-9) + try: + threads = [threading.Thread(target=opener)] + threads += [threading.Thread(target=logger) for k in range(10)] + with threading_helper.start_threads(threads): + start.set() + time.sleep(0.1) + stop = True + finally: + sys.setswitchinterval(orig_si) + + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py index 30faaaf83bec96..5b816574f2c335 100644 --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -2807,6 +2807,25 @@ def check_format(expected, format, *args): check_format('repr=abc', b'repr=%V', 'abc', b'xyz') + # test %p + # We cannot test the exact result, + # because it returns a hex representation of a C pointer, + # which is going to be different each time. But, we can test the format. + p_format_regex = r'^0x[a-zA-Z0-9]{3,}$' + p_format1 = PyUnicode_FromFormat(b'%p', 'abc') + self.assertIsInstance(p_format1, str) + self.assertRegex(p_format1, p_format_regex) + + p_format2 = PyUnicode_FromFormat(b'%p %p', '123456', b'xyz') + self.assertIsInstance(p_format2, str) + self.assertRegex(p_format2, + r'0x[a-zA-Z0-9]{3,} 0x[a-zA-Z0-9]{3,}') + + # Extra args are ignored: + p_format3 = PyUnicode_FromFormat(b'%p', '123456', None, b'xyz') + self.assertIsInstance(p_format3, str) + self.assertRegex(p_format3, p_format_regex) + # Test string decode from parameter of %s using utf-8. # b'\xe4\xba\xba\xe6\xb0\x91' is utf-8 encoded byte sequence of # '\u4eba\u6c11' diff --git a/Lib/test/test_zoneinfo/test_zoneinfo.py b/Lib/test/test_zoneinfo/test_zoneinfo.py index a2172f3ac21d08..fd0e3bc032ec0c 100644 --- a/Lib/test/test_zoneinfo/test_zoneinfo.py +++ b/Lib/test/test_zoneinfo/test_zoneinfo.py @@ -404,6 +404,19 @@ def test_time_fixed_offset(self): class CZoneInfoTest(ZoneInfoTest): module = c_zoneinfo + def test_signatures(self): + """Ensure that C module has valid method signatures.""" + import inspect + + must_have_signatures = ( + self.klass.clear_cache, + self.klass.no_cache, + self.klass.from_file, + ) + for method in must_have_signatures: + with self.subTest(method=method): + inspect.Signature.from_callable(method) + def test_fold_mutate(self): """Test that fold isn't mutated when no change is necessary. diff --git a/Lib/test/typinganndata/__init__.py b/Lib/test/typinganndata/__init__.py new file mode 100644 index 00000000000000..e69de29bb2d1d6 diff --git a/Lib/test/typinganndata/ann_module9.py b/Lib/test/typinganndata/ann_module9.py new file mode 100644 index 00000000000000..952217393e1ff7 --- /dev/null +++ b/Lib/test/typinganndata/ann_module9.py @@ -0,0 +1,14 @@ +# Test ``inspect.formatannotation`` +# https://github.com/python/cpython/issues/96073 + +from typing import Union, List + +ann = Union[List[str], int] + +# mock typing._type_repr behaviour +class A: ... + +A.__module__ = 'testModule.typing' +A.__qualname__ = 'A' + +ann1 = Union[List[A], int] diff --git a/Lib/timeit.py b/Lib/timeit.py index 9dfd454936e6b8..0cf8db67723a49 100755 --- a/Lib/timeit.py +++ b/Lib/timeit.py @@ -259,10 +259,9 @@ def main(args=None, *, _wrap_timer=None): args = sys.argv[1:] import getopt try: - opts, args = getopt.getopt(args, "n:u:s:r:tcpvh", + opts, args = getopt.getopt(args, "n:u:s:r:pvh", ["number=", "setup=", "repeat=", - "time", "clock", "process", - "verbose", "unit=", "help"]) + "process", "verbose", "unit=", "help"]) except getopt.error as err: print(err) print("use -h/--help for command line help") diff --git a/Lib/types.py b/Lib/types.py index 2e73fbc4501337..f8353126cb527c 100644 --- a/Lib/types.py +++ b/Lib/types.py @@ -60,7 +60,7 @@ def _m(self): pass GetSetDescriptorType = type(FunctionType.__code__) MemberDescriptorType = type(FunctionType.__globals__) -del sys, _f, _g, _C, _c, _ag # Not for export +del sys, _f, _g, _C, _c, _ag, _cell_factory # Not for export # Provide a PEP 3115 compliant mechanism for class creation diff --git a/Makefile.pre.in b/Makefile.pre.in index 11118354f15def..4602db69d15af9 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -2038,6 +2038,7 @@ TESTSUBDIRS= distutils/tests \ test/test_zoneinfo test/test_zoneinfo/data \ test/test_unittest test/test_unittest/testmock \ test/tracedmodules \ + test/typinganndata \ test/xmltestdata test/xmltestdata/c14n-20 \ test/ziptestdata diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-09-08-20-58-10.gh-issue-64373.AfCi36.rst b/Misc/NEWS.d/next/Core and Builtins/2022-09-08-20-58-10.gh-issue-64373.AfCi36.rst new file mode 100644 index 00000000000000..e7e13cb3a00c33 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-09-08-20-58-10.gh-issue-64373.AfCi36.rst @@ -0,0 +1 @@ +Convert :mod:`_functools` to argument clinic. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-10-06-14-14-28.gh-issue-97955.Nq5VXD.rst b/Misc/NEWS.d/next/Core and Builtins/2022-10-06-14-14-28.gh-issue-97955.Nq5VXD.rst new file mode 100644 index 00000000000000..e21794df4f1840 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-10-06-14-14-28.gh-issue-97955.Nq5VXD.rst @@ -0,0 +1 @@ +Migrate :mod:`zoneinfo` to Argument Clinic. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-10-06-23-13-34.gh-issue-97997.JQaJKF.rst b/Misc/NEWS.d/next/Core and Builtins/2022-10-06-23-13-34.gh-issue-97997.JQaJKF.rst new file mode 100644 index 00000000000000..5cb5e2126638be --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-10-06-23-13-34.gh-issue-97997.JQaJKF.rst @@ -0,0 +1 @@ +Add running column offset to the tokenizer state to avoid calculating AST column information with pointer arithmetic. diff --git a/Misc/NEWS.d/next/Library/2022-07-17-22-31-32.gh-issue-90085.c4FWcS.rst b/Misc/NEWS.d/next/Library/2022-07-17-22-31-32.gh-issue-90085.c4FWcS.rst new file mode 100644 index 00000000000000..37952adc8f1a92 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-07-17-22-31-32.gh-issue-90085.c4FWcS.rst @@ -0,0 +1,3 @@ +Remove ``-c/--clock`` and ``-t/--time`` CLI options of :mod:`timeit`. +The options had been deprecated since Python 3.3 and the functionality +was removed in Python 3.7. Patch by Shantanu Jain. diff --git a/Misc/NEWS.d/next/Library/2022-07-27-19-47-51.gh-issue-83901.OSw06c.rst b/Misc/NEWS.d/next/Library/2022-07-27-19-47-51.gh-issue-83901.OSw06c.rst new file mode 100644 index 00000000000000..da407905a886fd --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-07-27-19-47-51.gh-issue-83901.OSw06c.rst @@ -0,0 +1 @@ +Improve :meth:`Signature.bind ` error message for missing keyword-only arguments. diff --git a/Misc/NEWS.d/next/Library/2022-08-27-23-16-09.gh-issue-96346.jJX14I.rst b/Misc/NEWS.d/next/Library/2022-08-27-23-16-09.gh-issue-96346.jJX14I.rst new file mode 100644 index 00000000000000..9883348b9c3e24 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-08-27-23-16-09.gh-issue-96346.jJX14I.rst @@ -0,0 +1 @@ +Use double caching for compiled RE patterns. diff --git a/Misc/NEWS.d/next/Library/2022-08-29-12-35-28.gh-issue-96073.WaGstf.rst b/Misc/NEWS.d/next/Library/2022-08-29-12-35-28.gh-issue-96073.WaGstf.rst new file mode 100644 index 00000000000000..8f20588c4c585e --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-08-29-12-35-28.gh-issue-96073.WaGstf.rst @@ -0,0 +1 @@ +In :mod:`inspect`, fix overeager replacement of "``typing.``" in formatting annotations. diff --git a/Misc/NEWS.d/next/Library/2022-08-30-12-32-00.gh-issue-96415.6W7ORH.rst b/Misc/NEWS.d/next/Library/2022-08-30-12-32-00.gh-issue-96415.6W7ORH.rst new file mode 100644 index 00000000000000..0c93abf6dfcf16 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-08-30-12-32-00.gh-issue-96415.6W7ORH.rst @@ -0,0 +1 @@ +Remove ``types._cell_factory`` from module namespace. diff --git a/Misc/NEWS.d/next/Library/2022-10-05-20-52-17.gh-issue-97646.Q4fVww.rst b/Misc/NEWS.d/next/Library/2022-10-05-20-52-17.gh-issue-97646.Q4fVww.rst new file mode 100644 index 00000000000000..6eed16c8e8bbfb --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-10-05-20-52-17.gh-issue-97646.Q4fVww.rst @@ -0,0 +1 @@ +Replace deprecated ``application/javascript`` with ``text/javascript`` in :mod:`mimetypes`. See :rfc:`9239`. Patch by Noam Cohen. diff --git a/Modules/_csv.c b/Modules/_csv.c index d34d0a1296ae72..25bf86f78e8fb9 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -704,7 +704,7 @@ parse_process_char(ReaderObj *self, _csvstate *module_state, Py_UCS4 c) self->state = ESCAPED_CHAR; } else if (c == ' ' && dialect->skipinitialspace) - /* ignore space at start of field */ + /* ignore spaces at start of field */ ; else if (c == dialect->delimiter) { /* save empty field */ @@ -1647,9 +1647,9 @@ PyDoc_STRVAR(csv_module_doc, " quoting character. It defaults to '\"'.\n" " * delimiter - specifies a one-character string to use as the\n" " field separator. It defaults to ','.\n" -" * skipinitialspace - specifies how to interpret whitespace which\n" -" immediately follows a delimiter. It defaults to False, which\n" -" means that whitespace immediately following a delimiter is part\n" +" * skipinitialspace - specifies how to interpret spaces which\n" +" immediately follow a delimiter. It defaults to False, which\n" +" means that spaces immediately following a delimiter is part\n" " of the following field.\n" " * lineterminator - specifies the character sequence which should\n" " terminate rows.\n" diff --git a/Modules/_functoolsmodule.c b/Modules/_functoolsmodule.c index 3abb7fd710aeb4..e4036e166921d9 100644 --- a/Modules/_functoolsmodule.c +++ b/Modules/_functoolsmodule.c @@ -7,6 +7,13 @@ #include "pycore_tuple.h" // _PyTuple_ITEMS() #include "structmember.h" // PyMemberDef +#include "clinic/_functoolsmodule.c.h" +/*[clinic input] +module _functools +class _functools._lru_cache_wrapper "PyObject *" "&lru_cache_type_spec" +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=bece4053896b09c0]*/ + /* _functools module written and maintained by Hye-Shik Chang with adaptations by Raymond Hettinger @@ -58,6 +65,7 @@ get_functools_state_by_type(PyTypeObject *type) return get_functools_state(module); } +// Not converted to argument clinic, because of `*args, **kwargs` arguments. static PyObject * partial_new(PyTypeObject *type, PyObject *args, PyObject *kw) { @@ -282,6 +290,7 @@ partial_setvectorcall(partialobject *pto) } +// Not converted to argument clinic, because of `*args, **kwargs` arguments. static PyObject * partial_call(partialobject *pto, PyObject *args, PyObject *kwargs) { @@ -625,33 +634,37 @@ keyobject_richcompare(PyObject *ko, PyObject *other, int op) return answer; } +/*[clinic input] +_functools.cmp_to_key + + mycmp: object + Function that compares two objects. + +Convert a cmp= function into a key= function. +[clinic start generated code]*/ + static PyObject * -functools_cmp_to_key(PyObject *self, PyObject *args, PyObject *kwds) +_functools_cmp_to_key_impl(PyObject *module, PyObject *mycmp) +/*[clinic end generated code: output=71eaad0f4fc81f33 input=d1b76f231c0dfeb3]*/ { - PyObject *cmp; - static char *kwargs[] = {"mycmp", NULL}; keyobject *object; _functools_state *state; - if (!PyArg_ParseTupleAndKeywords(args, kwds, "O:cmp_to_key", kwargs, &cmp)) - return NULL; - - state = get_functools_state(self); + state = get_functools_state(module); object = PyObject_GC_New(keyobject, state->keyobject_type); if (!object) return NULL; - Py_INCREF(cmp); - object->cmp = cmp; + Py_INCREF(mycmp); + object->cmp = mycmp; object->object = NULL; PyObject_GC_Track(object); return (PyObject *)object; } -PyDoc_STRVAR(functools_cmp_to_key_doc, -"Convert a cmp= function into a key= function."); - /* reduce (used to be a builtin) ********************************************/ +// Not converted to argument clinic, because of `args` in-place modification. +// AC will affect performance. static PyObject * functools_reduce(PyObject *self, PyObject *args) { @@ -1299,25 +1312,41 @@ lru_cache_descr_get(PyObject *self, PyObject *obj, PyObject *type) return PyMethod_New(self, obj); } +/*[clinic input] +_functools._lru_cache_wrapper.cache_info + +Report cache statistics +[clinic start generated code]*/ + static PyObject * -lru_cache_cache_info(lru_cache_object *self, PyObject *unused) +_functools__lru_cache_wrapper_cache_info_impl(PyObject *self) +/*[clinic end generated code: output=cc796a0b06dbd717 input=f05e5b6ebfe38645]*/ { - if (self->maxsize == -1) { - return PyObject_CallFunction(self->cache_info_type, "nnOn", - self->hits, self->misses, Py_None, - PyDict_GET_SIZE(self->cache)); - } - return PyObject_CallFunction(self->cache_info_type, "nnnn", - self->hits, self->misses, self->maxsize, - PyDict_GET_SIZE(self->cache)); + lru_cache_object *_self = (lru_cache_object *) self; + if (_self->maxsize == -1) { + return PyObject_CallFunction(_self->cache_info_type, "nnOn", + _self->hits, _self->misses, Py_None, + PyDict_GET_SIZE(_self->cache)); + } + return PyObject_CallFunction(_self->cache_info_type, "nnnn", + _self->hits, _self->misses, _self->maxsize, + PyDict_GET_SIZE(_self->cache)); } +/*[clinic input] +_functools._lru_cache_wrapper.cache_clear + +Clear the cache and cache statistics +[clinic start generated code]*/ + static PyObject * -lru_cache_cache_clear(lru_cache_object *self, PyObject *unused) +_functools__lru_cache_wrapper_cache_clear_impl(PyObject *self) +/*[clinic end generated code: output=58423b35efc3e381 input=6ca59dba09b12584]*/ { - lru_list_elem *list = lru_cache_unlink_list(self); - self->hits = self->misses = 0; - PyDict_Clear(self->cache); + lru_cache_object *_self = (lru_cache_object *) self; + lru_list_elem *list = lru_cache_unlink_list(_self); + _self->hits = _self->misses = 0; + PyDict_Clear(_self->cache); lru_cache_clear_list(list); Py_RETURN_NONE; } @@ -1381,8 +1410,8 @@ cache_info_type: namedtuple class with the fields:\n\ ); static PyMethodDef lru_cache_methods[] = { - {"cache_info", (PyCFunction)lru_cache_cache_info, METH_NOARGS}, - {"cache_clear", (PyCFunction)lru_cache_cache_clear, METH_NOARGS}, + _FUNCTOOLS__LRU_CACHE_WRAPPER_CACHE_INFO_METHODDEF + _FUNCTOOLS__LRU_CACHE_WRAPPER_CACHE_CLEAR_METHODDEF {"__reduce__", (PyCFunction)lru_cache_reduce, METH_NOARGS}, {"__copy__", (PyCFunction)lru_cache_copy, METH_VARARGS}, {"__deepcopy__", (PyCFunction)lru_cache_deepcopy, METH_VARARGS}, @@ -1432,8 +1461,7 @@ PyDoc_STRVAR(_functools_doc, static PyMethodDef _functools_methods[] = { {"reduce", functools_reduce, METH_VARARGS, functools_reduce_doc}, - {"cmp_to_key", _PyCFunction_CAST(functools_cmp_to_key), - METH_VARARGS | METH_KEYWORDS, functools_cmp_to_key_doc}, + _FUNCTOOLS_CMP_TO_KEY_METHODDEF {NULL, NULL} /* sentinel */ }; diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 923c66a7943fea..9515cc685ac892 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -5210,6 +5210,7 @@ dict_watch_callback(PyDict_WatchEvent event, Py_DECREF(msg); return -1; } + Py_DECREF(msg); return 0; } @@ -5224,8 +5225,10 @@ dict_watch_callback_second(PyDict_WatchEvent event, return -1; } if (PyList_Append(g_dict_watch_events, msg) < 0) { + Py_DECREF(msg); return -1; } + Py_DECREF(msg); return 0; } diff --git a/Modules/_zoneinfo.c b/Modules/_zoneinfo.c index 207340adec152f..36b25bf3c05469 100644 --- a/Modules/_zoneinfo.c +++ b/Modules/_zoneinfo.c @@ -12,6 +12,13 @@ #include "datetime.h" +#include "clinic/_zoneinfo.c.h" +/*[clinic input] +module zoneinfo +class zoneinfo.ZoneInfo "PyObject *" "PyTypeObject *" +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=d12c73c0eef36df8]*/ + // Imports static PyObject *io_open = NULL; static PyObject *_tzpath_find_tzfile = NULL; @@ -338,20 +345,25 @@ zoneinfo_dealloc(PyObject *obj_self) Py_TYPE(self)->tp_free((PyObject *)self); } +/*[clinic input] +@classmethod +zoneinfo.ZoneInfo.from_file + + file_obj: object + / + key: object = None + +Create a ZoneInfo file from a file object. +[clinic start generated code]*/ + static PyObject * -zoneinfo_from_file(PyTypeObject *type, PyObject *args, PyObject *kwargs) +zoneinfo_ZoneInfo_from_file_impl(PyTypeObject *type, PyObject *file_obj, + PyObject *key) +/*[clinic end generated code: output=68ed2022404ae5be input=ccfe73708133d2e4]*/ { - PyObject *file_obj = NULL; PyObject *file_repr = NULL; - PyObject *key = Py_None; PyZoneInfo_ZoneInfo *self = NULL; - static char *kwlist[] = {"", "key", NULL}; - if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O|O", kwlist, &file_obj, - &key)) { - return NULL; - } - PyObject *obj_self = (PyObject *)(type->tp_alloc(type, 0)); self = (PyZoneInfo_ZoneInfo *)obj_self; if (self == NULL) { @@ -379,16 +391,20 @@ zoneinfo_from_file(PyTypeObject *type, PyObject *args, PyObject *kwargs) return NULL; } +/*[clinic input] +@classmethod +zoneinfo.ZoneInfo.no_cache + + key: object + +Get a new instance of ZoneInfo, bypassing the cache. +[clinic start generated code]*/ + static PyObject * -zoneinfo_no_cache(PyTypeObject *cls, PyObject *args, PyObject *kwargs) +zoneinfo_ZoneInfo_no_cache_impl(PyTypeObject *type, PyObject *key) +/*[clinic end generated code: output=751c6894ad66f91b input=bb24afd84a80ba46]*/ { - static char *kwlist[] = {"key", NULL}; - PyObject *key = NULL; - if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O", kwlist, &key)) { - return NULL; - } - - PyObject *out = zoneinfo_new_instance(cls, key); + PyObject *out = zoneinfo_new_instance(type, key); if (out != NULL) { ((PyZoneInfo_ZoneInfo *)out)->source = SOURCE_NOCACHE; } @@ -396,18 +412,20 @@ zoneinfo_no_cache(PyTypeObject *cls, PyObject *args, PyObject *kwargs) return out; } -static PyObject * -zoneinfo_clear_cache(PyObject *cls, PyObject *args, PyObject *kwargs) -{ - PyObject *only_keys = NULL; - static char *kwlist[] = {"only_keys", NULL}; +/*[clinic input] +@classmethod +zoneinfo.ZoneInfo.clear_cache - if (!(PyArg_ParseTupleAndKeywords(args, kwargs, "|$O", kwlist, - &only_keys))) { - return NULL; - } + * + only_keys: object = None - PyTypeObject *type = (PyTypeObject *)cls; +Clear the ZoneInfo cache. +[clinic start generated code]*/ + +static PyObject * +zoneinfo_ZoneInfo_clear_cache_impl(PyTypeObject *type, PyObject *only_keys) +/*[clinic end generated code: output=eec0a3276f07bd90 input=8cff0182a95f295b]*/ +{ PyObject *weak_cache = get_weak_cache(type); if (only_keys == NULL || only_keys == Py_None) { @@ -2545,15 +2563,9 @@ zoneinfo_init_subclass(PyTypeObject *cls, PyObject *args, PyObject **kwargs) ///// // Specify the ZoneInfo type static PyMethodDef zoneinfo_methods[] = { - {"clear_cache", (PyCFunction)(void (*)(void))zoneinfo_clear_cache, - METH_VARARGS | METH_KEYWORDS | METH_CLASS, - PyDoc_STR("Clear the ZoneInfo cache.")}, - {"no_cache", (PyCFunction)(void (*)(void))zoneinfo_no_cache, - METH_VARARGS | METH_KEYWORDS | METH_CLASS, - PyDoc_STR("Get a new instance of ZoneInfo, bypassing the cache.")}, - {"from_file", (PyCFunction)(void (*)(void))zoneinfo_from_file, - METH_VARARGS | METH_KEYWORDS | METH_CLASS, - PyDoc_STR("Create a ZoneInfo file from a file object.")}, + ZONEINFO_ZONEINFO_CLEAR_CACHE_METHODDEF + ZONEINFO_ZONEINFO_NO_CACHE_METHODDEF + ZONEINFO_ZONEINFO_FROM_FILE_METHODDEF {"utcoffset", (PyCFunction)zoneinfo_utcoffset, METH_O, PyDoc_STR("Retrieve a timedelta representing the UTC offset in a zone at " "the given datetime.")}, diff --git a/Modules/clinic/_functoolsmodule.c.h b/Modules/clinic/_functoolsmodule.c.h new file mode 100644 index 00000000000000..9c79e6430413bf --- /dev/null +++ b/Modules/clinic/_functoolsmodule.c.h @@ -0,0 +1,104 @@ +/*[clinic input] +preserve +[clinic start generated code]*/ + +#if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE) +# include "pycore_gc.h" // PyGC_Head +# include "pycore_runtime.h" // _Py_ID() +#endif + + +PyDoc_STRVAR(_functools_cmp_to_key__doc__, +"cmp_to_key($module, /, mycmp)\n" +"--\n" +"\n" +"Convert a cmp= function into a key= function.\n" +"\n" +" mycmp\n" +" Function that compares two objects."); + +#define _FUNCTOOLS_CMP_TO_KEY_METHODDEF \ + {"cmp_to_key", _PyCFunction_CAST(_functools_cmp_to_key), METH_FASTCALL|METH_KEYWORDS, _functools_cmp_to_key__doc__}, + +static PyObject * +_functools_cmp_to_key_impl(PyObject *module, PyObject *mycmp); + +static PyObject * +_functools_cmp_to_key(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + #if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE) + + #define NUM_KEYWORDS 1 + static struct { + PyGC_Head _this_is_not_used; + PyObject_VAR_HEAD + PyObject *ob_item[NUM_KEYWORDS]; + } _kwtuple = { + .ob_base = PyVarObject_HEAD_INIT(&PyTuple_Type, NUM_KEYWORDS) + .ob_item = { &_Py_ID(mycmp), }, + }; + #undef NUM_KEYWORDS + #define KWTUPLE (&_kwtuple.ob_base.ob_base) + + #else // !Py_BUILD_CORE + # define KWTUPLE NULL + #endif // !Py_BUILD_CORE + + static const char * const _keywords[] = {"mycmp", NULL}; + static _PyArg_Parser _parser = { + .keywords = _keywords, + .fname = "cmp_to_key", + .kwtuple = KWTUPLE, + }; + #undef KWTUPLE + PyObject *argsbuf[1]; + PyObject *mycmp; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { + goto exit; + } + mycmp = args[0]; + return_value = _functools_cmp_to_key_impl(module, mycmp); + +exit: + return return_value; +} + +PyDoc_STRVAR(_functools__lru_cache_wrapper_cache_info__doc__, +"cache_info($self, /)\n" +"--\n" +"\n" +"Report cache statistics"); + +#define _FUNCTOOLS__LRU_CACHE_WRAPPER_CACHE_INFO_METHODDEF \ + {"cache_info", (PyCFunction)_functools__lru_cache_wrapper_cache_info, METH_NOARGS, _functools__lru_cache_wrapper_cache_info__doc__}, + +static PyObject * +_functools__lru_cache_wrapper_cache_info_impl(PyObject *self); + +static PyObject * +_functools__lru_cache_wrapper_cache_info(PyObject *self, PyObject *Py_UNUSED(ignored)) +{ + return _functools__lru_cache_wrapper_cache_info_impl(self); +} + +PyDoc_STRVAR(_functools__lru_cache_wrapper_cache_clear__doc__, +"cache_clear($self, /)\n" +"--\n" +"\n" +"Clear the cache and cache statistics"); + +#define _FUNCTOOLS__LRU_CACHE_WRAPPER_CACHE_CLEAR_METHODDEF \ + {"cache_clear", (PyCFunction)_functools__lru_cache_wrapper_cache_clear, METH_NOARGS, _functools__lru_cache_wrapper_cache_clear__doc__}, + +static PyObject * +_functools__lru_cache_wrapper_cache_clear_impl(PyObject *self); + +static PyObject * +_functools__lru_cache_wrapper_cache_clear(PyObject *self, PyObject *Py_UNUSED(ignored)) +{ + return _functools__lru_cache_wrapper_cache_clear_impl(self); +} +/*[clinic end generated code: output=7e7f3bcf9ed61f23 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/_zoneinfo.c.h b/Modules/clinic/_zoneinfo.c.h new file mode 100644 index 00000000000000..78fcbfa9411bb8 --- /dev/null +++ b/Modules/clinic/_zoneinfo.c.h @@ -0,0 +1,188 @@ +/*[clinic input] +preserve +[clinic start generated code]*/ + +#if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE) +# include "pycore_gc.h" // PyGC_Head +# include "pycore_runtime.h" // _Py_ID() +#endif + + +PyDoc_STRVAR(zoneinfo_ZoneInfo_from_file__doc__, +"from_file($type, file_obj, /, key=None)\n" +"--\n" +"\n" +"Create a ZoneInfo file from a file object."); + +#define ZONEINFO_ZONEINFO_FROM_FILE_METHODDEF \ + {"from_file", _PyCFunction_CAST(zoneinfo_ZoneInfo_from_file), METH_FASTCALL|METH_KEYWORDS|METH_CLASS, zoneinfo_ZoneInfo_from_file__doc__}, + +static PyObject * +zoneinfo_ZoneInfo_from_file_impl(PyTypeObject *type, PyObject *file_obj, + PyObject *key); + +static PyObject * +zoneinfo_ZoneInfo_from_file(PyTypeObject *type, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + #if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE) + + #define NUM_KEYWORDS 1 + static struct { + PyGC_Head _this_is_not_used; + PyObject_VAR_HEAD + PyObject *ob_item[NUM_KEYWORDS]; + } _kwtuple = { + .ob_base = PyVarObject_HEAD_INIT(&PyTuple_Type, NUM_KEYWORDS) + .ob_item = { &_Py_ID(key), }, + }; + #undef NUM_KEYWORDS + #define KWTUPLE (&_kwtuple.ob_base.ob_base) + + #else // !Py_BUILD_CORE + # define KWTUPLE NULL + #endif // !Py_BUILD_CORE + + static const char * const _keywords[] = {"", "key", NULL}; + static _PyArg_Parser _parser = { + .keywords = _keywords, + .fname = "from_file", + .kwtuple = KWTUPLE, + }; + #undef KWTUPLE + PyObject *argsbuf[2]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 1; + PyObject *file_obj; + PyObject *key = Py_None; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 2, 0, argsbuf); + if (!args) { + goto exit; + } + file_obj = args[0]; + if (!noptargs) { + goto skip_optional_pos; + } + key = args[1]; +skip_optional_pos: + return_value = zoneinfo_ZoneInfo_from_file_impl(type, file_obj, key); + +exit: + return return_value; +} + +PyDoc_STRVAR(zoneinfo_ZoneInfo_no_cache__doc__, +"no_cache($type, /, key)\n" +"--\n" +"\n" +"Get a new instance of ZoneInfo, bypassing the cache."); + +#define ZONEINFO_ZONEINFO_NO_CACHE_METHODDEF \ + {"no_cache", _PyCFunction_CAST(zoneinfo_ZoneInfo_no_cache), METH_FASTCALL|METH_KEYWORDS|METH_CLASS, zoneinfo_ZoneInfo_no_cache__doc__}, + +static PyObject * +zoneinfo_ZoneInfo_no_cache_impl(PyTypeObject *type, PyObject *key); + +static PyObject * +zoneinfo_ZoneInfo_no_cache(PyTypeObject *type, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + #if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE) + + #define NUM_KEYWORDS 1 + static struct { + PyGC_Head _this_is_not_used; + PyObject_VAR_HEAD + PyObject *ob_item[NUM_KEYWORDS]; + } _kwtuple = { + .ob_base = PyVarObject_HEAD_INIT(&PyTuple_Type, NUM_KEYWORDS) + .ob_item = { &_Py_ID(key), }, + }; + #undef NUM_KEYWORDS + #define KWTUPLE (&_kwtuple.ob_base.ob_base) + + #else // !Py_BUILD_CORE + # define KWTUPLE NULL + #endif // !Py_BUILD_CORE + + static const char * const _keywords[] = {"key", NULL}; + static _PyArg_Parser _parser = { + .keywords = _keywords, + .fname = "no_cache", + .kwtuple = KWTUPLE, + }; + #undef KWTUPLE + PyObject *argsbuf[1]; + PyObject *key; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { + goto exit; + } + key = args[0]; + return_value = zoneinfo_ZoneInfo_no_cache_impl(type, key); + +exit: + return return_value; +} + +PyDoc_STRVAR(zoneinfo_ZoneInfo_clear_cache__doc__, +"clear_cache($type, /, *, only_keys=None)\n" +"--\n" +"\n" +"Clear the ZoneInfo cache."); + +#define ZONEINFO_ZONEINFO_CLEAR_CACHE_METHODDEF \ + {"clear_cache", _PyCFunction_CAST(zoneinfo_ZoneInfo_clear_cache), METH_FASTCALL|METH_KEYWORDS|METH_CLASS, zoneinfo_ZoneInfo_clear_cache__doc__}, + +static PyObject * +zoneinfo_ZoneInfo_clear_cache_impl(PyTypeObject *type, PyObject *only_keys); + +static PyObject * +zoneinfo_ZoneInfo_clear_cache(PyTypeObject *type, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + #if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE) + + #define NUM_KEYWORDS 1 + static struct { + PyGC_Head _this_is_not_used; + PyObject_VAR_HEAD + PyObject *ob_item[NUM_KEYWORDS]; + } _kwtuple = { + .ob_base = PyVarObject_HEAD_INIT(&PyTuple_Type, NUM_KEYWORDS) + .ob_item = { &_Py_ID(only_keys), }, + }; + #undef NUM_KEYWORDS + #define KWTUPLE (&_kwtuple.ob_base.ob_base) + + #else // !Py_BUILD_CORE + # define KWTUPLE NULL + #endif // !Py_BUILD_CORE + + static const char * const _keywords[] = {"only_keys", NULL}; + static _PyArg_Parser _parser = { + .keywords = _keywords, + .fname = "clear_cache", + .kwtuple = KWTUPLE, + }; + #undef KWTUPLE + PyObject *argsbuf[1]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 0; + PyObject *only_keys = Py_None; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 0, 0, 0, argsbuf); + if (!args) { + goto exit; + } + if (!noptargs) { + goto skip_optional_kwonly; + } + only_keys = args[0]; +skip_optional_kwonly: + return_value = zoneinfo_ZoneInfo_clear_cache_impl(type, only_keys); + +exit: + return return_value; +} +/*[clinic end generated code: output=d2da73ef66146b83 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h index 7fac07804686b4..31bd01d1c3c1fb 100644 --- a/Modules/clinic/posixmodule.c.h +++ b/Modules/clinic/posixmodule.c.h @@ -1849,7 +1849,8 @@ PyDoc_STRVAR(os_mkdir__doc__, "dir_fd may not be implemented on your platform.\n" " If it is unavailable, using it will raise a NotImplementedError.\n" "\n" -"The mode argument is ignored on Windows."); +"The mode argument is ignored on Windows. Where it is used, the current umask\n" +"value is first masked out."); #define OS_MKDIR_METHODDEF \ {"mkdir", _PyCFunction_CAST(os_mkdir), METH_FASTCALL|METH_KEYWORDS, os_mkdir__doc__}, @@ -11367,4 +11368,4 @@ os_waitstatus_to_exitcode(PyObject *module, PyObject *const *args, Py_ssize_t na #ifndef OS_WAITSTATUS_TO_EXITCODE_METHODDEF #define OS_WAITSTATUS_TO_EXITCODE_METHODDEF #endif /* !defined(OS_WAITSTATUS_TO_EXITCODE_METHODDEF) */ -/*[clinic end generated code: output=dd43d388b442c96d input=a9049054013a1b77]*/ +/*[clinic end generated code: output=471ab8f2ad3d46b0 input=a9049054013a1b77]*/ diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index a72d57771c2294..1e556fc904b77b 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -4539,12 +4539,13 @@ If dir_fd is not None, it should be a file descriptor open to a directory, dir_fd may not be implemented on your platform. If it is unavailable, using it will raise a NotImplementedError. -The mode argument is ignored on Windows. +The mode argument is ignored on Windows. Where it is used, the current umask +value is first masked out. [clinic start generated code]*/ static PyObject * os_mkdir_impl(PyObject *module, path_t *path, int mode, int dir_fd) -/*[clinic end generated code: output=a70446903abe821f input=e965f68377e9b1ce]*/ +/*[clinic end generated code: output=a70446903abe821f input=a61722e1576fab03]*/ { int result; #ifdef HAVE_MKDIRAT diff --git a/Modules/syslogmodule.c b/Modules/syslogmodule.c index c409fe968f8894..1593eea94a62bf 100644 --- a/Modules/syslogmodule.c +++ b/Modules/syslogmodule.c @@ -235,7 +235,7 @@ syslog_setlogmask(PyObject *self, PyObject *args) if (!PyArg_ParseTuple(args, "l;mask for priority", &maskpri)) return NULL; - if (PySys_Audit("syslog.setlogmask", "(O)", args ? args : Py_None) < 0) { + if (PySys_Audit("syslog.setlogmask", "l", maskpri) < 0) { return NULL; } omaskpri = setlogmask(maskpri); diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index c5d3e580247cc1..1c356d3d47c945 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -37,6 +37,11 @@ #define TABSIZE 8 #define MAKE_TOKEN(token_type) token_setup(tok, token, token_type, p_start, p_end) +#define MAKE_TYPE_COMMENT_TOKEN(token_type, col_offset, end_col_offset) (\ + type_comment_token_setup(tok, token, token_type, col_offset, end_col_offset, p_start, p_end)) +#define ADVANCE_LINENO() \ + tok->lineno++; \ + tok->col_offset = 0; /* Forward */ static struct tok_state *tok_new(void); @@ -73,6 +78,8 @@ tok_new(void) tok->pendin = 0; tok->prompt = tok->nextprompt = NULL; tok->lineno = 0; + tok->starting_col_offset = -1; + tok->col_offset = -1; tok->level = 0; tok->altindstack[0] = 0; tok->decoding_state = STATE_INIT; @@ -871,7 +878,7 @@ tok_underflow_string(struct tok_state *tok) { tok->buf = tok->cur; } tok->line_start = tok->cur; - tok->lineno++; + ADVANCE_LINENO(); tok->inp = end; return 1; } @@ -930,7 +937,7 @@ tok_underflow_interactive(struct tok_state *tok) { else if (tok->start != NULL) { Py_ssize_t cur_multi_line_start = tok->multi_line_start - tok->buf; size_t size = strlen(newtok); - tok->lineno++; + ADVANCE_LINENO(); if (!tok_reserve_buf(tok, size + 1)) { PyMem_Free(tok->buf); tok->buf = NULL; @@ -943,7 +950,7 @@ tok_underflow_interactive(struct tok_state *tok) { tok->multi_line_start = tok->buf + cur_multi_line_start; } else { - tok->lineno++; + ADVANCE_LINENO(); PyMem_Free(tok->buf); tok->buf = newtok; tok->cur = tok->buf; @@ -998,7 +1005,7 @@ tok_underflow_file(struct tok_state *tok) { *tok->inp = '\0'; } - tok->lineno++; + ADVANCE_LINENO(); if (tok->decoding_state != STATE_NORMAL) { if (tok->lineno > 2) { tok->decoding_state = STATE_NORMAL; @@ -1056,6 +1063,7 @@ tok_nextc(struct tok_state *tok) int rc; for (;;) { if (tok->cur != tok->inp) { + tok->col_offset++; return Py_CHARMASK(*tok->cur++); /* Fast path */ } if (tok->done != E_OK) { @@ -1104,6 +1112,7 @@ tok_backup(struct tok_state *tok, int c) if ((int)(unsigned char)*tok->cur != c) { Py_FatalError("tok_backup: wrong character"); } + tok->col_offset--; } } @@ -1390,6 +1399,19 @@ tok_continuation_line(struct tok_state *tok) { return c; } +static int +type_comment_token_setup(struct tok_state *tok, struct token *token, int type, int col_offset, + int end_col_offset, const char *start, const char *end) +{ + token->level = tok->level; + token->lineno = token->end_lineno = tok->lineno; + token->col_offset = col_offset; + token->end_col_offset = end_col_offset; + token->start = start; + token->end = end; + return type; +} + static int token_setup(struct tok_state *tok, struct token *token, int type, const char *start, const char *end) { @@ -1397,14 +1419,13 @@ token_setup(struct tok_state *tok, struct token *token, int type, const char *st token->level = tok->level; token->lineno = type == STRING ? tok->first_lineno : tok->lineno; token->end_lineno = tok->lineno; - token->col_offset = -1; - token->end_col_offset = -1; + token->col_offset = token->end_col_offset = -1; token->start = start; token->end = end; + if (start != NULL && end != NULL) { - const char *line_start = type == STRING ? tok->multi_line_start : tok->line_start; - token->col_offset = (start >= line_start) ? (int)(start - line_start) : -1; - token->end_col_offset = (end >= tok->line_start) ? (int)(end - tok->line_start) : -1; + token->col_offset = tok->starting_col_offset; + token->end_col_offset = tok->col_offset; } return type; } @@ -1419,6 +1440,7 @@ tok_get(struct tok_state *tok, struct token *token) const char *p_end = NULL; nextline: tok->start = NULL; + tok->starting_col_offset = -1; blankline = 0; /* Get indentation level */ @@ -1518,6 +1540,7 @@ tok_get(struct tok_state *tok, struct token *token) } tok->start = tok->cur; + tok->starting_col_offset = tok->col_offset; /* Return pending indents/dedents */ if (tok->pendin != 0) { @@ -1565,10 +1588,12 @@ tok_get(struct tok_state *tok, struct token *token) /* Set start of current token */ tok->start = tok->cur == NULL ? NULL : tok->cur - 1; + tok->starting_col_offset = tok->col_offset - 1; /* Skip comment, unless it's a type comment */ if (c == '#') { const char *prefix, *p, *type_start; + int current_starting_col_offset; while (c != EOF && c != '\n') { c = tok_nextc(tok); @@ -1576,14 +1601,17 @@ tok_get(struct tok_state *tok, struct token *token) if (tok->type_comments) { p = tok->start; + current_starting_col_offset = tok->starting_col_offset; prefix = type_comment_prefix; while (*prefix && p < tok->cur) { if (*prefix == ' ') { while (*p == ' ' || *p == '\t') { p++; + current_starting_col_offset++; } } else if (*prefix == *p) { p++; + current_starting_col_offset++; } else { break; } @@ -1594,7 +1622,9 @@ tok_get(struct tok_state *tok, struct token *token) /* This is a type comment if we matched all of type_comment_prefix. */ if (!*prefix) { int is_type_ignore = 1; + // +6 in order to skip the word 'ignore' const char *ignore_end = p + 6; + const int ignore_end_col_offset = current_starting_col_offset + 6; tok_backup(tok, c); /* don't eat the newline or EOF */ type_start = p; @@ -1615,11 +1645,11 @@ tok_get(struct tok_state *tok, struct token *token) tok_nextc(tok); tok->atbol = 1; } - return MAKE_TOKEN(TYPE_IGNORE); + return MAKE_TYPE_COMMENT_TOKEN(TYPE_IGNORE, ignore_end_col_offset, tok->col_offset); } else { p_start = type_start; p_end = tok->cur; - return MAKE_TOKEN(TYPE_COMMENT); + return MAKE_TYPE_COMMENT_TOKEN(TYPE_COMMENT, current_starting_col_offset, tok->col_offset); } } } diff --git a/Parser/tokenizer.h b/Parser/tokenizer.h index 5b8c7f314386ec..2542d30e1da0ed 100644 --- a/Parser/tokenizer.h +++ b/Parser/tokenizer.h @@ -57,6 +57,8 @@ struct tok_state { int lineno; /* Current line number */ int first_lineno; /* First line of a single line or multi line string expression (cf. issue 16806) */ + int starting_col_offset; /* The column offset at the beginning of a token */ + int col_offset; /* Current col offset */ int level; /* () [] {} Parentheses nesting level */ /* Used to allow free continuations inside them */ char parenstack[MAXLEVEL]; diff --git a/Python/mysnprintf.c b/Python/mysnprintf.c index cd69198011e3c9..2a505d14f82c99 100644 --- a/Python/mysnprintf.c +++ b/Python/mysnprintf.c @@ -9,6 +9,7 @@ would have been written had the buffer not been too small, and to set the last byte of the buffer to \0. At least MS _vsnprintf returns a negative value instead, and fills the entire buffer with non-\0 data. + Unlike C99, our wrappers do not support passing a null buffer. The wrappers ensure that str[size-1] is always \0 upon return. diff --git a/Tools/clinic/clinic.py b/Tools/clinic/clinic.py index 30a676328706f6..a8687e3470a185 100755 --- a/Tools/clinic/clinic.py +++ b/Tools/clinic/clinic.py @@ -495,7 +495,8 @@ def permute_optional_groups(left, required, right): result = [] if not required: - assert not left + if left: + raise ValueError("required is empty but left is not") accumulator = [] counts = set()