From 267a3897b91bd70657d82165746a7c0e6adb3723 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Tue, 4 Jun 2019 21:55:17 +0200 Subject: [PATCH 0001/2163] Post v3.8.0b1 --- Include/patchlevel.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h index 881558bb44f612..4a52f833a52550 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 1 /* Version as a string */ -#define PY_VERSION "3.8.0b1" +#define PY_VERSION "3.8.0b1+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. From 99a5178cd143cc880f081dc5f53903fefa378681 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 4 Jun 2019 16:21:19 -0700 Subject: [PATCH 0002/2163] Doc: Python 3.9 in sidebar and version switcher. (GH-13824) (cherry picked from commit 59e7bbcaa4d0d556591f774c5ea4869c41fa95b0) Co-authored-by: Julien Palard --- Doc/tools/static/switchers.js | 3 ++- Doc/tools/templates/indexsidebar.html | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Doc/tools/static/switchers.js b/Doc/tools/static/switchers.js index 346b31494e60f9..fa298a76b0fe10 100644 --- a/Doc/tools/static/switchers.js +++ b/Doc/tools/static/switchers.js @@ -10,7 +10,8 @@ '(?:release/\\d.\\d[\\x\\d\\.]*)']; var all_versions = { - '3.8': 'dev (3.8)', + '3.9': 'dev (3.9)', + '3.8': 'pre (3.8)', '3.7': '3.7', '3.6': '3.6', '3.5': '3.5', diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html index 3666af92f0da47..4fd7423430ca81 100644 --- a/Doc/tools/templates/indexsidebar.html +++ b/Doc/tools/templates/indexsidebar.html @@ -2,7 +2,8 @@

{% trans %}Download{% endtrans %}

{% trans %}Download these documents{% endtrans %}

{% trans %}Docs by version{% endtrans %}

diff --git a/Lib/idlelib/help.py b/Lib/idlelib/help.py index e19d3615016f2f..ba2c6ec554812b 100644 --- a/Lib/idlelib/help.py +++ b/Lib/idlelib/help.py @@ -62,6 +62,7 @@ def __init__(self, text): self.simplelist = False # simple list (no double spacing) self.toc = [] # pair headers with text indexes for toc self.header = '' # text within header tags for toc + self.prevtag = None # info about previous tag (was opener, tag) def indent(self, amt=1): self.level += amt @@ -78,8 +79,11 @@ def handle_starttag(self, tag, attrs): self.show = True # start of main content elif tag == 'div' and class_ == 'sphinxsidebar': self.show = False # end of main content - elif tag == 'p' and class_ != 'first': - s = '\n\n' + elif tag == 'p' and self.prevtag and not self.prevtag[0]: + # begin a new block for

tags after a closed tag + # avoid extra lines, e.g. after

 tags
+            lastline = self.text.get('end-1c linestart', 'end-1c')
+            s = '\n\n' if lastline and not lastline.isspace() else '\n'
         elif tag == 'span' and class_ == 'pre':
             self.chartags = 'pre'
         elif tag == 'span' and class_ == 'versionmodified':
@@ -120,6 +124,7 @@ def handle_starttag(self, tag, attrs):
             self.tags = tag
         if self.show:
             self.text.insert('end', s, (self.tags, self.chartags))
+        self.prevtag = (True, tag)
 
     def handle_endtag(self, tag):
         "Handle endtags in help.html."
@@ -139,6 +144,7 @@ def handle_endtag(self, tag):
             self.tags = ''
         elif tag in ['ul', 'dd', 'ol']:
             self.indent(amt=-1)
+        self.prevtag = (False, tag)
 
     def handle_data(self, data):
         "Handle date segments in help.html."

From 4dd1c9d9c2bca4744c70c9556b7051f4465ede3e Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 3 Sep 2019 20:03:37 -0700
Subject: [PATCH 0485/2163] closes bpo-37966: Fully implement the UAX GH-15
 quick-check algorithm. (GH-15558)

The purpose of the `unicodedata.is_normalized` function is to answer
the question `str == unicodedata.normalized(form, str)` more
efficiently than writing just that, by using the "quick check"
optimization described in the Unicode standard in UAX GH-15.

However, it turns out the code doesn't implement the full algorithm
from the standard, and as a result we often miss the optimization and
end up having to compute the whole normalized string after all.

Implement the standard's algorithm.  This greatly speeds up
`unicodedata.is_normalized` in many cases where our partial variant
of quick-check had been returning MAYBE and the standard algorithm
returns NO.

At a quick test on my desktop, the existing code takes about 4.4 ms/MB
(so 4.4 ns per byte) when the partial quick-check returns MAYBE and it
has to do the slow normalize-and-compare:

  $ build.base/python -m timeit -s 'import unicodedata; s = "\uf900"*500000' \
      -- 'unicodedata.is_normalized("NFD", s)'
  50 loops, best of 5: 4.39 msec per loop

With this patch, it gets the answer instantly (58 ns) on the same 1 MB
string:

  $ build.dev/python -m timeit -s 'import unicodedata; s = "\uf900"*500000' \
      -- 'unicodedata.is_normalized("NFD", s)'
  5000000 loops, best of 5: 58.2 nsec per loop

This restores a small optimization that the original version of this
code had for the `unicodedata.normalize` use case.

With this, that case is actually faster than in master!

$ build.base/python -m timeit -s 'import unicodedata; s = "\u0338"*500000' \
    -- 'unicodedata.normalize("NFD", s)'
500 loops, best of 5: 561 usec per loop

$ build.dev/python -m timeit -s 'import unicodedata; s = "\u0338"*500000' \
    -- 'unicodedata.normalize("NFD", s)'
500 loops, best of 5: 512 usec per loop
(cherry picked from commit 2f09413947d1ce0043de62ed2346f9a2b4e5880b)

Co-authored-by: Greg Price 
---
 Doc/whatsnew/3.8.rst                          |  5 +-
 Lib/test/test_unicodedata.py                  |  2 +
 .../2019-08-27-21-21-36.bpo-37966.5OBLez.rst  |  3 +
 Modules/unicodedata.c                         | 75 +++++++++++++------
 4 files changed, 59 insertions(+), 26 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-08-27-21-21-36.bpo-37966.5OBLez.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index bcdb60d86d8553..4a1362d943c809 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -1090,8 +1090,9 @@ unicodedata
   `_ release.
 
 * New function :func:`~unicodedata.is_normalized` can be used to verify a string
-  is in a specific normal form. (Contributed by Max Belanger and David Euresti in
-  :issue:`32285`).
+  is in a specific normal form, often much faster than by actually normalizing
+  the string.  (Contributed by Max Belanger, David Euresti, and Greg Price in
+  :issue:`32285` and :issue:`37966`).
 
 
 unittest
diff --git a/Lib/test/test_unicodedata.py b/Lib/test/test_unicodedata.py
index a52b6de547fbc9..07d717688b0c59 100644
--- a/Lib/test/test_unicodedata.py
+++ b/Lib/test/test_unicodedata.py
@@ -220,6 +220,8 @@ def test_issue29456(self):
         self.assertEqual(self.db.normalize('NFC', u11a7_str_a), u11a7_str_b)
         self.assertEqual(self.db.normalize('NFC', u11c3_str_a), u11c3_str_b)
 
+    # For tests of unicodedata.is_normalized / self.db.is_normalized ,
+    # see test_normalization.py .
 
     def test_east_asian_width(self):
         eaw = self.db.east_asian_width
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-08-27-21-21-36.bpo-37966.5OBLez.rst b/Misc/NEWS.d/next/Core and Builtins/2019-08-27-21-21-36.bpo-37966.5OBLez.rst
new file mode 100644
index 00000000000000..6b9d69c5b3a9a4
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-08-27-21-21-36.bpo-37966.5OBLez.rst	
@@ -0,0 +1,3 @@
+The implementation of :func:`~unicodedata.is_normalized` has been greatly
+sped up on strings that aren't normalized, by implementing the full
+normalization-quick-check algorithm from the Unicode standard.
diff --git a/Modules/unicodedata.c b/Modules/unicodedata.c
index ae0d4e46f9a409..5e8ba602d66848 100644
--- a/Modules/unicodedata.c
+++ b/Modules/unicodedata.c
@@ -19,6 +19,8 @@
 #include "ucnhash.h"
 #include "structmember.h"
 
+#include 
+
 _Py_IDENTIFIER(NFC);
 _Py_IDENTIFIER(NFD);
 _Py_IDENTIFIER(NFKC);
@@ -775,25 +777,40 @@ nfc_nfkc(PyObject *self, PyObject *input, int k)
     return result;
 }
 
-typedef enum {YES, NO, MAYBE} NormalMode;
-
-/* Return YES if the input is certainly normalized, NO or MAYBE if it might not be. */
-static NormalMode
-is_normalized(PyObject *self, PyObject *input, int nfc, int k)
+// This needs to match the logic in makeunicodedata.py
+// which constructs the quickcheck data.
+typedef enum {YES = 0, MAYBE = 1, NO = 2} QuickcheckResult;
+
+/* Run the Unicode normalization "quickcheck" algorithm.
+ *
+ * Return YES or NO if quickcheck determines the input is certainly
+ * normalized or certainly not, and MAYBE if quickcheck is unable to
+ * tell.
+ *
+ * If `yes_only` is true, then return MAYBE as soon as we determine
+ * the answer is not YES.
+ *
+ * For background and details on the algorithm, see UAX #15:
+ *   https://www.unicode.org/reports/tr15/#Detecting_Normalization_Forms
+ */
+static QuickcheckResult
+is_normalized_quickcheck(PyObject *self, PyObject *input,
+                         int nfc, int k, bool yes_only)
 {
-    Py_ssize_t i, len;
-    int kind;
-    void *data;
-    unsigned char prev_combining = 0, quickcheck_mask;
-
     /* An older version of the database is requested, quickchecks must be
        disabled. */
     if (self && UCD_Check(self))
         return NO;
 
-    /* The two quickcheck bits at this shift mean 0=Yes, 1=Maybe, 2=No,
-       as described in http://unicode.org/reports/tr15/#Annex8. */
-    quickcheck_mask = 3 << ((nfc ? 4 : 0) + (k ? 2 : 0));
+    Py_ssize_t i, len;
+    int kind;
+    void *data;
+    unsigned char prev_combining = 0;
+
+    /* The two quickcheck bits at this shift have type QuickcheckResult. */
+    int quickcheck_shift = (nfc ? 4 : 0) + (k ? 2 : 0);
+
+    QuickcheckResult result = YES; /* certainly normalized, unless we find something */
 
     i = 0;
     kind = PyUnicode_KIND(input);
@@ -802,16 +819,26 @@ is_normalized(PyObject *self, PyObject *input, int nfc, int k)
     while (i < len) {
         Py_UCS4 ch = PyUnicode_READ(kind, data, i++);
         const _PyUnicode_DatabaseRecord *record = _getrecord_ex(ch);
-        unsigned char combining = record->combining;
-        unsigned char quickcheck = record->normalization_quick_check;
 
-        if (quickcheck & quickcheck_mask)
-            return MAYBE; /* this string might need normalization */
+        unsigned char combining = record->combining;
         if (combining && prev_combining > combining)
             return NO; /* non-canonical sort order, not normalized */
         prev_combining = combining;
+
+        unsigned char quickcheck_whole = record->normalization_quick_check;
+        if (yes_only) {
+            if (quickcheck_whole & (3 << quickcheck_shift))
+                return MAYBE;
+        } else {
+            switch ((quickcheck_whole >> quickcheck_shift) & 3) {
+            case NO:
+              return NO;
+            case MAYBE:
+              result = MAYBE; /* this string might need normalization */
+            }
+        }
     }
-    return YES; /* certainly normalized */
+    return result;
 }
 
 /*[clinic input]
@@ -844,7 +871,7 @@ unicodedata_UCD_is_normalized_impl(PyObject *self, PyObject *form,
     PyObject *result;
     int nfc = 0;
     int k = 0;
-    NormalMode m;
+    QuickcheckResult m;
 
     PyObject *cmp;
     int match = 0;
@@ -867,7 +894,7 @@ unicodedata_UCD_is_normalized_impl(PyObject *self, PyObject *form,
         return NULL;
     }
 
-    m = is_normalized(self, input, nfc, k);
+    m = is_normalized_quickcheck(self, input, nfc, k, false);
 
     if (m == MAYBE) {
         cmp = (nfc ? nfc_nfkc : nfd_nfkd)(self, input, k);
@@ -913,28 +940,28 @@ unicodedata_UCD_normalize_impl(PyObject *self, PyObject *form,
     }
 
     if (_PyUnicode_EqualToASCIIId(form, &PyId_NFC)) {
-        if (is_normalized(self, input, 1, 0) == YES) {
+        if (is_normalized_quickcheck(self, input, 1, 0, true) == YES) {
             Py_INCREF(input);
             return input;
         }
         return nfc_nfkc(self, input, 0);
     }
     if (_PyUnicode_EqualToASCIIId(form, &PyId_NFKC)) {
-        if (is_normalized(self, input, 1, 1) == YES) {
+        if (is_normalized_quickcheck(self, input, 1, 1, true) == YES) {
             Py_INCREF(input);
             return input;
         }
         return nfc_nfkc(self, input, 1);
     }
     if (_PyUnicode_EqualToASCIIId(form, &PyId_NFD)) {
-        if (is_normalized(self, input, 0, 0) == YES) {
+        if (is_normalized_quickcheck(self, input, 0, 0, true) == YES) {
             Py_INCREF(input);
             return input;
         }
         return nfd_nfkd(self, input, 0);
     }
     if (_PyUnicode_EqualToASCIIId(form, &PyId_NFKD)) {
-        if (is_normalized(self, input, 0, 1) == YES) {
+        if (is_normalized_quickcheck(self, input, 0, 1, true) == YES) {
             Py_INCREF(input);
             return input;
         }

From 5e194f57c08898cb8da457c6584402a2be2c828d Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 3 Sep 2019 23:10:45 -0700
Subject: [PATCH 0486/2163] Fix grammar in asyncio-dev.rst (GH-15672)

Automerge-Triggered-By: @ned-deily
(cherry picked from commit 675d17cec49ed7f7eb01d1bc3ae6999c728e070d)

Co-authored-by: Roger Iyengar 
---
 Doc/library/asyncio-dev.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst
index b7288036192979..101e7817a95e98 100644
--- a/Doc/library/asyncio-dev.rst
+++ b/Doc/library/asyncio-dev.rst
@@ -119,7 +119,7 @@ all concurrent asyncio Tasks and IO operations would be delayed
 by 1 second.
 
 An executor can be used to run a task in a different thread or even in
-a different process to avoid blocking block the OS thread with the
+a different process to avoid blocking the OS thread with the
 event loop.  See the :meth:`loop.run_in_executor` method for more
 details.
 

From cad7abf8abe657b696b9c8deb4b727e0cefaf36d Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 4 Sep 2019 15:18:05 -0700
Subject: [PATCH 0487/2163] bpo-38030: Fix os.stat failures on block devices on
 Windows (GH-15681)

(cherry picked from commit 772ec0fad57412daa53d16d7019b6b2fe6e94942)

Co-authored-by: Steve Dower 
---
 Lib/test/test_os.py                           |  8 ++++++++
 .../2019-09-04-14-01-08.bpo-38030._USdtk.rst  |  1 +
 Modules/posixmodule.c                         | 19 +++++++++++++------
 3 files changed, 22 insertions(+), 6 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Windows/2019-09-04-14-01-08.bpo-38030._USdtk.rst

diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
index 440cd6c1cf73c3..8ff0296fad0bb4 100644
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -591,6 +591,14 @@ def test_access_denied(self):
         result = os.stat(fname)
         self.assertNotEqual(result.st_size, 0)
 
+    @unittest.skipUnless(sys.platform == "win32", "Win32 specific tests")
+    def test_stat_block_device(self):
+        # bpo-38030: os.stat fails for block devices
+        # Test a filename like "//./C:"
+        fname = "//./" + os.path.splitdrive(os.getcwd())[0]
+        result = os.stat(fname)
+        self.assertEqual(result.st_mode, stat.S_IFBLK)
+
 
 class UtimeTests(unittest.TestCase):
     def setUp(self):
diff --git a/Misc/NEWS.d/next/Windows/2019-09-04-14-01-08.bpo-38030._USdtk.rst b/Misc/NEWS.d/next/Windows/2019-09-04-14-01-08.bpo-38030._USdtk.rst
new file mode 100644
index 00000000000000..f1be8a1e1c86f0
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-09-04-14-01-08.bpo-38030._USdtk.rst
@@ -0,0 +1 @@
+Fixes :func:`os.stat` failing for block devices on Windows
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index b9e8c0d94c20e1..771c5616152cfd 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -1793,13 +1793,13 @@ win32_xstat_impl(const wchar_t *path, struct _Py_stat_struct *result,
             case ERROR_INVALID_PARAMETER:
             case ERROR_INVALID_FUNCTION:
             case ERROR_NOT_SUPPORTED:
-                retval = -1;
+                /* Volumes and physical disks are block devices, e.g.
+                   \\.\C: and \\.\PhysicalDrive0. */
+                memset(result, 0, sizeof(*result));
+                result->st_mode = 0x6000; /* S_IFBLK */
                 goto cleanup;
             }
-            /* Volumes and physical disks are block devices, e.g.
-               \\.\C: and \\.\PhysicalDrive0. */
-            memset(result, 0, sizeof(*result));
-            result->st_mode = 0x6000; /* S_IFBLK */
+            retval = -1;
             goto cleanup;
         }
     }
@@ -1826,7 +1826,14 @@ win32_xstat_impl(const wchar_t *path, struct _Py_stat_struct *result,
 
 cleanup:
     if (hFile != INVALID_HANDLE_VALUE) {
-        CloseHandle(hFile);
+        /* Preserve last error if we are failing */
+        error = retval ? GetLastError() : 0;
+        if (!CloseHandle(hFile)) {
+            retval = -1;
+        } else if (retval) {
+            /* Restore last error */
+            SetLastError(error);
+        }
     }
 
     return retval;

From 29825a33926db63bb73601ba9e76ecd3c6cfa0f6 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 4 Sep 2019 17:39:34 -0700
Subject: [PATCH 0488/2163] Fix idlelib.help comments (GH-15669)

(cherry picked from commit 6cd9666ce93658ae91f07b396aa6932b362a61d3)

Co-authored-by: Terry Jan Reedy 
---
 Lib/idlelib/help.py | 49 +++++++++++++++++++++++----------------------
 1 file changed, 25 insertions(+), 24 deletions(-)

diff --git a/Lib/idlelib/help.py b/Lib/idlelib/help.py
index ba2c6ec554812b..9f63ea0d3990e6 100644
--- a/Lib/idlelib/help.py
+++ b/Lib/idlelib/help.py
@@ -50,21 +50,22 @@ class HelpParser(HTMLParser):
     """
     def __init__(self, text):
         HTMLParser.__init__(self, convert_charrefs=True)
-        self.text = text         # text widget we're rendering into
-        self.tags = ''           # current block level text tags to apply
-        self.chartags = ''       # current character level text tags
-        self.show = False        # used so we exclude page navigation
-        self.hdrlink = False     # used so we don't show header links
-        self.level = 0           # indentation level
-        self.pre = False         # displaying preformatted text
-        self.hprefix = ''        # prefix such as '25.5' to strip from headings
-        self.nested_dl = False   # if we're in a nested 
- self.simplelist = False # simple list (no double spacing) - self.toc = [] # pair headers with text indexes for toc - self.header = '' # text within header tags for toc - self.prevtag = None # info about previous tag (was opener, tag) + self.text = text # Text widget we're rendering into. + self.tags = '' # Current block level text tags to apply. + self.chartags = '' # Current character level text tags. + self.show = False # Exclude html page navigation. + self.hdrlink = False # Exclude html header links. + self.level = 0 # Track indentation level. + self.pre = False # Displaying preformatted text? + self.hprefix = '' # Heading prefix (like '25.5'?) to remove. + self.nested_dl = False # In a nested
? + self.simplelist = False # In a simple list (no double spacing)? + self.toc = [] # Pair headers with text indexes for toc. + self.header = '' # Text within header tags for toc. + self.prevtag = None # Previous tag info (opener?, tag). def indent(self, amt=1): + "Change indent (+1, 0, -1) and tags." self.level += amt self.tags = '' if self.level == 0 else 'l'+str(self.level) @@ -76,12 +77,12 @@ def handle_starttag(self, tag, attrs): class_ = v s = '' if tag == 'div' and class_ == 'section': - self.show = True # start of main content + self.show = True # Start main content. elif tag == 'div' and class_ == 'sphinxsidebar': - self.show = False # end of main content + self.show = False # End main content. elif tag == 'p' and self.prevtag and not self.prevtag[0]: - # begin a new block for

tags after a closed tag - # avoid extra lines, e.g. after

 tags
+            # Begin a new block for 

tags after a closed tag. + # Avoid extra lines, e.g. after

 tags.
             lastline = self.text.get('end-1c linestart', 'end-1c')
             s = '\n\n' if lastline and not lastline.isspace() else '\n'
         elif tag == 'span' and class_ == 'pre':
@@ -103,7 +104,7 @@ def handle_starttag(self, tag, attrs):
         elif tag == 'li':
             s = '\n* ' if self.simplelist else '\n\n* '
         elif tag == 'dt':
-            s = '\n\n' if not self.nested_dl else '\n'  # avoid extra line
+            s = '\n\n' if not self.nested_dl else '\n'  # Avoid extra line.
             self.nested_dl = False
         elif tag == 'dd':
             self.indent()
@@ -129,12 +130,13 @@ def handle_starttag(self, tag, attrs):
     def handle_endtag(self, tag):
         "Handle endtags in help.html."
         if tag in ['h1', 'h2', 'h3']:
-            self.indent(0)  # clear tag, reset indent
+            assert self.level == 0
             if self.show:
                 indent = ('        ' if tag == 'h3' else
                           '    ' if tag == 'h2' else
                           '')
                 self.toc.append((indent+self.header, self.text.index('insert')))
+            self.tags = ''
         elif tag in ['span', 'em']:
             self.chartags = ''
         elif tag == 'a':
@@ -143,7 +145,7 @@ def handle_endtag(self, tag):
             self.pre = False
             self.tags = ''
         elif tag in ['ul', 'dd', 'ol']:
-            self.indent(amt=-1)
+            self.indent(-1)
         self.prevtag = (False, tag)
 
     def handle_data(self, data):
@@ -169,7 +171,7 @@ def __init__(self, parent, filename):
         "Configure tags and feed file to parser."
         uwide = idleConf.GetOption('main', 'EditorWindow', 'width', type='int')
         uhigh = idleConf.GetOption('main', 'EditorWindow', 'height', type='int')
-        uhigh = 3 * uhigh // 4  # lines average 4/3 of editor line height
+        uhigh = 3 * uhigh // 4  # Lines average 4/3 of editor line height.
         Text.__init__(self, parent, wrap='word', highlightthickness=0,
                       padx=5, borderwidth=0, width=uwide, height=uhigh)
 
@@ -209,7 +211,6 @@ class HelpFrame(Frame):
     "Display html text, scrollbar, and toc."
     def __init__(self, parent, filename):
         Frame.__init__(self, parent)
-        # keep references to widgets for test access.
         self.text = text = HelpText(self, filename)
         self['background'] = text['background']
         self.toc = toc = self.toc_menu(text)
@@ -217,7 +218,7 @@ def __init__(self, parent, filename):
         text['yscrollcommand'] = scroll.set
 
         self.rowconfigure(0, weight=1)
-        self.columnconfigure(1, weight=1)  # text
+        self.columnconfigure(1, weight=1)  # Only expand the text widget.
         toc.grid(row=0, column=0, sticky='nw')
         text.grid(row=0, column=1, sticky='nsew')
         scroll.grid(row=0, column=2, sticky='ns')
@@ -279,7 +280,7 @@ def show_idlehelp(parent):
     "Create HelpWindow; called from Idle Help event handler."
     filename = join(abspath(dirname(__file__)), 'help.html')
     if not isfile(filename):
-        # try copy_strip, present message
+        # Try copy_strip, present message.
         return
     HelpWindow(parent, filename, 'IDLE Help (%s)' % python_version())
 

From 6d7a786d2e4b48a6b50614e042ace9ff996f0238 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 4 Sep 2019 17:54:59 -0700
Subject: [PATCH 0489/2163] bpo-22347: Update mimetypes.guess_type to allow
 proper parsing of URLs (GH-15522)

https://bugs.python.org/issue22347
(cherry picked from commit 87bd2071c756188b6cd577889fb1682831142ceb)

Co-authored-by: Dong-hee Na 
---
 Lib/mimetypes.py                                          | 3 ++-
 Lib/test/test_mimetypes.py                                | 8 ++++++++
 Lib/test/test_urllib2.py                                  | 2 +-
 .../next/Library/2019-08-27-01-03-26.bpo-22347._TRpYr.rst | 2 ++
 4 files changed, 13 insertions(+), 2 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2019-08-27-01-03-26.bpo-22347._TRpYr.rst

diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py
index 01a16fdf9aa1b3..f38005c9d29598 100644
--- a/Lib/mimetypes.py
+++ b/Lib/mimetypes.py
@@ -114,7 +114,8 @@ def guess_type(self, url, strict=True):
         but non-standard types.
         """
         url = os.fspath(url)
-        scheme, url = urllib.parse._splittype(url)
+        p = urllib.parse.urlparse(url)
+        scheme, url = p.scheme, p.path
         if scheme == 'data':
             # syntax of data URLs:
             # dataurl   := "data:" [ mediatype ] [ ";base64" ] "," data
diff --git a/Lib/test/test_mimetypes.py b/Lib/test/test_mimetypes.py
index bfd5eeedaa77b4..7761c3fe867a7e 100644
--- a/Lib/test/test_mimetypes.py
+++ b/Lib/test/test_mimetypes.py
@@ -51,6 +51,14 @@ def test_non_standard_types(self):
         eq(self.db.guess_type('foo.xul', strict=False), ('text/xul', None))
         eq(self.db.guess_extension('image/jpg', strict=False), '.jpg')
 
+    def test_url(self):
+        result = self.db.guess_type('http://host.html')
+        msg = 'URL only has a host name, not a file'
+        self.assertSequenceEqual(result, (None, None), msg)
+        result = self.db.guess_type('http://example.com/host.html')
+        msg = 'Should be text/html'
+        self.assertSequenceEqual(result, ('text/html', None), msg)
+
     def test_guess_all_types(self):
         eq = self.assertEqual
         unless = self.assertTrue
diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py
index c6d275e16b097f..810ee5e6ea28bf 100644
--- a/Lib/test/test_urllib2.py
+++ b/Lib/test/test_urllib2.py
@@ -744,7 +744,7 @@ def connect_ftp(self, user, passwd, host, port, dirs,
              ["foo", "bar"], "", None),
             ("ftp://localhost/baz.gif;type=a",
              "localhost", ftplib.FTP_PORT, "", "", "A",
-             [], "baz.gif", None),  # XXX really this should guess image/gif
+             [], "baz.gif", "image/gif"),
             ]:
             req = Request(url)
             req.timeout = None
diff --git a/Misc/NEWS.d/next/Library/2019-08-27-01-03-26.bpo-22347._TRpYr.rst b/Misc/NEWS.d/next/Library/2019-08-27-01-03-26.bpo-22347._TRpYr.rst
new file mode 100644
index 00000000000000..1a3c19938217c4
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-08-27-01-03-26.bpo-22347._TRpYr.rst
@@ -0,0 +1,2 @@
+Update mimetypes.guess_type to allow proper parsing of URLs with only a host name.
+Patch by Dong-hee Na.

From 6ad0a2c45f78020f7994e47620c1cf7b225f8197 Mon Sep 17 00:00:00 2001
From: Abhilash Raj 
Date: Wed, 4 Sep 2019 18:20:40 -0700
Subject: [PATCH 0490/2163] [3.8] bpo-37764: Fix infinite loop when parsing
 unstructured email headers. (GH-15239) (GH-15686)

Fixes a case in which email._header_value_parser.get_unstructured hangs the system for some invalid headers. This covers the cases in which the header contains either:
- a case without trailing whitespace
- an invalid encoded word

https://bugs.python.org/issue37764

This fix should also be backported to 3.7 and 3.8

https://bugs.python.org/issue37764
(cherry picked from commit c5b242f87f31286ad38991bc3868cf4cfbf2b681)

Co-authored-by: Ashwin Ramaswami 
---
 Lib/email/_header_value_parser.py             | 19 ++++++++++++++---
 .../test_email/test__header_value_parser.py   | 16 ++++++++++++++
 Lib/test/test_email/test_email.py             | 21 +++++++++++++++++++
 Misc/ACKS                                     |  1 +
 .../2019-08-27-01-13-05.bpo-37764.qv67PQ.rst  |  1 +
 5 files changed, 55 insertions(+), 3 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Security/2019-08-27-01-13-05.bpo-37764.qv67PQ.rst

diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py
index b5003943ab0d97..16c19907d68d59 100644
--- a/Lib/email/_header_value_parser.py
+++ b/Lib/email/_header_value_parser.py
@@ -935,6 +935,10 @@ def __str__(self):
         return ''
 
 
+class _InvalidEwError(errors.HeaderParseError):
+    """Invalid encoded word found while parsing headers."""
+
+
 # XXX these need to become classes and used as instances so
 # that a program can't change them in a parse tree and screw
 # up other parse trees.  Maybe should have  tests for that, too.
@@ -1039,7 +1043,10 @@ def get_encoded_word(value):
         raise errors.HeaderParseError(
             "expected encoded word but found {}".format(value))
     remstr = ''.join(remainder)
-    if len(remstr) > 1 and remstr[0] in hexdigits and remstr[1] in hexdigits:
+    if (len(remstr) > 1 and
+        remstr[0] in hexdigits and
+        remstr[1] in hexdigits and
+        tok.count('?') < 2):
         # The ? after the CTE was followed by an encoded word escape (=XX).
         rest, *remainder = remstr.split('?=', 1)
         tok = tok + '?=' + rest
@@ -1051,7 +1058,7 @@ def get_encoded_word(value):
     try:
         text, charset, lang, defects = _ew.decode('=?' + tok + '?=')
     except ValueError:
-        raise errors.HeaderParseError(
+        raise _InvalidEwError(
             "encoded word format invalid: '{}'".format(ew.cte))
     ew.charset = charset
     ew.lang = lang
@@ -1101,9 +1108,12 @@ def get_unstructured(value):
             token, value = get_fws(value)
             unstructured.append(token)
             continue
+        valid_ew = True
         if value.startswith('=?'):
             try:
                 token, value = get_encoded_word(value)
+            except _InvalidEwError:
+                valid_ew = False
             except errors.HeaderParseError:
                 # XXX: Need to figure out how to register defects when
                 # appropriate here.
@@ -1125,7 +1135,10 @@ def get_unstructured(value):
         # Split in the middle of an atom if there is a rfc2047 encoded word
         # which does not have WSP on both sides. The defect will be registered
         # the next time through the loop.
-        if rfc2047_matcher.search(tok):
+        # This needs to only be performed when the encoded word is valid;
+        # otherwise, performing it on an invalid encoded word can cause
+        # the parser to go in an infinite loop.
+        if valid_ew and rfc2047_matcher.search(tok):
             tok, *remainder = value.partition('=?')
         vtext = ValueTerminal(tok, 'vtext')
         _validate_xtext(vtext)
diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py
index bad4333dbc4351..dd33b065c804bc 100644
--- a/Lib/test/test_email/test__header_value_parser.py
+++ b/Lib/test/test_email/test__header_value_parser.py
@@ -383,6 +383,22 @@ def test_get_unstructured_ew_without_trailing_whitespace(self):
             [errors.InvalidHeaderDefect],
             '')
 
+    def test_get_unstructured_without_trailing_whitespace_hang_case(self):
+        self._test_get_x(self._get_unst,
+            '=?utf-8?q?somevalue?=aa',
+            'somevalueaa',
+            'somevalueaa',
+            [errors.InvalidHeaderDefect],
+            '')
+
+    def test_get_unstructured_invalid_ew(self):
+        self._test_get_x(self._get_unst,
+            '=?utf-8?q?=somevalue?=',
+            '=?utf-8?q?=somevalue?=',
+            '=?utf-8?q?=somevalue?=',
+            [],
+            '')
+
     # get_qp_ctext
 
     def test_get_qp_ctext_only(self):
diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py
index aa775881c5521a..5414cf070cc12f 100644
--- a/Lib/test/test_email/test_email.py
+++ b/Lib/test/test_email/test_email.py
@@ -5381,6 +5381,27 @@ def test_rfc2231_unencoded_then_encoded_segments(self):
         eq(language, 'en-us')
         eq(s, 'My Document For You')
 
+    def test_should_not_hang_on_invalid_ew_messages(self):
+        messages = ["""From: user@host.com
+To: user@host.com
+Bad-Header:
+ =?us-ascii?Q?LCSwrV11+IB0rSbSker+M9vWR7wEDSuGqmHD89Gt=ea0nJFSaiz4vX3XMJPT4vrE?=
+ =?us-ascii?Q?xGUZeOnp0o22pLBB7CYLH74Js=wOlK6Tfru2U47qR?=
+ =?us-ascii?Q?72OfyEY2p2=2FrA9xNFyvH+fBTCmazxwzF8nGkK6D?=
+
+Hello!
+""", """From: ����� �������� 
+To: "xxx" 
+Subject:   ��� ���������� ����� ����� � ��������� �� ����
+MIME-Version: 1.0
+Content-Type: text/plain; charset="windows-1251";
+Content-Transfer-Encoding: 8bit
+
+�� ����� � ���� ������ ��� ��������
+"""]
+        for m in messages:
+            with self.subTest(m=m):
+                msg = email.message_from_string(m)
 
 
 # Tests to ensure that signed parts of an email are completely preserved, as
diff --git a/Misc/ACKS b/Misc/ACKS
index 24e327a5f86e99..def874b0071e6f 100644
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -1330,6 +1330,7 @@ Burton Radons
 Abhilash Raj
 Shorya Raj
 Dhushyanth Ramasamy
+Ashwin Ramaswami
 Jeff Ramnani
 Bayard Randel
 Varpu Rantala
diff --git a/Misc/NEWS.d/next/Security/2019-08-27-01-13-05.bpo-37764.qv67PQ.rst b/Misc/NEWS.d/next/Security/2019-08-27-01-13-05.bpo-37764.qv67PQ.rst
new file mode 100644
index 00000000000000..27fa8e192f0c07
--- /dev/null
+++ b/Misc/NEWS.d/next/Security/2019-08-27-01-13-05.bpo-37764.qv67PQ.rst
@@ -0,0 +1 @@
+Fixes email._header_value_parser.get_unstructured going into an infinite loop for a specific case in which the email header does not have trailing whitespace, and the case in which it contains an invalid encoded word. Patch by Ashwin Ramaswami.
\ No newline at end of file

From 9c2654d1aa85968fede1b888fba86aebc06c5be6 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 4 Sep 2019 18:53:47 -0700
Subject: [PATCH 0491/2163] bpo-37902: IDLE: Add scrolling for IDLE browsers.
 (GH-15368)

Modify the wheel event handler so it can also be used for module, path, and stack browsers.
Patch by George Zhang.
(cherry picked from commit 2cd902585815582eb059e3b40e014ebe4e7fdee7)

Co-authored-by: GeeTransit 
---
 Lib/idlelib/NEWS.txt                          |  3 ++
 Lib/idlelib/editor.py                         | 25 +++------------
 Lib/idlelib/idle_test/test_multicall.py       |  8 +++++
 Lib/idlelib/idle_test/test_tree.py            | 29 ++++++++++++++++-
 Lib/idlelib/tree.py                           | 31 +++++++++++++++++++
 Misc/ACKS                                     |  1 +
 .../2019-08-21-16-02-49.bpo-37902._R_adE.rst  |  2 ++
 7 files changed, 78 insertions(+), 21 deletions(-)
 create mode 100644 Misc/NEWS.d/next/IDLE/2019-08-21-16-02-49.bpo-37902._R_adE.rst

diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt
index 47c2291d237727..c9e846a6fba667 100644
--- a/Lib/idlelib/NEWS.txt
+++ b/Lib/idlelib/NEWS.txt
@@ -3,6 +3,9 @@ Released on 2019-10-20?
 ======================================
 
 
+bpo-37092: Add mousewheel scrolling for IDLE module, path, and stack
+browsers.  Patch by George Zhang.
+
 bpo-35771: To avoid occasional spurious test_idle failures on slower
 machines, increase the ``hover_delay`` in test_tooltip.
 
diff --git a/Lib/idlelib/editor.py b/Lib/idlelib/editor.py
index 793ed3afaed0ff..5cbf704ab27b6a 100644
--- a/Lib/idlelib/editor.py
+++ b/Lib/idlelib/editor.py
@@ -26,6 +26,7 @@
 from idlelib import query
 from idlelib import replace
 from idlelib import search
+from idlelib.tree import wheel_event
 from idlelib import window
 
 # The default tab setting for a Text widget, in average-width characters.
@@ -151,9 +152,10 @@ def __init__(self, flist=None, filename=None, key=None, root=None):
         else:
             # Elsewhere, use right-click for popup menus.
             text.bind("<3>",self.right_menu_event)
-        text.bind('', self.mousescroll)
-        text.bind('', self.mousescroll)
-        text.bind('', self.mousescroll)
+
+        text.bind('', wheel_event)
+        text.bind('', wheel_event)
+        text.bind('', wheel_event)
         text.bind('', self.handle_winconfig)
         text.bind("<>", self.cut)
         text.bind("<>", self.copy)
@@ -502,23 +504,6 @@ def handle_yview(self, event, *args):
         self.text.yview(event, *args)
         return 'break'
 
-    def mousescroll(self, event):
-        """Handle scrollwheel event.
-
-        For wheel up, event.delta = 120*n on Windows, -1*n on darwin,
-        where n can be > 1 if one scrolls fast.  Flicking the wheel
-        generates up to maybe 20 events with n up to 10 or more 1.
-        Macs use wheel down (delta = 1*n) to scroll up, so positive
-        delta means to scroll up on both systems.
-
-        X-11 sends Control-Button-4 event instead.
-        """
-        up = {EventType.MouseWheel: event.delta > 0,
-              EventType.Button: event.num == 4}
-        lines = -5 if up[event.type] else 5
-        self.text.yview_scroll(lines, 'units')
-        return 'break'
-
     rmenu = None
 
     def right_menu_event(self, event):
diff --git a/Lib/idlelib/idle_test/test_multicall.py b/Lib/idlelib/idle_test/test_multicall.py
index 68156a743d7b9b..ba582bb3ca51b4 100644
--- a/Lib/idlelib/idle_test/test_multicall.py
+++ b/Lib/idlelib/idle_test/test_multicall.py
@@ -35,6 +35,14 @@ def test_init(self):
         mctext = self.mc(self.root)
         self.assertIsInstance(mctext._MultiCall__binders, list)
 
+    def test_yview(self):
+        # Added for tree.wheel_event
+        # (it depends on yview to not be overriden)
+        mc = self.mc
+        self.assertIs(mc.yview, Text.yview)
+        mctext = self.mc(self.root)
+        self.assertIs(mctext.yview.__func__, Text.yview)
+
 
 if __name__ == '__main__':
     unittest.main(verbosity=2)
diff --git a/Lib/idlelib/idle_test/test_tree.py b/Lib/idlelib/idle_test/test_tree.py
index 9be9abee361f08..b3e4c10cf9e38e 100644
--- a/Lib/idlelib/idle_test/test_tree.py
+++ b/Lib/idlelib/idle_test/test_tree.py
@@ -4,7 +4,7 @@
 import unittest
 from test.support import requires
 requires('gui')
-from tkinter import Tk
+from tkinter import Tk, EventType, SCROLL
 
 
 class TreeTest(unittest.TestCase):
@@ -29,5 +29,32 @@ def test_init(self):
         node.expand()
 
 
+class TestScrollEvent(unittest.TestCase):
+
+    def test_wheel_event(self):
+        # Fake widget class containing `yview` only.
+        class _Widget:
+            def __init__(widget, *expected):
+                widget.expected = expected
+            def yview(widget, *args):
+                self.assertTupleEqual(widget.expected, args)
+        # Fake event class
+        class _Event:
+            pass
+        #        (type, delta, num, amount)
+        tests = ((EventType.MouseWheel, 120, -1, -5),
+                 (EventType.MouseWheel, -120, -1, 5),
+                 (EventType.ButtonPress, -1, 4, -5),
+                 (EventType.ButtonPress, -1, 5, 5))
+
+        event = _Event()
+        for ty, delta, num, amount in tests:
+            event.type = ty
+            event.delta = delta
+            event.num = num
+            res = tree.wheel_event(event, _Widget(SCROLL, amount, "units"))
+            self.assertEqual(res, "break")
+
+
 if __name__ == '__main__':
     unittest.main(verbosity=2)
diff --git a/Lib/idlelib/tree.py b/Lib/idlelib/tree.py
index 21426cbb33e0da..6229be4e5a8ad5 100644
--- a/Lib/idlelib/tree.py
+++ b/Lib/idlelib/tree.py
@@ -56,6 +56,30 @@ def listicons(icondir=ICONDIR):
             column = 0
     root.images = images
 
+def wheel_event(event, widget=None):
+    """Handle scrollwheel event.
+
+    For wheel up, event.delta = 120*n on Windows, -1*n on darwin,
+    where n can be > 1 if one scrolls fast.  Flicking the wheel
+    generates up to maybe 20 events with n up to 10 or more 1.
+    Macs use wheel down (delta = 1*n) to scroll up, so positive
+    delta means to scroll up on both systems.
+
+    X-11 sends Control-Button-4,5 events instead.
+
+    The widget parameter is needed so browser label bindings can pass
+    the underlying canvas.
+
+    This function depends on widget.yview to not be overridden by
+    a subclass.
+    """
+    up = {EventType.MouseWheel: event.delta > 0,
+          EventType.ButtonPress: event.num == 4}
+    lines = -5 if up[event.type] else 5
+    widget = event.widget if widget is None else widget
+    widget.yview(SCROLL, lines, 'units')
+    return 'break'
+
 
 class TreeNode:
 
@@ -260,6 +284,9 @@ def drawtext(self):
                                        anchor="nw", window=self.label)
         self.label.bind("<1>", self.select_or_edit)
         self.label.bind("", self.flip)
+        self.label.bind("", lambda e: wheel_event(e, self.canvas))
+        self.label.bind("", lambda e: wheel_event(e, self.canvas))
+        self.label.bind("", lambda e: wheel_event(e, self.canvas))
         self.text_id = id
 
     def select_or_edit(self, event=None):
@@ -410,6 +437,7 @@ def GetSubList(self):
 # A canvas widget with scroll bars and some useful bindings
 
 class ScrolledCanvas:
+
     def __init__(self, master, **opts):
         if 'yscrollincrement' not in opts:
             opts['yscrollincrement'] = 17
@@ -431,6 +459,9 @@ def __init__(self, master, **opts):
         self.canvas.bind("", self.page_down)
         self.canvas.bind("", self.unit_up)
         self.canvas.bind("", self.unit_down)
+        self.canvas.bind("", wheel_event)
+        self.canvas.bind("", wheel_event)
+        self.canvas.bind("", wheel_event)
         #if isinstance(master, Toplevel) or isinstance(master, Tk):
         self.canvas.bind("", self.zoom_height)
         self.canvas.focus_set()
diff --git a/Misc/ACKS b/Misc/ACKS
index def874b0071e6f..290f8ea33269b3 100644
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -1863,6 +1863,7 @@ Nickolai Zeldovich
 Yuxiao Zeng
 Uwe Zessin
 Cheng Zhang
+George Zhang
 Kai Zhu
 Tarek Ziadé
 Jelle Zijlstra
diff --git a/Misc/NEWS.d/next/IDLE/2019-08-21-16-02-49.bpo-37902._R_adE.rst b/Misc/NEWS.d/next/IDLE/2019-08-21-16-02-49.bpo-37902._R_adE.rst
new file mode 100644
index 00000000000000..24b4142484695c
--- /dev/null
+++ b/Misc/NEWS.d/next/IDLE/2019-08-21-16-02-49.bpo-37902._R_adE.rst
@@ -0,0 +1,2 @@
+Add mousewheel scrolling for IDLE module, path, and stack browsers.
+Patch by George Zhang.

From bdcbb83c6640c2b30d0e1a2a497e9ba34cc53b26 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 4 Sep 2019 21:25:59 -0700
Subject: [PATCH 0492/2163] bpo-38026: fix inspect.getattr_static (GH-15676)

It should avoid dynamic lookup including `isinstance`.

This is a regression caused by GH-5351.
(cherry picked from commit 8f9cc8771ffb8d0e21be287eaed42ae06087acca)

Co-authored-by: Inada Naoki 
---
 Lib/inspect.py                                                | 4 ++--
 .../next/Library/2019-09-04-20-34-14.bpo-38026.0LLRX-.rst     | 2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2019-09-04-20-34-14.bpo-38026.0LLRX-.rst

diff --git a/Lib/inspect.py b/Lib/inspect.py
index 99a580bd2f2350..a616f2d49b7d96 100644
--- a/Lib/inspect.py
+++ b/Lib/inspect.py
@@ -1558,7 +1558,7 @@ def _shadowed_dict(klass):
         except KeyError:
             pass
         else:
-            if not (isinstance(class_dict, types.GetSetDescriptorType) and
+            if not (type(class_dict) is types.GetSetDescriptorType and
                     class_dict.__name__ == "__dict__" and
                     class_dict.__objclass__ is entry):
                 return class_dict
@@ -1580,7 +1580,7 @@ def getattr_static(obj, attr, default=_sentinel):
         klass = type(obj)
         dict_attr = _shadowed_dict(klass)
         if (dict_attr is _sentinel or
-            isinstance(dict_attr, types.MemberDescriptorType)):
+            type(dict_attr) is types.MemberDescriptorType):
             instance_result = _check_instance(obj, attr)
     else:
         klass = obj
diff --git a/Misc/NEWS.d/next/Library/2019-09-04-20-34-14.bpo-38026.0LLRX-.rst b/Misc/NEWS.d/next/Library/2019-09-04-20-34-14.bpo-38026.0LLRX-.rst
new file mode 100644
index 00000000000000..166cd867514cae
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-04-20-34-14.bpo-38026.0LLRX-.rst
@@ -0,0 +1,2 @@
+Fixed :func:`inspect.getattr_static` used ``isinstance`` while it should
+avoid dynamic lookup.

From dafbe3265657dc5a5c46e762080023f0aa25ec58 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Thu, 5 Sep 2019 00:42:22 -0700
Subject: [PATCH 0493/2163] bpo-36324:  Apply review comments from Allen Downey
 (GH-15693) (GH-15694)

(cherry picked from commit e4810b2a6c1d0db1a27ad046831b8fa3b57967a4)

Co-authored-by: Raymond Hettinger 
---
 Doc/library/statistics.rst | 129 +++++++++++++++++++------------------
 Lib/statistics.py          |  38 +++++------
 Misc/ACKS                  |   1 +
 3 files changed, 83 insertions(+), 85 deletions(-)

diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst
index cbe2b8223faf9b..9f9fc86b3a1350 100644
--- a/Doc/library/statistics.rst
+++ b/Doc/library/statistics.rst
@@ -26,10 +26,10 @@ numeric (:class:`Real`-valued) data.
    Unless explicitly noted otherwise, these functions support :class:`int`,
    :class:`float`, :class:`decimal.Decimal` and :class:`fractions.Fraction`.
    Behaviour with other types (whether in the numeric tower or not) is
-   currently unsupported.  Mixed types are also undefined and
-   implementation-dependent.  If your input data consists of mixed types,
-   you may be able to use :func:`map` to ensure a consistent result, e.g.
-   ``map(float, input_data)``.
+   currently unsupported.  Collections with a mix of types are also undefined
+   and implementation-dependent.  If your input data consists of mixed types,
+   you may be able to use :func:`map` to ensure a consistent result, for
+   example: ``map(float, input_data)``.
 
 Averages and measures of central location
 -----------------------------------------
@@ -102,11 +102,9 @@ However, for reading convenience, most of the examples show sorted sequences.
    .. note::
 
       The mean is strongly affected by outliers and is not a robust estimator
-      for central location: the mean is not necessarily a typical example of the
-      data points.  For more robust, although less efficient, measures of
-      central location, see :func:`median` and :func:`mode`.  (In this case,
-      "efficient" refers to statistical efficiency rather than computational
-      efficiency.)
+      for central location: the mean is not necessarily a typical example of
+      the data points.  For more robust measures of central location, see
+      :func:`median` and :func:`mode`.
 
       The sample mean gives an unbiased estimate of the true population mean,
       which means that, taken on average over all the possible samples,
@@ -120,9 +118,8 @@ However, for reading convenience, most of the examples show sorted sequences.
    Convert *data* to floats and compute the arithmetic mean.
 
    This runs faster than the :func:`mean` function and it always returns a
-   :class:`float`.  The result is highly accurate but not as perfect as
-   :func:`mean`.  If the input dataset is empty, raises a
-   :exc:`StatisticsError`.
+   :class:`float`.  The *data* may be a sequence or iterator.  If the input
+   dataset is empty, raises a :exc:`StatisticsError`.
 
    .. doctest::
 
@@ -136,15 +133,20 @@ However, for reading convenience, most of the examples show sorted sequences.
 
    Convert *data* to floats and compute the geometric mean.
 
+   The geometric mean indicates the central tendency or typical value of the
+   *data* using the product of the values (as opposed to the arithmetic mean
+   which uses their sum).
+
    Raises a :exc:`StatisticsError` if the input dataset is empty,
    if it contains a zero, or if it contains a negative value.
+   The *data* may be a sequence or iterator.
 
    No special efforts are made to achieve exact results.
    (However, this may change in the future.)
 
    .. doctest::
 
-      >>> round(geometric_mean([54, 24, 36]), 9)
+      >>> round(geometric_mean([54, 24, 36]), 1)
       36.0
 
    .. versionadded:: 3.8
@@ -174,7 +176,7 @@ However, for reading convenience, most of the examples show sorted sequences.
       3.6
 
    Using the arithmetic mean would give an average of about 5.167, which
-   is too high.
+   is well over the aggregate P/E ratio.
 
    :exc:`StatisticsError` is raised if *data* is empty, or any element
    is less than zero.
@@ -312,10 +314,10 @@ However, for reading convenience, most of the examples show sorted sequences.
    The mode (when it exists) is the most typical value and serves as a
    measure of central location.
 
-   If there are multiple modes, returns the first one encountered in the *data*.
-   If the smallest or largest of multiple modes is desired instead, use
-   ``min(multimode(data))`` or ``max(multimode(data))``.  If the input *data* is
-   empty, :exc:`StatisticsError` is raised.
+   If there are multiple modes with the same frequency, returns the first one
+   encountered in the *data*.  If the smallest or largest of those is
+   desired instead, use ``min(multimode(data))`` or ``max(multimode(data))``.
+   If the input *data* is empty, :exc:`StatisticsError` is raised.
 
    ``mode`` assumes discrete data, and returns a single value. This is the
    standard treatment of the mode as commonly taught in schools:
@@ -325,8 +327,8 @@ However, for reading convenience, most of the examples show sorted sequences.
       >>> mode([1, 1, 2, 3, 3, 3, 3, 4])
       3
 
-   The mode is unique in that it is the only statistic which also applies
-   to nominal (non-numeric) data:
+   The mode is unique in that it is the only statistic in this package that
+   also applies to nominal (non-numeric) data:
 
    .. doctest::
 
@@ -368,15 +370,16 @@ However, for reading convenience, most of the examples show sorted sequences.
 
 .. function:: pvariance(data, mu=None)
 
-   Return the population variance of *data*, a non-empty iterable of real-valued
-   numbers.  Variance, or second moment about the mean, is a measure of the
-   variability (spread or dispersion) of data.  A large variance indicates that
-   the data is spread out; a small variance indicates it is clustered closely
-   around the mean.
+   Return the population variance of *data*, a non-empty sequence or iterator
+   of real-valued numbers.  Variance, or second moment about the mean, is a
+   measure of the variability (spread or dispersion) of data.  A large
+   variance indicates that the data is spread out; a small variance indicates
+   it is clustered closely around the mean.
 
-   If the optional second argument *mu* is given, it should be the mean of
-   *data*.  If it is missing or ``None`` (the default), the mean is
-   automatically calculated.
+   If the optional second argument *mu* is given, it is typically the mean of
+   the *data*.  It can also be used to compute the second moment around a
+   point that is not the mean.  If it is missing or ``None`` (the default),
+   the arithmetic mean is automatically calculated.
 
    Use this function to calculate the variance from the entire population.  To
    estimate the variance from a sample, the :func:`variance` function is usually
@@ -401,10 +404,6 @@ However, for reading convenience, most of the examples show sorted sequences.
       >>> pvariance(data, mu)
       1.25
 
-   This function does not attempt to verify that you have passed the actual mean
-   as *mu*.  Using arbitrary values for *mu* may lead to invalid or impossible
-   results.
-
    Decimals and Fractions are supported:
 
    .. doctest::
@@ -423,11 +422,11 @@ However, for reading convenience, most of the examples show sorted sequences.
       Ď².  When called on a sample instead, this is the biased sample variance
       s², also known as variance with N degrees of freedom.
 
-      If you somehow know the true population mean ÎĽ, you may use this function
-      to calculate the variance of a sample, giving the known population mean as
-      the second argument.  Provided the data points are representative
-      (e.g. independent and identically distributed), the result will be an
-      unbiased estimate of the population variance.
+      If you somehow know the true population mean ÎĽ, you may use this
+      function to calculate the variance of a sample, giving the known
+      population mean as the second argument.  Provided the data points are a
+      random sample of the population, the result will be an unbiased estimate
+      of the population variance.
 
 
 .. function:: stdev(data, xbar=None)
@@ -502,19 +501,19 @@ However, for reading convenience, most of the examples show sorted sequences.
       :func:`pvariance` function as the *mu* parameter to get the variance of a
       sample.
 
-.. function:: quantiles(dist, *, n=4, method='exclusive')
+.. function:: quantiles(data, *, n=4, method='exclusive')
 
-   Divide *dist* into *n* continuous intervals with equal probability.
+   Divide *data* into *n* continuous intervals with equal probability.
    Returns a list of ``n - 1`` cut points separating the intervals.
 
    Set *n* to 4 for quartiles (the default).  Set *n* to 10 for deciles.  Set
    *n* to 100 for percentiles which gives the 99 cuts points that separate
-   *dist* in to 100 equal sized groups.  Raises :exc:`StatisticsError` if *n*
+   *data* in to 100 equal sized groups.  Raises :exc:`StatisticsError` if *n*
    is not least 1.
 
-   The *dist* can be any iterable containing sample data or it can be an
+   The *data* can be any iterable containing sample data or it can be an
    instance of a class that defines an :meth:`~inv_cdf` method.  For meaningful
-   results, the number of data points in *dist* should be larger than *n*.
+   results, the number of data points in *data* should be larger than *n*.
    Raises :exc:`StatisticsError` if there are not at least two data points.
 
    For sample data, the cut points are linearly interpolated from the
@@ -523,7 +522,7 @@ However, for reading convenience, most of the examples show sorted sequences.
    cut-point will evaluate to ``104``.
 
    The *method* for computing quantiles can be varied depending on
-   whether the data in *dist* includes or excludes the lowest and
+   whether the data in *data* includes or excludes the lowest and
    highest possible values from the population.
 
    The default *method* is "exclusive" and is used for data sampled from
@@ -535,14 +534,14 @@ However, for reading convenience, most of the examples show sorted sequences.
 
    Setting the *method* to "inclusive" is used for describing population
    data or for samples that are known to include the most extreme values
-   from the population.  The minimum value in *dist* is treated as the 0th
+   from the population.  The minimum value in *data* is treated as the 0th
    percentile and the maximum value is treated as the 100th percentile.
    The portion of the population falling below the *i-th* of *m* sorted
    data points is computed as ``(i - 1) / (m - 1)``.  Given 11 sample
    values, the method sorts them and assigns the following percentiles:
    0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%.
 
-   If *dist* is an instance of a class that defines an
+   If *data* is an instance of a class that defines an
    :meth:`~inv_cdf` method, setting *method* has no effect.
 
    .. doctest::
@@ -580,7 +579,7 @@ A single exception is defined:
 :class:`NormalDist` is a tool for creating and manipulating normal
 distributions of a `random variable
 `_.  It is a
-composite class that treats the mean and standard deviation of data
+class that treats the mean and standard deviation of data
 measurements as a single entity.
 
 Normal distributions arise from the `Central Limit Theorem
@@ -616,13 +615,14 @@ of applications in statistics.
 
     .. classmethod:: NormalDist.from_samples(data)
 
-       Makes a normal distribution instance computed from sample data.  The
-       *data* can be any :term:`iterable` and should consist of values that
-       can be converted to type :class:`float`.
+       Makes a normal distribution instance with *mu* and *sigma* parameters
+       estimated from the *data* using :func:`fmean` and :func:`stdev`.
 
-       If *data* does not contain at least two elements, raises
-       :exc:`StatisticsError` because it takes at least one point to estimate
-       a central value and at least two points to estimate dispersion.
+       The *data* can be any :term:`iterable` and should consist of values
+       that can be converted to type :class:`float`.  If *data* does not
+       contain at least two elements, raises :exc:`StatisticsError` because it
+       takes at least one point to estimate a central value and at least two
+       points to estimate dispersion.
 
     .. method:: NormalDist.samples(n, *, seed=None)
 
@@ -636,10 +636,10 @@ of applications in statistics.
     .. method:: NormalDist.pdf(x)
 
        Using a `probability density function (pdf)
-       `_,
-       compute the relative likelihood that a random variable *X* will be near
-       the given value *x*.  Mathematically, it is the ratio ``P(x <= X <
-       x+dx) / dx``.
+       `_, compute
+       the relative likelihood that a random variable *X* will be near the
+       given value *x*.  Mathematically, it is the limit of the ratio ``P(x <=
+       X < x+dx) / dx`` as *dx* approaches zero.
 
        The relative likelihood is computed as the probability of a sample
        occurring in a narrow range divided by the width of the range (hence
@@ -667,8 +667,10 @@ of applications in statistics.
 
     .. method:: NormalDist.overlap(other)
 
-       Returns a value between 0.0 and 1.0 giving the overlapping area for
-       the two probability density functions.
+       Measures the agreement between two normal probability distributions.
+       Returns a value between 0.0 and 1.0 giving `the overlapping area for
+       the two probability density functions
+       `_.
 
     Instances of :class:`NormalDist` support addition, subtraction,
     multiplication and division by a constant.  These operations
@@ -740,12 +742,11 @@ Carlo simulation `_:
     ...     return (3*x + 7*x*y - 5*y) / (11 * z)
     ...
     >>> n = 100_000
-    >>> seed = 86753099035768
-    >>> X = NormalDist(10, 2.5).samples(n, seed=seed)
-    >>> Y = NormalDist(15, 1.75).samples(n, seed=seed)
-    >>> Z = NormalDist(50, 1.25).samples(n, seed=seed)
-    >>> NormalDist.from_samples(map(model, X, Y, Z))     # doctest: +SKIP
-    NormalDist(mu=1.8661894803304777, sigma=0.65238717376862)
+    >>> X = NormalDist(10, 2.5).samples(n, seed=3652260728)
+    >>> Y = NormalDist(15, 1.75).samples(n, seed=4582495471)
+    >>> Z = NormalDist(50, 1.25).samples(n, seed=6582483453)
+    >>> quantiles(map(model, X, Y, Z))       # doctest: +SKIP
+    [1.4591308524824727, 1.8035946855390597, 2.175091447274739]
 
 Normal distributions commonly arise in machine learning problems.
 
diff --git a/Lib/statistics.py b/Lib/statistics.py
index c7d6568145e0fa..4b172662770fb0 100644
--- a/Lib/statistics.py
+++ b/Lib/statistics.py
@@ -322,7 +322,6 @@ def fmean(data):
     """Convert data to floats and compute the arithmetic mean.
 
     This runs faster than the mean() function and it always returns a float.
-    The result is highly accurate but not as perfect as mean().
     If the input dataset is empty, it raises a StatisticsError.
 
     >>> fmean([3.5, 4.0, 5.25])
@@ -538,15 +537,16 @@ def mode(data):
     ``mode`` assumes discrete data, and returns a single value. This is the
     standard treatment of the mode as commonly taught in schools:
 
-    >>> mode([1, 1, 2, 3, 3, 3, 3, 4])
-    3
+        >>> mode([1, 1, 2, 3, 3, 3, 3, 4])
+        3
 
     This also works with nominal (non-numeric) data:
 
-    >>> mode(["red", "blue", "blue", "red", "green", "red", "red"])
-    'red'
+        >>> mode(["red", "blue", "blue", "red", "green", "red", "red"])
+        'red'
 
-    If there are multiple modes, return the first one encountered.
+    If there are multiple modes with same frequency, return the first one
+    encountered:
 
         >>> mode(['red', 'red', 'green', 'blue', 'blue'])
         'red'
@@ -615,28 +615,28 @@ def multimode(data):
 # position is that fewer options make for easier choices and that
 # external packages can be used for anything more advanced.
 
-def quantiles(dist, /, *, n=4, method='exclusive'):
-    """Divide *dist* into *n* continuous intervals with equal probability.
+def quantiles(data, /, *, n=4, method='exclusive'):
+    """Divide *data* into *n* continuous intervals with equal probability.
 
     Returns a list of (n - 1) cut points separating the intervals.
 
     Set *n* to 4 for quartiles (the default).  Set *n* to 10 for deciles.
     Set *n* to 100 for percentiles which gives the 99 cuts points that
-    separate *dist* in to 100 equal sized groups.
+    separate *data* in to 100 equal sized groups.
 
-    The *dist* can be any iterable containing sample data or it can be
+    The *data* can be any iterable containing sample data or it can be
     an instance of a class that defines an inv_cdf() method.  For sample
     data, the cut points are linearly interpolated between data points.
 
-    If *method* is set to *inclusive*, *dist* is treated as population
+    If *method* is set to *inclusive*, *data* is treated as population
     data.  The minimum value is treated as the 0th percentile and the
     maximum value is treated as the 100th percentile.
     """
     if n < 1:
         raise StatisticsError('n must be at least 1')
-    if hasattr(dist, 'inv_cdf'):
-        return [dist.inv_cdf(i / n) for i in range(1, n)]
-    data = sorted(dist)
+    if hasattr(data, 'inv_cdf'):
+        return [data.inv_cdf(i / n) for i in range(1, n)]
+    data = sorted(data)
     ld = len(data)
     if ld < 2:
         raise StatisticsError('must have at least two data points')
@@ -745,7 +745,7 @@ def variance(data, xbar=None):
 def pvariance(data, mu=None):
     """Return the population variance of ``data``.
 
-    data should be an iterable of Real-valued numbers, with at least one
+    data should be a sequence or iterator of Real-valued numbers, with at least one
     value. The optional argument mu, if given, should be the mean of
     the data. If it is missing or None, the mean is automatically calculated.
 
@@ -766,10 +766,6 @@ def pvariance(data, mu=None):
     >>> pvariance(data, mu)
     1.25
 
-    This function does not check that ``mu`` is actually the mean of ``data``.
-    Giving arbitrary values for ``mu`` may lead to invalid or impossible
-    results.
-
     Decimals and Fractions are supported:
 
     >>> from decimal import Decimal as D
@@ -913,8 +909,8 @@ def __init__(self, mu=0.0, sigma=1.0):
         "NormalDist where mu is the mean and sigma is the standard deviation."
         if sigma < 0.0:
             raise StatisticsError('sigma must be non-negative')
-        self._mu = mu
-        self._sigma = sigma
+        self._mu = float(mu)
+        self._sigma = float(sigma)
 
     @classmethod
     def from_samples(cls, data):
diff --git a/Misc/ACKS b/Misc/ACKS
index 290f8ea33269b3..55aab6b6b2b324 100644
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -414,6 +414,7 @@ Dima Dorfman
 Yves Dorfsman
 Michael Dorman
 Steve Dower
+Allen Downey
 Cesar Douady
 Dean Draayer
 Fred L. Drake, Jr.

From f5649bfe7622447b302ef55e4db3a96b5840f8e8 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Thu, 5 Sep 2019 01:10:40 -0700
Subject: [PATCH 0494/2163] bpo-36324:  Apply review comment from Jake
 Vanderplas (GH-15695) (GH-15696)

(cherry picked from commit 9b51570ffd0494c07dafe10c7d2afe865754694c)

Co-authored-by: Raymond Hettinger 
---
 Doc/library/statistics.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst
index 9f9fc86b3a1350..2ddd393a459368 100644
--- a/Doc/library/statistics.rst
+++ b/Doc/library/statistics.rst
@@ -727,9 +727,9 @@ Find the `quartiles `_ and `deciles
 
 .. doctest::
 
-    >>> [round(sat.inv_cdf(p)) for p in (0.25, 0.50, 0.75)]
+    >>> list(map(round, quantiles(sat)))
     [928, 1060, 1192]
-    >>> [round(sat.inv_cdf(p / 10)) for p in range(1, 10)]
+    >>> list(map(round, quantiles(sat, n=10)))
     [810, 896, 958, 1011, 1060, 1109, 1162, 1224, 1310]
 
 To estimate the distribution for a model than isn't easy to solve

From 7eaeddad75cc736d327b8ece9380ef6ee23a0d9a Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Thu, 5 Sep 2019 04:17:41 -0700
Subject: [PATCH 0495/2163] Correct minor gramatical mistake in sys.settrace
 doc (GH-15637)

(cherry picked from commit 3038e87ba848023470f571242a8bb5a206c24430)

Co-authored-by: Andre Delfino 
---
 Doc/library/sys.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
index 09a987ca32ace9..be1af371d428a5 100644
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -1276,7 +1276,8 @@ always available.
 
    The trace function is invoked (with *event* set to ``'call'``) whenever a new
    local scope is entered; it should return a reference to a local trace
-   function to be used that scope, or ``None`` if the scope shouldn't be traced.
+   function to be used for the new scope, or ``None`` if the scope shouldn't be
+   traced.
 
    The local trace function should return a reference to itself (or to another
    function for further tracing in that scope), or ``None`` to turn off tracing

From b8c66779c7003071f1a330428d58bbbb34c7ae12 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Thu, 5 Sep 2019 23:19:13 -0700
Subject: [PATCH 0496/2163] More refinements to the statistics docs (GH-15713)
 (GH-15715)

(cherry picked from commit d8c93aa5d29d3cab537357018d5806a57452a8fe)

Co-authored-by: Raymond Hettinger 
---
 Doc/library/statistics.rst | 60 +++++++++++++++++++++-----------------
 1 file changed, 33 insertions(+), 27 deletions(-)

diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst
index 2ddd393a459368..9e39828a5700c8 100644
--- a/Doc/library/statistics.rst
+++ b/Doc/library/statistics.rst
@@ -19,17 +19,21 @@
 --------------
 
 This module provides functions for calculating mathematical statistics of
-numeric (:class:`Real`-valued) data.
-
-.. note::
-
-   Unless explicitly noted otherwise, these functions support :class:`int`,
-   :class:`float`, :class:`decimal.Decimal` and :class:`fractions.Fraction`.
-   Behaviour with other types (whether in the numeric tower or not) is
-   currently unsupported.  Collections with a mix of types are also undefined
-   and implementation-dependent.  If your input data consists of mixed types,
-   you may be able to use :func:`map` to ensure a consistent result, for
-   example: ``map(float, input_data)``.
+numeric (:class:`~numbers.Real`-valued) data.
+
+The module is not intended to be a competitor to third-party libraries such
+as `NumPy `_, `SciPy `_, or
+proprietary full-featured statistics packages aimed at professional
+statisticians such as Minitab, SAS and Matlab. It is aimed at the level of
+graphing and scientific calculators.
+
+Unless explicitly noted, these functions support :class:`int`,
+:class:`float`, :class:`~decimal.Decimal` and :class:`~fractions.Fraction`.
+Behaviour with other types (whether in the numeric tower or not) is
+currently unsupported.  Collections with a mix of types are also undefined
+and implementation-dependent.  If your input data consists of mixed types,
+you may be able to use :func:`map` to ensure a consistent result, for
+example: ``map(float, input_data)``.
 
 Averages and measures of central location
 -----------------------------------------
@@ -107,7 +111,7 @@ However, for reading convenience, most of the examples show sorted sequences.
       :func:`median` and :func:`mode`.
 
       The sample mean gives an unbiased estimate of the true population mean,
-      which means that, taken on average over all the possible samples,
+      so that when taken on average over all the possible samples,
       ``mean(sample)`` converges on the true mean of the entire population.  If
       *data* represents the entire population rather than a sample, then
       ``mean(data)`` is equivalent to calculating the true population mean ÎĽ.
@@ -163,8 +167,16 @@ However, for reading convenience, most of the examples show sorted sequences.
    will be equivalent to ``3/(1/a + 1/b + 1/c)``.
 
    The harmonic mean is a type of average, a measure of the central
-   location of the data.  It is often appropriate when averaging quantities
-   which are rates or ratios, for example speeds. For example:
+   location of the data.  It is often appropriate when averaging
+   rates or ratios, for example speeds.
+
+   Suppose a car travels 10 km at 40 km/hr, then another 10 km at 60 km/hr.
+   What is the average speed?
+
+   .. doctest::
+
+      >>> harmonic_mean([40, 60])
+      48.0
 
    Suppose an investor purchases an equal value of shares in each of
    three companies, with P/E (price/earning) ratios of 2.5, 3 and 10.
@@ -175,9 +187,6 @@ However, for reading convenience, most of the examples show sorted sequences.
       >>> harmonic_mean([2.5, 3, 10])  # For an equal investment portfolio.
       3.6
 
-   Using the arithmetic mean would give an average of about 5.167, which
-   is well over the aggregate P/E ratio.
-
    :exc:`StatisticsError` is raised if *data* is empty, or any element
    is less than zero.
 
@@ -190,9 +199,9 @@ However, for reading convenience, most of the examples show sorted sequences.
    middle two" method.  If *data* is empty, :exc:`StatisticsError` is raised.
    *data* can be a sequence or iterator.
 
-   The median is a robust measure of central location, and is less affected by
-   the presence of outliers in your data.  When the number of data points is
-   odd, the middle data point is returned:
+   The median is a robust measure of central location and is less affected by
+   the presence of outliers.  When the number of data points is odd, the
+   middle data point is returned:
 
    .. doctest::
 
@@ -210,13 +219,10 @@ However, for reading convenience, most of the examples show sorted sequences.
    This is suited for when your data is discrete, and you don't mind that the
    median may not be an actual data point.
 
-   If your data is ordinal (supports order operations) but not numeric (doesn't
-   support addition), you should use :func:`median_low` or :func:`median_high`
+   If the data is ordinal (supports order operations) but not numeric (doesn't
+   support addition), consider using :func:`median_low` or :func:`median_high`
    instead.
 
-   .. seealso:: :func:`median_low`, :func:`median_high`, :func:`median_grouped`
-
-
 .. function:: median_low(data)
 
    Return the low median of numeric data.  If *data* is empty,
@@ -319,7 +325,7 @@ However, for reading convenience, most of the examples show sorted sequences.
    desired instead, use ``min(multimode(data))`` or ``max(multimode(data))``.
    If the input *data* is empty, :exc:`StatisticsError` is raised.
 
-   ``mode`` assumes discrete data, and returns a single value. This is the
+   ``mode`` assumes discrete data and returns a single value. This is the
    standard treatment of the mode as commonly taught in schools:
 
    .. doctest::
@@ -522,7 +528,7 @@ However, for reading convenience, most of the examples show sorted sequences.
    cut-point will evaluate to ``104``.
 
    The *method* for computing quantiles can be varied depending on
-   whether the data in *data* includes or excludes the lowest and
+   whether the *data* includes or excludes the lowest and
    highest possible values from the population.
 
    The default *method* is "exclusive" and is used for data sampled from

From 4d1abedce9422473af2ac78047e55cde73208208 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Fri, 6 Sep 2019 02:14:31 -0700
Subject: [PATCH 0497/2163] bpo-37380: subprocess: don't use _active on win
 (GH-14360) (GH-15707)

As noted by @eryksun in [1] and [2], using _cleanup and _active(in
__del__) is not necessary on Windows, since:

> Unlike Unix, a process in Windows doesn't have to be waited on by
> its parent to avoid a zombie. Keeping the handle open will actually
> create a zombie until the next _cleanup() call, which may be never
> if Popen() isn't called again.

This patch simply defines `subprocess._active` as `None`, for which we already
have the proper logic in place in `subprocess.Popen.__del__`, that prevents it
from trying to append the process to the `_active`. This patch also defines
`subprocess._cleanup` as a noop for Windows.

[1] https://bugs.python.org/issue37380GH-msg346333
[2] https://bugs.python.org/issue36067GH-msg336262

Signed-off-by: Ruslan Kuprieiev 
(cherry picked from commit 042821ae3cf537e01963c9ec85d1a454d921e826)

Co-authored-by: Ruslan Kuprieiev 
---
 Lib/subprocess.py                             | 48 ++++++++++++-------
 Lib/test/test_subprocess.py                   | 34 +++++++++----
 .../2019-06-25-04-15-22.bpo-37380.tPxjuz.rst  |  2 +
 3 files changed, 59 insertions(+), 25 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Windows/2019-06-25-04-15-22.bpo-37380.tPxjuz.rst

diff --git a/Lib/subprocess.py b/Lib/subprocess.py
index c0bda96cbc0339..5bbeba47a37432 100644
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -218,22 +218,38 @@ def __repr__(self):
         _PopenSelector = selectors.SelectSelector
 
 
-# This lists holds Popen instances for which the underlying process had not
-# exited at the time its __del__ method got called: those processes are wait()ed
-# for synchronously from _cleanup() when a new Popen object is created, to avoid
-# zombie processes.
-_active = []
-
-def _cleanup():
-    for inst in _active[:]:
-        res = inst._internal_poll(_deadstate=sys.maxsize)
-        if res is not None:
-            try:
-                _active.remove(inst)
-            except ValueError:
-                # This can happen if two threads create a new Popen instance.
-                # It's harmless that it was already removed, so ignore.
-                pass
+if _mswindows:
+    # On Windows we just need to close `Popen._handle` when we no longer need
+    # it, so that the kernel can free it. `Popen._handle` gets closed
+    # implicitly when the `Popen` instance is finalized (see `Handle.__del__`,
+    # which is calling `CloseHandle` as requested in [1]), so there is nothing
+    # for `_cleanup` to do.
+    #
+    # [1] https://docs.microsoft.com/en-us/windows/desktop/ProcThread/
+    # creating-processes
+    _active = None
+
+    def _cleanup():
+        pass
+else:
+    # This lists holds Popen instances for which the underlying process had not
+    # exited at the time its __del__ method got called: those processes are
+    # wait()ed for synchronously from _cleanup() when a new Popen object is
+    # created, to avoid zombie processes.
+    _active = []
+
+    def _cleanup():
+        if _active is None:
+            return
+        for inst in _active[:]:
+            res = inst._internal_poll(_deadstate=sys.maxsize)
+            if res is not None:
+                try:
+                    _active.remove(inst)
+                except ValueError:
+                    # This can happen if two threads create a new Popen instance.
+                    # It's harmless that it was already removed, so ignore.
+                    pass
 
 PIPE = -1
 STDOUT = -2
diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index 9bfc21123cc016..e58d0925df3bac 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -52,10 +52,14 @@ def setUp(self):
         support.reap_children()
 
     def tearDown(self):
-        for inst in subprocess._active:
-            inst.wait()
-        subprocess._cleanup()
-        self.assertFalse(subprocess._active, "subprocess._active not empty")
+        if not mswindows:
+            # subprocess._active is not used on Windows and is set to None.
+            for inst in subprocess._active:
+                inst.wait()
+            subprocess._cleanup()
+            self.assertFalse(
+                subprocess._active, "subprocess._active not empty"
+            )
         self.doCleanups()
         support.reap_children()
 
@@ -2672,8 +2676,12 @@ def test_zombie_fast_process_del(self):
         with support.check_warnings(('', ResourceWarning)):
             p = None
 
-        # check that p is in the active processes list
-        self.assertIn(ident, [id(o) for o in subprocess._active])
+        if mswindows:
+            # subprocess._active is not used on Windows and is set to None.
+            self.assertIsNone(subprocess._active)
+        else:
+            # check that p is in the active processes list
+            self.assertIn(ident, [id(o) for o in subprocess._active])
 
     def test_leak_fast_process_del_killed(self):
         # Issue #12650: on Unix, if Popen.__del__() was called before the
@@ -2694,8 +2702,12 @@ def test_leak_fast_process_del_killed(self):
             p = None
 
         os.kill(pid, signal.SIGKILL)
-        # check that p is in the active processes list
-        self.assertIn(ident, [id(o) for o in subprocess._active])
+        if mswindows:
+            # subprocess._active is not used on Windows and is set to None.
+            self.assertIsNone(subprocess._active)
+        else:
+            # check that p is in the active processes list
+            self.assertIn(ident, [id(o) for o in subprocess._active])
 
         # let some time for the process to exit, and create a new Popen: this
         # should trigger the wait() of p
@@ -2707,7 +2719,11 @@ def test_leak_fast_process_del_killed(self):
                 pass
         # p should have been wait()ed on, and removed from the _active list
         self.assertRaises(OSError, os.waitpid, pid, 0)
-        self.assertNotIn(ident, [id(o) for o in subprocess._active])
+        if mswindows:
+            # subprocess._active is not used on Windows and is set to None.
+            self.assertIsNone(subprocess._active)
+        else:
+            self.assertNotIn(ident, [id(o) for o in subprocess._active])
 
     def test_close_fds_after_preexec(self):
         fd_status = support.findfile("fd_status.py", subdir="subprocessdata")
diff --git a/Misc/NEWS.d/next/Windows/2019-06-25-04-15-22.bpo-37380.tPxjuz.rst b/Misc/NEWS.d/next/Windows/2019-06-25-04-15-22.bpo-37380.tPxjuz.rst
new file mode 100644
index 00000000000000..facce27954a633
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-06-25-04-15-22.bpo-37380.tPxjuz.rst
@@ -0,0 +1,2 @@
+Don't collect unfinished processes with ``subprocess._active`` on Windows to
+cleanup later. Patch by Ruslan Kuprieiev.

From 4009a8522d774c36918005527dfb0975f389c8c2 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Fri, 6 Sep 2019 11:14:49 -0700
Subject: [PATCH 0498/2163] bpo-38041: Refine IDLE Shell restart lines. 
 (GH-15709)

 Restart lines now always start with '=' and never end with ' ' and fill the width of the window unless that would require ending with ' ', which could be wrapped by itself and possible confusing the user.
(cherry picked from commit 38da805d563422cf1bb9cd9be24c73806840fe30)

Co-authored-by: Terry Jan Reedy 
---
 Lib/idlelib/NEWS.txt                          |  4 ++++
 Lib/idlelib/idle_test/test_pyshell.py         | 22 +++++++++++++++++++
 Lib/idlelib/pyshell.py                        | 18 ++++++++++++---
 .../2019-09-05-23-12-13.bpo-38041.nxmGGK.rst  |  3 +++
 4 files changed, 44 insertions(+), 3 deletions(-)
 create mode 100644 Misc/NEWS.d/next/IDLE/2019-09-05-23-12-13.bpo-38041.nxmGGK.rst

diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt
index c9e846a6fba667..559ffd0cf4f30b 100644
--- a/Lib/idlelib/NEWS.txt
+++ b/Lib/idlelib/NEWS.txt
@@ -3,6 +3,10 @@ Released on 2019-10-20?
 ======================================
 
 
+bpo-38401: Shell restart lines now fill the window width, always start
+with '=', and avoid wrapping unnecessarily. The line will still wrap
+if the included file name is long relative to the width.
+
 bpo-37092: Add mousewheel scrolling for IDLE module, path, and stack
 browsers.  Patch by George Zhang.
 
diff --git a/Lib/idlelib/idle_test/test_pyshell.py b/Lib/idlelib/idle_test/test_pyshell.py
index 581444ca5ef21f..4a096676f25796 100644
--- a/Lib/idlelib/idle_test/test_pyshell.py
+++ b/Lib/idlelib/idle_test/test_pyshell.py
@@ -7,6 +7,28 @@
 from tkinter import Tk
 
 
+class FunctionTest(unittest.TestCase):
+    # Test stand-alone module level non-gui functions.
+
+    def test_restart_line_wide(self):
+        eq = self.assertEqual
+        for file, mul, extra in (('', 22, ''), ('finame', 21, '=')):
+            width = 60
+            bar = mul * '='
+            with self.subTest(file=file, bar=bar):
+                file = file or 'Shell'
+                line = pyshell.restart_line(width, file)
+                eq(len(line), width)
+                eq(line, f"{bar+extra} RESTART: {file} {bar}")
+
+    def test_restart_line_narrow(self):
+        expect, taglen = "= RESTART: Shell", 16
+        for width in (taglen-1, taglen, taglen+1):
+            with self.subTest(width=width):
+                self.assertEqual(pyshell.restart_line(width, ''), expect)
+        self.assertEqual(pyshell.restart_line(taglen+2, ''), expect+' =')
+
+
 class PyShellFileListTest(unittest.TestCase):
 
     @classmethod
diff --git a/Lib/idlelib/pyshell.py b/Lib/idlelib/pyshell.py
index ea9465568bd93f..08716a9733b759 100755
--- a/Lib/idlelib/pyshell.py
+++ b/Lib/idlelib/pyshell.py
@@ -387,6 +387,19 @@ def handle_EOF(self):
         "Override the base class - just re-raise EOFError"
         raise EOFError
 
+def restart_line(width, filename):  # See bpo-38141.
+    """Return width long restart line formatted with filename.
+
+    Fill line with balanced '='s, with any extras and at least one at
+    the beginning.  Do not end with a trailing space.
+    """
+    tag = f"= RESTART: {filename or 'Shell'} ="
+    if width >= len(tag):
+        div, mod = divmod((width -len(tag)), 2)
+        return f"{(div+mod)*'='}{tag}{div*'='}"
+    else:
+        return tag[:-2]  # Remove ' ='.
+
 
 class ModifiedInterpreter(InteractiveInterpreter):
 
@@ -491,9 +504,8 @@ def restart_subprocess(self, with_cwd=False, filename=''):
         console.stop_readline()
         # annotate restart in shell window and mark it
         console.text.delete("iomark", "end-1c")
-        tag = 'RESTART: ' + (filename if filename else 'Shell')
-        halfbar = ((int(console.width) -len(tag) - 4) // 2) * '='
-        console.write("\n{0} {1} {0}".format(halfbar, tag))
+        console.write('\n')
+        console.write(restart_line(console.width, filename))
         console.text.mark_set("restart", "end-1c")
         console.text.mark_gravity("restart", "left")
         if not filename:
diff --git a/Misc/NEWS.d/next/IDLE/2019-09-05-23-12-13.bpo-38041.nxmGGK.rst b/Misc/NEWS.d/next/IDLE/2019-09-05-23-12-13.bpo-38041.nxmGGK.rst
new file mode 100644
index 00000000000000..0aa254e8ca70f0
--- /dev/null
+++ b/Misc/NEWS.d/next/IDLE/2019-09-05-23-12-13.bpo-38041.nxmGGK.rst
@@ -0,0 +1,3 @@
+Shell restart lines now fill the window width, always start with '=',
+and avoid wrapping unnecessarily. The line will still wrap if the
+included file name is long relative to the width.

From cc51a6d7c7b6b06fb537860428347d88776d802b Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Sat, 7 Sep 2019 00:12:34 -0700
Subject: [PATCH 0499/2163] bpo-20806: Reference both times(2) and times(3) and
 link to MSDN. (GH-15479)

(cherry picked from commit 3ccdbc33385a849c60a268def578cb06b8d41be6)

Co-authored-by: Joannah Nanjekye <33177550+nanjekyejoannah@users.noreply.github.com>
---
 Doc/library/os.rst | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Doc/library/os.rst b/Doc/library/os.rst
index 6cbfb743284157..67fe36b54e2f87 100644
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -3879,7 +3879,9 @@ written in Python, such as a mail server's external command delivery program.
    :attr:`children_system`, and :attr:`elapsed` in that order.
 
    See the Unix manual page
-   :manpage:`times(2)` or the corresponding Windows Platform API documentation.
+   :manpage:`times(2)` and :manpage:`times(3)` manual page on Unix or `the GetProcessTimes MSDN
+   `
+   _ on Windows.
    On Windows, only :attr:`user` and :attr:`system` are known; the other
    attributes are zero.
 

From 3be4b107490be27470cb9a101a8dfecf446fd992 Mon Sep 17 00:00:00 2001
From: Serhiy Storchaka 
Date: Sun, 8 Sep 2019 13:17:24 +0300
Subject: [PATCH 0500/2163] [3.8] Correct Roman-numeral example in Unicode
 HOWTO. (GH-15541). (GH-15728)

(cherry picked from commit 32a960f8e1015b64b4b955b3d62920c5903d4c6f)

Co-authored-by: Greg Price 
---
 Doc/howto/unicode.rst | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst
index 24c3235e4add94..51bd64bfc232ca 100644
--- a/Doc/howto/unicode.rst
+++ b/Doc/howto/unicode.rst
@@ -57,14 +57,14 @@ their corresponding code points:
    ...
    007B    '{'; LEFT CURLY BRACKET
    ...
-   2167    'â…¦': ROMAN NUMERAL EIGHT
-   2168    'â…¨': ROMAN NUMERAL NINE
+   2167    'â…§'; ROMAN NUMERAL EIGHT
+   2168    'â…¨'; ROMAN NUMERAL NINE
    ...
-   265E    '♞': BLACK CHESS KNIGHT
-   265F    '♟': BLACK CHESS PAWN
+   265E    '♞'; BLACK CHESS KNIGHT
+   265F    '♟'; BLACK CHESS PAWN
    ...
-   1F600   'đź€': GRINNING FACE
-   1F609   'đź‰': WINKING FACE
+   1F600   'đź€'; GRINNING FACE
+   1F609   'đź‰'; WINKING FACE
    ...
 
 Strictly, these definitions imply that it's meaningless to say 'this is

From cc1bdf91d53b1a4751be84ef607e24e69a327a9b Mon Sep 17 00:00:00 2001
From: Raymond Hettinger 
Date: Sun, 8 Sep 2019 18:40:06 -0700
Subject: [PATCH 0501/2163] [3.8] bpo-36018: Address more reviewer feedback
 (GH-15733) (GH-15734)

---
 Doc/library/statistics.rst  | 41 ++++++++++++++++++++++++-------------
 Lib/statistics.py           | 32 ++++++++++++++++++++++++-----
 Lib/test/test_statistics.py | 35 ++++++++++++++-----------------
 3 files changed, 69 insertions(+), 39 deletions(-)

diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst
index 9e39828a5700c8..bdd706d0a93e3c 100644
--- a/Doc/library/statistics.rst
+++ b/Doc/library/statistics.rst
@@ -514,15 +514,14 @@ However, for reading convenience, most of the examples show sorted sequences.
 
    Set *n* to 4 for quartiles (the default).  Set *n* to 10 for deciles.  Set
    *n* to 100 for percentiles which gives the 99 cuts points that separate
-   *data* in to 100 equal sized groups.  Raises :exc:`StatisticsError` if *n*
+   *data* into 100 equal sized groups.  Raises :exc:`StatisticsError` if *n*
    is not least 1.
 
-   The *data* can be any iterable containing sample data or it can be an
-   instance of a class that defines an :meth:`~inv_cdf` method.  For meaningful
+   The *data* can be any iterable containing sample data.  For meaningful
    results, the number of data points in *data* should be larger than *n*.
    Raises :exc:`StatisticsError` if there are not at least two data points.
 
-   For sample data, the cut points are linearly interpolated from the
+   The cut points are linearly interpolated from the
    two nearest data points.  For example, if a cut point falls one-third
    of the distance between two sample values, ``100`` and ``112``, the
    cut-point will evaluate to ``104``.
@@ -547,9 +546,6 @@ However, for reading convenience, most of the examples show sorted sequences.
    values, the method sorts them and assigns the following percentiles:
    0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%.
 
-   If *data* is an instance of a class that defines an
-   :meth:`~inv_cdf` method, setting *method* has no effect.
-
    .. doctest::
 
         # Decile cut points for empirically sampled data
@@ -561,11 +557,6 @@ However, for reading convenience, most of the examples show sorted sequences.
         >>> [round(q, 1) for q in quantiles(data, n=10)]
         [81.0, 86.2, 89.0, 99.4, 102.5, 103.6, 106.0, 109.8, 111.0]
 
-        >>> # Quartile cut points for the standard normal distibution
-        >>> Z = NormalDist()
-        >>> [round(q, 4) for q in quantiles(Z, n=4)]
-        [-0.6745, 0.0, 0.6745]
-
    .. versionadded:: 3.8
 
 
@@ -607,6 +598,18 @@ of applications in statistics.
        `_ of a normal
        distribution.
 
+    .. attribute:: median
+
+       A read-only property for the `median
+       `_ of a normal
+       distribution.
+
+    .. attribute:: mode
+
+       A read-only property for the `mode
+       `_ of a normal
+       distribution.
+
     .. attribute:: stdev
 
        A read-only property for the `standard deviation
@@ -678,6 +681,16 @@ of applications in statistics.
        the two probability density functions
        `_.
 
+    .. method:: NormalDist.quantiles()
+
+        Divide the normal distribution into *n* continuous intervals with
+        equal probability.  Returns a list of (n - 1) cut points separating
+        the intervals.
+
+        Set *n* to 4 for quartiles (the default).  Set *n* to 10 for deciles.
+        Set *n* to 100 for percentiles which gives the 99 cuts points that
+        separate the normal distribution into 100 equal sized groups.
+
     Instances of :class:`NormalDist` support addition, subtraction,
     multiplication and division by a constant.  These operations
     are used for translation and scaling.  For example:
@@ -733,9 +746,9 @@ Find the `quartiles `_ and `deciles
 
 .. doctest::
 
-    >>> list(map(round, quantiles(sat)))
+    >>> list(map(round, sat.quantiles()))
     [928, 1060, 1192]
-    >>> list(map(round, quantiles(sat, n=10)))
+    >>> list(map(round, sat.quantiles(n=10)))
     [810, 896, 958, 1011, 1060, 1109, 1162, 1224, 1310]
 
 To estimate the distribution for a model than isn't easy to solve
diff --git a/Lib/statistics.py b/Lib/statistics.py
index 4b172662770fb0..70c48d605d1969 100644
--- a/Lib/statistics.py
+++ b/Lib/statistics.py
@@ -624,9 +624,8 @@ def quantiles(data, /, *, n=4, method='exclusive'):
     Set *n* to 100 for percentiles which gives the 99 cuts points that
     separate *data* in to 100 equal sized groups.
 
-    The *data* can be any iterable containing sample data or it can be
-    an instance of a class that defines an inv_cdf() method.  For sample
-    data, the cut points are linearly interpolated between data points.
+    The *data* can be any iterable containing sample.
+    The cut points are linearly interpolated between data points.
 
     If *method* is set to *inclusive*, *data* is treated as population
     data.  The minimum value is treated as the 0th percentile and the
@@ -634,8 +633,6 @@ def quantiles(data, /, *, n=4, method='exclusive'):
     """
     if n < 1:
         raise StatisticsError('n must be at least 1')
-    if hasattr(data, 'inv_cdf'):
-        return [data.inv_cdf(i / n) for i in range(1, n)]
     data = sorted(data)
     ld = len(data)
     if ld < 2:
@@ -955,6 +952,17 @@ def inv_cdf(self, p):
             raise StatisticsError('cdf() not defined when sigma at or below zero')
         return _normal_dist_inv_cdf(p, self._mu, self._sigma)
 
+    def quantiles(self, n=4):
+        """Divide into *n* continuous intervals with equal probability.
+
+        Returns a list of (n - 1) cut points separating the intervals.
+
+        Set *n* to 4 for quartiles (the default).  Set *n* to 10 for deciles.
+        Set *n* to 100 for percentiles which gives the 99 cuts points that
+        separate the normal distribution in to 100 equal sized groups.
+        """
+        return [self.inv_cdf(i / n) for i in range(1, n)]
+
     def overlap(self, other):
         """Compute the overlapping coefficient (OVL) between two normal distributions.
 
@@ -994,6 +1002,20 @@ def mean(self):
         "Arithmetic mean of the normal distribution."
         return self._mu
 
+    @property
+    def median(self):
+        "Return the median of the normal distribution"
+        return self._mu
+
+    @property
+    def mode(self):
+        """Return the mode of the normal distribution
+
+        The mode is the value x where which the probability density
+        function (pdf) takes its maximum value.
+        """
+        return self._mu
+
     @property
     def stdev(self):
         "Standard deviation of the normal distribution."
diff --git a/Lib/test/test_statistics.py b/Lib/test/test_statistics.py
index 01b317c3281eff..af26473e8fdfc3 100644
--- a/Lib/test/test_statistics.py
+++ b/Lib/test/test_statistics.py
@@ -2198,16 +2198,6 @@ def f(x):
             exp = list(map(f, expected))
             act = quantiles(map(f, data), n=n)
             self.assertTrue(all(math.isclose(e, a) for e, a in zip(exp, act)))
-        # Quartiles of a standard normal distribution
-        for n, expected in [
-            (1, []),
-            (2, [0.0]),
-            (3, [-0.4307, 0.4307]),
-            (4 ,[-0.6745, 0.0, 0.6745]),
-                ]:
-            actual = quantiles(statistics.NormalDist(), n=n)
-            self.assertTrue(all(math.isclose(e, a, abs_tol=0.0001)
-                            for e, a in zip(expected, actual)))
         # Q2 agrees with median()
         for k in range(2, 60):
             data = random.choices(range(100), k=k)
@@ -2248,16 +2238,6 @@ def f(x):
             exp = list(map(f, expected))
             act = quantiles(map(f, data), n=n, method="inclusive")
             self.assertTrue(all(math.isclose(e, a) for e, a in zip(exp, act)))
-        # Quartiles of a standard normal distribution
-        for n, expected in [
-            (1, []),
-            (2, [0.0]),
-            (3, [-0.4307, 0.4307]),
-            (4 ,[-0.6745, 0.0, 0.6745]),
-                ]:
-            actual = quantiles(statistics.NormalDist(), n=n, method="inclusive")
-            self.assertTrue(all(math.isclose(e, a, abs_tol=0.0001)
-                            for e, a in zip(expected, actual)))
         # Natural deciles
         self.assertEqual(quantiles([0, 100], n=10, method='inclusive'),
                          [10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0])
@@ -2546,6 +2526,19 @@ def test_inv_cdf(self):
         # Special values
         self.assertTrue(math.isnan(Z.inv_cdf(float('NaN'))))
 
+    def test_quantiles(self):
+        # Quartiles of a standard normal distribution
+        Z = self.module.NormalDist()
+        for n, expected in [
+            (1, []),
+            (2, [0.0]),
+            (3, [-0.4307, 0.4307]),
+            (4 ,[-0.6745, 0.0, 0.6745]),
+                ]:
+            actual = Z.quantiles(n=n)
+            self.assertTrue(all(math.isclose(e, a, abs_tol=0.0001)
+                            for e, a in zip(expected, actual)))
+
     def test_overlap(self):
         NormalDist = self.module.NormalDist
 
@@ -2612,6 +2605,8 @@ def overlap_numeric(X, Y, *, steps=8_192, z=5):
     def test_properties(self):
         X = self.module.NormalDist(100, 15)
         self.assertEqual(X.mean, 100)
+        self.assertEqual(X.median, 100)
+        self.assertEqual(X.mode, 100)
         self.assertEqual(X.stdev, 15)
         self.assertEqual(X.variance, 225)
 

From 6e3809c7ce9fbee11c3a3f89dd7e89829b7581ac Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 02:07:51 -0700
Subject: [PATCH 0502/2163] bpo-34410: Fix a crash in the tee iterator when
 re-enter it. (GH-15625)

RuntimeError is now raised in this case.
(cherry picked from commit 526a01467b3277f9fcf7f91e66c23321caa1245d)

Co-authored-by: Serhiy Storchaka 
---
 Doc/library/itertools.rst                     |  4 ++
 Lib/test/test_itertools.py                    | 37 +++++++++++++++++++
 .../2019-08-31-01-52-59.bpo-34410.7KbWZQ.rst  |  2 +
 Modules/itertoolsmodule.c                     |  9 +++++
 4 files changed, 52 insertions(+)
 create mode 100644 Misc/NEWS.d/next/Library/2019-08-31-01-52-59.bpo-34410.7KbWZQ.rst

diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst
index a3f403a5b40b9c..8d134d43806593 100644
--- a/Doc/library/itertools.rst
+++ b/Doc/library/itertools.rst
@@ -645,6 +645,10 @@ loops that truncate the stream.
    used anywhere else; otherwise, the *iterable* could get advanced without
    the tee objects being informed.
 
+   ``tee`` iterators are not threadsafe. A :exc:`RuntimeError` may be
+   raised when using simultaneously iterators returned by the same :func:`tee`
+   call, even if the original *iterable* is threadsafe.
+
    This itertool may require significant auxiliary storage (depending on how
    much temporary data needs to be stored). In general, if one iterator uses
    most or all of the data before another iterator starts, it is faster to use
diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
index 98b8c83731899a..eaa6197bec395c 100644
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -11,6 +11,7 @@
 from functools import reduce
 import sys
 import struct
+import threading
 maxsize = support.MAX_Py_ssize_t
 minsize = -maxsize-1
 
@@ -1494,6 +1495,42 @@ def test_tee_del_backward(self):
             del forward, backward
             raise
 
+    def test_tee_reenter(self):
+        class I:
+            first = True
+            def __iter__(self):
+                return self
+            def __next__(self):
+                first = self.first
+                self.first = False
+                if first:
+                    return next(b)
+
+        a, b = tee(I())
+        with self.assertRaisesRegex(RuntimeError, "tee"):
+            next(a)
+
+    def test_tee_concurrent(self):
+        start = threading.Event()
+        finish = threading.Event()
+        class I:
+            def __iter__(self):
+                return self
+            def __next__(self):
+                start.set()
+                finish.wait()
+
+        a, b = tee(I())
+        thread = threading.Thread(target=next, args=[a])
+        thread.start()
+        try:
+            start.wait()
+            with self.assertRaisesRegex(RuntimeError, "tee"):
+                next(b)
+        finally:
+            finish.set()
+            thread.join()
+
     def test_StopIteration(self):
         self.assertRaises(StopIteration, next, zip())
 
diff --git a/Misc/NEWS.d/next/Library/2019-08-31-01-52-59.bpo-34410.7KbWZQ.rst b/Misc/NEWS.d/next/Library/2019-08-31-01-52-59.bpo-34410.7KbWZQ.rst
new file mode 100644
index 00000000000000..64e778ee0913c2
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-08-31-01-52-59.bpo-34410.7KbWZQ.rst
@@ -0,0 +1,2 @@
+Fixed a crash in the :func:`tee` iterator when re-enter it. RuntimeError is
+now raised in this case.
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
index 22c04f29353e26..eba59ba1b88034 100644
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -443,6 +443,7 @@ typedef struct {
     PyObject_HEAD
     PyObject *it;
     int numread;                /* 0 <= numread <= LINKCELLS */
+    int running;
     PyObject *nextlink;
     PyObject *(values[LINKCELLS]);
 } teedataobject;
@@ -465,6 +466,7 @@ teedataobject_newinternal(PyObject *it)
     if (tdo == NULL)
         return NULL;
 
+    tdo->running = 0;
     tdo->numread = 0;
     tdo->nextlink = NULL;
     Py_INCREF(it);
@@ -493,7 +495,14 @@ teedataobject_getitem(teedataobject *tdo, int i)
     else {
         /* this is the lead iterator, so fetch more data */
         assert(i == tdo->numread);
+        if (tdo->running) {
+            PyErr_SetString(PyExc_RuntimeError,
+                            "cannot re-enter the tee iterator");
+            return NULL;
+        }
+        tdo->running = 1;
         value = PyIter_Next(tdo->it);
+        tdo->running = 0;
         if (value == NULL)
             return NULL;
         tdo->numread++;

From e103732f5df13a97f610a8b80883895f7a273573 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 02:50:30 -0700
Subject: [PATCH 0503/2163] bpo-37445: Include FORMAT_MESSAGE_IGNORE_INSERTS in
 FormatMessageW() calls (GH-14462)

If FormatMessageW() is passed the FORMAT_MESSAGE_FROM_SYSTEM flag without FORMAT_MESSAGE_IGNORE_INSERTS, it will fail if there are insert sequences in the message definition.
(cherry picked from commit a6563650c835d50f7302971a5b145e94f9d0dc68)

Co-authored-by: Zackery Spytz 
---
 .../next/Windows/2019-06-28-18-10-29.bpo-37445.LsdYO6.rst     | 2 ++
 Modules/_ctypes/callproc.c                                    | 4 +++-
 Modules/overlapped.c                                          | 3 ++-
 PC/bdist_wininst/install.c                                    | 3 ++-
 4 files changed, 9 insertions(+), 3 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Windows/2019-06-28-18-10-29.bpo-37445.LsdYO6.rst

diff --git a/Misc/NEWS.d/next/Windows/2019-06-28-18-10-29.bpo-37445.LsdYO6.rst b/Misc/NEWS.d/next/Windows/2019-06-28-18-10-29.bpo-37445.LsdYO6.rst
new file mode 100644
index 00000000000000..e4805b4e02fb12
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-06-28-18-10-29.bpo-37445.LsdYO6.rst
@@ -0,0 +1,2 @@
+Include the ``FORMAT_MESSAGE_IGNORE_INSERTS`` flag in ``FormatMessageW()``
+calls.
diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c
index 1de86e48d719b5..d9bdd9881333d9 100644
--- a/Modules/_ctypes/callproc.c
+++ b/Modules/_ctypes/callproc.c
@@ -242,7 +242,9 @@ static WCHAR *FormatError(DWORD code)
 {
     WCHAR *lpMsgBuf;
     DWORD n;
-    n = FormatMessageW(FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM,
+    n = FormatMessageW(FORMAT_MESSAGE_ALLOCATE_BUFFER |
+                       FORMAT_MESSAGE_FROM_SYSTEM |
+                       FORMAT_MESSAGE_IGNORE_INSERTS,
                        NULL,
                        code,
                        MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), /* Default language */
diff --git a/Modules/overlapped.c b/Modules/overlapped.c
index 44a0a5a834633f..52ed0bc284bcc0 100644
--- a/Modules/overlapped.c
+++ b/Modules/overlapped.c
@@ -500,7 +500,8 @@ overlapped_FormatMessage(PyObject *ignore, PyObject *args)
         return NULL;
 
     n = FormatMessageW(FORMAT_MESSAGE_ALLOCATE_BUFFER |
-                       FORMAT_MESSAGE_FROM_SYSTEM,
+                       FORMAT_MESSAGE_FROM_SYSTEM |
+                       FORMAT_MESSAGE_IGNORE_INSERTS,
                        NULL,
                        code,
                        MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),
diff --git a/PC/bdist_wininst/install.c b/PC/bdist_wininst/install.c
index 6d01ad5c2d50e7..72d9837fe4f7f0 100644
--- a/PC/bdist_wininst/install.c
+++ b/PC/bdist_wininst/install.c
@@ -938,7 +938,8 @@ static BOOL SystemError(int error, char *msg)
         LPVOID lpMsgBuf;
         FormatMessage(
             FORMAT_MESSAGE_ALLOCATE_BUFFER |
-            FORMAT_MESSAGE_FROM_SYSTEM,
+            FORMAT_MESSAGE_FROM_SYSTEM |
+            FORMAT_MESSAGE_IGNORE_INSERTS,
             NULL,
             error,
             MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),

From 14f7de72b62ec8e73a8a57b25ad8fef230dc6a3c Mon Sep 17 00:00:00 2001
From: Zachary Ware 
Date: Mon, 9 Sep 2019 04:56:38 -0500
Subject: [PATCH 0504/2163] [3.8] bpo-15817: gdbinit: Document commands after
 defining them (GH-15021) (#15744)

The gdb manual[1] says the following for "document":

  The command commandname must already be defined.

[1] https://sourceware.org/gdb/current/onlinedocs/gdb/Define.html

And indeed when trying to use the gdbinit file with gdb 8.3, I get:

  .../cpython/Misc/gdbinit:17: Error in sourced command file:
  Undefined command: "pyo".  Try "help".

Fix this by moving all documentation blocks after the define blocks.

This was introduced in GH-6384.
(cherry picked from commit 1f86fdcfc57270ee569cc58269a4e08afe7608ec)

Authored-by: Florian Bruhin 
---
 Misc/gdbinit | 46 +++++++++++++++++++++++-----------------------
 1 file changed, 23 insertions(+), 23 deletions(-)

diff --git a/Misc/gdbinit b/Misc/gdbinit
index afefe0818e4c77..45e79fcf6f4682 100644
--- a/Misc/gdbinit
+++ b/Misc/gdbinit
@@ -14,30 +14,27 @@
 # with embedded macros that you may find superior to what is in here.
 # See Tools/gdb/libpython.py and http://bugs.python.org/issue8032.
 
-document pyo
-  Prints a representation of the object to stderr, along with the
-  number of reference counts it currently has and the hex address the
-  object is allocated at.  The argument must be a PyObject*
-end
 define pyo
     # side effect of calling _PyObject_Dump is to dump the object's
     # info - assigning just prevents gdb from printing the
     # NULL return value
     set $_unused_void = _PyObject_Dump($arg0)
 end
-
-document pyg
+document pyo
   Prints a representation of the object to stderr, along with the
   number of reference counts it currently has and the hex address the
-  object is allocated at.  The argument must be a PyGC_Head*
+  object is allocated at.  The argument must be a PyObject*
 end
+
 define pyg
     print _PyGC_Dump($arg0)
 end
-
-document pylocals
-  Print the local variables of the current frame.
+document pyg
+  Prints a representation of the object to stderr, along with the
+  number of reference counts it currently has and the hex address the
+  object is allocated at.  The argument must be a PyGC_Head*
 end
+
 define pylocals
     set $_i = 0
     while $_i < f->f_code->co_nlocals
@@ -50,6 +47,9 @@ define pylocals
         set $_i = $_i + 1
     end
 end
+document pylocals
+  Print the local variables of the current frame.
+end
 
 # A rewrite of the Python interpreter's line number calculator in GDB's
 # command language
@@ -75,13 +75,13 @@ define lineno
     printf "%d", $__li
 end
 
-document pyframev
-  Print the current frame - verbose
-end
 define pyframev
     pyframe
     pylocals
 end
+document pyframev
+  Print the current frame - verbose
+end
 
 define pyframe
     set $__fn = PyUnicode_AsUTF8(f->f_code->co_filename)
@@ -134,9 +134,6 @@ end
 # the interpreter you may will have to change the functions you compare with
 # $pc.
 
-document pystack
-  Print the entire Python call stack
-end
 define pystack
     while $pc < Py_Main || $pc > Py_GetArgcArgv
         if $pc > PyEval_EvalFrameEx && $pc < _PyEval_EvalFrameDefault
@@ -146,10 +143,10 @@ define pystack
     end
     select-frame 0
 end
-
-document pystackv
-  Print the entire Python call stack - verbose mode
+document pystack
+  Print the entire Python call stack
 end
+
 define pystackv
     while $pc < Py_Main || $pc > Py_GetArgcArgv
         if $pc > PyEval_EvalFrameEx && $pc < _PyEval_EvalFrameDefault
@@ -159,10 +156,10 @@ define pystackv
     end
     select-frame 0
 end
-
-document pu
-  Generally useful macro to print a Unicode string
+document pystackv
+  Print the entire Python call stack - verbose mode
 end
+
 def pu
   set $uni = $arg0
   set $i = 0
@@ -174,3 +171,6 @@ def pu
     end
   end
 end
+document pu
+  Generally useful macro to print a Unicode string
+end

From ebca7eb093f31052ff9f245b306d38941c28a1ad Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 03:11:00 -0700
Subject: [PATCH 0505/2163] bpo-32587: Make winreg.REG_MULTI_SZ support
 zero-length strings (GH-13239)

* bpo-32587: Make winreg.REG_MULTI_SZ support PendingFileRenameOperations

* Address review comments.
(cherry picked from commit e223ba13d8d871ee58570dfca4e82a591189cc2f)

Co-authored-by: Zackery Spytz 
---
 Lib/test/test_winreg.py                       |  1 +
 .../2019-05-10-15-25-44.bpo-32587.-0g2O3.rst  |  1 +
 PC/winreg.c                                   | 41 +++++++++++--------
 3 files changed, 27 insertions(+), 16 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Windows/2019-05-10-15-25-44.bpo-32587.-0g2O3.rst

diff --git a/Lib/test/test_winreg.py b/Lib/test/test_winreg.py
index 91a2bbc066b155..5c25ec8f7ec674 100644
--- a/Lib/test/test_winreg.py
+++ b/Lib/test/test_winreg.py
@@ -41,6 +41,7 @@
     ("String Val",    "A string value",                        REG_SZ),
     ("StringExpand",  "The path is %path%",                    REG_EXPAND_SZ),
     ("Multi-string",  ["Lots", "of", "string", "values"],      REG_MULTI_SZ),
+    ("Multi-nul",     ["", "", "", ""],                        REG_MULTI_SZ),
     ("Raw Data",      b"binary\x00data",                       REG_BINARY),
     ("Big String",    "x"*(2**14-1),                           REG_SZ),
     ("Big Binary",    b"x"*(2**14),                            REG_BINARY),
diff --git a/Misc/NEWS.d/next/Windows/2019-05-10-15-25-44.bpo-32587.-0g2O3.rst b/Misc/NEWS.d/next/Windows/2019-05-10-15-25-44.bpo-32587.-0g2O3.rst
new file mode 100644
index 00000000000000..41483aa8b74a9f
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-05-10-15-25-44.bpo-32587.-0g2O3.rst
@@ -0,0 +1 @@
+Make :data:`winreg.REG_MULTI_SZ` support zero-length strings.
diff --git a/PC/winreg.c b/PC/winreg.c
index d0df7ef0ad47f4..37bc2c72dcdee1 100644
--- a/PC/winreg.c
+++ b/PC/winreg.c
@@ -518,11 +518,18 @@ fixupMultiSZ(wchar_t **str, wchar_t *data, int len)
     int i;
     wchar_t *Q;
 
-    Q = data + len;
-    for (P = data, i = 0; P < Q && *P != '\0'; P++, i++) {
+    if (len > 0 && data[len - 1] == '\0') {
+        Q = data + len - 1;
+    }
+    else {
+        Q = data + len;
+    }
+
+    for (P = data, i = 0; P < Q; P++, i++) {
         str[i] = P;
-        for (; P < Q && *P != '\0'; P++)
+        for (; P < Q && *P != '\0'; P++) {
             ;
+        }
     }
 }
 
@@ -530,12 +537,20 @@ static int
 countStrings(wchar_t *data, int len)
 {
     int strings;
-    wchar_t *P;
-    wchar_t *Q = data + len;
+    wchar_t *P, *Q;
+
+    if (len > 0 && data[len - 1] == '\0') {
+        Q = data + len - 1;
+    }
+    else {
+        Q = data + len;
+    }
 
-    for (P = data, strings = 0; P < Q && *P != '\0'; P++, strings++)
-        for (; P < Q && *P != '\0'; P++)
+    for (P = data, strings = 0; P < Q; P++, strings++) {
+        for (; P < Q && *P != '\0'; P++) {
             ;
+        }
+    }
     return strings;
 }
 
@@ -749,21 +764,15 @@ Reg2Py(BYTE *retDataBuf, DWORD retDataSize, DWORD typ)
                 }
                 for (index = 0; index < s; index++)
                 {
-                    size_t len = wcslen(str[index]);
-                    if (len > INT_MAX) {
-                        PyErr_SetString(PyExc_OverflowError,
-                            "registry string is too long for a Python string");
-                        Py_DECREF(obData);
-                        PyMem_Free(str);
-                        return NULL;
-                    }
-                    PyObject *uni = PyUnicode_FromWideChar(str[index], len);
+                    size_t slen = wcsnlen(str[index], len);
+                    PyObject *uni = PyUnicode_FromWideChar(str[index], slen);
                     if (uni == NULL) {
                         Py_DECREF(obData);
                         PyMem_Free(str);
                         return NULL;
                     }
                     PyList_SET_ITEM(obData, index, uni);
+                    len -= slen + 1;
                 }
                 PyMem_Free(str);
 

From c837ad408e85eed9d20ba8331751df15e14f6aef Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 03:25:04 -0700
Subject: [PATCH 0506/2163] bpo-37936: Avoid ignoring files that we actually do
 track. (GH-15451)

There were about 14 files that are actually in the repo but that are
covered by the rules in .gitignore.

Git itself takes no notice of what .gitignore says about files that
it's already tracking... but the discrepancy can be confusing to a
human that adds a new file unexpectedly covered by these rules, as
well as to non-Git software that looks at .gitignore but doesn't
implement this wrinkle in its semantics.  (E.g., `rg`.)

Several of these are from rules that apply more broadly than
intended: for example, `Makefile` applies to `Doc/Makefile` and
`Tools/freeze/test/Makefile`, whereas `/Makefile` means only the
`Makefile` at the repo's root.

And the `Modules/Setup` rule simply wasn't updated after 961d54c5c.

https://bugs.python.org/issue37936
(cherry picked from commit 5e5e9515029f70836003a8cfb30433166fcc8db7)

Co-authored-by: Greg Price 
---
 .gitignore                                                 | 7 ++++---
 .../next/Build/2019-08-24-00-29-40.bpo-37936.QrORqA.rst    | 2 ++
 2 files changed, 6 insertions(+), 3 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Build/2019-08-24-00-29-40.bpo-37936.QrORqA.rst

diff --git a/.gitignore b/.gitignore
index 9c0c2ef07a1dc8..648f07e765b122 100644
--- a/.gitignore
+++ b/.gitignore
@@ -27,12 +27,12 @@ Include/pydtrace_probes.h
 Lib/distutils/command/*.pdb
 Lib/lib2to3/*.pickle
 Lib/test/data/*
-Makefile
+!Lib/test/data/README
+/Makefile
 Makefile.pre
 Misc/python.pc
 Misc/python-embed.pc
 Misc/python-config.sh
-Modules/Setup
 Modules/Setup.config
 Modules/Setup.local
 Modules/config.c
@@ -78,6 +78,7 @@ config.log
 config.status
 config.status.lineno
 core
+!Tools/msi/core/
 db_home
 .hg/
 .idea/
@@ -88,7 +89,7 @@ libpython*.dylib
 libpython*.dll
 platform
 pybuilddir.txt
-pyconfig.h
+/pyconfig.h
 python-config
 python-config.py
 python.bat
diff --git a/Misc/NEWS.d/next/Build/2019-08-24-00-29-40.bpo-37936.QrORqA.rst b/Misc/NEWS.d/next/Build/2019-08-24-00-29-40.bpo-37936.QrORqA.rst
new file mode 100644
index 00000000000000..4c6486103881af
--- /dev/null
+++ b/Misc/NEWS.d/next/Build/2019-08-24-00-29-40.bpo-37936.QrORqA.rst
@@ -0,0 +1,2 @@
+The :file:`.gitignore` file no longer applies to any files that are in fact
+tracked in the Git repository.  Patch by Greg Price.

From eb02196bd95ea12fcccff3960f36601596811570 Mon Sep 17 00:00:00 2001
From: Steve Dower 
Date: Mon, 9 Sep 2019 03:36:04 -0700
Subject: [PATCH 0507/2163] bpo-11953: Extend table of Windows WSA* error codes
 (GH-15004)

---
 Lib/socket.py                                 | 82 ++++++++++++++++++-
 .../2019-07-29-21-39-45.bpo-11953.4Hpwf9.rst  |  1 +
 2 files changed, 82 insertions(+), 1 deletion(-)
 create mode 100644 Misc/NEWS.d/next/Library/2019-07-29-21-39-45.bpo-11953.4Hpwf9.rst

diff --git a/Lib/socket.py b/Lib/socket.py
index 0dd8ec70e168aa..af2ed0e76a4981 100644
--- a/Lib/socket.py
+++ b/Lib/socket.py
@@ -109,21 +109,101 @@ def _intenum_converter(value, enum_klass):
 # WSA error codes
 if sys.platform.lower().startswith("win"):
     errorTab = {}
+    errorTab[6] = "Specified event object handle is invalid."
+    errorTab[8] = "Insufficient memory available."
+    errorTab[87] = "One or more parameters are invalid."
+    errorTab[995] = "Overlapped operation aborted."
+    errorTab[996] = "Overlapped I/O event object not in signaled state."
+    errorTab[997] = "Overlapped operation will complete later."
     errorTab[10004] = "The operation was interrupted."
     errorTab[10009] = "A bad file handle was passed."
     errorTab[10013] = "Permission denied."
-    errorTab[10014] = "A fault occurred on the network??" # WSAEFAULT
+    errorTab[10014] = "A fault occurred on the network??"  # WSAEFAULT
     errorTab[10022] = "An invalid operation was attempted."
+    errorTab[10024] = "Too many open files."
     errorTab[10035] = "The socket operation would block"
     errorTab[10036] = "A blocking operation is already in progress."
+    errorTab[10037] = "Operation already in progress."
+    errorTab[10038] = "Socket operation on nonsocket."
+    errorTab[10039] = "Destination address required."
+    errorTab[10040] = "Message too long."
+    errorTab[10041] = "Protocol wrong type for socket."
+    errorTab[10042] = "Bad protocol option."
+    errorTab[10043] = "Protocol not supported."
+    errorTab[10044] = "Socket type not supported."
+    errorTab[10045] = "Operation not supported."
+    errorTab[10046] = "Protocol family not supported."
+    errorTab[10047] = "Address family not supported by protocol family."
     errorTab[10048] = "The network address is in use."
+    errorTab[10049] = "Cannot assign requested address."
+    errorTab[10050] = "Network is down."
+    errorTab[10051] = "Network is unreachable."
+    errorTab[10052] = "Network dropped connection on reset."
+    errorTab[10053] = "Software caused connection abort."
     errorTab[10054] = "The connection has been reset."
+    errorTab[10055] = "No buffer space available."
+    errorTab[10056] = "Socket is already connected."
+    errorTab[10057] = "Socket is not connected."
     errorTab[10058] = "The network has been shut down."
+    errorTab[10059] = "Too many references."
     errorTab[10060] = "The operation timed out."
     errorTab[10061] = "Connection refused."
+    errorTab[10062] = "Cannot translate name."
     errorTab[10063] = "The name is too long."
     errorTab[10064] = "The host is down."
     errorTab[10065] = "The host is unreachable."
+    errorTab[10066] = "Directory not empty."
+    errorTab[10067] = "Too many processes."
+    errorTab[10068] = "User quota exceeded."
+    errorTab[10069] = "Disk quota exceeded."
+    errorTab[10070] = "Stale file handle reference."
+    errorTab[10071] = "Item is remote."
+    errorTab[10091] = "Network subsystem is unavailable."
+    errorTab[10092] = "Winsock.dll version out of range."
+    errorTab[10093] = "Successful WSAStartup not yet performed."
+    errorTab[10101] = "Graceful shutdown in progress."
+    errorTab[10102] = "No more results from WSALookupServiceNext."
+    errorTab[10103] = "Call has been canceled."
+    errorTab[10104] = "Procedure call table is invalid."
+    errorTab[10105] = "Service provider is invalid."
+    errorTab[10106] = "Service provider failed to initialize."
+    errorTab[10107] = "System call failure."
+    errorTab[10108] = "Service not found."
+    errorTab[10109] = "Class type not found."
+    errorTab[10110] = "No more results from WSALookupServiceNext."
+    errorTab[10111] = "Call was canceled."
+    errorTab[10112] = "Database query was refused."
+    errorTab[11001] = "Host not found."
+    errorTab[11002] = "Nonauthoritative host not found."
+    errorTab[11003] = "This is a nonrecoverable error."
+    errorTab[11004] = "Valid name, no data record requested type."
+    errorTab[11005] = "QoS receivers."
+    errorTab[11006] = "QoS senders."
+    errorTab[11007] = "No QoS senders."
+    errorTab[11008] = "QoS no receivers."
+    errorTab[11009] = "QoS request confirmed."
+    errorTab[11010] = "QoS admission error."
+    errorTab[11011] = "QoS policy failure."
+    errorTab[11012] = "QoS bad style."
+    errorTab[11013] = "QoS bad object."
+    errorTab[11014] = "QoS traffic control error."
+    errorTab[11015] = "QoS generic error."
+    errorTab[11016] = "QoS service type error."
+    errorTab[11017] = "QoS flowspec error."
+    errorTab[11018] = "Invalid QoS provider buffer."
+    errorTab[11019] = "Invalid QoS filter style."
+    errorTab[11020] = "Invalid QoS filter style."
+    errorTab[11021] = "Incorrect QoS filter count."
+    errorTab[11022] = "Invalid QoS object length."
+    errorTab[11023] = "Incorrect QoS flow count."
+    errorTab[11024] = "Unrecognized QoS object."
+    errorTab[11025] = "Invalid QoS policy object."
+    errorTab[11026] = "Invalid QoS flow descriptor."
+    errorTab[11027] = "Invalid QoS provider-specific flowspec."
+    errorTab[11028] = "Invalid QoS provider-specific filterspec."
+    errorTab[11029] = "Invalid QoS shape discard mode object."
+    errorTab[11030] = "Invalid QoS shaping rate object."
+    errorTab[11031] = "Reserved policy QoS element type."
     __all__.append("errorTab")
 
 
diff --git a/Misc/NEWS.d/next/Library/2019-07-29-21-39-45.bpo-11953.4Hpwf9.rst b/Misc/NEWS.d/next/Library/2019-07-29-21-39-45.bpo-11953.4Hpwf9.rst
new file mode 100644
index 00000000000000..62f3c44c284ea0
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-07-29-21-39-45.bpo-11953.4Hpwf9.rst
@@ -0,0 +1 @@
+Completing WSA* error codes in :mod:`socket`.

From 68e401fa0b1a3407bce395ad893535f65107ee6e Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 03:37:55 -0700
Subject: [PATCH 0508/2163] bpo-37705: Improve the implementation of
 winerror_to_errno() (GH-15623)

winerror_to_errno() is no longer automatically generated.
Do not rely on the old _dosmapperr() function.
Add ERROR_NO_UNICODE_TRANSLATION (1113) -> EILSEQ.
(cherry picked from commit 19052a11314e7be7ba003fd6cdbb5400a5d77d96)

Co-authored-by: Zackery Spytz 
---
 .../2019-08-30-15-15-22.bpo-37705.2o4NWW.rst  |   1 +
 PC/errmap.h                                   | 214 +++++++++++-------
 PC/generrmap.c                                |  32 ---
 3 files changed, 138 insertions(+), 109 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Windows/2019-08-30-15-15-22.bpo-37705.2o4NWW.rst
 delete mode 100644 PC/generrmap.c

diff --git a/Misc/NEWS.d/next/Windows/2019-08-30-15-15-22.bpo-37705.2o4NWW.rst b/Misc/NEWS.d/next/Windows/2019-08-30-15-15-22.bpo-37705.2o4NWW.rst
new file mode 100644
index 00000000000000..a374c3a2965a6c
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-08-30-15-15-22.bpo-37705.2o4NWW.rst
@@ -0,0 +1 @@
+Improve the implementation of ``winerror_to_errno()``.
diff --git a/PC/errmap.h b/PC/errmap.h
index 985f673a4649c8..a7489ab75c6561 100644
--- a/PC/errmap.h
+++ b/PC/errmap.h
@@ -1,80 +1,140 @@
-/* Generated file. Do not edit. */
-int winerror_to_errno(int winerror)
+int
+winerror_to_errno(int winerror)
 {
-    switch(winerror) {
-        case 2: return 2;
-        case 3: return 2;
-        case 4: return 24;
-        case 5: return 13;
-        case 6: return 9;
-        case 7: return 12;
-        case 8: return 12;
-        case 9: return 12;
-        case 10: return 7;
-        case 11: return 8;
-        case 15: return 2;
-        case 16: return 13;
-        case 17: return 18;
-        case 18: return 2;
-        case 19: return 13;
-        case 20: return 13;
-        case 21: return 13;
-        case 22: return 13;
-        case 23: return 13;
-        case 24: return 13;
-        case 25: return 13;
-        case 26: return 13;
-        case 27: return 13;
-        case 28: return 13;
-        case 29: return 13;
-        case 30: return 13;
-        case 31: return 13;
-        case 32: return 13;
-        case 33: return 13;
-        case 34: return 13;
-        case 35: return 13;
-        case 36: return 13;
-        case 53: return 2;
-        case 65: return 13;
-        case 67: return 2;
-        case 80: return 17;
-        case 82: return 13;
-        case 83: return 13;
-        case 89: return 11;
-        case 108: return 13;
-        case 109: return 32;
-        case 112: return 28;
-        case 114: return 9;
-        case 128: return 10;
-        case 129: return 10;
-        case 130: return 9;
-        case 132: return 13;
-        case 145: return 41;
-        case 158: return 13;
-        case 161: return 2;
-        case 164: return 11;
-        case 167: return 13;
-        case 183: return 17;
-        case 188: return 8;
-        case 189: return 8;
-        case 190: return 8;
-        case 191: return 8;
-        case 192: return 8;
-        case 193: return 8;
-        case 194: return 8;
-        case 195: return 8;
-        case 196: return 8;
-        case 197: return 8;
-        case 198: return 8;
-        case 199: return 8;
-        case 200: return 8;
-        case 201: return 8;
-        case 202: return 8;
-        case 206: return 2;
-        case 215: return 11;
-        case 232: return 32;
-        case 267: return 20;
-        case 1816: return 12;
-        default: return EINVAL;
+    // Unwrap FACILITY_WIN32 HRESULT errors.
+    if ((winerror & 0xFFFF0000) == 0x80070000) {
+        winerror &= 0x0000FFFF;
+    }
+
+    // Winsock error codes (10000-11999) are errno values.
+    if (winerror >= 10000 && winerror < 12000) {
+        switch (winerror) {
+        case WSAEINTR:
+        case WSAEBADF:
+        case WSAEACCES:
+        case WSAEFAULT:
+        case WSAEINVAL:
+        case WSAEMFILE:
+            // Winsock definitions of errno values. See WinSock2.h
+            return winerror - 10000;
+        default:
+            return winerror;
+        }
+    }
+
+    switch (winerror) {
+    case ERROR_FILE_NOT_FOUND:            //    2
+    case ERROR_PATH_NOT_FOUND:            //    3
+    case ERROR_INVALID_DRIVE:             //   15
+    case ERROR_NO_MORE_FILES:             //   18
+    case ERROR_BAD_NETPATH:               //   53
+    case ERROR_BAD_NET_NAME:              //   67
+    case ERROR_BAD_PATHNAME:              //  161
+    case ERROR_FILENAME_EXCED_RANGE:      //  206
+        return ENOENT;
+
+    case ERROR_BAD_ENVIRONMENT:           //   10
+        return E2BIG;
+
+    case ERROR_BAD_FORMAT:                //   11
+    case ERROR_INVALID_STARTING_CODESEG:  //  188
+    case ERROR_INVALID_STACKSEG:          //  189
+    case ERROR_INVALID_MODULETYPE:        //  190
+    case ERROR_INVALID_EXE_SIGNATURE:     //  191
+    case ERROR_EXE_MARKED_INVALID:        //  192
+    case ERROR_BAD_EXE_FORMAT:            //  193
+    case ERROR_ITERATED_DATA_EXCEEDS_64k: //  194
+    case ERROR_INVALID_MINALLOCSIZE:      //  195
+    case ERROR_DYNLINK_FROM_INVALID_RING: //  196
+    case ERROR_IOPL_NOT_ENABLED:          //  197
+    case ERROR_INVALID_SEGDPL:            //  198
+    case ERROR_AUTODATASEG_EXCEEDS_64k:   //  199
+    case ERROR_RING2SEG_MUST_BE_MOVABLE:  //  200
+    case ERROR_RELOC_CHAIN_XEEDS_SEGLIM:  //  201
+    case ERROR_INFLOOP_IN_RELOC_CHAIN:    //  202
+        return ENOEXEC;
+
+    case ERROR_INVALID_HANDLE:            //    6
+    case ERROR_INVALID_TARGET_HANDLE:     //  114
+    case ERROR_DIRECT_ACCESS_HANDLE:      //  130
+        return EBADF;
+
+    case ERROR_WAIT_NO_CHILDREN:          //  128
+    case ERROR_CHILD_NOT_COMPLETE:        //  129
+        return ECHILD;
+
+    case ERROR_NO_PROC_SLOTS:             //   89
+    case ERROR_MAX_THRDS_REACHED:         //  164
+    case ERROR_NESTING_NOT_ALLOWED:       //  215
+        return EAGAIN;
+
+    case ERROR_ARENA_TRASHED:             //    7
+    case ERROR_NOT_ENOUGH_MEMORY:         //    8
+    case ERROR_INVALID_BLOCK:             //    9
+    case ERROR_NOT_ENOUGH_QUOTA:          // 1816
+        return ENOMEM;
+
+    case ERROR_ACCESS_DENIED:             //    5
+    case ERROR_CURRENT_DIRECTORY:         //   16
+    case ERROR_WRITE_PROTECT:             //   19
+    case ERROR_BAD_UNIT:                  //   20
+    case ERROR_NOT_READY:                 //   21
+    case ERROR_BAD_COMMAND:               //   22
+    case ERROR_CRC:                       //   23
+    case ERROR_BAD_LENGTH:                //   24
+    case ERROR_SEEK:                      //   25
+    case ERROR_NOT_DOS_DISK:              //   26
+    case ERROR_SECTOR_NOT_FOUND:          //   27
+    case ERROR_OUT_OF_PAPER:              //   28
+    case ERROR_WRITE_FAULT:               //   29
+    case ERROR_READ_FAULT:                //   30
+    case ERROR_GEN_FAILURE:               //   31
+    case ERROR_SHARING_VIOLATION:         //   32
+    case ERROR_LOCK_VIOLATION:            //   33
+    case ERROR_WRONG_DISK:                //   34
+    case ERROR_SHARING_BUFFER_EXCEEDED:   //   36
+    case ERROR_NETWORK_ACCESS_DENIED:     //   65
+    case ERROR_CANNOT_MAKE:               //   82
+    case ERROR_FAIL_I24:                  //   83
+    case ERROR_DRIVE_LOCKED:              //  108
+    case ERROR_SEEK_ON_DEVICE:            //  132
+    case ERROR_NOT_LOCKED:                //  158
+    case ERROR_LOCK_FAILED:               //  167
+    case 35:                              //   35 (undefined)
+        return EACCES;
+
+    case ERROR_FILE_EXISTS:               //   80
+    case ERROR_ALREADY_EXISTS:            //  183
+        return EEXIST;
+
+    case ERROR_NOT_SAME_DEVICE:           //   17
+        return EXDEV;
+
+    case ERROR_DIRECTORY:                 //  267 (bpo-12802)
+        return ENOTDIR;
+
+    case ERROR_TOO_MANY_OPEN_FILES:       //    4
+        return EMFILE;
+
+    case ERROR_DISK_FULL:                 //  112
+        return ENOSPC;
+
+    case ERROR_BROKEN_PIPE:               //  109
+    case ERROR_NO_DATA:                   //  232 (bpo-13063)
+        return EPIPE;
+
+    case ERROR_DIR_NOT_EMPTY:             //  145
+        return ENOTEMPTY;
+
+    case ERROR_NO_UNICODE_TRANSLATION:    // 1113
+        return EILSEQ;
+
+    case ERROR_INVALID_FUNCTION:          //    1
+    case ERROR_INVALID_ACCESS:            //   12
+    case ERROR_INVALID_DATA:              //   13
+    case ERROR_INVALID_PARAMETER:         //   87
+    case ERROR_NEGATIVE_SEEK:             //  131
+    default:
+        return EINVAL;
     }
 }
diff --git a/PC/generrmap.c b/PC/generrmap.c
deleted file mode 100644
index 953344c0d79597..00000000000000
--- a/PC/generrmap.c
+++ /dev/null
@@ -1,32 +0,0 @@
-#include 
-#include 
-#include 
-#include 
-#include 
-
-/* Extract the mapping of Win32 error codes to errno */
-
-int main()
-{
-    int i;
-    _setmode(fileno(stdout), O_BINARY);
-    printf("/* Generated file. Do not edit. */\n");
-    printf("int winerror_to_errno(int winerror)\n");
-    printf("{\n    switch(winerror) {\n");
-    for(i=1; i < 65000; i++) {
-        _dosmaperr(i);
-        if (errno == EINVAL) {
-            /* Issue #12802 */
-            if (i == ERROR_DIRECTORY)
-                errno = ENOTDIR;
-            /* Issue #13063 */
-            else if (i == ERROR_NO_DATA)
-                errno = EPIPE;
-            else
-                continue;
-        }
-        printf("        case %d: return %d;\n", i, errno);
-    }
-    printf("        default: return EINVAL;\n");
-    printf("    }\n}\n");
-}

From 87a5a331eab5a99538d60a6dab25bdf299a68e3e Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 04:25:21 -0700
Subject: [PATCH 0509/2163] bpo-36250: ignore ValueError from signal in
 non-main thread (GH-12251)

Authored-By: blueyed 
(cherry picked from commit 8d64bfafdffd9f866bb6ac2e5b4c4bdfcb16aea0)

Co-authored-by: Daniel Hahler 
---
 Lib/pdb.py                                    |  8 +++--
 Lib/test/test_pdb.py                          | 29 +++++++++++++++++++
 .../2019-03-09-16-04-12.bpo-36250.tSK4N1.rst  |  2 ++
 3 files changed, 37 insertions(+), 2 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2019-03-09-16-04-12.bpo-36250.tSK4N1.rst

diff --git a/Lib/pdb.py b/Lib/pdb.py
index 69fd8bd6efb0e5..8c1c96163ed913 100755
--- a/Lib/pdb.py
+++ b/Lib/pdb.py
@@ -340,8 +340,12 @@ def preloop(self):
     def interaction(self, frame, traceback):
         # Restore the previous signal handler at the Pdb prompt.
         if Pdb._previous_sigint_handler:
-            signal.signal(signal.SIGINT, Pdb._previous_sigint_handler)
-            Pdb._previous_sigint_handler = None
+            try:
+                signal.signal(signal.SIGINT, Pdb._previous_sigint_handler)
+            except ValueError:  # ValueError: signal only works in main thread
+                pass
+            else:
+                Pdb._previous_sigint_handler = None
         if self.setup(frame, traceback):
             # no interaction desired at this time (happens if .pdbrc contains
             # a command like "continue")
diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
index 16d245a5602aba..4c38e919a83b78 100644
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -1333,6 +1333,35 @@ def start_pdb():
         self.assertNotIn('Error', stdout.decode(),
                          "Got an error running test script under PDB")
 
+    def test_issue36250(self):
+
+        with open(support.TESTFN, 'wb') as f:
+            f.write(textwrap.dedent("""
+                import threading
+                import pdb
+
+                evt = threading.Event()
+
+                def start_pdb():
+                    evt.wait()
+                    pdb.Pdb(readrc=False).set_trace()
+
+                t = threading.Thread(target=start_pdb)
+                t.start()
+                pdb.Pdb(readrc=False).set_trace()
+                evt.set()
+                t.join()""").encode('ascii'))
+        cmd = [sys.executable, '-u', support.TESTFN]
+        proc = subprocess.Popen(cmd,
+            stdout=subprocess.PIPE,
+            stdin=subprocess.PIPE,
+            stderr=subprocess.STDOUT,
+            )
+        self.addCleanup(proc.stdout.close)
+        stdout, stderr = proc.communicate(b'cont\ncont\n')
+        self.assertNotIn('Error', stdout.decode(),
+                         "Got an error running test script under PDB")
+
     def test_issue16180(self):
         # A syntax error in the debuggee.
         script = "def f: pass\n"
diff --git a/Misc/NEWS.d/next/Library/2019-03-09-16-04-12.bpo-36250.tSK4N1.rst b/Misc/NEWS.d/next/Library/2019-03-09-16-04-12.bpo-36250.tSK4N1.rst
new file mode 100644
index 00000000000000..8d9fbcbb1cba59
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-03-09-16-04-12.bpo-36250.tSK4N1.rst
@@ -0,0 +1,2 @@
+Ignore ``ValueError`` from ``signal`` with ``interaction`` in non-main
+thread.

From 252267925d3e74cfaf5216ecb0839c89c2a1baa8 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 04:29:10 -0700
Subject: [PATCH 0510/2163] [3.8] Doc: Fix PDF build (NoUri). (GH-15739)
 (GH-15754)

(cherry picked from commit 63c98ed2d21d22b46f3517fd7dfd88f0c1521299)


Co-authored-by: Julien Palard 

Automerge-Triggered-By: @JulienPalard
---
 Doc/tools/extensions/pyspecific.py | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py
index 28b8bda31146a5..f41077b0761521 100644
--- a/Doc/tools/extensions/pyspecific.py
+++ b/Doc/tools/extensions/pyspecific.py
@@ -22,6 +22,10 @@
 
 from sphinx import addnodes
 from sphinx.builders import Builder
+try:
+    from sphinx.errors import NoUri
+except ImportError:
+    from sphinx.environment import NoUri
 from sphinx.locale import translators
 from sphinx.util import status_iterator, logging
 from sphinx.util.nodes import split_explicit_title
@@ -569,10 +573,13 @@ def process_audit_events(app, doctree, fromdocname):
         for i, (doc, label) in backlinks:
             if isinstance(label, str):
                 ref = nodes.reference("", nodes.Text("[{}]".format(i)), internal=True)
-                ref['refuri'] = "{}#{}".format(
-                    app.builder.get_relative_uri(fromdocname, doc),
-                    label,
-                )
+                try:
+                    ref['refuri'] = "{}#{}".format(
+                        app.builder.get_relative_uri(fromdocname, doc),
+                        label,
+                    )
+                except NoUri:
+                    continue
                 node += ref
         row += nodes.entry('', node)
 

From bee8bfe5f440c2dde7f5af189febdbf81b27abd5 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 04:42:43 -0700
Subject: [PATCH 0511/2163] bpo-37212: Preserve keyword argument order in
 unittest.mock.call and error messages (GH-14310)

(cherry picked from commit 9d607061c9c888913ae2c18543775cf360d55f27)

Co-authored-by: Xtreak 
---
 Lib/unittest/mock.py                                        | 2 +-
 Lib/unittest/test/testmock/testmock.py                      | 6 +++---
 .../next/Library/2019-06-22-22-00-35.bpo-37212.Zhv-tq.rst   | 2 ++
 3 files changed, 6 insertions(+), 4 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2019-06-22-22-00-35.bpo-37212.Zhv-tq.rst

diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py
index 4c76f53f3870c0..7a4fcf4e3aa97f 100644
--- a/Lib/unittest/mock.py
+++ b/Lib/unittest/mock.py
@@ -2320,7 +2320,7 @@ def _format_call_signature(name, args, kwargs):
     formatted_args = ''
     args_string = ', '.join([repr(arg) for arg in args])
     kwargs_string = ', '.join([
-        '%s=%r' % (key, value) for key, value in sorted(kwargs.items())
+        '%s=%r' % (key, value) for key, value in kwargs.items()
     ])
     if args_string:
         formatted_args = args_string
diff --git a/Lib/unittest/test/testmock/testmock.py b/Lib/unittest/test/testmock/testmock.py
index d3a1e89da81a1e..413ee689510193 100644
--- a/Lib/unittest/test/testmock/testmock.py
+++ b/Lib/unittest/test/testmock/testmock.py
@@ -1563,11 +1563,11 @@ def test_assert_called_once_message_not_called(self):
             m.assert_called_once()
         self.assertNotIn("Calls:", str(e.exception))
 
-    #Issue21256 printout of keyword args should be in deterministic order
-    def test_sorted_call_signature(self):
+    #Issue37212 printout of keyword args now preserves the original order
+    def test_ordered_call_signature(self):
         m = Mock()
         m.hello(name='hello', daddy='hero')
-        text = "call(daddy='hero', name='hello')"
+        text = "call(name='hello', daddy='hero')"
         self.assertEqual(repr(m.hello.call_args), text)
 
     #Issue21270 overrides tuple methods for mock.call objects
diff --git a/Misc/NEWS.d/next/Library/2019-06-22-22-00-35.bpo-37212.Zhv-tq.rst b/Misc/NEWS.d/next/Library/2019-06-22-22-00-35.bpo-37212.Zhv-tq.rst
new file mode 100644
index 00000000000000..520a0229aa9d09
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-06-22-22-00-35.bpo-37212.Zhv-tq.rst
@@ -0,0 +1,2 @@
+:func:`unittest.mock.call` now preserves the order of keyword arguments in
+repr output. Patch by Karthikeyan Singaravelan.

From 12acb5b9c8c21e486083483bded422d8443c38a7 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 06:01:41 -0700
Subject: [PATCH 0512/2163] bpo-34652 again: Remove lchmod from the default
 AC_CHECK_FUNCS list. (GH-15758)

(cherry picked from commit bed04b664729e3e6fcee42daa108936360bac6ea)

Co-authored-by: Benjamin Peterson 
---
 aclocal.m4   | 74 +++-------------------------------------------------
 configure    | 16 ++++++++++--
 configure.ac |  2 +-
 3 files changed, 18 insertions(+), 74 deletions(-)

diff --git a/aclocal.m4 b/aclocal.m4
index 3d6b1a375fdca3..85f00dd5fac7f2 100644
--- a/aclocal.m4
+++ b/aclocal.m4
@@ -12,9 +12,9 @@
 # PARTICULAR PURPOSE.
 
 m4_ifndef([AC_CONFIG_MACRO_DIRS], [m4_defun([_AM_CONFIG_MACRO_DIRS], [])m4_defun([AC_CONFIG_MACRO_DIRS], [_AM_CONFIG_MACRO_DIRS($@)])])
-# pkg.m4 - Macros to locate and utilise pkg-config.   -*- Autoconf -*-
-# serial 11 (pkg-config-0.29.1)
-
+dnl pkg.m4 - Macros to locate and utilise pkg-config.   -*- Autoconf -*-
+dnl serial 11 (pkg-config-0.29.1)
+dnl
 dnl Copyright © 2004 Scott James Remnant .
 dnl Copyright © 2012-2015 Dan Nicholson 
 dnl
@@ -288,73 +288,5 @@ AS_VAR_COPY([$1], [pkg_cv_][$1])
 AS_VAR_IF([$1], [""], [$5], [$4])dnl
 ])dnl PKG_CHECK_VAR
 
-dnl PKG_WITH_MODULES(VARIABLE-PREFIX, MODULES,
-dnl   [ACTION-IF-FOUND],[ACTION-IF-NOT-FOUND],
-dnl   [DESCRIPTION], [DEFAULT])
-dnl ------------------------------------------
-dnl
-dnl Prepare a "--with-" configure option using the lowercase
-dnl [VARIABLE-PREFIX] name, merging the behaviour of AC_ARG_WITH and
-dnl PKG_CHECK_MODULES in a single macro.
-AC_DEFUN([PKG_WITH_MODULES],
-[
-m4_pushdef([with_arg], m4_tolower([$1]))
-
-m4_pushdef([description],
-           [m4_default([$5], [build with ]with_arg[ support])])
-
-m4_pushdef([def_arg], [m4_default([$6], [auto])])
-m4_pushdef([def_action_if_found], [AS_TR_SH([with_]with_arg)=yes])
-m4_pushdef([def_action_if_not_found], [AS_TR_SH([with_]with_arg)=no])
-
-m4_case(def_arg,
-            [yes],[m4_pushdef([with_without], [--without-]with_arg)],
-            [m4_pushdef([with_without],[--with-]with_arg)])
-
-AC_ARG_WITH(with_arg,
-     AS_HELP_STRING(with_without, description[ @<:@default=]def_arg[@:>@]),,
-    [AS_TR_SH([with_]with_arg)=def_arg])
-
-AS_CASE([$AS_TR_SH([with_]with_arg)],
-            [yes],[PKG_CHECK_MODULES([$1],[$2],$3,$4)],
-            [auto],[PKG_CHECK_MODULES([$1],[$2],
-                                        [m4_n([def_action_if_found]) $3],
-                                        [m4_n([def_action_if_not_found]) $4])])
-
-m4_popdef([with_arg])
-m4_popdef([description])
-m4_popdef([def_arg])
-
-])dnl PKG_WITH_MODULES
-
-dnl PKG_HAVE_WITH_MODULES(VARIABLE-PREFIX, MODULES,
-dnl   [DESCRIPTION], [DEFAULT])
-dnl -----------------------------------------------
-dnl
-dnl Convenience macro to trigger AM_CONDITIONAL after PKG_WITH_MODULES
-dnl check._[VARIABLE-PREFIX] is exported as make variable.
-AC_DEFUN([PKG_HAVE_WITH_MODULES],
-[
-PKG_WITH_MODULES([$1],[$2],,,[$3],[$4])
-
-AM_CONDITIONAL([HAVE_][$1],
-               [test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"])
-])dnl PKG_HAVE_WITH_MODULES
-
-dnl PKG_HAVE_DEFINE_WITH_MODULES(VARIABLE-PREFIX, MODULES,
-dnl   [DESCRIPTION], [DEFAULT])
-dnl ------------------------------------------------------
-dnl
-dnl Convenience macro to run AM_CONDITIONAL and AC_DEFINE after
-dnl PKG_WITH_MODULES check. HAVE_[VARIABLE-PREFIX] is exported as make
-dnl and preprocessor variable.
-AC_DEFUN([PKG_HAVE_DEFINE_WITH_MODULES],
-[
-PKG_HAVE_WITH_MODULES([$1],[$2],[$3],[$4])
-
-AS_IF([test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"],
-        [AC_DEFINE([HAVE_][$1], 1, [Enable ]m4_tolower([$1])[ support])])
-])dnl PKG_HAVE_DEFINE_WITH_MODULES
-
 m4_include([m4/ax_c_float_words_bigendian.m4])
 m4_include([m4/ax_check_openssl.m4])
diff --git a/configure b/configure
index 3cd9b8866c7246..59466c6123d6d8 100755
--- a/configure
+++ b/configure
@@ -786,6 +786,7 @@ infodir
 docdir
 oldincludedir
 includedir
+runstatedir
 localstatedir
 sharedstatedir
 sysconfdir
@@ -899,6 +900,7 @@ datadir='${datarootdir}'
 sysconfdir='${prefix}/etc'
 sharedstatedir='${prefix}/com'
 localstatedir='${prefix}/var'
+runstatedir='${localstatedir}/run'
 includedir='${prefix}/include'
 oldincludedir='/usr/include'
 docdir='${datarootdir}/doc/${PACKAGE_TARNAME}'
@@ -1151,6 +1153,15 @@ do
   | -silent | --silent | --silen | --sile | --sil)
     silent=yes ;;
 
+  -runstatedir | --runstatedir | --runstatedi | --runstated \
+  | --runstate | --runstat | --runsta | --runst | --runs \
+  | --run | --ru | --r)
+    ac_prev=runstatedir ;;
+  -runstatedir=* | --runstatedir=* | --runstatedi=* | --runstated=* \
+  | --runstate=* | --runstat=* | --runsta=* | --runst=* | --runs=* \
+  | --run=* | --ru=* | --r=*)
+    runstatedir=$ac_optarg ;;
+
   -sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb)
     ac_prev=sbindir ;;
   -sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \
@@ -1288,7 +1299,7 @@ fi
 for ac_var in	exec_prefix prefix bindir sbindir libexecdir datarootdir \
 		datadir sysconfdir sharedstatedir localstatedir includedir \
 		oldincludedir docdir infodir htmldir dvidir pdfdir psdir \
-		libdir localedir mandir
+		libdir localedir mandir runstatedir
 do
   eval ac_val=\$$ac_var
   # Remove trailing slashes.
@@ -1441,6 +1452,7 @@ Fine tuning of the installation directories:
   --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
   --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
   --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
+  --runstatedir=DIR       modifiable per-process data [LOCALSTATEDIR/run]
   --libdir=DIR            object code libraries [EPREFIX/lib]
   --includedir=DIR        C header files [PREFIX/include]
   --oldincludedir=DIR     C header files for non-gcc [/usr/include]
@@ -11494,7 +11506,7 @@ for ac_func in alarm accept4 setitimer getitimer bind_textdomain_codeset chown \
  getgrouplist getgroups getlogin getloadavg getpeername getpgid getpid \
  getpriority getresuid getresgid getpwent getpwnam_r getpwuid_r getspnam getspent getsid getwd \
  if_nameindex \
- initgroups kill killpg lchmod lchown lockf linkat lstat lutimes mmap \
+ initgroups kill killpg lchown lockf linkat lstat lutimes mmap \
  memrchr mbrtowc mkdirat mkfifo \
  madvise mkfifoat mknod mknodat mktime mremap nice openat pathconf pause pipe2 plock poll \
  posix_fallocate posix_fadvise posix_spawn posix_spawnp pread preadv preadv2 \
diff --git a/configure.ac b/configure.ac
index 033a93cd3f3b28..0689c70e1e99a4 100644
--- a/configure.ac
+++ b/configure.ac
@@ -3544,7 +3544,7 @@ AC_CHECK_FUNCS(alarm accept4 setitimer getitimer bind_textdomain_codeset chown \
  getgrouplist getgroups getlogin getloadavg getpeername getpgid getpid \
  getpriority getresuid getresgid getpwent getpwnam_r getpwuid_r getspnam getspent getsid getwd \
  if_nameindex \
- initgroups kill killpg lchmod lchown lockf linkat lstat lutimes mmap \
+ initgroups kill killpg lchown lockf linkat lstat lutimes mmap \
  memrchr mbrtowc mkdirat mkfifo \
  madvise mkfifoat mknod mknodat mktime mremap nice openat pathconf pause pipe2 plock poll \
  posix_fallocate posix_fadvise posix_spawn posix_spawnp pread preadv preadv2 \

From 2a4a982cbccf66dd8d29439dbd232c79fe3ec44e Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 06:02:42 -0700
Subject: [PATCH 0513/2163] bpo-37283: Ensure command-line and unattend.xml
 setting override previously detected states in Windows installer (GH-15759)

(cherry picked from commit 3a0ddbcdfcbc0f4372905fabf81e093f1b043e99)

Co-authored-by: Steve Dower 
---
 .../2019-09-09-12-22-23.bpo-37283.8NvOkU.rst  |  2 ++
 .../PythonBootstrapperApplication.cpp         | 30 +++++++++++++++----
 Tools/msi/bundle/bundle.wxs                   |  3 +-
 3 files changed, 27 insertions(+), 8 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Windows/2019-09-09-12-22-23.bpo-37283.8NvOkU.rst

diff --git a/Misc/NEWS.d/next/Windows/2019-09-09-12-22-23.bpo-37283.8NvOkU.rst b/Misc/NEWS.d/next/Windows/2019-09-09-12-22-23.bpo-37283.8NvOkU.rst
new file mode 100644
index 00000000000000..973047839301e7
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-09-09-12-22-23.bpo-37283.8NvOkU.rst
@@ -0,0 +1,2 @@
+Ensure command-line and unattend.xml setting override previously detected
+states in Windows installer.
diff --git a/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
index 7cd8fb8e058303..570798de1aafa8 100644
--- a/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
+++ b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
@@ -727,9 +727,13 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication {
                 BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Failed to load AssociateFiles state: error code 0x%08X", hr);
             }
 
-            _engine->SetVariableNumeric(L"Include_launcher", 1);
+            LONGLONG includeLauncher;
+            if (FAILED(BalGetNumericVariable(L"Include_launcher", &includeLauncher))
+                || includeLauncher == -1) {
+                _engine->SetVariableNumeric(L"Include_launcher", 1);
+                _engine->SetVariableNumeric(L"InstallLauncherAllUsers", fPerMachine ? 1 : 0);
+            }
             _engine->SetVariableNumeric(L"DetectedOldLauncher", 1);
-            _engine->SetVariableNumeric(L"InstallLauncherAllUsers", fPerMachine ? 1 : 0);
         }
         return CheckCanceled() ? IDCANCEL : IDNOACTION;
     }
@@ -796,6 +800,12 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication {
             }
         }
 
+        LONGLONG includeLauncher;
+        if (SUCCEEDED(BalGetNumericVariable(L"Include_launcher", &includeLauncher))
+            && includeLauncher != -1) {
+            detectedLauncher = FALSE;
+        }
+
         if (detectedLauncher) {
             /* When we detect the current version of the launcher. */
             _engine->SetVariableNumeric(L"Include_launcher", 1);
@@ -819,6 +829,14 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication {
             _baFunction->OnDetectComplete();
         }
 
+        if (SUCCEEDED(hrStatus)) {
+            LONGLONG includeLauncher;
+            if (SUCCEEDED(BalGetNumericVariable(L"Include_launcher", &includeLauncher))
+                && includeLauncher == -1) {
+                _engine->SetVariableNumeric(L"Include_launcher", 1);
+            }
+        }
+
         if (SUCCEEDED(hrStatus)) {
             hrStatus = EvaluateConditions();
         }
@@ -1451,6 +1469,10 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication {
         hr = ParseOverridableVariablesFromXml(pixdManifest);
         BalExitOnFailure(hr, "Failed to read overridable variables.");
 
+        if (_command.action == BOOTSTRAPPER_ACTION_MODIFY) {
+            LoadOptionalFeatureStates(_engine);
+        }
+
         hr = ParseVariablesFromUnattendXml();
         ExitOnFailure(hr, "Failed to read unattend.ini file.");
 
@@ -1478,10 +1500,6 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication {
         hr = UpdateUIStrings(_command.action);
         BalExitOnFailure(hr, "Failed to load UI strings.");
 
-        if (_command.action == BOOTSTRAPPER_ACTION_MODIFY) {
-            LoadOptionalFeatureStates(_engine);
-        }
-
         GetBundleFileVersion();
         // don't fail if we couldn't get the version info; best-effort only
     LExit:
diff --git a/Tools/msi/bundle/bundle.wxs b/Tools/msi/bundle/bundle.wxs
index f6cff6fc351dc1..ddd6870f625526 100644
--- a/Tools/msi/bundle/bundle.wxs
+++ b/Tools/msi/bundle/bundle.wxs
@@ -71,11 +71,10 @@
     
     
     
+    
     
-    
     
     
-    
     
     
     

From 248387f3f1182087da8f432c989ac6f18315b50f Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 06:28:07 -0700
Subject: [PATCH 0514/2163] bpo-37589: Add a few missing dependencies on .h
 files in the Makefile. (GH-15757)

The missing dependencies prevented incremental builds from working when you touched any
of these files. Based on GH-14758 by @vemakereporter.
(cherry picked from commit b4612f5d54aced5bc37f1b85bf50b4cafa2480f0)

Co-authored-by: T. Wouters 
---
 Makefile.pre.in | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/Makefile.pre.in b/Makefile.pre.in
index 6a9f4b52704d22..b02f78412f167b 100644
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -305,8 +305,9 @@ POBJS=		\
 PARSER_OBJS=	$(POBJS) Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o
 
 PARSER_HEADERS= \
-		$(srcdir)/Parser/parser.h \
+		$(srcdir)/Include/grammar.h \
 		$(srcdir)/Include/parsetok.h \
+		$(srcdir)/Parser/parser.h \
 		$(srcdir)/Parser/tokenizer.h
 
 ##########################################################################
@@ -865,7 +866,7 @@ regen-symbol: $(srcdir)/Include/graminit.h
 		$(srcdir)/Include/graminit.h \
 		$(srcdir)/Lib/symbol.py
 
-Python/compile.o Python/symtable.o Python/ast_unparse.o Python/ast.o: $(srcdir)/Include/graminit.h $(srcdir)/Include/Python-ast.h
+Python/compile.o Python/symtable.o Python/ast_unparse.o Python/ast.o Python/future.o Parser/parsetok.o: $(srcdir)/Include/graminit.h $(srcdir)/Include/Python-ast.h
 
 Python/getplatform.o: $(srcdir)/Python/getplatform.c
 		$(CC) -c $(PY_CORE_CFLAGS) -DPLATFORM='"$(MACHDEP)"' -o $@ $(srcdir)/Python/getplatform.c
@@ -965,11 +966,11 @@ PYTHON_HEADERS= \
 		$(srcdir)/Include/abstract.h \
 		$(srcdir)/Include/asdl.h \
 		$(srcdir)/Include/ast.h \
-		$(srcdir)/Include/bltinmodule.h \
 		$(srcdir)/Include/bitset.h \
+		$(srcdir)/Include/bltinmodule.h \
 		$(srcdir)/Include/boolobject.h \
-		$(srcdir)/Include/bytes_methods.h \
 		$(srcdir)/Include/bytearrayobject.h \
+		$(srcdir)/Include/bytes_methods.h \
 		$(srcdir)/Include/bytesobject.h \
 		$(srcdir)/Include/cellobject.h \
 		$(srcdir)/Include/ceval.h \
@@ -978,6 +979,7 @@ PYTHON_HEADERS= \
 		$(srcdir)/Include/codecs.h \
 		$(srcdir)/Include/compile.h \
 		$(srcdir)/Include/complexobject.h \
+		$(srcdir)/Include/context.h \
 		$(srcdir)/Include/descrobject.h \
 		$(srcdir)/Include/dictobject.h \
 		$(srcdir)/Include/dtoa.h \
@@ -1007,6 +1009,7 @@ PYTHON_HEADERS= \
 		$(srcdir)/Include/node.h \
 		$(srcdir)/Include/object.h \
 		$(srcdir)/Include/objimpl.h \
+		$(srcdir)/Include/odictobject.h \
 		$(srcdir)/Include/opcode.h \
 		$(srcdir)/Include/osdefs.h \
 		$(srcdir)/Include/osmodule.h \
@@ -1021,15 +1024,15 @@ PYTHON_HEADERS= \
 		$(srcdir)/Include/pyfpe.h \
 		$(srcdir)/Include/pyhash.h \
 		$(srcdir)/Include/pylifecycle.h \
-		$(srcdir)/Include/pymath.h \
+		$(srcdir)/Include/pymacconfig.h \
 		$(srcdir)/Include/pymacro.h \
+		$(srcdir)/Include/pymath.h \
 		$(srcdir)/Include/pymem.h \
 		$(srcdir)/Include/pyport.h \
 		$(srcdir)/Include/pystate.h \
-		$(srcdir)/Include/context.h \
 		$(srcdir)/Include/pystrcmp.h \
-		$(srcdir)/Include/pystrtod.h \
 		$(srcdir)/Include/pystrhex.h \
+		$(srcdir)/Include/pystrtod.h \
 		$(srcdir)/Include/pythonrun.h \
 		$(srcdir)/Include/pythread.h \
 		$(srcdir)/Include/pytime.h \
@@ -1040,6 +1043,7 @@ PYTHON_HEADERS= \
 		$(srcdir)/Include/structseq.h \
 		$(srcdir)/Include/symtable.h \
 		$(srcdir)/Include/sysmodule.h \
+		$(srcdir)/Include/token.h \
 		$(srcdir)/Include/traceback.h \
 		$(srcdir)/Include/tracemalloc.h \
 		$(srcdir)/Include/tupleobject.h \

From 5e053eb98eb0d65a8e0f00b3641f9907198aace3 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 06:42:50 -0700
Subject: [PATCH 0515/2163] Fix typo in math.prod example (GH-15614)

(cherry picked from commit 1a8de82d3a30ecc7ed18a5ad51a0e17417ebfb89)

Co-authored-by: Ashwin Vishnu <9155111+ashwinvis@users.noreply.github.com>
---
 Doc/whatsnew/3.8.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 4a1362d943c809..2b4eb63d61f658 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -767,7 +767,7 @@ numbers::
 
     >>> prior = 0.8
     >>> likelihoods = [0.625, 0.84, 0.30]
-    >>> (link: http://math.prod) math.prod(likelihoods, start=prior)
+    >>> math.prod(likelihoods, start=prior)
     0.126
 
 (Contributed by Pablo Galindo in :issue:`35606`)

From 5d695b6b7bcccf5f028cdacd986096de15bc0ca6 Mon Sep 17 00:00:00 2001
From: Steve Dower 
Date: Mon, 9 Sep 2019 06:48:22 -0700
Subject: [PATCH 0516/2163] bpo-37702: Fix SSL's certificate-store leak on
 Windows (GH-15632)

ssl_collect_certificates function in _ssl.c has a memory leak.
Calling CertOpenStore() and CertAddStoreToCollection(), a store's refcnt gets incremented by 2.
But CertCloseStore() is called only once and the refcnt leaves 1.
---
 .../next/Windows/2019-07-29-16-49-31.bpo-37702.Lj2f5e.rst       | 2 ++
 Modules/_ssl.c                                                  | 1 +
 2 files changed, 3 insertions(+)
 create mode 100644 Misc/NEWS.d/next/Windows/2019-07-29-16-49-31.bpo-37702.Lj2f5e.rst

diff --git a/Misc/NEWS.d/next/Windows/2019-07-29-16-49-31.bpo-37702.Lj2f5e.rst b/Misc/NEWS.d/next/Windows/2019-07-29-16-49-31.bpo-37702.Lj2f5e.rst
new file mode 100644
index 00000000000000..67d53d4c46276b
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-07-29-16-49-31.bpo-37702.Lj2f5e.rst
@@ -0,0 +1,2 @@
+Fix memory leak on Windows in creating an SSLContext object or
+running urllib.request.urlopen('https://...').
\ No newline at end of file
diff --git a/Modules/_ssl.c b/Modules/_ssl.c
index 089aa3b24a0261..6f91b488252851 100644
--- a/Modules/_ssl.c
+++ b/Modules/_ssl.c
@@ -5581,6 +5581,7 @@ ssl_collect_certificates(const char *store_name)
             if (result) {
                 ++storesAdded;
             }
+            CertCloseStore(hSystemStore, 0);  /* flag must be 0 */
         }
     }
     if (storesAdded == 0) {

From 44729c9f5198211faf533da49fa0fa26693a1993 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 07:23:37 -0700
Subject: [PATCH 0517/2163] bpo-32587: Fixes unsafe downcast in PC/winreg.c
 (GH-15766)

(cherry picked from commit ef66f31ce21cd759cc0c618c5c42ba6da0a06834)

Co-authored-by: Steve Dower 
---
 PC/winreg.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/PC/winreg.c b/PC/winreg.c
index 37bc2c72dcdee1..72a7c380beefe1 100644
--- a/PC/winreg.c
+++ b/PC/winreg.c
@@ -772,7 +772,7 @@ Reg2Py(BYTE *retDataBuf, DWORD retDataSize, DWORD typ)
                         return NULL;
                     }
                     PyList_SET_ITEM(obData, index, uni);
-                    len -= slen + 1;
+                    len -= Py_SAFE_DOWNCAST(slen + 1, size_t, int);
                 }
                 PyMem_Free(str);
 

From b150d0bf1bb4c3203bb3293625e32aed01b25887 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 07:42:35 -0700
Subject: [PATCH 0518/2163] bpo-38037: Fix reference counters in signal module
 (GH-15753)

(cherry picked from commit 77643c486fd22d8030e0d82c13012684b4ab6df1)

Co-authored-by: animalize 
---
 .../2019-09-09-18-39-23.bpo-38037.B0UgFU.rst  |  1 +
 Modules/signalmodule.c                        | 25 +++++++++++++------
 2 files changed, 18 insertions(+), 8 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2019-09-09-18-39-23.bpo-38037.B0UgFU.rst

diff --git a/Misc/NEWS.d/next/Library/2019-09-09-18-39-23.bpo-38037.B0UgFU.rst b/Misc/NEWS.d/next/Library/2019-09-09-18-39-23.bpo-38037.B0UgFU.rst
new file mode 100644
index 00000000000000..607673bd9994b3
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-09-18-39-23.bpo-38037.B0UgFU.rst
@@ -0,0 +1 @@
+Fix reference counters in the :mod:`signal` module.
diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c
index 7698984ff3afe1..9aca70599689b6 100644
--- a/Modules/signalmodule.c
+++ b/Modules/signalmodule.c
@@ -1329,7 +1329,7 @@ static struct PyModuleDef signalmodule = {
 PyMODINIT_FUNC
 PyInit__signal(void)
 {
-    PyObject *m, *d, *x;
+    PyObject *m, *d;
     int i;
 
     /* Create the module and add the functions */
@@ -1350,13 +1350,17 @@ PyInit__signal(void)
     /* Add some symbolic constants to the module */
     d = PyModule_GetDict(m);
 
-    x = DefaultHandler = PyLong_FromVoidPtr((void *)SIG_DFL);
-    if (PyModule_AddObject(m, "SIG_DFL", x))
+    DefaultHandler = PyLong_FromVoidPtr((void *)SIG_DFL);
+    if (!DefaultHandler ||
+        PyDict_SetItemString(d, "SIG_DFL", DefaultHandler) < 0) {
         goto finally;
+    }
 
-    x = IgnoreHandler = PyLong_FromVoidPtr((void *)SIG_IGN);
-    if (PyModule_AddObject(m, "SIG_IGN", x))
+    IgnoreHandler = PyLong_FromVoidPtr((void *)SIG_IGN);
+    if (!IgnoreHandler ||
+        PyDict_SetItemString(d, "SIG_IGN", IgnoreHandler) < 0) {
         goto finally;
+    }
 
     if (PyModule_AddIntMacro(m, NSIG))
         goto finally;
@@ -1374,8 +1378,8 @@ PyInit__signal(void)
          goto finally;
 #endif
 
-    x = IntHandler = PyDict_GetItemString(d, "default_int_handler");
-    if (!x)
+    IntHandler = PyDict_GetItemString(d, "default_int_handler");
+    if (!IntHandler)
         goto finally;
     Py_INCREF(IntHandler);
 
@@ -1568,8 +1572,10 @@ PyInit__signal(void)
 #if defined (HAVE_SETITIMER) || defined (HAVE_GETITIMER)
     ItimerError = PyErr_NewException("signal.ItimerError",
             PyExc_OSError, NULL);
-    if (PyModule_AddObject(m, "ItimerError", ItimerError))
+    if (!ItimerError ||
+        PyDict_SetItemString(d, "ItimerError", ItimerError) < 0) {
         goto finally;
+    }
 #endif
 
 #ifdef CTRL_C_EVENT
@@ -1615,6 +1621,9 @@ finisignal(void)
     Py_CLEAR(IntHandler);
     Py_CLEAR(DefaultHandler);
     Py_CLEAR(IgnoreHandler);
+#ifdef HAVE_GETITIMER
+    Py_CLEAR(ItimerError);
+#endif
 }
 
 

From cabcbbe7a518d53cd7dbb915ceccf4970c2d0ea5 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 07:52:34 -0700
Subject: [PATCH 0519/2163] bpo-36797: Fix a dead link in  Doc/distutils/apiref
 (GH-15700) (GH-15704)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

https://bugs.python.org/issue36797
(cherry picked from commit 2c2b561967c1916855399f809e30ae0ba7e09ae2)

Co-authored-by: Miro HronÄŤok 
---
 Doc/distutils/apiref.rst                                      | 4 ++--
 .../Documentation/2019-09-05-14-47-51.bpo-36797.KN9Ga5.rst    | 1 +
 2 files changed, 3 insertions(+), 2 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Documentation/2019-09-05-14-47-51.bpo-36797.KN9Ga5.rst

diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst
index 937f19f57be95a..a42d2d3559f2b2 100644
--- a/Doc/distutils/apiref.rst
+++ b/Doc/distutils/apiref.rst
@@ -6,12 +6,12 @@ API Reference
 
 .. seealso::
 
-   `New and changed setup.py arguments in setuptools `_
+   `New and changed setup.py arguments in setuptools`_
       The ``setuptools`` project adds new capabilities to the ``setup`` function
       and other APIs, makes the API consistent across different Python versions,
       and is hence recommended over using ``distutils`` directly.
 
-.. _setuptools-setup-py: https://setuptools.readthedocs.io/en/latest/setuptools.html#new-and-changed-setup-keywords
+.. _New and changed setup.py arguments in setuptools: https://setuptools.readthedocs.io/en/latest/setuptools.html#new-and-changed-setup-keywords
 
 .. include:: ./_setuptools_disclaimer.rst
 
diff --git a/Misc/NEWS.d/next/Documentation/2019-09-05-14-47-51.bpo-36797.KN9Ga5.rst b/Misc/NEWS.d/next/Documentation/2019-09-05-14-47-51.bpo-36797.KN9Ga5.rst
new file mode 100644
index 00000000000000..62a3c176b1c7f9
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2019-09-05-14-47-51.bpo-36797.KN9Ga5.rst
@@ -0,0 +1 @@
+Fix a dead link in the distutils API Reference.

From 3bd4bed78a0b068e28bcf2242d33aed227c2532c Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 08:01:15 -0700
Subject: [PATCH 0520/2163] bpo-34596: Fallback to a default reason when
 @unittest.skip is uncalled (GH-9082) (#15781)

* bpo-34596: Fallback to a default reason when @unittest.skip is uncalled

* Change default reason to empty string

* Fix rst formatting of NEWS entry
(cherry picked from commit d5fd75c53fad7049fc640c9a6162d35f0c5bea03)

Co-authored-by: Naitree Zhu 
---
 Lib/unittest/case.py                                  |  5 +++++
 Lib/unittest/test/test_skipping.py                    | 11 +++++++++++
 .../Tests/2018-09-07-01-18-27.bpo-34596.r2-EGd.rst    |  2 ++
 3 files changed, 18 insertions(+)
 create mode 100644 Misc/NEWS.d/next/Tests/2018-09-07-01-18-27.bpo-34596.r2-EGd.rst

diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py
index b363c635100726..b639c64d02a7aa 100644
--- a/Lib/unittest/case.py
+++ b/Lib/unittest/case.py
@@ -10,6 +10,7 @@
 import collections
 import contextlib
 import traceback
+import types
 
 from . import result
 from .util import (strclass, safe_repr, _count_diff_all_purpose,
@@ -122,6 +123,10 @@ def skip_wrapper(*args, **kwargs):
         test_item.__unittest_skip__ = True
         test_item.__unittest_skip_why__ = reason
         return test_item
+    if isinstance(reason, types.FunctionType):
+        test_item = reason
+        reason = ''
+        return decorator(test_item)
     return decorator
 
 def skipIf(condition, reason):
diff --git a/Lib/unittest/test/test_skipping.py b/Lib/unittest/test/test_skipping.py
index 71f7b70e479d2e..1c178a95f750ff 100644
--- a/Lib/unittest/test/test_skipping.py
+++ b/Lib/unittest/test/test_skipping.py
@@ -255,6 +255,17 @@ def test_1(self):
         suite.run(result)
         self.assertEqual(result.skipped, [(test, "testing")])
 
+    def test_skip_without_reason(self):
+        class Foo(unittest.TestCase):
+            @unittest.skip
+            def test_1(self):
+                pass
+
+        result = unittest.TestResult()
+        test = Foo("test_1")
+        suite = unittest.TestSuite([test])
+        suite.run(result)
+        self.assertEqual(result.skipped, [(test, "")])
 
 if __name__ == "__main__":
     unittest.main()
diff --git a/Misc/NEWS.d/next/Tests/2018-09-07-01-18-27.bpo-34596.r2-EGd.rst b/Misc/NEWS.d/next/Tests/2018-09-07-01-18-27.bpo-34596.r2-EGd.rst
new file mode 100644
index 00000000000000..156e8aa8945de4
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2018-09-07-01-18-27.bpo-34596.r2-EGd.rst
@@ -0,0 +1,2 @@
+Fallback to a default reason when :func:`unittest.skip` is uncalled. Patch by
+Naitree Zhu.

From 0d4396c04cba5ac2b66fdaa23c01db84b1b54227 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 08:20:36 -0700
Subject: [PATCH 0521/2163] bpo-38059: Using sys.exit() over exit() in
 inspect.py (GH-15666)

Constants added by the site module like exit() "should not be used in programs"
(cherry picked from commit e3c59a75279b0df4e7553d6f0031e202de434e43)

Co-authored-by: Alan Yee 
---
 Lib/inspect.py                                                | 4 ++--
 .../next/Library/2019-09-08-11-36-50.bpo-38059.8SA6co.rst     | 1 +
 2 files changed, 3 insertions(+), 2 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2019-09-08-11-36-50.bpo-38059.8SA6co.rst

diff --git a/Lib/inspect.py b/Lib/inspect.py
index a616f2d49b7d96..0a57749ccdd458 100644
--- a/Lib/inspect.py
+++ b/Lib/inspect.py
@@ -3118,7 +3118,7 @@ def _main():
                                                     type(exc).__name__,
                                                     exc)
         print(msg, file=sys.stderr)
-        exit(2)
+        sys.exit(2)
 
     if has_attrs:
         parts = attrs.split(".")
@@ -3128,7 +3128,7 @@ def _main():
 
     if module.__name__ in sys.builtin_module_names:
         print("Can't get info for builtin modules.", file=sys.stderr)
-        exit(1)
+        sys.exit(1)
 
     if args.details:
         print('Target: {}'.format(target))
diff --git a/Misc/NEWS.d/next/Library/2019-09-08-11-36-50.bpo-38059.8SA6co.rst b/Misc/NEWS.d/next/Library/2019-09-08-11-36-50.bpo-38059.8SA6co.rst
new file mode 100644
index 00000000000000..001952ae126137
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-08-11-36-50.bpo-38059.8SA6co.rst
@@ -0,0 +1 @@
+inspect.py now uses sys.exit() instead of exit()

From b6ef8f2beb90678a4a54218d6169040afbbf9fe1 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 09:12:01 -0700
Subject: [PATCH 0522/2163] bpo-37876: Tests for ROT-13 codec (GH-15314)

The Rot-13 codec is for educational use but does not have unit tests,
dragging down test coverage. This adds a few very simple tests.
(cherry picked from commit b3b48c81f09d1472010937f1331c5a208a2a2d48)

Co-authored-by: Zeth 
---
 Lib/test/test_codecs.py                       | 37 +++++++++++++++++++
 .../2019-08-16-16-15-14.bpo-37876.m3k1w3.rst  |  1 +
 2 files changed, 38 insertions(+)
 create mode 100644 Misc/NEWS.d/next/Tests/2019-08-16-16-15-14.bpo-37876.m3k1w3.rst

diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
index ba7f4847468a3e..b37525bf660430 100644
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -3343,5 +3343,42 @@ def test_decode_unsupported_error_handler(self):
         self.assertEqual(str(cm.exception), 'unsupported error handler')
 
 
+class Rot13Test(unittest.TestCase):
+    """Test the educational ROT-13 codec."""
+    def test_encode(self):
+        ciphertext = codecs.encode("Caesar liked ciphers", 'rot-13')
+        self.assertEqual(ciphertext, 'Pnrfne yvxrq pvcuref')
+
+    def test_decode(self):
+        plaintext = codecs.decode('Rg gh, Oehgr?', 'rot-13')
+        self.assertEqual(plaintext, 'Et tu, Brute?')
+
+    def test_incremental_encode(self):
+        encoder = codecs.getincrementalencoder('rot-13')()
+        ciphertext = encoder.encode('ABBA nag Cheryl Baker')
+        self.assertEqual(ciphertext, 'NOON ant Purely Onxre')
+
+    def test_incremental_decode(self):
+        decoder = codecs.getincrementaldecoder('rot-13')()
+        plaintext = decoder.decode('terra Ares envy tha')
+        self.assertEqual(plaintext, 'green Nerf rail gun')
+
+
+class Rot13UtilTest(unittest.TestCase):
+    """Test the ROT-13 codec via rot13 function,
+    i.e. the user has done something like:
+    $ echo "Hello World" | python -m encodings.rot_13
+    """
+    def test_rot13_func(self):
+        infile = io.StringIO('Gb or, be abg gb or, gung vf gur dhrfgvba')
+        outfile = io.StringIO()
+        encodings.rot_13.rot13(infile, outfile)
+        outfile.seek(0)
+        plain_text = outfile.read()
+        self.assertEqual(
+            plain_text,
+            'To be, or not to be, that is the question')
+
+
 if __name__ == "__main__":
     unittest.main()
diff --git a/Misc/NEWS.d/next/Tests/2019-08-16-16-15-14.bpo-37876.m3k1w3.rst b/Misc/NEWS.d/next/Tests/2019-08-16-16-15-14.bpo-37876.m3k1w3.rst
new file mode 100644
index 00000000000000..45702fc6351032
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-08-16-16-15-14.bpo-37876.m3k1w3.rst
@@ -0,0 +1 @@
+Add tests for ROT-13 codec.
\ No newline at end of file

From 78d15faf6c522619098e94be3e7f6d88a9e61123 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 09:24:16 -0700
Subject: [PATCH 0523/2163] bpo-38006: Avoid closure in
 weakref.WeakValueDictionary (GH-15641)

weakref.WeakValueDictionary defines a local remove() function used as
callback for weak references. This function was created with a
closure.  Modify the implementation to avoid the closure.
(cherry picked from commit a2af05a0d3f0da06b8d432f52efa3ecf29038532)

Co-authored-by: Victor Stinner 
---
 Lib/test/test_weakref.py                                     | 5 +++++
 Lib/weakref.py                                               | 4 ++--
 .../next/Library/2019-09-02-13-37-27.bpo-38006.Y7vA0Q.rst    | 3 +++
 3 files changed, 10 insertions(+), 2 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2019-09-02-13-37-27.bpo-38006.Y7vA0Q.rst

diff --git a/Lib/test/test_weakref.py b/Lib/test/test_weakref.py
index 6f15c03ac5292f..d9e1b201f65676 100644
--- a/Lib/test/test_weakref.py
+++ b/Lib/test/test_weakref.py
@@ -1785,6 +1785,11 @@ def test_threaded_weak_value_dict_deepcopy(self):
         # copying should not result in a crash.
         self.check_threaded_weak_dict_copy(weakref.WeakValueDictionary, True)
 
+    @support.cpython_only
+    def test_remove_closure(self):
+        d = weakref.WeakValueDictionary()
+        self.assertIsNone(d._remove.__closure__)
+
 
 from test import mapping_tests
 
diff --git a/Lib/weakref.py b/Lib/weakref.py
index 8d71af653b7ec4..9d7008947f06d4 100644
--- a/Lib/weakref.py
+++ b/Lib/weakref.py
@@ -108,12 +108,12 @@ def remove(wr, selfref=ref(self), _atomic_removal=_remove_dead_weakref):
                 else:
                     # Atomic removal is necessary since this function
                     # can be called asynchronously by the GC
-                    _atomic_removal(d, wr.key)
+                    _atomic_removal(self.data, wr.key)
         self._remove = remove
         # A list of keys to be removed
         self._pending_removals = []
         self._iterating = set()
-        self.data = d = {}
+        self.data = {}
         self.update(other, **kw)
 
     def _commit_removals(self):
diff --git a/Misc/NEWS.d/next/Library/2019-09-02-13-37-27.bpo-38006.Y7vA0Q.rst b/Misc/NEWS.d/next/Library/2019-09-02-13-37-27.bpo-38006.Y7vA0Q.rst
new file mode 100644
index 00000000000000..ff064ad3f1a492
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-02-13-37-27.bpo-38006.Y7vA0Q.rst
@@ -0,0 +1,3 @@
+weakref.WeakValueDictionary defines a local remove() function used as
+callback for weak references. This function was created with a closure.
+Modify the implementation to avoid the closure.

From 2d5594fac21a81a06f82c3605318dfa96e72398f Mon Sep 17 00:00:00 2001
From: Steve Dower 
Date: Mon, 9 Sep 2019 09:45:18 -0700
Subject: [PATCH 0524/2163] bpo-20490: Improve circular import error message
 (GH-15308)

---
 Lib/test/test_import/__init__.py                  | 10 ++++++++++
 .../data/circular_imports/from_cycle1.py          |  2 ++
 .../data/circular_imports/from_cycle2.py          |  2 ++
 .../2019-08-15-12-48-36.bpo-20490.-hXeEn.rst      |  2 ++
 Python/ceval.c                                    | 15 +++++++++++----
 5 files changed, 27 insertions(+), 4 deletions(-)
 create mode 100644 Lib/test/test_import/data/circular_imports/from_cycle1.py
 create mode 100644 Lib/test/test_import/data/circular_imports/from_cycle2.py
 create mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-08-15-12-48-36.bpo-20490.-hXeEn.rst

diff --git a/Lib/test/test_import/__init__.py b/Lib/test/test_import/__init__.py
index 50406d9aa1d9cd..7c24f0e502323e 100644
--- a/Lib/test/test_import/__init__.py
+++ b/Lib/test/test_import/__init__.py
@@ -1326,6 +1326,16 @@ def test_crossreference2(self):
         self.assertIn('partially initialized module', errmsg)
         self.assertIn('circular import', errmsg)
 
+    def test_circular_from_import(self):
+        with self.assertRaises(ImportError) as cm:
+            import test.test_import.data.circular_imports.from_cycle1
+        self.assertIn(
+            "cannot import name 'b' from partially initialized module "
+            "'test.test_import.data.circular_imports.from_cycle1' "
+            "(most likely due to a circular import)",
+            str(cm.exception),
+        )
+
 
 if __name__ == '__main__':
     # Test needs to be a package, so we can do relative imports.
diff --git a/Lib/test/test_import/data/circular_imports/from_cycle1.py b/Lib/test/test_import/data/circular_imports/from_cycle1.py
new file mode 100644
index 00000000000000..aacfd5f46fca36
--- /dev/null
+++ b/Lib/test/test_import/data/circular_imports/from_cycle1.py
@@ -0,0 +1,2 @@
+from .from_cycle2 import a
+b = 1
diff --git a/Lib/test/test_import/data/circular_imports/from_cycle2.py b/Lib/test/test_import/data/circular_imports/from_cycle2.py
new file mode 100644
index 00000000000000..62a66e1cfd028e
--- /dev/null
+++ b/Lib/test/test_import/data/circular_imports/from_cycle2.py
@@ -0,0 +1,2 @@
+from .from_cycle1 import b
+a = 1
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-08-15-12-48-36.bpo-20490.-hXeEn.rst b/Misc/NEWS.d/next/Core and Builtins/2019-08-15-12-48-36.bpo-20490.-hXeEn.rst
new file mode 100644
index 00000000000000..dfee480e1aa706
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-08-15-12-48-36.bpo-20490.-hXeEn.rst	
@@ -0,0 +1,2 @@
+Improve import error message for partially initialized module on circular
+``from`` imports - by Anthony Sottile.
diff --git a/Python/ceval.c b/Python/ceval.c
index 546a4264d8ada2..07ec3293adf1c4 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -5236,10 +5236,17 @@ import_from(PyThreadState *tstate, PyObject *v, PyObject *name)
         PyErr_SetImportError(errmsg, pkgname, NULL);
     }
     else {
-        errmsg = PyUnicode_FromFormat(
-            "cannot import name %R from %R (%S)",
-            name, pkgname_or_unknown, pkgpath
-        );
+        _Py_IDENTIFIER(__spec__);
+        PyObject *spec = _PyObject_GetAttrId(v, &PyId___spec__);
+        Py_XINCREF(spec);
+        const char *fmt =
+            _PyModuleSpec_IsInitializing(spec) ?
+            "cannot import name %R from partially initialized module %R "
+            "(most likely due to a circular import) (%S)" :
+            "cannot import name %R from %R (%S)";
+        Py_XDECREF(spec);
+
+        errmsg = PyUnicode_FromFormat(fmt, name, pkgname_or_unknown, pkgpath);
         /* NULL checks for errmsg and pkgname done by PyErr_SetImportError. */
         PyErr_SetImportError(errmsg, pkgname, pkgpath);
     }

From 5731172bb1e958b1d80b18eaf88d3f2f93cfccdd Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 10:18:09 -0700
Subject: [PATCH 0525/2163] bpo-38070: visit_decref() calls _PyObject_IsFreed()
 (GH-15782)

In debug mode, visit_decref() now calls _PyObject_IsFreed() to ensure
that the object is not freed. If it's freed, the program fails with
an assertion error and Python dumps informations about the freed
object.
(cherry picked from commit d91d4de31745fc1ed4c7e6c208917827c9c472b6)

Co-authored-by: Victor Stinner 
---
 Modules/gcmodule.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index 2741edc0a0211c..b2ee56623cd484 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -376,6 +376,8 @@ static int
 visit_decref(PyObject *op, void *data)
 {
     assert(op != NULL);
+    _PyObject_ASSERT(op, !_PyObject_IsFreed(op));
+
     if (PyObject_IS_GC(op)) {
         PyGC_Head *gc = AS_GC(op);
         /* We're only interested in gc_refs for objects in the

From 0468a85cc4a3bd870f7e119ef8e6a2af0177ac56 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 10:18:51 -0700
Subject: [PATCH 0526/2163] Clarify that shutil's copy functions can accept
 path-like values (GH-15141)

(cherry picked from commit 9488a5289de2ceecdfd2098cd70d215f96c4e745)

Co-authored-by: Boris Verhovsky 
---
 Doc/library/shutil.rst | 8 +++++---
 Lib/shutil.py          | 3 ++-
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst
index a51e068ebd593a..88c5f62dd4c99f 100644
--- a/Doc/library/shutil.rst
+++ b/Doc/library/shutil.rst
@@ -52,7 +52,7 @@ Directory and files operations
 
    Copy the contents (no metadata) of the file named *src* to a file named
    *dst* and return *dst* in the most efficient way possible.
-   *src* and *dst* are path names given as strings.
+   *src* and *dst* are path-like objects or path names given as strings.
 
    *dst* must be the complete target file name; look at :func:`~shutil.copy`
    for a copy that accepts a target directory path.  If *src* and *dst*
@@ -92,7 +92,8 @@ Directory and files operations
 .. function:: copymode(src, dst, *, follow_symlinks=True)
 
    Copy the permission bits from *src* to *dst*.  The file contents, owner, and
-   group are unaffected.  *src* and *dst* are path names given as strings.
+   group are unaffected.  *src* and *dst* are path-like objects or path names
+   given as strings.
    If *follow_symlinks* is false, and both *src* and *dst* are symbolic links,
    :func:`copymode` will attempt to modify the mode of *dst* itself (rather
    than the file it points to).  This functionality is not available on every
@@ -108,7 +109,8 @@ Directory and files operations
    Copy the permission bits, last access time, last modification time, and
    flags from *src* to *dst*.  On Linux, :func:`copystat` also copies the
    "extended attributes" where possible.  The file contents, owner, and
-   group are unaffected.  *src* and *dst* are path names given as strings.
+   group are unaffected.  *src* and *dst* are path-like objects or path
+   names given as strings.
 
    If *follow_symlinks* is false, and *src* and *dst* both
    refer to symbolic links, :func:`copystat` will operate on
diff --git a/Lib/shutil.py b/Lib/shutil.py
index 561ad71508cc8d..5c1255a6713de6 100644
--- a/Lib/shutil.py
+++ b/Lib/shutil.py
@@ -330,7 +330,8 @@ def copystat(src, dst, *, follow_symlinks=True):
     Copy the permission bits, last access time, last modification time, and
     flags from `src` to `dst`. On Linux, copystat() also copies the "extended
     attributes" where possible. The file contents, owner, and group are
-    unaffected. `src` and `dst` are path names given as strings.
+    unaffected. `src` and `dst` are path-like objects or path names given as
+    strings.
 
     If the optional flag `follow_symlinks` is not set, symlinks aren't
     followed if and only if both `src` and `dst` are symlinks.

From eadf6b8787e979920c4fb6845797c33d270d2729 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 10:31:31 -0700
Subject: [PATCH 0527/2163] bpo-35803: Document and test dir=PathLike for
 tempfile (GH-11644)

Co-Authored-By: Ammar Askar 
(cherry picked from commit 370138ba9c29595df773cc057d17ea26fb6f21bd)

Co-authored-by: Anthony Sottile 
---
 Doc/library/tempfile.rst                         |  6 ++++++
 Lib/test/test_tempfile.py                        | 16 ++++++++++++++--
 .../2019-01-21-14-30-59.bpo-35803.yae6Lq.rst     |  2 ++
 3 files changed, 22 insertions(+), 2 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Documentation/2019-01-21-14-30-59.bpo-35803.yae6Lq.rst

diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst
index 0793e43df0145c..fff7a7a03eb81d 100644
--- a/Doc/library/tempfile.rst
+++ b/Doc/library/tempfile.rst
@@ -191,6 +191,9 @@ The module defines the following user-callable items:
       *suffix* and *prefix* now accept and default to ``None`` to cause
       an appropriate default value to be used.
 
+   .. versionchanged:: 3.6
+      The *dir* parameter now accepts a :term:`path-like object`.
+
 
 .. function:: mkdtemp(suffix=None, prefix=None, dir=None)
 
@@ -214,6 +217,9 @@ The module defines the following user-callable items:
       *suffix* and *prefix* now accept and default to ``None`` to cause
       an appropriate default value to be used.
 
+   .. versionchanged:: 3.6
+      The *dir* parameter now accepts a :term:`path-like object`.
+
 
 .. function:: gettempdir()
 
diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py
index bd4db839331b49..f995f6c9bfaf00 100644
--- a/Lib/test/test_tempfile.py
+++ b/Lib/test/test_tempfile.py
@@ -3,6 +3,7 @@
 import errno
 import io
 import os
+import pathlib
 import signal
 import sys
 import re
@@ -56,6 +57,9 @@ def test_infer_return_type_multiples_and_none(self):
         with self.assertRaises(TypeError):
             tempfile._infer_return_type(b'', None, '')
 
+    def test_infer_return_type_pathlib(self):
+        self.assertIs(str, tempfile._infer_return_type(pathlib.Path('/')))
+
 
 # Common functionality.
 
@@ -79,8 +83,13 @@ def nameCheck(self, name, dir, pre, suf):
         nsuf  = nbase[len(nbase)-len(suf):]
 
         if dir is not None:
-            self.assertIs(type(name), str if type(dir) is str else bytes,
-                          "unexpected return type")
+            self.assertIs(
+                type(name),
+                str
+                if type(dir) is str or isinstance(dir, os.PathLike) else
+                bytes,
+                "unexpected return type",
+            )
         if pre is not None:
             self.assertIs(type(name), str if type(pre) is str else bytes,
                           "unexpected return type")
@@ -425,6 +434,7 @@ def test_choose_directory(self):
         dir = tempfile.mkdtemp()
         try:
             self.do_create(dir=dir).write(b"blat")
+            self.do_create(dir=pathlib.Path(dir)).write(b"blat")
         finally:
             os.rmdir(dir)
 
@@ -659,6 +669,7 @@ def test_choose_directory(self):
         dir = tempfile.mkdtemp()
         try:
             self.do_create(dir=dir)
+            self.do_create(dir=pathlib.Path(dir))
         finally:
             os.rmdir(dir)
 
@@ -728,6 +739,7 @@ def test_choose_directory(self):
         dir = tempfile.mkdtemp()
         try:
             os.rmdir(self.do_create(dir=dir))
+            os.rmdir(self.do_create(dir=pathlib.Path(dir)))
         finally:
             os.rmdir(dir)
 
diff --git a/Misc/NEWS.d/next/Documentation/2019-01-21-14-30-59.bpo-35803.yae6Lq.rst b/Misc/NEWS.d/next/Documentation/2019-01-21-14-30-59.bpo-35803.yae6Lq.rst
new file mode 100644
index 00000000000000..b8394560e54869
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2019-01-21-14-30-59.bpo-35803.yae6Lq.rst
@@ -0,0 +1,2 @@
+Document and test that ``tempfile`` functions may accept a
+:term:`path-like object` for the ``dir`` argument.  Patch by Anthony Sottile.

From 421ef39968a5e59cf09fe1983f76be1c6866c637 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 10:44:46 -0700
Subject: [PATCH 0528/2163] Fix punctuation in `os.execvpe` docstring.
 (GH-15051)

(cherry picked from commit fb6807b043ab586428225920373e551b0432bc40)

Co-authored-by: Hasan Ramezani 
---
 Lib/os.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Lib/os.py b/Lib/os.py
index 79ff7a22d92e03..200902528cbfe9 100644
--- a/Lib/os.py
+++ b/Lib/os.py
@@ -568,7 +568,7 @@ def execvpe(file, args, env):
     """execvpe(file, args, env)
 
     Execute the executable file (which is searched for along $PATH)
-    with argument list args and environment env , replacing the
+    with argument list args and environment env, replacing the
     current process.
     args may be a list or tuple of strings. """
     _execvpe(file, args, env)

From e83296314583ba9c7952d6539c63aae803a197ae Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 10:59:18 -0700
Subject: [PATCH 0529/2163] bpo-37649: Fix exec_prefix check (GH-14897)

(cherry picked from commit 09090d04ef8d2f4c94157b852d3d530a51e13d22)

Co-authored-by: Orivej Desh 
---
 Modules/getpath.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Modules/getpath.c b/Modules/getpath.c
index 5f807381880283..dc0574851c3b72 100644
--- a/Modules/getpath.c
+++ b/Modules/getpath.c
@@ -1142,7 +1142,7 @@ calculate_init(PyCalculatePath *calculate,
         return DECODE_LOCALE_ERR("PREFIX define", len);
     }
     calculate->exec_prefix = Py_DecodeLocale(EXEC_PREFIX, &len);
-    if (!calculate->prefix) {
+    if (!calculate->exec_prefix) {
         return DECODE_LOCALE_ERR("EXEC_PREFIX define", len);
     }
     calculate->lib_python = Py_DecodeLocale("lib/python" VERSION, &len);

From c1c04cbc24c11cd7a47579af3faffee05a16acd7 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 11:41:13 -0700
Subject: [PATCH 0530/2163] bpo-36502: Update link to UAX GH-44, the Unicode
 doc on the UCD. (GH-15301)

The link we have points to the version from Unicode 6.0.0, dated 2010.
There have been numerous updates to it since then:
  https://www.unicode.org/reports/tr44/GH-Modifications

Change the link to one that points to the current version. Also, use HTTPS.
(cherry picked from commit 64c6ac74e254d31f93fcc74bf02b3daa7d3e3f25)

Co-authored-by: Greg Price 
---
 Doc/library/unicodedata.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Doc/library/unicodedata.rst b/Doc/library/unicodedata.rst
index ee790c0cd00bbc..225384cf391e99 100644
--- a/Doc/library/unicodedata.rst
+++ b/Doc/library/unicodedata.rst
@@ -22,7 +22,7 @@ this database is compiled from the `UCD version 12.1.0
 
 The module uses the same names and symbols as defined by Unicode
 Standard Annex #44, `"Unicode Character Database"
-`_.  It defines the
+`_.  It defines the
 following functions:
 
 

From 99df5e837334b62c29c979bb0806f525778a4f3e Mon Sep 17 00:00:00 2001
From: Zachary Ware 
Date: Mon, 9 Sep 2019 23:11:23 +0100
Subject: [PATCH 0531/2163] [3.8] bpo-34293: Fix PDF documentation paper size
 (GH-8585) (GH-15816)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The "A4" pdfs were previously the wrong size due to a change in the options in Sphinx 1.5.

See also sphinx-doc/sphinxGH-5235
(cherry picked from commit b5381f669718aa19690f42f3b8bd88f03045b9d2)

Authored-by: Jean-François B 
---
 Doc/Makefile                                                | 6 +++++-
 .../Documentation/2018-07-31-15-38-26.bpo-34293.yHupAL.rst  | 1 +
 2 files changed, 6 insertions(+), 1 deletion(-)
 create mode 100644 Misc/NEWS.d/next/Documentation/2018-07-31-15-38-26.bpo-34293.yHupAL.rst

diff --git a/Doc/Makefile b/Doc/Makefile
index 6f86728ea834c3..05eeab9124df63 100644
--- a/Doc/Makefile
+++ b/Doc/Makefile
@@ -13,7 +13,11 @@ SOURCES      =
 DISTVERSION  = $(shell $(PYTHON) tools/extensions/patchlevel.py)
 SPHINXERRORHANDLING = -W
 
-ALLSPHINXOPTS = -b $(BUILDER) -d build/doctrees -D latex_elements.papersize=$(PAPER) \
+# Internal variables.
+PAPEROPT_a4     = -D latex_elements.papersize=a4paper
+PAPEROPT_letter = -D latex_elements.papersize=letterpaper
+
+ALLSPHINXOPTS = -b $(BUILDER) -d build/doctrees $(PAPEROPT_$(PAPER)) \
                 $(SPHINXOPTS) $(SPHINXERRORHANDLING) . build/$(BUILDER) $(SOURCES)
 
 .PHONY: help build html htmlhelp latex text changes linkcheck \
diff --git a/Misc/NEWS.d/next/Documentation/2018-07-31-15-38-26.bpo-34293.yHupAL.rst b/Misc/NEWS.d/next/Documentation/2018-07-31-15-38-26.bpo-34293.yHupAL.rst
new file mode 100644
index 00000000000000..912a3ad48d403a
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2018-07-31-15-38-26.bpo-34293.yHupAL.rst
@@ -0,0 +1 @@
+Fix the Doc/Makefile regarding PAPER environment variable and PDF builds

From 29bde48ade5dbd5d88cfe309653014c84bebb89c Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Mon, 9 Sep 2019 20:29:01 -0700
Subject: [PATCH 0532/2163] bpo-38077: IDLE no longer adds 'argv' to the user
 namespace (GH-15818)

This only happened when initializing the subprocess to run a module.
This recent bug only affected 3.7.4 and 3.8.0b2 to 3.8.0b4.
(cherry picked from commit c59295a1ca304f37ca136dd7efca9e560db27d28)

Co-authored-by: Terry Jan Reedy 
---
 Lib/idlelib/NEWS.txt                                           | 3 +++
 Lib/idlelib/runscript.py                                       | 2 +-
 Misc/NEWS.d/next/IDLE/2019-09-09-22-08-36.bpo-38077.Mzpfe2.rst | 2 ++
 3 files changed, 6 insertions(+), 1 deletion(-)
 create mode 100644 Misc/NEWS.d/next/IDLE/2019-09-09-22-08-36.bpo-38077.Mzpfe2.rst

diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt
index 559ffd0cf4f30b..3ccc5c07fb0046 100644
--- a/Lib/idlelib/NEWS.txt
+++ b/Lib/idlelib/NEWS.txt
@@ -3,6 +3,9 @@ Released on 2019-10-20?
 ======================================
 
 
+bpo-38077: IDLE no longer adds 'argv' to the user namespace when
+initializing it.  This bug only affected 3.7.4 and 3.8.0b2 to 3.8.0b4.
+
 bpo-38401: Shell restart lines now fill the window width, always start
 with '=', and avoid wrapping unnecessarily. The line will still wrap
 if the included file name is long relative to the width.
diff --git a/Lib/idlelib/runscript.py b/Lib/idlelib/runscript.py
index f97cf528cce682..de73bf8458151f 100644
--- a/Lib/idlelib/runscript.py
+++ b/Lib/idlelib/runscript.py
@@ -164,7 +164,7 @@ def _run_module_event(self, event, *, customize=False):
                 _sys.argv = argv
             import os as _os
             _os.chdir({dirname!r})
-            del _sys, _basename, _os
+            del _sys, argv, _basename, _os
             \n""")
         interp.prepend_syspath(filename)
         # XXX KBK 03Jul04 When run w/o subprocess, runtime warnings still
diff --git a/Misc/NEWS.d/next/IDLE/2019-09-09-22-08-36.bpo-38077.Mzpfe2.rst b/Misc/NEWS.d/next/IDLE/2019-09-09-22-08-36.bpo-38077.Mzpfe2.rst
new file mode 100644
index 00000000000000..ba1fd55defaa28
--- /dev/null
+++ b/Misc/NEWS.d/next/IDLE/2019-09-09-22-08-36.bpo-38077.Mzpfe2.rst
@@ -0,0 +1,2 @@
+IDLE no longer adds 'argv' to the user namespace when initializing it.  This
+bug only affected 3.7.4 and 3.8.0b2 to 3.8.0b4.

From d4391aa5eb4767e19b7b380a836413e7c47f1fb4 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 00:31:34 -0700
Subject: [PATCH 0533/2163] bpo-37383: Updates docs to reflect AsyncMock
 call_count after await. (GH-15761)

* bpo-351428: Updates documentation to reflect AsyncMock call_count after await.

* Adds skip and fixes warning.

* Removes extra >>>.

* Adds ... in front of await mock().
(cherry picked from commit b9f65f01fd761da7799f36d29b54518399d3458e)

Co-authored-by: Lisa Roach 
---
 Doc/library/unittest.mock.rst | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst
index 304ba532274ddb..b2547546f3d6d7 100644
--- a/Doc/library/unittest.mock.rst
+++ b/Doc/library/unittest.mock.rst
@@ -514,6 +514,20 @@ the *new_callable* argument to :func:`patch`.
             >>> mock.call_count
             2
 
+        For :class:`AsyncMock` the :attr:`call_count` is only iterated if the function
+        has been awaited:
+
+            >>> mock = AsyncMock()
+            >>> mock()  # doctest: +SKIP
+            
+            >>> mock.call_count
+            0
+            >>> async def main():
+            ...     await mock()
+            ...
+            >>> asyncio.run(main())
+            >>> mock.call_count
+            1
 
     .. attribute:: return_value
 

From 74b7413d3a9be5e06b16eb2b9a6bdc1f3fe44bbb Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 00:31:56 -0700
Subject: [PATCH 0534/2163] bpo-37662: Documented
 venv.EnvBuilder.upgrade_dependencies(). (GH-15768)

(cherry picked from commit 264e034f990240e2aa379d8484b15b9e70c1fad5)

Co-authored-by: Vinay Sajip 
---
 Doc/library/venv.rst | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst
index 1e825c3c21e843..a834ac75eb1b51 100644
--- a/Doc/library/venv.rst
+++ b/Doc/library/venv.rst
@@ -186,6 +186,14 @@ creation according to their needs, the :class:`EnvBuilder` class.
         Installs activation scripts appropriate to the platform into the virtual
         environment.
 
+    .. method:: upgrade_dependencies(context)
+
+       Upgrades the core venv dependency packages (currently ``pip`` and
+       ``setuptools``) in the environment. This is done by shelling out to the
+       ``pip`` executable in the environment.
+
+       .. versionadded:: 3.8
+
     .. method:: post_setup(context)
 
         A placeholder method which can be overridden in third party

From fdd17abc51e363ab19d248375d717512b8b26966 Mon Sep 17 00:00:00 2001
From: Steve Dower 
Date: Tue, 10 Sep 2019 02:02:04 -0700
Subject: [PATCH 0535/2163] bpo-35941: Fix performance regression in SSL
 certificate code (GH-12610)

Accumulate certificates in a set instead of doing a costly list contain
operation. A Windows cert store can easily contain over hundred
certificates. The old code would result in way over 5,000 comparison
operations

Signed-off-by: Christian Heimes 
---
 Lib/test/test_ssl.py |  4 ++--
 Modules/_ssl.c       | 56 +++++++++++++++++++++-----------------------
 2 files changed, 29 insertions(+), 31 deletions(-)

diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
index 879d944f327e97..48c4fecabd14fd 100644
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -832,8 +832,8 @@ def test_enum_certificates(self):
                 cert, enc, trust = element
                 self.assertIsInstance(cert, bytes)
                 self.assertIn(enc, {"x509_asn", "pkcs_7_asn"})
-                self.assertIsInstance(trust, (set, bool))
-                if isinstance(trust, set):
+                self.assertIsInstance(trust, (frozenset, set, bool))
+                if isinstance(trust, (frozenset, set)):
                     trust_oids.update(trust)
 
         serverAuth = "1.3.6.1.5.5.7.3.1"
diff --git a/Modules/_ssl.c b/Modules/_ssl.c
index 6f91b488252851..64060c64c4df28 100644
--- a/Modules/_ssl.c
+++ b/Modules/_ssl.c
@@ -5517,7 +5517,7 @@ parseKeyUsage(PCCERT_CONTEXT pCertCtx, DWORD flags)
         }
         return PyErr_SetFromWindowsErr(error);
     }
-    retval = PySet_New(NULL);
+    retval = PyFrozenSet_New(NULL);
     if (retval == NULL) {
         goto error;
     }
@@ -5592,20 +5592,6 @@ ssl_collect_certificates(const char *store_name)
     return hCollectionStore;
 }
 
-/* code from Objects/listobject.c */
-
-static int
-list_contains(PyListObject *a, PyObject *el)
-{
-    Py_ssize_t i;
-    int cmp;
-
-    for (i = 0, cmp = 0 ; cmp == 0 && i < Py_SIZE(a); ++i)
-        cmp = PyObject_RichCompareBool(el, PyList_GET_ITEM(a, i),
-                                           Py_EQ);
-    return cmp;
-}
-
 /*[clinic input]
 _ssl.enum_certificates
     store_name: str
@@ -5628,7 +5614,7 @@ _ssl_enum_certificates_impl(PyObject *module, const char *store_name)
     PyObject *keyusage = NULL, *cert = NULL, *enc = NULL, *tup = NULL;
     PyObject *result = NULL;
 
-    result = PyList_New(0);
+    result = PySet_New(NULL);
     if (result == NULL) {
         return NULL;
     }
@@ -5668,11 +5654,10 @@ _ssl_enum_certificates_impl(PyObject *module, const char *store_name)
         enc = NULL;
         PyTuple_SET_ITEM(tup, 2, keyusage);
         keyusage = NULL;
-        if (!list_contains((PyListObject*)result, tup)) {
-            if (PyList_Append(result, tup) < 0) {
-                Py_CLEAR(result);
-                break;
-            }
+        if (PySet_Add(result, tup) == -1) {
+            Py_CLEAR(result);
+            Py_CLEAR(tup);
+            break;
         }
         Py_CLEAR(tup);
     }
@@ -5696,7 +5681,14 @@ _ssl_enum_certificates_impl(PyObject *module, const char *store_name)
         return PyErr_SetFromWindowsErr(GetLastError());
     }
 
-    return result;
+    /* convert set to list */
+    if (result == NULL) {
+        return NULL;
+    } else {
+        PyObject *lst = PySequence_List(result);
+        Py_DECREF(result);
+        return lst;
+    }
 }
 
 /*[clinic input]
@@ -5720,7 +5712,7 @@ _ssl_enum_crls_impl(PyObject *module, const char *store_name)
     PyObject *crl = NULL, *enc = NULL, *tup = NULL;
     PyObject *result = NULL;
 
-    result = PyList_New(0);
+    result = PySet_New(NULL);
     if (result == NULL) {
         return NULL;
     }
@@ -5750,11 +5742,10 @@ _ssl_enum_crls_impl(PyObject *module, const char *store_name)
         PyTuple_SET_ITEM(tup, 1, enc);
         enc = NULL;
 
-        if (!list_contains((PyListObject*)result, tup)) {
-            if (PyList_Append(result, tup) < 0) {
-                Py_CLEAR(result);
-                break;
-            }
+        if (PySet_Add(result, tup) == -1) {
+            Py_CLEAR(result);
+            Py_CLEAR(tup);
+            break;
         }
         Py_CLEAR(tup);
     }
@@ -5776,7 +5767,14 @@ _ssl_enum_crls_impl(PyObject *module, const char *store_name)
         Py_XDECREF(result);
         return PyErr_SetFromWindowsErr(GetLastError());
     }
-    return result;
+    /* convert set to list */
+    if (result == NULL) {
+        return NULL;
+    } else {
+        PyObject *lst = PySequence_List(result);
+        Py_DECREF(result);
+        return lst;
+    }
 }
 
 #endif /* _MSC_VER */

From ccaea525885e41c5f1e566bb68698847faaa82ca Mon Sep 17 00:00:00 2001
From: Victor Stinner 
Date: Tue, 10 Sep 2019 11:57:31 +0200
Subject: [PATCH 0536/2163] Revert "bpo-33418: Add tp_clear for function object
 (GH-8058)" (GH-15826)

This reverts commit 3c452404ae178b742967589a0bb4a5ec768d76e0.
---
 Objects/funcobject.c | 34 +++++++++++++---------------------
 1 file changed, 13 insertions(+), 21 deletions(-)

diff --git a/Objects/funcobject.c b/Objects/funcobject.c
index df5cc2d3f57024..53aebf12f550fd 100644
--- a/Objects/funcobject.c
+++ b/Objects/funcobject.c
@@ -570,31 +570,23 @@ func_new_impl(PyTypeObject *type, PyCodeObject *code, PyObject *globals,
     return (PyObject *)newfunc;
 }
 
-static int
-func_clear(PyFunctionObject *op)
-{
-    Py_CLEAR(op->func_code);
-    Py_CLEAR(op->func_globals);
-    Py_CLEAR(op->func_module);
-    Py_CLEAR(op->func_name);
-    Py_CLEAR(op->func_defaults);
-    Py_CLEAR(op->func_kwdefaults);
-    Py_CLEAR(op->func_doc);
-    Py_CLEAR(op->func_dict);
-    Py_CLEAR(op->func_closure);
-    Py_CLEAR(op->func_annotations);
-    Py_CLEAR(op->func_qualname);
-    return 0;
-}
-
 static void
 func_dealloc(PyFunctionObject *op)
 {
     _PyObject_GC_UNTRACK(op);
-    if (op->func_weakreflist != NULL) {
+    if (op->func_weakreflist != NULL)
         PyObject_ClearWeakRefs((PyObject *) op);
-    }
-    (void)func_clear(op);
+    Py_DECREF(op->func_code);
+    Py_DECREF(op->func_globals);
+    Py_XDECREF(op->func_module);
+    Py_DECREF(op->func_name);
+    Py_XDECREF(op->func_defaults);
+    Py_XDECREF(op->func_kwdefaults);
+    Py_XDECREF(op->func_doc);
+    Py_XDECREF(op->func_dict);
+    Py_XDECREF(op->func_closure);
+    Py_XDECREF(op->func_annotations);
+    Py_XDECREF(op->func_qualname);
     PyObject_GC_Del(op);
 }
 
@@ -669,7 +661,7 @@ PyTypeObject PyFunction_Type = {
     Py_TPFLAGS_METHOD_DESCRIPTOR,               /* tp_flags */
     func_new__doc__,                            /* tp_doc */
     (traverseproc)func_traverse,                /* tp_traverse */
-    (inquiry)func_clear,                        /* tp_clear */
+    0,                                          /* tp_clear */
     0,                                          /* tp_richcompare */
     offsetof(PyFunctionObject, func_weakreflist), /* tp_weaklistoffset */
     0,                                          /* tp_iter */

From 18fa272601a1fa2c5a978b27c0c8dfd7c3ca90af Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 03:13:36 -0700
Subject: [PATCH 0537/2163] Fix typo in dict object comment (GH-15814)

(cherry picked from commit 359143c68659d165f52320d368667e0eff279dc5)

Co-authored-by: dalgarno <32097481+dalgarno@users.noreply.github.com>
---
 Objects/dictobject.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index e417cd2119c6df..31f962d16b25ed 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -39,7 +39,7 @@ Size of indices is dk_size.  Type of each index in indices is vary on dk_size:
 * int32 for 2**16 <= dk_size <= 2**31
 * int64 for 2**32 <= dk_size
 
-dk_entries is array of PyDictKeyEntry.  It's size is USABLE_FRACTION(dk_size).
+dk_entries is array of PyDictKeyEntry.  Its size is USABLE_FRACTION(dk_size).
 DK_ENTRIES(dk) can be used to get pointer to entries.
 
 NOTE: Since negative value is used for DKIX_EMPTY and DKIX_DUMMY, type of

From 7d41e400d6acc799f49e754998f1963630252e4f Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 03:54:48 -0700
Subject: [PATCH 0538/2163] Remove macOS tests from Travis. (GH-15809)

Azure runs macOS, so we don't need Travis to do it.
(cherry picked from commit e45b217ae0e83cfb367e9769cae59b9a5505f9b4)

Co-authored-by: Benjamin Peterson 
---
 .travis.yml | 10 ----------
 1 file changed, 10 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 1d470728a536a8..519dd9c652792d 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -73,16 +73,6 @@ matrix:
         - make -C Doc/ PYTHON=../python venv
       script:
         xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W -j4" doctest
-    - name: "Mac OS X tests"
-      os: osx
-      language: c
-      compiler: clang
-      # Testing under macOS is optional until testing stability has been demonstrated.
-      env: OPTIONAL=true
-      before_install:
-        # Python 3 is needed for Argument Clinic and multissl
-        - HOMEBREW_NO_AUTO_UPDATE=1 brew install xz python3
-        - export PATH=$(brew --prefix)/bin:$(brew --prefix)/sbin:$PATH
     - name: "Test code coverage (Python)"
       os: linux
       language: c

From ab74e52f768be5048faf2a11e78822533afebcb7 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 04:08:52 -0700
Subject: [PATCH 0539/2163] bpo-37052: Add examples for mocking async iterators
 and context managers (GH-14660)

Add examples for mocking asynchronous iterators and asynchronous context managers.

https://bugs.python.org/issue37052
(cherry picked from commit c8dfa7333d6317d7cd8c5c7366023f5a668e3f91)

Co-authored-by: Xtreak 
---
 Doc/library/unittest.mock-examples.rst | 39 ++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/Doc/library/unittest.mock-examples.rst b/Doc/library/unittest.mock-examples.rst
index 811f0fb1ce9397..cf6b671b5beeb4 100644
--- a/Doc/library/unittest.mock-examples.rst
+++ b/Doc/library/unittest.mock-examples.rst
@@ -12,6 +12,7 @@
 
 .. testsetup::
 
+    import asyncio
     import unittest
     from unittest.mock import Mock, MagicMock, patch, call, sentinel
 
@@ -276,6 +277,44 @@ function returns is what the call returns:
     2
 
 
+Mocking asynchronous iterators
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Python 3.8, ``MagicMock`` has support to mock :ref:`async-iterators`
+through ``__aiter__``. The :attr:`~Mock.return_value` attribute of ``__aiter__``
+can be used to set the return values to be used for iteration.
+
+    >>> mock = MagicMock()
+    >>> mock.__aiter__.return_value = [1, 2, 3]
+    >>> async def main():
+    ...     return [i async for i in mock]
+    >>> asyncio.run(main())
+    [1, 2, 3]
+
+
+Mocking asynchronous context manager
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Python 3.8, ``MagicMock`` has support to mock
+:ref:`async-context-managers` through ``__aenter__`` and ``__aexit__``. The
+return value of ``__aenter__`` is an :class:`AsyncMock`.
+
+    >>> class AsyncContextManager:
+    ...
+    ...     async def __aenter__(self):
+    ...         return self
+    ...
+    ...     async def __aexit__(self):
+    ...         pass
+    >>> mock_instance = MagicMock(AsyncContextManager())
+    >>> async def main():
+    ...     async with mock_instance as result:
+    ...         pass
+    >>> asyncio.run(main())
+    >>> mock_instance.__aenter__.assert_called_once()
+    >>> mock_instance.__aexit__.assert_called_once()
+
+
 Creating a Mock from an Existing Object
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

From bb8fc8bd309419c159b632068dff73c3c76d478c Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 04:26:54 -0700
Subject: [PATCH 0540/2163] bpo-36373: Deprecate explicit loop parameter in all
 public asyncio APIs [locks] (GH-13920)

This PR deprecate explicit loop parameters in all public asyncio APIs

This issues is split to be easier to review.

Third step: locks.py

https://bugs.python.org/issue36373
(cherry picked from commit 537877d85d1c27d2c2f5189e39da64a7a0c413d3)

Co-authored-by: Emmanuel Arias 
---
 Doc/library/asyncio-sync.rst         |  18 ++
 Lib/asyncio/locks.py                 |  41 +++--
 Lib/test/test_asyncio/test_events.py | 241 ++++++++++++++-------------
 Lib/test/test_asyncio/test_locks.py  | 198 ++++++++++++++--------
 Lib/test/test_asyncio/test_pep492.py |  26 +--
 Lib/test/test_asyncio/test_queues.py | 135 ++++++++++-----
 Lib/test/test_asyncio/test_tasks.py  |  25 ++-
 7 files changed, 419 insertions(+), 265 deletions(-)

diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst
index 79f6b02d85e2e5..cc8c29dc3b6bc5 100644
--- a/Doc/library/asyncio-sync.rst
+++ b/Doc/library/asyncio-sync.rst
@@ -59,6 +59,9 @@ Lock
        finally:
            lock.release()
 
+   .. deprecated-removed:: 3.8 3.10
+      The *loop* parameter.
+
    .. coroutinemethod:: acquire()
 
       Acquire the lock.
@@ -101,6 +104,10 @@ Event
    :meth:`clear` method.  The :meth:`wait` method blocks until the
    flag is set to *true*.  The flag is set to *false* initially.
 
+
+   .. deprecated-removed:: 3.8 3.10
+      The *loop* parameter.
+
    .. _asyncio_example_sync_event:
 
    Example::
@@ -173,6 +180,10 @@ Condition
    ``None``.  In the latter case a new Lock object is created
    automatically.
 
+
+   .. deprecated-removed:: 3.8 3.10
+      The *loop* parameter.
+
    The preferred way to use a Condition is an :keyword:`async with`
    statement::
 
@@ -269,6 +280,10 @@ Semaphore
    internal counter (``1`` by default). If the given value is
    less than ``0`` a :exc:`ValueError` is raised.
 
+
+   .. deprecated-removed:: 3.8 3.10
+      The *loop* parameter.
+
    The preferred way to use a Semaphore is an :keyword:`async with`
    statement::
 
@@ -322,6 +337,9 @@ BoundedSemaphore
    increases the internal counter above the initial *value*.
 
 
+   .. deprecated-removed:: 3.8 3.10
+      The *loop* parameter.
+
 ---------
 
 
diff --git a/Lib/asyncio/locks.py b/Lib/asyncio/locks.py
index 1324eefb5ff460..f63d4cedbbb7c3 100644
--- a/Lib/asyncio/locks.py
+++ b/Lib/asyncio/locks.py
@@ -160,10 +160,13 @@ class Lock(_ContextManagerMixin):
     def __init__(self, *, loop=None):
         self._waiters = None
         self._locked = False
-        if loop is not None:
-            self._loop = loop
-        else:
+        if loop is None:
             self._loop = events.get_event_loop()
+        else:
+            self._loop = loop
+            warnings.warn("The loop argument is deprecated since Python 3.8, "
+                          "and scheduled for removal in Python 3.10.",
+                          DeprecationWarning, stacklevel=2)
 
     def __repr__(self):
         res = super().__repr__()
@@ -253,10 +256,13 @@ class Event:
     def __init__(self, *, loop=None):
         self._waiters = collections.deque()
         self._value = False
-        if loop is not None:
-            self._loop = loop
-        else:
+        if loop is None:
             self._loop = events.get_event_loop()
+        else:
+            self._loop = loop
+            warnings.warn("The loop argument is deprecated since Python 3.8, "
+                          "and scheduled for removal in Python 3.10.",
+                          DeprecationWarning, stacklevel=2)
 
     def __repr__(self):
         res = super().__repr__()
@@ -317,10 +323,13 @@ class Condition(_ContextManagerMixin):
     """
 
     def __init__(self, lock=None, *, loop=None):
-        if loop is not None:
-            self._loop = loop
-        else:
+        if loop is None:
             self._loop = events.get_event_loop()
+        else:
+            self._loop = loop
+            warnings.warn("The loop argument is deprecated since Python 3.8, "
+                          "and scheduled for removal in Python 3.10.",
+                          DeprecationWarning, stacklevel=2)
 
         if lock is None:
             lock = Lock(loop=self._loop)
@@ -445,10 +454,13 @@ def __init__(self, value=1, *, loop=None):
             raise ValueError("Semaphore initial value must be >= 0")
         self._value = value
         self._waiters = collections.deque()
-        if loop is not None:
-            self._loop = loop
-        else:
+        if loop is None:
             self._loop = events.get_event_loop()
+        else:
+            self._loop = loop
+            warnings.warn("The loop argument is deprecated since Python 3.8, "
+                          "and scheduled for removal in Python 3.10.",
+                          DeprecationWarning, stacklevel=2)
 
     def __repr__(self):
         res = super().__repr__()
@@ -508,6 +520,11 @@ class BoundedSemaphore(Semaphore):
     """
 
     def __init__(self, value=1, *, loop=None):
+        if loop:
+            warnings.warn("The loop argument is deprecated since Python 3.8, "
+                          "and scheduled for removal in Python 3.10.",
+                          DeprecationWarning, stacklevel=2)
+
         self._bound_value = value
         super().__init__(value, loop=loop)
 
diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py
index e5ad72fe5ba85e..95dac72b6d7efd 100644
--- a/Lib/test/test_asyncio/test_events.py
+++ b/Lib/test/test_asyncio/test_events.py
@@ -1718,19 +1718,20 @@ def test_subprocess_exec(self):
         connect = self.loop.subprocess_exec(
                         functools.partial(MySubprocessProtocol, self.loop),
                         sys.executable, prog)
-        transp, proto = self.loop.run_until_complete(connect)
-        self.assertIsInstance(proto, MySubprocessProtocol)
-        self.loop.run_until_complete(proto.connected)
-        self.assertEqual('CONNECTED', proto.state)
+        with self.assertWarns(DeprecationWarning):
+            transp, proto = self.loop.run_until_complete(connect)
+            self.assertIsInstance(proto, MySubprocessProtocol)
+            self.loop.run_until_complete(proto.connected)
+            self.assertEqual('CONNECTED', proto.state)
 
-        stdin = transp.get_pipe_transport(0)
-        stdin.write(b'Python The Winner')
-        self.loop.run_until_complete(proto.got_data[1].wait())
-        with test_utils.disable_logger():
-            transp.close()
-        self.loop.run_until_complete(proto.completed)
-        self.check_killed(proto.returncode)
-        self.assertEqual(b'Python The Winner', proto.data[1])
+            stdin = transp.get_pipe_transport(0)
+            stdin.write(b'Python The Winner')
+            self.loop.run_until_complete(proto.got_data[1].wait())
+            with test_utils.disable_logger():
+                transp.close()
+            self.loop.run_until_complete(proto.completed)
+            self.check_killed(proto.returncode)
+            self.assertEqual(b'Python The Winner', proto.data[1])
 
     def test_subprocess_interactive(self):
         prog = os.path.join(os.path.dirname(__file__), 'echo.py')
@@ -1738,47 +1739,52 @@ def test_subprocess_interactive(self):
         connect = self.loop.subprocess_exec(
                         functools.partial(MySubprocessProtocol, self.loop),
                         sys.executable, prog)
-        transp, proto = self.loop.run_until_complete(connect)
-        self.assertIsInstance(proto, MySubprocessProtocol)
-        self.loop.run_until_complete(proto.connected)
-        self.assertEqual('CONNECTED', proto.state)
 
-        stdin = transp.get_pipe_transport(0)
-        stdin.write(b'Python ')
-        self.loop.run_until_complete(proto.got_data[1].wait())
-        proto.got_data[1].clear()
-        self.assertEqual(b'Python ', proto.data[1])
+        with self.assertWarns(DeprecationWarning):
+            transp, proto = self.loop.run_until_complete(connect)
+            self.assertIsInstance(proto, MySubprocessProtocol)
+            self.loop.run_until_complete(proto.connected)
+            self.assertEqual('CONNECTED', proto.state)
 
-        stdin.write(b'The Winner')
-        self.loop.run_until_complete(proto.got_data[1].wait())
-        self.assertEqual(b'Python The Winner', proto.data[1])
+            stdin = transp.get_pipe_transport(0)
+            stdin.write(b'Python ')
+            self.loop.run_until_complete(proto.got_data[1].wait())
+            proto.got_data[1].clear()
+            self.assertEqual(b'Python ', proto.data[1])
 
-        with test_utils.disable_logger():
-            transp.close()
-        self.loop.run_until_complete(proto.completed)
-        self.check_killed(proto.returncode)
+            stdin.write(b'The Winner')
+            self.loop.run_until_complete(proto.got_data[1].wait())
+            self.assertEqual(b'Python The Winner', proto.data[1])
+
+            with test_utils.disable_logger():
+                transp.close()
+            self.loop.run_until_complete(proto.completed)
+            self.check_killed(proto.returncode)
 
     def test_subprocess_shell(self):
-        connect = self.loop.subprocess_shell(
-                        functools.partial(MySubprocessProtocol, self.loop),
-                        'echo Python')
-        transp, proto = self.loop.run_until_complete(connect)
-        self.assertIsInstance(proto, MySubprocessProtocol)
-        self.loop.run_until_complete(proto.connected)
+        with self.assertWarns(DeprecationWarning):
+            connect = self.loop.subprocess_shell(
+                            functools.partial(MySubprocessProtocol, self.loop),
+                            'echo Python')
+            transp, proto = self.loop.run_until_complete(connect)
+            self.assertIsInstance(proto, MySubprocessProtocol)
+            self.loop.run_until_complete(proto.connected)
 
-        transp.get_pipe_transport(0).close()
-        self.loop.run_until_complete(proto.completed)
-        self.assertEqual(0, proto.returncode)
-        self.assertTrue(all(f.done() for f in proto.disconnects.values()))
-        self.assertEqual(proto.data[1].rstrip(b'\r\n'), b'Python')
-        self.assertEqual(proto.data[2], b'')
-        transp.close()
+            transp.get_pipe_transport(0).close()
+            self.loop.run_until_complete(proto.completed)
+            self.assertEqual(0, proto.returncode)
+            self.assertTrue(all(f.done() for f in proto.disconnects.values()))
+            self.assertEqual(proto.data[1].rstrip(b'\r\n'), b'Python')
+            self.assertEqual(proto.data[2], b'')
+            transp.close()
 
     def test_subprocess_exitcode(self):
         connect = self.loop.subprocess_shell(
                         functools.partial(MySubprocessProtocol, self.loop),
                         'exit 7', stdin=None, stdout=None, stderr=None)
-        transp, proto = self.loop.run_until_complete(connect)
+
+        with self.assertWarns(DeprecationWarning):
+            transp, proto = self.loop.run_until_complete(connect)
         self.assertIsInstance(proto, MySubprocessProtocol)
         self.loop.run_until_complete(proto.completed)
         self.assertEqual(7, proto.returncode)
@@ -1788,7 +1794,8 @@ def test_subprocess_close_after_finish(self):
         connect = self.loop.subprocess_shell(
                         functools.partial(MySubprocessProtocol, self.loop),
                         'exit 7', stdin=None, stdout=None, stderr=None)
-        transp, proto = self.loop.run_until_complete(connect)
+        with self.assertWarns(DeprecationWarning):
+            transp, proto = self.loop.run_until_complete(connect)
         self.assertIsInstance(proto, MySubprocessProtocol)
         self.assertIsNone(transp.get_pipe_transport(0))
         self.assertIsNone(transp.get_pipe_transport(1))
@@ -1803,14 +1810,16 @@ def test_subprocess_kill(self):
         connect = self.loop.subprocess_exec(
                         functools.partial(MySubprocessProtocol, self.loop),
                         sys.executable, prog)
-        transp, proto = self.loop.run_until_complete(connect)
-        self.assertIsInstance(proto, MySubprocessProtocol)
-        self.loop.run_until_complete(proto.connected)
 
-        transp.kill()
-        self.loop.run_until_complete(proto.completed)
-        self.check_killed(proto.returncode)
-        transp.close()
+        with self.assertWarns(DeprecationWarning):
+            transp, proto = self.loop.run_until_complete(connect)
+            self.assertIsInstance(proto, MySubprocessProtocol)
+            self.loop.run_until_complete(proto.connected)
+
+            transp.kill()
+            self.loop.run_until_complete(proto.completed)
+            self.check_killed(proto.returncode)
+            transp.close()
 
     def test_subprocess_terminate(self):
         prog = os.path.join(os.path.dirname(__file__), 'echo.py')
@@ -1818,14 +1827,16 @@ def test_subprocess_terminate(self):
         connect = self.loop.subprocess_exec(
                         functools.partial(MySubprocessProtocol, self.loop),
                         sys.executable, prog)
-        transp, proto = self.loop.run_until_complete(connect)
-        self.assertIsInstance(proto, MySubprocessProtocol)
-        self.loop.run_until_complete(proto.connected)
 
-        transp.terminate()
-        self.loop.run_until_complete(proto.completed)
-        self.check_terminated(proto.returncode)
-        transp.close()
+        with self.assertWarns(DeprecationWarning):
+            transp, proto = self.loop.run_until_complete(connect)
+            self.assertIsInstance(proto, MySubprocessProtocol)
+            self.loop.run_until_complete(proto.connected)
+
+            transp.terminate()
+            self.loop.run_until_complete(proto.completed)
+            self.check_terminated(proto.returncode)
+            transp.close()
 
     @unittest.skipIf(sys.platform == 'win32', "Don't have SIGHUP")
     def test_subprocess_send_signal(self):
@@ -1839,14 +1850,16 @@ def test_subprocess_send_signal(self):
             connect = self.loop.subprocess_exec(
                             functools.partial(MySubprocessProtocol, self.loop),
                             sys.executable, prog)
-            transp, proto = self.loop.run_until_complete(connect)
-            self.assertIsInstance(proto, MySubprocessProtocol)
-            self.loop.run_until_complete(proto.connected)
 
-            transp.send_signal(signal.SIGHUP)
-            self.loop.run_until_complete(proto.completed)
-            self.assertEqual(-signal.SIGHUP, proto.returncode)
-            transp.close()
+            with self.assertWarns(DeprecationWarning):
+                transp, proto = self.loop.run_until_complete(connect)
+                self.assertIsInstance(proto, MySubprocessProtocol)
+                self.loop.run_until_complete(proto.connected)
+
+                transp.send_signal(signal.SIGHUP)
+                self.loop.run_until_complete(proto.completed)
+                self.assertEqual(-signal.SIGHUP, proto.returncode)
+                transp.close()
         finally:
             signal.signal(signal.SIGHUP, old_handler)
 
@@ -1856,19 +1869,21 @@ def test_subprocess_stderr(self):
         connect = self.loop.subprocess_exec(
                         functools.partial(MySubprocessProtocol, self.loop),
                         sys.executable, prog)
-        transp, proto = self.loop.run_until_complete(connect)
-        self.assertIsInstance(proto, MySubprocessProtocol)
-        self.loop.run_until_complete(proto.connected)
 
-        stdin = transp.get_pipe_transport(0)
-        stdin.write(b'test')
+        with self.assertWarns(DeprecationWarning):
+            transp, proto = self.loop.run_until_complete(connect)
+            self.assertIsInstance(proto, MySubprocessProtocol)
+            self.loop.run_until_complete(proto.connected)
 
-        self.loop.run_until_complete(proto.completed)
+            stdin = transp.get_pipe_transport(0)
+            stdin.write(b'test')
 
-        transp.close()
-        self.assertEqual(b'OUT:test', proto.data[1])
-        self.assertTrue(proto.data[2].startswith(b'ERR:test'), proto.data[2])
-        self.assertEqual(0, proto.returncode)
+            self.loop.run_until_complete(proto.completed)
+
+            transp.close()
+            self.assertEqual(b'OUT:test', proto.data[1])
+            self.assertTrue(proto.data[2].startswith(b'ERR:test'), proto.data[2])
+            self.assertEqual(0, proto.returncode)
 
     def test_subprocess_stderr_redirect_to_stdout(self):
         prog = os.path.join(os.path.dirname(__file__), 'echo2.py')
@@ -1876,22 +1891,24 @@ def test_subprocess_stderr_redirect_to_stdout(self):
         connect = self.loop.subprocess_exec(
                         functools.partial(MySubprocessProtocol, self.loop),
                         sys.executable, prog, stderr=subprocess.STDOUT)
-        transp, proto = self.loop.run_until_complete(connect)
-        self.assertIsInstance(proto, MySubprocessProtocol)
-        self.loop.run_until_complete(proto.connected)
 
-        stdin = transp.get_pipe_transport(0)
-        self.assertIsNotNone(transp.get_pipe_transport(1))
-        self.assertIsNone(transp.get_pipe_transport(2))
+        with self.assertWarns(DeprecationWarning):
+            transp, proto = self.loop.run_until_complete(connect)
+            self.assertIsInstance(proto, MySubprocessProtocol)
+            self.loop.run_until_complete(proto.connected)
 
-        stdin.write(b'test')
-        self.loop.run_until_complete(proto.completed)
-        self.assertTrue(proto.data[1].startswith(b'OUT:testERR:test'),
-                        proto.data[1])
-        self.assertEqual(b'', proto.data[2])
+            stdin = transp.get_pipe_transport(0)
+            self.assertIsNotNone(transp.get_pipe_transport(1))
+            self.assertIsNone(transp.get_pipe_transport(2))
 
-        transp.close()
-        self.assertEqual(0, proto.returncode)
+            stdin.write(b'test')
+            self.loop.run_until_complete(proto.completed)
+            self.assertTrue(proto.data[1].startswith(b'OUT:testERR:test'),
+                            proto.data[1])
+            self.assertEqual(b'', proto.data[2])
+
+            transp.close()
+            self.assertEqual(0, proto.returncode)
 
     def test_subprocess_close_client_stream(self):
         prog = os.path.join(os.path.dirname(__file__), 'echo3.py')
@@ -1899,32 +1916,33 @@ def test_subprocess_close_client_stream(self):
         connect = self.loop.subprocess_exec(
                         functools.partial(MySubprocessProtocol, self.loop),
                         sys.executable, prog)
-        transp, proto = self.loop.run_until_complete(connect)
-        self.assertIsInstance(proto, MySubprocessProtocol)
-        self.loop.run_until_complete(proto.connected)
+        with self.assertWarns(DeprecationWarning):
+            transp, proto = self.loop.run_until_complete(connect)
+            self.assertIsInstance(proto, MySubprocessProtocol)
+            self.loop.run_until_complete(proto.connected)
 
-        stdin = transp.get_pipe_transport(0)
-        stdout = transp.get_pipe_transport(1)
-        stdin.write(b'test')
-        self.loop.run_until_complete(proto.got_data[1].wait())
-        self.assertEqual(b'OUT:test', proto.data[1])
+            stdin = transp.get_pipe_transport(0)
+            stdout = transp.get_pipe_transport(1)
+            stdin.write(b'test')
+            self.loop.run_until_complete(proto.got_data[1].wait())
+            self.assertEqual(b'OUT:test', proto.data[1])
 
-        stdout.close()
-        self.loop.run_until_complete(proto.disconnects[1])
-        stdin.write(b'xxx')
-        self.loop.run_until_complete(proto.got_data[2].wait())
-        if sys.platform != 'win32':
-            self.assertEqual(b'ERR:BrokenPipeError', proto.data[2])
-        else:
-            # After closing the read-end of a pipe, writing to the
-            # write-end using os.write() fails with errno==EINVAL and
-            # GetLastError()==ERROR_INVALID_NAME on Windows!?!  (Using
-            # WriteFile() we get ERROR_BROKEN_PIPE as expected.)
-            self.assertEqual(b'ERR:OSError', proto.data[2])
-        with test_utils.disable_logger():
-            transp.close()
-        self.loop.run_until_complete(proto.completed)
-        self.check_killed(proto.returncode)
+            stdout.close()
+            self.loop.run_until_complete(proto.disconnects[1])
+            stdin.write(b'xxx')
+            self.loop.run_until_complete(proto.got_data[2].wait())
+            if sys.platform != 'win32':
+                self.assertEqual(b'ERR:BrokenPipeError', proto.data[2])
+            else:
+                # After closing the read-end of a pipe, writing to the
+                # write-end using os.write() fails with errno==EINVAL and
+                # GetLastError()==ERROR_INVALID_NAME on Windows!?!  (Using
+                # WriteFile() we get ERROR_BROKEN_PIPE as expected.)
+                self.assertEqual(b'ERR:OSError', proto.data[2])
+            with test_utils.disable_logger():
+                transp.close()
+            self.loop.run_until_complete(proto.completed)
+            self.check_killed(proto.returncode)
 
     def test_subprocess_wait_no_same_group(self):
         # start the new process in a new session
@@ -1938,7 +1956,6 @@ def test_subprocess_wait_no_same_group(self):
         self.assertEqual(7, proto.returncode)
 
     def test_subprocess_exec_invalid_args(self):
-
         async def connect(**kwds):
             await self.loop.subprocess_exec(
                 asyncio.SubprocessProtocol,
diff --git a/Lib/test/test_asyncio/test_locks.py b/Lib/test/test_asyncio/test_locks.py
index 5063a1da448e65..d69b56dcda2254 100644
--- a/Lib/test/test_asyncio/test_locks.py
+++ b/Lib/test/test_asyncio/test_locks.py
@@ -28,10 +28,12 @@ def setUp(self):
 
     def test_ctor_loop(self):
         loop = mock.Mock()
-        lock = asyncio.Lock(loop=loop)
+        with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=loop)
         self.assertIs(lock._loop, loop)
 
-        lock = asyncio.Lock(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
         self.assertIs(lock._loop, self.loop)
 
     def test_ctor_noloop(self):
@@ -40,7 +42,8 @@ def test_ctor_noloop(self):
         self.assertIs(lock._loop, self.loop)
 
     def test_repr(self):
-        lock = asyncio.Lock(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
         self.assertTrue(repr(lock).endswith('[unlocked]>'))
         self.assertTrue(RGX_REPR.match(repr(lock)))
 
@@ -55,9 +58,10 @@ def acquire_lock():
         self.assertTrue(RGX_REPR.match(repr(lock)))
 
     def test_lock(self):
-        lock = asyncio.Lock(loop=self.loop)
-
         with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
+
+
             @asyncio.coroutine
             def acquire_lock():
                 with self.assertWarns(DeprecationWarning):
@@ -74,14 +78,14 @@ def acquire_lock():
     def test_lock_by_with_statement(self):
         loop = asyncio.new_event_loop()  # don't use TestLoop quirks
         self.set_event_loop(loop)
-        primitives = [
-            asyncio.Lock(loop=loop),
-            asyncio.Condition(loop=loop),
-            asyncio.Semaphore(loop=loop),
-            asyncio.BoundedSemaphore(loop=loop),
-        ]
-
         with self.assertWarns(DeprecationWarning):
+            primitives = [
+                asyncio.Lock(loop=loop),
+                asyncio.Condition(loop=loop),
+                asyncio.Semaphore(loop=loop),
+                asyncio.BoundedSemaphore(loop=loop),
+            ]
+
             @asyncio.coroutine
             def test(lock):
                 yield from asyncio.sleep(0.01)
@@ -99,7 +103,8 @@ def test(lock):
             self.assertFalse(primitive.locked())
 
     def test_acquire(self):
-        lock = asyncio.Lock(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
         result = []
 
         self.assertTrue(self.loop.run_until_complete(lock.acquire()))
@@ -150,7 +155,8 @@ async def c3(result):
         self.assertTrue(t3.result())
 
     def test_acquire_cancel(self):
-        lock = asyncio.Lock(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
         self.assertTrue(self.loop.run_until_complete(lock.acquire()))
 
         task = asyncio.Task(lock.acquire(), loop=self.loop)
@@ -175,7 +181,8 @@ def test_cancel_race(self):
         # B's waiter; instead, it should move on to C's waiter.
 
         # Setup: A has the lock, b and c are waiting.
-        lock = asyncio.Lock(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
 
         async def lockit(name, blocker):
             await lock.acquire()
@@ -211,7 +218,8 @@ def test_cancel_release_race(self):
         # Issue 32734
         # Acquire 4 locks, cancel second, release first
         # and 2 locks are taken at once.
-        lock = asyncio.Lock(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
         lock_count = 0
         call_count = 0
 
@@ -256,7 +264,8 @@ def trigger():
         self.assertTrue(t3.cancelled())
 
     def test_finished_waiter_cancelled(self):
-        lock = asyncio.Lock(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
 
         ta = asyncio.Task(lock.acquire(), loop=self.loop)
         test_utils.run_briefly(self.loop)
@@ -278,12 +287,14 @@ def test_finished_waiter_cancelled(self):
         self.assertTrue(tb.cancelled())
 
     def test_release_not_acquired(self):
-        lock = asyncio.Lock(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
 
         self.assertRaises(RuntimeError, lock.release)
 
     def test_release_no_waiters(self):
-        lock = asyncio.Lock(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
         self.loop.run_until_complete(lock.acquire())
         self.assertTrue(lock.locked())
 
@@ -291,9 +302,9 @@ def test_release_no_waiters(self):
         self.assertFalse(lock.locked())
 
     def test_context_manager(self):
-        lock = asyncio.Lock(loop=self.loop)
-
         with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
+
             @asyncio.coroutine
             def acquire_lock():
                 with self.assertWarns(DeprecationWarning):
@@ -305,9 +316,9 @@ def acquire_lock():
         self.assertFalse(lock.locked())
 
     def test_context_manager_cant_reuse(self):
-        lock = asyncio.Lock(loop=self.loop)
-
         with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
+
             @asyncio.coroutine
             def acquire_lock():
                 with self.assertWarns(DeprecationWarning):
@@ -325,7 +336,8 @@ def acquire_lock():
                 pass
 
     def test_context_manager_no_yield(self):
-        lock = asyncio.Lock(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
 
         try:
             with lock:
@@ -346,10 +358,12 @@ def setUp(self):
 
     def test_ctor_loop(self):
         loop = mock.Mock()
-        ev = asyncio.Event(loop=loop)
+        with self.assertWarns(DeprecationWarning):
+            ev = asyncio.Event(loop=loop)
         self.assertIs(ev._loop, loop)
 
-        ev = asyncio.Event(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            ev = asyncio.Event(loop=self.loop)
         self.assertIs(ev._loop, self.loop)
 
     def test_ctor_noloop(self):
@@ -358,7 +372,8 @@ def test_ctor_noloop(self):
         self.assertIs(ev._loop, self.loop)
 
     def test_repr(self):
-        ev = asyncio.Event(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            ev = asyncio.Event(loop=self.loop)
         self.assertTrue(repr(ev).endswith('[unset]>'))
         match = RGX_REPR.match(repr(ev))
         self.assertEqual(match.group('extras'), 'unset')
@@ -372,7 +387,8 @@ def test_repr(self):
         self.assertTrue(RGX_REPR.match(repr(ev)))
 
     def test_wait(self):
-        ev = asyncio.Event(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            ev = asyncio.Event(loop=self.loop)
         self.assertFalse(ev.is_set())
 
         result = []
@@ -409,14 +425,16 @@ async def c3(result):
         self.assertIsNone(t3.result())
 
     def test_wait_on_set(self):
-        ev = asyncio.Event(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            ev = asyncio.Event(loop=self.loop)
         ev.set()
 
         res = self.loop.run_until_complete(ev.wait())
         self.assertTrue(res)
 
     def test_wait_cancel(self):
-        ev = asyncio.Event(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            ev = asyncio.Event(loop=self.loop)
 
         wait = asyncio.Task(ev.wait(), loop=self.loop)
         self.loop.call_soon(wait.cancel)
@@ -426,7 +444,8 @@ def test_wait_cancel(self):
         self.assertFalse(ev._waiters)
 
     def test_clear(self):
-        ev = asyncio.Event(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            ev = asyncio.Event(loop=self.loop)
         self.assertFalse(ev.is_set())
 
         ev.set()
@@ -436,7 +455,8 @@ def test_clear(self):
         self.assertFalse(ev.is_set())
 
     def test_clear_with_waiters(self):
-        ev = asyncio.Event(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            ev = asyncio.Event(loop=self.loop)
         result = []
 
         async def c1(result):
@@ -472,19 +492,22 @@ def setUp(self):
 
     def test_ctor_loop(self):
         loop = mock.Mock()
-        cond = asyncio.Condition(loop=loop)
-        self.assertIs(cond._loop, loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=loop)
+            self.assertIs(cond._loop, loop)
 
-        cond = asyncio.Condition(loop=self.loop)
-        self.assertIs(cond._loop, self.loop)
+            cond = asyncio.Condition(loop=self.loop)
+            self.assertIs(cond._loop, self.loop)
 
     def test_ctor_noloop(self):
-        asyncio.set_event_loop(self.loop)
-        cond = asyncio.Condition()
-        self.assertIs(cond._loop, self.loop)
+        with self.assertWarns(DeprecationWarning):
+            asyncio.set_event_loop(self.loop)
+            cond = asyncio.Condition()
+            self.assertIs(cond._loop, self.loop)
 
     def test_wait(self):
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
         result = []
 
         async def c1(result):
@@ -547,7 +570,8 @@ async def c3(result):
         self.assertTrue(t3.result())
 
     def test_wait_cancel(self):
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
         self.loop.run_until_complete(cond.acquire())
 
         wait = asyncio.Task(cond.wait(), loop=self.loop)
@@ -559,7 +583,8 @@ def test_wait_cancel(self):
         self.assertTrue(cond.locked())
 
     def test_wait_cancel_contested(self):
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
 
         self.loop.run_until_complete(cond.acquire())
         self.assertTrue(cond.locked())
@@ -585,7 +610,8 @@ def test_wait_cancel_contested(self):
 
     def test_wait_cancel_after_notify(self):
         # See bpo-32841
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
         waited = False
 
         async def wait_on_cond():
@@ -609,13 +635,15 @@ async def wait_on_cond():
         self.assertTrue(waited)
 
     def test_wait_unacquired(self):
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
         self.assertRaises(
             RuntimeError,
             self.loop.run_until_complete, cond.wait())
 
     def test_wait_for(self):
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
         presult = False
 
         def predicate():
@@ -652,7 +680,8 @@ async def c1(result):
         self.assertTrue(t.result())
 
     def test_wait_for_unacquired(self):
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
 
         # predicate can return true immediately
         res = self.loop.run_until_complete(cond.wait_for(lambda: [1, 2, 3]))
@@ -664,7 +693,8 @@ def test_wait_for_unacquired(self):
             cond.wait_for(lambda: False))
 
     def test_notify(self):
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
         result = []
 
         async def c1(result):
@@ -716,7 +746,8 @@ async def c3(result):
         self.assertTrue(t3.result())
 
     def test_notify_all(self):
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
 
         result = []
 
@@ -752,15 +783,18 @@ async def c2(result):
         self.assertTrue(t2.result())
 
     def test_notify_unacquired(self):
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
         self.assertRaises(RuntimeError, cond.notify)
 
     def test_notify_all_unacquired(self):
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
         self.assertRaises(RuntimeError, cond.notify_all)
 
     def test_repr(self):
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
         self.assertTrue('unlocked' in repr(cond))
         self.assertTrue(RGX_REPR.match(repr(cond)))
 
@@ -776,7 +810,8 @@ def test_repr(self):
         self.assertTrue(RGX_REPR.match(repr(cond)))
 
     def test_context_manager(self):
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
 
         with self.assertWarns(DeprecationWarning):
             @asyncio.coroutine
@@ -790,7 +825,8 @@ def acquire_cond():
         self.assertFalse(cond.locked())
 
     def test_context_manager_no_yield(self):
-        cond = asyncio.Condition(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            cond = asyncio.Condition(loop=self.loop)
 
         try:
             with cond:
@@ -803,8 +839,9 @@ def test_context_manager_no_yield(self):
         self.assertFalse(cond.locked())
 
     def test_explicit_lock(self):
-        lock = asyncio.Lock(loop=self.loop)
-        cond = asyncio.Condition(lock, loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
+            cond = asyncio.Condition(lock, loop=self.loop)
 
         self.assertIs(cond._lock, lock)
         self.assertIs(cond._loop, lock._loop)
@@ -812,10 +849,10 @@ def test_explicit_lock(self):
     def test_ambiguous_loops(self):
         loop = self.new_test_loop()
         self.addCleanup(loop.close)
-
-        lock = asyncio.Lock(loop=self.loop)
-        with self.assertRaises(ValueError):
-            asyncio.Condition(lock, loop=loop)
+        with self.assertWarns(DeprecationWarning):
+            lock = asyncio.Lock(loop=self.loop)
+            with self.assertRaises(ValueError):
+                asyncio.Condition(lock, loop=loop)
 
     def test_timeout_in_block(self):
         loop = asyncio.new_event_loop()
@@ -827,7 +864,8 @@ async def task_timeout():
                 with self.assertRaises(asyncio.TimeoutError):
                     await asyncio.wait_for(condition.wait(), timeout=0.5)
 
-        loop.run_until_complete(task_timeout())
+        with self.assertWarns(DeprecationWarning):
+            loop.run_until_complete(task_timeout())
 
 
 class SemaphoreTests(test_utils.TestCase):
@@ -838,10 +876,12 @@ def setUp(self):
 
     def test_ctor_loop(self):
         loop = mock.Mock()
-        sem = asyncio.Semaphore(loop=loop)
+        with self.assertWarns(DeprecationWarning):
+            sem = asyncio.Semaphore(loop=loop)
         self.assertIs(sem._loop, loop)
 
-        sem = asyncio.Semaphore(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            sem = asyncio.Semaphore(loop=self.loop)
         self.assertIs(sem._loop, self.loop)
 
     def test_ctor_noloop(self):
@@ -850,11 +890,13 @@ def test_ctor_noloop(self):
         self.assertIs(sem._loop, self.loop)
 
     def test_initial_value_zero(self):
-        sem = asyncio.Semaphore(0, loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            sem = asyncio.Semaphore(0, loop=self.loop)
         self.assertTrue(sem.locked())
 
     def test_repr(self):
-        sem = asyncio.Semaphore(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            sem = asyncio.Semaphore(loop=self.loop)
         self.assertTrue(repr(sem).endswith('[unlocked, value:1]>'))
         self.assertTrue(RGX_REPR.match(repr(sem)))
 
@@ -872,7 +914,8 @@ def test_repr(self):
         self.assertTrue(RGX_REPR.match(repr(sem)))
 
     def test_semaphore(self):
-        sem = asyncio.Semaphore(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            sem = asyncio.Semaphore(loop=self.loop)
         self.assertEqual(1, sem._value)
 
         with self.assertWarns(DeprecationWarning):
@@ -895,7 +938,8 @@ def test_semaphore_value(self):
         self.assertRaises(ValueError, asyncio.Semaphore, -1)
 
     def test_acquire(self):
-        sem = asyncio.Semaphore(3, loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            sem = asyncio.Semaphore(3, loop=self.loop)
         result = []
 
         self.assertTrue(self.loop.run_until_complete(sem.acquire()))
@@ -956,7 +1000,8 @@ async def c4(result):
         self.loop.run_until_complete(asyncio.gather(*race_tasks))
 
     def test_acquire_cancel(self):
-        sem = asyncio.Semaphore(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            sem = asyncio.Semaphore(loop=self.loop)
         self.loop.run_until_complete(sem.acquire())
 
         acquire = asyncio.Task(sem.acquire(), loop=self.loop)
@@ -968,7 +1013,8 @@ def test_acquire_cancel(self):
                         all(waiter.done() for waiter in sem._waiters))
 
     def test_acquire_cancel_before_awoken(self):
-        sem = asyncio.Semaphore(value=0, loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            sem = asyncio.Semaphore(value=0, loop=self.loop)
 
         t1 = asyncio.Task(sem.acquire(), loop=self.loop)
         t2 = asyncio.Task(sem.acquire(), loop=self.loop)
@@ -990,7 +1036,8 @@ def test_acquire_cancel_before_awoken(self):
         test_utils.run_briefly(self.loop)
 
     def test_acquire_hang(self):
-        sem = asyncio.Semaphore(value=0, loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            sem = asyncio.Semaphore(value=0, loop=self.loop)
 
         t1 = asyncio.Task(sem.acquire(), loop=self.loop)
         t2 = asyncio.Task(sem.acquire(), loop=self.loop)
@@ -1004,12 +1051,14 @@ def test_acquire_hang(self):
         self.assertTrue(sem.locked())
 
     def test_release_not_acquired(self):
-        sem = asyncio.BoundedSemaphore(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            sem = asyncio.BoundedSemaphore(loop=self.loop)
 
         self.assertRaises(ValueError, sem.release)
 
     def test_release_no_waiters(self):
-        sem = asyncio.Semaphore(loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            sem = asyncio.Semaphore(loop=self.loop)
         self.loop.run_until_complete(sem.acquire())
         self.assertTrue(sem.locked())
 
@@ -1017,9 +1066,9 @@ def test_release_no_waiters(self):
         self.assertFalse(sem.locked())
 
     def test_context_manager(self):
-        sem = asyncio.Semaphore(2, loop=self.loop)
-
         with self.assertWarns(DeprecationWarning):
+            sem = asyncio.Semaphore(2, loop=self.loop)
+
             @asyncio.coroutine
             def acquire_lock():
                 with self.assertWarns(DeprecationWarning):
@@ -1035,7 +1084,8 @@ def acquire_lock():
         self.assertEqual(2, sem._value)
 
     def test_context_manager_no_yield(self):
-        sem = asyncio.Semaphore(2, loop=self.loop)
+        with self.assertWarns(DeprecationWarning):
+            sem = asyncio.Semaphore(2, loop=self.loop)
 
         try:
             with sem:
diff --git a/Lib/test/test_asyncio/test_pep492.py b/Lib/test/test_asyncio/test_pep492.py
index a5cf37ded7c964..f327bbd1d66da7 100644
--- a/Lib/test/test_asyncio/test_pep492.py
+++ b/Lib/test/test_asyncio/test_pep492.py
@@ -43,12 +43,13 @@ def setUp(self):
 class LockTests(BaseTest):
 
     def test_context_manager_async_with(self):
-        primitives = [
-            asyncio.Lock(loop=self.loop),
-            asyncio.Condition(loop=self.loop),
-            asyncio.Semaphore(loop=self.loop),
-            asyncio.BoundedSemaphore(loop=self.loop),
-        ]
+        with self.assertWarns(DeprecationWarning):
+            primitives = [
+                asyncio.Lock(loop=self.loop),
+                asyncio.Condition(loop=self.loop),
+                asyncio.Semaphore(loop=self.loop),
+                asyncio.BoundedSemaphore(loop=self.loop),
+            ]
 
         async def test(lock):
             await asyncio.sleep(0.01)
@@ -65,12 +66,13 @@ async def test(lock):
             self.assertFalse(primitive.locked())
 
     def test_context_manager_with_await(self):
-        primitives = [
-            asyncio.Lock(loop=self.loop),
-            asyncio.Condition(loop=self.loop),
-            asyncio.Semaphore(loop=self.loop),
-            asyncio.BoundedSemaphore(loop=self.loop),
-        ]
+        with self.assertWarns(DeprecationWarning):
+            primitives = [
+                asyncio.Lock(loop=self.loop),
+                asyncio.Condition(loop=self.loop),
+                asyncio.Semaphore(loop=self.loop),
+                asyncio.BoundedSemaphore(loop=self.loop),
+            ]
 
         async def test(lock):
             await asyncio.sleep(0.01)
diff --git a/Lib/test/test_asyncio/test_queues.py b/Lib/test/test_asyncio/test_queues.py
index b0f0f9c8fdb7c1..02e8e43ccee698 100644
--- a/Lib/test/test_asyncio/test_queues.py
+++ b/Lib/test/test_asyncio/test_queues.py
@@ -35,7 +35,8 @@ def gen():
 
         loop = self.new_test_loop(gen)
 
-        q = asyncio.Queue(loop=loop)
+        with self.assertWarns(DeprecationWarning):
+            q = asyncio.Queue(loop=loop)
         self.assertTrue(fn(q).startswith('
Date: Tue, 10 Sep 2019 05:50:14 -0700
Subject: [PATCH 0541/2163] bpo-36373: Deprecate explicit loop parameter in all
 public asyncio APIs [queue] (GH-13950)

This PR deprecate explicit loop parameters in all public asyncio APIs

This issues is split to be easier to review.

fourth step: queue.py

https://bugs.python.org/issue36373
(cherry picked from commit 9008be303a89bfab8c3314c6a42330b5523adc8b)

Co-authored-by: Emmanuel Arias 
---
 Doc/library/asyncio-queue.rst | 4 ++++
 Lib/asyncio/queues.py         | 4 ++++
 2 files changed, 8 insertions(+)

diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst
index 7be1023c80cc66..e7775da54f5891 100644
--- a/Doc/library/asyncio-queue.rst
+++ b/Doc/library/asyncio-queue.rst
@@ -32,6 +32,10 @@ Queue
    the queue is always known and can be returned by calling the
    :meth:`qsize` method.
 
+   .. deprecated-removed:: 3.8 3.10
+      The *loop* parameter.
+
+
    This class is :ref:`not thread safe `.
 
    .. attribute:: maxsize
diff --git a/Lib/asyncio/queues.py b/Lib/asyncio/queues.py
index 2a8d3e76044109..c96b4a0c1ba327 100644
--- a/Lib/asyncio/queues.py
+++ b/Lib/asyncio/queues.py
@@ -2,6 +2,7 @@
 
 import collections
 import heapq
+import warnings
 
 from . import events
 from . import locks
@@ -34,6 +35,9 @@ def __init__(self, maxsize=0, *, loop=None):
             self._loop = events.get_event_loop()
         else:
             self._loop = loop
+            warnings.warn("The loop argument is deprecated since Python 3.8, "
+                          "and scheduled for removal in Python 3.10.",
+                          DeprecationWarning, stacklevel=2)
         self._maxsize = maxsize
 
         # Futures.

From eb1bc48c74f4f8af88b5276729f9652201e46324 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 05:55:12 -0700
Subject: [PATCH 0542/2163] bpo-37619: update_one_slot() should not ignore
 wrapper descriptors for wrong type (GH-15838)

(cherry picked from commit 57ea33560662e0f20a3b0334bb20065771edf4da)

Co-authored-by: Jeroen Demeyer 
---
 Lib/test/test_descr.py                        | 12 ++++++++++++
 .../2019-07-18-11-50-49.bpo-37619.X6Lulo.rst  |  3 +++
 Objects/typeobject.c                          | 19 +++++++++++++------
 3 files changed, 28 insertions(+), 6 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-07-18-11-50-49.bpo-37619.X6Lulo.rst

diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py
index 741cb6dce285fb..c4d900a8c1bcb9 100644
--- a/Lib/test/test_descr.py
+++ b/Lib/test/test_descr.py
@@ -4649,6 +4649,18 @@ class X(dict):
         self.assertEqual(x["y"], 42)
         self.assertEqual(x, -x)
 
+    def test_wrong_class_slot_wrapper(self):
+        # Check bpo-37619: a wrapper descriptor taken from the wrong class
+        # should raise an exception instead of silently being ignored
+        class A(int):
+            __eq__ = str.__eq__
+            __add__ = str.__add__
+        a = A()
+        with self.assertRaises(TypeError):
+            a == a
+        with self.assertRaises(TypeError):
+            a + a
+
     def test_slot_shadows_class_variable(self):
         with self.assertRaises(ValueError) as cm:
             class X:
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-07-18-11-50-49.bpo-37619.X6Lulo.rst b/Misc/NEWS.d/next/Core and Builtins/2019-07-18-11-50-49.bpo-37619.X6Lulo.rst
new file mode 100644
index 00000000000000..8723d3d9e9da09
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-07-18-11-50-49.bpo-37619.X6Lulo.rst	
@@ -0,0 +1,3 @@
+When adding a wrapper descriptor from one class to a different class
+(for example, setting ``__add__ = str.__add__`` on an ``int`` subclass),
+an exception is correctly raised when the operator is called.
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
index 8ece21827519f2..f42caedf6cc208 100644
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -7243,14 +7243,21 @@ update_one_slot(PyTypeObject *type, slotdef *p)
             if (tptr == NULL || tptr == ptr)
                 generic = p->function;
             d = (PyWrapperDescrObject *)descr;
-            if (d->d_base->wrapper == p->wrapper &&
+            if ((specific == NULL || specific == d->d_wrapped) &&
+                d->d_base->wrapper == p->wrapper &&
                 PyType_IsSubtype(type, PyDescr_TYPE(d)))
             {
-                if (specific == NULL ||
-                    specific == d->d_wrapped)
-                    specific = d->d_wrapped;
-                else
-                    use_generic = 1;
+                specific = d->d_wrapped;
+            }
+            else {
+                /* We cannot use the specific slot function because either
+                   - it is not unique: there are multiple methods for this
+                     slot and they conflict
+                   - the signature is wrong (as checked by the ->wrapper
+                     comparison above)
+                   - it's wrapping the wrong class
+                 */
+                use_generic = 1;
             }
         }
         else if (Py_TYPE(descr) == &PyCFunction_Type &&

From eaa1b09412cc82ba830fd731016278af26b9099b Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 06:15:19 -0700
Subject: [PATCH 0543/2163] docs: Add references to AsyncMock in
 unittest.mock.patch (GH-13681)

Update the docs as patch can now return an AsyncMock if the patched
object is an async function.
(cherry picked from commit f5e7f39d2916ed150e80381faed125f405a11e11)

Co-authored-by: Mario Corchero 
---
 Doc/library/unittest.mock.rst | 20 ++++++++++++++++----
 Lib/unittest/mock.py          |  9 +++++----
 2 files changed, 21 insertions(+), 8 deletions(-)

diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst
index b2547546f3d6d7..04ff8a61da3c56 100644
--- a/Doc/library/unittest.mock.rst
+++ b/Doc/library/unittest.mock.rst
@@ -1321,8 +1321,10 @@ patch
     is patched with a *new* object. When the function/with statement exits
     the patch is undone.
 
-    If *new* is omitted, then the target is replaced with a
-    :class:`MagicMock`. If :func:`patch` is used as a decorator and *new* is
+    If *new* is omitted, then the target is replaced with an
+    :class:`AsyncMock` if the patched object is an async function or
+    a :class:`MagicMock` otherwise.
+    If :func:`patch` is used as a decorator and *new* is
     omitted, the created mock is passed in as an extra argument to the
     decorated function. If :func:`patch` is used as a context manager the created
     mock is returned by the context manager.
@@ -1340,8 +1342,8 @@ patch
     patch to pass in the object being mocked as the spec/spec_set object.
 
     *new_callable* allows you to specify a different class, or callable object,
-    that will be called to create the *new* object. By default :class:`MagicMock` is
-    used.
+    that will be called to create the *new* object. By default :class:`AsyncMock`
+    is used for async functions and :class:`MagicMock` for the rest.
 
     A more powerful form of *spec* is *autospec*. If you set ``autospec=True``
     then the mock will be created with a spec from the object being replaced.
@@ -1505,6 +1507,10 @@ work as expected::
     ...
     >>> test()
 
+.. versionchanged:: 3.8
+
+    :func:`patch` now returns an :class:`AsyncMock` if the target is an async function.
+
 
 patch.object
 ~~~~~~~~~~~~
@@ -2289,6 +2295,12 @@ See :ref:`auto-speccing` for examples of how to use auto-speccing with
 :func:`create_autospec` and the *autospec* argument to :func:`patch`.
 
 
+.. versionchanged:: 3.8
+
+    :func:`create_autospec` now returns an :class:`AsyncMock` if the target is
+    an async function.
+
+
 ANY
 ~~~
 
diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py
index 7a4fcf4e3aa97f..915883db880291 100644
--- a/Lib/unittest/mock.py
+++ b/Lib/unittest/mock.py
@@ -1631,8 +1631,9 @@ def patch(
     is patched with a `new` object. When the function/with statement exits
     the patch is undone.
 
-    If `new` is omitted, then the target is replaced with a
-    `MagicMock`. If `patch` is used as a decorator and `new` is
+    If `new` is omitted, then the target is replaced with an
+    `AsyncMock if the patched object is an async function or a
+    `MagicMock` otherwise. If `patch` is used as a decorator and `new` is
     omitted, the created mock is passed in as an extra argument to the
     decorated function. If `patch` is used as a context manager the created
     mock is returned by the context manager.
@@ -1650,8 +1651,8 @@ def patch(
     patch to pass in the object being mocked as the spec/spec_set object.
 
     `new_callable` allows you to specify a different class, or callable object,
-    that will be called to create the `new` object. By default `MagicMock` is
-    used.
+    that will be called to create the `new` object. By default `AsyncMock` is
+    used for async functions and `MagicMock` for the rest.
 
     A more powerful form of `spec` is `autospec`. If you set `autospec=True`
     then the mock will be created with a spec from the object being replaced.

From c3008dd480d645678409273eecbfd24bbc9669d7 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 06:16:00 -0700
Subject: [PATCH 0544/2163] bpo-37251: Removes __code__ check from
 _is_async_obj. (GH-15830)

(cherry picked from commit f1a297acb60b88917712450ebd3cfa707e6efd6b)

Co-authored-by: Lisa Roach 
---
 Lib/unittest/mock.py                              |  5 ++---
 Lib/unittest/test/testmock/testasync.py           | 15 +++++++++++++++
 .../2019-09-10-10-59-50.bpo-37251.8zn2o3.rst      |  3 +++
 3 files changed, 20 insertions(+), 3 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2019-09-10-10-59-50.bpo-37251.8zn2o3.rst

diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py
index 915883db880291..7bfbfc92eef9fb 100644
--- a/Lib/unittest/mock.py
+++ b/Lib/unittest/mock.py
@@ -46,10 +46,9 @@
 _safe_super = super
 
 def _is_async_obj(obj):
-    if getattr(obj, '__code__', None):
-        return asyncio.iscoroutinefunction(obj) or inspect.isawaitable(obj)
-    else:
+    if _is_instance_mock(obj) and not isinstance(obj, AsyncMock):
         return False
+    return asyncio.iscoroutinefunction(obj) or inspect.isawaitable(obj)
 
 
 def _is_async_func(func):
diff --git a/Lib/unittest/test/testmock/testasync.py b/Lib/unittest/test/testmock/testasync.py
index fa906e4f7152f8..4660bea3e1235d 100644
--- a/Lib/unittest/test/testmock/testasync.py
+++ b/Lib/unittest/test/testmock/testasync.py
@@ -18,6 +18,10 @@ async def async_method(self):
     def normal_method(self):
         pass
 
+class AwaitableClass:
+    def __await__(self):
+        yield
+
 async def async_func():
     pass
 
@@ -160,6 +164,10 @@ def test_create_autospec_instance(self):
         with self.assertRaises(RuntimeError):
             create_autospec(async_func, instance=True)
 
+    def test_create_autospec_awaitable_class(self):
+        awaitable_mock = create_autospec(spec=AwaitableClass())
+        self.assertIsInstance(create_autospec(awaitable_mock), AsyncMock)
+
     def test_create_autospec(self):
         spec = create_autospec(async_func_args)
         awaitable = spec(1, 2, c=3)
@@ -321,6 +329,13 @@ def test_is_child_AsyncMock(self):
         self.assertIsInstance(mock.normal_method, MagicMock)
         self.assertIsInstance(mock, MagicMock)
 
+    def test_magicmock_lambda_spec(self):
+        mock_obj = MagicMock()
+        mock_obj.mock_func = MagicMock(spec=lambda x: x)
+
+        with patch.object(mock_obj, "mock_func") as cm:
+            self.assertIsInstance(cm, MagicMock)
+
 
 class AsyncArguments(unittest.TestCase):
     def test_add_return_value(self):
diff --git a/Misc/NEWS.d/next/Library/2019-09-10-10-59-50.bpo-37251.8zn2o3.rst b/Misc/NEWS.d/next/Library/2019-09-10-10-59-50.bpo-37251.8zn2o3.rst
new file mode 100644
index 00000000000000..27fd1e46963c72
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-10-10-59-50.bpo-37251.8zn2o3.rst
@@ -0,0 +1,3 @@
+Remove `__code__` check in AsyncMock that incorrectly
+evaluated function specs as async objects but failed to evaluate classes
+with `__await__` but no `__code__` attribute defined as async objects.

From c43f26eca35f22707a52fb8f3fbfc9340639b58d Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 06:21:57 -0700
Subject: [PATCH 0545/2163] closes bpo-25461: Update os.walk() docstring to
 match the online docs. (GH-11836)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

(cherry picked from commit 734f1202a50641eb2c4bfbcd5b75247c1dc99a8f)

Co-authored-by: Bernt Røskar Brenna 
---
 Lib/os.py | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/Lib/os.py b/Lib/os.py
index 200902528cbfe9..52d3f1d7415854 100644
--- a/Lib/os.py
+++ b/Lib/os.py
@@ -300,10 +300,11 @@ def walk(top, topdown=True, onerror=None, followlinks=False):
     (e.g., via del or slice assignment), and walk will only recurse into the
     subdirectories whose names remain in dirnames; this can be used to prune the
     search, or to impose a specific order of visiting.  Modifying dirnames when
-    topdown is false is ineffective, since the directories in dirnames have
-    already been generated by the time dirnames itself is generated. No matter
-    the value of topdown, the list of subdirectories is retrieved before the
-    tuples for the directory and its subdirectories are generated.
+    topdown is false has no effect on the behavior of os.walk(), since the
+    directories in dirnames have already been generated by the time dirnames
+    itself is generated. No matter the value of topdown, the list of
+    subdirectories is retrieved before the tuples for the directory and its
+    subdirectories are generated.
 
     By default errors from the os.scandir() call are ignored.  If
     optional arg 'onerror' is specified, it should be a function; it

From 5cf8155bbd2c65572295b26e96c221763b3322f0 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 06:32:56 -0700
Subject: [PATCH 0546/2163] bpo-21018: added missing documentation about
 escaping characters for configparser (GH-6137) (GH-15846)

Document how $ and % can be escaped in configparser.
(cherry picked from commit 9a94093189417adddd6b59d6c80cc5544630c8aa)

Co-authored-by: Arun Persaud 
---
 Doc/library/configparser.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst
index 04b52dc7b21531..739477f55fddda 100644
--- a/Doc/library/configparser.rst
+++ b/Doc/library/configparser.rst
@@ -313,6 +313,8 @@ from ``get()`` calls.
       my_dir: %(home_dir)s/lumberjack
       my_pictures: %(my_dir)s/Pictures
 
+      [Escape]
+      gain: 80%%  # use a %% to escape the % sign (% is the only character that needs to be escaped)
 
    In the example above, :class:`ConfigParser` with *interpolation* set to
    ``BasicInterpolation()`` would resolve ``%(home_dir)s`` to the value of
@@ -346,6 +348,9 @@ from ``get()`` calls.
       my_dir: ${home_dir}/lumberjack
       my_pictures: ${my_dir}/Pictures
 
+      [Escape]
+      cost: $$80  # use a $$ to escape the $ sign ($ is the only character that needs to be escaped)
+
    Values from other sections can be fetched as well:
 
    .. code-block:: ini

From f12ff05bc07ac087743a4615c4af5f66b73f8d2c Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 06:44:32 -0700
Subject: [PATCH 0547/2163] bpo-38066: Hide internal Stream methods (GH-15762)

feed_eof(), feed_data(), set_exception(), and set_transport() are prefixed with underscore now.

https://bugs.python.org/issue38066
(cherry picked from commit 12c122ae958a55c9874ed4c7d7805ceb084411d7)

Co-authored-by: Andrew Svetlov 
---
 Lib/asyncio/streams.py                        |  49 ++++-
 Lib/asyncio/subprocess.py                     |  10 +-
 Lib/test/test_asyncio/test_pep492.py          |   4 +-
 Lib/test/test_asyncio/test_streams.py         | 182 ++++++++++--------
 .../2019-09-09-14-39-47.bpo-38066.l9mWv-.rst  |   2 +
 5 files changed, 155 insertions(+), 92 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2019-09-09-14-39-47.bpo-38066.l9mWv-.rst

diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py
index 1d3e487b1d932d..01aa6ffd49e431 100644
--- a/Lib/asyncio/streams.py
+++ b/Lib/asyncio/streams.py
@@ -1133,9 +1133,9 @@ def connection_lost(self, exc):
         stream = self._stream
         if stream is not None:
             if exc is None:
-                stream.feed_eof()
+                stream._feed_eof()
             else:
-                stream.set_exception(exc)
+                stream._set_exception(exc)
         if not self._closed.done():
             if exc is None:
                 self._closed.set_result(None)
@@ -1147,12 +1147,12 @@ def connection_lost(self, exc):
     def data_received(self, data):
         stream = self._stream
         if stream is not None:
-            stream.feed_data(data)
+            stream._feed_data(data)
 
     def eof_received(self):
         stream = self._stream
         if stream is not None:
-            stream.feed_eof()
+            stream._feed_eof()
         if self._over_ssl:
             # Prevent a warning in SSLProtocol.eof_received:
             # "returning true from eof_received()
@@ -1219,7 +1219,7 @@ def connection_made(self, transport):
         stream = self._stream
         if stream is None:
             return
-        stream.set_transport(transport)
+        stream._set_transport(transport)
         stream._protocol = self
 
     def connection_lost(self, exc):
@@ -1351,6 +1351,11 @@ def is_server_side(self):
 
     @property
     def transport(self):
+        warnings.warn("Stream.transport attribute is deprecated "
+                      "since Python 3.8 and is scheduled for removal in 3.10; "
+                      "it is an internal API",
+                      DeprecationWarning,
+                      stacklevel=2)
         return self._transport
 
     def write(self, data):
@@ -1366,7 +1371,7 @@ def writelines(self, data):
     def _fast_drain(self):
         # The helper tries to use fast-path to return already existing
         # complete future object if underlying transport is not paused
-        #and actual waiting for writing resume is not needed
+        # and actual waiting for writing resume is not needed
         exc = self.exception()
         if exc is not None:
             fut = self._loop.create_future()
@@ -1450,6 +1455,14 @@ def exception(self):
         return self._exception
 
     def set_exception(self, exc):
+        warnings.warn("Stream.set_exception() is deprecated "
+                      "since Python 3.8 and is scheduled for removal in 3.10; "
+                      "it is an internal API",
+                      DeprecationWarning,
+                      stacklevel=2)
+        self._set_exception(exc)
+
+    def _set_exception(self, exc):
         self._exception = exc
 
         waiter = self._waiter
@@ -1467,6 +1480,14 @@ def _wakeup_waiter(self):
                 waiter.set_result(None)
 
     def set_transport(self, transport):
+        warnings.warn("Stream.set_transport() is deprecated "
+                      "since Python 3.8 and is scheduled for removal in 3.10; "
+                      "it is an internal API",
+                      DeprecationWarning,
+                      stacklevel=2)
+        self._set_transport(transport)
+
+    def _set_transport(self, transport):
         if transport is self._transport:
             return
         assert self._transport is None, 'Transport already set'
@@ -1478,6 +1499,14 @@ def _maybe_resume_transport(self):
             self._transport.resume_reading()
 
     def feed_eof(self):
+        warnings.warn("Stream.feed_eof() is deprecated "
+                      "since Python 3.8 and is scheduled for removal in 3.10; "
+                      "it is an internal API",
+                      DeprecationWarning,
+                      stacklevel=2)
+        self._feed_eof()
+
+    def _feed_eof(self):
         self._eof = True
         self._wakeup_waiter()
 
@@ -1486,6 +1515,14 @@ def at_eof(self):
         return self._eof and not self._buffer
 
     def feed_data(self, data):
+        warnings.warn("Stream.feed_data() is deprecated "
+                      "since Python 3.8 and is scheduled for removal in 3.10; "
+                      "it is an internal API",
+                      DeprecationWarning,
+                      stacklevel=2)
+        self._feed_data(data)
+
+    def _feed_data(self, data):
         _ensure_can_read(self._mode)
         assert not self._eof, 'feed_data after feed_eof'
 
diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py
index e6bec71d6c7dac..e6c1c8b61cc3c6 100644
--- a/Lib/asyncio/subprocess.py
+++ b/Lib/asyncio/subprocess.py
@@ -50,7 +50,7 @@ def connection_made(self, transport):
                                          limit=self._limit,
                                          loop=self._loop,
                                          _asyncio_internal=True)
-            self.stdout.set_transport(stdout_transport)
+            self.stdout._set_transport(stdout_transport)
             self._pipe_fds.append(1)
 
         stderr_transport = transport.get_pipe_transport(2)
@@ -61,7 +61,7 @@ def connection_made(self, transport):
                                          limit=self._limit,
                                          loop=self._loop,
                                          _asyncio_internal=True)
-            self.stderr.set_transport(stderr_transport)
+            self.stderr._set_transport(stderr_transport)
             self._pipe_fds.append(2)
 
         stdin_transport = transport.get_pipe_transport(0)
@@ -80,7 +80,7 @@ def pipe_data_received(self, fd, data):
         else:
             reader = None
         if reader is not None:
-            reader.feed_data(data)
+            reader._feed_data(data)
 
     def pipe_connection_lost(self, fd, exc):
         if fd == 0:
@@ -101,9 +101,9 @@ def pipe_connection_lost(self, fd, exc):
             reader = None
         if reader is not None:
             if exc is None:
-                reader.feed_eof()
+                reader._feed_eof()
             else:
-                reader.set_exception(exc)
+                reader._set_exception(exc)
 
         if fd in self._pipe_fds:
             self._pipe_fds.remove(fd)
diff --git a/Lib/test/test_asyncio/test_pep492.py b/Lib/test/test_asyncio/test_pep492.py
index f327bbd1d66da7..dfb68b8e85d49f 100644
--- a/Lib/test/test_asyncio/test_pep492.py
+++ b/Lib/test/test_asyncio/test_pep492.py
@@ -98,8 +98,8 @@ def test_readline(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(DATA)
-        stream.feed_eof()
+        stream._feed_data(DATA)
+        stream._feed_eof()
 
         async def reader():
             data = []
diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py
index eab71e80308fef..428c863d9fa676 100644
--- a/Lib/test/test_asyncio/test_streams.py
+++ b/Lib/test/test_asyncio/test_streams.py
@@ -165,7 +165,7 @@ def test_feed_empty_data(self):
                                 loop=self.loop,
                                 _asyncio_internal=True)
 
-        stream.feed_data(b'')
+        stream._feed_data(b'')
         self.assertEqual(b'', stream._buffer)
 
     def test_feed_nonempty_data(self):
@@ -173,7 +173,7 @@ def test_feed_nonempty_data(self):
                                 loop=self.loop,
                                 _asyncio_internal=True)
 
-        stream.feed_data(self.DATA)
+        stream._feed_data(self.DATA)
         self.assertEqual(self.DATA, stream._buffer)
 
     def test_read_zero(self):
@@ -181,7 +181,7 @@ def test_read_zero(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(self.DATA)
+        stream._feed_data(self.DATA)
 
         data = self.loop.run_until_complete(stream.read(0))
         self.assertEqual(b'', data)
@@ -195,7 +195,7 @@ def test_read(self):
         read_task = asyncio.Task(stream.read(30), loop=self.loop)
 
         def cb():
-            stream.feed_data(self.DATA)
+            stream._feed_data(self.DATA)
         self.loop.call_soon(cb)
 
         data = self.loop.run_until_complete(read_task)
@@ -207,8 +207,8 @@ def test_read_line_breaks(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(b'line1')
-        stream.feed_data(b'line2')
+        stream._feed_data(b'line1')
+        stream._feed_data(b'line2')
 
         data = self.loop.run_until_complete(stream.read(5))
 
@@ -223,7 +223,7 @@ def test_read_eof(self):
         read_task = asyncio.Task(stream.read(1024), loop=self.loop)
 
         def cb():
-            stream.feed_eof()
+            stream._feed_eof()
         self.loop.call_soon(cb)
 
         data = self.loop.run_until_complete(read_task)
@@ -238,9 +238,9 @@ def test_read_until_eof(self):
         read_task = asyncio.Task(stream.read(-1), loop=self.loop)
 
         def cb():
-            stream.feed_data(b'chunk1\n')
-            stream.feed_data(b'chunk2')
-            stream.feed_eof()
+            stream._feed_data(b'chunk1\n')
+            stream._feed_data(b'chunk2')
+            stream._feed_eof()
         self.loop.call_soon(cb)
 
         data = self.loop.run_until_complete(read_task)
@@ -252,12 +252,12 @@ def test_read_exception(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(b'line\n')
+        stream._feed_data(b'line\n')
 
         data = self.loop.run_until_complete(stream.read(2))
         self.assertEqual(b'li', data)
 
-        stream.set_exception(ValueError())
+        stream._set_exception(ValueError())
         self.assertRaises(
             ValueError, self.loop.run_until_complete, stream.read(2))
 
@@ -276,7 +276,7 @@ def test_read_limit(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 limit=3, loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(b'chunk')
+        stream._feed_data(b'chunk')
         data = self.loop.run_until_complete(stream.read(5))
         self.assertEqual(b'chunk', data)
         self.assertEqual(b'', stream._buffer)
@@ -287,13 +287,13 @@ def test_readline(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(b'chunk1 ')
+        stream._feed_data(b'chunk1 ')
         read_task = asyncio.Task(stream.readline(), loop=self.loop)
 
         def cb():
-            stream.feed_data(b'chunk2 ')
-            stream.feed_data(b'chunk3 ')
-            stream.feed_data(b'\n chunk4')
+            stream._feed_data(b'chunk2 ')
+            stream._feed_data(b'chunk3 ')
+            stream._feed_data(b'\n chunk4')
         self.loop.call_soon(cb)
 
         line = self.loop.run_until_complete(read_task)
@@ -307,8 +307,8 @@ def test_readline_limit_with_existing_data(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 limit=3, loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(b'li')
-        stream.feed_data(b'ne1\nline2\n')
+        stream._feed_data(b'li')
+        stream._feed_data(b'ne1\nline2\n')
 
         self.assertRaises(
             ValueError, self.loop.run_until_complete, stream.readline())
@@ -318,9 +318,9 @@ def test_readline_limit_with_existing_data(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 limit=3, loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(b'li')
-        stream.feed_data(b'ne1')
-        stream.feed_data(b'li')
+        stream._feed_data(b'li')
+        stream._feed_data(b'ne1')
+        stream._feed_data(b'li')
 
         self.assertRaises(
             ValueError, self.loop.run_until_complete, stream.readline())
@@ -337,14 +337,14 @@ def test_at_eof(self):
                                 _asyncio_internal=True)
         self.assertFalse(stream.at_eof())
 
-        stream.feed_data(b'some data\n')
+        stream._feed_data(b'some data\n')
         self.assertFalse(stream.at_eof())
 
         self.loop.run_until_complete(stream.readline())
         self.assertFalse(stream.at_eof())
 
-        stream.feed_data(b'some data\n')
-        stream.feed_eof()
+        stream._feed_data(b'some data\n')
+        stream._feed_eof()
         self.loop.run_until_complete(stream.readline())
         self.assertTrue(stream.at_eof())
 
@@ -356,10 +356,10 @@ def test_readline_limit(self):
                                 limit=7, loop=self.loop,
                                 _asyncio_internal=True)
         def cb():
-            stream.feed_data(b'chunk1')
-            stream.feed_data(b'chunk2')
-            stream.feed_data(b'chunk3\n')
-            stream.feed_eof()
+            stream._feed_data(b'chunk1')
+            stream._feed_data(b'chunk2')
+            stream._feed_data(b'chunk3\n')
+            stream._feed_eof()
         self.loop.call_soon(cb)
 
         self.assertRaises(
@@ -372,10 +372,10 @@ def cb():
                                 limit=7, loop=self.loop,
                                 _asyncio_internal=True)
         def cb():
-            stream.feed_data(b'chunk1')
-            stream.feed_data(b'chunk2\n')
-            stream.feed_data(b'chunk3\n')
-            stream.feed_eof()
+            stream._feed_data(b'chunk1')
+            stream._feed_data(b'chunk2\n')
+            stream._feed_data(b'chunk3\n')
+            stream._feed_eof()
         self.loop.call_soon(cb)
 
         self.assertRaises(
@@ -386,17 +386,17 @@ def cb():
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 limit=7, loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(b'1234567\n')
+        stream._feed_data(b'1234567\n')
         line = self.loop.run_until_complete(stream.readline())
         self.assertEqual(b'1234567\n', line)
         self.assertEqual(b'', stream._buffer)
 
-        stream.feed_data(b'12345678\n')
+        stream._feed_data(b'12345678\n')
         with self.assertRaises(ValueError) as cm:
             self.loop.run_until_complete(stream.readline())
         self.assertEqual(b'', stream._buffer)
 
-        stream.feed_data(b'12345678')
+        stream._feed_data(b'12345678')
         with self.assertRaises(ValueError) as cm:
             self.loop.run_until_complete(stream.readline())
         self.assertEqual(b'', stream._buffer)
@@ -407,8 +407,8 @@ def test_readline_nolimit_nowait(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(self.DATA[:6])
-        stream.feed_data(self.DATA[6:])
+        stream._feed_data(self.DATA[:6])
+        stream._feed_data(self.DATA[6:])
 
         line = self.loop.run_until_complete(stream.readline())
 
@@ -419,8 +419,8 @@ def test_readline_eof(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(b'some data')
-        stream.feed_eof()
+        stream._feed_data(b'some data')
+        stream._feed_eof()
 
         line = self.loop.run_until_complete(stream.readline())
         self.assertEqual(b'some data', line)
@@ -429,7 +429,7 @@ def test_readline_empty_eof(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_eof()
+        stream._feed_eof()
 
         line = self.loop.run_until_complete(stream.readline())
         self.assertEqual(b'', line)
@@ -438,7 +438,7 @@ def test_readline_read_byte_count(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(self.DATA)
+        stream._feed_data(self.DATA)
 
         self.loop.run_until_complete(stream.readline())
 
@@ -451,12 +451,12 @@ def test_readline_exception(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(b'line\n')
+        stream._feed_data(b'line\n')
 
         data = self.loop.run_until_complete(stream.readline())
         self.assertEqual(b'line\n', data)
 
-        stream.set_exception(ValueError())
+        stream._set_exception(ValueError())
         self.assertRaises(
             ValueError, self.loop.run_until_complete, stream.readline())
         self.assertEqual(b'', stream._buffer)
@@ -473,17 +473,17 @@ def test_readuntil_multi_chunks(self):
                                 loop=self.loop,
                                 _asyncio_internal=True)
 
-        stream.feed_data(b'lineAAA')
+        stream._feed_data(b'lineAAA')
         data = self.loop.run_until_complete(stream.readuntil(separator=b'AAA'))
         self.assertEqual(b'lineAAA', data)
         self.assertEqual(b'', stream._buffer)
 
-        stream.feed_data(b'lineAAA')
+        stream._feed_data(b'lineAAA')
         data = self.loop.run_until_complete(stream.readuntil(b'AAA'))
         self.assertEqual(b'lineAAA', data)
         self.assertEqual(b'', stream._buffer)
 
-        stream.feed_data(b'lineAAAxxx')
+        stream._feed_data(b'lineAAAxxx')
         data = self.loop.run_until_complete(stream.readuntil(b'AAA'))
         self.assertEqual(b'lineAAA', data)
         self.assertEqual(b'xxx', stream._buffer)
@@ -493,34 +493,34 @@ def test_readuntil_multi_chunks_1(self):
                                 loop=self.loop,
                                 _asyncio_internal=True)
 
-        stream.feed_data(b'QWEaa')
-        stream.feed_data(b'XYaa')
-        stream.feed_data(b'a')
+        stream._feed_data(b'QWEaa')
+        stream._feed_data(b'XYaa')
+        stream._feed_data(b'a')
         data = self.loop.run_until_complete(stream.readuntil(b'aaa'))
         self.assertEqual(b'QWEaaXYaaa', data)
         self.assertEqual(b'', stream._buffer)
 
-        stream.feed_data(b'QWEaa')
-        stream.feed_data(b'XYa')
-        stream.feed_data(b'aa')
+        stream._feed_data(b'QWEaa')
+        stream._feed_data(b'XYa')
+        stream._feed_data(b'aa')
         data = self.loop.run_until_complete(stream.readuntil(b'aaa'))
         self.assertEqual(b'QWEaaXYaaa', data)
         self.assertEqual(b'', stream._buffer)
 
-        stream.feed_data(b'aaa')
+        stream._feed_data(b'aaa')
         data = self.loop.run_until_complete(stream.readuntil(b'aaa'))
         self.assertEqual(b'aaa', data)
         self.assertEqual(b'', stream._buffer)
 
-        stream.feed_data(b'Xaaa')
+        stream._feed_data(b'Xaaa')
         data = self.loop.run_until_complete(stream.readuntil(b'aaa'))
         self.assertEqual(b'Xaaa', data)
         self.assertEqual(b'', stream._buffer)
 
-        stream.feed_data(b'XXX')
-        stream.feed_data(b'a')
-        stream.feed_data(b'a')
-        stream.feed_data(b'a')
+        stream._feed_data(b'XXX')
+        stream._feed_data(b'a')
+        stream._feed_data(b'a')
+        stream._feed_data(b'a')
         data = self.loop.run_until_complete(stream.readuntil(b'aaa'))
         self.assertEqual(b'XXXaaa', data)
         self.assertEqual(b'', stream._buffer)
@@ -529,8 +529,8 @@ def test_readuntil_eof(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(b'some dataAA')
-        stream.feed_eof()
+        stream._feed_data(b'some dataAA')
+        stream._feed_eof()
 
         with self.assertRaises(asyncio.IncompleteReadError) as cm:
             self.loop.run_until_complete(stream.readuntil(b'AAA'))
@@ -542,7 +542,7 @@ def test_readuntil_limit_found_sep(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop, limit=3,
                                 _asyncio_internal=True)
-        stream.feed_data(b'some dataAA')
+        stream._feed_data(b'some dataAA')
 
         with self.assertRaisesRegex(asyncio.LimitOverrunError,
                                     'not found') as cm:
@@ -550,7 +550,7 @@ def test_readuntil_limit_found_sep(self):
 
         self.assertEqual(b'some dataAA', stream._buffer)
 
-        stream.feed_data(b'A')
+        stream._feed_data(b'A')
         with self.assertRaisesRegex(asyncio.LimitOverrunError,
                                     'is found') as cm:
             self.loop.run_until_complete(stream.readuntil(b'AAA'))
@@ -562,7 +562,7 @@ def test_readexactly_zero_or_less(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(self.DATA)
+        stream._feed_data(self.DATA)
 
         data = self.loop.run_until_complete(stream.readexactly(0))
         self.assertEqual(b'', data)
@@ -582,9 +582,9 @@ def test_readexactly(self):
         read_task = asyncio.Task(stream.readexactly(n), loop=self.loop)
 
         def cb():
-            stream.feed_data(self.DATA)
-            stream.feed_data(self.DATA)
-            stream.feed_data(self.DATA)
+            stream._feed_data(self.DATA)
+            stream._feed_data(self.DATA)
+            stream._feed_data(self.DATA)
         self.loop.call_soon(cb)
 
         data = self.loop.run_until_complete(read_task)
@@ -595,7 +595,7 @@ def test_readexactly_limit(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 limit=3, loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(b'chunk')
+        stream._feed_data(b'chunk')
         data = self.loop.run_until_complete(stream.readexactly(5))
         self.assertEqual(b'chunk', data)
         self.assertEqual(b'', stream._buffer)
@@ -609,8 +609,8 @@ def test_readexactly_eof(self):
         read_task = asyncio.Task(stream.readexactly(n), loop=self.loop)
 
         def cb():
-            stream.feed_data(self.DATA)
-            stream.feed_eof()
+            stream._feed_data(self.DATA)
+            stream._feed_eof()
         self.loop.call_soon(cb)
 
         with self.assertRaises(asyncio.IncompleteReadError) as cm:
@@ -625,12 +625,12 @@ def test_readexactly_exception(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(b'line\n')
+        stream._feed_data(b'line\n')
 
         data = self.loop.run_until_complete(stream.readexactly(2))
         self.assertEqual(b'li', data)
 
-        stream.set_exception(ValueError())
+        stream._set_exception(ValueError())
         self.assertRaises(
             ValueError, self.loop.run_until_complete, stream.readexactly(2))
 
@@ -641,7 +641,7 @@ def test_exception(self):
         self.assertIsNone(stream.exception())
 
         exc = ValueError()
-        stream.set_exception(exc)
+        stream._set_exception(exc)
         self.assertIs(stream.exception(), exc)
 
     def test_exception_waiter(self):
@@ -650,7 +650,7 @@ def test_exception_waiter(self):
                                 _asyncio_internal=True)
 
         async def set_err():
-            stream.set_exception(ValueError())
+            stream._set_exception(ValueError())
 
         t1 = asyncio.Task(stream.readline(), loop=self.loop)
         t2 = asyncio.Task(set_err(), loop=self.loop)
@@ -669,7 +669,7 @@ def test_exception_cancel(self):
         t.cancel()
         test_utils.run_briefly(self.loop)
         # The following line fails if set_exception() isn't careful.
-        stream.set_exception(RuntimeError('message'))
+        stream._set_exception(RuntimeError('message'))
         test_utils.run_briefly(self.loop)
         self.assertIs(stream._waiter, None)
 
@@ -993,14 +993,14 @@ def test___repr__eof(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_eof()
+        stream._feed_eof()
         self.assertEqual("", repr(stream))
 
     def test___repr__data(self):
         stream = asyncio.Stream(mode=asyncio.StreamMode.READ,
                                 loop=self.loop,
                                 _asyncio_internal=True)
-        stream.feed_data(b'data')
+        stream._feed_data(b'data')
         self.assertEqual("", repr(stream))
 
     def test___repr__exception(self):
@@ -1008,7 +1008,7 @@ def test___repr__exception(self):
                                 loop=self.loop,
                                 _asyncio_internal=True)
         exc = RuntimeError()
-        stream.set_exception(exc)
+        stream._set_exception(exc)
         self.assertEqual("",
                          repr(stream))
 
@@ -1260,7 +1260,7 @@ async def inner():
             stream = asyncio.Stream(mode=asyncio.StreamMode.WRITE,
                                     _asyncio_internal=True)
             with self.assertRaisesRegex(RuntimeError, "The stream is write-only"):
-                stream.feed_data(b'data')
+                stream._feed_data(b'data')
             with self.assertRaisesRegex(RuntimeError, "The stream is write-only"):
                 await stream.readline()
             with self.assertRaisesRegex(RuntimeError, "The stream is write-only"):
@@ -1785,6 +1785,30 @@ def test_stream_ctor_forbidden(self):
                                     "by asyncio internals only"):
             asyncio.Stream(asyncio.StreamMode.READWRITE)
 
+    def test_deprecated_methods(self):
+        async def f():
+            return asyncio.Stream(mode=asyncio.StreamMode.READWRITE,
+                                  _asyncio_internal=True)
+
+        stream = self.loop.run_until_complete(f())
+
+        tr = mock.Mock()
+
+        with self.assertWarns(DeprecationWarning):
+            stream.set_transport(tr)
+
+        with self.assertWarns(DeprecationWarning):
+            stream.transport is tr
+
+        with self.assertWarns(DeprecationWarning):
+            stream.feed_data(b'data')
+
+        with self.assertWarns(DeprecationWarning):
+            stream.feed_eof()
+
+        with self.assertWarns(DeprecationWarning):
+            stream.set_exception(ConnectionResetError("test"))
+
 
 if __name__ == '__main__':
     unittest.main()
diff --git a/Misc/NEWS.d/next/Library/2019-09-09-14-39-47.bpo-38066.l9mWv-.rst b/Misc/NEWS.d/next/Library/2019-09-09-14-39-47.bpo-38066.l9mWv-.rst
new file mode 100644
index 00000000000000..a69924fe2aa410
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-09-14-39-47.bpo-38066.l9mWv-.rst
@@ -0,0 +1,2 @@
+Hide internal asyncio.Stream methods: feed_eof(), feed_data(),
+set_exception() and set_transport().

From 206e4c3d3547b024935ea9655f960061dffbb80f Mon Sep 17 00:00:00 2001
From: Steve Dower 
Date: Tue, 10 Sep 2019 15:29:28 +0100
Subject: [PATCH 0548/2163] bpo-38087: Fix case sensitivity in test_pathlib and
 test_ntpath (GH-15850)

---
 Lib/test/test_ntpath.py                       | 196 ++++++++++--------
 Lib/test/test_pathlib.py                      |  35 ++--
 .../2019-09-10-14-21-40.bpo-38087.--eIib.rst  |   1 +
 3 files changed, 125 insertions(+), 107 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Windows/2019-09-10-14-21-40.bpo-38087.--eIib.rst

diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py
index 285fb69dc1e88f..c5c96e32d7473c 100644
--- a/Lib/test/test_ntpath.py
+++ b/Lib/test/test_ntpath.py
@@ -23,10 +23,18 @@
     HAVE_GETFINALPATHNAME = True
 
 
+def _norm(path):
+    if isinstance(path, (bytes, str, os.PathLike)):
+        return ntpath.normcase(os.fsdecode(path))
+    elif hasattr(path, "__iter__"):
+        return tuple(ntpath.normcase(os.fsdecode(p)) for p in path)
+    return path
+
+
 def tester(fn, wantResult):
     fn = fn.replace("\\", "\\\\")
     gotResult = eval(fn)
-    if wantResult != gotResult:
+    if wantResult != gotResult and _norm(wantResult) != _norm(gotResult):
         raise TestFailed("%s should return: %s but returned: %s" \
               %(str(fn), str(wantResult), str(gotResult)))
 
@@ -42,16 +50,22 @@ def tester(fn, wantResult):
     with warnings.catch_warnings():
         warnings.simplefilter("ignore", DeprecationWarning)
         gotResult = eval(fn)
-    if isinstance(wantResult, str):
-        wantResult = os.fsencode(wantResult)
-    elif isinstance(wantResult, tuple):
-        wantResult = tuple(os.fsencode(r) for r in wantResult)
-    if wantResult != gotResult:
+    if _norm(wantResult) != _norm(gotResult):
         raise TestFailed("%s should return: %s but returned: %s" \
               %(str(fn), str(wantResult), repr(gotResult)))
 
 
-class TestNtpath(unittest.TestCase):
+class NtpathTestCase(unittest.TestCase):
+    def assertPathEqual(self, path1, path2):
+        if path1 == path2 or _norm(path1) == _norm(path2):
+            return
+        self.assertEqual(path1, path2)
+
+    def assertPathIn(self, path, pathset):
+        self.assertIn(_norm(path), _norm(pathset))
+
+
+class TestNtpath(NtpathTestCase):
     def test_splitext(self):
         tester('ntpath.splitext("foo.ext")', ('foo', '.ext'))
         tester('ntpath.splitext("/foo/foo.ext")', ('/foo/foo', '.ext'))
@@ -232,8 +246,8 @@ def test_realpath_basic(self):
         self.addCleanup(support.unlink, ABSTFN + "1")
 
         os.symlink(ABSTFN, ABSTFN + "1")
-        self.assertEqual(ntpath.realpath(ABSTFN + "1"), ABSTFN)
-        self.assertEqual(ntpath.realpath(os.fsencode(ABSTFN + "1")),
+        self.assertPathEqual(ntpath.realpath(ABSTFN + "1"), ABSTFN)
+        self.assertPathEqual(ntpath.realpath(os.fsencode(ABSTFN + "1")),
                          os.fsencode(ABSTFN))
 
     @support.skip_unless_symlink
@@ -245,7 +259,7 @@ def test_realpath_relative(self):
         self.addCleanup(support.unlink, ABSTFN + "1")
 
         os.symlink(ABSTFN, ntpath.relpath(ABSTFN + "1"))
-        self.assertEqual(ntpath.realpath(ABSTFN + "1"), ABSTFN)
+        self.assertPathEqual(ntpath.realpath(ABSTFN + "1"), ABSTFN)
 
     @support.skip_unless_symlink
     @unittest.skipUnless(HAVE_GETFINALPATHNAME, 'need _getfinalpathname')
@@ -268,39 +282,39 @@ def test_realpath_broken_symlinks(self):
             os.symlink(ABSTFN + r"\broken", "broken4")
             os.symlink(r"recursive\..\broken", "broken5")
 
-            self.assertEqual(ntpath.realpath("broken"),
-                             ABSTFN + r"\missing")
-            self.assertEqual(ntpath.realpath(r"broken\foo"),
-                             ABSTFN + r"\missing\foo")
-            self.assertEqual(ntpath.realpath(r"broken1"),
-                             ABSTFN + r"\missing\bar")
-            self.assertEqual(ntpath.realpath(r"broken1\baz"),
-                             ABSTFN + r"\missing\bar\baz")
-            self.assertEqual(ntpath.realpath("broken2"),
-                             ABSTFN + r"\missing")
-            self.assertEqual(ntpath.realpath("broken3"),
-                             ABSTFN + r"\missing")
-            self.assertEqual(ntpath.realpath("broken4"),
-                             ABSTFN + r"\missing")
-            self.assertEqual(ntpath.realpath("broken5"),
-                             ABSTFN + r"\missing")
-
-            self.assertEqual(ntpath.realpath(b"broken"),
-                             os.fsencode(ABSTFN + r"\missing"))
-            self.assertEqual(ntpath.realpath(rb"broken\foo"),
-                             os.fsencode(ABSTFN + r"\missing\foo"))
-            self.assertEqual(ntpath.realpath(rb"broken1"),
-                             os.fsencode(ABSTFN + r"\missing\bar"))
-            self.assertEqual(ntpath.realpath(rb"broken1\baz"),
-                             os.fsencode(ABSTFN + r"\missing\bar\baz"))
-            self.assertEqual(ntpath.realpath(b"broken2"),
-                             os.fsencode(ABSTFN + r"\missing"))
-            self.assertEqual(ntpath.realpath(rb"broken3"),
-                             os.fsencode(ABSTFN + r"\missing"))
-            self.assertEqual(ntpath.realpath(b"broken4"),
-                             os.fsencode(ABSTFN + r"\missing"))
-            self.assertEqual(ntpath.realpath(b"broken5"),
-                             os.fsencode(ABSTFN + r"\missing"))
+            self.assertPathEqual(ntpath.realpath("broken"),
+                                 ABSTFN + r"\missing")
+            self.assertPathEqual(ntpath.realpath(r"broken\foo"),
+                                 ABSTFN + r"\missing\foo")
+            self.assertPathEqual(ntpath.realpath(r"broken1"),
+                                 ABSTFN + r"\missing\bar")
+            self.assertPathEqual(ntpath.realpath(r"broken1\baz"),
+                                 ABSTFN + r"\missing\bar\baz")
+            self.assertPathEqual(ntpath.realpath("broken2"),
+                                 ABSTFN + r"\missing")
+            self.assertPathEqual(ntpath.realpath("broken3"),
+                                 ABSTFN + r"\missing")
+            self.assertPathEqual(ntpath.realpath("broken4"),
+                                 ABSTFN + r"\missing")
+            self.assertPathEqual(ntpath.realpath("broken5"),
+                                 ABSTFN + r"\missing")
+
+            self.assertPathEqual(ntpath.realpath(b"broken"),
+                                 os.fsencode(ABSTFN + r"\missing"))
+            self.assertPathEqual(ntpath.realpath(rb"broken\foo"),
+                                 os.fsencode(ABSTFN + r"\missing\foo"))
+            self.assertPathEqual(ntpath.realpath(rb"broken1"),
+                                 os.fsencode(ABSTFN + r"\missing\bar"))
+            self.assertPathEqual(ntpath.realpath(rb"broken1\baz"),
+                                 os.fsencode(ABSTFN + r"\missing\bar\baz"))
+            self.assertPathEqual(ntpath.realpath(b"broken2"),
+                                 os.fsencode(ABSTFN + r"\missing"))
+            self.assertPathEqual(ntpath.realpath(rb"broken3"),
+                                 os.fsencode(ABSTFN + r"\missing"))
+            self.assertPathEqual(ntpath.realpath(b"broken4"),
+                                 os.fsencode(ABSTFN + r"\missing"))
+            self.assertPathEqual(ntpath.realpath(b"broken5"),
+                                 os.fsencode(ABSTFN + r"\missing"))
 
     @support.skip_unless_symlink
     @unittest.skipUnless(HAVE_GETFINALPATHNAME, 'need _getfinalpathname')
@@ -318,39 +332,39 @@ def test_realpath_symlink_loops(self):
         P = "\\\\?\\"
 
         os.symlink(ABSTFN, ABSTFN)
-        self.assertEqual(ntpath.realpath(ABSTFN), P + ABSTFN)
+        self.assertPathEqual(ntpath.realpath(ABSTFN), P + ABSTFN)
 
         # cycles are non-deterministic as to which path is returned, but
         # it will always be the fully resolved path of one member of the cycle
         os.symlink(ABSTFN + "1", ABSTFN + "2")
         os.symlink(ABSTFN + "2", ABSTFN + "1")
         expected = (P + ABSTFN + "1", P + ABSTFN + "2")
-        self.assertIn(ntpath.realpath(ABSTFN + "1"), expected)
-        self.assertIn(ntpath.realpath(ABSTFN + "2"), expected)
-
-        self.assertIn(ntpath.realpath(ABSTFN + "1\\x"),
-                      (ntpath.join(r, "x") for r in expected))
-        self.assertEqual(ntpath.realpath(ABSTFN + "1\\.."),
-                         ntpath.dirname(ABSTFN))
-        self.assertEqual(ntpath.realpath(ABSTFN + "1\\..\\x"),
-                         ntpath.dirname(ABSTFN) + "\\x")
+        self.assertPathIn(ntpath.realpath(ABSTFN + "1"), expected)
+        self.assertPathIn(ntpath.realpath(ABSTFN + "2"), expected)
+
+        self.assertPathIn(ntpath.realpath(ABSTFN + "1\\x"),
+                          (ntpath.join(r, "x") for r in expected))
+        self.assertPathEqual(ntpath.realpath(ABSTFN + "1\\.."),
+                             ntpath.dirname(ABSTFN))
+        self.assertPathEqual(ntpath.realpath(ABSTFN + "1\\..\\x"),
+                             ntpath.dirname(ABSTFN) + "\\x")
         os.symlink(ABSTFN + "x", ABSTFN + "y")
-        self.assertEqual(ntpath.realpath(ABSTFN + "1\\..\\"
-                                         + ntpath.basename(ABSTFN) + "y"),
-                         ABSTFN + "x")
-        self.assertIn(ntpath.realpath(ABSTFN + "1\\..\\"
-                                      + ntpath.basename(ABSTFN) + "1"),
-                      expected)
+        self.assertPathEqual(ntpath.realpath(ABSTFN + "1\\..\\"
+                                             + ntpath.basename(ABSTFN) + "y"),
+                             ABSTFN + "x")
+        self.assertPathIn(ntpath.realpath(ABSTFN + "1\\..\\"
+                                          + ntpath.basename(ABSTFN) + "1"),
+                          expected)
 
         os.symlink(ntpath.basename(ABSTFN) + "a\\b", ABSTFN + "a")
-        self.assertEqual(ntpath.realpath(ABSTFN + "a"), P + ABSTFN + "a")
+        self.assertPathEqual(ntpath.realpath(ABSTFN + "a"), P + ABSTFN + "a")
 
         os.symlink("..\\" + ntpath.basename(ntpath.dirname(ABSTFN))
                    + "\\" + ntpath.basename(ABSTFN) + "c", ABSTFN + "c")
-        self.assertEqual(ntpath.realpath(ABSTFN + "c"), P + ABSTFN + "c")
+        self.assertPathEqual(ntpath.realpath(ABSTFN + "c"), P + ABSTFN + "c")
 
         # Test using relative path as well.
-        self.assertEqual(ntpath.realpath(ntpath.basename(ABSTFN)), P + ABSTFN)
+        self.assertPathEqual(ntpath.realpath(ntpath.basename(ABSTFN)), P + ABSTFN)
 
     @support.skip_unless_symlink
     @unittest.skipUnless(HAVE_GETFINALPATHNAME, 'need _getfinalpathname')
@@ -369,10 +383,10 @@ def test_realpath_symlink_prefix(self):
             f.write(b'1')
         os.symlink("\\\\?\\" + ABSTFN + "3.", ABSTFN + "3.link")
 
-        self.assertEqual(ntpath.realpath(ABSTFN + "3link"),
-                         ABSTFN + "3")
-        self.assertEqual(ntpath.realpath(ABSTFN + "3.link"),
-                         "\\\\?\\" + ABSTFN + "3.")
+        self.assertPathEqual(ntpath.realpath(ABSTFN + "3link"),
+                             ABSTFN + "3")
+        self.assertPathEqual(ntpath.realpath(ABSTFN + "3.link"),
+                             "\\\\?\\" + ABSTFN + "3.")
 
         # Resolved paths should be usable to open target files
         with open(ntpath.realpath(ABSTFN + "3link"), "rb") as f:
@@ -381,10 +395,10 @@ def test_realpath_symlink_prefix(self):
             self.assertEqual(f.read(), b'1')
 
         # When the prefix is included, it is not stripped
-        self.assertEqual(ntpath.realpath("\\\\?\\" + ABSTFN + "3link"),
-                         "\\\\?\\" + ABSTFN + "3")
-        self.assertEqual(ntpath.realpath("\\\\?\\" + ABSTFN + "3.link"),
-                         "\\\\?\\" + ABSTFN + "3.")
+        self.assertPathEqual(ntpath.realpath("\\\\?\\" + ABSTFN + "3link"),
+                             "\\\\?\\" + ABSTFN + "3")
+        self.assertPathEqual(ntpath.realpath("\\\\?\\" + ABSTFN + "3.link"),
+                             "\\\\?\\" + ABSTFN + "3.")
 
     def test_expandvars(self):
         with support.EnvironmentVarGuard() as env:
@@ -658,7 +672,7 @@ class NtCommonTest(test_genericpath.CommonTest, unittest.TestCase):
     attributes = ['relpath']
 
 
-class PathLikeTests(unittest.TestCase):
+class PathLikeTests(NtpathTestCase):
 
     path = ntpath
 
@@ -669,67 +683,67 @@ def setUp(self):
         with open(self.file_name, 'xb', 0) as file:
             file.write(b"test_ntpath.PathLikeTests")
 
-    def assertPathEqual(self, func):
-        self.assertEqual(func(self.file_path), func(self.file_name))
+    def _check_function(self, func):
+        self.assertPathEqual(func(self.file_path), func(self.file_name))
 
     def test_path_normcase(self):
-        self.assertPathEqual(self.path.normcase)
+        self._check_function(self.path.normcase)
 
     def test_path_isabs(self):
-        self.assertPathEqual(self.path.isabs)
+        self._check_function(self.path.isabs)
 
     def test_path_join(self):
         self.assertEqual(self.path.join('a', FakePath('b'), 'c'),
                          self.path.join('a', 'b', 'c'))
 
     def test_path_split(self):
-        self.assertPathEqual(self.path.split)
+        self._check_function(self.path.split)
 
     def test_path_splitext(self):
-        self.assertPathEqual(self.path.splitext)
+        self._check_function(self.path.splitext)
 
     def test_path_splitdrive(self):
-        self.assertPathEqual(self.path.splitdrive)
+        self._check_function(self.path.splitdrive)
 
     def test_path_basename(self):
-        self.assertPathEqual(self.path.basename)
+        self._check_function(self.path.basename)
 
     def test_path_dirname(self):
-        self.assertPathEqual(self.path.dirname)
+        self._check_function(self.path.dirname)
 
     def test_path_islink(self):
-        self.assertPathEqual(self.path.islink)
+        self._check_function(self.path.islink)
 
     def test_path_lexists(self):
-        self.assertPathEqual(self.path.lexists)
+        self._check_function(self.path.lexists)
 
     def test_path_ismount(self):
-        self.assertPathEqual(self.path.ismount)
+        self._check_function(self.path.ismount)
 
     def test_path_expanduser(self):
-        self.assertPathEqual(self.path.expanduser)
+        self._check_function(self.path.expanduser)
 
     def test_path_expandvars(self):
-        self.assertPathEqual(self.path.expandvars)
+        self._check_function(self.path.expandvars)
 
     def test_path_normpath(self):
-        self.assertPathEqual(self.path.normpath)
+        self._check_function(self.path.normpath)
 
     def test_path_abspath(self):
-        self.assertPathEqual(self.path.abspath)
+        self._check_function(self.path.abspath)
 
     def test_path_realpath(self):
-        self.assertPathEqual(self.path.realpath)
+        self._check_function(self.path.realpath)
 
     def test_path_relpath(self):
-        self.assertPathEqual(self.path.relpath)
+        self._check_function(self.path.relpath)
 
     def test_path_commonpath(self):
         common_path = self.path.commonpath([self.file_path, self.file_name])
-        self.assertEqual(common_path, self.file_name)
+        self.assertPathEqual(common_path, self.file_name)
 
     def test_path_isdir(self):
-        self.assertPathEqual(self.path.isdir)
+        self._check_function(self.path.isdir)
 
 
 if __name__ == "__main__":
diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py
index f74524d992a129..027331f95911e8 100644
--- a/Lib/test/test_pathlib.py
+++ b/Lib/test/test_pathlib.py
@@ -1274,10 +1274,13 @@ def assertFileNotFound(self, func, *args, **kwargs):
             func(*args, **kwargs)
         self.assertEqual(cm.exception.errno, errno.ENOENT)
 
+    def assertEqualNormCase(self, path_a, path_b):
+        self.assertEqual(os.path.normcase(path_a), os.path.normcase(path_b))
+
     def _test_cwd(self, p):
         q = self.cls(os.getcwd())
         self.assertEqual(p, q)
-        self.assertEqual(str(p), str(q))
+        self.assertEqualNormCase(str(p), str(q))
         self.assertIs(type(p), type(q))
         self.assertTrue(p.is_absolute())
 
@@ -1288,7 +1291,7 @@ def test_cwd(self):
     def _test_home(self, p):
         q = self.cls(os.path.expanduser('~'))
         self.assertEqual(p, q)
-        self.assertEqual(str(p), str(q))
+        self.assertEqualNormCase(str(p), str(q))
         self.assertIs(type(p), type(q))
         self.assertTrue(p.is_absolute())
 
@@ -1496,14 +1499,14 @@ def test_resolve_common(self):
             p.resolve(strict=True)
         self.assertEqual(cm.exception.errno, errno.ENOENT)
         # Non-strict
-        self.assertEqual(str(p.resolve(strict=False)),
-                         os.path.join(BASE, 'foo'))
+        self.assertEqualNormCase(str(p.resolve(strict=False)),
+                                 os.path.join(BASE, 'foo'))
         p = P(BASE, 'foo', 'in', 'spam')
-        self.assertEqual(str(p.resolve(strict=False)),
-                         os.path.join(BASE, 'foo', 'in', 'spam'))
+        self.assertEqualNormCase(str(p.resolve(strict=False)),
+                                 os.path.join(BASE, 'foo', 'in', 'spam'))
         p = P(BASE, '..', 'foo', 'in', 'spam')
-        self.assertEqual(str(p.resolve(strict=False)),
-                         os.path.abspath(os.path.join('foo', 'in', 'spam')))
+        self.assertEqualNormCase(str(p.resolve(strict=False)),
+                                 os.path.abspath(os.path.join('foo', 'in', 'spam')))
         # These are all relative symlinks.
         p = P(BASE, 'dirB', 'fileB')
         self._check_resolve_relative(p, p)
@@ -2050,16 +2053,16 @@ def _check_complex_symlinks(self, link0_target):
         # Resolve absolute paths.
         p = (P / 'link0').resolve()
         self.assertEqual(p, P)
-        self.assertEqual(str(p), BASE)
+        self.assertEqualNormCase(str(p), BASE)
         p = (P / 'link1').resolve()
         self.assertEqual(p, P)
-        self.assertEqual(str(p), BASE)
+        self.assertEqualNormCase(str(p), BASE)
         p = (P / 'link2').resolve()
         self.assertEqual(p, P)
-        self.assertEqual(str(p), BASE)
+        self.assertEqualNormCase(str(p), BASE)
         p = (P / 'link3').resolve()
         self.assertEqual(p, P)
-        self.assertEqual(str(p), BASE)
+        self.assertEqualNormCase(str(p), BASE)
 
         # Resolve relative paths.
         old_path = os.getcwd()
@@ -2067,16 +2070,16 @@ def _check_complex_symlinks(self, link0_target):
         try:
             p = self.cls('link0').resolve()
             self.assertEqual(p, P)
-            self.assertEqual(str(p), BASE)
+            self.assertEqualNormCase(str(p), BASE)
             p = self.cls('link1').resolve()
             self.assertEqual(p, P)
-            self.assertEqual(str(p), BASE)
+            self.assertEqualNormCase(str(p), BASE)
             p = self.cls('link2').resolve()
             self.assertEqual(p, P)
-            self.assertEqual(str(p), BASE)
+            self.assertEqualNormCase(str(p), BASE)
             p = self.cls('link3').resolve()
             self.assertEqual(p, P)
-            self.assertEqual(str(p), BASE)
+            self.assertEqualNormCase(str(p), BASE)
         finally:
             os.chdir(old_path)
 
diff --git a/Misc/NEWS.d/next/Windows/2019-09-10-14-21-40.bpo-38087.--eIib.rst b/Misc/NEWS.d/next/Windows/2019-09-10-14-21-40.bpo-38087.--eIib.rst
new file mode 100644
index 00000000000000..ca625cc775a209
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-09-10-14-21-40.bpo-38087.--eIib.rst
@@ -0,0 +1 @@
+Fix case sensitivity in test_pathlib and test_ntpath.

From 45bc3928e232603a97451dea3106d824b0f7a392 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 07:31:07 -0700
Subject: [PATCH 0549/2163] bpo-38090: Fix reference leak in ceval.c (GH-15848)

(cherry picked from commit a511c7a4961a684db1f8d0ed438822d87d7d3dcf)

Co-authored-by: Pablo Galindo 
---
 Python/ceval.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/Python/ceval.c b/Python/ceval.c
index 07ec3293adf1c4..3306fb9728e8cd 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -5238,7 +5238,6 @@ import_from(PyThreadState *tstate, PyObject *v, PyObject *name)
     else {
         _Py_IDENTIFIER(__spec__);
         PyObject *spec = _PyObject_GetAttrId(v, &PyId___spec__);
-        Py_XINCREF(spec);
         const char *fmt =
             _PyModuleSpec_IsInitializing(spec) ?
             "cannot import name %R from partially initialized module %R "

From ed99bb9ca68b37cfaec3629afa67d70289f3ffc7 Mon Sep 17 00:00:00 2001
From: Steve Dower 
Date: Tue, 10 Sep 2019 15:31:26 +0100
Subject: [PATCH 0550/2163] bpo-37913: document that __length_hint__ can return
 NotImplemented (GH-15383)

---
 Doc/reference/datamodel.rst | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
index 8813f57587f01a..c6c6e4075039cd 100644
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -2155,7 +2155,9 @@ through the container; for mappings, :meth:`__iter__` should be the same as
 
    Called to implement :func:`operator.length_hint`. Should return an estimated
    length for the object (which may be greater or less than the actual length).
-   The length must be an integer ``>=`` 0. This method is purely an
+   The length must be an integer ``>=`` 0. The return value may also be
+   :const:`NotImplemented`, which is treated the same as if the
+   ``__length_hint__`` method didn't exist at all. This method is purely an
    optimization and is never required for correctness.
 
    .. versionadded:: 3.4

From 21dacea6ad1c49e1cafbe8bac803ddc22a534274 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 07:37:56 -0700
Subject: [PATCH 0551/2163] Fix calling order of PyEval_InitThreads. (GH-15836)

As described in Doc/c-api/init.rst, PyEval_InitThreads() cannot be called
before Py_Initialize() function.
(cherry picked from commit 9e61066355b4b55c873d56f5f106a22463b56862)

Co-authored-by: Kenta Murata 
---
 Modules/_ctypes/callbacks.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Modules/_ctypes/callbacks.c b/Modules/_ctypes/callbacks.c
index 97463b599bc004..d2d9a6587d7995 100644
--- a/Modules/_ctypes/callbacks.c
+++ b/Modules/_ctypes/callbacks.c
@@ -422,8 +422,8 @@ CThunkObject *_ctypes_alloc_callback(PyObject *callable,
 static void LoadPython(void)
 {
     if (!Py_IsInitialized()) {
-        PyEval_InitThreads();
         Py_Initialize();
+        PyEval_InitThreads();
     }
 }
 

From 2ed0ac6bf17d599bc246c513c32710c154e50a64 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 07:42:26 -0700
Subject: [PATCH 0552/2163] bpo-38088: Fixes distutils not finding
 vcruntime140.dll with only v142 toolset installed (GH-15849)

(cherry picked from commit cd8221152dd235ec5d06e3d9d0d8787645bbac8e)

Co-authored-by: Steve Dower 
---
 Lib/distutils/_msvccompiler.py                                  | 2 +-
 .../next/Windows/2019-09-10-14-17-25.bpo-38088.FOvWSM.rst       | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)
 create mode 100644 Misc/NEWS.d/next/Windows/2019-09-10-14-17-25.bpo-38088.FOvWSM.rst

diff --git a/Lib/distutils/_msvccompiler.py b/Lib/distutils/_msvccompiler.py
index 6e14f330d7f4d2..e8e4b717b9736f 100644
--- a/Lib/distutils/_msvccompiler.py
+++ b/Lib/distutils/_msvccompiler.py
@@ -107,7 +107,7 @@ def _find_vcvarsall(plat_spec):
 
     if best_dir:
         vcredist = os.path.join(best_dir, "..", "..", "redist", "MSVC", "**",
-            vcruntime_plat, "Microsoft.VC141.CRT", "vcruntime140.dll")
+            vcruntime_plat, "Microsoft.VC14*.CRT", "vcruntime140.dll")
         try:
             import glob
             vcruntime = glob.glob(vcredist, recursive=True)[-1]
diff --git a/Misc/NEWS.d/next/Windows/2019-09-10-14-17-25.bpo-38088.FOvWSM.rst b/Misc/NEWS.d/next/Windows/2019-09-10-14-17-25.bpo-38088.FOvWSM.rst
new file mode 100644
index 00000000000000..37bdeeadd894a2
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-09-10-14-17-25.bpo-38088.FOvWSM.rst
@@ -0,0 +1,2 @@
+Fixes distutils not finding vcruntime140.dll with only the v142 toolset
+installed.

From 41c965f93007b53710b9f63918f6710dde321226 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 07:51:29 -0700
Subject: [PATCH 0553/2163] Fix subprocess docstring typo (GH-15812)

(cherry picked from commit 182e1d1f849757439f2031504f142fa4e1251611)

Co-authored-by: Matthias 
---
 Lib/subprocess.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Lib/subprocess.py b/Lib/subprocess.py
index 5bbeba47a37432..aed7292541e034 100644
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -395,7 +395,7 @@ def check_output(*popenargs, timeout=None, **kwargs):
     b'when in the course of barman events\n'
 
     By default, all communication is in bytes, and therefore any "input"
-    should be bytes, and the return value wil be bytes.  If in text mode,
+    should be bytes, and the return value will be bytes.  If in text mode,
     any "input" should be a string, and the return value will be a string
     decoded according to locale encoding, or by "encoding" if set. Text mode
     is triggered by setting any of text, encoding, errors or universal_newlines.

From 313f80192a7396ea3e5ab107175afe4c5a017ab8 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 07:51:43 -0700
Subject: [PATCH 0554/2163] bpo-37504: Fix documentation build with texinfo
 builder (GH-14606)

In the table model used by docutils, the `cols` attribute of `tgroup`
nodes is mandatory, see [1]. It is used in texinfo builder in [2].

[1]: https://www.oasis-open.org/specs/tm9901.htmGH-AEN348
[2]: https://github.com/sphinx-doc/sphinx/blob/v2.1.2/sphinx/writers/texinfo.pyGH-L1129

* Doc: Add texinfo support to the Makefile
(cherry picked from commit c3d679fd398f42a2e489fbe3dab17fac1fb2439c)

Co-authored-by: Dmitry Shachnev 
---
 Doc/Makefile                       | 19 ++++++++++++++++++-
 Doc/tools/extensions/pyspecific.py |  1 +
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/Doc/Makefile b/Doc/Makefile
index 05eeab9124df63..f589b6e75f6180 100644
--- a/Doc/Makefile
+++ b/Doc/Makefile
@@ -20,7 +20,7 @@ PAPEROPT_letter = -D latex_elements.papersize=letterpaper
 ALLSPHINXOPTS = -b $(BUILDER) -d build/doctrees $(PAPEROPT_$(PAPER)) \
                 $(SPHINXOPTS) $(SPHINXERRORHANDLING) . build/$(BUILDER) $(SOURCES)
 
-.PHONY: help build html htmlhelp latex text changes linkcheck \
+.PHONY: help build html htmlhelp latex text texinfo changes linkcheck \
 	suspicious coverage doctest pydoc-topics htmlview clean dist check serve \
 	autobuild-dev autobuild-stable venv
 
@@ -33,6 +33,7 @@ help:
 	@echo "  htmlhelp   to make HTML files and a HTML help project"
 	@echo "  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
 	@echo "  text       to make plain text files"
+	@echo "  texinfo    to make Texinfo file"
 	@echo "  epub       to make EPUB files"
 	@echo "  changes    to make an overview over all changed/added/deprecated items"
 	@echo "  linkcheck  to check all external links for integrity"
@@ -89,6 +90,11 @@ text: BUILDER = text
 text: build
 	@echo "Build finished; the text files are in build/text."
 
+texinfo: BUILDER = texinfo
+texinfo: build
+	@echo "Build finished; the python.texi file is in build/texinfo."
+	@echo "Run \`make info' in that directory to run it through makeinfo."
+
 epub: BUILDER = epub
 epub: build
 	@echo "Build finished; the epub files are in build/epub."
@@ -183,6 +189,17 @@ dist:
 	make epub
 	cp -pPR build/epub/Python.epub dist/python-$(DISTVERSION)-docs.epub
 
+	# archive the texinfo build
+	rm -rf build/texinfo
+	make texinfo
+	make info --directory=build/texinfo
+	cp -pPR build/texinfo dist/python-$(DISTVERSION)-docs-texinfo
+	tar -C dist -cf dist/python-$(DISTVERSION)-docs-texinfo.tar python-$(DISTVERSION)-docs-texinfo
+	bzip2 -9 -k dist/python-$(DISTVERSION)-docs-texinfo.tar
+	(cd dist; zip -q -r -9 python-$(DISTVERSION)-docs-texinfo.zip python-$(DISTVERSION)-docs-texinfo)
+	rm -r dist/python-$(DISTVERSION)-docs-texinfo
+	rm dist/python-$(DISTVERSION)-docs-texinfo.tar
+
 check:
 	$(PYTHON) tools/rstlint.py -i tools -i $(VENVDIR) -i README.rst
 
diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py
index f41077b0761521..82dfc6bb263b01 100644
--- a/Doc/tools/extensions/pyspecific.py
+++ b/Doc/tools/extensions/pyspecific.py
@@ -540,6 +540,7 @@ def process_audit_events(app, doctree, fromdocname):
         nodes.colspec(colwidth=30),
         nodes.colspec(colwidth=55),
         nodes.colspec(colwidth=15),
+        cols=3,
     )
     head = nodes.thead()
     body = nodes.tbody()

From 97c2f68a4c0bf4e69a1e769928d7928fab73a666 Mon Sep 17 00:00:00 2001
From: "Jason R. Coombs" 
Date: Tue, 10 Sep 2019 15:58:29 +0100
Subject: [PATCH 0555/2163] [3.8] bpo-38086: Sync importlib.metadata with
 importlib_metadata 0.21. (GH-15840) (#15861)

https://gitlab.com/python-devs/importlib_metadata/-/tags/0.21.
(cherry picked from commit 17499d82702432955d8e442a1871ff276ca64bc5)

Co-authored-by: Jason R. Coombs 
---
 Doc/library/importlib.metadata.rst            |   30 +-
 Doc/tools/susp-ignored.csv                    |    2 +-
 Lib/importlib/_bootstrap_external.py          |   18 +-
 Lib/importlib/metadata.py                     |   63 +-
 Lib/test/test_importlib/test_main.py          |    4 +
 Lib/test/test_importlib/test_metadata_api.py  |   30 +-
 .../2019-09-10-11-42-59.bpo-38086.w5TlG-.rst  |    1 +
 Python/importlib_external.h                   | 1219 ++++++++---------
 8 files changed, 709 insertions(+), 658 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2019-09-10-11-42-59.bpo-38086.w5TlG-.rst

diff --git a/Doc/library/importlib.metadata.rst b/Doc/library/importlib.metadata.rst
index 23c8cdeb21c25d..4a4a8f7dfa0f87 100644
--- a/Doc/library/importlib.metadata.rst
+++ b/Doc/library/importlib.metadata.rst
@@ -172,10 +172,10 @@ Distribution requirements
 -------------------------
 
 To get the full set of requirements for a distribution, use the ``requires()``
-function.  Note that this returns an iterator::
+function::
 
-    >>> list(requires('wheel'))  # doctest: +SKIP
-    ["pytest (>=3.0.0) ; extra == 'test'"]
+    >>> requires('wheel')  # doctest: +SKIP
+    ["pytest (>=3.0.0) ; extra == 'test'", "pytest-cov ; extra == 'test'"]
 
 
 Distributions
@@ -224,23 +224,25 @@ The abstract class :py:class:`importlib.abc.MetaPathFinder` defines the
 interface expected of finders by Python's import system.
 ``importlib.metadata`` extends this protocol by looking for an optional
 ``find_distributions`` callable on the finders from
-``sys.meta_path``.  If the finder has this method, it must return
-an iterator over instances of the ``Distribution`` abstract class. This
-method must have the signature::
+``sys.meta_path`` and presents this extended interface as the
+``DistributionFinder`` abstract base class, which defines this abstract
+method::
 
-    def find_distributions(name=None, path=None):
+    @abc.abstractmethod
+    def find_distributions(context=DistributionFinder.Context()):
         """Return an iterable of all Distribution instances capable of
-        loading the metadata for packages matching the name
-        (or all names if not supplied) along the paths in the list
-        of directories ``path`` (defaults to sys.path).
+        loading the metadata for packages for the indicated ``context``.
         """
 
+The ``DistributionFinder.Context`` object provides ``.path`` and ``.name``
+properties indicating the path to search and names to match and may
+supply other relevant context.
+
 What this means in practice is that to support finding distribution package
 metadata in locations other than the file system, you should derive from
-``Distribution`` and implement the ``load_metadata()`` method.  This takes a
-single argument which is the name of the package whose metadata is being
-found.  This instance of the ``Distribution`` base abstract class is what your
-finder's ``find_distributions()`` method should return.
+``Distribution`` and implement the ``load_metadata()`` method. Then from
+your finder, return instances of this derived ``Distribution`` in the
+``find_distributions()`` method.
 
 
 .. _`entry point API`: https://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points
diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv
index d6252ed2ffa24a..c97c859fe8915e 100644
--- a/Doc/tools/susp-ignored.csv
+++ b/Doc/tools/susp-ignored.csv
@@ -350,4 +350,4 @@ whatsnew/changelog,,::,error::BytesWarning
 whatsnew/changelog,,::,default::BytesWarning
 whatsnew/changelog,,::,default::DeprecationWarning
 library/importlib.metadata,,:main,"EntryPoint(name='wheel', value='wheel.cli:main', group='console_scripts')"
-library/importlib.metadata,,`,of directories ``path`` (defaults to sys.path).
+library/importlib.metadata,,`,loading the metadata for packages for the indicated ``context``.
diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py
index 67bd1d3cf99e98..ba78c57eebc331 100644
--- a/Lib/importlib/_bootstrap_external.py
+++ b/Lib/importlib/_bootstrap_external.py
@@ -1369,21 +1369,19 @@ def find_module(cls, fullname, path=None):
         return spec.loader
 
     @classmethod
-    def find_distributions(cls, name=None, path=None):
+    def find_distributions(self, context=None):
         """
         Find distributions.
 
         Return an iterable of all Distribution instances capable of
-        loading the metadata for packages matching the ``name``
-        (or all names if not supplied) along the paths in the list
-        of directories ``path`` (defaults to sys.path).
+        loading the metadata for packages matching ``context.name``
+        (or all names if ``None`` indicated) along the paths in the list
+        of directories ``context.path``.
         """
-        import re
-        from importlib.metadata import PathDistribution
-        if path is None:
-            path = sys.path
-        pattern = '.*' if name is None else re.escape(name)
-        found = cls._search_paths(pattern, path)
+        from importlib.metadata import PathDistribution, DistributionFinder
+        if context is None:
+            context = DistributionFinder.Context()
+        found = self._search_paths(context.pattern, context.path)
         return map(PathDistribution, found)
 
     @classmethod
diff --git a/Lib/importlib/metadata.py b/Lib/importlib/metadata.py
index 3b46142231ec21..e23076654fe456 100644
--- a/Lib/importlib/metadata.py
+++ b/Lib/importlib/metadata.py
@@ -19,6 +19,7 @@
 
 __all__ = [
     'Distribution',
+    'DistributionFinder',
     'PackageNotFoundError',
     'distribution',
     'distributions',
@@ -158,7 +159,7 @@ def from_name(cls, name):
             metadata cannot be found.
         """
         for resolver in cls._discover_resolvers():
-            dists = resolver(name)
+            dists = resolver(DistributionFinder.Context(name=name))
             dist = next(dists, None)
             if dist is not None:
                 return dist
@@ -166,16 +167,33 @@ def from_name(cls, name):
             raise PackageNotFoundError(name)
 
     @classmethod
-    def discover(cls):
+    def discover(cls, **kwargs):
         """Return an iterable of Distribution objects for all packages.
 
+        Pass a ``context`` or pass keyword arguments for constructing
+        a context.
+
+        :context: A ``DistributionFinder.Context`` object.
         :return: Iterable of Distribution objects for all packages.
         """
+        context = kwargs.pop('context', None)
+        if context and kwargs:
+            raise ValueError("cannot accept context and kwargs")
+        context = context or DistributionFinder.Context(**kwargs)
         return itertools.chain.from_iterable(
-            resolver()
+            resolver(context)
             for resolver in cls._discover_resolvers()
             )
 
+    @staticmethod
+    def at(path):
+        """Return a Distribution for the indicated metadata path
+
+        :param path: a string or path-like object
+        :return: a concrete Distribution instance for the path
+        """
+        return PathDistribution(pathlib.Path(path))
+
     @staticmethod
     def _discover_resolvers():
         """Search the meta_path for resolvers."""
@@ -215,7 +233,7 @@ def entry_points(self):
     def files(self):
         """Files in this distribution.
 
-        :return: Iterable of PackagePath for this distribution or None
+        :return: List of PackagePath for this distribution or None
 
         Result is `None` if the metadata file that enumerates files
         (i.e. RECORD for dist-info or SOURCES.txt for egg-info) is
@@ -231,7 +249,7 @@ def make_file(name, hash=None, size_str=None):
             result.dist = self
             return result
 
-        return file_lines and starmap(make_file, csv.reader(file_lines))
+        return file_lines and list(starmap(make_file, csv.reader(file_lines)))
 
     def _read_files_distinfo(self):
         """
@@ -251,7 +269,8 @@ def _read_files_egginfo(self):
     @property
     def requires(self):
         """Generated requirements specified for this Distribution"""
-        return self._read_dist_info_reqs() or self._read_egg_info_reqs()
+        reqs = self._read_dist_info_reqs() or self._read_egg_info_reqs()
+        return reqs and list(reqs)
 
     def _read_dist_info_reqs(self):
         return self.metadata.get_all('Requires-Dist')
@@ -312,15 +331,35 @@ class DistributionFinder(MetaPathFinder):
     A MetaPathFinder capable of discovering installed distributions.
     """
 
+    class Context:
+
+        name = None
+        """
+        Specific name for which a distribution finder should match.
+        """
+
+        def __init__(self, **kwargs):
+            vars(self).update(kwargs)
+
+        @property
+        def path(self):
+            """
+            The path that a distribution finder should search.
+            """
+            return vars(self).get('path', sys.path)
+
+        @property
+        def pattern(self):
+            return '.*' if self.name is None else re.escape(self.name)
+
     @abc.abstractmethod
-    def find_distributions(self, name=None, path=None):
+    def find_distributions(self, context=Context()):
         """
         Find distributions.
 
         Return an iterable of all Distribution instances capable of
-        loading the metadata for packages matching the ``name``
-        (or all names if not supplied) along the paths in the list
-        of directories ``path`` (defaults to sys.path).
+        loading the metadata for packages matching the ``context``,
+        a DistributionFinder.Context instance.
         """
 
 
@@ -352,12 +391,12 @@ def distribution(package):
     return Distribution.from_name(package)
 
 
-def distributions():
+def distributions(**kwargs):
     """Get all ``Distribution`` instances in the current environment.
 
     :return: An iterable of ``Distribution`` instances.
     """
-    return Distribution.discover()
+    return Distribution.discover(**kwargs)
 
 
 def metadata(package):
diff --git a/Lib/test/test_importlib/test_main.py b/Lib/test/test_importlib/test_main.py
index 3d7da819b3434c..4d5b1273d9d103 100644
--- a/Lib/test/test_importlib/test_main.py
+++ b/Lib/test/test_importlib/test_main.py
@@ -162,6 +162,10 @@ def test_package_discovery(self):
             for dist in dists
             )
 
+    def test_invalid_usage(self):
+        with self.assertRaises(ValueError):
+            list(distributions(context='something', name='else'))
+
 
 class DirectoryTest(fixtures.OnSysPath, fixtures.SiteDir, unittest.TestCase):
     def test_egg_info(self):
diff --git a/Lib/test/test_importlib/test_metadata_api.py b/Lib/test/test_importlib/test_metadata_api.py
index af3bab3558ef66..1d7b29ae05fd10 100644
--- a/Lib/test/test_importlib/test_metadata_api.py
+++ b/Lib/test/test_importlib/test_metadata_api.py
@@ -1,7 +1,6 @@
 import re
 import textwrap
 import unittest
-import itertools
 
 from collections.abc import Iterator
 
@@ -61,9 +60,7 @@ def test_metadata_for_this_package(self):
         assert 'Topic :: Software Development :: Libraries' in classifiers
 
     @staticmethod
-    def _test_files(files_iter):
-        assert isinstance(files_iter, Iterator), files_iter
-        files = list(files_iter)
+    def _test_files(files):
         root = files[0].root
         for file in files:
             assert file.root == root
@@ -99,16 +96,18 @@ def test_requires_egg_info_file(self):
         requirements = requires('egginfo-file')
         self.assertIsNone(requirements)
 
-    def test_requires(self):
+    def test_requires_egg_info(self):
         deps = requires('egginfo-pkg')
+        assert len(deps) == 2
         assert any(
             dep == 'wheel >= 1.0; python_version >= "2.7"'
             for dep in deps
             )
 
     def test_requires_dist_info(self):
-        deps = list(requires('distinfo-pkg'))
-        assert deps and all(deps)
+        deps = requires('distinfo-pkg')
+        assert len(deps) == 2
+        assert all(deps)
         assert 'wheel >= 1.0' in deps
         assert "pytest; extra == 'test'" in deps
 
@@ -143,11 +142,20 @@ def test_more_complex_deps_requires_text(self):
 
 class OffSysPathTests(fixtures.DistInfoPkgOffPath, unittest.TestCase):
     def test_find_distributions_specified_path(self):
-        dists = itertools.chain.from_iterable(
-            resolver(path=[str(self.site_dir)])
-            for resolver in Distribution._discover_resolvers()
-            )
+        dists = Distribution.discover(path=[str(self.site_dir)])
         assert any(
             dist.metadata['Name'] == 'distinfo-pkg'
             for dist in dists
             )
+
+    def test_distribution_at_pathlib(self):
+        """Demonstrate how to load metadata direct from a directory.
+        """
+        dist_info_path = self.site_dir / 'distinfo_pkg-1.0.0.dist-info'
+        dist = Distribution.at(dist_info_path)
+        assert dist.version == '1.0.0'
+
+    def test_distribution_at_str(self):
+        dist_info_path = self.site_dir / 'distinfo_pkg-1.0.0.dist-info'
+        dist = Distribution.at(str(dist_info_path))
+        assert dist.version == '1.0.0'
diff --git a/Misc/NEWS.d/next/Library/2019-09-10-11-42-59.bpo-38086.w5TlG-.rst b/Misc/NEWS.d/next/Library/2019-09-10-11-42-59.bpo-38086.w5TlG-.rst
new file mode 100644
index 00000000000000..40db53c97da4dc
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-10-11-42-59.bpo-38086.w5TlG-.rst
@@ -0,0 +1 @@
+Update importlib.metadata with changes from `importlib_metadata 0.21 `_.
\ No newline at end of file
diff --git a/Python/importlib_external.h b/Python/importlib_external.h
index 9bcf05057d1fd1..1e3415e2efbdfb 100644
--- a/Python/importlib_external.h
+++ b/Python/importlib_external.h
@@ -2237,618 +2237,617 @@ const unsigned char _Py_M__importlib_bootstrap_external[] = {
     0,0,0,114,206,0,0,0,78,5,0,0,115,8,0,0,
     0,0,8,12,1,8,1,4,1,122,22,80,97,116,104,70,
     105,110,100,101,114,46,102,105,110,100,95,109,111,100,117,108,
-    101,99,3,0,0,0,0,0,0,0,0,0,0,0,7,0,
-    0,0,4,0,0,0,67,0,0,0,115,78,0,0,0,100,
-    1,100,2,108,0,125,3,100,1,100,3,108,1,109,2,125,
-    4,1,0,124,2,100,2,107,8,114,34,116,3,106,4,125,
-    2,124,1,100,2,107,8,114,46,100,4,110,8,124,3,160,
-    5,124,1,161,1,125,5,124,0,160,6,124,5,124,2,161,
-    2,125,6,116,7,124,4,124,6,131,2,83,0,41,5,97,
-    37,1,0,0,10,32,32,32,32,32,32,32,32,70,105,110,
-    100,32,100,105,115,116,114,105,98,117,116,105,111,110,115,46,
-    10,10,32,32,32,32,32,32,32,32,82,101,116,117,114,110,
-    32,97,110,32,105,116,101,114,97,98,108,101,32,111,102,32,
-    97,108,108,32,68,105,115,116,114,105,98,117,116,105,111,110,
-    32,105,110,115,116,97,110,99,101,115,32,99,97,112,97,98,
-    108,101,32,111,102,10,32,32,32,32,32,32,32,32,108,111,
-    97,100,105,110,103,32,116,104,101,32,109,101,116,97,100,97,
-    116,97,32,102,111,114,32,112,97,99,107,97,103,101,115,32,
-    109,97,116,99,104,105,110,103,32,116,104,101,32,96,96,110,
+    101,99,2,0,0,0,0,0,0,0,0,0,0,0,5,0,
+    0,0,4,0,0,0,67,0,0,0,115,58,0,0,0,100,
+    1,100,2,108,0,109,1,125,2,109,2,125,3,1,0,124,
+    1,100,3,107,8,114,32,124,3,160,3,161,0,125,1,124,
+    0,160,4,124,1,106,5,124,1,106,6,161,2,125,4,116,
+    7,124,2,124,4,131,2,83,0,41,4,97,32,1,0,0,
+    10,32,32,32,32,32,32,32,32,70,105,110,100,32,100,105,
+    115,116,114,105,98,117,116,105,111,110,115,46,10,10,32,32,
+    32,32,32,32,32,32,82,101,116,117,114,110,32,97,110,32,
+    105,116,101,114,97,98,108,101,32,111,102,32,97,108,108,32,
+    68,105,115,116,114,105,98,117,116,105,111,110,32,105,110,115,
+    116,97,110,99,101,115,32,99,97,112,97,98,108,101,32,111,
+    102,10,32,32,32,32,32,32,32,32,108,111,97,100,105,110,
+    103,32,116,104,101,32,109,101,116,97,100,97,116,97,32,102,
+    111,114,32,112,97,99,107,97,103,101,115,32,109,97,116,99,
+    104,105,110,103,32,96,96,99,111,110,116,101,120,116,46,110,
     97,109,101,96,96,10,32,32,32,32,32,32,32,32,40,111,
-    114,32,97,108,108,32,110,97,109,101,115,32,105,102,32,110,
-    111,116,32,115,117,112,112,108,105,101,100,41,32,97,108,111,
-    110,103,32,116,104,101,32,112,97,116,104,115,32,105,110,32,
-    116,104,101,32,108,105,115,116,10,32,32,32,32,32,32,32,
-    32,111,102,32,100,105,114,101,99,116,111,114,105,101,115,32,
-    96,96,112,97,116,104,96,96,32,40,100,101,102,97,117,108,
-    116,115,32,116,111,32,115,121,115,46,112,97,116,104,41,46,
-    10,32,32,32,32,32,32,32,32,114,73,0,0,0,78,41,
-    1,218,16,80,97,116,104,68,105,115,116,114,105,98,117,116,
-    105,111,110,122,2,46,42,41,8,218,2,114,101,90,18,105,
-    109,112,111,114,116,108,105,98,46,109,101,116,97,100,97,116,
-    97,114,59,1,0,0,114,8,0,0,0,114,44,0,0,0,
-    90,6,101,115,99,97,112,101,218,13,95,115,101,97,114,99,
-    104,95,112,97,116,104,115,218,3,109,97,112,41,7,114,193,
-    0,0,0,114,117,0,0,0,114,44,0,0,0,114,60,1,
-    0,0,114,59,1,0,0,218,7,112,97,116,116,101,114,110,
-    90,5,102,111,117,110,100,114,3,0,0,0,114,3,0,0,
-    0,114,6,0,0,0,218,18,102,105,110,100,95,100,105,115,
-    116,114,105,98,117,116,105,111,110,115,91,5,0,0,115,14,
-    0,0,0,0,10,8,1,12,1,8,1,6,1,22,1,12,
-    1,122,29,80,97,116,104,70,105,110,100,101,114,46,102,105,
-    110,100,95,100,105,115,116,114,105,98,117,116,105,111,110,115,
-    99,3,0,0,0,0,0,0,0,0,0,0,0,4,0,0,
-    0,6,0,0,0,3,0,0,0,115,44,0,0,0,100,1,
-    100,2,108,0,125,3,124,3,106,1,160,2,135,0,135,1,
-    102,2,100,3,100,4,132,8,116,3,136,0,106,4,124,2,
-    131,2,68,0,131,1,161,1,83,0,41,5,122,49,70,105,
-    110,100,32,109,101,116,97,100,97,116,97,32,100,105,114,101,
-    99,116,111,114,105,101,115,32,105,110,32,112,97,116,104,115,
-    32,104,101,117,114,105,115,116,105,99,97,108,108,121,46,114,
-    73,0,0,0,78,99,1,0,0,0,0,0,0,0,0,0,
-    0,0,2,0,0,0,5,0,0,0,51,0,0,0,115,26,
-    0,0,0,124,0,93,18,125,1,136,0,160,0,124,1,136,
-    1,161,2,86,0,1,0,113,2,100,0,83,0,114,110,0,
-    0,0,41,1,218,12,95,115,101,97,114,99,104,95,112,97,
-    116,104,41,2,114,32,0,0,0,114,44,0,0,0,169,2,
-    114,193,0,0,0,114,63,1,0,0,114,3,0,0,0,114,
-    6,0,0,0,114,19,1,0,0,113,5,0,0,115,4,0,
-    0,0,4,2,2,255,122,43,80,97,116,104,70,105,110,100,
-    101,114,46,95,115,101,97,114,99,104,95,112,97,116,104,115,
-    46,60,108,111,99,97,108,115,62,46,60,103,101,110,101,120,
-    112,114,62,41,5,218,9,105,116,101,114,116,111,111,108,115,
-    90,5,99,104,97,105,110,90,13,102,114,111,109,95,105,116,
-    101,114,97,98,108,101,114,62,1,0,0,218,12,95,115,119,
-    105,116,99,104,95,112,97,116,104,41,4,114,193,0,0,0,
-    114,63,1,0,0,90,5,112,97,116,104,115,114,67,1,0,
-    0,114,3,0,0,0,114,66,1,0,0,114,6,0,0,0,
-    114,61,1,0,0,109,5,0,0,115,8,0,0,0,0,3,
-    8,1,18,2,10,254,122,24,80,97,116,104,70,105,110,100,
-    101,114,46,95,115,101,97,114,99,104,95,112,97,116,104,115,
-    99,1,0,0,0,0,0,0,0,0,0,0,0,5,0,0,
-    0,10,0,0,0,67,0,0,0,115,96,0,0,0,100,1,
-    100,2,108,0,109,1,125,1,1,0,100,1,100,0,108,2,
-    125,2,100,1,100,0,108,3,125,3,100,3,125,4,124,4,
-    114,48,116,4,106,5,160,6,124,0,161,1,114,86,124,1,
-    116,7,131,1,143,24,1,0,124,2,160,8,124,0,161,1,
-    87,0,2,0,53,0,81,0,82,0,163,0,83,0,81,0,
-    82,0,88,0,124,3,160,8,124,0,161,1,83,0,41,4,
-    78,114,73,0,0,0,41,1,218,8,115,117,112,112,114,101,
-    115,115,70,41,9,90,10,99,111,110,116,101,120,116,108,105,
-    98,114,69,1,0,0,218,7,122,105,112,102,105,108,101,218,
-    7,112,97,116,104,108,105,98,90,2,111,115,114,44,0,0,
-    0,90,6,105,115,102,105,108,101,218,9,69,120,99,101,112,
-    116,105,111,110,90,4,80,97,116,104,41,5,114,44,0,0,
-    0,114,69,1,0,0,114,70,1,0,0,114,71,1,0,0,
-    90,13,80,89,80,89,95,79,80,69,78,95,66,85,71,114,
-    3,0,0,0,114,3,0,0,0,114,6,0,0,0,114,68,
-    1,0,0,118,5,0,0,115,16,0,0,0,0,2,12,1,
-    8,1,8,1,4,1,16,1,10,1,28,1,122,23,80,97,
-    116,104,70,105,110,100,101,114,46,95,115,119,105,116,99,104,
-    95,112,97,116,104,99,3,0,0,0,0,0,0,0,0,0,
-    0,0,6,0,0,0,5,0,0,0,67,0,0,0,115,44,
-    0,0,0,100,1,100,0,108,0,125,3,100,2,125,4,124,
-    4,106,1,124,1,100,3,141,1,125,5,124,3,106,2,124,
-    5,124,2,106,3,124,3,106,4,100,4,141,3,83,0,41,
-    5,78,114,73,0,0,0,122,32,123,112,97,116,116,101,114,
-    110,125,40,45,46,42,41,63,92,46,40,100,105,115,116,124,
-    101,103,103,41,45,105,110,102,111,169,1,114,63,1,0,0,
-    169,1,114,83,0,0,0,41,5,114,60,1,0,0,114,62,
-    0,0,0,90,5,109,97,116,99,104,114,117,0,0,0,218,
-    10,73,71,78,79,82,69,67,65,83,69,169,6,114,193,0,
-    0,0,218,10,110,111,114,109,97,108,105,122,101,100,114,41,
-    1,0,0,114,60,1,0,0,90,8,116,101,109,112,108,97,
-    116,101,90,8,109,97,110,105,102,101,115,116,114,3,0,0,
-    0,114,3,0,0,0,114,6,0,0,0,218,13,95,109,97,
-    116,99,104,101,115,95,105,110,102,111,129,5,0,0,115,8,
-    0,0,0,0,2,8,1,4,1,12,1,122,24,80,97,116,
-    104,70,105,110,100,101,114,46,95,109,97,116,99,104,101,115,
-    95,105,110,102,111,99,3,0,0,0,0,0,0,0,0,0,
-    0,0,6,0,0,0,5,0,0,0,67,0,0,0,115,46,
-    0,0,0,100,1,100,0,108,0,125,3,100,2,125,4,124,
-    4,106,1,124,1,100,3,141,1,125,5,124,3,106,2,124,
-    5,116,3,124,2,131,1,124,3,106,4,100,4,141,3,83,
-    0,41,5,78,114,73,0,0,0,122,30,123,112,97,116,116,
-    101,114,110,125,45,46,42,92,46,101,103,103,91,92,92,47,
-    93,69,71,71,45,73,78,70,79,114,73,1,0,0,114,74,
-    1,0,0,41,5,114,60,1,0,0,114,62,0,0,0,90,
-    6,115,101,97,114,99,104,114,85,0,0,0,114,75,1,0,
-    0,114,76,1,0,0,114,3,0,0,0,114,3,0,0,0,
-    114,6,0,0,0,218,15,95,109,97,116,99,104,101,115,95,
-    108,101,103,97,99,121,136,5,0,0,115,8,0,0,0,0,
-    2,8,1,4,1,12,1,122,26,80,97,116,104,70,105,110,
-    100,101,114,46,95,109,97,116,99,104,101,115,95,108,101,103,
-    97,99,121,99,3,0,0,0,0,0,0,0,0,0,0,0,
-    3,0,0,0,4,0,0,0,3,0,0,0,115,48,0,0,
-    0,124,1,160,0,161,0,115,12,100,1,83,0,124,2,160,
-    1,100,2,100,3,161,2,137,1,135,0,135,1,102,2,100,
-    4,100,5,132,8,124,1,160,2,161,0,68,0,131,1,83,
-    0,41,6,78,114,3,0,0,0,250,1,45,114,45,0,0,
-    0,99,1,0,0,0,0,0,0,0,0,0,0,0,2,0,
-    0,0,5,0,0,0,51,0,0,0,115,42,0,0,0,124,
-    0,93,34,125,1,136,0,160,0,136,1,124,1,161,2,115,
-    30,136,0,160,1,136,1,124,1,161,2,114,2,124,1,86,
-    0,1,0,113,2,100,0,83,0,114,110,0,0,0,41,2,
-    114,78,1,0,0,114,79,1,0,0,41,2,114,32,0,0,
-    0,114,41,1,0,0,169,2,114,193,0,0,0,114,77,1,
-    0,0,114,3,0,0,0,114,6,0,0,0,114,19,1,0,
-    0,148,5,0,0,115,8,0,0,0,4,0,2,1,12,1,
-    12,254,122,42,80,97,116,104,70,105,110,100,101,114,46,95,
-    115,101,97,114,99,104,95,112,97,116,104,46,60,108,111,99,
-    97,108,115,62,46,60,103,101,110,101,120,112,114,62,41,3,
-    90,6,105,115,95,100,105,114,114,67,0,0,0,90,7,105,
-    116,101,114,100,105,114,41,3,114,193,0,0,0,90,4,114,
-    111,111,116,114,63,1,0,0,114,3,0,0,0,114,81,1,
-    0,0,114,6,0,0,0,114,65,1,0,0,143,5,0,0,
-    115,8,0,0,0,0,2,8,1,4,1,12,1,122,23,80,
-    97,116,104,70,105,110,100,101,114,46,95,115,101,97,114,99,
-    104,95,112,97,116,104,41,1,78,41,2,78,78,41,1,78,
-    41,2,78,78,41,19,114,125,0,0,0,114,124,0,0,0,
-    114,126,0,0,0,114,127,0,0,0,114,207,0,0,0,114,
-    46,1,0,0,114,52,1,0,0,114,54,1,0,0,114,55,
-    1,0,0,114,58,1,0,0,114,203,0,0,0,114,206,0,
-    0,0,114,64,1,0,0,114,61,1,0,0,218,12,115,116,
-    97,116,105,99,109,101,116,104,111,100,114,68,1,0,0,114,
-    78,1,0,0,114,79,1,0,0,114,65,1,0,0,114,3,
-    0,0,0,114,3,0,0,0,114,3,0,0,0,114,6,0,
-    0,0,114,45,1,0,0,214,4,0,0,115,54,0,0,0,
-    8,2,4,2,2,1,10,9,2,1,10,12,2,1,10,21,
-    2,1,10,14,2,1,12,31,2,1,12,23,2,1,12,12,
-    2,1,12,17,2,1,10,8,2,1,10,10,2,1,10,6,
-    2,1,10,6,2,1,114,45,1,0,0,99,0,0,0,0,
-    0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,
-    64,0,0,0,115,90,0,0,0,101,0,90,1,100,0,90,
-    2,100,1,90,3,100,2,100,3,132,0,90,4,100,4,100,
-    5,132,0,90,5,101,6,90,7,100,6,100,7,132,0,90,
-    8,100,8,100,9,132,0,90,9,100,19,100,11,100,12,132,
-    1,90,10,100,13,100,14,132,0,90,11,101,12,100,15,100,
-    16,132,0,131,1,90,13,100,17,100,18,132,0,90,14,100,
-    10,83,0,41,20,218,10,70,105,108,101,70,105,110,100,101,
-    114,122,172,70,105,108,101,45,98,97,115,101,100,32,102,105,
-    110,100,101,114,46,10,10,32,32,32,32,73,110,116,101,114,
-    97,99,116,105,111,110,115,32,119,105,116,104,32,116,104,101,
-    32,102,105,108,101,32,115,121,115,116,101,109,32,97,114,101,
-    32,99,97,99,104,101,100,32,102,111,114,32,112,101,114,102,
-    111,114,109,97,110,99,101,44,32,98,101,105,110,103,10,32,
-    32,32,32,114,101,102,114,101,115,104,101,100,32,119,104,101,
-    110,32,116,104,101,32,100,105,114,101,99,116,111,114,121,32,
-    116,104,101,32,102,105,110,100,101,114,32,105,115,32,104,97,
-    110,100,108,105,110,103,32,104,97,115,32,98,101,101,110,32,
-    109,111,100,105,102,105,101,100,46,10,10,32,32,32,32,99,
-    2,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,
-    6,0,0,0,7,0,0,0,115,84,0,0,0,103,0,125,
-    3,124,2,68,0,93,32,92,2,137,0,125,4,124,3,160,
-    0,135,0,102,1,100,1,100,2,132,8,124,4,68,0,131,
-    1,161,1,1,0,113,8,124,3,124,0,95,1,124,1,112,
-    54,100,3,124,0,95,2,100,4,124,0,95,3,116,4,131,
-    0,124,0,95,5,116,4,131,0,124,0,95,6,100,5,83,
-    0,41,6,122,154,73,110,105,116,105,97,108,105,122,101,32,
-    119,105,116,104,32,116,104,101,32,112,97,116,104,32,116,111,
-    32,115,101,97,114,99,104,32,111,110,32,97,110,100,32,97,
-    32,118,97,114,105,97,98,108,101,32,110,117,109,98,101,114,
-    32,111,102,10,32,32,32,32,32,32,32,32,50,45,116,117,
-    112,108,101,115,32,99,111,110,116,97,105,110,105,110,103,32,
-    116,104,101,32,108,111,97,100,101,114,32,97,110,100,32,116,
-    104,101,32,102,105,108,101,32,115,117,102,102,105,120,101,115,
-    32,116,104,101,32,108,111,97,100,101,114,10,32,32,32,32,
-    32,32,32,32,114,101,99,111,103,110,105,122,101,115,46,99,
+    114,32,97,108,108,32,110,97,109,101,115,32,105,102,32,96,
+    96,78,111,110,101,96,96,32,105,110,100,105,99,97,116,101,
+    100,41,32,97,108,111,110,103,32,116,104,101,32,112,97,116,
+    104,115,32,105,110,32,116,104,101,32,108,105,115,116,10,32,
+    32,32,32,32,32,32,32,111,102,32,100,105,114,101,99,116,
+    111,114,105,101,115,32,96,96,99,111,110,116,101,120,116,46,
+    112,97,116,104,96,96,46,10,32,32,32,32,32,32,32,32,
+    114,73,0,0,0,41,2,218,16,80,97,116,104,68,105,115,
+    116,114,105,98,117,116,105,111,110,218,18,68,105,115,116,114,
+    105,98,117,116,105,111,110,70,105,110,100,101,114,78,41,8,
+    90,18,105,109,112,111,114,116,108,105,98,46,109,101,116,97,
+    100,97,116,97,114,59,1,0,0,114,60,1,0,0,90,7,
+    67,111,110,116,101,120,116,218,13,95,115,101,97,114,99,104,
+    95,112,97,116,104,115,218,7,112,97,116,116,101,114,110,114,
+    44,0,0,0,218,3,109,97,112,41,5,114,119,0,0,0,
+    90,7,99,111,110,116,101,120,116,114,59,1,0,0,114,60,
+    1,0,0,90,5,102,111,117,110,100,114,3,0,0,0,114,
+    3,0,0,0,114,6,0,0,0,218,18,102,105,110,100,95,
+    100,105,115,116,114,105,98,117,116,105,111,110,115,91,5,0,
+    0,115,10,0,0,0,0,10,16,1,8,1,8,1,16,1,
+    122,29,80,97,116,104,70,105,110,100,101,114,46,102,105,110,
+    100,95,100,105,115,116,114,105,98,117,116,105,111,110,115,99,
+    3,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,
+    6,0,0,0,3,0,0,0,115,44,0,0,0,100,1,100,
+    2,108,0,125,3,124,3,106,1,160,2,135,0,135,1,102,
+    2,100,3,100,4,132,8,116,3,136,0,106,4,124,2,131,
+    2,68,0,131,1,161,1,83,0,41,5,122,49,70,105,110,
+    100,32,109,101,116,97,100,97,116,97,32,100,105,114,101,99,
+    116,111,114,105,101,115,32,105,110,32,112,97,116,104,115,32,
+    104,101,117,114,105,115,116,105,99,97,108,108,121,46,114,73,
+    0,0,0,78,99,1,0,0,0,0,0,0,0,0,0,0,
+    0,2,0,0,0,5,0,0,0,51,0,0,0,115,26,0,
+    0,0,124,0,93,18,125,1,136,0,160,0,124,1,136,1,
+    161,2,86,0,1,0,113,2,100,0,83,0,114,110,0,0,
+    0,41,1,218,12,95,115,101,97,114,99,104,95,112,97,116,
+    104,41,2,114,32,0,0,0,114,44,0,0,0,169,2,114,
+    193,0,0,0,114,62,1,0,0,114,3,0,0,0,114,6,
+    0,0,0,114,19,1,0,0,111,5,0,0,115,4,0,0,
+    0,4,2,2,255,122,43,80,97,116,104,70,105,110,100,101,
+    114,46,95,115,101,97,114,99,104,95,112,97,116,104,115,46,
+    60,108,111,99,97,108,115,62,46,60,103,101,110,101,120,112,
+    114,62,41,5,218,9,105,116,101,114,116,111,111,108,115,90,
+    5,99,104,97,105,110,90,13,102,114,111,109,95,105,116,101,
+    114,97,98,108,101,114,63,1,0,0,218,12,95,115,119,105,
+    116,99,104,95,112,97,116,104,41,4,114,193,0,0,0,114,
+    62,1,0,0,90,5,112,97,116,104,115,114,67,1,0,0,
+    114,3,0,0,0,114,66,1,0,0,114,6,0,0,0,114,
+    61,1,0,0,107,5,0,0,115,8,0,0,0,0,3,8,
+    1,18,2,10,254,122,24,80,97,116,104,70,105,110,100,101,
+    114,46,95,115,101,97,114,99,104,95,112,97,116,104,115,99,
+    1,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,
+    10,0,0,0,67,0,0,0,115,96,0,0,0,100,1,100,
+    2,108,0,109,1,125,1,1,0,100,1,100,0,108,2,125,
+    2,100,1,100,0,108,3,125,3,100,3,125,4,124,4,114,
+    48,116,4,106,5,160,6,124,0,161,1,114,86,124,1,116,
+    7,131,1,143,24,1,0,124,2,160,8,124,0,161,1,87,
+    0,2,0,53,0,81,0,82,0,163,0,83,0,81,0,82,
+    0,88,0,124,3,160,8,124,0,161,1,83,0,41,4,78,
+    114,73,0,0,0,41,1,218,8,115,117,112,112,114,101,115,
+    115,70,41,9,90,10,99,111,110,116,101,120,116,108,105,98,
+    114,69,1,0,0,218,7,122,105,112,102,105,108,101,218,7,
+    112,97,116,104,108,105,98,90,2,111,115,114,44,0,0,0,
+    90,6,105,115,102,105,108,101,218,9,69,120,99,101,112,116,
+    105,111,110,90,4,80,97,116,104,41,5,114,44,0,0,0,
+    114,69,1,0,0,114,70,1,0,0,114,71,1,0,0,90,
+    13,80,89,80,89,95,79,80,69,78,95,66,85,71,114,3,
+    0,0,0,114,3,0,0,0,114,6,0,0,0,114,68,1,
+    0,0,116,5,0,0,115,16,0,0,0,0,2,12,1,8,
+    1,8,1,4,1,16,1,10,1,28,1,122,23,80,97,116,
+    104,70,105,110,100,101,114,46,95,115,119,105,116,99,104,95,
+    112,97,116,104,99,3,0,0,0,0,0,0,0,0,0,0,
+    0,6,0,0,0,5,0,0,0,67,0,0,0,115,44,0,
+    0,0,100,1,100,0,108,0,125,3,100,2,125,4,124,4,
+    106,1,124,1,100,3,141,1,125,5,124,3,106,2,124,5,
+    124,2,106,3,124,3,106,4,100,4,141,3,83,0,41,5,
+    78,114,73,0,0,0,122,32,123,112,97,116,116,101,114,110,
+    125,40,45,46,42,41,63,92,46,40,100,105,115,116,124,101,
+    103,103,41,45,105,110,102,111,169,1,114,62,1,0,0,169,
+    1,114,83,0,0,0,41,5,218,2,114,101,114,62,0,0,
+    0,90,5,109,97,116,99,104,114,117,0,0,0,218,10,73,
+    71,78,79,82,69,67,65,83,69,169,6,114,193,0,0,0,
+    218,10,110,111,114,109,97,108,105,122,101,100,114,41,1,0,
+    0,114,75,1,0,0,90,8,116,101,109,112,108,97,116,101,
+    90,8,109,97,110,105,102,101,115,116,114,3,0,0,0,114,
+    3,0,0,0,114,6,0,0,0,218,13,95,109,97,116,99,
+    104,101,115,95,105,110,102,111,127,5,0,0,115,8,0,0,
+    0,0,2,8,1,4,1,12,1,122,24,80,97,116,104,70,
+    105,110,100,101,114,46,95,109,97,116,99,104,101,115,95,105,
+    110,102,111,99,3,0,0,0,0,0,0,0,0,0,0,0,
+    6,0,0,0,5,0,0,0,67,0,0,0,115,46,0,0,
+    0,100,1,100,0,108,0,125,3,100,2,125,4,124,4,106,
+    1,124,1,100,3,141,1,125,5,124,3,106,2,124,5,116,
+    3,124,2,131,1,124,3,106,4,100,4,141,3,83,0,41,
+    5,78,114,73,0,0,0,122,30,123,112,97,116,116,101,114,
+    110,125,45,46,42,92,46,101,103,103,91,92,92,47,93,69,
+    71,71,45,73,78,70,79,114,73,1,0,0,114,74,1,0,
+    0,41,5,114,75,1,0,0,114,62,0,0,0,90,6,115,
+    101,97,114,99,104,114,85,0,0,0,114,76,1,0,0,114,
+    77,1,0,0,114,3,0,0,0,114,3,0,0,0,114,6,
+    0,0,0,218,15,95,109,97,116,99,104,101,115,95,108,101,
+    103,97,99,121,134,5,0,0,115,8,0,0,0,0,2,8,
+    1,4,1,12,1,122,26,80,97,116,104,70,105,110,100,101,
+    114,46,95,109,97,116,99,104,101,115,95,108,101,103,97,99,
+    121,99,3,0,0,0,0,0,0,0,0,0,0,0,3,0,
+    0,0,4,0,0,0,3,0,0,0,115,48,0,0,0,124,
+    1,160,0,161,0,115,12,100,1,83,0,124,2,160,1,100,
+    2,100,3,161,2,137,1,135,0,135,1,102,2,100,4,100,
+    5,132,8,124,1,160,2,161,0,68,0,131,1,83,0,41,
+    6,78,114,3,0,0,0,250,1,45,114,45,0,0,0,99,
     1,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,
-    3,0,0,0,51,0,0,0,115,22,0,0,0,124,0,93,
-    14,125,1,124,1,136,0,102,2,86,0,1,0,113,2,100,
-    0,83,0,114,110,0,0,0,114,3,0,0,0,114,16,1,
-    0,0,169,1,114,140,0,0,0,114,3,0,0,0,114,6,
-    0,0,0,114,19,1,0,0,168,5,0,0,115,4,0,0,
-    0,4,0,2,0,122,38,70,105,108,101,70,105,110,100,101,
-    114,46,95,95,105,110,105,116,95,95,46,60,108,111,99,97,
-    108,115,62,46,60,103,101,110,101,120,112,114,62,114,71,0,
-    0,0,114,105,0,0,0,78,41,7,114,167,0,0,0,218,
-    8,95,108,111,97,100,101,114,115,114,44,0,0,0,218,11,
-    95,112,97,116,104,95,109,116,105,109,101,218,3,115,101,116,
-    218,11,95,112,97,116,104,95,99,97,99,104,101,218,19,95,
-    114,101,108,97,120,101,100,95,112,97,116,104,95,99,97,99,
-    104,101,41,5,114,119,0,0,0,114,44,0,0,0,218,14,
-    108,111,97,100,101,114,95,100,101,116,97,105,108,115,90,7,
-    108,111,97,100,101,114,115,114,189,0,0,0,114,3,0,0,
-    0,114,84,1,0,0,114,6,0,0,0,114,209,0,0,0,
-    162,5,0,0,115,16,0,0,0,0,4,4,1,12,1,26,
-    1,6,2,10,1,6,1,8,1,122,19,70,105,108,101,70,
-    105,110,100,101,114,46,95,95,105,110,105,116,95,95,99,1,
-    0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,2,
-    0,0,0,67,0,0,0,115,10,0,0,0,100,1,124,0,
-    95,0,100,2,83,0,41,3,122,31,73,110,118,97,108,105,
-    100,97,116,101,32,116,104,101,32,100,105,114,101,99,116,111,
-    114,121,32,109,116,105,109,101,46,114,105,0,0,0,78,41,
-    1,114,86,1,0,0,114,246,0,0,0,114,3,0,0,0,
-    114,3,0,0,0,114,6,0,0,0,114,46,1,0,0,176,
-    5,0,0,115,2,0,0,0,0,2,122,28,70,105,108,101,
-    70,105,110,100,101,114,46,105,110,118,97,108,105,100,97,116,
-    101,95,99,97,99,104,101,115,99,2,0,0,0,0,0,0,
-    0,0,0,0,0,3,0,0,0,3,0,0,0,67,0,0,
-    0,115,42,0,0,0,124,0,160,0,124,1,161,1,125,2,
-    124,2,100,1,107,8,114,26,100,1,103,0,102,2,83,0,
-    124,2,106,1,124,2,106,2,112,38,103,0,102,2,83,0,
-    41,2,122,197,84,114,121,32,116,111,32,102,105,110,100,32,
-    97,32,108,111,97,100,101,114,32,102,111,114,32,116,104,101,
-    32,115,112,101,99,105,102,105,101,100,32,109,111,100,117,108,
-    101,44,32,111,114,32,116,104,101,32,110,97,109,101,115,112,
-    97,99,101,10,32,32,32,32,32,32,32,32,112,97,99,107,
-    97,103,101,32,112,111,114,116,105,111,110,115,46,32,82,101,
-    116,117,114,110,115,32,40,108,111,97,100,101,114,44,32,108,
-    105,115,116,45,111,102,45,112,111,114,116,105,111,110,115,41,
-    46,10,10,32,32,32,32,32,32,32,32,84,104,105,115,32,
-    109,101,116,104,111,100,32,105,115,32,100,101,112,114,101,99,
-    97,116,101,100,46,32,32,85,115,101,32,102,105,110,100,95,
-    115,112,101,99,40,41,32,105,110,115,116,101,97,100,46,10,
-    10,32,32,32,32,32,32,32,32,78,41,3,114,203,0,0,
-    0,114,140,0,0,0,114,178,0,0,0,41,3,114,119,0,
-    0,0,114,139,0,0,0,114,187,0,0,0,114,3,0,0,
-    0,114,3,0,0,0,114,6,0,0,0,114,137,0,0,0,
-    182,5,0,0,115,8,0,0,0,0,7,10,1,8,1,8,
-    1,122,22,70,105,108,101,70,105,110,100,101,114,46,102,105,
-    110,100,95,108,111,97,100,101,114,99,6,0,0,0,0,0,
-    0,0,0,0,0,0,7,0,0,0,6,0,0,0,67,0,
-    0,0,115,26,0,0,0,124,1,124,2,124,3,131,2,125,
-    6,116,0,124,2,124,3,124,6,124,4,100,1,141,4,83,
-    0,41,2,78,114,177,0,0,0,41,1,114,190,0,0,0,
-    41,7,114,119,0,0,0,114,188,0,0,0,114,139,0,0,
-    0,114,44,0,0,0,90,4,115,109,115,108,114,202,0,0,
-    0,114,140,0,0,0,114,3,0,0,0,114,3,0,0,0,
-    114,6,0,0,0,114,58,1,0,0,194,5,0,0,115,8,
-    0,0,0,0,1,10,1,8,1,2,255,122,20,70,105,108,
-    101,70,105,110,100,101,114,46,95,103,101,116,95,115,112,101,
-    99,78,99,3,0,0,0,0,0,0,0,0,0,0,0,14,
-    0,0,0,8,0,0,0,67,0,0,0,115,98,1,0,0,
-    100,1,125,3,124,1,160,0,100,2,161,1,100,3,25,0,
-    125,4,122,24,116,1,124,0,106,2,112,34,116,3,160,4,
-    161,0,131,1,106,5,125,5,87,0,110,24,4,0,116,6,
-    107,10,114,66,1,0,1,0,1,0,100,4,125,5,89,0,
-    110,2,88,0,124,5,124,0,106,7,107,3,114,92,124,0,
-    160,8,161,0,1,0,124,5,124,0,95,7,116,9,131,0,
-    114,114,124,0,106,10,125,6,124,4,160,11,161,0,125,7,
-    110,10,124,0,106,12,125,6,124,4,125,7,124,7,124,6,
-    107,6,114,218,116,13,124,0,106,2,124,4,131,2,125,8,
-    124,0,106,14,68,0,93,58,92,2,125,9,125,10,100,5,
-    124,9,23,0,125,11,116,13,124,8,124,11,131,2,125,12,
-    116,15,124,12,131,1,114,150,124,0,160,16,124,10,124,1,
-    124,12,124,8,103,1,124,2,161,5,2,0,1,0,83,0,
-    113,150,116,17,124,8,131,1,125,3,124,0,106,14,68,0,
-    93,82,92,2,125,9,125,10,116,13,124,0,106,2,124,4,
-    124,9,23,0,131,2,125,12,116,18,106,19,100,6,124,12,
-    100,3,100,7,141,3,1,0,124,7,124,9,23,0,124,6,
-    107,6,114,224,116,15,124,12,131,1,114,224,124,0,160,16,
-    124,10,124,1,124,12,100,8,124,2,161,5,2,0,1,0,
-    83,0,113,224,124,3,144,1,114,94,116,18,160,19,100,9,
-    124,8,161,2,1,0,116,18,160,20,124,1,100,8,161,2,
-    125,13,124,8,103,1,124,13,95,21,124,13,83,0,100,8,
-    83,0,41,10,122,111,84,114,121,32,116,111,32,102,105,110,
-    100,32,97,32,115,112,101,99,32,102,111,114,32,116,104,101,
-    32,115,112,101,99,105,102,105,101,100,32,109,111,100,117,108,
-    101,46,10,10,32,32,32,32,32,32,32,32,82,101,116,117,
-    114,110,115,32,116,104,101,32,109,97,116,99,104,105,110,103,
-    32,115,112,101,99,44,32,111,114,32,78,111,110,101,32,105,
-    102,32,110,111,116,32,102,111,117,110,100,46,10,32,32,32,
-    32,32,32,32,32,70,114,71,0,0,0,114,28,0,0,0,
-    114,105,0,0,0,114,209,0,0,0,122,9,116,114,121,105,
-    110,103,32,123,125,41,1,90,9,118,101,114,98,111,115,105,
-    116,121,78,122,25,112,111,115,115,105,98,108,101,32,110,97,
-    109,101,115,112,97,99,101,32,102,111,114,32,123,125,41,22,
-    114,41,0,0,0,114,49,0,0,0,114,44,0,0,0,114,
-    2,0,0,0,114,55,0,0,0,114,10,1,0,0,114,50,
-    0,0,0,114,86,1,0,0,218,11,95,102,105,108,108,95,
-    99,97,99,104,101,114,7,0,0,0,114,89,1,0,0,114,
-    106,0,0,0,114,88,1,0,0,114,38,0,0,0,114,85,
-    1,0,0,114,54,0,0,0,114,58,1,0,0,114,56,0,
-    0,0,114,134,0,0,0,114,149,0,0,0,114,183,0,0,
-    0,114,178,0,0,0,41,14,114,119,0,0,0,114,139,0,
-    0,0,114,202,0,0,0,90,12,105,115,95,110,97,109,101,
-    115,112,97,99,101,90,11,116,97,105,108,95,109,111,100,117,
-    108,101,114,169,0,0,0,90,5,99,97,99,104,101,90,12,
-    99,97,99,104,101,95,109,111,100,117,108,101,90,9,98,97,
-    115,101,95,112,97,116,104,114,17,1,0,0,114,188,0,0,
-    0,90,13,105,110,105,116,95,102,105,108,101,110,97,109,101,
-    90,9,102,117,108,108,95,112,97,116,104,114,187,0,0,0,
+    5,0,0,0,51,0,0,0,115,42,0,0,0,124,0,93,
+    34,125,1,136,0,160,0,136,1,124,1,161,2,115,30,136,
+    0,160,1,136,1,124,1,161,2,114,2,124,1,86,0,1,
+    0,113,2,100,0,83,0,114,110,0,0,0,41,2,114,79,
+    1,0,0,114,80,1,0,0,41,2,114,32,0,0,0,114,
+    41,1,0,0,169,2,114,193,0,0,0,114,78,1,0,0,
+    114,3,0,0,0,114,6,0,0,0,114,19,1,0,0,146,
+    5,0,0,115,8,0,0,0,4,0,2,1,12,1,12,254,
+    122,42,80,97,116,104,70,105,110,100,101,114,46,95,115,101,
+    97,114,99,104,95,112,97,116,104,46,60,108,111,99,97,108,
+    115,62,46,60,103,101,110,101,120,112,114,62,41,3,90,6,
+    105,115,95,100,105,114,114,67,0,0,0,90,7,105,116,101,
+    114,100,105,114,41,3,114,193,0,0,0,90,4,114,111,111,
+    116,114,62,1,0,0,114,3,0,0,0,114,82,1,0,0,
+    114,6,0,0,0,114,65,1,0,0,141,5,0,0,115,8,
+    0,0,0,0,2,8,1,4,1,12,1,122,23,80,97,116,
+    104,70,105,110,100,101,114,46,95,115,101,97,114,99,104,95,
+    112,97,116,104,41,1,78,41,2,78,78,41,1,78,41,1,
+    78,41,19,114,125,0,0,0,114,124,0,0,0,114,126,0,
+    0,0,114,127,0,0,0,114,207,0,0,0,114,46,1,0,
+    0,114,52,1,0,0,114,54,1,0,0,114,55,1,0,0,
+    114,58,1,0,0,114,203,0,0,0,114,206,0,0,0,114,
+    64,1,0,0,114,61,1,0,0,218,12,115,116,97,116,105,
+    99,109,101,116,104,111,100,114,68,1,0,0,114,79,1,0,
+    0,114,80,1,0,0,114,65,1,0,0,114,3,0,0,0,
     114,3,0,0,0,114,3,0,0,0,114,6,0,0,0,114,
-    203,0,0,0,199,5,0,0,115,74,0,0,0,0,5,4,
-    1,14,1,2,1,24,1,14,1,10,1,10,1,8,1,6,
-    2,6,1,6,1,10,2,6,1,4,2,8,1,12,1,14,
-    1,8,1,10,1,8,1,26,4,8,2,14,1,16,1,16,
-    1,12,1,8,1,10,1,2,0,2,255,10,2,6,1,12,
-    1,12,1,8,1,4,1,122,20,70,105,108,101,70,105,110,
-    100,101,114,46,102,105,110,100,95,115,112,101,99,99,1,0,
-    0,0,0,0,0,0,0,0,0,0,9,0,0,0,10,0,
-    0,0,67,0,0,0,115,190,0,0,0,124,0,106,0,125,
-    1,122,22,116,1,160,2,124,1,112,22,116,1,160,3,161,
-    0,161,1,125,2,87,0,110,30,4,0,116,4,116,5,116,
-    6,102,3,107,10,114,58,1,0,1,0,1,0,103,0,125,
-    2,89,0,110,2,88,0,116,7,106,8,160,9,100,1,161,
-    1,115,84,116,10,124,2,131,1,124,0,95,11,110,74,116,
-    10,131,0,125,3,124,2,68,0,93,56,125,4,124,4,160,
-    12,100,2,161,1,92,3,125,5,125,6,125,7,124,6,114,
-    136,100,3,160,13,124,5,124,7,160,14,161,0,161,2,125,
-    8,110,4,124,5,125,8,124,3,160,15,124,8,161,1,1,
-    0,113,94,124,3,124,0,95,11,116,7,106,8,160,9,116,
-    16,161,1,114,186,100,4,100,5,132,0,124,2,68,0,131,
-    1,124,0,95,17,100,6,83,0,41,7,122,68,70,105,108,
-    108,32,116,104,101,32,99,97,99,104,101,32,111,102,32,112,
-    111,116,101,110,116,105,97,108,32,109,111,100,117,108,101,115,
-    32,97,110,100,32,112,97,99,107,97,103,101,115,32,102,111,
-    114,32,116,104,105,115,32,100,105,114,101,99,116,111,114,121,
-    46,114,0,0,0,0,114,71,0,0,0,114,61,0,0,0,
-    99,1,0,0,0,0,0,0,0,0,0,0,0,2,0,0,
-    0,4,0,0,0,83,0,0,0,115,20,0,0,0,104,0,
-    124,0,93,12,125,1,124,1,160,0,161,0,146,2,113,4,
-    83,0,114,3,0,0,0,41,1,114,106,0,0,0,41,2,
-    114,32,0,0,0,90,2,102,110,114,3,0,0,0,114,3,
-    0,0,0,114,6,0,0,0,218,9,60,115,101,116,99,111,
-    109,112,62,20,6,0,0,115,4,0,0,0,6,0,2,0,
-    122,41,70,105,108,101,70,105,110,100,101,114,46,95,102,105,
-    108,108,95,99,97,99,104,101,46,60,108,111,99,97,108,115,
-    62,46,60,115,101,116,99,111,109,112,62,78,41,18,114,44,
-    0,0,0,114,2,0,0,0,114,7,1,0,0,114,55,0,
-    0,0,114,3,1,0,0,218,15,80,101,114,109,105,115,115,
-    105,111,110,69,114,114,111,114,218,18,78,111,116,65,68,105,
-    114,101,99,116,111,114,121,69,114,114,111,114,114,8,0,0,
-    0,114,9,0,0,0,114,10,0,0,0,114,87,1,0,0,
-    114,88,1,0,0,114,101,0,0,0,114,62,0,0,0,114,
-    106,0,0,0,218,3,97,100,100,114,11,0,0,0,114,89,
-    1,0,0,41,9,114,119,0,0,0,114,44,0,0,0,114,
-    8,1,0,0,90,21,108,111,119,101,114,95,115,117,102,102,
-    105,120,95,99,111,110,116,101,110,116,115,114,41,1,0,0,
-    114,117,0,0,0,114,29,1,0,0,114,17,1,0,0,90,
-    8,110,101,119,95,110,97,109,101,114,3,0,0,0,114,3,
-    0,0,0,114,6,0,0,0,114,91,1,0,0,247,5,0,
-    0,115,34,0,0,0,0,2,6,1,2,1,22,1,20,3,
-    10,3,12,1,12,7,6,1,8,1,16,1,4,1,18,2,
-    4,1,12,1,6,1,12,1,122,22,70,105,108,101,70,105,
-    110,100,101,114,46,95,102,105,108,108,95,99,97,99,104,101,
-    99,1,0,0,0,0,0,0,0,0,0,0,0,3,0,0,
-    0,3,0,0,0,7,0,0,0,115,18,0,0,0,135,0,
-    135,1,102,2,100,1,100,2,132,8,125,2,124,2,83,0,
-    41,3,97,20,1,0,0,65,32,99,108,97,115,115,32,109,
-    101,116,104,111,100,32,119,104,105,99,104,32,114,101,116,117,
-    114,110,115,32,97,32,99,108,111,115,117,114,101,32,116,111,
-    32,117,115,101,32,111,110,32,115,121,115,46,112,97,116,104,
-    95,104,111,111,107,10,32,32,32,32,32,32,32,32,119,104,
-    105,99,104,32,119,105,108,108,32,114,101,116,117,114,110,32,
-    97,110,32,105,110,115,116,97,110,99,101,32,117,115,105,110,
-    103,32,116,104,101,32,115,112,101,99,105,102,105,101,100,32,
-    108,111,97,100,101,114,115,32,97,110,100,32,116,104,101,32,
-    112,97,116,104,10,32,32,32,32,32,32,32,32,99,97,108,
-    108,101,100,32,111,110,32,116,104,101,32,99,108,111,115,117,
-    114,101,46,10,10,32,32,32,32,32,32,32,32,73,102,32,
-    116,104,101,32,112,97,116,104,32,99,97,108,108,101,100,32,
-    111,110,32,116,104,101,32,99,108,111,115,117,114,101,32,105,
-    115,32,110,111,116,32,97,32,100,105,114,101,99,116,111,114,
-    121,44,32,73,109,112,111,114,116,69,114,114,111,114,32,105,
-    115,10,32,32,32,32,32,32,32,32,114,97,105,115,101,100,
-    46,10,10,32,32,32,32,32,32,32,32,99,1,0,0,0,
-    0,0,0,0,0,0,0,0,1,0,0,0,4,0,0,0,
-    19,0,0,0,115,34,0,0,0,116,0,124,0,131,1,115,
-    20,116,1,100,1,124,0,100,2,141,2,130,1,136,0,124,
-    0,102,1,136,1,158,2,142,0,83,0,41,3,122,45,80,
-    97,116,104,32,104,111,111,107,32,102,111,114,32,105,109,112,
-    111,114,116,108,105,98,46,109,97,99,104,105,110,101,114,121,
-    46,70,105,108,101,70,105,110,100,101,114,46,122,30,111,110,
-    108,121,32,100,105,114,101,99,116,111,114,105,101,115,32,97,
-    114,101,32,115,117,112,112,111,114,116,101,100,114,48,0,0,
-    0,41,2,114,56,0,0,0,114,118,0,0,0,114,48,0,
-    0,0,169,2,114,193,0,0,0,114,90,1,0,0,114,3,
-    0,0,0,114,6,0,0,0,218,24,112,97,116,104,95,104,
-    111,111,107,95,102,111,114,95,70,105,108,101,70,105,110,100,
-    101,114,32,6,0,0,115,6,0,0,0,0,2,8,1,12,
-    1,122,54,70,105,108,101,70,105,110,100,101,114,46,112,97,
-    116,104,95,104,111,111,107,46,60,108,111,99,97,108,115,62,
-    46,112,97,116,104,95,104,111,111,107,95,102,111,114,95,70,
-    105,108,101,70,105,110,100,101,114,114,3,0,0,0,41,3,
-    114,193,0,0,0,114,90,1,0,0,114,97,1,0,0,114,
-    3,0,0,0,114,96,1,0,0,114,6,0,0,0,218,9,
-    112,97,116,104,95,104,111,111,107,22,6,0,0,115,4,0,
-    0,0,0,10,14,6,122,20,70,105,108,101,70,105,110,100,
-    101,114,46,112,97,116,104,95,104,111,111,107,99,1,0,0,
-    0,0,0,0,0,0,0,0,0,1,0,0,0,3,0,0,
-    0,67,0,0,0,115,12,0,0,0,100,1,160,0,124,0,
-    106,1,161,1,83,0,41,2,78,122,16,70,105,108,101,70,
-    105,110,100,101,114,40,123,33,114,125,41,41,2,114,62,0,
-    0,0,114,44,0,0,0,114,246,0,0,0,114,3,0,0,
-    0,114,3,0,0,0,114,6,0,0,0,114,39,1,0,0,
-    40,6,0,0,115,2,0,0,0,0,1,122,19,70,105,108,
-    101,70,105,110,100,101,114,46,95,95,114,101,112,114,95,95,
-    41,1,78,41,15,114,125,0,0,0,114,124,0,0,0,114,
-    126,0,0,0,114,127,0,0,0,114,209,0,0,0,114,46,
-    1,0,0,114,143,0,0,0,114,206,0,0,0,114,137,0,
-    0,0,114,58,1,0,0,114,203,0,0,0,114,91,1,0,
-    0,114,207,0,0,0,114,98,1,0,0,114,39,1,0,0,
-    114,3,0,0,0,114,3,0,0,0,114,3,0,0,0,114,
-    6,0,0,0,114,83,1,0,0,153,5,0,0,115,22,0,
-    0,0,8,2,4,7,8,14,8,4,4,2,8,12,8,5,
-    10,48,8,31,2,1,10,17,114,83,1,0,0,99,4,0,
-    0,0,0,0,0,0,0,0,0,0,6,0,0,0,8,0,
-    0,0,67,0,0,0,115,146,0,0,0,124,0,160,0,100,
-    1,161,1,125,4,124,0,160,0,100,2,161,1,125,5,124,
-    4,115,66,124,5,114,36,124,5,106,1,125,4,110,30,124,
-    2,124,3,107,2,114,56,116,2,124,1,124,2,131,2,125,
-    4,110,10,116,3,124,1,124,2,131,2,125,4,124,5,115,
-    84,116,4,124,1,124,2,124,4,100,3,141,3,125,5,122,
-    36,124,5,124,0,100,2,60,0,124,4,124,0,100,1,60,
-    0,124,2,124,0,100,4,60,0,124,3,124,0,100,5,60,
-    0,87,0,110,20,4,0,116,5,107,10,114,140,1,0,1,
-    0,1,0,89,0,110,2,88,0,100,0,83,0,41,6,78,
-    218,10,95,95,108,111,97,100,101,114,95,95,218,8,95,95,
-    115,112,101,99,95,95,114,84,1,0,0,90,8,95,95,102,
-    105,108,101,95,95,90,10,95,95,99,97,99,104,101,100,95,
-    95,41,6,218,3,103,101,116,114,140,0,0,0,114,15,1,
-    0,0,114,9,1,0,0,114,190,0,0,0,114,72,1,0,
-    0,41,6,90,2,110,115,114,117,0,0,0,90,8,112,97,
-    116,104,110,97,109,101,90,9,99,112,97,116,104,110,97,109,
-    101,114,140,0,0,0,114,187,0,0,0,114,3,0,0,0,
-    114,3,0,0,0,114,6,0,0,0,218,14,95,102,105,120,
-    95,117,112,95,109,111,100,117,108,101,46,6,0,0,115,34,
-    0,0,0,0,2,10,1,10,1,4,1,4,1,8,1,8,
-    1,12,2,10,1,4,1,14,1,2,1,8,1,8,1,8,
-    1,12,1,14,2,114,102,1,0,0,99,0,0,0,0,0,
-    0,0,0,0,0,0,0,3,0,0,0,3,0,0,0,67,
-    0,0,0,115,38,0,0,0,116,0,116,1,160,2,161,0,
-    102,2,125,0,116,3,116,4,102,2,125,1,116,5,116,6,
-    102,2,125,2,124,0,124,1,124,2,103,3,83,0,41,1,
-    122,95,82,101,116,117,114,110,115,32,97,32,108,105,115,116,
-    32,111,102,32,102,105,108,101,45,98,97,115,101,100,32,109,
-    111,100,117,108,101,32,108,111,97,100,101,114,115,46,10,10,
-    32,32,32,32,69,97,99,104,32,105,116,101,109,32,105,115,
-    32,97,32,116,117,112,108,101,32,40,108,111,97,100,101,114,
-    44,32,115,117,102,102,105,120,101,115,41,46,10,32,32,32,
-    32,41,7,114,252,0,0,0,114,163,0,0,0,218,18,101,
-    120,116,101,110,115,105,111,110,95,115,117,102,102,105,120,101,
-    115,114,9,1,0,0,114,102,0,0,0,114,15,1,0,0,
-    114,89,0,0,0,41,3,90,10,101,120,116,101,110,115,105,
-    111,110,115,90,6,115,111,117,114,99,101,90,8,98,121,116,
-    101,99,111,100,101,114,3,0,0,0,114,3,0,0,0,114,
-    6,0,0,0,114,184,0,0,0,69,6,0,0,115,8,0,
-    0,0,0,5,12,1,8,1,8,1,114,184,0,0,0,99,
-    1,0,0,0,0,0,0,0,0,0,0,0,12,0,0,0,
-    9,0,0,0,67,0,0,0,115,178,1,0,0,124,0,97,
-    0,116,0,106,1,97,1,116,0,106,2,97,2,116,1,106,
-    3,116,4,25,0,125,1,100,1,68,0,93,48,125,2,124,
-    2,116,1,106,3,107,7,114,56,116,0,160,5,124,2,161,
-    1,125,3,110,10,116,1,106,3,124,2,25,0,125,3,116,
-    6,124,1,124,2,124,3,131,3,1,0,113,30,100,2,100,
-    3,103,1,102,2,100,4,100,5,100,3,103,2,102,2,102,
-    2,125,4,124,4,68,0,93,110,92,2,125,5,125,6,116,
-    7,100,6,100,7,132,0,124,6,68,0,131,1,131,1,115,
-    136,116,8,130,1,124,6,100,8,25,0,125,7,124,5,116,
-    1,106,3,107,6,114,170,116,1,106,3,124,5,25,0,125,
-    8,1,0,113,226,113,106,122,20,116,0,160,5,124,5,161,
-    1,125,8,87,0,1,0,113,226,87,0,113,106,4,0,116,
-    9,107,10,114,214,1,0,1,0,1,0,89,0,113,106,89,
-    0,113,106,88,0,113,106,116,9,100,9,131,1,130,1,116,
-    6,124,1,100,10,124,8,131,3,1,0,116,6,124,1,100,
-    11,124,7,131,3,1,0,116,6,124,1,100,12,100,13,160,
-    10,124,6,161,1,131,3,1,0,116,6,124,1,100,14,100,
-    15,100,16,132,0,124,6,68,0,131,1,131,3,1,0,116,
-    0,160,5,100,17,161,1,125,9,116,6,124,1,100,17,124,
-    9,131,3,1,0,116,0,160,5,100,18,161,1,125,10,116,
-    6,124,1,100,18,124,10,131,3,1,0,124,5,100,4,107,
-    2,144,1,114,110,116,0,160,5,100,19,161,1,125,11,116,
-    6,124,1,100,20,124,11,131,3,1,0,116,6,124,1,100,
-    21,116,11,131,0,131,3,1,0,116,12,160,13,116,2,160,
-    14,161,0,161,1,1,0,124,5,100,4,107,2,144,1,114,
-    174,116,15,160,16,100,22,161,1,1,0,100,23,116,12,107,
-    6,144,1,114,174,100,24,116,17,95,18,100,25,83,0,41,
-    26,122,205,83,101,116,117,112,32,116,104,101,32,112,97,116,
-    104,45,98,97,115,101,100,32,105,109,112,111,114,116,101,114,
-    115,32,102,111,114,32,105,109,112,111,114,116,108,105,98,32,
-    98,121,32,105,109,112,111,114,116,105,110,103,32,110,101,101,
-    100,101,100,10,32,32,32,32,98,117,105,108,116,45,105,110,
-    32,109,111,100,117,108,101,115,32,97,110,100,32,105,110,106,
-    101,99,116,105,110,103,32,116,104,101,109,32,105,110,116,111,
-    32,116,104,101,32,103,108,111,98,97,108,32,110,97,109,101,
-    115,112,97,99,101,46,10,10,32,32,32,32,79,116,104,101,
-    114,32,99,111,109,112,111,110,101,110,116,115,32,97,114,101,
-    32,101,120,116,114,97,99,116,101,100,32,102,114,111,109,32,
-    116,104,101,32,99,111,114,101,32,98,111,111,116,115,116,114,
-    97,112,32,109,111,100,117,108,101,46,10,10,32,32,32,32,
-    41,4,114,64,0,0,0,114,75,0,0,0,218,8,98,117,
-    105,108,116,105,110,115,114,160,0,0,0,90,5,112,111,115,
-    105,120,250,1,47,90,2,110,116,250,1,92,99,1,0,0,
+    45,1,0,0,214,4,0,0,115,54,0,0,0,8,2,4,
+    2,2,1,10,9,2,1,10,12,2,1,10,21,2,1,10,
+    14,2,1,12,31,2,1,12,23,2,1,12,12,2,1,12,
+    15,2,1,10,8,2,1,10,10,2,1,10,6,2,1,10,
+    6,2,1,114,45,1,0,0,99,0,0,0,0,0,0,0,
+    0,0,0,0,0,0,0,0,0,3,0,0,0,64,0,0,
+    0,115,90,0,0,0,101,0,90,1,100,0,90,2,100,1,
+    90,3,100,2,100,3,132,0,90,4,100,4,100,5,132,0,
+    90,5,101,6,90,7,100,6,100,7,132,0,90,8,100,8,
+    100,9,132,0,90,9,100,19,100,11,100,12,132,1,90,10,
+    100,13,100,14,132,0,90,11,101,12,100,15,100,16,132,0,
+    131,1,90,13,100,17,100,18,132,0,90,14,100,10,83,0,
+    41,20,218,10,70,105,108,101,70,105,110,100,101,114,122,172,
+    70,105,108,101,45,98,97,115,101,100,32,102,105,110,100,101,
+    114,46,10,10,32,32,32,32,73,110,116,101,114,97,99,116,
+    105,111,110,115,32,119,105,116,104,32,116,104,101,32,102,105,
+    108,101,32,115,121,115,116,101,109,32,97,114,101,32,99,97,
+    99,104,101,100,32,102,111,114,32,112,101,114,102,111,114,109,
+    97,110,99,101,44,32,98,101,105,110,103,10,32,32,32,32,
+    114,101,102,114,101,115,104,101,100,32,119,104,101,110,32,116,
+    104,101,32,100,105,114,101,99,116,111,114,121,32,116,104,101,
+    32,102,105,110,100,101,114,32,105,115,32,104,97,110,100,108,
+    105,110,103,32,104,97,115,32,98,101,101,110,32,109,111,100,
+    105,102,105,101,100,46,10,10,32,32,32,32,99,2,0,0,
+    0,0,0,0,0,0,0,0,0,5,0,0,0,6,0,0,
+    0,7,0,0,0,115,84,0,0,0,103,0,125,3,124,2,
+    68,0,93,32,92,2,137,0,125,4,124,3,160,0,135,0,
+    102,1,100,1,100,2,132,8,124,4,68,0,131,1,161,1,
+    1,0,113,8,124,3,124,0,95,1,124,1,112,54,100,3,
+    124,0,95,2,100,4,124,0,95,3,116,4,131,0,124,0,
+    95,5,116,4,131,0,124,0,95,6,100,5,83,0,41,6,
+    122,154,73,110,105,116,105,97,108,105,122,101,32,119,105,116,
+    104,32,116,104,101,32,112,97,116,104,32,116,111,32,115,101,
+    97,114,99,104,32,111,110,32,97,110,100,32,97,32,118,97,
+    114,105,97,98,108,101,32,110,117,109,98,101,114,32,111,102,
+    10,32,32,32,32,32,32,32,32,50,45,116,117,112,108,101,
+    115,32,99,111,110,116,97,105,110,105,110,103,32,116,104,101,
+    32,108,111,97,100,101,114,32,97,110,100,32,116,104,101,32,
+    102,105,108,101,32,115,117,102,102,105,120,101,115,32,116,104,
+    101,32,108,111,97,100,101,114,10,32,32,32,32,32,32,32,
+    32,114,101,99,111,103,110,105,122,101,115,46,99,1,0,0,
     0,0,0,0,0,0,0,0,0,2,0,0,0,3,0,0,
-    0,115,0,0,0,115,26,0,0,0,124,0,93,18,125,1,
-    116,0,124,1,131,1,100,0,107,2,86,0,1,0,113,2,
-    100,1,83,0,41,2,114,39,0,0,0,78,41,1,114,22,
-    0,0,0,41,2,114,32,0,0,0,114,95,0,0,0,114,
-    3,0,0,0,114,3,0,0,0,114,6,0,0,0,114,19,
-    1,0,0,105,6,0,0,115,4,0,0,0,4,0,2,0,
-    122,25,95,115,101,116,117,112,46,60,108,111,99,97,108,115,
-    62,46,60,103,101,110,101,120,112,114,62,114,73,0,0,0,
-    122,30,105,109,112,111,114,116,108,105,98,32,114,101,113,117,
-    105,114,101,115,32,112,111,115,105,120,32,111,114,32,110,116,
-    114,2,0,0,0,114,35,0,0,0,114,31,0,0,0,114,
-    40,0,0,0,114,58,0,0,0,99,1,0,0,0,0,0,
-    0,0,0,0,0,0,2,0,0,0,4,0,0,0,83,0,
-    0,0,115,22,0,0,0,104,0,124,0,93,14,125,1,100,
-    0,124,1,155,0,157,2,146,2,113,4,83,0,41,1,114,
-    74,0,0,0,114,3,0,0,0,41,2,114,32,0,0,0,
-    218,1,115,114,3,0,0,0,114,3,0,0,0,114,6,0,
-    0,0,114,92,1,0,0,121,6,0,0,115,4,0,0,0,
-    6,0,2,0,122,25,95,115,101,116,117,112,46,60,108,111,
-    99,97,108,115,62,46,60,115,101,116,99,111,109,112,62,90,
-    7,95,116,104,114,101,97,100,90,8,95,119,101,97,107,114,
-    101,102,90,6,119,105,110,114,101,103,114,192,0,0,0,114,
-    7,0,0,0,122,4,46,112,121,119,122,6,95,100,46,112,
-    121,100,84,78,41,19,114,134,0,0,0,114,8,0,0,0,
-    114,163,0,0,0,114,31,1,0,0,114,125,0,0,0,90,
-    18,95,98,117,105,108,116,105,110,95,102,114,111,109,95,110,
-    97,109,101,114,129,0,0,0,218,3,97,108,108,114,23,0,
-    0,0,114,118,0,0,0,114,36,0,0,0,114,13,0,0,
-    0,114,21,1,0,0,114,167,0,0,0,114,103,1,0,0,
-    114,102,0,0,0,114,186,0,0,0,114,191,0,0,0,114,
-    195,0,0,0,41,12,218,17,95,98,111,111,116,115,116,114,
-    97,112,95,109,111,100,117,108,101,90,11,115,101,108,102,95,
-    109,111,100,117,108,101,90,12,98,117,105,108,116,105,110,95,
-    110,97,109,101,90,14,98,117,105,108,116,105,110,95,109,111,
-    100,117,108,101,90,10,111,115,95,100,101,116,97,105,108,115,
-    90,10,98,117,105,108,116,105,110,95,111,115,114,31,0,0,
-    0,114,35,0,0,0,90,9,111,115,95,109,111,100,117,108,
-    101,90,13,116,104,114,101,97,100,95,109,111,100,117,108,101,
-    90,14,119,101,97,107,114,101,102,95,109,111,100,117,108,101,
-    90,13,119,105,110,114,101,103,95,109,111,100,117,108,101,114,
-    3,0,0,0,114,3,0,0,0,114,6,0,0,0,218,6,
-    95,115,101,116,117,112,80,6,0,0,115,78,0,0,0,0,
-    8,4,1,6,1,6,3,10,1,8,1,10,1,12,2,10,
-    1,14,3,22,1,12,2,22,1,8,1,10,1,10,1,6,
-    2,2,1,10,1,10,1,14,1,12,2,8,1,12,1,12,
-    1,18,1,22,3,10,1,12,3,10,1,12,3,10,1,10,
-    1,12,3,14,1,14,1,10,1,10,1,10,1,114,110,1,
-    0,0,99,1,0,0,0,0,0,0,0,0,0,0,0,2,
-    0,0,0,4,0,0,0,67,0,0,0,115,50,0,0,0,
-    116,0,124,0,131,1,1,0,116,1,131,0,125,1,116,2,
-    106,3,160,4,116,5,106,6,124,1,142,0,103,1,161,1,
-    1,0,116,2,106,7,160,8,116,9,161,1,1,0,100,1,
-    83,0,41,2,122,41,73,110,115,116,97,108,108,32,116,104,
-    101,32,112,97,116,104,45,98,97,115,101,100,32,105,109,112,
-    111,114,116,32,99,111,109,112,111,110,101,110,116,115,46,78,
-    41,10,114,110,1,0,0,114,184,0,0,0,114,8,0,0,
-    0,114,51,1,0,0,114,167,0,0,0,114,83,1,0,0,
-    114,98,1,0,0,218,9,109,101,116,97,95,112,97,116,104,
-    114,186,0,0,0,114,45,1,0,0,41,2,114,109,1,0,
-    0,90,17,115,117,112,112,111,114,116,101,100,95,108,111,97,
-    100,101,114,115,114,3,0,0,0,114,3,0,0,0,114,6,
-    0,0,0,218,8,95,105,110,115,116,97,108,108,145,6,0,
-    0,115,8,0,0,0,0,2,8,1,6,1,20,1,114,112,
-    1,0,0,41,1,114,60,0,0,0,41,1,78,41,3,78,
-    78,78,41,2,114,73,0,0,0,114,73,0,0,0,41,1,
-    84,41,1,78,41,1,78,41,63,114,127,0,0,0,114,12,
-    0,0,0,90,37,95,67,65,83,69,95,73,78,83,69,78,
-    83,73,84,73,86,69,95,80,76,65,84,70,79,82,77,83,
-    95,66,89,84,69,83,95,75,69,89,114,11,0,0,0,114,
-    13,0,0,0,114,20,0,0,0,114,27,0,0,0,114,29,
-    0,0,0,114,38,0,0,0,114,47,0,0,0,114,49,0,
-    0,0,114,53,0,0,0,114,54,0,0,0,114,56,0,0,
-    0,114,59,0,0,0,114,69,0,0,0,218,4,116,121,112,
-    101,218,8,95,95,99,111,100,101,95,95,114,162,0,0,0,
-    114,18,0,0,0,114,148,0,0,0,114,17,0,0,0,114,
-    24,0,0,0,114,236,0,0,0,114,92,0,0,0,114,88,
-    0,0,0,114,102,0,0,0,114,89,0,0,0,90,23,68,
-    69,66,85,71,95,66,89,84,69,67,79,68,69,95,83,85,
-    70,70,73,88,69,83,90,27,79,80,84,73,77,73,90,69,
-    68,95,66,89,84,69,67,79,68,69,95,83,85,70,70,73,
-    88,69,83,114,98,0,0,0,114,103,0,0,0,114,109,0,
-    0,0,114,113,0,0,0,114,115,0,0,0,114,136,0,0,
-    0,114,143,0,0,0,114,152,0,0,0,114,156,0,0,0,
-    114,158,0,0,0,114,165,0,0,0,114,170,0,0,0,114,
-    171,0,0,0,114,176,0,0,0,218,6,111,98,106,101,99,
-    116,114,185,0,0,0,114,190,0,0,0,114,191,0,0,0,
-    114,208,0,0,0,114,221,0,0,0,114,239,0,0,0,114,
-    9,1,0,0,114,15,1,0,0,114,21,1,0,0,114,252,
-    0,0,0,114,22,1,0,0,114,43,1,0,0,114,45,1,
-    0,0,114,83,1,0,0,114,102,1,0,0,114,184,0,0,
-    0,114,110,1,0,0,114,112,1,0,0,114,3,0,0,0,
-    114,3,0,0,0,114,3,0,0,0,114,6,0,0,0,218,
-    8,60,109,111,100,117,108,101,62,1,0,0,0,115,126,0,
-    0,0,4,22,4,1,4,1,2,1,2,255,4,4,8,17,
-    8,5,8,5,8,6,8,6,8,12,8,10,8,9,8,5,
-    8,7,8,9,10,22,10,127,0,13,16,1,12,2,4,1,
-    4,2,6,2,6,2,8,2,16,71,8,40,8,19,8,12,
-    8,12,8,28,8,17,8,33,8,28,8,24,10,13,10,10,
-    10,11,8,14,6,3,4,1,2,255,12,68,14,64,14,29,
-    16,127,0,17,14,72,18,45,18,26,4,3,18,53,14,63,
-    14,42,14,127,0,68,14,127,0,22,10,23,8,11,8,65,
+    0,51,0,0,0,115,22,0,0,0,124,0,93,14,125,1,
+    124,1,136,0,102,2,86,0,1,0,113,2,100,0,83,0,
+    114,110,0,0,0,114,3,0,0,0,114,16,1,0,0,169,
+    1,114,140,0,0,0,114,3,0,0,0,114,6,0,0,0,
+    114,19,1,0,0,166,5,0,0,115,4,0,0,0,4,0,
+    2,0,122,38,70,105,108,101,70,105,110,100,101,114,46,95,
+    95,105,110,105,116,95,95,46,60,108,111,99,97,108,115,62,
+    46,60,103,101,110,101,120,112,114,62,114,71,0,0,0,114,
+    105,0,0,0,78,41,7,114,167,0,0,0,218,8,95,108,
+    111,97,100,101,114,115,114,44,0,0,0,218,11,95,112,97,
+    116,104,95,109,116,105,109,101,218,3,115,101,116,218,11,95,
+    112,97,116,104,95,99,97,99,104,101,218,19,95,114,101,108,
+    97,120,101,100,95,112,97,116,104,95,99,97,99,104,101,41,
+    5,114,119,0,0,0,114,44,0,0,0,218,14,108,111,97,
+    100,101,114,95,100,101,116,97,105,108,115,90,7,108,111,97,
+    100,101,114,115,114,189,0,0,0,114,3,0,0,0,114,85,
+    1,0,0,114,6,0,0,0,114,209,0,0,0,160,5,0,
+    0,115,16,0,0,0,0,4,4,1,12,1,26,1,6,2,
+    10,1,6,1,8,1,122,19,70,105,108,101,70,105,110,100,
+    101,114,46,95,95,105,110,105,116,95,95,99,1,0,0,0,
+    0,0,0,0,0,0,0,0,1,0,0,0,2,0,0,0,
+    67,0,0,0,115,10,0,0,0,100,1,124,0,95,0,100,
+    2,83,0,41,3,122,31,73,110,118,97,108,105,100,97,116,
+    101,32,116,104,101,32,100,105,114,101,99,116,111,114,121,32,
+    109,116,105,109,101,46,114,105,0,0,0,78,41,1,114,87,
+    1,0,0,114,246,0,0,0,114,3,0,0,0,114,3,0,
+    0,0,114,6,0,0,0,114,46,1,0,0,174,5,0,0,
+    115,2,0,0,0,0,2,122,28,70,105,108,101,70,105,110,
+    100,101,114,46,105,110,118,97,108,105,100,97,116,101,95,99,
+    97,99,104,101,115,99,2,0,0,0,0,0,0,0,0,0,
+    0,0,3,0,0,0,3,0,0,0,67,0,0,0,115,42,
+    0,0,0,124,0,160,0,124,1,161,1,125,2,124,2,100,
+    1,107,8,114,26,100,1,103,0,102,2,83,0,124,2,106,
+    1,124,2,106,2,112,38,103,0,102,2,83,0,41,2,122,
+    197,84,114,121,32,116,111,32,102,105,110,100,32,97,32,108,
+    111,97,100,101,114,32,102,111,114,32,116,104,101,32,115,112,
+    101,99,105,102,105,101,100,32,109,111,100,117,108,101,44,32,
+    111,114,32,116,104,101,32,110,97,109,101,115,112,97,99,101,
+    10,32,32,32,32,32,32,32,32,112,97,99,107,97,103,101,
+    32,112,111,114,116,105,111,110,115,46,32,82,101,116,117,114,
+    110,115,32,40,108,111,97,100,101,114,44,32,108,105,115,116,
+    45,111,102,45,112,111,114,116,105,111,110,115,41,46,10,10,
+    32,32,32,32,32,32,32,32,84,104,105,115,32,109,101,116,
+    104,111,100,32,105,115,32,100,101,112,114,101,99,97,116,101,
+    100,46,32,32,85,115,101,32,102,105,110,100,95,115,112,101,
+    99,40,41,32,105,110,115,116,101,97,100,46,10,10,32,32,
+    32,32,32,32,32,32,78,41,3,114,203,0,0,0,114,140,
+    0,0,0,114,178,0,0,0,41,3,114,119,0,0,0,114,
+    139,0,0,0,114,187,0,0,0,114,3,0,0,0,114,3,
+    0,0,0,114,6,0,0,0,114,137,0,0,0,180,5,0,
+    0,115,8,0,0,0,0,7,10,1,8,1,8,1,122,22,
+    70,105,108,101,70,105,110,100,101,114,46,102,105,110,100,95,
+    108,111,97,100,101,114,99,6,0,0,0,0,0,0,0,0,
+    0,0,0,7,0,0,0,6,0,0,0,67,0,0,0,115,
+    26,0,0,0,124,1,124,2,124,3,131,2,125,6,116,0,
+    124,2,124,3,124,6,124,4,100,1,141,4,83,0,41,2,
+    78,114,177,0,0,0,41,1,114,190,0,0,0,41,7,114,
+    119,0,0,0,114,188,0,0,0,114,139,0,0,0,114,44,
+    0,0,0,90,4,115,109,115,108,114,202,0,0,0,114,140,
+    0,0,0,114,3,0,0,0,114,3,0,0,0,114,6,0,
+    0,0,114,58,1,0,0,192,5,0,0,115,8,0,0,0,
+    0,1,10,1,8,1,2,255,122,20,70,105,108,101,70,105,
+    110,100,101,114,46,95,103,101,116,95,115,112,101,99,78,99,
+    3,0,0,0,0,0,0,0,0,0,0,0,14,0,0,0,
+    8,0,0,0,67,0,0,0,115,98,1,0,0,100,1,125,
+    3,124,1,160,0,100,2,161,1,100,3,25,0,125,4,122,
+    24,116,1,124,0,106,2,112,34,116,3,160,4,161,0,131,
+    1,106,5,125,5,87,0,110,24,4,0,116,6,107,10,114,
+    66,1,0,1,0,1,0,100,4,125,5,89,0,110,2,88,
+    0,124,5,124,0,106,7,107,3,114,92,124,0,160,8,161,
+    0,1,0,124,5,124,0,95,7,116,9,131,0,114,114,124,
+    0,106,10,125,6,124,4,160,11,161,0,125,7,110,10,124,
+    0,106,12,125,6,124,4,125,7,124,7,124,6,107,6,114,
+    218,116,13,124,0,106,2,124,4,131,2,125,8,124,0,106,
+    14,68,0,93,58,92,2,125,9,125,10,100,5,124,9,23,
+    0,125,11,116,13,124,8,124,11,131,2,125,12,116,15,124,
+    12,131,1,114,150,124,0,160,16,124,10,124,1,124,12,124,
+    8,103,1,124,2,161,5,2,0,1,0,83,0,113,150,116,
+    17,124,8,131,1,125,3,124,0,106,14,68,0,93,82,92,
+    2,125,9,125,10,116,13,124,0,106,2,124,4,124,9,23,
+    0,131,2,125,12,116,18,106,19,100,6,124,12,100,3,100,
+    7,141,3,1,0,124,7,124,9,23,0,124,6,107,6,114,
+    224,116,15,124,12,131,1,114,224,124,0,160,16,124,10,124,
+    1,124,12,100,8,124,2,161,5,2,0,1,0,83,0,113,
+    224,124,3,144,1,114,94,116,18,160,19,100,9,124,8,161,
+    2,1,0,116,18,160,20,124,1,100,8,161,2,125,13,124,
+    8,103,1,124,13,95,21,124,13,83,0,100,8,83,0,41,
+    10,122,111,84,114,121,32,116,111,32,102,105,110,100,32,97,
+    32,115,112,101,99,32,102,111,114,32,116,104,101,32,115,112,
+    101,99,105,102,105,101,100,32,109,111,100,117,108,101,46,10,
+    10,32,32,32,32,32,32,32,32,82,101,116,117,114,110,115,
+    32,116,104,101,32,109,97,116,99,104,105,110,103,32,115,112,
+    101,99,44,32,111,114,32,78,111,110,101,32,105,102,32,110,
+    111,116,32,102,111,117,110,100,46,10,32,32,32,32,32,32,
+    32,32,70,114,71,0,0,0,114,28,0,0,0,114,105,0,
+    0,0,114,209,0,0,0,122,9,116,114,121,105,110,103,32,
+    123,125,41,1,90,9,118,101,114,98,111,115,105,116,121,78,
+    122,25,112,111,115,115,105,98,108,101,32,110,97,109,101,115,
+    112,97,99,101,32,102,111,114,32,123,125,41,22,114,41,0,
+    0,0,114,49,0,0,0,114,44,0,0,0,114,2,0,0,
+    0,114,55,0,0,0,114,10,1,0,0,114,50,0,0,0,
+    114,87,1,0,0,218,11,95,102,105,108,108,95,99,97,99,
+    104,101,114,7,0,0,0,114,90,1,0,0,114,106,0,0,
+    0,114,89,1,0,0,114,38,0,0,0,114,86,1,0,0,
+    114,54,0,0,0,114,58,1,0,0,114,56,0,0,0,114,
+    134,0,0,0,114,149,0,0,0,114,183,0,0,0,114,178,
+    0,0,0,41,14,114,119,0,0,0,114,139,0,0,0,114,
+    202,0,0,0,90,12,105,115,95,110,97,109,101,115,112,97,
+    99,101,90,11,116,97,105,108,95,109,111,100,117,108,101,114,
+    169,0,0,0,90,5,99,97,99,104,101,90,12,99,97,99,
+    104,101,95,109,111,100,117,108,101,90,9,98,97,115,101,95,
+    112,97,116,104,114,17,1,0,0,114,188,0,0,0,90,13,
+    105,110,105,116,95,102,105,108,101,110,97,109,101,90,9,102,
+    117,108,108,95,112,97,116,104,114,187,0,0,0,114,3,0,
+    0,0,114,3,0,0,0,114,6,0,0,0,114,203,0,0,
+    0,197,5,0,0,115,74,0,0,0,0,5,4,1,14,1,
+    2,1,24,1,14,1,10,1,10,1,8,1,6,2,6,1,
+    6,1,10,2,6,1,4,2,8,1,12,1,14,1,8,1,
+    10,1,8,1,26,4,8,2,14,1,16,1,16,1,12,1,
+    8,1,10,1,2,0,2,255,10,2,6,1,12,1,12,1,
+    8,1,4,1,122,20,70,105,108,101,70,105,110,100,101,114,
+    46,102,105,110,100,95,115,112,101,99,99,1,0,0,0,0,
+    0,0,0,0,0,0,0,9,0,0,0,10,0,0,0,67,
+    0,0,0,115,190,0,0,0,124,0,106,0,125,1,122,22,
+    116,1,160,2,124,1,112,22,116,1,160,3,161,0,161,1,
+    125,2,87,0,110,30,4,0,116,4,116,5,116,6,102,3,
+    107,10,114,58,1,0,1,0,1,0,103,0,125,2,89,0,
+    110,2,88,0,116,7,106,8,160,9,100,1,161,1,115,84,
+    116,10,124,2,131,1,124,0,95,11,110,74,116,10,131,0,
+    125,3,124,2,68,0,93,56,125,4,124,4,160,12,100,2,
+    161,1,92,3,125,5,125,6,125,7,124,6,114,136,100,3,
+    160,13,124,5,124,7,160,14,161,0,161,2,125,8,110,4,
+    124,5,125,8,124,3,160,15,124,8,161,1,1,0,113,94,
+    124,3,124,0,95,11,116,7,106,8,160,9,116,16,161,1,
+    114,186,100,4,100,5,132,0,124,2,68,0,131,1,124,0,
+    95,17,100,6,83,0,41,7,122,68,70,105,108,108,32,116,
+    104,101,32,99,97,99,104,101,32,111,102,32,112,111,116,101,
+    110,116,105,97,108,32,109,111,100,117,108,101,115,32,97,110,
+    100,32,112,97,99,107,97,103,101,115,32,102,111,114,32,116,
+    104,105,115,32,100,105,114,101,99,116,111,114,121,46,114,0,
+    0,0,0,114,71,0,0,0,114,61,0,0,0,99,1,0,
+    0,0,0,0,0,0,0,0,0,0,2,0,0,0,4,0,
+    0,0,83,0,0,0,115,20,0,0,0,104,0,124,0,93,
+    12,125,1,124,1,160,0,161,0,146,2,113,4,83,0,114,
+    3,0,0,0,41,1,114,106,0,0,0,41,2,114,32,0,
+    0,0,90,2,102,110,114,3,0,0,0,114,3,0,0,0,
+    114,6,0,0,0,218,9,60,115,101,116,99,111,109,112,62,
+    18,6,0,0,115,4,0,0,0,6,0,2,0,122,41,70,
+    105,108,101,70,105,110,100,101,114,46,95,102,105,108,108,95,
+    99,97,99,104,101,46,60,108,111,99,97,108,115,62,46,60,
+    115,101,116,99,111,109,112,62,78,41,18,114,44,0,0,0,
+    114,2,0,0,0,114,7,1,0,0,114,55,0,0,0,114,
+    3,1,0,0,218,15,80,101,114,109,105,115,115,105,111,110,
+    69,114,114,111,114,218,18,78,111,116,65,68,105,114,101,99,
+    116,111,114,121,69,114,114,111,114,114,8,0,0,0,114,9,
+    0,0,0,114,10,0,0,0,114,88,1,0,0,114,89,1,
+    0,0,114,101,0,0,0,114,62,0,0,0,114,106,0,0,
+    0,218,3,97,100,100,114,11,0,0,0,114,90,1,0,0,
+    41,9,114,119,0,0,0,114,44,0,0,0,114,8,1,0,
+    0,90,21,108,111,119,101,114,95,115,117,102,102,105,120,95,
+    99,111,110,116,101,110,116,115,114,41,1,0,0,114,117,0,
+    0,0,114,29,1,0,0,114,17,1,0,0,90,8,110,101,
+    119,95,110,97,109,101,114,3,0,0,0,114,3,0,0,0,
+    114,6,0,0,0,114,92,1,0,0,245,5,0,0,115,34,
+    0,0,0,0,2,6,1,2,1,22,1,20,3,10,3,12,
+    1,12,7,6,1,8,1,16,1,4,1,18,2,4,1,12,
+    1,6,1,12,1,122,22,70,105,108,101,70,105,110,100,101,
+    114,46,95,102,105,108,108,95,99,97,99,104,101,99,1,0,
+    0,0,0,0,0,0,0,0,0,0,3,0,0,0,3,0,
+    0,0,7,0,0,0,115,18,0,0,0,135,0,135,1,102,
+    2,100,1,100,2,132,8,125,2,124,2,83,0,41,3,97,
+    20,1,0,0,65,32,99,108,97,115,115,32,109,101,116,104,
+    111,100,32,119,104,105,99,104,32,114,101,116,117,114,110,115,
+    32,97,32,99,108,111,115,117,114,101,32,116,111,32,117,115,
+    101,32,111,110,32,115,121,115,46,112,97,116,104,95,104,111,
+    111,107,10,32,32,32,32,32,32,32,32,119,104,105,99,104,
+    32,119,105,108,108,32,114,101,116,117,114,110,32,97,110,32,
+    105,110,115,116,97,110,99,101,32,117,115,105,110,103,32,116,
+    104,101,32,115,112,101,99,105,102,105,101,100,32,108,111,97,
+    100,101,114,115,32,97,110,100,32,116,104,101,32,112,97,116,
+    104,10,32,32,32,32,32,32,32,32,99,97,108,108,101,100,
+    32,111,110,32,116,104,101,32,99,108,111,115,117,114,101,46,
+    10,10,32,32,32,32,32,32,32,32,73,102,32,116,104,101,
+    32,112,97,116,104,32,99,97,108,108,101,100,32,111,110,32,
+    116,104,101,32,99,108,111,115,117,114,101,32,105,115,32,110,
+    111,116,32,97,32,100,105,114,101,99,116,111,114,121,44,32,
+    73,109,112,111,114,116,69,114,114,111,114,32,105,115,10,32,
+    32,32,32,32,32,32,32,114,97,105,115,101,100,46,10,10,
+    32,32,32,32,32,32,32,32,99,1,0,0,0,0,0,0,
+    0,0,0,0,0,1,0,0,0,4,0,0,0,19,0,0,
+    0,115,34,0,0,0,116,0,124,0,131,1,115,20,116,1,
+    100,1,124,0,100,2,141,2,130,1,136,0,124,0,102,1,
+    136,1,158,2,142,0,83,0,41,3,122,45,80,97,116,104,
+    32,104,111,111,107,32,102,111,114,32,105,109,112,111,114,116,
+    108,105,98,46,109,97,99,104,105,110,101,114,121,46,70,105,
+    108,101,70,105,110,100,101,114,46,122,30,111,110,108,121,32,
+    100,105,114,101,99,116,111,114,105,101,115,32,97,114,101,32,
+    115,117,112,112,111,114,116,101,100,114,48,0,0,0,41,2,
+    114,56,0,0,0,114,118,0,0,0,114,48,0,0,0,169,
+    2,114,193,0,0,0,114,91,1,0,0,114,3,0,0,0,
+    114,6,0,0,0,218,24,112,97,116,104,95,104,111,111,107,
+    95,102,111,114,95,70,105,108,101,70,105,110,100,101,114,30,
+    6,0,0,115,6,0,0,0,0,2,8,1,12,1,122,54,
+    70,105,108,101,70,105,110,100,101,114,46,112,97,116,104,95,
+    104,111,111,107,46,60,108,111,99,97,108,115,62,46,112,97,
+    116,104,95,104,111,111,107,95,102,111,114,95,70,105,108,101,
+    70,105,110,100,101,114,114,3,0,0,0,41,3,114,193,0,
+    0,0,114,91,1,0,0,114,98,1,0,0,114,3,0,0,
+    0,114,97,1,0,0,114,6,0,0,0,218,9,112,97,116,
+    104,95,104,111,111,107,20,6,0,0,115,4,0,0,0,0,
+    10,14,6,122,20,70,105,108,101,70,105,110,100,101,114,46,
+    112,97,116,104,95,104,111,111,107,99,1,0,0,0,0,0,
+    0,0,0,0,0,0,1,0,0,0,3,0,0,0,67,0,
+    0,0,115,12,0,0,0,100,1,160,0,124,0,106,1,161,
+    1,83,0,41,2,78,122,16,70,105,108,101,70,105,110,100,
+    101,114,40,123,33,114,125,41,41,2,114,62,0,0,0,114,
+    44,0,0,0,114,246,0,0,0,114,3,0,0,0,114,3,
+    0,0,0,114,6,0,0,0,114,39,1,0,0,38,6,0,
+    0,115,2,0,0,0,0,1,122,19,70,105,108,101,70,105,
+    110,100,101,114,46,95,95,114,101,112,114,95,95,41,1,78,
+    41,15,114,125,0,0,0,114,124,0,0,0,114,126,0,0,
+    0,114,127,0,0,0,114,209,0,0,0,114,46,1,0,0,
+    114,143,0,0,0,114,206,0,0,0,114,137,0,0,0,114,
+    58,1,0,0,114,203,0,0,0,114,92,1,0,0,114,207,
+    0,0,0,114,99,1,0,0,114,39,1,0,0,114,3,0,
+    0,0,114,3,0,0,0,114,3,0,0,0,114,6,0,0,
+    0,114,84,1,0,0,151,5,0,0,115,22,0,0,0,8,
+    2,4,7,8,14,8,4,4,2,8,12,8,5,10,48,8,
+    31,2,1,10,17,114,84,1,0,0,99,4,0,0,0,0,
+    0,0,0,0,0,0,0,6,0,0,0,8,0,0,0,67,
+    0,0,0,115,146,0,0,0,124,0,160,0,100,1,161,1,
+    125,4,124,0,160,0,100,2,161,1,125,5,124,4,115,66,
+    124,5,114,36,124,5,106,1,125,4,110,30,124,2,124,3,
+    107,2,114,56,116,2,124,1,124,2,131,2,125,4,110,10,
+    116,3,124,1,124,2,131,2,125,4,124,5,115,84,116,4,
+    124,1,124,2,124,4,100,3,141,3,125,5,122,36,124,5,
+    124,0,100,2,60,0,124,4,124,0,100,1,60,0,124,2,
+    124,0,100,4,60,0,124,3,124,0,100,5,60,0,87,0,
+    110,20,4,0,116,5,107,10,114,140,1,0,1,0,1,0,
+    89,0,110,2,88,0,100,0,83,0,41,6,78,218,10,95,
+    95,108,111,97,100,101,114,95,95,218,8,95,95,115,112,101,
+    99,95,95,114,85,1,0,0,90,8,95,95,102,105,108,101,
+    95,95,90,10,95,95,99,97,99,104,101,100,95,95,41,6,
+    218,3,103,101,116,114,140,0,0,0,114,15,1,0,0,114,
+    9,1,0,0,114,190,0,0,0,114,72,1,0,0,41,6,
+    90,2,110,115,114,117,0,0,0,90,8,112,97,116,104,110,
+    97,109,101,90,9,99,112,97,116,104,110,97,109,101,114,140,
+    0,0,0,114,187,0,0,0,114,3,0,0,0,114,3,0,
+    0,0,114,6,0,0,0,218,14,95,102,105,120,95,117,112,
+    95,109,111,100,117,108,101,44,6,0,0,115,34,0,0,0,
+    0,2,10,1,10,1,4,1,4,1,8,1,8,1,12,2,
+    10,1,4,1,14,1,2,1,8,1,8,1,8,1,12,1,
+    14,2,114,103,1,0,0,99,0,0,0,0,0,0,0,0,
+    0,0,0,0,3,0,0,0,3,0,0,0,67,0,0,0,
+    115,38,0,0,0,116,0,116,1,160,2,161,0,102,2,125,
+    0,116,3,116,4,102,2,125,1,116,5,116,6,102,2,125,
+    2,124,0,124,1,124,2,103,3,83,0,41,1,122,95,82,
+    101,116,117,114,110,115,32,97,32,108,105,115,116,32,111,102,
+    32,102,105,108,101,45,98,97,115,101,100,32,109,111,100,117,
+    108,101,32,108,111,97,100,101,114,115,46,10,10,32,32,32,
+    32,69,97,99,104,32,105,116,101,109,32,105,115,32,97,32,
+    116,117,112,108,101,32,40,108,111,97,100,101,114,44,32,115,
+    117,102,102,105,120,101,115,41,46,10,32,32,32,32,41,7,
+    114,252,0,0,0,114,163,0,0,0,218,18,101,120,116,101,
+    110,115,105,111,110,95,115,117,102,102,105,120,101,115,114,9,
+    1,0,0,114,102,0,0,0,114,15,1,0,0,114,89,0,
+    0,0,41,3,90,10,101,120,116,101,110,115,105,111,110,115,
+    90,6,115,111,117,114,99,101,90,8,98,121,116,101,99,111,
+    100,101,114,3,0,0,0,114,3,0,0,0,114,6,0,0,
+    0,114,184,0,0,0,67,6,0,0,115,8,0,0,0,0,
+    5,12,1,8,1,8,1,114,184,0,0,0,99,1,0,0,
+    0,0,0,0,0,0,0,0,0,12,0,0,0,9,0,0,
+    0,67,0,0,0,115,178,1,0,0,124,0,97,0,116,0,
+    106,1,97,1,116,0,106,2,97,2,116,1,106,3,116,4,
+    25,0,125,1,100,1,68,0,93,48,125,2,124,2,116,1,
+    106,3,107,7,114,56,116,0,160,5,124,2,161,1,125,3,
+    110,10,116,1,106,3,124,2,25,0,125,3,116,6,124,1,
+    124,2,124,3,131,3,1,0,113,30,100,2,100,3,103,1,
+    102,2,100,4,100,5,100,3,103,2,102,2,102,2,125,4,
+    124,4,68,0,93,110,92,2,125,5,125,6,116,7,100,6,
+    100,7,132,0,124,6,68,0,131,1,131,1,115,136,116,8,
+    130,1,124,6,100,8,25,0,125,7,124,5,116,1,106,3,
+    107,6,114,170,116,1,106,3,124,5,25,0,125,8,1,0,
+    113,226,113,106,122,20,116,0,160,5,124,5,161,1,125,8,
+    87,0,1,0,113,226,87,0,113,106,4,0,116,9,107,10,
+    114,214,1,0,1,0,1,0,89,0,113,106,89,0,113,106,
+    88,0,113,106,116,9,100,9,131,1,130,1,116,6,124,1,
+    100,10,124,8,131,3,1,0,116,6,124,1,100,11,124,7,
+    131,3,1,0,116,6,124,1,100,12,100,13,160,10,124,6,
+    161,1,131,3,1,0,116,6,124,1,100,14,100,15,100,16,
+    132,0,124,6,68,0,131,1,131,3,1,0,116,0,160,5,
+    100,17,161,1,125,9,116,6,124,1,100,17,124,9,131,3,
+    1,0,116,0,160,5,100,18,161,1,125,10,116,6,124,1,
+    100,18,124,10,131,3,1,0,124,5,100,4,107,2,144,1,
+    114,110,116,0,160,5,100,19,161,1,125,11,116,6,124,1,
+    100,20,124,11,131,3,1,0,116,6,124,1,100,21,116,11,
+    131,0,131,3,1,0,116,12,160,13,116,2,160,14,161,0,
+    161,1,1,0,124,5,100,4,107,2,144,1,114,174,116,15,
+    160,16,100,22,161,1,1,0,100,23,116,12,107,6,144,1,
+    114,174,100,24,116,17,95,18,100,25,83,0,41,26,122,205,
+    83,101,116,117,112,32,116,104,101,32,112,97,116,104,45,98,
+    97,115,101,100,32,105,109,112,111,114,116,101,114,115,32,102,
+    111,114,32,105,109,112,111,114,116,108,105,98,32,98,121,32,
+    105,109,112,111,114,116,105,110,103,32,110,101,101,100,101,100,
+    10,32,32,32,32,98,117,105,108,116,45,105,110,32,109,111,
+    100,117,108,101,115,32,97,110,100,32,105,110,106,101,99,116,
+    105,110,103,32,116,104,101,109,32,105,110,116,111,32,116,104,
+    101,32,103,108,111,98,97,108,32,110,97,109,101,115,112,97,
+    99,101,46,10,10,32,32,32,32,79,116,104,101,114,32,99,
+    111,109,112,111,110,101,110,116,115,32,97,114,101,32,101,120,
+    116,114,97,99,116,101,100,32,102,114,111,109,32,116,104,101,
+    32,99,111,114,101,32,98,111,111,116,115,116,114,97,112,32,
+    109,111,100,117,108,101,46,10,10,32,32,32,32,41,4,114,
+    64,0,0,0,114,75,0,0,0,218,8,98,117,105,108,116,
+    105,110,115,114,160,0,0,0,90,5,112,111,115,105,120,250,
+    1,47,90,2,110,116,250,1,92,99,1,0,0,0,0,0,
+    0,0,0,0,0,0,2,0,0,0,3,0,0,0,115,0,
+    0,0,115,26,0,0,0,124,0,93,18,125,1,116,0,124,
+    1,131,1,100,0,107,2,86,0,1,0,113,2,100,1,83,
+    0,41,2,114,39,0,0,0,78,41,1,114,22,0,0,0,
+    41,2,114,32,0,0,0,114,95,0,0,0,114,3,0,0,
+    0,114,3,0,0,0,114,6,0,0,0,114,19,1,0,0,
+    103,6,0,0,115,4,0,0,0,4,0,2,0,122,25,95,
+    115,101,116,117,112,46,60,108,111,99,97,108,115,62,46,60,
+    103,101,110,101,120,112,114,62,114,73,0,0,0,122,30,105,
+    109,112,111,114,116,108,105,98,32,114,101,113,117,105,114,101,
+    115,32,112,111,115,105,120,32,111,114,32,110,116,114,2,0,
+    0,0,114,35,0,0,0,114,31,0,0,0,114,40,0,0,
+    0,114,58,0,0,0,99,1,0,0,0,0,0,0,0,0,
+    0,0,0,2,0,0,0,4,0,0,0,83,0,0,0,115,
+    22,0,0,0,104,0,124,0,93,14,125,1,100,0,124,1,
+    155,0,157,2,146,2,113,4,83,0,41,1,114,74,0,0,
+    0,114,3,0,0,0,41,2,114,32,0,0,0,218,1,115,
+    114,3,0,0,0,114,3,0,0,0,114,6,0,0,0,114,
+    93,1,0,0,119,6,0,0,115,4,0,0,0,6,0,2,
+    0,122,25,95,115,101,116,117,112,46,60,108,111,99,97,108,
+    115,62,46,60,115,101,116,99,111,109,112,62,90,7,95,116,
+    104,114,101,97,100,90,8,95,119,101,97,107,114,101,102,90,
+    6,119,105,110,114,101,103,114,192,0,0,0,114,7,0,0,
+    0,122,4,46,112,121,119,122,6,95,100,46,112,121,100,84,
+    78,41,19,114,134,0,0,0,114,8,0,0,0,114,163,0,
+    0,0,114,31,1,0,0,114,125,0,0,0,90,18,95,98,
+    117,105,108,116,105,110,95,102,114,111,109,95,110,97,109,101,
+    114,129,0,0,0,218,3,97,108,108,114,23,0,0,0,114,
+    118,0,0,0,114,36,0,0,0,114,13,0,0,0,114,21,
+    1,0,0,114,167,0,0,0,114,104,1,0,0,114,102,0,
+    0,0,114,186,0,0,0,114,191,0,0,0,114,195,0,0,
+    0,41,12,218,17,95,98,111,111,116,115,116,114,97,112,95,
+    109,111,100,117,108,101,90,11,115,101,108,102,95,109,111,100,
+    117,108,101,90,12,98,117,105,108,116,105,110,95,110,97,109,
+    101,90,14,98,117,105,108,116,105,110,95,109,111,100,117,108,
+    101,90,10,111,115,95,100,101,116,97,105,108,115,90,10,98,
+    117,105,108,116,105,110,95,111,115,114,31,0,0,0,114,35,
+    0,0,0,90,9,111,115,95,109,111,100,117,108,101,90,13,
+    116,104,114,101,97,100,95,109,111,100,117,108,101,90,14,119,
+    101,97,107,114,101,102,95,109,111,100,117,108,101,90,13,119,
+    105,110,114,101,103,95,109,111,100,117,108,101,114,3,0,0,
+    0,114,3,0,0,0,114,6,0,0,0,218,6,95,115,101,
+    116,117,112,78,6,0,0,115,78,0,0,0,0,8,4,1,
+    6,1,6,3,10,1,8,1,10,1,12,2,10,1,14,3,
+    22,1,12,2,22,1,8,1,10,1,10,1,6,2,2,1,
+    10,1,10,1,14,1,12,2,8,1,12,1,12,1,18,1,
+    22,3,10,1,12,3,10,1,12,3,10,1,10,1,12,3,
+    14,1,14,1,10,1,10,1,10,1,114,111,1,0,0,99,
+    1,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,
+    4,0,0,0,67,0,0,0,115,50,0,0,0,116,0,124,
+    0,131,1,1,0,116,1,131,0,125,1,116,2,106,3,160,
+    4,116,5,106,6,124,1,142,0,103,1,161,1,1,0,116,
+    2,106,7,160,8,116,9,161,1,1,0,100,1,83,0,41,
+    2,122,41,73,110,115,116,97,108,108,32,116,104,101,32,112,
+    97,116,104,45,98,97,115,101,100,32,105,109,112,111,114,116,
+    32,99,111,109,112,111,110,101,110,116,115,46,78,41,10,114,
+    111,1,0,0,114,184,0,0,0,114,8,0,0,0,114,51,
+    1,0,0,114,167,0,0,0,114,84,1,0,0,114,99,1,
+    0,0,218,9,109,101,116,97,95,112,97,116,104,114,186,0,
+    0,0,114,45,1,0,0,41,2,114,110,1,0,0,90,17,
+    115,117,112,112,111,114,116,101,100,95,108,111,97,100,101,114,
+    115,114,3,0,0,0,114,3,0,0,0,114,6,0,0,0,
+    218,8,95,105,110,115,116,97,108,108,143,6,0,0,115,8,
+    0,0,0,0,2,8,1,6,1,20,1,114,113,1,0,0,
+    41,1,114,60,0,0,0,41,1,78,41,3,78,78,78,41,
+    2,114,73,0,0,0,114,73,0,0,0,41,1,84,41,1,
+    78,41,1,78,41,63,114,127,0,0,0,114,12,0,0,0,
+    90,37,95,67,65,83,69,95,73,78,83,69,78,83,73,84,
+    73,86,69,95,80,76,65,84,70,79,82,77,83,95,66,89,
+    84,69,83,95,75,69,89,114,11,0,0,0,114,13,0,0,
+    0,114,20,0,0,0,114,27,0,0,0,114,29,0,0,0,
+    114,38,0,0,0,114,47,0,0,0,114,49,0,0,0,114,
+    53,0,0,0,114,54,0,0,0,114,56,0,0,0,114,59,
+    0,0,0,114,69,0,0,0,218,4,116,121,112,101,218,8,
+    95,95,99,111,100,101,95,95,114,162,0,0,0,114,18,0,
+    0,0,114,148,0,0,0,114,17,0,0,0,114,24,0,0,
+    0,114,236,0,0,0,114,92,0,0,0,114,88,0,0,0,
+    114,102,0,0,0,114,89,0,0,0,90,23,68,69,66,85,
+    71,95,66,89,84,69,67,79,68,69,95,83,85,70,70,73,
+    88,69,83,90,27,79,80,84,73,77,73,90,69,68,95,66,
+    89,84,69,67,79,68,69,95,83,85,70,70,73,88,69,83,
+    114,98,0,0,0,114,103,0,0,0,114,109,0,0,0,114,
+    113,0,0,0,114,115,0,0,0,114,136,0,0,0,114,143,
+    0,0,0,114,152,0,0,0,114,156,0,0,0,114,158,0,
+    0,0,114,165,0,0,0,114,170,0,0,0,114,171,0,0,
+    0,114,176,0,0,0,218,6,111,98,106,101,99,116,114,185,
+    0,0,0,114,190,0,0,0,114,191,0,0,0,114,208,0,
+    0,0,114,221,0,0,0,114,239,0,0,0,114,9,1,0,
+    0,114,15,1,0,0,114,21,1,0,0,114,252,0,0,0,
+    114,22,1,0,0,114,43,1,0,0,114,45,1,0,0,114,
+    84,1,0,0,114,103,1,0,0,114,184,0,0,0,114,111,
+    1,0,0,114,113,1,0,0,114,3,0,0,0,114,3,0,
+    0,0,114,3,0,0,0,114,6,0,0,0,218,8,60,109,
+    111,100,117,108,101,62,1,0,0,0,115,126,0,0,0,4,
+    22,4,1,4,1,2,1,2,255,4,4,8,17,8,5,8,
+    5,8,6,8,6,8,12,8,10,8,9,8,5,8,7,8,
+    9,10,22,10,127,0,13,16,1,12,2,4,1,4,2,6,
+    2,6,2,8,2,16,71,8,40,8,19,8,12,8,12,8,
+    28,8,17,8,33,8,28,8,24,10,13,10,10,10,11,8,
+    14,6,3,4,1,2,255,12,68,14,64,14,29,16,127,0,
+    17,14,72,18,45,18,26,4,3,18,53,14,63,14,42,14,
+    127,0,66,14,127,0,22,10,23,8,11,8,65,
 };

From 80722308820c112f0e473f2353f550d5aeb90352 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 08:04:56 -0700
Subject: [PATCH 0556/2163] Note regarding + mode truncation applies to both
 text and binary mode (GH-11314) (GH-15869)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* Improve doc on open's mode +

* Improve wording

* Address comment from Rémi
(cherry picked from commit c1d8c1cb8e90a54a3daaa7fcdb8d6ca7f08d6a73)

Co-authored-by: Andre Delfino 
---
 Doc/library/functions.rst | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
index c748b086d8acae..a4097b0b30eca2 100644
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -1068,12 +1068,12 @@ are always available.  They are listed here in alphabetical order.
    ``'a'``   open for writing, appending to the end of the file if it exists
    ``'b'``   binary mode
    ``'t'``   text mode (default)
-   ``'+'``   open a disk file for updating (reading and writing)
+   ``'+'``   open for updating (reading and writing)
    ========= ===============================================================
 
    The default mode is ``'r'`` (open for reading text, synonym of ``'rt'``).
-   For binary read-write access, the mode ``'w+b'`` opens and truncates the file
-   to 0 bytes.  ``'r+b'`` opens the file without truncation.
+   Modes ``'w+'`` and ``'w+b'`` opens and truncates the file.  Modes ``'r+'``
+   and ``'r+b'`` opens the file with no truncation.
 
    As mentioned in the :ref:`io-overview`, Python distinguishes between binary
    and text I/O.  Files opened in binary mode (including ``'b'`` in the *mode*

From 58ef7d341c79f649da275bb1d5c11f668d7bac9e Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 08:42:56 -0700
Subject: [PATCH 0557/2163] bpo-33602: Doc: Remove set and queue references
 from Data Types (GH-7055) (GH-15875)

(cherry picked from commit 912108891db52c2067889be1f4ce5713839807cd)

Co-authored-by: Andre Delfino 
---
 Doc/library/datatypes.rst | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/Doc/library/datatypes.rst b/Doc/library/datatypes.rst
index 48af0823aa4912..94010c0e391b0b 100644
--- a/Doc/library/datatypes.rst
+++ b/Doc/library/datatypes.rst
@@ -5,13 +5,14 @@ Data Types
 **********
 
 The modules described in this chapter provide a variety of specialized data
-types such as dates and times, fixed-type arrays, heap queues, synchronized
-queues, and sets.
+types such as dates and times, fixed-type arrays, heap queues, double-ended
+queues, and enumerations.
 
 Python also provides some built-in data types, in particular,
 :class:`dict`, :class:`list`, :class:`set` and :class:`frozenset`, and
 :class:`tuple`.  The :class:`str` class is used to hold
-Unicode strings, and the :class:`bytes` class is used to hold binary data.
+Unicode strings, and the :class:`bytes` and :class:`bytearray` classes are used
+to hold binary data.
 
 The following modules are documented in this chapter:
 

From d42a4fdc630c54352701a466a9e512bee68b5c48 Mon Sep 17 00:00:00 2001
From: Victor Stinner 
Date: Tue, 10 Sep 2019 17:54:51 +0200
Subject: [PATCH 0558/2163] bpo-37531: Enhance regrtest multiprocess timeout
 (GH-15345) (GH-15871)

* Write a message when killing a worker process
* Put a timeout on the second popen.communicate() call
  (after killing the process)
* Put a timeout on popen.wait() call
* Catch popen.kill() and popen.wait() exceptions

(cherry picked from commit de2d9eed8bc628533e1628b843cc4c7a5010f6e5)
---
 Lib/test/libregrtest/runtest_mp.py            | 81 ++++++++++++++-----
 .../2019-08-20-19-24-19.bpo-37531.wRoXfU.rst  |  3 +
 2 files changed, 66 insertions(+), 18 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Tests/2019-08-20-19-24-19.bpo-37531.wRoXfU.rst

diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index eed643291162e5..c22479b7976f00 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -126,6 +126,38 @@ def __repr__(self):
             info.append(f'pid={popen.pid}')
         return '<%s>' % ' '.join(info)
 
+    def _kill(self):
+        dt = time.monotonic() - self.start_time
+
+        popen = self._popen
+        pid = popen.pid
+        print("Kill worker process %s running for %.1f sec" % (pid, dt),
+              file=sys.stderr, flush=True)
+
+        try:
+            popen.kill()
+            return True
+        except OSError as exc:
+            print("WARNING: Failed to kill worker process %s: %r" % (pid, exc),
+                  file=sys.stderr, flush=True)
+            return False
+
+    def _close_wait(self):
+        popen = self._popen
+
+        # stdout and stderr must be closed to ensure that communicate()
+        # does not hang
+        popen.stdout.close()
+        popen.stderr.close()
+
+        try:
+            popen.wait(JOIN_TIMEOUT)
+        except (subprocess.TimeoutExpired, OSError) as exc:
+            print("WARNING: Failed to wait for worker process %s "
+                  "completion (timeout=%.1f sec): %r"
+                  % (popen.pid, JOIN_TIMEOUT, exc),
+                  file=sys.stderr, flush=True)
+
     def kill(self):
         """
         Kill the current process (if any).
@@ -135,15 +167,13 @@ def kill(self):
         """
         self._killed = True
 
-        popen = self._popen
-        if popen is None:
+        if self._popen is None:
             return
-        popen.kill()
-        # stdout and stderr must be closed to ensure that communicate()
-        # does not hang
-        popen.stdout.close()
-        popen.stderr.close()
-        popen.wait()
+
+        if not self._kill():
+            return
+
+        self._close_wait()
 
     def mp_result_error(self, test_name, error_type, stdout='', stderr='',
                         err_msg=None):
@@ -151,6 +181,23 @@ def mp_result_error(self, test_name, error_type, stdout='', stderr='',
         result = TestResult(test_name, error_type, test_time, None)
         return MultiprocessResult(result, stdout, stderr, err_msg)
 
+    def _timedout(self, test_name):
+        self._kill()
+
+        stdout = sterr = ''
+        popen = self._popen
+        try:
+            stdout, stderr = popen.communicate(timeout=JOIN_TIMEOUT)
+        except (subprocess.TimeoutExpired, OSError) as exc:
+            print("WARNING: Failed to read worker process %s output "
+                  "(timeout=%.1f sec): %r"
+                  % (popen.pid, exc, timeout),
+                  file=sys.stderr, flush=True)
+
+        self._close_wait()
+
+        return self.mp_result_error(test_name, TIMEOUT, stdout, stderr)
+
     def _runtest(self, test_name):
         try:
             self.start_time = time.monotonic()
@@ -158,7 +205,7 @@ def _runtest(self, test_name):
 
             self._popen = run_test_in_subprocess(test_name, self.ns)
             popen = self._popen
-            with popen:
+            try:
                 try:
                     if self._killed:
                         # If kill() has been called before self._popen is set,
@@ -175,12 +222,7 @@ def _runtest(self, test_name):
                             # on reading closed stdout/stderr
                             raise ExitThread
 
-                        popen.kill()
-                        stdout, stderr = popen.communicate()
-                        self.kill()
-
-                        return self.mp_result_error(test_name, TIMEOUT,
-                                                    stdout, stderr)
+                        return self._timedout(test_name)
                     except OSError:
                         if self._killed:
                             # kill() has been called: communicate() fails
@@ -190,8 +232,10 @@ def _runtest(self, test_name):
                 except:
                     self.kill()
                     raise
+            finally:
+                self._close_wait()
 
-            retcode = popen.wait()
+            retcode = popen.returncode
         finally:
             self.current_test_name = None
             self._popen = None
@@ -286,10 +330,11 @@ def wait_workers(self):
                 if not worker.is_alive():
                     break
                 dt = time.monotonic() - start_time
-                print("Wait for regrtest worker %r for %.1f sec" % (worker, dt))
+                print("Wait for regrtest worker %r for %.1f sec" % (worker, dt),
+                      flush=True)
                 if dt > JOIN_TIMEOUT:
                     print("Warning -- failed to join a regrtest worker %s"
-                          % worker)
+                          % worker, flush=True)
                     break
 
     def _get_result(self):
diff --git a/Misc/NEWS.d/next/Tests/2019-08-20-19-24-19.bpo-37531.wRoXfU.rst b/Misc/NEWS.d/next/Tests/2019-08-20-19-24-19.bpo-37531.wRoXfU.rst
new file mode 100644
index 00000000000000..59500ce67a01ee
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-08-20-19-24-19.bpo-37531.wRoXfU.rst
@@ -0,0 +1,3 @@
+Enhance regrtest multiprocess timeout: write a message when killing a worker
+process, catch popen.kill() and popen.wait() exceptions, put a timeout on the
+second call to popen.communicate().

From 6c588a00ed88dc621023771d5c50167423d0eea3 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 09:00:09 -0700
Subject: [PATCH 0559/2163] Correct info about "f.read(size)". (GH13852)

In text mode, the "size" parameter indicates the number of characters, not bytes.
(cherry picked from commit faff81c05f838b0b7a64bbc8c53c02a9b04bb79d)

Co-authored-by: William Andrea 
---
 Doc/tutorial/inputoutput.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst
index fc2bd5578c4cf1..9fe1276ec9d150 100644
--- a/Doc/tutorial/inputoutput.rst
+++ b/Doc/tutorial/inputoutput.rst
@@ -358,8 +358,8 @@ To read a file's contents, call ``f.read(size)``, which reads some quantity of
 data and returns it as a string (in text mode) or bytes object (in binary mode).
 *size* is an optional numeric argument.  When *size* is omitted or negative, the
 entire contents of the file will be read and returned; it's your problem if the
-file is twice as large as your machine's memory. Otherwise, at most *size* bytes
-are read and returned.
+file is twice as large as your machine's memory. Otherwise, at most *size*
+characters (in text mode) or *size* bytes (in binary mode) are read and returned.
 If the end of the file has been reached, ``f.read()`` will return an empty
 string (``''``).  ::
 

From cdce233f61349d029640b2dd2c0403e502050e39 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 09:20:16 -0700
Subject: [PATCH 0560/2163] bpo-38089: Move Azure Pipelines to latest VM
 versions and make macOS tests optional (GH-15851)

(cherry picked from commit 801f925998cc393260f36f5ac77369fef2373ad1)

Co-authored-by: Steve Dower 
---
 .azure-pipelines/ci.yml                                       | 4 ++--
 .azure-pipelines/macos-steps.yml                              | 2 ++
 .azure-pipelines/pr.yml                                       | 4 ++--
 .../next/macOS/2019-09-10-14-24-35.bpo-38089.eedgyD.rst       | 1 +
 4 files changed, 7 insertions(+), 4 deletions(-)
 create mode 100644 Misc/NEWS.d/next/macOS/2019-09-10-14-24-35.bpo-38089.eedgyD.rst

diff --git a/.azure-pipelines/ci.yml b/.azure-pipelines/ci.yml
index fcfac85ed9c442..12ff7d4c55c985 100644
--- a/.azure-pipelines/ci.yml
+++ b/.azure-pipelines/ci.yml
@@ -42,7 +42,7 @@ jobs:
     testRunPlatform: macos
 
   pool:
-    vmImage: xcode9-macos10.13
+    vmImage: macos-10.14
 
   steps:
   - template: ./macos-steps.yml
@@ -131,7 +131,7 @@ jobs:
   condition: and(succeeded(), eq(dependencies.Prebuild.outputs['tests.run'], 'true'))
 
   pool:
-    vmImage: vs2017-win2016
+    vmImage: windows-2019
 
   strategy:
     matrix:
diff --git a/.azure-pipelines/macos-steps.yml b/.azure-pipelines/macos-steps.yml
index 647081689454ac..d2ca580a93d7dd 100644
--- a/.azure-pipelines/macos-steps.yml
+++ b/.azure-pipelines/macos-steps.yml
@@ -14,6 +14,8 @@ steps:
 
 - script: make buildbottest TESTOPTS="-j4 -uall,-cpu --junit-xml=$(build.binariesDirectory)/test-results.xml"
   displayName: 'Tests'
+  continueOnError: true
+  timeoutInMinutes: 30
 
 - task: PublishTestResults@2
   displayName: 'Publish Test Results'
diff --git a/.azure-pipelines/pr.yml b/.azure-pipelines/pr.yml
index 2486f88a63fb15..e3e24ae2a4b07f 100644
--- a/.azure-pipelines/pr.yml
+++ b/.azure-pipelines/pr.yml
@@ -40,7 +40,7 @@ jobs:
     testRunPlatform: macos
 
   pool:
-    vmImage: xcode9-macos10.13
+    vmImage: macos-10.14
 
   steps:
   - template: ./macos-steps.yml
@@ -131,7 +131,7 @@ jobs:
   condition: and(succeeded(), eq(dependencies.Prebuild.outputs['tests.run'], 'true'))
 
   pool:
-    vmImage: vs2017-win2016
+    vmImage: windows-2019
 
   strategy:
     matrix:
diff --git a/Misc/NEWS.d/next/macOS/2019-09-10-14-24-35.bpo-38089.eedgyD.rst b/Misc/NEWS.d/next/macOS/2019-09-10-14-24-35.bpo-38089.eedgyD.rst
new file mode 100644
index 00000000000000..41349848621ec1
--- /dev/null
+++ b/Misc/NEWS.d/next/macOS/2019-09-10-14-24-35.bpo-38089.eedgyD.rst
@@ -0,0 +1 @@
+Move Azure Pipelines to latest VM versions and make macOS tests optional

From 30a8fd7e73802e2df94829b004d45ff18b9abc91 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 09:24:06 -0700
Subject: [PATCH 0561/2163] Correct minor grammatical mistake in open docs
 (GH-15865)

(cherry picked from commit 05184515f924c3880067aad87940dd466f751585)

Co-authored-by: Andre Delfino 
---
 Doc/library/functions.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
index a4097b0b30eca2..9ea7b7b55a97a3 100644
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -1072,8 +1072,8 @@ are always available.  They are listed here in alphabetical order.
    ========= ===============================================================
 
    The default mode is ``'r'`` (open for reading text, synonym of ``'rt'``).
-   Modes ``'w+'`` and ``'w+b'`` opens and truncates the file.  Modes ``'r+'``
-   and ``'r+b'`` opens the file with no truncation.
+   Modes ``'w+'`` and ``'w+b'`` open and truncate the file.  Modes ``'r+'``
+   and ``'r+b'`` open the file with no truncation.
 
    As mentioned in the :ref:`io-overview`, Python distinguishes between binary
    and text I/O.  Files opened in binary mode (including ``'b'`` in the *mode*

From 78c3949407580593f92968ea268630c36ebe6634 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 09:38:07 -0700
Subject: [PATCH 0562/2163] Docs: Small tweaks to c-api/introGH-Include_Files
 (GH-14698)

(cherry picked from commit b6dafe51399f5c6313a00529118a6052b466942f)

Co-authored-by: Kyle Stanley 
---
 Doc/c-api/intro.rst | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/Doc/c-api/intro.rst b/Doc/c-api/intro.rst
index a1c8d34a7ea02f..80eebd89ad3f25 100644
--- a/Doc/c-api/intro.rst
+++ b/Doc/c-api/intro.rst
@@ -69,10 +69,12 @@ standard headers) have one of the prefixes ``Py`` or ``_Py``.  Names beginning
 with ``_Py`` are for internal use by the Python implementation and should not be
 used by extension writers. Structure member names do not have a reserved prefix.
 
-**Important:** user code should never define names that begin with ``Py`` or
-``_Py``.  This confuses the reader, and jeopardizes the portability of the user
-code to future Python versions, which may define additional names beginning with
-one of these prefixes.
+.. note::
+
+   User code should never define names that begin with ``Py`` or ``_Py``. This
+   confuses the reader, and jeopardizes the portability of the user code to
+   future Python versions, which may define additional names beginning with one
+   of these prefixes.
 
 The header files are typically installed with Python.  On Unix, these  are
 located in the directories :file:`{prefix}/include/pythonversion/` and
@@ -90,9 +92,9 @@ multi-platform builds since the platform independent headers under
 :envvar:`prefix` include the platform specific headers from
 :envvar:`exec_prefix`.
 
-C++ users should note that though the API is defined entirely using C, the
-header files do properly declare the entry points to be ``extern "C"``, so there
-is no need to do anything special to use the API from C++.
+C++ users should note that although the API is defined entirely using C, the
+header files properly declare the entry points to be ``extern "C"``. As a result,
+there is no need to do anything special to use the API from C++.
 
 
 Useful macros

From 63909cdc2ff4eec58cc2b56426f2c725ccd4ba2b Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 10:13:55 -0700
Subject: [PATCH 0563/2163] bpo-37574: Mention helper functions for find_spec
 documentation (GH-14739)

(cherry picked from commit 9cbb97b29eac4b23e916a3233f26b60ac69e335b)

Co-authored-by: jdkandersson <51036209+jdkandersson@users.noreply.github.com>
---
 Doc/library/importlib.rst | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst
index cb8360caca5e32..47e3dd03a687a8 100644
--- a/Doc/library/importlib.rst
+++ b/Doc/library/importlib.rst
@@ -274,6 +274,8 @@ ABC hierarchy::
       parent package. If a spec cannot be found, ``None`` is returned.
       When passed in, ``target`` is a module object that the finder may
       use to make a more educated guess about what spec to return.
+      :func:`importlib.util.spec_from_loader` may be useful for implementing
+      concrete ``MetaPathFinders``.
 
       .. versionadded:: 3.4
 
@@ -323,7 +325,8 @@ ABC hierarchy::
       within the :term:`path entry` to which it is assigned.  If a spec
       cannot be found, ``None`` is returned.  When passed in, ``target``
       is a module object that the finder may use to make a more educated
-      guess about what spec to return.
+      guess about what spec to return. :func:`importlib.util.spec_from_loader`
+      may be useful for implementing concrete ``PathEntryFinders``.
 
       .. versionadded:: 3.4
 

From 98224d24d1d69383ccc6336e87e362f986c99169 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 10:46:52 -0700
Subject: [PATCH 0564/2163] Remove unneeded assignment in PyBytes_Concat()
 (GH-15274)

The `wb.len = -1` assignment is unneeded since its introduction in 161d695fb0455ce52530d4f43a9eac4c738f64bb as `PyObject_GetBuffer` always fills it in.
(cherry picked from commit afdeb189e97033b54cef44a7490d89d2013cb4e5)

Co-authored-by: Sergey Fedoseev 
---
 Objects/bytesobject.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
index 06c87b0c62d3bb..9358e5effd5674 100644
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -2939,7 +2939,6 @@ PyBytes_Concat(PyObject **pv, PyObject *w)
         Py_ssize_t oldsize;
         Py_buffer wb;
 
-        wb.len = -1;
         if (PyObject_GetBuffer(w, &wb, PyBUF_SIMPLE) != 0) {
             PyErr_Format(PyExc_TypeError, "can't concat %.100s to %.100s",
                          Py_TYPE(w)->tp_name, Py_TYPE(*pv)->tp_name);

From 74b0291b03db60dd244d31e9c97407cccb8d30dd Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Tue, 10 Sep 2019 15:57:54 -0700
Subject: [PATCH 0565/2163] bpo-28494: Test existing zipfile working behavior.
 (GH-15853)

Add unittests for executables with a zipfile appended to test_zipfile, as zipfile.is_zipfile and zipfile.ZipFile work properly on these today.
(cherry picked from commit 3f4db4a0bab073b768fae958e93288bd5d24eadd)

Co-authored-by: Gregory P. Smith 
---
 Lib/test/test_zipfile.py                      |  40 ++++++++++++++++++
 Lib/test/ziptestdata/README.md                |  35 +++++++++++++++
 Lib/test/ziptestdata/exe_with_z64             | Bin 0 -> 978 bytes
 Lib/test/ziptestdata/exe_with_zip             | Bin 0 -> 990 bytes
 Lib/test/ziptestdata/header.sh                |  24 +++++++++++
 .../ziptestdata/testdata_module_inside_zip.py |   2 +
 6 files changed, 101 insertions(+)
 create mode 100644 Lib/test/ziptestdata/README.md
 create mode 100755 Lib/test/ziptestdata/exe_with_z64
 create mode 100755 Lib/test/ziptestdata/exe_with_zip
 create mode 100755 Lib/test/ziptestdata/header.sh
 create mode 100644 Lib/test/ziptestdata/testdata_module_inside_zip.py

diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py
index 8e437e5cb2b90c..955e540f96e627 100644
--- a/Lib/test/test_zipfile.py
+++ b/Lib/test/test_zipfile.py
@@ -5,6 +5,8 @@
 import pathlib
 import posixpath
 import struct
+import subprocess
+import sys
 import time
 import unittest
 import zipfile
@@ -2443,6 +2445,44 @@ def build_alpharep_fixture():
     return zf
 
 
+class TestExecutablePrependedZip(unittest.TestCase):
+    """Test our ability to open zip files with an executable prepended."""
+
+    def setUp(self):
+        self.exe_zip = findfile('exe_with_zip', subdir='ziptestdata')
+        self.exe_zip64 = findfile('exe_with_z64', subdir='ziptestdata')
+
+    def _test_zip_works(self, name):
+        # bpo-28494 sanity check: ensure is_zipfile works on these.
+        self.assertTrue(zipfile.is_zipfile(name),
+                        f'is_zipfile failed on {name}')
+        # Ensure we can operate on these via ZipFile.
+        with zipfile.ZipFile(name) as zipfp:
+            for n in zipfp.namelist():
+                data = zipfp.read(n)
+                self.assertIn(b'FAVORITE_NUMBER', data)
+
+    def test_read_zip_with_exe_prepended(self):
+        self._test_zip_works(self.exe_zip)
+
+    def test_read_zip64_with_exe_prepended(self):
+        self._test_zip_works(self.exe_zip64)
+
+    @unittest.skipUnless(sys.executable, 'sys.executable required.')
+    @unittest.skipUnless(os.access('/bin/bash', os.X_OK),
+                         'Test relies on #!/bin/bash working.')
+    def test_execute_zip2(self):
+        output = subprocess.check_output([self.exe_zip, sys.executable])
+        self.assertIn(b'number in executable: 5', output)
+
+    @unittest.skipUnless(sys.executable, 'sys.executable required.')
+    @unittest.skipUnless(os.access('/bin/bash', os.X_OK),
+                         'Test relies on #!/bin/bash working.')
+    def test_execute_zip64(self):
+        output = subprocess.check_output([self.exe_zip64, sys.executable])
+        self.assertIn(b'number in executable: 5', output)
+
+
 class TestPath(unittest.TestCase):
     def setUp(self):
         self.fixtures = contextlib.ExitStack()
diff --git a/Lib/test/ziptestdata/README.md b/Lib/test/ziptestdata/README.md
new file mode 100644
index 00000000000000..6b9147db76e178
--- /dev/null
+++ b/Lib/test/ziptestdata/README.md
@@ -0,0 +1,35 @@
+# Test data for `test_zipfile`
+
+The test executables in this directory are created manually from header.sh and
+the `testdata_module_inside_zip.py` file.  You must have infozip's zip utility
+installed (`apt install zip` on Debian).
+
+## Purpose
+
+These are used to test executable files with an appended zipfile, in a scenario
+where the executable is _not_ a Python interpreter itself so our automatic
+zipimport machinery (that'd look for `__main__.py`) is not being used.
+
+## Updating the test executables
+
+If you update header.sh or the testdata_module_inside_zip.py file, rerun the
+commands below.  These are expected to be rarely changed, if ever.
+
+### Standard old format (2.0) zip file
+
+```
+zip -0 zip2.zip testdata_module_inside_zip.py
+cat header.sh zip2.zip >exe_with_zip
+rm zip2.zip
+```
+
+### Modern format (4.5) zip64 file
+
+Redirecting from stdin forces infozip's zip tool to create a zip64.
+
+```
+zip -0 zip64.zip
+cat header.sh zip64.zip >exe_with_z64
+rm zip64.zip
+```
+
diff --git a/Lib/test/ziptestdata/exe_with_z64 b/Lib/test/ziptestdata/exe_with_z64
new file mode 100755
index 0000000000000000000000000000000000000000..82b03cf39d919d9de05866b4f7cdf1dfe47e6eb3
GIT binary patch
literal 978
zcmaJ<-EY${5KkEl$xl4ME4rnV(xNp330}q~w6RoFg|_N41e7XrQ)jhEoXByzLbV6}
z7bN~nxXZ`b5aOcPclPh+`)uy)&!pO)@qEF01K%5u#vZQ0`QQ{+-#hbQJlMzfN
zumhbn*t?s5Bd=_jPG5pq2*m(Jgo_mHo-#sbTHp%FGB+?21c5M360YVDOC^Boi)A8|
zaqW`1mIj`)NHXt(_xjvFK6&e598YZ!YZ3l8f{q6rI6U+Qr@^orj6V8rh65&(EY$|m
zyw<+SERwNcOz}kI84m>G
zSHN@NP(AKCZFVWm;@bWsvo1d0s^NQ(q;qlPXs1m?Of5j_0ahSNH4rM0DoR1B`pzXg
zmbq!Q2?j9dhGVD|)zyN}i{}esyQ-xKTZG$#>tt`JC1{4sF91y#s`x7`^Rh*e)YvZy
zgkqqkaUCw?O1P{lg45-zR7)d3Et45`xQsPi8a|7~fpf#r#O@xyAC7yz7Yxqdop@t=
z+GecTY{GH*C{6^9iZVG|$~dMm;TcwVF6O`^njW)|c@d2ZNMpBKJnC=V?N}s_K0g`u
zf>&q1Drr~`txm&wV0p#0b-g#i7nomB!y-wOlGog%8hujhFq@*CrC0V>0$BJLY}9Yu
zdAxPoGdZHaQ8}dT$9GygqyF~x9%(2wjr1B?@B4I!vMx6ZdG|^ES=ode_3v$y*}#wR
GCH6PMMGy`E

literal 0
HcmV?d00001

diff --git a/Lib/test/ziptestdata/exe_with_zip b/Lib/test/ziptestdata/exe_with_zip
new file mode 100755
index 0000000000000000000000000000000000000000..c833cdf9f934b08fc449da077efae44aa9a06f18
GIT binary patch
literal 990
zcmah{&2G~`5Ozt6#BxUB0BGt|a)MeHBzh^WP)i*V6_O}*5D+43W3QVP_S)KAHz`pi
zcm>WJc^aOC$6(e@N~!o+%den-3;)Sr%Y!Z0+w(d{LAMq3-uf@P
z9m3N*lNvI$JbmPO%o9e4pea*14H@ji{DR<>2{Yf&&6LZ;8JC$DI=^T*Ba%xlbR%}U
zITKu*!h8w30IGn(BDw1{$&~BKrT>oSEll57hHpZeMQq=ZPSXIfv;d*Is6d=aFi`;)
zaRyv0|GCCbxYCWL2?L0zrbu-GbtR)wnZ5)z7h1BgVd6I7ve+xfDrk(z4*+%OisT#$
zRkbMQ68mL{7!IasRE86N#$2)x!D-R6OmfXY6zLc{TyYHxO~(n_b*@}Av|9(SyZyHB
z1)agGL$7a-nuOHrbvUS!;zZ!62(4hslf;Y(%~9cqML=USJ$k}b$;JhQk>6X~JFcw~
z%d9)^A9mZpvl9=`=Dly-vourMXb_;{MX9UeQ7N~ZpAY<7R_*(II{NZyIx1$jt(Dau
zHOneZ9ejjVI+sG|%rH|rlgP`o7b`AXUNIxrip1vZklyjijR&>AvAb(Xm+RYSv;Bwb
WTE+Dm&))IcO#@!RC&c}$ajc&zD=F;&

literal 0
HcmV?d00001

diff --git a/Lib/test/ziptestdata/header.sh b/Lib/test/ziptestdata/header.sh
new file mode 100755
index 00000000000000..52dc91acf7400c
--- /dev/null
+++ b/Lib/test/ziptestdata/header.sh
@@ -0,0 +1,24 @@
+#!/bin/bash
+INTERPRETER_UNDER_TEST="$1"
+if [[ ! -x "${INTERPRETER_UNDER_TEST}" ]]; then
+    echo "Interpreter must be the command line argument."
+    exit 4
+fi
+EXECUTABLE="$0" exec "${INTERPRETER_UNDER_TEST}" -E - <
Date: Tue, 10 Sep 2019 23:56:10 -0700
Subject: [PATCH 0566/2163] bpo-38034: Fix typo in logging.handlers.rst
 (GH-15708) (GH-15893)

(cherry picked from commit efd5741ae953e50a6654e04cf731da86a1307296)

Co-authored-by: wwuck <301402+wwuck@users.noreply.github.com>
---
 Doc/library/logging.handlers.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst
index 32919c1a2e2909..0e9870408ff43d 100644
--- a/Doc/library/logging.handlers.rst
+++ b/Doc/library/logging.handlers.rst
@@ -1046,7 +1046,7 @@ possible, while any potentially slow operations (such as sending an email via
    versions - to always pass each message to each handler.
 
    .. versionchanged:: 3.5
-      The ``respect_handler_levels`` argument was added.
+      The ``respect_handler_level`` argument was added.
 
    .. method:: dequeue(block)
 

From 872c85a1796290baef89a20dbde819c4da45830c Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 11 Sep 2019 03:05:14 -0700
Subject: [PATCH 0567/2163] bpo-37424: Avoid a hang in subprocess.run timeout
 output capture (GH-14490)

Fixes a possible hang when using a timeout on subprocess.run() while
capturing output. If the child process spawned its own children or otherwise
connected its stdout or stderr handles with another process, we could hang
after the timeout was reached and our child was killed when attempting to read
final output from the pipes.
(cherry picked from commit 580d2782f70f8e0bed7ec20abb03d740cb83b5da)

Co-authored-by: Gregory P. Smith 
---
 Lib/subprocess.py                             | 36 ++++++++++++++-----
 Lib/test/test_subprocess.py                   | 21 +++++++++++
 .../2019-07-04-13-00-20.bpo-37424.0i1MR-.rst  |  5 +++
 3 files changed, 53 insertions(+), 9 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2019-07-04-13-00-20.bpo-37424.0i1MR-.rst

diff --git a/Lib/subprocess.py b/Lib/subprocess.py
index aed7292541e034..85b9ea07854669 100644
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -489,11 +489,20 @@ def run(*popenargs,
     with Popen(*popenargs, **kwargs) as process:
         try:
             stdout, stderr = process.communicate(input, timeout=timeout)
-        except TimeoutExpired:
+        except TimeoutExpired as exc:
             process.kill()
-            stdout, stderr = process.communicate()
-            raise TimeoutExpired(process.args, timeout, output=stdout,
-                                 stderr=stderr)
+            if _mswindows:
+                # Windows accumulates the output in a single blocking
+                # read() call run on child threads, with the timeout
+                # being done in a join() on those threads.  communicate()
+                # _after_ kill() is required to collect that and add it
+                # to the exception.
+                exc.stdout, exc.stderr = process.communicate()
+            else:
+                # POSIX _communicate already populated the output so
+                # far into the TimeoutExpired exception.
+                process.wait()
+            raise
         except:  # Including KeyboardInterrupt, communicate handled that.
             process.kill()
             # We don't call process.wait() as .__exit__ does that for us.
@@ -1050,12 +1059,16 @@ def _remaining_time(self, endtime):
             return endtime - _time()
 
 
-    def _check_timeout(self, endtime, orig_timeout):
+    def _check_timeout(self, endtime, orig_timeout, stdout_seq, stderr_seq,
+                       skip_check_and_raise=False):
         """Convenience for checking if a timeout has expired."""
         if endtime is None:
             return
-        if _time() > endtime:
-            raise TimeoutExpired(self.args, orig_timeout)
+        if skip_check_and_raise or _time() > endtime:
+            raise TimeoutExpired(
+                    self.args, orig_timeout,
+                    output=b''.join(stdout_seq) if stdout_seq else None,
+                    stderr=b''.join(stderr_seq) if stderr_seq else None)
 
 
     def wait(self, timeout=None):
@@ -1843,10 +1856,15 @@ def _communicate(self, input, endtime, orig_timeout):
                 while selector.get_map():
                     timeout = self._remaining_time(endtime)
                     if timeout is not None and timeout < 0:
-                        raise TimeoutExpired(self.args, orig_timeout)
+                        self._check_timeout(endtime, orig_timeout,
+                                            stdout, stderr,
+                                            skip_check_and_raise=True)
+                        raise RuntimeError(  # Impossible :)
+                            '_check_timeout(..., skip_check_and_raise=True) '
+                            'failed to raise TimeoutExpired.')
 
                     ready = selector.select(timeout)
-                    self._check_timeout(endtime, orig_timeout)
+                    self._check_timeout(endtime, orig_timeout, stdout, stderr)
 
                     # XXX Rewrite these to use non-blocking I/O on the file
                     # objects; they are no longer using C stdio!
diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index e58d0925df3bac..69ac584667d3ee 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -10,6 +10,7 @@
 import errno
 import tempfile
 import time
+import traceback
 import selectors
 import sysconfig
 import select
@@ -1557,6 +1558,26 @@ def test_stderr_with_capture_output_arg(self):
         self.assertIn('stderr', c.exception.args[0])
         self.assertIn('capture_output', c.exception.args[0])
 
+    # This test _might_ wind up a bit fragile on loaded build+test machines
+    # as it depends on the timing with wide enough margins for normal situations
+    # but does assert that it happened "soon enough" to believe the right thing
+    # happened.
+    @unittest.skipIf(mswindows, "requires posix like 'sleep' shell command")
+    def test_run_with_shell_timeout_and_capture_output(self):
+        """Output capturing after a timeout mustn't hang forever on open filehandles."""
+        before_secs = time.monotonic()
+        try:
+            subprocess.run('sleep 3', shell=True, timeout=0.1,
+                           capture_output=True)  # New session unspecified.
+        except subprocess.TimeoutExpired as exc:
+            after_secs = time.monotonic()
+            stacks = traceback.format_exc()  # assertRaises doesn't give this.
+        else:
+            self.fail("TimeoutExpired not raised.")
+        self.assertLess(after_secs - before_secs, 1.5,
+                        msg="TimeoutExpired was delayed! Bad traceback:\n```\n"
+                        f"{stacks}```")
+
 
 @unittest.skipIf(mswindows, "POSIX specific tests")
 class POSIXProcessTestCase(BaseTestCase):
diff --git a/Misc/NEWS.d/next/Library/2019-07-04-13-00-20.bpo-37424.0i1MR-.rst b/Misc/NEWS.d/next/Library/2019-07-04-13-00-20.bpo-37424.0i1MR-.rst
new file mode 100644
index 00000000000000..b98a17e241e077
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-07-04-13-00-20.bpo-37424.0i1MR-.rst
@@ -0,0 +1,5 @@
+Fixes a possible hang when using a timeout on `subprocess.run()` while
+capturing output.  If the child process spawned its own children or
+otherwise connected its stdout or stderr handles with another process, we
+could hang after the timeout was reached and our child was killed when
+attempting to read final output from the pipes.

From df935b5f0bcc40522c7aac1e844aa176cd8bbdef Mon Sep 17 00:00:00 2001
From: Zachary Ware 
Date: Wed, 11 Sep 2019 11:31:12 +0100
Subject: [PATCH 0568/2163] [3.8] bpo-37936: Systematically distinguish rooted
 vs. unrooted in .gitignore (GH-15823) (GH-15900)

A root cause of bpo-37936 is that it's easy to write a .gitignore
rule that's intended to apply to a specific file (e.g., the
`pyconfig.h` generated by `./configure`) but actually applies to all
similarly-named files in the tree (e.g., `PC/pyconfig.h`.)

Specifically, any rule with no non-trailing slashes is applied in an
"unrooted" way, to files anywhere in the tree.  This means that if we
write the rules in the most obvious-looking way, then

 * for specific files we want to ignore that happen to be in
   subdirectories (like `Modules/config.c`), the rule will work
   as intended, staying "rooted" to the top of the tree; but

 * when a specific file we want to ignore happens to be at the root of
   the repo (like `platform`), then the obvious rule (`platform`) will
   apply much more broadly than intended: if someone tries to add a
   file or directory named `platform` somewhere else in the tree, it
   will unexpectedly get ignored.

That's surprising behavior that can make the .gitignore file's
behavior feel finicky and unpredictable.

To avoid it, we can simply always give a rule "rooted" behavior when
that's what's intended, by systematically using leading slashes.

Further, to help make the pattern obvious when looking at the file and
minimize any need for thinking about the syntax when adding new rules:
separate the rules into one group for each type, with brief comments
identifying them.

For most of these rules it's clear whether they're meant to be rooted
or unrooted, but in a handful of cases I've only guessed.  In that
case the safer default (the choice that won't hide information) is the
narrower, rooted meaning, with a leading slash.  If for some of these
the unrooted meaning is desired after all, it'll be easy to move them
to the unrooted section at the top.

(cherry picked from commit 455122a0094c8cfdf7e062eccc5e5b5885c75e8b)

Co-authored-by: Greg Price 
---
 .gitignore                                    | 103 ++++++++++--------
 Lib/test/libregrtest/runtest.py               |   4 +-
 .../2019-09-10-00-54-48.bpo-37936.E7XEwu.rst  |   5 +
 3 files changed, 61 insertions(+), 51 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Build/2019-09-10-00-54-48.bpo-37936.E7XEwu.rst

diff --git a/.gitignore b/.gitignore
index 648f07e765b122..b662374b1255cb 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,11 +1,14 @@
-# Two-trick pony for OSX and other case insensitive file systems:
-# Ignore ./python binary on Unix but still look into ./Python/ directory.
-/python
-!/Python/
+#####
+# First, rules intended to apply in all subdirectories.
+# These contain no slash, or only a trailing slash.
 
 *.cover
 *.iml
 *.o
+*.a
+*.so*
+*.dylib
+*.dll
 *.orig
 *.pyc
 *.pyd
@@ -18,6 +21,31 @@
 *.profraw
 *.dyn
 .gdb_history
+.purify
+__pycache__
+.hg/
+.svn/
+.idea/
+tags
+TAGS
+.vs/
+.vscode/
+gmon.out
+.coverage
+.mypy_cache/
+
+*.exe
+!Lib/distutils/command/*.exe
+
+# Ignore core dumps... but not Tools/msi/core/ or the like.
+core
+!core/
+
+
+#####
+# Then, rules meant for a specific location relative to the repo root.
+# These must contain a non-trailing slash (and may also have a trailing slash.)
+
 Doc/build/
 Doc/venv/
 Doc/.venv/
@@ -29,7 +57,7 @@ Lib/lib2to3/*.pickle
 Lib/test/data/*
 !Lib/test/data/README
 /Makefile
-Makefile.pre
+/Makefile.pre
 Misc/python.pc
 Misc/python-embed.pc
 Misc/python-config.sh
@@ -38,12 +66,9 @@ Modules/Setup.local
 Modules/config.c
 Modules/ld_so_aix
 Programs/_freeze_importlib
-Programs/_freeze_importlib.exe
 Programs/_testembed
-Programs/_testembed.exe
 PC/python_nt*.h
 PC/pythonnt_rc*.h
-PC/*/*.exe
 PC/*/*.exp
 PC/*/*.lib
 PC/*/*.bsc
@@ -62,52 +87,34 @@ PCbuild/*-pgi
 PCbuild/*-pgo
 PCbuild/*.VC.db
 PCbuild/*.VC.opendb
-PCbuild/.vs/
 PCbuild/amd64/
 PCbuild/arm32/
 PCbuild/arm64/
 PCbuild/obj/
 PCbuild/win32/
-.purify
-__pycache__
-autom4te.cache
-build/
-buildno
-config.cache
-config.log
-config.status
-config.status.lineno
-core
-!Tools/msi/core/
-db_home
-.hg/
-.idea/
-ipch/
-libpython*.a
-libpython*.so*
-libpython*.dylib
-libpython*.dll
-platform
-pybuilddir.txt
+/autom4te.cache
+/build/
+/config.cache
+/config.log
+/config.status
+/config.status.lineno
+/platform
+/pybuilddir.txt
 /pyconfig.h
-python-config
-python-config.py
-python.bat
-python.exe
-python-gdb.py
-python.exe-gdb.py
-reflog.txt
-.svn/
-tags
-TAGS
-.coverage
-coverage/
-externals/
-htmlcov/
+/python-config
+/python-config.py
+/python.bat
+/python-gdb.py
+/python.exe-gdb.py
+/reflog.txt
+/coverage/
+/externals/
+/htmlcov/
 Tools/msi/obj
 Tools/ssl/amd64
 Tools/ssl/win32
-.vs/
-.vscode/
-gmon.out
-.mypy_cache/
+
+# Two-trick pony for OSX and other case insensitive file systems:
+# Ignore ./python binary on Unix but still look into ./Python/ directory.
+/python
+!/Python/
diff --git a/Lib/test/libregrtest/runtest.py b/Lib/test/libregrtest/runtest.py
index e7dce180cb3753..eeb108bb447394 100644
--- a/Lib/test/libregrtest/runtest.py
+++ b/Lib/test/libregrtest/runtest.py
@@ -313,9 +313,7 @@ def cleanup_test_droppings(test_name, verbose):
     # since if a test leaves a file open, it cannot be deleted by name (while
     # there's nothing we can do about that here either, we can display the
     # name of the offending test, which is a real help).
-    for name in (support.TESTFN,
-                 "db_home",
-                ):
+    for name in (support.TESTFN,):
         if not os.path.exists(name):
             continue
 
diff --git a/Misc/NEWS.d/next/Build/2019-09-10-00-54-48.bpo-37936.E7XEwu.rst b/Misc/NEWS.d/next/Build/2019-09-10-00-54-48.bpo-37936.E7XEwu.rst
new file mode 100644
index 00000000000000..5ded61eafe947c
--- /dev/null
+++ b/Misc/NEWS.d/next/Build/2019-09-10-00-54-48.bpo-37936.E7XEwu.rst
@@ -0,0 +1,5 @@
+The :file:`.gitignore` file systematically keeps "rooted", with a
+non-trailing slash, all the rules that are meant to apply to files in a
+specific place in the repo.  Previously, when the intended file to ignore
+happened to be at the root of the repo, we'd most often accidentally also
+ignore files and directories with the same name anywhere in the tree.

From 80db4b4be54ccdb5b67821506b6db2b27bd7c28a Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 11 Sep 2019 03:39:36 -0700
Subject: [PATCH 0569/2163] bpo-16438: Doc: confusing text regarding numeric
 precedence corrected (GH-10521)

(cherry picked from commit 4576b5431bd597df7581fe3c852b315e47e4b230)

Co-authored-by: Anjali 
---
 Doc/library/stdtypes.rst | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
index 0f7c369ea51f9a..dade2cd69fee98 100644
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -265,9 +265,8 @@ which is narrower than complex.  Comparisons between numbers of mixed type use
 the same rule. [2]_ The constructors :func:`int`, :func:`float`, and
 :func:`complex` can be used to produce numbers of a specific type.
 
-All numeric types (except complex) support the following operations, sorted by
-ascending priority (all numeric operations have a higher priority than
-comparison operations):
+All numeric types (except complex) support the following operations (for priorities of
+the operations, see :ref:`operator-summary`):
 
 +---------------------+---------------------------------+---------+--------------------+
 | Operation           | Result                          | Notes   | Full documentation |

From 4601f7a49fe8ed00c4b6b70b0eda2b3922568e9b Mon Sep 17 00:00:00 2001
From: Andrew Svetlov 
Date: Wed, 11 Sep 2019 13:40:36 +0300
Subject: [PATCH 0570/2163] [3.8] bpo-36373: Fix deprecation warnings
 (GH-15889) (GH-15901)

https://bugs.python.org/issue36373
(cherry picked from commit 7264e92b718d307cc499b2f10eab7644b00f0499)

Co-authored-by: Andrew Svetlov 
---
 Lib/asyncio/locks.py                 |  2 +-
 Lib/asyncio/queues.py                |  2 +-
 Lib/test/test_asyncio/test_locks.py  |  7 +++----
 Lib/test/test_asyncio/test_queues.py |  3 +--
 Lib/unittest/async_case.py           | 10 ++++++----
 5 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/Lib/asyncio/locks.py b/Lib/asyncio/locks.py
index f63d4cedbbb7c3..d94daeb5a173f5 100644
--- a/Lib/asyncio/locks.py
+++ b/Lib/asyncio/locks.py
@@ -332,7 +332,7 @@ def __init__(self, lock=None, *, loop=None):
                           DeprecationWarning, stacklevel=2)
 
         if lock is None:
-            lock = Lock(loop=self._loop)
+            lock = Lock(loop=loop)
         elif lock._loop is not self._loop:
             raise ValueError("loop argument must agree with lock")
 
diff --git a/Lib/asyncio/queues.py b/Lib/asyncio/queues.py
index c96b4a0c1ba327..390ae9a6821c4d 100644
--- a/Lib/asyncio/queues.py
+++ b/Lib/asyncio/queues.py
@@ -45,7 +45,7 @@ def __init__(self, maxsize=0, *, loop=None):
         # Futures.
         self._putters = collections.deque()
         self._unfinished_tasks = 0
-        self._finished = locks.Event(loop=self._loop)
+        self._finished = locks.Event(loop=loop)
         self._finished.set()
         self._init(maxsize)
 
diff --git a/Lib/test/test_asyncio/test_locks.py b/Lib/test/test_asyncio/test_locks.py
index d69b56dcda2254..a9953b4b2a2de2 100644
--- a/Lib/test/test_asyncio/test_locks.py
+++ b/Lib/test/test_asyncio/test_locks.py
@@ -500,10 +500,9 @@ def test_ctor_loop(self):
             self.assertIs(cond._loop, self.loop)
 
     def test_ctor_noloop(self):
-        with self.assertWarns(DeprecationWarning):
-            asyncio.set_event_loop(self.loop)
-            cond = asyncio.Condition()
-            self.assertIs(cond._loop, self.loop)
+        asyncio.set_event_loop(self.loop)
+        cond = asyncio.Condition()
+        self.assertIs(cond._loop, self.loop)
 
     def test_wait(self):
         with self.assertWarns(DeprecationWarning):
diff --git a/Lib/test/test_asyncio/test_queues.py b/Lib/test/test_asyncio/test_queues.py
index 02e8e43ccee698..171176c9fc5309 100644
--- a/Lib/test/test_asyncio/test_queues.py
+++ b/Lib/test/test_asyncio/test_queues.py
@@ -83,8 +83,7 @@ def test_ctor_loop(self):
 
     def test_ctor_noloop(self):
         asyncio.set_event_loop(self.loop)
-        with self.assertWarns(DeprecationWarning):
-            q = asyncio.Queue()
+        q = asyncio.Queue()
         self.assertIs(q._loop, self.loop)
 
     def test_repr(self):
diff --git a/Lib/unittest/async_case.py b/Lib/unittest/async_case.py
index a3c8bfb9eca758..1bc1312c8c2ee9 100644
--- a/Lib/unittest/async_case.py
+++ b/Lib/unittest/async_case.py
@@ -89,8 +89,9 @@ def _callMaybeAsync(self, func, /, *args, **kwargs):
         else:
             return ret
 
-    async def _asyncioLoopRunner(self):
-        queue = self._asyncioCallsQueue
+    async def _asyncioLoopRunner(self, fut):
+        self._asyncioCallsQueue = queue = asyncio.Queue()
+        fut.set_result(None)
         while True:
             query = await queue.get()
             queue.task_done()
@@ -113,8 +114,9 @@ def _setupAsyncioLoop(self):
         asyncio.set_event_loop(loop)
         loop.set_debug(True)
         self._asyncioTestLoop = loop
-        self._asyncioCallsQueue = asyncio.Queue(loop=loop)
-        self._asyncioCallsTask = loop.create_task(self._asyncioLoopRunner())
+        fut = loop.create_future()
+        self._asyncioCallsTask = loop.create_task(self._asyncioLoopRunner(fut))
+        loop.run_until_complete(fut)
 
     def _tearDownAsyncioLoop(self):
         assert self._asyncioTestLoop is not None

From 57491de7c33c5886c4cae2f468b474d9b4e6fed2 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 11 Sep 2019 03:43:30 -0700
Subject: [PATCH 0571/2163] bpo-38081: Fixes ntpath.realpath('NUL') (GH-15899)

(cherry picked from commit 92521fea5d0d4aeb9b6a3c3fdd4654af700ad5c8)

Co-authored-by: Steve Dower 
---
 Lib/ntpath.py                                       | 13 +++++++------
 Lib/test/test_ntpath.py                             |  4 ++++
 .../2019-09-11-10-22-01.bpo-38081.8JhzjD.rst        |  1 +
 3 files changed, 12 insertions(+), 6 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Windows/2019-09-11-10-22-01.bpo-38081.8JhzjD.rst

diff --git a/Lib/ntpath.py b/Lib/ntpath.py
index 1d22d5f1dc2ab5..1b5e16f95f1659 100644
--- a/Lib/ntpath.py
+++ b/Lib/ntpath.py
@@ -535,9 +535,10 @@ def _readlink_deep(path, seen=None):
             try:
                 path = _nt_readlink(path)
             except OSError as ex:
-                # Stop on file (2) or directory (3) not found, or
-                # paths that are not reparse points (4390)
-                if ex.winerror in (2, 3, 4390):
+                # Stop on incorrect function (1), file (2) or
+                # directory (3) not found, or paths that are
+                # not reparse points (4390)
+                if ex.winerror in (1, 2, 3, 4390):
                     break
                 raise
             except ValueError:
@@ -553,9 +554,9 @@ def _getfinalpathname_nonstrict(path):
         except OSError:
             pass
 
-        # Allow file (2) or directory (3) not found, invalid syntax (123),
-        # and symlinks that cannot be followed (1921)
-        allowed_winerror = 2, 3, 123, 1921
+        # Allow file (2) or directory (3) not found, incorrect parameter (87),
+        # invalid syntax (123), and symlinks that cannot be followed (1921)
+        allowed_winerror = 2, 3, 87, 123, 1921
 
         # Non-strict algorithm is to find as much of the target directory
         # as we can and join the rest.
diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py
index c5c96e32d7473c..2f0faf94726fd2 100644
--- a/Lib/test/test_ntpath.py
+++ b/Lib/test/test_ntpath.py
@@ -400,6 +400,10 @@ def test_realpath_symlink_prefix(self):
         self.assertPathEqual(ntpath.realpath("\\\\?\\" + ABSTFN + "3.link"),
                              "\\\\?\\" + ABSTFN + "3.")
 
+    @unittest.skipUnless(HAVE_GETFINALPATHNAME, 'need _getfinalpathname')
+    def test_realpath_nul(self):
+        tester("ntpath.realpath('NUL')", r'\\.\NUL')
+
     def test_expandvars(self):
         with support.EnvironmentVarGuard() as env:
             env.clear()
diff --git a/Misc/NEWS.d/next/Windows/2019-09-11-10-22-01.bpo-38081.8JhzjD.rst b/Misc/NEWS.d/next/Windows/2019-09-11-10-22-01.bpo-38081.8JhzjD.rst
new file mode 100644
index 00000000000000..e9d7a30eed9f9c
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-09-11-10-22-01.bpo-38081.8JhzjD.rst
@@ -0,0 +1 @@
+Prevent error calling :func:`os.path.realpath` on ``'NUL'``.

From 690a16d455603500a0c6df0bd87e49c9b37a6950 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 11 Sep 2019 03:46:53 -0700
Subject: [PATCH 0572/2163] bpo-37585: Add clarification regarding comparing
 dict.values() (GH-14954)

(cherry picked from commit 6472ece5a0fe82809d3aa0ffb281796fcd252d76)

Co-authored-by: Kyle Stanley 
---
 Doc/library/stdtypes.rst | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
index dade2cd69fee98..32bd7d2b894d7d 100644
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -4343,6 +4343,14 @@ pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098:
       Return a new view of the dictionary's values.  See the
       :ref:`documentation of view objects `.
 
+      An equality comparison between one ``dict.values()`` view and another
+      will always return ``False``. This also applies when comparing
+      ``dict.values()`` to itself::
+
+         >>> d = {'a': 1}
+         >>> d.values() == d.values()
+         False
+
    Dictionaries compare equal if and only if they have the same ``(key,
    value)`` pairs. Order comparisons ('<', '<=', '>=', '>') raise
    :exc:`TypeError`.

From 4e914ab29f6d48a9fd045ea6a25dbf9e2fb603f9 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 11 Sep 2019 03:53:16 -0700
Subject: [PATCH 0573/2163] bpo-25810: Clarify eval() docs, it does not
 keywords (GH-15173)

(cherry picked from commit 7a0023e8d17566eb32c836b65c33663303a2224f)

Co-authored-by: smokephil 
---
 Doc/library/functions.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
index 9ea7b7b55a97a3..2f3ef4f7fc76fb 100644
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -454,7 +454,7 @@ are always available.  They are listed here in alphabetical order.
               n += 1
 
 
-.. function:: eval(expression, globals=None, locals=None)
+.. function:: eval(expression[, globals[, locals]])
 
    The arguments are a string and optional globals and locals.  If provided,
    *globals* must be a dictionary.  If provided, *locals* can be any mapping

From 7acb22e6e9061f85988c0c6c5ee25ebdf2950841 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 11 Sep 2019 04:09:53 -0700
Subject: [PATCH 0574/2163] bpo-28494: install ziptestdata to fix install bot
 (GH-15902)

(cherry picked from commit c37447481ec8f6d0e49d0587ec0de3f9e7d56b28)

Co-authored-by: Gregory P. Smith 
---
 Makefile.pre.in | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile.pre.in b/Makefile.pre.in
index b02f78412f167b..502317aa0c783a 100644
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -1372,6 +1372,7 @@ LIBSUBDIRS=	tkinter tkinter/test tkinter/test/test_tkinter \
 		test/test_importlib/source \
 		test/test_importlib/zipdata01 \
 		test/test_importlib/zipdata02 \
+		test/ziptestdata \
 		asyncio \
 		test/test_asyncio \
 		collections concurrent concurrent/futures encodings \

From 0ba5dbd992d68d7df23396148ad55624200a1dbc Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 11 Sep 2019 04:17:14 -0700
Subject: [PATCH 0575/2163] bpo-32972: Document IsolatedAsyncioTestCase of
 unittest module (GH-15878) (GH-15918)

* Document `unittest.IsolatedAsyncioTestCase` API
* Add a simple example with respect to order of evaluation of setup and teardown calls.

https://bugs.python.org/issue32972

Automerge-Triggered-By: @asvetlov
(cherry picked from commit 6a9fd66f6e4445a418c43c92585b9e06d76df4b1)

Co-authored-by: Xtreak 
---
 Doc/library/unittest.rst | 76 ++++++++++++++++++++++++++++++++++++++++
 Doc/whatsnew/3.8.rst     | 26 ++++++++++++++
 Misc/NEWS.d/3.8.0b1.rst  |  2 +-
 3 files changed, 103 insertions(+), 1 deletion(-)

diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
index 5ec4b40856ae91..cbee94e52c0aee 100644
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -1486,8 +1486,84 @@ Test cases
       .. versionadded:: 3.8
 
 
+.. class:: IsolatedAsyncioTestCase(methodName='runTest')
 
+   This class provides an API similar to :class:`TestCase` and also accepts
+   coroutines as test functions.
 
+   .. versionadded:: 3.8
+
+   .. coroutinemethod:: asyncSetUp()
+
+      Method called to prepare the test fixture. This is called after :meth:`setUp`.
+      This is called immediately before calling the test method; other than
+      :exc:`AssertionError` or :exc:`SkipTest`, any exception raised by this method
+      will be considered an error rather than a test failure. The default implementation
+      does nothing.
+
+   .. coroutinemethod:: asyncTearDown()
+
+      Method called immediately after the test method has been called and the
+      result recorded.  This is called before :meth:`tearDown`. This is called even if
+      the test method raised an exception, so the implementation in subclasses may need
+      to be particularly careful about checking internal state.  Any exception, other than
+      :exc:`AssertionError` or :exc:`SkipTest`, raised by this method will be
+      considered an additional error rather than a test failure (thus increasing
+      the total number of reported errors). This method will only be called if
+      the :meth:`asyncSetUp` succeeds, regardless of the outcome of the test method.
+      The default implementation does nothing.
+
+   .. method:: addAsyncCleanup(function, /, *args, **kwargs)
+
+      This method accepts a coroutine that can be used as a cleanup function.
+
+   .. method:: run(result=None)
+
+      Sets up a new event loop to run the test, collecting the result into
+      the :class:`TestResult` object passed as *result*.  If *result* is
+      omitted or ``None``, a temporary result object is created (by calling
+      the :meth:`defaultTestResult` method) and used. The result object is
+      returned to :meth:`run`'s caller. At the end of the test all the tasks
+      in the event loop are cancelled.
+
+
+   An example illustrating the order::
+
+      from unittest import IsolatedAsyncioTestCase
+
+      events = []
+
+
+      class Test(IsolatedAsyncioTestCase):
+
+
+          def setUp(self):
+              events.append("setUp")
+
+          async def asyncSetUp(self):
+              self._async_connection = await AsyncConnection()
+              events.append("asyncSetUp")
+
+          async def test_response(self):
+              events.append("test_response")
+              response = await self._async_connection.get("https://example.com")
+              self.assertEqual(response.status_code, 200)
+              self.addAsyncCleanup(self.on_cleanup)
+
+          def tearDown(self):
+              events.append("tearDown")
+
+          async def asyncTearDown(self):
+              await self._async_connection.close()
+              events.append("asyncTearDown")
+
+          async def on_cleanup(self):
+              events.append("cleanup")
+
+      if __name__ == "__main__":
+          unittest.main()
+
+   After running the test ``events`` would contain ``["setUp", "asyncSetUp", "test_response", "asyncTearDown", "tearDown", "cleanup"]``
 
 
 .. class:: FunctionTestCase(testFunc, setUp=None, tearDown=None, description=None)
diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 2b4eb63d61f658..07202ea4ff08d6 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -1111,6 +1111,32 @@ unittest
 * Several mock assert functions now also print a list of actual calls upon
   failure. (Contributed by Petter Strandmark in :issue:`35047`.)
 
+* :mod:`unittest` module gained support for coroutines to be used as test cases
+  with :class:`unittest.IsolatedAsyncioTestCase`.
+  (Contributed by Andrew Svetlov in :issue:`32972`.)
+
+  Example::
+
+    import unittest
+
+
+    class TestRequest(unittest.IsolatedAsyncioTestCase):
+
+        async def asyncSetUp(self):
+            self.connection = await AsyncConnection()
+
+        async def test_get(self):
+            response = await self.connection.get("https://example.com")
+            self.assertEqual(response.status_code, 200)
+
+        async def asyncTearDown(self):
+            await self.connection.close()
+
+
+    if __name__ == "__main__":
+        unittest.main()
+
+
 venv
 ----
 
diff --git a/Misc/NEWS.d/3.8.0b1.rst b/Misc/NEWS.d/3.8.0b1.rst
index 84b0350134fb10..43a88a37c5cb08 100644
--- a/Misc/NEWS.d/3.8.0b1.rst
+++ b/Misc/NEWS.d/3.8.0b1.rst
@@ -808,7 +808,7 @@ detect classes that can be passed to `hex()`, `oct()` and `bin()`.
 .. nonce: LoeUNh
 .. section: Library
 
-Implement ``unittest.AsyncTestCase`` to help testing asyncio-based code.
+Implement ``unittest.IsolatedAsyncioTestCase`` to help testing asyncio-based code.
 
 ..
 

From 0a6693a469cfb1dd5c8048d8cb4231a7b5883251 Mon Sep 17 00:00:00 2001
From: Brett Cannon <54418+brettcannon@users.noreply.github.com>
Date: Wed, 11 Sep 2019 12:38:22 +0100
Subject: [PATCH 0576/2163] [3.8] bpo-37409: fix relative import with no parent
 (GH-14956) (GH-15913)

Relative imports use resolve_name to get the absolute target name,
which first seeks the current module's absolute package name from the globals:
If __package__ (and __spec__.parent) are missing then
import uses __name__, truncating the last segment if
the module is a submodule rather than a package __init__.py
(which it guesses from whether __path__ is defined).

The __name__ attempt should fail if there is no parent package (top level modules),
if __name__ is '__main__' (-m entry points), or both (scripts).
That is, if both __name__ has no subcomponents and the module does not seem
to be a package __init__ module then import should fail..
(cherry picked from commit 92420b3e679959a7d0ce875875601a4cee45231e)

Co-authored-by: Ben Lewis 
---
 Lib/test/test_builtin.py                      |  4 ++++
 Lib/test/test_import/__init__.py              |  5 ++++
 Misc/ACKS                                     |  1 +
 .../2019-08-06-23-39-05.bpo-37409.1qwzn2.rst  |  2 ++
 Python/import.c                               | 23 +++++++++++--------
 5 files changed, 25 insertions(+), 10 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-08-06-23-39-05.bpo-37409.1qwzn2.rst

diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py
index 1100c49e9b8831..0bd4772dbb6e7d 100644
--- a/Lib/test/test_builtin.py
+++ b/Lib/test/test_builtin.py
@@ -160,6 +160,10 @@ def test_import(self):
         self.assertRaises(TypeError, __import__, 1, 2, 3, 4)
         self.assertRaises(ValueError, __import__, '')
         self.assertRaises(TypeError, __import__, 'sys', name='sys')
+        # Relative import outside of a package with no __package__ or __spec__ (bpo-37409).
+        self.assertRaises(ImportError, __import__, '',
+                          {'__package__': None, '__spec__': None, '__name__': '__main__'},
+                          locals={}, fromlist=('foo',), level=1)
         # embedded null character
         self.assertRaises(ModuleNotFoundError, __import__, 'string\x00')
 
diff --git a/Lib/test/test_import/__init__.py b/Lib/test/test_import/__init__.py
index 7c24f0e502323e..f037821dda2ce8 100644
--- a/Lib/test/test_import/__init__.py
+++ b/Lib/test/test_import/__init__.py
@@ -774,6 +774,11 @@ def check_relative():
         ns = dict(__package__=object())
         self.assertRaises(TypeError, check_relative)
 
+    def test_parentless_import_shadowed_by_global(self):
+        # Test as if this were done from the REPL where this error most commonly occurs (bpo-37409).
+        script_helper.assert_python_failure('-W', 'ignore', '-c',
+            "foo = 1; from . import foo")
+
     def test_absolute_import_without_future(self):
         # If explicit relative import syntax is used, then do not try
         # to perform an absolute import in the face of failure.
diff --git a/Misc/ACKS b/Misc/ACKS
index 55aab6b6b2b324..adc15c94cd315d 100644
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -969,6 +969,7 @@ Alain Leufroy
 Mark Levinson
 Mark Levitt
 Ivan Levkivskyi
+Ben Lewis
 William Lewis
 Akira Li
 Robert Li
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-08-06-23-39-05.bpo-37409.1qwzn2.rst b/Misc/NEWS.d/next/Core and Builtins/2019-08-06-23-39-05.bpo-37409.1qwzn2.rst
new file mode 100644
index 00000000000000..9cfa7154802015
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-08-06-23-39-05.bpo-37409.1qwzn2.rst	
@@ -0,0 +1,2 @@
+Ensure explicit relative imports from interactive sessions and scripts (having no parent package) always raise ImportError, rather than treating the current module as the package.
+Patch by Ben Lewis.
diff --git a/Python/import.c b/Python/import.c
index ab7db6bc17f6b1..495012d1c7da66 100644
--- a/Python/import.c
+++ b/Python/import.c
@@ -1601,22 +1601,20 @@ resolve_name(PyObject *name, PyObject *globals, int level)
             if (dot == -2) {
                 goto error;
             }
-
-            if (dot >= 0) {
-                PyObject *substr = PyUnicode_Substring(package, 0, dot);
-                if (substr == NULL) {
-                    goto error;
-                }
-                Py_SETREF(package, substr);
+            else if (dot == -1) {
+                goto no_parent_error;
             }
+            PyObject *substr = PyUnicode_Substring(package, 0, dot);
+            if (substr == NULL) {
+                goto error;
+            }
+            Py_SETREF(package, substr);
         }
     }
 
     last_dot = PyUnicode_GET_LENGTH(package);
     if (last_dot == 0) {
-        PyErr_SetString(PyExc_ImportError,
-                "attempted relative import with no known parent package");
-        goto error;
+        goto no_parent_error;
     }
 
     for (level_up = 1; level_up < level; level_up += 1) {
@@ -1642,6 +1640,11 @@ resolve_name(PyObject *name, PyObject *globals, int level)
     Py_DECREF(base);
     return abs_name;
 
+  no_parent_error:
+    PyErr_SetString(PyExc_ImportError,
+                     "attempted relative import "
+                     "with no known parent package");
+
   error:
     Py_XDECREF(package);
     return NULL;

From f3e430b07975c84cf34c927851df234d04d5753f Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 11 Sep 2019 04:47:16 -0700
Subject: [PATCH 0577/2163] bpo-35066: Make trailing percent test more
 portable. (GH-15907)

Different libc implementations have different behavior when presented with trailing % in strftime strings. To make test_strftime_trailing_percent more portable, compare the output of datetime.strftime directly to that of time.strftime rather than hardcoding.
(cherry picked from commit f2173ae38fa49235c3cdc28ae2ca2e19a375a596)

Co-authored-by: Benjamin Peterson 
---
 Lib/test/datetimetester.py | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py
index b440e5ab5fa65b..d1a3c2ff9abc21 100644
--- a/Lib/test/datetimetester.py
+++ b/Lib/test/datetimetester.py
@@ -1449,15 +1449,20 @@ def test_strftime(self):
         t.strftime("%f")
 
     def test_strftime_trailing_percent(self):
-        # bpo-35066: make sure trailing '%' doesn't cause
-        # datetime's strftime to complain
+        # bpo-35066: Make sure trailing '%' doesn't cause datetime's strftime to
+        # complain. Different libcs have different handling of trailing
+        # percents, so we simply check datetime's strftime acts the same as
+        # time.strftime.
         t = self.theclass(2005, 3, 2)
         try:
             _time.strftime('%')
         except ValueError:
             self.skipTest('time module does not support trailing %')
-        self.assertEqual(t.strftime('%'), '%')
-        self.assertEqual(t.strftime("m:%m d:%d y:%y %"), "m:03 d:02 y:05 %")
+        self.assertEqual(t.strftime('%'), _time.strftime('%', t.timetuple()))
+        self.assertEqual(
+            t.strftime("m:%m d:%d y:%y %"),
+            _time.strftime("m:03 d:02 y:05 %", t.timetuple()),
+        )
 
     def test_format(self):
         dt = self.theclass(2007, 9, 10)

From e3bd941e4e6f4465f17a0e5a4a6bdf4ea0afdd0d Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 11 Sep 2019 05:09:23 -0700
Subject: [PATCH 0578/2163] =?UTF-8?q?bpo-20504=20:=20in=20cgi.py,=20fix=20?=
 =?UTF-8?q?bug=20when=20a=20multipart/form-data=20request=20has=E2=80=A6?=
 =?UTF-8?q?=20(GH-10638)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* bpo-20504 : in cgi.py, fix bug when a multipart/form-data request has no content-length header

* Add Misc/NEWS.d/next file.

* Add rst formatting for NEWS.d/next file

* Reaplce assert by self.assertEqual
(cherry picked from commit 2d7cacacc310b65b43e7e2de89e7722291dea6a4)

Co-authored-by: Pierre Quentel 
---
 Lib/cgi.py                                      |  8 +++++---
 Lib/test/test_cgi.py                            | 17 +++++++++++++++++
 .../2018-11-21-18-05-50.bpo-20504.kG0ub5.rst    |  2 ++
 3 files changed, 24 insertions(+), 3 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2018-11-21-18-05-50.bpo-20504.kG0ub5.rst

diff --git a/Lib/cgi.py b/Lib/cgi.py
index b96bd1f0fe39ac..c22c71b3878516 100755
--- a/Lib/cgi.py
+++ b/Lib/cgi.py
@@ -461,7 +461,7 @@ def __init__(self, fp=None, headers=None, outerboundary=b'',
             if maxlen and clen > maxlen:
                 raise ValueError('Maximum content length exceeded')
         self.length = clen
-        if self.limit is None and clen:
+        if self.limit is None and clen >= 0:
             self.limit = clen
 
         self.list = self.file = None
@@ -642,8 +642,10 @@ def read_multi(self, environ, keep_blank_values, strict_parsing):
             if 'content-length' in headers:
                 del headers['content-length']
 
+            limit = None if self.limit is None \
+                else self.limit - self.bytes_read
             part = klass(self.fp, headers, ib, environ, keep_blank_values,
-                         strict_parsing,self.limit-self.bytes_read,
+                         strict_parsing, limit,
                          self.encoding, self.errors, max_num_fields)
 
             if max_num_fields is not None:
@@ -734,7 +736,7 @@ def read_lines_to_outerboundary(self):
         last_line_lfend = True
         _read = 0
         while 1:
-            if _read >= self.limit:
+            if self.limit is not None and _read >= self.limit:
                 break
             line = self.fp.readline(1<<16) # bytes
             self.bytes_read += len(line)
diff --git a/Lib/test/test_cgi.py b/Lib/test/test_cgi.py
index 0922555982596c..ab8677199f32e7 100644
--- a/Lib/test/test_cgi.py
+++ b/Lib/test/test_cgi.py
@@ -352,6 +352,23 @@ def test_fieldstorage_part_content_length(self):
         self.assertEqual(fs.list[0].name, 'submit-name')
         self.assertEqual(fs.list[0].value, 'Larry')
 
+    def test_field_storage_multipart_no_content_length(self):
+        fp = BytesIO(b"""--MyBoundary
+Content-Disposition: form-data; name="my-arg"; filename="foo"
+
+Test
+
+--MyBoundary--
+""")
+        env = {
+            "REQUEST_METHOD": "POST",
+            "CONTENT_TYPE": "multipart/form-data; boundary=MyBoundary",
+            "wsgi.input": fp,
+        }
+        fields = cgi.FieldStorage(fp, environ=env)
+
+        self.assertEqual(len(fields["my-arg"].file.read()), 5)
+
     def test_fieldstorage_as_context_manager(self):
         fp = BytesIO(b'x' * 10)
         env = {'REQUEST_METHOD': 'PUT'}
diff --git a/Misc/NEWS.d/next/Library/2018-11-21-18-05-50.bpo-20504.kG0ub5.rst b/Misc/NEWS.d/next/Library/2018-11-21-18-05-50.bpo-20504.kG0ub5.rst
new file mode 100644
index 00000000000000..726329ad0d65ad
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-11-21-18-05-50.bpo-20504.kG0ub5.rst
@@ -0,0 +1,2 @@
+Fixes a bug in :mod:`cgi` module when a multipart/form-data request has no
+`Content-Length` header.

From 44e36e80456dabaeb59c6e2a93e0c1322bfeb179 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 11 Sep 2019 05:24:55 -0700
Subject: [PATCH 0579/2163] bpo-35603: Add a note on difflib table header
 interpreted as HTML (GH-11439)

(cherry picked from commit c78dae8d2b890d487e428dce00c7f600612cce7b)

Co-authored-by: Xtreak 
---
 Doc/library/difflib.rst | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst
index e245ab81cfb949..c2a19dc019bb37 100644
--- a/Doc/library/difflib.rst
+++ b/Doc/library/difflib.rst
@@ -127,6 +127,10 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module.
       the next difference highlight at the top of the browser without any leading
       context).
 
+      .. note::
+         *fromdesc* and *todesc* are interpreted as unescaped HTML and should be
+         properly escaped while receiving input from untrusted sources.
+
       .. versionchanged:: 3.5
          *charset* keyword-only argument was added.  The default charset of
          HTML document changed from ``'ISO-8859-1'`` to ``'utf-8'``.

From e0dd713370f94d0f4a59604cc769e0cc643b7740 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 11 Sep 2019 05:28:06 -0700
Subject: [PATCH 0580/2163] bpo-38103: fix conflicting labels in the docs.
 (GH-15906)

(cherry picked from commit 2d8d597bb8f882a7677db5a2739df0e617098634)

Co-authored-by: Ezio Melotti 
---
 Doc/c-api/typeobj.rst      | 4 ++--
 Doc/distutils/examples.rst | 6 +++---
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst
index 638fb0c32d788b..f8e30a0380b9df 100644
--- a/Doc/c-api/typeobj.rst
+++ b/Doc/c-api/typeobj.rst
@@ -20,7 +20,7 @@ functionality.  The fields of the type object are examined in detail in this
 section.  The fields will be described in the order in which they occur in the
 structure.
 
-In addition to the following quick reference, the :ref:`examples`
+In addition to the following quick reference, the :ref:`typedef-examples`
 section provides at-a-glance insight into the meaning and use of
 :c:type:`PyTypeObject`.
 
@@ -2450,7 +2450,7 @@ Slot Type typedefs
 .. c:type:: int (*objobjargproc)(PyObject *, PyObject *, PyObject *)
 
 
-.. _examples:
+.. _typedef-examples:
 
 Examples
 ========
diff --git a/Doc/distutils/examples.rst b/Doc/distutils/examples.rst
index 4ac552c7c6997a..44f70831d0bf87 100644
--- a/Doc/distutils/examples.rst
+++ b/Doc/distutils/examples.rst
@@ -1,8 +1,8 @@
 .. _distutils_examples:
 
-********
-Examples
-********
+******************
+Distutils Examples
+******************
 
 .. include:: ./_setuptools_disclaimer.rst
 

From 01ae0e269899c244c6cf779dc3a7ebdd29248859 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 11 Sep 2019 05:32:16 -0700
Subject: [PATCH 0581/2163] Minor ReST formatting fixes in subprocess docs
 (GH-14876)

(cherry picked from commit 1a13efb7e05b545def26f29c954751fdb6b22fa3)

Co-authored-by: Tim Hoffmann <2836374+timhoffm@users.noreply.github.com>
---
 Doc/library/subprocess.rst | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
index 167ed9a6ead497..954e0fec11828a 100644
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -1048,7 +1048,7 @@ calls these functions.
    Run the command described by *args*.  Wait for command to complete, then
    return the :attr:`~Popen.returncode` attribute.
 
-   Code needing to capture stdout or stderr should use :func:`run` instead:
+   Code needing to capture stdout or stderr should use :func:`run` instead::
 
        run(...).returncode
 
@@ -1076,7 +1076,7 @@ calls these functions.
    :exc:`CalledProcessError` object will have the return code in the
    :attr:`~CalledProcessError.returncode` attribute.
 
-   Code needing to capture stdout or stderr should use :func:`run` instead:
+   Code needing to capture stdout or stderr should use :func:`run` instead::
 
        run(..., check=True)
 
@@ -1198,8 +1198,8 @@ becomes::
    p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
    output = p2.communicate()[0]
 
-The p1.stdout.close() call after starting the p2 is important in order for p1
-to receive a SIGPIPE if p2 exits before p1.
+The ``p1.stdout.close()`` call after starting the p2 is important in order for
+p1 to receive a SIGPIPE if p2 exits before p1.
 
 Alternatively, for trusted input, the shell's own pipeline support may still
 be used directly:

From 3b92ddb7612fd8fe2be95ff3d8022ed18335b13f Mon Sep 17 00:00:00 2001
From: Vinay Sajip 
Date: Wed, 11 Sep 2019 13:39:52 +0100
Subject: [PATCH 0582/2163] [3.8] bpo-35168: Make shlex.punctuation_chars
 read-only (GH-11631) (GH-15927)

(cherry picked from commit 972cf5c06a5ba16ad243a442dbb9c15307fbed95)

Co-authored-by: Alex 
---
 Doc/library/shlex.rst                                     | 8 +++++---
 Lib/shlex.py                                              | 6 +++++-
 Lib/test/test_shlex.py                                    | 7 +++++++
 .../next/Library/2019-01-22-09-23-20.bpo-35168.UGv2yW.rst | 1 +
 4 files changed, 18 insertions(+), 4 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2019-01-22-09-23-20.bpo-35168.UGv2yW.rst

diff --git a/Doc/library/shlex.rst b/Doc/library/shlex.rst
index a8421fdb7008ce..adc23da6d491b7 100644
--- a/Doc/library/shlex.rst
+++ b/Doc/library/shlex.rst
@@ -113,7 +113,9 @@ The :mod:`shlex` module defines the following class:
    characters, those characters will be used as the punctuation characters.  Any
    characters in the :attr:`wordchars` attribute that appear in
    *punctuation_chars* will be removed from :attr:`wordchars`.  See
-   :ref:`improved-shell-compatibility` for more information.
+   :ref:`improved-shell-compatibility` for more information. *punctuation_chars*
+   can be set only upon :class:`~shlex.shlex` instance creation and can't be
+   modified later.
 
    .. versionchanged:: 3.6
       The *punctuation_chars* parameter was added.
@@ -317,8 +319,8 @@ variables which either control lexical analysis or can be used for debugging:
 
 .. attribute:: shlex.punctuation_chars
 
-   Characters that will be considered punctuation. Runs of punctuation
-   characters will be returned as a single token. However, note that no
+   A read-only property. Characters that will be considered punctuation. Runs of
+   punctuation characters will be returned as a single token. However, note that no
    semantic validity checking will be performed: for example, '>>>' could be
    returned as a token, even though it may not be recognised as such by shells.
 
diff --git a/Lib/shlex.py b/Lib/shlex.py
index edea077948608a..ae0f5ddec18716 100644
--- a/Lib/shlex.py
+++ b/Lib/shlex.py
@@ -55,7 +55,7 @@ def __init__(self, instream=None, infile=None, posix=False,
             punctuation_chars = ''
         elif punctuation_chars is True:
             punctuation_chars = '();<>|&'
-        self.punctuation_chars = punctuation_chars
+        self._punctuation_chars = punctuation_chars
         if punctuation_chars:
             # _pushback_chars is a push back queue used by lookahead logic
             self._pushback_chars = deque()
@@ -65,6 +65,10 @@ def __init__(self, instream=None, infile=None, posix=False,
             t = self.wordchars.maketrans(dict.fromkeys(punctuation_chars))
             self.wordchars = self.wordchars.translate(t)
 
+    @property
+    def punctuation_chars(self):
+        return self._punctuation_chars
+
     def push_token(self, tok):
         "Push a token onto the stack popped by the get_token method"
         if self.debug >= 1:
diff --git a/Lib/test/test_shlex.py b/Lib/test/test_shlex.py
index 376c5e88d3819d..a21ccd2fdf44a7 100644
--- a/Lib/test/test_shlex.py
+++ b/Lib/test/test_shlex.py
@@ -353,6 +353,13 @@ def testJoinRoundtrip(self):
                 resplit = shlex.split(joined)
                 self.assertEqual(split_command, resplit)
 
+    def testPunctuationCharsReadOnly(self):
+        punctuation_chars = "/|$%^"
+        shlex_instance = shlex.shlex(punctuation_chars=punctuation_chars)
+        self.assertEqual(shlex_instance.punctuation_chars, punctuation_chars)
+        with self.assertRaises(AttributeError):
+            shlex_instance.punctuation_chars = False
+
 
 # Allow this test to be used with old shlex.py
 if not getattr(shlex, "split", None):
diff --git a/Misc/NEWS.d/next/Library/2019-01-22-09-23-20.bpo-35168.UGv2yW.rst b/Misc/NEWS.d/next/Library/2019-01-22-09-23-20.bpo-35168.UGv2yW.rst
new file mode 100644
index 00000000000000..10684fdbde2f19
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-01-22-09-23-20.bpo-35168.UGv2yW.rst
@@ -0,0 +1 @@
+:attr:`shlex.shlex.punctuation_chars` is now a read-only property.

From 43fb3bb223338511a7aee9b55d75af4a415134dc Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Wed, 11 Sep 2019 06:02:25 -0700
Subject: [PATCH 0583/2163] bpo-35649: update http client example (GH-11441)
 (GH-15930)

(cherry picked from commit 62cf6981425c6a6b136c5e2abef853364f535e9d)

Co-authored-by: Ashwin Ramaswami 
---
 Doc/library/http.client.rst | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst
index 4e761cd39a0109..48bc35ca76cc07 100644
--- a/Doc/library/http.client.rst
+++ b/Doc/library/http.client.rst
@@ -516,8 +516,11 @@ Here is an example session that uses the ``GET`` method::
    >>> # The following example demonstrates reading data in chunks.
    >>> conn.request("GET", "/")
    >>> r1 = conn.getresponse()
-   >>> while not r1.closed:
-   ...     print(r1.read(200))  # 200 bytes
+   >>> while True:
+   ...     chunk = r1.read(200)  # 200 bytes
+   ...     if not chunk:
+   ...          break
+   ...     print(repr(chunk))
    b'\n 0x%0*lx (%lu bytes)\n",
-        label,
-        (int)sizeof(entropy) * 2, entropy,
-        (unsigned long)sizeof(entropy));
+    fprintf(stderr, "Entropy: %s --> 0x%0*lx (%lu bytes)\n", label,
+            (int)sizeof(entropy) * 2, entropy, (unsigned long)sizeof(entropy));
   }
   return entropy;
 }
 
 static unsigned long
-generate_hash_secret_salt(XML_Parser parser)
-{
+generate_hash_secret_salt(XML_Parser parser) {
   unsigned long entropy;
   (void)parser;
 
@@ -848,20 +811,20 @@ generate_hash_secret_salt(XML_Parser parser)
   return ENTROPY_DEBUG("arc4random", entropy);
 #else
   /* Try high quality providers first .. */
-#ifdef _WIN32
-  if (writeRandomBytes_RtlGenRandom((void *)&entropy, sizeof(entropy))) {
-    return ENTROPY_DEBUG("RtlGenRandom", entropy);
+#  ifdef _WIN32
+  if (writeRandomBytes_rand_s((void *)&entropy, sizeof(entropy))) {
+    return ENTROPY_DEBUG("rand_s", entropy);
   }
-#elif defined(HAVE_GETRANDOM) || defined(HAVE_SYSCALL_GETRANDOM)
+#  elif defined(HAVE_GETRANDOM) || defined(HAVE_SYSCALL_GETRANDOM)
   if (writeRandomBytes_getrandom_nonblock((void *)&entropy, sizeof(entropy))) {
     return ENTROPY_DEBUG("getrandom", entropy);
   }
-#endif
-#if ! defined(_WIN32) && defined(XML_DEV_URANDOM)
+#  endif
+#  if ! defined(_WIN32) && defined(XML_DEV_URANDOM)
   if (writeRandomBytes_dev_urandom((void *)&entropy, sizeof(entropy))) {
     return ENTROPY_DEBUG("/dev/urandom", entropy);
   }
-#endif  /* ! defined(_WIN32) && defined(XML_DEV_URANDOM) */
+#  endif /* ! defined(_WIN32) && defined(XML_DEV_URANDOM) */
   /* .. and self-made low quality for backup: */
 
   /* Process ID is 0 bits entropy if attacker has local access */
@@ -872,7 +835,7 @@ generate_hash_secret_salt(XML_Parser parser)
     return ENTROPY_DEBUG("fallback(4)", entropy * 2147483647);
   } else {
     return ENTROPY_DEBUG("fallback(8)",
-        entropy * (unsigned long)2305843009213693951ULL);
+                         entropy * (unsigned long)2305843009213693951ULL);
   }
 #endif
 }
@@ -884,49 +847,43 @@ get_hash_secret_salt(XML_Parser parser) {
   return parser->m_hash_secret_salt;
 }
 
-static XML_Bool  /* only valid for root parser */
-startParsing(XML_Parser parser)
-{
-    /* hash functions must be initialized before setContext() is called */
-    if (parser->m_hash_secret_salt == 0)
-      parser->m_hash_secret_salt = generate_hash_secret_salt(parser);
-    if (parser->m_ns) {
-      /* implicit context only set for root parser, since child
-         parsers (i.e. external entity parsers) will inherit it
-      */
-      return setContext(parser, implicitContext);
-    }
-    return XML_TRUE;
+static XML_Bool /* only valid for root parser */
+startParsing(XML_Parser parser) {
+  /* hash functions must be initialized before setContext() is called */
+  if (parser->m_hash_secret_salt == 0)
+    parser->m_hash_secret_salt = generate_hash_secret_salt(parser);
+  if (parser->m_ns) {
+    /* implicit context only set for root parser, since child
+       parsers (i.e. external entity parsers) will inherit it
+    */
+    return setContext(parser, implicitContext);
+  }
+  return XML_TRUE;
 }
 
 XML_Parser XMLCALL
 XML_ParserCreate_MM(const XML_Char *encodingName,
                     const XML_Memory_Handling_Suite *memsuite,
-                    const XML_Char *nameSep)
-{
+                    const XML_Char *nameSep) {
   return parserCreate(encodingName, memsuite, nameSep, NULL);
 }
 
 static XML_Parser
 parserCreate(const XML_Char *encodingName,
-             const XML_Memory_Handling_Suite *memsuite,
-             const XML_Char *nameSep,
-             DTD *dtd)
-{
+             const XML_Memory_Handling_Suite *memsuite, const XML_Char *nameSep,
+             DTD *dtd) {
   XML_Parser parser;
 
   if (memsuite) {
     XML_Memory_Handling_Suite *mtemp;
-    parser = (XML_Parser)
-      memsuite->malloc_fcn(sizeof(struct XML_ParserStruct));
+    parser = (XML_Parser)memsuite->malloc_fcn(sizeof(struct XML_ParserStruct));
     if (parser != NULL) {
       mtemp = (XML_Memory_Handling_Suite *)&(parser->m_mem);
       mtemp->malloc_fcn = memsuite->malloc_fcn;
       mtemp->realloc_fcn = memsuite->realloc_fcn;
       mtemp->free_fcn = memsuite->free_fcn;
     }
-  }
-  else {
+  } else {
     XML_Memory_Handling_Suite *mtemp;
     parser = (XML_Parser)malloc(sizeof(struct XML_ParserStruct));
     if (parser != NULL) {
@@ -937,27 +894,30 @@ parserCreate(const XML_Char *encodingName,
     }
   }
 
-  if (!parser)
+  if (! parser)
     return parser;
 
   parser->m_buffer = NULL;
   parser->m_bufferLim = NULL;
 
   parser->m_attsSize = INIT_ATTS_SIZE;
-  parser->m_atts = (ATTRIBUTE *)MALLOC(parser, parser->m_attsSize * sizeof(ATTRIBUTE));
+  parser->m_atts
+      = (ATTRIBUTE *)MALLOC(parser, parser->m_attsSize * sizeof(ATTRIBUTE));
   if (parser->m_atts == NULL) {
     FREE(parser, parser);
     return NULL;
   }
 #ifdef XML_ATTR_INFO
-  parser->m_attInfo = (XML_AttrInfo*)MALLOC(parser, parser->m_attsSize * sizeof(XML_AttrInfo));
+  parser->m_attInfo = (XML_AttrInfo *)MALLOC(
+      parser, parser->m_attsSize * sizeof(XML_AttrInfo));
   if (parser->m_attInfo == NULL) {
     FREE(parser, parser->m_atts);
     FREE(parser, parser);
     return NULL;
   }
 #endif
-  parser->m_dataBuf = (XML_Char *)MALLOC(parser, INIT_DATA_BUF_SIZE * sizeof(XML_Char));
+  parser->m_dataBuf
+      = (XML_Char *)MALLOC(parser, INIT_DATA_BUF_SIZE * sizeof(XML_Char));
   if (parser->m_dataBuf == NULL) {
     FREE(parser, parser->m_atts);
 #ifdef XML_ATTR_INFO
@@ -1007,7 +967,7 @@ parserCreate(const XML_Char *encodingName,
   poolInit(&parser->m_temp2Pool, &(parser->m_mem));
   parserInit(parser, encodingName);
 
-  if (encodingName && !parser->m_protocolEncodingName) {
+  if (encodingName && ! parser->m_protocolEncodingName) {
     XML_ParserFree(parser);
     return NULL;
   }
@@ -1016,8 +976,7 @@ parserCreate(const XML_Char *encodingName,
     parser->m_ns = XML_TRUE;
     parser->m_internalEncoding = XmlGetInternalEncodingNS();
     parser->m_namespaceSeparator = *nameSep;
-  }
-  else {
+  } else {
     parser->m_internalEncoding = XmlGetInternalEncoding();
   }
 
@@ -1025,8 +984,7 @@ parserCreate(const XML_Char *encodingName,
 }
 
 static void
-parserInit(XML_Parser parser, const XML_Char *encodingName)
-{
+parserInit(XML_Parser parser, const XML_Char *encodingName) {
   parser->m_processor = prologInitProcessor;
   XmlPrologStateInit(&parser->m_prologState);
   if (encodingName != NULL) {
@@ -1099,8 +1057,7 @@ parserInit(XML_Parser parser, const XML_Char *encodingName)
 
 /* moves list of bindings to m_freeBindingList */
 static void FASTCALL
-moveToFreeBindingList(XML_Parser parser, BINDING *bindings)
-{
+moveToFreeBindingList(XML_Parser parser, BINDING *bindings) {
   while (bindings) {
     BINDING *b = bindings;
     bindings = bindings->nextTagBinding;
@@ -1110,13 +1067,12 @@ moveToFreeBindingList(XML_Parser parser, BINDING *bindings)
 }
 
 XML_Bool XMLCALL
-XML_ParserReset(XML_Parser parser, const XML_Char *encodingName)
-{
+XML_ParserReset(XML_Parser parser, const XML_Char *encodingName) {
   TAG *tStk;
   OPEN_INTERNAL_ENTITY *openEntityList;
 
   if (parser == NULL)
-      return XML_FALSE;
+    return XML_FALSE;
 
   if (parser->m_parentParser)
     return XML_FALSE;
@@ -1152,15 +1108,15 @@ XML_ParserReset(XML_Parser parser, const XML_Char *encodingName)
 }
 
 enum XML_Status XMLCALL
-XML_SetEncoding(XML_Parser parser, const XML_Char *encodingName)
-{
+XML_SetEncoding(XML_Parser parser, const XML_Char *encodingName) {
   if (parser == NULL)
-      return XML_STATUS_ERROR;
+    return XML_STATUS_ERROR;
   /* Block after XML_Parse()/XML_ParseBuffer() has been called.
      XXX There's no way for the caller to determine which of the
      XXX possible error cases caused the XML_STATUS_ERROR return.
   */
-  if (parser->m_parsingStatus.parsing == XML_PARSING || parser->m_parsingStatus.parsing == XML_SUSPENDED)
+  if (parser->m_parsingStatus.parsing == XML_PARSING
+      || parser->m_parsingStatus.parsing == XML_SUSPENDED)
     return XML_STATUS_ERROR;
 
   /* Get rid of any previous encoding name */
@@ -1172,17 +1128,15 @@ XML_SetEncoding(XML_Parser parser, const XML_Char *encodingName)
   else {
     /* Copy the new encoding name into allocated memory */
     parser->m_protocolEncodingName = copyString(encodingName, &(parser->m_mem));
-    if (!parser->m_protocolEncodingName)
+    if (! parser->m_protocolEncodingName)
       return XML_STATUS_ERROR;
   }
   return XML_STATUS_OK;
 }
 
 XML_Parser XMLCALL
-XML_ExternalEntityParserCreate(XML_Parser oldParser,
-                               const XML_Char *context,
-                               const XML_Char *encodingName)
-{
+XML_ExternalEntityParserCreate(XML_Parser oldParser, const XML_Char *context,
+                               const XML_Char *encodingName) {
   XML_Parser parser = oldParser;
   DTD *newDtd = NULL;
   DTD *oldDtd;
@@ -1206,7 +1160,7 @@ XML_ExternalEntityParserCreate(XML_Parser oldParser,
   XML_AttlistDeclHandler oldAttlistDeclHandler;
   XML_EntityDeclHandler oldEntityDeclHandler;
   XML_XmlDeclHandler oldXmlDeclHandler;
-  ELEMENT_TYPE * oldDeclElementType;
+  ELEMENT_TYPE *oldDeclElementType;
 
   void *oldUserData;
   void *oldHandlerArg;
@@ -1269,7 +1223,7 @@ XML_ExternalEntityParserCreate(XML_Parser oldParser,
   oldhash_secret_salt = parser->m_hash_secret_salt;
 
 #ifdef XML_DTD
-  if (!context)
+  if (! context)
     newDtd = oldDtd;
 #endif /* XML_DTD */
 
@@ -1282,12 +1236,11 @@ XML_ExternalEntityParserCreate(XML_Parser oldParser,
     XML_Char tmp[2];
     *tmp = parser->m_namespaceSeparator;
     parser = parserCreate(encodingName, &parser->m_mem, tmp, newDtd);
-  }
-  else {
+  } else {
     parser = parserCreate(encodingName, &parser->m_mem, NULL, newDtd);
   }
 
-  if (!parser)
+  if (! parser)
     return NULL;
 
   parser->m_startElementHandler = oldStartElementHandler;
@@ -1327,21 +1280,20 @@ XML_ExternalEntityParserCreate(XML_Parser oldParser,
   parser->m_prologState.inEntityValue = oldInEntityValue;
   if (context) {
 #endif /* XML_DTD */
-    if (!dtdCopy(oldParser, parser->m_dtd, oldDtd, &parser->m_mem)
-      || !setContext(parser, context)) {
+    if (! dtdCopy(oldParser, parser->m_dtd, oldDtd, &parser->m_mem)
+        || ! setContext(parser, context)) {
       XML_ParserFree(parser);
       return NULL;
     }
     parser->m_processor = externalEntityInitProcessor;
 #ifdef XML_DTD
-  }
-  else {
-    /* The DTD instance referenced by parser->m_dtd is shared between the document's
-       root parser and external PE parsers, therefore one does not need to
-       call setContext. In addition, one also *must* not call setContext,
-       because this would overwrite existing prefix->binding pointers in
-       parser->m_dtd with ones that get destroyed with the external PE parser.
-       This would leave those prefixes with dangling pointers.
+  } else {
+    /* The DTD instance referenced by parser->m_dtd is shared between the
+       document's root parser and external PE parsers, therefore one does not
+       need to call setContext. In addition, one also *must* not call
+       setContext, because this would overwrite existing prefix->binding
+       pointers in parser->m_dtd with ones that get destroyed with the external
+       PE parser. This would leave those prefixes with dangling pointers.
     */
     parser->m_isParamEntity = XML_TRUE;
     XmlPrologStateInitExternalEntity(&parser->m_prologState);
@@ -1352,11 +1304,10 @@ XML_ExternalEntityParserCreate(XML_Parser oldParser,
 }
 
 static void FASTCALL
-destroyBindings(BINDING *bindings, XML_Parser parser)
-{
+destroyBindings(BINDING *bindings, XML_Parser parser) {
   for (;;) {
     BINDING *b = bindings;
-    if (!b)
+    if (! b)
       break;
     bindings = b->nextTagBinding;
     FREE(parser, b->uri);
@@ -1365,8 +1316,7 @@ destroyBindings(BINDING *bindings, XML_Parser parser)
 }
 
 void XMLCALL
-XML_ParserFree(XML_Parser parser)
-{
+XML_ParserFree(XML_Parser parser) {
   TAG *tagList;
   OPEN_INTERNAL_ENTITY *entityList;
   if (parser == NULL)
@@ -1411,11 +1361,12 @@ XML_ParserFree(XML_Parser parser)
   /* external parameter entity parsers share the DTD structure
      parser->m_dtd with the root parser, so we must not destroy it
   */
-  if (!parser->m_isParamEntity && parser->m_dtd)
+  if (! parser->m_isParamEntity && parser->m_dtd)
 #else
   if (parser->m_dtd)
 #endif /* XML_DTD */
-    dtdDestroy(parser->m_dtd, (XML_Bool)!parser->m_parentParser, &parser->m_mem);
+    dtdDestroy(parser->m_dtd, (XML_Bool)! parser->m_parentParser,
+               &parser->m_mem);
   FREE(parser, (void *)parser->m_atts);
 #ifdef XML_ATTR_INFO
   FREE(parser, (void *)parser->m_attInfo);
@@ -1431,20 +1382,19 @@ XML_ParserFree(XML_Parser parser)
 }
 
 void XMLCALL
-XML_UseParserAsHandlerArg(XML_Parser parser)
-{
+XML_UseParserAsHandlerArg(XML_Parser parser) {
   if (parser != NULL)
     parser->m_handlerArg = parser;
 }
 
 enum XML_Error XMLCALL
-XML_UseForeignDTD(XML_Parser parser, XML_Bool useDTD)
-{
+XML_UseForeignDTD(XML_Parser parser, XML_Bool useDTD) {
   if (parser == NULL)
     return XML_ERROR_INVALID_ARGUMENT;
 #ifdef XML_DTD
   /* block after XML_Parse()/XML_ParseBuffer() has been called */
-  if (parser->m_parsingStatus.parsing == XML_PARSING || parser->m_parsingStatus.parsing == XML_SUSPENDED)
+  if (parser->m_parsingStatus.parsing == XML_PARSING
+      || parser->m_parsingStatus.parsing == XML_SUSPENDED)
     return XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING;
   parser->m_useForeignDTD = useDTD;
   return XML_ERROR_NONE;
@@ -1454,19 +1404,18 @@ XML_UseForeignDTD(XML_Parser parser, XML_Bool useDTD)
 }
 
 void XMLCALL
-XML_SetReturnNSTriplet(XML_Parser parser, int do_nst)
-{
+XML_SetReturnNSTriplet(XML_Parser parser, int do_nst) {
   if (parser == NULL)
     return;
   /* block after XML_Parse()/XML_ParseBuffer() has been called */
-  if (parser->m_parsingStatus.parsing == XML_PARSING || parser->m_parsingStatus.parsing == XML_SUSPENDED)
+  if (parser->m_parsingStatus.parsing == XML_PARSING
+      || parser->m_parsingStatus.parsing == XML_SUSPENDED)
     return;
   parser->m_ns_triplets = do_nst ? XML_TRUE : XML_FALSE;
 }
 
 void XMLCALL
-XML_SetUserData(XML_Parser parser, void *p)
-{
+XML_SetUserData(XML_Parser parser, void *p) {
   if (parser == NULL)
     return;
   if (parser->m_handlerArg == parser->m_userData)
@@ -1476,49 +1425,43 @@ XML_SetUserData(XML_Parser parser, void *p)
 }
 
 enum XML_Status XMLCALL
-XML_SetBase(XML_Parser parser, const XML_Char *p)
-{
+XML_SetBase(XML_Parser parser, const XML_Char *p) {
   if (parser == NULL)
     return XML_STATUS_ERROR;
   if (p) {
     p = poolCopyString(&parser->m_dtd->pool, p);
-    if (!p)
+    if (! p)
       return XML_STATUS_ERROR;
     parser->m_curBase = p;
-  }
-  else
+  } else
     parser->m_curBase = NULL;
   return XML_STATUS_OK;
 }
 
-const XML_Char * XMLCALL
-XML_GetBase(XML_Parser parser)
-{
+const XML_Char *XMLCALL
+XML_GetBase(XML_Parser parser) {
   if (parser == NULL)
     return NULL;
   return parser->m_curBase;
 }
 
 int XMLCALL
-XML_GetSpecifiedAttributeCount(XML_Parser parser)
-{
+XML_GetSpecifiedAttributeCount(XML_Parser parser) {
   if (parser == NULL)
     return -1;
   return parser->m_nSpecifiedAtts;
 }
 
 int XMLCALL
-XML_GetIdAttributeIndex(XML_Parser parser)
-{
+XML_GetIdAttributeIndex(XML_Parser parser) {
   if (parser == NULL)
     return -1;
   return parser->m_idAttIndex;
 }
 
 #ifdef XML_ATTR_INFO
-const XML_AttrInfo * XMLCALL
-XML_GetAttributeInfo(XML_Parser parser)
-{
+const XML_AttrInfo *XMLCALL
+XML_GetAttributeInfo(XML_Parser parser) {
   if (parser == NULL)
     return NULL;
   return parser->m_attInfo;
@@ -1526,10 +1469,8 @@ XML_GetAttributeInfo(XML_Parser parser)
 #endif
 
 void XMLCALL
-XML_SetElementHandler(XML_Parser parser,
-                      XML_StartElementHandler start,
-                      XML_EndElementHandler end)
-{
+XML_SetElementHandler(XML_Parser parser, XML_StartElementHandler start,
+                      XML_EndElementHandler end) {
   if (parser == NULL)
     return;
   parser->m_startElementHandler = start;
@@ -1537,39 +1478,33 @@ XML_SetElementHandler(XML_Parser parser,
 }
 
 void XMLCALL
-XML_SetStartElementHandler(XML_Parser parser,
-                           XML_StartElementHandler start) {
+XML_SetStartElementHandler(XML_Parser parser, XML_StartElementHandler start) {
   if (parser != NULL)
     parser->m_startElementHandler = start;
 }
 
 void XMLCALL
-XML_SetEndElementHandler(XML_Parser parser,
-                         XML_EndElementHandler end) {
+XML_SetEndElementHandler(XML_Parser parser, XML_EndElementHandler end) {
   if (parser != NULL)
     parser->m_endElementHandler = end;
 }
 
 void XMLCALL
 XML_SetCharacterDataHandler(XML_Parser parser,
-                            XML_CharacterDataHandler handler)
-{
+                            XML_CharacterDataHandler handler) {
   if (parser != NULL)
     parser->m_characterDataHandler = handler;
 }
 
 void XMLCALL
 XML_SetProcessingInstructionHandler(XML_Parser parser,
-                                    XML_ProcessingInstructionHandler handler)
-{
+                                    XML_ProcessingInstructionHandler handler) {
   if (parser != NULL)
     parser->m_processingInstructionHandler = handler;
 }
 
 void XMLCALL
-XML_SetCommentHandler(XML_Parser parser,
-                      XML_CommentHandler handler)
-{
+XML_SetCommentHandler(XML_Parser parser, XML_CommentHandler handler) {
   if (parser != NULL)
     parser->m_commentHandler = handler;
 }
@@ -1577,8 +1512,7 @@ XML_SetCommentHandler(XML_Parser parser,
 void XMLCALL
 XML_SetCdataSectionHandler(XML_Parser parser,
                            XML_StartCdataSectionHandler start,
-                           XML_EndCdataSectionHandler end)
-{
+                           XML_EndCdataSectionHandler end) {
   if (parser == NULL)
     return;
   parser->m_startCdataSectionHandler = start;
@@ -1600,9 +1534,7 @@ XML_SetEndCdataSectionHandler(XML_Parser parser,
 }
 
 void XMLCALL
-XML_SetDefaultHandler(XML_Parser parser,
-                      XML_DefaultHandler handler)
-{
+XML_SetDefaultHandler(XML_Parser parser, XML_DefaultHandler handler) {
   if (parser == NULL)
     return;
   parser->m_defaultHandler = handler;
@@ -1610,9 +1542,7 @@ XML_SetDefaultHandler(XML_Parser parser,
 }
 
 void XMLCALL
-XML_SetDefaultHandlerExpand(XML_Parser parser,
-                            XML_DefaultHandler handler)
-{
+XML_SetDefaultHandlerExpand(XML_Parser parser, XML_DefaultHandler handler) {
   if (parser == NULL)
     return;
   parser->m_defaultHandler = handler;
@@ -1620,10 +1550,8 @@ XML_SetDefaultHandlerExpand(XML_Parser parser,
 }
 
 void XMLCALL
-XML_SetDoctypeDeclHandler(XML_Parser parser,
-                          XML_StartDoctypeDeclHandler start,
-                          XML_EndDoctypeDeclHandler end)
-{
+XML_SetDoctypeDeclHandler(XML_Parser parser, XML_StartDoctypeDeclHandler start,
+                          XML_EndDoctypeDeclHandler end) {
   if (parser == NULL)
     return;
   parser->m_startDoctypeDeclHandler = start;
@@ -1638,24 +1566,20 @@ XML_SetStartDoctypeDeclHandler(XML_Parser parser,
 }
 
 void XMLCALL
-XML_SetEndDoctypeDeclHandler(XML_Parser parser,
-                             XML_EndDoctypeDeclHandler end) {
+XML_SetEndDoctypeDeclHandler(XML_Parser parser, XML_EndDoctypeDeclHandler end) {
   if (parser != NULL)
     parser->m_endDoctypeDeclHandler = end;
 }
 
 void XMLCALL
 XML_SetUnparsedEntityDeclHandler(XML_Parser parser,
-                                 XML_UnparsedEntityDeclHandler handler)
-{
+                                 XML_UnparsedEntityDeclHandler handler) {
   if (parser != NULL)
     parser->m_unparsedEntityDeclHandler = handler;
 }
 
 void XMLCALL
-XML_SetNotationDeclHandler(XML_Parser parser,
-                           XML_NotationDeclHandler handler)
-{
+XML_SetNotationDeclHandler(XML_Parser parser, XML_NotationDeclHandler handler) {
   if (parser != NULL)
     parser->m_notationDeclHandler = handler;
 }
@@ -1663,8 +1587,7 @@ XML_SetNotationDeclHandler(XML_Parser parser,
 void XMLCALL
 XML_SetNamespaceDeclHandler(XML_Parser parser,
                             XML_StartNamespaceDeclHandler start,
-                            XML_EndNamespaceDeclHandler end)
-{
+                            XML_EndNamespaceDeclHandler end) {
   if (parser == NULL)
     return;
   parser->m_startNamespaceDeclHandler = start;
@@ -1687,23 +1610,20 @@ XML_SetEndNamespaceDeclHandler(XML_Parser parser,
 
 void XMLCALL
 XML_SetNotStandaloneHandler(XML_Parser parser,
-                            XML_NotStandaloneHandler handler)
-{
+                            XML_NotStandaloneHandler handler) {
   if (parser != NULL)
     parser->m_notStandaloneHandler = handler;
 }
 
 void XMLCALL
 XML_SetExternalEntityRefHandler(XML_Parser parser,
-                                XML_ExternalEntityRefHandler handler)
-{
+                                XML_ExternalEntityRefHandler handler) {
   if (parser != NULL)
     parser->m_externalEntityRefHandler = handler;
 }
 
 void XMLCALL
-XML_SetExternalEntityRefHandlerArg(XML_Parser parser, void *arg)
-{
+XML_SetExternalEntityRefHandlerArg(XML_Parser parser, void *arg) {
   if (parser == NULL)
     return;
   if (arg)
@@ -1714,17 +1634,14 @@ XML_SetExternalEntityRefHandlerArg(XML_Parser parser, void *arg)
 
 void XMLCALL
 XML_SetSkippedEntityHandler(XML_Parser parser,
-                            XML_SkippedEntityHandler handler)
-{
+                            XML_SkippedEntityHandler handler) {
   if (parser != NULL)
     parser->m_skippedEntityHandler = handler;
 }
 
 void XMLCALL
 XML_SetUnknownEncodingHandler(XML_Parser parser,
-                              XML_UnknownEncodingHandler handler,
-                              void *data)
-{
+                              XML_UnknownEncodingHandler handler, void *data) {
   if (parser == NULL)
     return;
   parser->m_unknownEncodingHandler = handler;
@@ -1732,44 +1649,37 @@ XML_SetUnknownEncodingHandler(XML_Parser parser,
 }
 
 void XMLCALL
-XML_SetElementDeclHandler(XML_Parser parser,
-                          XML_ElementDeclHandler eldecl)
-{
+XML_SetElementDeclHandler(XML_Parser parser, XML_ElementDeclHandler eldecl) {
   if (parser != NULL)
     parser->m_elementDeclHandler = eldecl;
 }
 
 void XMLCALL
-XML_SetAttlistDeclHandler(XML_Parser parser,
-                          XML_AttlistDeclHandler attdecl)
-{
+XML_SetAttlistDeclHandler(XML_Parser parser, XML_AttlistDeclHandler attdecl) {
   if (parser != NULL)
     parser->m_attlistDeclHandler = attdecl;
 }
 
 void XMLCALL
-XML_SetEntityDeclHandler(XML_Parser parser,
-                         XML_EntityDeclHandler handler)
-{
+XML_SetEntityDeclHandler(XML_Parser parser, XML_EntityDeclHandler handler) {
   if (parser != NULL)
     parser->m_entityDeclHandler = handler;
 }
 
 void XMLCALL
-XML_SetXmlDeclHandler(XML_Parser parser,
-                      XML_XmlDeclHandler handler) {
+XML_SetXmlDeclHandler(XML_Parser parser, XML_XmlDeclHandler handler) {
   if (parser != NULL)
     parser->m_xmlDeclHandler = handler;
 }
 
 int XMLCALL
 XML_SetParamEntityParsing(XML_Parser parser,
-                          enum XML_ParamEntityParsing peParsing)
-{
+                          enum XML_ParamEntityParsing peParsing) {
   if (parser == NULL)
     return 0;
   /* block after XML_Parse()/XML_ParseBuffer() has been called */
-  if (parser->m_parsingStatus.parsing == XML_PARSING || parser->m_parsingStatus.parsing == XML_SUSPENDED)
+  if (parser->m_parsingStatus.parsing == XML_PARSING
+      || parser->m_parsingStatus.parsing == XML_SUSPENDED)
     return 0;
 #ifdef XML_DTD
   parser->m_paramEntityParsing = peParsing;
@@ -1780,23 +1690,21 @@ XML_SetParamEntityParsing(XML_Parser parser,
 }
 
 int XMLCALL
-XML_SetHashSalt(XML_Parser parser,
-                unsigned long hash_salt)
-{
+XML_SetHashSalt(XML_Parser parser, unsigned long hash_salt) {
   if (parser == NULL)
     return 0;
   if (parser->m_parentParser)
     return XML_SetHashSalt(parser->m_parentParser, hash_salt);
   /* block after XML_Parse()/XML_ParseBuffer() has been called */
-  if (parser->m_parsingStatus.parsing == XML_PARSING || parser->m_parsingStatus.parsing == XML_SUSPENDED)
+  if (parser->m_parsingStatus.parsing == XML_PARSING
+      || parser->m_parsingStatus.parsing == XML_SUSPENDED)
     return 0;
   parser->m_hash_secret_salt = hash_salt;
   return 1;
 }
 
 enum XML_Status XMLCALL
-XML_Parse(XML_Parser parser, const char *s, int len, int isFinal)
-{
+XML_Parse(XML_Parser parser, const char *s, int len, int isFinal) {
   if ((parser == NULL) || (len < 0) || ((s == NULL) && (len != 0))) {
     if (parser != NULL)
       parser->m_errorCode = XML_ERROR_INVALID_ARGUMENT;
@@ -1810,7 +1718,7 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal)
     parser->m_errorCode = XML_ERROR_FINISHED;
     return XML_STATUS_ERROR;
   case XML_INITIALIZED:
-    if (parser->m_parentParser == NULL && !startParsing(parser)) {
+    if (parser->m_parentParser == NULL && ! startParsing(parser)) {
       parser->m_errorCode = XML_ERROR_NO_MEMORY;
       return XML_STATUS_ERROR;
     }
@@ -1821,7 +1729,7 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal)
 
   if (len == 0) {
     parser->m_parsingStatus.finalBuffer = (XML_Bool)isFinal;
-    if (!isFinal)
+    if (! isFinal)
       return XML_STATUS_OK;
     parser->m_positionPtr = parser->m_bufferPtr;
     parser->m_parseEndPtr = parser->m_bufferEnd;
@@ -1830,7 +1738,9 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal)
        data are the final chunk of input, then we have to check them again
        to detect errors based on that fact.
     */
-    parser->m_errorCode = parser->m_processor(parser, parser->m_bufferPtr, parser->m_parseEndPtr, &parser->m_bufferPtr);
+    parser->m_errorCode
+        = parser->m_processor(parser, parser->m_bufferPtr,
+                              parser->m_parseEndPtr, &parser->m_bufferPtr);
 
     if (parser->m_errorCode == XML_ERROR_NONE) {
       switch (parser->m_parsingStatus.parsing) {
@@ -1847,7 +1757,8 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal)
          *
          * LCOV_EXCL_START
          */
-        XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, parser->m_bufferPtr, &parser->m_position);
+        XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr,
+                          parser->m_bufferPtr, &parser->m_position);
         parser->m_positionPtr = parser->m_bufferPtr;
         return XML_STATUS_SUSPENDED;
         /* LCOV_EXCL_STOP */
@@ -1870,23 +1781,23 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal)
     enum XML_Status result;
     /* Detect overflow (a+b > MAX <==> b > MAX-a) */
     if (len > ((XML_Size)-1) / 2 - parser->m_parseEndByteIndex) {
-       parser->m_errorCode = XML_ERROR_NO_MEMORY;
-       parser->m_eventPtr = parser->m_eventEndPtr = NULL;
-       parser->m_processor = errorProcessor;
-       return XML_STATUS_ERROR;
+      parser->m_errorCode = XML_ERROR_NO_MEMORY;
+      parser->m_eventPtr = parser->m_eventEndPtr = NULL;
+      parser->m_processor = errorProcessor;
+      return XML_STATUS_ERROR;
     }
     parser->m_parseEndByteIndex += len;
     parser->m_positionPtr = s;
     parser->m_parsingStatus.finalBuffer = (XML_Bool)isFinal;
 
-    parser->m_errorCode = parser->m_processor(parser, s, parser->m_parseEndPtr = s + len, &end);
+    parser->m_errorCode
+        = parser->m_processor(parser, s, parser->m_parseEndPtr = s + len, &end);
 
     if (parser->m_errorCode != XML_ERROR_NONE) {
       parser->m_eventEndPtr = parser->m_eventPtr;
       parser->m_processor = errorProcessor;
       return XML_STATUS_ERROR;
-    }
-    else {
+    } else {
       switch (parser->m_parsingStatus.parsing) {
       case XML_SUSPENDED:
         result = XML_STATUS_SUSPENDED;
@@ -1903,10 +1814,12 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal)
       }
     }
 
-    XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, end, &parser->m_position);
+    XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, end,
+                      &parser->m_position);
     nLeftOver = s + len - end;
     if (nLeftOver) {
-      if (parser->m_buffer == NULL || nLeftOver > parser->m_bufferLim - parser->m_buffer) {
+      if (parser->m_buffer == NULL
+          || nLeftOver > parser->m_bufferLim - parser->m_buffer) {
         /* avoid _signed_ integer overflow */
         char *temp = NULL;
         const int bytesToAllocate = (int)((unsigned)len * 2U);
@@ -1932,7 +1845,7 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal)
     parser->m_eventEndPtr = parser->m_bufferPtr;
     return result;
   }
-#endif  /* not defined XML_CONTEXT_BYTES */
+#endif /* not defined XML_CONTEXT_BYTES */
   else {
     void *buff = XML_GetBuffer(parser, len);
     if (buff == NULL)
@@ -1945,8 +1858,7 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal)
 }
 
 enum XML_Status XMLCALL
-XML_ParseBuffer(XML_Parser parser, int len, int isFinal)
-{
+XML_ParseBuffer(XML_Parser parser, int len, int isFinal) {
   const char *start;
   enum XML_Status result = XML_STATUS_OK;
 
@@ -1960,7 +1872,7 @@ XML_ParseBuffer(XML_Parser parser, int len, int isFinal)
     parser->m_errorCode = XML_ERROR_FINISHED;
     return XML_STATUS_ERROR;
   case XML_INITIALIZED:
-    if (parser->m_parentParser == NULL && !startParsing(parser)) {
+    if (parser->m_parentParser == NULL && ! startParsing(parser)) {
       parser->m_errorCode = XML_ERROR_NO_MEMORY;
       return XML_STATUS_ERROR;
     }
@@ -1976,14 +1888,14 @@ XML_ParseBuffer(XML_Parser parser, int len, int isFinal)
   parser->m_parseEndByteIndex += len;
   parser->m_parsingStatus.finalBuffer = (XML_Bool)isFinal;
 
-  parser->m_errorCode = parser->m_processor(parser, start, parser->m_parseEndPtr, &parser->m_bufferPtr);
+  parser->m_errorCode = parser->m_processor(
+      parser, start, parser->m_parseEndPtr, &parser->m_bufferPtr);
 
   if (parser->m_errorCode != XML_ERROR_NONE) {
     parser->m_eventEndPtr = parser->m_eventPtr;
     parser->m_processor = errorProcessor;
     return XML_STATUS_ERROR;
-  }
-  else {
+  } else {
     switch (parser->m_parsingStatus.parsing) {
     case XML_SUSPENDED:
       result = XML_STATUS_SUSPENDED;
@@ -1994,18 +1906,18 @@ XML_ParseBuffer(XML_Parser parser, int len, int isFinal)
         parser->m_parsingStatus.parsing = XML_FINISHED;
         return result;
       }
-    default: ;  /* should not happen */
+    default:; /* should not happen */
     }
   }
 
-  XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, parser->m_bufferPtr, &parser->m_position);
+  XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr,
+                    parser->m_bufferPtr, &parser->m_position);
   parser->m_positionPtr = parser->m_bufferPtr;
   return result;
 }
 
-void * XMLCALL
-XML_GetBuffer(XML_Parser parser, int len)
-{
+void *XMLCALL
+XML_GetBuffer(XML_Parser parser, int len) {
   if (parser == NULL)
     return NULL;
   if (len < 0) {
@@ -2019,17 +1931,17 @@ XML_GetBuffer(XML_Parser parser, int len)
   case XML_FINISHED:
     parser->m_errorCode = XML_ERROR_FINISHED;
     return NULL;
-  default: ;
+  default:;
   }
 
   if (len > EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferEnd)) {
 #ifdef XML_CONTEXT_BYTES
     int keep;
-#endif  /* defined XML_CONTEXT_BYTES */
+#endif /* defined XML_CONTEXT_BYTES */
     /* Do not invoke signed arithmetic overflow: */
-    int neededSize = (int) ((unsigned)len +
-                            (unsigned)EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd,
-                                                          parser->m_bufferPtr));
+    int neededSize = (int)((unsigned)len
+                           + (unsigned)EXPAT_SAFE_PTR_DIFF(
+                               parser->m_bufferEnd, parser->m_bufferPtr));
     if (neededSize < 0) {
       parser->m_errorCode = XML_ERROR_NO_MEMORY;
       return NULL;
@@ -2039,13 +1951,18 @@ XML_GetBuffer(XML_Parser parser, int len)
     if (keep > XML_CONTEXT_BYTES)
       keep = XML_CONTEXT_BYTES;
     neededSize += keep;
-#endif  /* defined XML_CONTEXT_BYTES */
-    if (neededSize <= EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_buffer)) {
+#endif /* defined XML_CONTEXT_BYTES */
+    if (neededSize
+        <= EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_buffer)) {
 #ifdef XML_CONTEXT_BYTES
       if (keep < EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer)) {
-          int offset = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer) - keep;
-        /* The buffer pointers cannot be NULL here; we have at least some bytes in the buffer */
-        memmove(parser->m_buffer, &parser->m_buffer[offset], parser->m_bufferEnd - parser->m_bufferPtr + keep);
+        int offset
+            = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer)
+              - keep;
+        /* The buffer pointers cannot be NULL here; we have at least some bytes
+         * in the buffer */
+        memmove(parser->m_buffer, &parser->m_buffer[offset],
+                parser->m_bufferEnd - parser->m_bufferPtr + keep);
         parser->m_bufferEnd -= offset;
         parser->m_bufferPtr -= offset;
       }
@@ -2053,20 +1970,21 @@ XML_GetBuffer(XML_Parser parser, int len)
       if (parser->m_buffer && parser->m_bufferPtr) {
         memmove(parser->m_buffer, parser->m_bufferPtr,
                 EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr));
-        parser->m_bufferEnd = parser->m_buffer +
-            EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr);
+        parser->m_bufferEnd
+            = parser->m_buffer
+              + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr);
         parser->m_bufferPtr = parser->m_buffer;
       }
-#endif  /* not defined XML_CONTEXT_BYTES */
-    }
-    else {
+#endif /* not defined XML_CONTEXT_BYTES */
+    } else {
       char *newBuf;
-      int bufferSize = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferPtr);
+      int bufferSize
+          = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferPtr);
       if (bufferSize == 0)
         bufferSize = INIT_BUFFER_SIZE;
       do {
         /* Do not invoke signed arithmetic overflow: */
-        bufferSize = (int) (2U * (unsigned) bufferSize);
+        bufferSize = (int)(2U * (unsigned)bufferSize);
       } while (bufferSize < neededSize && bufferSize > 0);
       if (bufferSize <= 0) {
         parser->m_errorCode = XML_ERROR_NO_MEMORY;
@@ -2080,18 +1998,17 @@ XML_GetBuffer(XML_Parser parser, int len)
       parser->m_bufferLim = newBuf + bufferSize;
 #ifdef XML_CONTEXT_BYTES
       if (parser->m_bufferPtr) {
-        int keep = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer);
-        if (keep > XML_CONTEXT_BYTES)
-          keep = XML_CONTEXT_BYTES;
         memcpy(newBuf, &parser->m_bufferPtr[-keep],
-               EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr) + keep);
+               EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)
+                   + keep);
         FREE(parser, parser->m_buffer);
         parser->m_buffer = newBuf;
-        parser->m_bufferEnd = parser->m_buffer +
-            EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr) + keep;
+        parser->m_bufferEnd
+            = parser->m_buffer
+              + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)
+              + keep;
         parser->m_bufferPtr = parser->m_buffer + keep;
-      }
-      else {
+      } else {
         /* This must be a brand new buffer with no data in it yet */
         parser->m_bufferEnd = newBuf;
         parser->m_bufferPtr = parser->m_buffer = newBuf;
@@ -2101,15 +2018,15 @@ XML_GetBuffer(XML_Parser parser, int len)
         memcpy(newBuf, parser->m_bufferPtr,
                EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr));
         FREE(parser, parser->m_buffer);
-        parser->m_bufferEnd = newBuf +
-            EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr);
-      }
-      else {
+        parser->m_bufferEnd
+            = newBuf
+              + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr);
+      } else {
         /* This must be a brand new buffer with no data in it yet */
         parser->m_bufferEnd = newBuf;
       }
       parser->m_bufferPtr = parser->m_buffer = newBuf;
-#endif  /* not defined XML_CONTEXT_BYTES */
+#endif /* not defined XML_CONTEXT_BYTES */
     }
     parser->m_eventPtr = parser->m_eventEndPtr = NULL;
     parser->m_positionPtr = NULL;
@@ -2118,8 +2035,7 @@ XML_GetBuffer(XML_Parser parser, int len)
 }
 
 enum XML_Status XMLCALL
-XML_StopParser(XML_Parser parser, XML_Bool resumable)
-{
+XML_StopParser(XML_Parser parser, XML_Bool resumable) {
   if (parser == NULL)
     return XML_STATUS_ERROR;
   switch (parser->m_parsingStatus.parsing) {
@@ -2142,16 +2058,14 @@ XML_StopParser(XML_Parser parser, XML_Bool resumable)
       }
 #endif
       parser->m_parsingStatus.parsing = XML_SUSPENDED;
-    }
-    else
+    } else
       parser->m_parsingStatus.parsing = XML_FINISHED;
   }
   return XML_STATUS_OK;
 }
 
 enum XML_Status XMLCALL
-XML_ResumeParser(XML_Parser parser)
-{
+XML_ResumeParser(XML_Parser parser) {
   enum XML_Status result = XML_STATUS_OK;
 
   if (parser == NULL)
@@ -2162,14 +2076,14 @@ XML_ResumeParser(XML_Parser parser)
   }
   parser->m_parsingStatus.parsing = XML_PARSING;
 
-  parser->m_errorCode = parser->m_processor(parser, parser->m_bufferPtr, parser->m_parseEndPtr, &parser->m_bufferPtr);
+  parser->m_errorCode = parser->m_processor(
+      parser, parser->m_bufferPtr, parser->m_parseEndPtr, &parser->m_bufferPtr);
 
   if (parser->m_errorCode != XML_ERROR_NONE) {
     parser->m_eventEndPtr = parser->m_eventPtr;
     parser->m_processor = errorProcessor;
     return XML_STATUS_ERROR;
-  }
-  else {
+  } else {
     switch (parser->m_parsingStatus.parsing) {
     case XML_SUSPENDED:
       result = XML_STATUS_SUSPENDED;
@@ -2180,18 +2094,18 @@ XML_ResumeParser(XML_Parser parser)
         parser->m_parsingStatus.parsing = XML_FINISHED;
         return result;
       }
-    default: ;
+    default:;
     }
   }
 
-  XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, parser->m_bufferPtr, &parser->m_position);
+  XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr,
+                    parser->m_bufferPtr, &parser->m_position);
   parser->m_positionPtr = parser->m_bufferPtr;
   return result;
 }
 
 void XMLCALL
-XML_GetParsingStatus(XML_Parser parser, XML_ParsingStatus *status)
-{
+XML_GetParsingStatus(XML_Parser parser, XML_ParsingStatus *status) {
   if (parser == NULL)
     return;
   assert(status != NULL);
@@ -2199,26 +2113,24 @@ XML_GetParsingStatus(XML_Parser parser, XML_ParsingStatus *status)
 }
 
 enum XML_Error XMLCALL
-XML_GetErrorCode(XML_Parser parser)
-{
+XML_GetErrorCode(XML_Parser parser) {
   if (parser == NULL)
     return XML_ERROR_INVALID_ARGUMENT;
   return parser->m_errorCode;
 }
 
 XML_Index XMLCALL
-XML_GetCurrentByteIndex(XML_Parser parser)
-{
+XML_GetCurrentByteIndex(XML_Parser parser) {
   if (parser == NULL)
     return -1;
   if (parser->m_eventPtr)
-    return (XML_Index)(parser->m_parseEndByteIndex - (parser->m_parseEndPtr - parser->m_eventPtr));
+    return (XML_Index)(parser->m_parseEndByteIndex
+                       - (parser->m_parseEndPtr - parser->m_eventPtr));
   return -1;
 }
 
 int XMLCALL
-XML_GetCurrentByteCount(XML_Parser parser)
-{
+XML_GetCurrentByteCount(XML_Parser parser) {
   if (parser == NULL)
     return 0;
   if (parser->m_eventEndPtr && parser->m_eventPtr)
@@ -2226,9 +2138,8 @@ XML_GetCurrentByteCount(XML_Parser parser)
   return 0;
 }
 
-const char * XMLCALL
-XML_GetInputContext(XML_Parser parser, int *offset, int *size)
-{
+const char *XMLCALL
+XML_GetInputContext(XML_Parser parser, int *offset, int *size) {
 #ifdef XML_CONTEXT_BYTES
   if (parser == NULL)
     return NULL;
@@ -2236,7 +2147,7 @@ XML_GetInputContext(XML_Parser parser, int *offset, int *size)
     if (offset != NULL)
       *offset = (int)(parser->m_eventPtr - parser->m_buffer);
     if (size != NULL)
-      *size   = (int)(parser->m_bufferEnd - parser->m_buffer);
+      *size = (int)(parser->m_bufferEnd - parser->m_buffer);
     return parser->m_buffer;
   }
 #else
@@ -2244,82 +2155,76 @@ XML_GetInputContext(XML_Parser parser, int *offset, int *size)
   (void)offset;
   (void)size;
 #endif /* defined XML_CONTEXT_BYTES */
-  return (char *) 0;
+  return (char *)0;
 }
 
 XML_Size XMLCALL
-XML_GetCurrentLineNumber(XML_Parser parser)
-{
+XML_GetCurrentLineNumber(XML_Parser parser) {
   if (parser == NULL)
     return 0;
   if (parser->m_eventPtr && parser->m_eventPtr >= parser->m_positionPtr) {
-    XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, parser->m_eventPtr, &parser->m_position);
+    XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr,
+                      parser->m_eventPtr, &parser->m_position);
     parser->m_positionPtr = parser->m_eventPtr;
   }
   return parser->m_position.lineNumber + 1;
 }
 
 XML_Size XMLCALL
-XML_GetCurrentColumnNumber(XML_Parser parser)
-{
+XML_GetCurrentColumnNumber(XML_Parser parser) {
   if (parser == NULL)
     return 0;
   if (parser->m_eventPtr && parser->m_eventPtr >= parser->m_positionPtr) {
-    XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, parser->m_eventPtr, &parser->m_position);
+    XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr,
+                      parser->m_eventPtr, &parser->m_position);
     parser->m_positionPtr = parser->m_eventPtr;
   }
   return parser->m_position.columnNumber;
 }
 
 void XMLCALL
-XML_FreeContentModel(XML_Parser parser, XML_Content *model)
-{
+XML_FreeContentModel(XML_Parser parser, XML_Content *model) {
   if (parser != NULL)
     FREE(parser, model);
 }
 
-void * XMLCALL
-XML_MemMalloc(XML_Parser parser, size_t size)
-{
+void *XMLCALL
+XML_MemMalloc(XML_Parser parser, size_t size) {
   if (parser == NULL)
     return NULL;
   return MALLOC(parser, size);
 }
 
-void * XMLCALL
-XML_MemRealloc(XML_Parser parser, void *ptr, size_t size)
-{
+void *XMLCALL
+XML_MemRealloc(XML_Parser parser, void *ptr, size_t size) {
   if (parser == NULL)
     return NULL;
   return REALLOC(parser, ptr, size);
 }
 
 void XMLCALL
-XML_MemFree(XML_Parser parser, void *ptr)
-{
+XML_MemFree(XML_Parser parser, void *ptr) {
   if (parser != NULL)
     FREE(parser, ptr);
 }
 
 void XMLCALL
-XML_DefaultCurrent(XML_Parser parser)
-{
+XML_DefaultCurrent(XML_Parser parser) {
   if (parser == NULL)
     return;
   if (parser->m_defaultHandler) {
     if (parser->m_openInternalEntities)
-      reportDefault(parser,
-                    parser->m_internalEncoding,
+      reportDefault(parser, parser->m_internalEncoding,
                     parser->m_openInternalEntities->internalEventPtr,
                     parser->m_openInternalEntities->internalEventEndPtr);
     else
-      reportDefault(parser, parser->m_encoding, parser->m_eventPtr, parser->m_eventEndPtr);
+      reportDefault(parser, parser->m_encoding, parser->m_eventPtr,
+                    parser->m_eventEndPtr);
   }
 }
 
-const XML_LChar * XMLCALL
-XML_ErrorString(enum XML_Error code)
-{
+const XML_LChar *XMLCALL
+XML_ErrorString(enum XML_Error code) {
   switch (code) {
   case XML_ERROR_NONE:
     return NULL;
@@ -2401,21 +2306,22 @@ XML_ErrorString(enum XML_Error code)
     return XML_L("cannot suspend in external parameter entity");
   /* Added in 2.0.0. */
   case XML_ERROR_RESERVED_PREFIX_XML:
-    return XML_L("reserved prefix (xml) must not be undeclared or bound to another namespace name");
+    return XML_L(
+        "reserved prefix (xml) must not be undeclared or bound to another namespace name");
   case XML_ERROR_RESERVED_PREFIX_XMLNS:
     return XML_L("reserved prefix (xmlns) must not be declared or undeclared");
   case XML_ERROR_RESERVED_NAMESPACE_URI:
-    return XML_L("prefix must not be bound to one of the reserved namespace names");
+    return XML_L(
+        "prefix must not be bound to one of the reserved namespace names");
   /* Added in 2.2.5. */
-  case XML_ERROR_INVALID_ARGUMENT:  /* Constant added in 2.2.1, already */
+  case XML_ERROR_INVALID_ARGUMENT: /* Constant added in 2.2.1, already */
     return XML_L("invalid argument");
   }
   return NULL;
 }
 
-const XML_LChar * XMLCALL
+const XML_LChar *XMLCALL
 XML_ExpatVersion(void) {
-
   /* V1 is used to string-ize the version number. However, it would
      string-ize the actual version macro *names* unless we get them
      substituted before being passed to V1. CPP is defined to expand
@@ -2424,8 +2330,8 @@ XML_ExpatVersion(void) {
      with the correct numerals. */
   /* ### I'm assuming cpp is portable in this respect... */
 
-#define V1(a,b,c) XML_L(#a)XML_L(".")XML_L(#b)XML_L(".")XML_L(#c)
-#define V2(a,b,c) XML_L("expat_")V1(a,b,c)
+#define V1(a, b, c) XML_L(#a) XML_L(".") XML_L(#b) XML_L(".") XML_L(#c)
+#define V2(a, b, c) XML_L("expat_") V1(a, b, c)
 
   return V2(XML_MAJOR_VERSION, XML_MINOR_VERSION, XML_MICRO_VERSION);
 
@@ -2434,8 +2340,7 @@ XML_ExpatVersion(void) {
 }
 
 XML_Expat_Version XMLCALL
-XML_ExpatVersionInfo(void)
-{
+XML_ExpatVersionInfo(void) {
   XML_Expat_Version version;
 
   version.major = XML_MAJOR_VERSION;
@@ -2445,41 +2350,39 @@ XML_ExpatVersionInfo(void)
   return version;
 }
 
-const XML_Feature * XMLCALL
-XML_GetFeatureList(void)
-{
-  static const XML_Feature features[] = {
-    {XML_FEATURE_SIZEOF_XML_CHAR,  XML_L("sizeof(XML_Char)"),
-     sizeof(XML_Char)},
-    {XML_FEATURE_SIZEOF_XML_LCHAR, XML_L("sizeof(XML_LChar)"),
-     sizeof(XML_LChar)},
+const XML_Feature *XMLCALL
+XML_GetFeatureList(void) {
+  static const XML_Feature features[]
+      = {{XML_FEATURE_SIZEOF_XML_CHAR, XML_L("sizeof(XML_Char)"),
+          sizeof(XML_Char)},
+         {XML_FEATURE_SIZEOF_XML_LCHAR, XML_L("sizeof(XML_LChar)"),
+          sizeof(XML_LChar)},
 #ifdef XML_UNICODE
-    {XML_FEATURE_UNICODE,          XML_L("XML_UNICODE"), 0},
+         {XML_FEATURE_UNICODE, XML_L("XML_UNICODE"), 0},
 #endif
 #ifdef XML_UNICODE_WCHAR_T
-    {XML_FEATURE_UNICODE_WCHAR_T,  XML_L("XML_UNICODE_WCHAR_T"), 0},
+         {XML_FEATURE_UNICODE_WCHAR_T, XML_L("XML_UNICODE_WCHAR_T"), 0},
 #endif
 #ifdef XML_DTD
-    {XML_FEATURE_DTD,              XML_L("XML_DTD"), 0},
+         {XML_FEATURE_DTD, XML_L("XML_DTD"), 0},
 #endif
 #ifdef XML_CONTEXT_BYTES
-    {XML_FEATURE_CONTEXT_BYTES,    XML_L("XML_CONTEXT_BYTES"),
-     XML_CONTEXT_BYTES},
+         {XML_FEATURE_CONTEXT_BYTES, XML_L("XML_CONTEXT_BYTES"),
+          XML_CONTEXT_BYTES},
 #endif
 #ifdef XML_MIN_SIZE
-    {XML_FEATURE_MIN_SIZE,         XML_L("XML_MIN_SIZE"), 0},
+         {XML_FEATURE_MIN_SIZE, XML_L("XML_MIN_SIZE"), 0},
 #endif
 #ifdef XML_NS
-    {XML_FEATURE_NS,               XML_L("XML_NS"), 0},
+         {XML_FEATURE_NS, XML_L("XML_NS"), 0},
 #endif
 #ifdef XML_LARGE_SIZE
-    {XML_FEATURE_LARGE_SIZE,       XML_L("XML_LARGE_SIZE"), 0},
+         {XML_FEATURE_LARGE_SIZE, XML_L("XML_LARGE_SIZE"), 0},
 #endif
 #ifdef XML_ATTR_INFO
-    {XML_FEATURE_ATTR_INFO,        XML_L("XML_ATTR_INFO"), 0},
+         {XML_FEATURE_ATTR_INFO, XML_L("XML_ATTR_INFO"), 0},
 #endif
-    {XML_FEATURE_END,              NULL, 0}
-  };
+         {XML_FEATURE_END, NULL, 0}};
 
   return features;
 }
@@ -2490,8 +2393,7 @@ XML_GetFeatureList(void)
    permanent location, since the parse buffer is about to be discarded.
 */
 static XML_Bool
-storeRawNames(XML_Parser parser)
-{
+storeRawNames(XML_Parser parser) {
   TAG *tag = parser->m_tagStack;
   while (tag) {
     int bufSize;
@@ -2521,8 +2423,8 @@ storeRawNames(XML_Parser parser)
          then update it as well, since it will always point into tag->buf
       */
       if (tag->name.localPart)
-        tag->name.localPart = (XML_Char *)temp + (tag->name.localPart -
-                                                  (XML_Char *)tag->buf);
+        tag->name.localPart
+            = (XML_Char *)temp + (tag->name.localPart - (XML_Char *)tag->buf);
       tag->buf = temp;
       tag->bufEnd = temp + bufSize;
       rawNameBuf = temp + nameLen;
@@ -2535,26 +2437,21 @@ storeRawNames(XML_Parser parser)
 }
 
 static enum XML_Error PTRCALL
-contentProcessor(XML_Parser parser,
-                 const char *start,
-                 const char *end,
-                 const char **endPtr)
-{
-  enum XML_Error result = doContent(parser, 0, parser->m_encoding, start, end,
-                                    endPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer);
+contentProcessor(XML_Parser parser, const char *start, const char *end,
+                 const char **endPtr) {
+  enum XML_Error result
+      = doContent(parser, 0, parser->m_encoding, start, end, endPtr,
+                  (XML_Bool)! parser->m_parsingStatus.finalBuffer);
   if (result == XML_ERROR_NONE) {
-    if (!storeRawNames(parser))
+    if (! storeRawNames(parser))
       return XML_ERROR_NO_MEMORY;
   }
   return result;
 }
 
 static enum XML_Error PTRCALL
-externalEntityInitProcessor(XML_Parser parser,
-                            const char *start,
-                            const char *end,
-                            const char **endPtr)
-{
+externalEntityInitProcessor(XML_Parser parser, const char *start,
+                            const char *end, const char **endPtr) {
   enum XML_Error result = initializeEncoding(parser);
   if (result != XML_ERROR_NONE)
     return result;
@@ -2563,11 +2460,8 @@ externalEntityInitProcessor(XML_Parser parser,
 }
 
 static enum XML_Error PTRCALL
-externalEntityInitProcessor2(XML_Parser parser,
-                             const char *start,
-                             const char *end,
-                             const char **endPtr)
-{
+externalEntityInitProcessor2(XML_Parser parser, const char *start,
+                             const char *end, const char **endPtr) {
   const char *next = start; /* XmlContentTok doesn't always set the last arg */
   int tok = XmlContentTok(parser->m_encoding, start, end, &next);
   switch (tok) {
@@ -2577,21 +2471,21 @@ externalEntityInitProcessor2(XML_Parser parser,
        doContent (by detecting XML_TOK_NONE) without processing any xml text
        declaration - causing the error XML_ERROR_MISPLACED_XML_PI in doContent.
     */
-    if (next == end && !parser->m_parsingStatus.finalBuffer) {
+    if (next == end && ! parser->m_parsingStatus.finalBuffer) {
       *endPtr = next;
       return XML_ERROR_NONE;
     }
     start = next;
     break;
   case XML_TOK_PARTIAL:
-    if (!parser->m_parsingStatus.finalBuffer) {
+    if (! parser->m_parsingStatus.finalBuffer) {
       *endPtr = start;
       return XML_ERROR_NONE;
     }
     parser->m_eventPtr = start;
     return XML_ERROR_UNCLOSED_TOKEN;
   case XML_TOK_PARTIAL_CHAR:
-    if (!parser->m_parsingStatus.finalBuffer) {
+    if (! parser->m_parsingStatus.finalBuffer) {
       *endPtr = start;
       return XML_ERROR_NONE;
     }
@@ -2603,11 +2497,8 @@ externalEntityInitProcessor2(XML_Parser parser,
 }
 
 static enum XML_Error PTRCALL
-externalEntityInitProcessor3(XML_Parser parser,
-                             const char *start,
-                             const char *end,
-                             const char **endPtr)
-{
+externalEntityInitProcessor3(XML_Parser parser, const char *start,
+                             const char *end, const char **endPtr) {
   int tok;
   const char *next = start; /* XmlContentTok doesn't always set the last arg */
   parser->m_eventPtr = start;
@@ -2615,31 +2506,29 @@ externalEntityInitProcessor3(XML_Parser parser,
   parser->m_eventEndPtr = next;
 
   switch (tok) {
-  case XML_TOK_XML_DECL:
-    {
-      enum XML_Error result;
-      result = processXmlDecl(parser, 1, start, next);
-      if (result != XML_ERROR_NONE)
-        return result;
-      switch (parser->m_parsingStatus.parsing) {
-      case XML_SUSPENDED:
-        *endPtr = next;
-        return XML_ERROR_NONE;
-      case XML_FINISHED:
-        return XML_ERROR_ABORTED;
-      default:
-        start = next;
-      }
+  case XML_TOK_XML_DECL: {
+    enum XML_Error result;
+    result = processXmlDecl(parser, 1, start, next);
+    if (result != XML_ERROR_NONE)
+      return result;
+    switch (parser->m_parsingStatus.parsing) {
+    case XML_SUSPENDED:
+      *endPtr = next;
+      return XML_ERROR_NONE;
+    case XML_FINISHED:
+      return XML_ERROR_ABORTED;
+    default:
+      start = next;
     }
-    break;
+  } break;
   case XML_TOK_PARTIAL:
-    if (!parser->m_parsingStatus.finalBuffer) {
+    if (! parser->m_parsingStatus.finalBuffer) {
       *endPtr = start;
       return XML_ERROR_NONE;
     }
     return XML_ERROR_UNCLOSED_TOKEN;
   case XML_TOK_PARTIAL_CHAR:
-    if (!parser->m_parsingStatus.finalBuffer) {
+    if (! parser->m_parsingStatus.finalBuffer) {
       *endPtr = start;
       return XML_ERROR_NONE;
     }
@@ -2651,39 +2540,31 @@ externalEntityInitProcessor3(XML_Parser parser,
 }
 
 static enum XML_Error PTRCALL
-externalEntityContentProcessor(XML_Parser parser,
-                               const char *start,
-                               const char *end,
-                               const char **endPtr)
-{
-  enum XML_Error result = doContent(parser, 1, parser->m_encoding, start, end,
-                                    endPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer);
+externalEntityContentProcessor(XML_Parser parser, const char *start,
+                               const char *end, const char **endPtr) {
+  enum XML_Error result
+      = doContent(parser, 1, parser->m_encoding, start, end, endPtr,
+                  (XML_Bool)! parser->m_parsingStatus.finalBuffer);
   if (result == XML_ERROR_NONE) {
-    if (!storeRawNames(parser))
+    if (! storeRawNames(parser))
       return XML_ERROR_NO_MEMORY;
   }
   return result;
 }
 
 static enum XML_Error
-doContent(XML_Parser parser,
-          int startTagLevel,
-          const ENCODING *enc,
-          const char *s,
-          const char *end,
-          const char **nextPtr,
-          XML_Bool haveMore)
-{
+doContent(XML_Parser parser, int startTagLevel, const ENCODING *enc,
+          const char *s, const char *end, const char **nextPtr,
+          XML_Bool haveMore) {
   /* save one level of indirection */
-  DTD * const dtd = parser->m_dtd;
+  DTD *const dtd = parser->m_dtd;
 
   const char **eventPP;
   const char **eventEndPP;
   if (enc == parser->m_encoding) {
     eventPP = &parser->m_eventPtr;
     eventEndPP = &parser->m_eventEndPtr;
-  }
-  else {
+  } else {
     eventPP = &(parser->m_openInternalEntities->internalEventPtr);
     eventEndPP = &(parser->m_openInternalEntities->internalEventEndPtr);
   }
@@ -2703,8 +2584,7 @@ doContent(XML_Parser parser,
       if (parser->m_characterDataHandler) {
         XML_Char c = 0xA;
         parser->m_characterDataHandler(parser->m_handlerArg, &c, 1);
-      }
-      else if (parser->m_defaultHandler)
+      } else if (parser->m_defaultHandler)
         reportDefault(parser, enc, s, end);
       /* We are at the end of the final buffer, should we check for
          XML_SUSPENDED, XML_FINISHED?
@@ -2742,185 +2622,178 @@ doContent(XML_Parser parser,
         return XML_ERROR_NONE;
       }
       return XML_ERROR_PARTIAL_CHAR;
-    case XML_TOK_ENTITY_REF:
-      {
-        const XML_Char *name;
-        ENTITY *entity;
-        XML_Char ch = (XML_Char) XmlPredefinedEntityName(enc,
-                                              s + enc->minBytesPerChar,
-                                              next - enc->minBytesPerChar);
-        if (ch) {
-          if (parser->m_characterDataHandler)
-            parser->m_characterDataHandler(parser->m_handlerArg, &ch, 1);
-          else if (parser->m_defaultHandler)
-            reportDefault(parser, enc, s, next);
-          break;
-        }
-        name = poolStoreString(&dtd->pool, enc,
-                                s + enc->minBytesPerChar,
-                                next - enc->minBytesPerChar);
-        if (!name)
-          return XML_ERROR_NO_MEMORY;
-        entity = (ENTITY *)lookup(parser, &dtd->generalEntities, name, 0);
-        poolDiscard(&dtd->pool);
-        /* First, determine if a check for an existing declaration is needed;
-           if yes, check that the entity exists, and that it is internal,
-           otherwise call the skipped entity or default handler.
-        */
-        if (!dtd->hasParamEntityRefs || dtd->standalone) {
-          if (!entity)
-            return XML_ERROR_UNDEFINED_ENTITY;
-          else if (!entity->is_internal)
-            return XML_ERROR_ENTITY_DECLARED_IN_PE;
-        }
-        else if (!entity) {
+    case XML_TOK_ENTITY_REF: {
+      const XML_Char *name;
+      ENTITY *entity;
+      XML_Char ch = (XML_Char)XmlPredefinedEntityName(
+          enc, s + enc->minBytesPerChar, next - enc->minBytesPerChar);
+      if (ch) {
+        if (parser->m_characterDataHandler)
+          parser->m_characterDataHandler(parser->m_handlerArg, &ch, 1);
+        else if (parser->m_defaultHandler)
+          reportDefault(parser, enc, s, next);
+        break;
+      }
+      name = poolStoreString(&dtd->pool, enc, s + enc->minBytesPerChar,
+                             next - enc->minBytesPerChar);
+      if (! name)
+        return XML_ERROR_NO_MEMORY;
+      entity = (ENTITY *)lookup(parser, &dtd->generalEntities, name, 0);
+      poolDiscard(&dtd->pool);
+      /* First, determine if a check for an existing declaration is needed;
+         if yes, check that the entity exists, and that it is internal,
+         otherwise call the skipped entity or default handler.
+      */
+      if (! dtd->hasParamEntityRefs || dtd->standalone) {
+        if (! entity)
+          return XML_ERROR_UNDEFINED_ENTITY;
+        else if (! entity->is_internal)
+          return XML_ERROR_ENTITY_DECLARED_IN_PE;
+      } else if (! entity) {
+        if (parser->m_skippedEntityHandler)
+          parser->m_skippedEntityHandler(parser->m_handlerArg, name, 0);
+        else if (parser->m_defaultHandler)
+          reportDefault(parser, enc, s, next);
+        break;
+      }
+      if (entity->open)
+        return XML_ERROR_RECURSIVE_ENTITY_REF;
+      if (entity->notation)
+        return XML_ERROR_BINARY_ENTITY_REF;
+      if (entity->textPtr) {
+        enum XML_Error result;
+        if (! parser->m_defaultExpandInternalEntities) {
           if (parser->m_skippedEntityHandler)
-            parser->m_skippedEntityHandler(parser->m_handlerArg, name, 0);
+            parser->m_skippedEntityHandler(parser->m_handlerArg, entity->name,
+                                           0);
           else if (parser->m_defaultHandler)
             reportDefault(parser, enc, s, next);
           break;
         }
-        if (entity->open)
-          return XML_ERROR_RECURSIVE_ENTITY_REF;
-        if (entity->notation)
-          return XML_ERROR_BINARY_ENTITY_REF;
-        if (entity->textPtr) {
-          enum XML_Error result;
-          if (!parser->m_defaultExpandInternalEntities) {
-            if (parser->m_skippedEntityHandler)
-              parser->m_skippedEntityHandler(parser->m_handlerArg, entity->name, 0);
-            else if (parser->m_defaultHandler)
-              reportDefault(parser, enc, s, next);
-            break;
-          }
-          result = processInternalEntity(parser, entity, XML_FALSE);
-          if (result != XML_ERROR_NONE)
-            return result;
-        }
-        else if (parser->m_externalEntityRefHandler) {
-          const XML_Char *context;
-          entity->open = XML_TRUE;
-          context = getContext(parser);
-          entity->open = XML_FALSE;
-          if (!context)
-            return XML_ERROR_NO_MEMORY;
-          if (!parser->m_externalEntityRefHandler(parser->m_externalEntityRefHandlerArg,
-                                        context,
-                                        entity->base,
-                                        entity->systemId,
-                                        entity->publicId))
-            return XML_ERROR_EXTERNAL_ENTITY_HANDLING;
-          poolDiscard(&parser->m_tempPool);
-        }
-        else if (parser->m_defaultHandler)
-          reportDefault(parser, enc, s, next);
-        break;
-      }
+        result = processInternalEntity(parser, entity, XML_FALSE);
+        if (result != XML_ERROR_NONE)
+          return result;
+      } else if (parser->m_externalEntityRefHandler) {
+        const XML_Char *context;
+        entity->open = XML_TRUE;
+        context = getContext(parser);
+        entity->open = XML_FALSE;
+        if (! context)
+          return XML_ERROR_NO_MEMORY;
+        if (! parser->m_externalEntityRefHandler(
+                parser->m_externalEntityRefHandlerArg, context, entity->base,
+                entity->systemId, entity->publicId))
+          return XML_ERROR_EXTERNAL_ENTITY_HANDLING;
+        poolDiscard(&parser->m_tempPool);
+      } else if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+      break;
+    }
     case XML_TOK_START_TAG_NO_ATTS:
       /* fall through */
-    case XML_TOK_START_TAG_WITH_ATTS:
-      {
-        TAG *tag;
-        enum XML_Error result;
-        XML_Char *toPtr;
-        if (parser->m_freeTagList) {
-          tag = parser->m_freeTagList;
-          parser->m_freeTagList = parser->m_freeTagList->parent;
+    case XML_TOK_START_TAG_WITH_ATTS: {
+      TAG *tag;
+      enum XML_Error result;
+      XML_Char *toPtr;
+      if (parser->m_freeTagList) {
+        tag = parser->m_freeTagList;
+        parser->m_freeTagList = parser->m_freeTagList->parent;
+      } else {
+        tag = (TAG *)MALLOC(parser, sizeof(TAG));
+        if (! tag)
+          return XML_ERROR_NO_MEMORY;
+        tag->buf = (char *)MALLOC(parser, INIT_TAG_BUF_SIZE);
+        if (! tag->buf) {
+          FREE(parser, tag);
+          return XML_ERROR_NO_MEMORY;
         }
-        else {
-          tag = (TAG *)MALLOC(parser, sizeof(TAG));
-          if (!tag)
-            return XML_ERROR_NO_MEMORY;
-          tag->buf = (char *)MALLOC(parser, INIT_TAG_BUF_SIZE);
-          if (!tag->buf) {
-            FREE(parser, tag);
-            return XML_ERROR_NO_MEMORY;
+        tag->bufEnd = tag->buf + INIT_TAG_BUF_SIZE;
+      }
+      tag->bindings = NULL;
+      tag->parent = parser->m_tagStack;
+      parser->m_tagStack = tag;
+      tag->name.localPart = NULL;
+      tag->name.prefix = NULL;
+      tag->rawName = s + enc->minBytesPerChar;
+      tag->rawNameLength = XmlNameLength(enc, tag->rawName);
+      ++parser->m_tagLevel;
+      {
+        const char *rawNameEnd = tag->rawName + tag->rawNameLength;
+        const char *fromPtr = tag->rawName;
+        toPtr = (XML_Char *)tag->buf;
+        for (;;) {
+          int bufSize;
+          int convLen;
+          const enum XML_Convert_Result convert_res
+              = XmlConvert(enc, &fromPtr, rawNameEnd, (ICHAR **)&toPtr,
+                           (ICHAR *)tag->bufEnd - 1);
+          convLen = (int)(toPtr - (XML_Char *)tag->buf);
+          if ((fromPtr >= rawNameEnd)
+              || (convert_res == XML_CONVERT_INPUT_INCOMPLETE)) {
+            tag->name.strLen = convLen;
+            break;
           }
-          tag->bufEnd = tag->buf + INIT_TAG_BUF_SIZE;
-        }
-        tag->bindings = NULL;
-        tag->parent = parser->m_tagStack;
-        parser->m_tagStack = tag;
-        tag->name.localPart = NULL;
-        tag->name.prefix = NULL;
-        tag->rawName = s + enc->minBytesPerChar;
-        tag->rawNameLength = XmlNameLength(enc, tag->rawName);
-        ++parser->m_tagLevel;
-        {
-          const char *rawNameEnd = tag->rawName + tag->rawNameLength;
-          const char *fromPtr = tag->rawName;
-          toPtr = (XML_Char *)tag->buf;
-          for (;;) {
-            int bufSize;
-            int convLen;
-            const enum XML_Convert_Result convert_res = XmlConvert(enc,
-                       &fromPtr, rawNameEnd,
-                       (ICHAR **)&toPtr, (ICHAR *)tag->bufEnd - 1);
-            convLen = (int)(toPtr - (XML_Char *)tag->buf);
-            if ((fromPtr >= rawNameEnd) || (convert_res == XML_CONVERT_INPUT_INCOMPLETE)) {
-              tag->name.strLen = convLen;
-              break;
-            }
-            bufSize = (int)(tag->bufEnd - tag->buf) << 1;
-            {
-              char *temp = (char *)REALLOC(parser, tag->buf, bufSize);
-              if (temp == NULL)
-                return XML_ERROR_NO_MEMORY;
-              tag->buf = temp;
-              tag->bufEnd = temp + bufSize;
-              toPtr = (XML_Char *)temp + convLen;
-            }
+          bufSize = (int)(tag->bufEnd - tag->buf) << 1;
+          {
+            char *temp = (char *)REALLOC(parser, tag->buf, bufSize);
+            if (temp == NULL)
+              return XML_ERROR_NO_MEMORY;
+            tag->buf = temp;
+            tag->bufEnd = temp + bufSize;
+            toPtr = (XML_Char *)temp + convLen;
           }
         }
-        tag->name.str = (XML_Char *)tag->buf;
-        *toPtr = XML_T('\0');
-        result = storeAtts(parser, enc, s, &(tag->name), &(tag->bindings));
-        if (result)
-          return result;
-        if (parser->m_startElementHandler)
-          parser->m_startElementHandler(parser->m_handlerArg, tag->name.str,
-                              (const XML_Char **)parser->m_atts);
-        else if (parser->m_defaultHandler)
-          reportDefault(parser, enc, s, next);
-        poolClear(&parser->m_tempPool);
-        break;
       }
+      tag->name.str = (XML_Char *)tag->buf;
+      *toPtr = XML_T('\0');
+      result = storeAtts(parser, enc, s, &(tag->name), &(tag->bindings));
+      if (result)
+        return result;
+      if (parser->m_startElementHandler)
+        parser->m_startElementHandler(parser->m_handlerArg, tag->name.str,
+                                      (const XML_Char **)parser->m_atts);
+      else if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+      poolClear(&parser->m_tempPool);
+      break;
+    }
     case XML_TOK_EMPTY_ELEMENT_NO_ATTS:
       /* fall through */
-    case XML_TOK_EMPTY_ELEMENT_WITH_ATTS:
-      {
-        const char *rawName = s + enc->minBytesPerChar;
-        enum XML_Error result;
-        BINDING *bindings = NULL;
-        XML_Bool noElmHandlers = XML_TRUE;
-        TAG_NAME name;
-        name.str = poolStoreString(&parser->m_tempPool, enc, rawName,
-                                   rawName + XmlNameLength(enc, rawName));
-        if (!name.str)
-          return XML_ERROR_NO_MEMORY;
-        poolFinish(&parser->m_tempPool);
-        result = storeAtts(parser, enc, s, &name, &bindings);
-        if (result != XML_ERROR_NONE) {
-          freeBindings(parser, bindings);
-          return result;
-        }
-        poolFinish(&parser->m_tempPool);
-        if (parser->m_startElementHandler) {
-          parser->m_startElementHandler(parser->m_handlerArg, name.str, (const XML_Char **)parser->m_atts);
-          noElmHandlers = XML_FALSE;
-        }
-        if (parser->m_endElementHandler) {
-          if (parser->m_startElementHandler)
-            *eventPP = *eventEndPP;
-          parser->m_endElementHandler(parser->m_handlerArg, name.str);
-          noElmHandlers = XML_FALSE;
-        }
-        if (noElmHandlers && parser->m_defaultHandler)
-          reportDefault(parser, enc, s, next);
-        poolClear(&parser->m_tempPool);
+    case XML_TOK_EMPTY_ELEMENT_WITH_ATTS: {
+      const char *rawName = s + enc->minBytesPerChar;
+      enum XML_Error result;
+      BINDING *bindings = NULL;
+      XML_Bool noElmHandlers = XML_TRUE;
+      TAG_NAME name;
+      name.str = poolStoreString(&parser->m_tempPool, enc, rawName,
+                                 rawName + XmlNameLength(enc, rawName));
+      if (! name.str)
+        return XML_ERROR_NO_MEMORY;
+      poolFinish(&parser->m_tempPool);
+      result = storeAtts(parser, enc, s, &name, &bindings);
+      if (result != XML_ERROR_NONE) {
         freeBindings(parser, bindings);
+        return result;
+      }
+      poolFinish(&parser->m_tempPool);
+      if (parser->m_startElementHandler) {
+        parser->m_startElementHandler(parser->m_handlerArg, name.str,
+                                      (const XML_Char **)parser->m_atts);
+        noElmHandlers = XML_FALSE;
+      }
+      if (parser->m_endElementHandler) {
+        if (parser->m_startElementHandler)
+          *eventPP = *eventEndPP;
+        parser->m_endElementHandler(parser->m_handlerArg, name.str);
+        noElmHandlers = XML_FALSE;
       }
-      if ((parser->m_tagLevel == 0) && (parser->m_parsingStatus.parsing != XML_FINISHED)) {
+      if (noElmHandlers && parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+      poolClear(&parser->m_tempPool);
+      freeBindings(parser, bindings);
+    }
+      if ((parser->m_tagLevel == 0)
+          && (parser->m_parsingStatus.parsing != XML_FINISHED)) {
         if (parser->m_parsingStatus.parsing == XML_SUSPENDED)
           parser->m_processor = epilogProcessor;
         else
@@ -2937,7 +2810,7 @@ doContent(XML_Parser parser,
         parser->m_tagStack = tag->parent;
         tag->parent = parser->m_freeTagList;
         parser->m_freeTagList = tag;
-        rawName = s + enc->minBytesPerChar*2;
+        rawName = s + enc->minBytesPerChar * 2;
         len = XmlNameLength(enc, rawName);
         if (len != tag->rawNameLength
             || memcmp(tag->rawName, rawName, len) != 0) {
@@ -2957,86 +2830,89 @@ doContent(XML_Parser parser,
             */
             uri = (XML_Char *)tag->name.str + tag->name.uriLen;
             /* don't need to check for space - already done in storeAtts() */
-            while (*localPart) *uri++ = *localPart++;
+            while (*localPart)
+              *uri++ = *localPart++;
             prefix = (XML_Char *)tag->name.prefix;
             if (parser->m_ns_triplets && prefix) {
               *uri++ = parser->m_namespaceSeparator;
-              while (*prefix) *uri++ = *prefix++;
-             }
+              while (*prefix)
+                *uri++ = *prefix++;
+            }
             *uri = XML_T('\0');
           }
           parser->m_endElementHandler(parser->m_handlerArg, tag->name.str);
-        }
-        else if (parser->m_defaultHandler)
+        } else if (parser->m_defaultHandler)
           reportDefault(parser, enc, s, next);
         while (tag->bindings) {
           BINDING *b = tag->bindings;
           if (parser->m_endNamespaceDeclHandler)
-            parser->m_endNamespaceDeclHandler(parser->m_handlerArg, b->prefix->name);
+            parser->m_endNamespaceDeclHandler(parser->m_handlerArg,
+                                              b->prefix->name);
           tag->bindings = tag->bindings->nextTagBinding;
           b->nextTagBinding = parser->m_freeBindingList;
           parser->m_freeBindingList = b;
           b->prefix->binding = b->prevPrefixBinding;
         }
-        if (parser->m_tagLevel == 0)
-          return epilogProcessor(parser, next, end, nextPtr);
-      }
-      break;
-    case XML_TOK_CHAR_REF:
-      {
-        int n = XmlCharRefNumber(enc, s);
-        if (n < 0)
-          return XML_ERROR_BAD_CHAR_REF;
-        if (parser->m_characterDataHandler) {
-          XML_Char buf[XML_ENCODE_MAX];
-          parser->m_characterDataHandler(parser->m_handlerArg, buf, XmlEncode(n, (ICHAR *)buf));
+        if ((parser->m_tagLevel == 0)
+            && (parser->m_parsingStatus.parsing != XML_FINISHED)) {
+          if (parser->m_parsingStatus.parsing == XML_SUSPENDED)
+            parser->m_processor = epilogProcessor;
+          else
+            return epilogProcessor(parser, next, end, nextPtr);
         }
-        else if (parser->m_defaultHandler)
-          reportDefault(parser, enc, s, next);
       }
       break;
+    case XML_TOK_CHAR_REF: {
+      int n = XmlCharRefNumber(enc, s);
+      if (n < 0)
+        return XML_ERROR_BAD_CHAR_REF;
+      if (parser->m_characterDataHandler) {
+        XML_Char buf[XML_ENCODE_MAX];
+        parser->m_characterDataHandler(parser->m_handlerArg, buf,
+                                       XmlEncode(n, (ICHAR *)buf));
+      } else if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+    } break;
     case XML_TOK_XML_DECL:
       return XML_ERROR_MISPLACED_XML_PI;
     case XML_TOK_DATA_NEWLINE:
       if (parser->m_characterDataHandler) {
         XML_Char c = 0xA;
         parser->m_characterDataHandler(parser->m_handlerArg, &c, 1);
-      }
-      else if (parser->m_defaultHandler)
+      } else if (parser->m_defaultHandler)
         reportDefault(parser, enc, s, next);
       break;
-    case XML_TOK_CDATA_SECT_OPEN:
-      {
-        enum XML_Error result;
-        if (parser->m_startCdataSectionHandler)
-          parser->m_startCdataSectionHandler(parser->m_handlerArg);
-/* BEGIN disabled code */
-        /* Suppose you doing a transformation on a document that involves
-           changing only the character data.  You set up a defaultHandler
-           and a characterDataHandler.  The defaultHandler simply copies
-           characters through.  The characterDataHandler does the
-           transformation and writes the characters out escaping them as
-           necessary.  This case will fail to work if we leave out the
-           following two lines (because & and < inside CDATA sections will
-           be incorrectly escaped).
-
-           However, now we have a start/endCdataSectionHandler, so it seems
-           easier to let the user deal with this.
-        */
-        else if (0 && parser->m_characterDataHandler)
-          parser->m_characterDataHandler(parser->m_handlerArg, parser->m_dataBuf, 0);
-/* END disabled code */
-        else if (parser->m_defaultHandler)
-          reportDefault(parser, enc, s, next);
-        result = doCdataSection(parser, enc, &next, end, nextPtr, haveMore);
-        if (result != XML_ERROR_NONE)
-          return result;
-        else if (!next) {
-          parser->m_processor = cdataSectionProcessor;
-          return result;
-        }
+    case XML_TOK_CDATA_SECT_OPEN: {
+      enum XML_Error result;
+      if (parser->m_startCdataSectionHandler)
+        parser->m_startCdataSectionHandler(parser->m_handlerArg);
+      /* BEGIN disabled code */
+      /* Suppose you doing a transformation on a document that involves
+         changing only the character data.  You set up a defaultHandler
+         and a characterDataHandler.  The defaultHandler simply copies
+         characters through.  The characterDataHandler does the
+         transformation and writes the characters out escaping them as
+         necessary.  This case will fail to work if we leave out the
+         following two lines (because & and < inside CDATA sections will
+         be incorrectly escaped).
+
+         However, now we have a start/endCdataSectionHandler, so it seems
+         easier to let the user deal with this.
+      */
+      else if (0 && parser->m_characterDataHandler)
+        parser->m_characterDataHandler(parser->m_handlerArg, parser->m_dataBuf,
+                                       0);
+      /* END disabled code */
+      else if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+      result = doCdataSection(parser, enc, &next, end, nextPtr, haveMore);
+      if (result != XML_ERROR_NONE)
+        return result;
+      else if (! next) {
+        parser->m_processor = cdataSectionProcessor;
+        return result;
       }
-      break;
+    } break;
     case XML_TOK_TRAILING_RSQB:
       if (haveMore) {
         *nextPtr = s;
@@ -3046,15 +2922,14 @@ doContent(XML_Parser parser,
         if (MUST_CONVERT(enc, s)) {
           ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf;
           XmlConvert(enc, &s, end, &dataPtr, (ICHAR *)parser->m_dataBufEnd);
-          parser->m_characterDataHandler(parser->m_handlerArg, parser->m_dataBuf,
-                               (int)(dataPtr - (ICHAR *)parser->m_dataBuf));
-        }
-        else
-          parser->m_characterDataHandler(parser->m_handlerArg,
-                               (XML_Char *)s,
-                               (int)((XML_Char *)end - (XML_Char *)s));
-      }
-      else if (parser->m_defaultHandler)
+          parser->m_characterDataHandler(
+              parser->m_handlerArg, parser->m_dataBuf,
+              (int)(dataPtr - (ICHAR *)parser->m_dataBuf));
+        } else
+          parser->m_characterDataHandler(
+              parser->m_handlerArg, (XML_Char *)s,
+              (int)((XML_Char *)end - (XML_Char *)s));
+      } else if (parser->m_defaultHandler)
         reportDefault(parser, enc, s, end);
       /* We are at the end of the final buffer, should we check for
          XML_SUSPENDED, XML_FINISHED?
@@ -3069,37 +2944,34 @@ doContent(XML_Parser parser,
       }
       *nextPtr = end;
       return XML_ERROR_NONE;
-    case XML_TOK_DATA_CHARS:
-      {
-        XML_CharacterDataHandler charDataHandler = parser->m_characterDataHandler;
-        if (charDataHandler) {
-          if (MUST_CONVERT(enc, s)) {
-            for (;;) {
-              ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf;
-              const enum XML_Convert_Result convert_res = XmlConvert(enc, &s, next, &dataPtr, (ICHAR *)parser->m_dataBufEnd);
-              *eventEndPP = s;
-              charDataHandler(parser->m_handlerArg, parser->m_dataBuf,
-                              (int)(dataPtr - (ICHAR *)parser->m_dataBuf));
-              if ((convert_res == XML_CONVERT_COMPLETED) || (convert_res == XML_CONVERT_INPUT_INCOMPLETE))
-                break;
-              *eventPP = s;
-            }
+    case XML_TOK_DATA_CHARS: {
+      XML_CharacterDataHandler charDataHandler = parser->m_characterDataHandler;
+      if (charDataHandler) {
+        if (MUST_CONVERT(enc, s)) {
+          for (;;) {
+            ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf;
+            const enum XML_Convert_Result convert_res = XmlConvert(
+                enc, &s, next, &dataPtr, (ICHAR *)parser->m_dataBufEnd);
+            *eventEndPP = s;
+            charDataHandler(parser->m_handlerArg, parser->m_dataBuf,
+                            (int)(dataPtr - (ICHAR *)parser->m_dataBuf));
+            if ((convert_res == XML_CONVERT_COMPLETED)
+                || (convert_res == XML_CONVERT_INPUT_INCOMPLETE))
+              break;
+            *eventPP = s;
           }
-          else
-            charDataHandler(parser->m_handlerArg,
-                            (XML_Char *)s,
-                            (int)((XML_Char *)next - (XML_Char *)s));
-        }
-        else if (parser->m_defaultHandler)
-          reportDefault(parser, enc, s, next);
-      }
-      break;
+        } else
+          charDataHandler(parser->m_handlerArg, (XML_Char *)s,
+                          (int)((XML_Char *)next - (XML_Char *)s));
+      } else if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+    } break;
     case XML_TOK_PI:
-      if (!reportProcessingInstruction(parser, enc, s, next))
+      if (! reportProcessingInstruction(parser, enc, s, next))
         return XML_ERROR_NO_MEMORY;
       break;
     case XML_TOK_COMMENT:
-      if (!reportComment(parser, enc, s, next))
+      if (! reportComment(parser, enc, s, next))
         return XML_ERROR_NO_MEMORY;
       break;
     default:
@@ -3122,7 +2994,7 @@ doContent(XML_Parser parser,
       return XML_ERROR_NONE;
     case XML_FINISHED:
       return XML_ERROR_ABORTED;
-    default: ;
+    default:;
     }
   }
   /* not reached */
@@ -3133,8 +3005,7 @@ doContent(XML_Parser parser,
  * reused as appropriate.
  */
 static void
-freeBindings(XML_Parser parser, BINDING *bindings)
-{
+freeBindings(XML_Parser parser, BINDING *bindings) {
   while (bindings) {
     BINDING *b = bindings;
 
@@ -3142,7 +3013,7 @@ freeBindings(XML_Parser parser, BINDING *bindings)
      * binding in addBindings(), so call the end handler now.
      */
     if (parser->m_endNamespaceDeclHandler)
-        parser->m_endNamespaceDeclHandler(parser->m_handlerArg, b->prefix->name);
+      parser->m_endNamespaceDeclHandler(parser->m_handlerArg, b->prefix->name);
 
     bindings = bindings->nextTagBinding;
     b->nextTagBinding = parser->m_freeBindingList;
@@ -3162,14 +3033,12 @@ freeBindings(XML_Parser parser, BINDING *bindings)
    - generate namespace aware element name (URI, prefix)
 */
 static enum XML_Error
-storeAtts(XML_Parser parser, const ENCODING *enc,
-          const char *attStr, TAG_NAME *tagNamePtr,
-          BINDING **bindingsPtr)
-{
-  DTD * const dtd = parser->m_dtd;  /* save one level of indirection */
+storeAtts(XML_Parser parser, const ENCODING *enc, const char *attStr,
+          TAG_NAME *tagNamePtr, BINDING **bindingsPtr) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   ELEMENT_TYPE *elementType;
   int nDefaultAtts;
-  const XML_Char **appAtts;   /* the attribute list for the application */
+  const XML_Char **appAtts; /* the attribute list for the application */
   int attIndex = 0;
   int prefixLen;
   int i;
@@ -3180,16 +3049,17 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
   const XML_Char *localPart;
 
   /* lookup the element type name */
-  elementType = (ELEMENT_TYPE *)lookup(parser, &dtd->elementTypes, tagNamePtr->str,0);
-  if (!elementType) {
+  elementType
+      = (ELEMENT_TYPE *)lookup(parser, &dtd->elementTypes, tagNamePtr->str, 0);
+  if (! elementType) {
     const XML_Char *name = poolCopyString(&dtd->pool, tagNamePtr->str);
-    if (!name)
+    if (! name)
       return XML_ERROR_NO_MEMORY;
     elementType = (ELEMENT_TYPE *)lookup(parser, &dtd->elementTypes, name,
                                          sizeof(ELEMENT_TYPE));
-    if (!elementType)
+    if (! elementType)
       return XML_ERROR_NO_MEMORY;
-    if (parser->m_ns && !setElementTypePrefix(parser, elementType))
+    if (parser->m_ns && ! setElementTypePrefix(parser, elementType))
       return XML_ERROR_NO_MEMORY;
   }
   nDefaultAtts = elementType->nDefaultAtts;
@@ -3203,14 +3073,16 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
     XML_AttrInfo *temp2;
 #endif
     parser->m_attsSize = n + nDefaultAtts + INIT_ATTS_SIZE;
-    temp = (ATTRIBUTE *)REALLOC(parser, (void *)parser->m_atts, parser->m_attsSize * sizeof(ATTRIBUTE));
+    temp = (ATTRIBUTE *)REALLOC(parser, (void *)parser->m_atts,
+                                parser->m_attsSize * sizeof(ATTRIBUTE));
     if (temp == NULL) {
       parser->m_attsSize = oldAttsSize;
       return XML_ERROR_NO_MEMORY;
     }
     parser->m_atts = temp;
 #ifdef XML_ATTR_INFO
-    temp2 = (XML_AttrInfo *)REALLOC(parser, (void *)parser->m_attInfo, parser->m_attsSize * sizeof(XML_AttrInfo));
+    temp2 = (XML_AttrInfo *)REALLOC(parser, (void *)parser->m_attInfo,
+                                    parser->m_attsSize * sizeof(XML_AttrInfo));
     if (temp2 == NULL) {
       parser->m_attsSize = oldAttsSize;
       return XML_ERROR_NO_MEMORY;
@@ -3228,18 +3100,20 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
     XML_AttrInfo *currAttInfo = &parser->m_attInfo[i];
 #endif
     /* add the name and value to the attribute list */
-    ATTRIBUTE_ID *attId = getAttributeId(parser, enc, currAtt->name,
-                                         currAtt->name
-                                         + XmlNameLength(enc, currAtt->name));
-    if (!attId)
+    ATTRIBUTE_ID *attId
+        = getAttributeId(parser, enc, currAtt->name,
+                         currAtt->name + XmlNameLength(enc, currAtt->name));
+    if (! attId)
       return XML_ERROR_NO_MEMORY;
 #ifdef XML_ATTR_INFO
-    currAttInfo->nameStart = parser->m_parseEndByteIndex - (parser->m_parseEndPtr - currAtt->name);
-    currAttInfo->nameEnd = currAttInfo->nameStart +
-                           XmlNameLength(enc, currAtt->name);
-    currAttInfo->valueStart = parser->m_parseEndByteIndex -
-                            (parser->m_parseEndPtr - currAtt->valuePtr);
-    currAttInfo->valueEnd = parser->m_parseEndByteIndex - (parser->m_parseEndPtr - currAtt->valueEnd);
+    currAttInfo->nameStart
+        = parser->m_parseEndByteIndex - (parser->m_parseEndPtr - currAtt->name);
+    currAttInfo->nameEnd
+        = currAttInfo->nameStart + XmlNameLength(enc, currAtt->name);
+    currAttInfo->valueStart = parser->m_parseEndByteIndex
+                              - (parser->m_parseEndPtr - currAtt->valuePtr);
+    currAttInfo->valueEnd = parser->m_parseEndByteIndex
+                            - (parser->m_parseEndPtr - currAtt->valueEnd);
 #endif
     /* Detect duplicate attributes by their QNames. This does not work when
        namespace processing is turned on and different prefixes for the same
@@ -3252,7 +3126,7 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
     }
     (attId->name)[-1] = 1;
     appAtts[attIndex++] = attId->name;
-    if (!parser->m_atts[i].normalized) {
+    if (! parser->m_atts[i].normalized) {
       enum XML_Error result;
       XML_Bool isCdata = XML_TRUE;
 
@@ -3268,17 +3142,17 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
       }
 
       /* normalize the attribute value */
-      result = storeAttributeValue(parser, enc, isCdata,
-                                   parser->m_atts[i].valuePtr, parser->m_atts[i].valueEnd,
-                                   &parser->m_tempPool);
+      result = storeAttributeValue(
+          parser, enc, isCdata, parser->m_atts[i].valuePtr,
+          parser->m_atts[i].valueEnd, &parser->m_tempPool);
       if (result)
         return result;
       appAtts[attIndex] = poolStart(&parser->m_tempPool);
       poolFinish(&parser->m_tempPool);
-    }
-    else {
+    } else {
       /* the value did not need normalizing */
-      appAtts[attIndex] = poolStoreString(&parser->m_tempPool, enc, parser->m_atts[i].valuePtr,
+      appAtts[attIndex] = poolStoreString(&parser->m_tempPool, enc,
+                                          parser->m_atts[i].valuePtr,
                                           parser->m_atts[i].valueEnd);
       if (appAtts[attIndex] == 0)
         return XML_ERROR_NO_MEMORY;
@@ -3293,15 +3167,13 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
         if (result)
           return result;
         --attIndex;
-      }
-      else {
+      } else {
         /* deal with other prefixed names later */
         attIndex++;
         nPrefixes++;
         (attId->name)[-1] = 2;
       }
-    }
-    else
+    } else
       attIndex++;
   }
 
@@ -3313,29 +3185,26 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
         parser->m_idAttIndex = i;
         break;
       }
-  }
-  else
+  } else
     parser->m_idAttIndex = -1;
 
   /* do attribute defaulting */
   for (i = 0; i < nDefaultAtts; i++) {
     const DEFAULT_ATTRIBUTE *da = elementType->defaultAtts + i;
-    if (!(da->id->name)[-1] && da->value) {
+    if (! (da->id->name)[-1] && da->value) {
       if (da->id->prefix) {
         if (da->id->xmlns) {
           enum XML_Error result = addBinding(parser, da->id->prefix, da->id,
                                              da->value, bindingsPtr);
           if (result)
             return result;
-        }
-        else {
+        } else {
           (da->id->name)[-1] = 2;
           nPrefixes++;
           appAtts[attIndex++] = da->id->name;
           appAtts[attIndex++] = da->value;
         }
-      }
-      else {
+      } else {
         (da->id->name)[-1] = 1;
         appAtts[attIndex++] = da->id->name;
         appAtts[attIndex++] = da->value;
@@ -3348,31 +3217,34 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
      and clear flags that say whether attributes were specified */
   i = 0;
   if (nPrefixes) {
-    int j;  /* hash table index */
+    int j; /* hash table index */
     unsigned long version = parser->m_nsAttsVersion;
     int nsAttsSize = (int)1 << parser->m_nsAttsPower;
     unsigned char oldNsAttsPower = parser->m_nsAttsPower;
     /* size of hash table must be at least 2 * (# of prefixed attributes) */
-    if ((nPrefixes << 1) >> parser->m_nsAttsPower) {  /* true for m_nsAttsPower = 0 */
+    if ((nPrefixes << 1)
+        >> parser->m_nsAttsPower) { /* true for m_nsAttsPower = 0 */
       NS_ATT *temp;
       /* hash table size must also be a power of 2 and >= 8 */
-      while (nPrefixes >> parser->m_nsAttsPower++);
+      while (nPrefixes >> parser->m_nsAttsPower++)
+        ;
       if (parser->m_nsAttsPower < 3)
         parser->m_nsAttsPower = 3;
       nsAttsSize = (int)1 << parser->m_nsAttsPower;
-      temp = (NS_ATT *)REALLOC(parser, parser->m_nsAtts, nsAttsSize * sizeof(NS_ATT));
-      if (!temp) {
+      temp = (NS_ATT *)REALLOC(parser, parser->m_nsAtts,
+                               nsAttsSize * sizeof(NS_ATT));
+      if (! temp) {
         /* Restore actual size of memory in m_nsAtts */
         parser->m_nsAttsPower = oldNsAttsPower;
         return XML_ERROR_NO_MEMORY;
       }
       parser->m_nsAtts = temp;
-      version = 0;  /* force re-initialization of m_nsAtts hash table */
+      version = 0; /* force re-initialization of m_nsAtts hash table */
     }
     /* using a version flag saves us from initializing m_nsAtts every time */
-    if (!version) {  /* initialize version flags when version wraps around */
+    if (! version) { /* initialize version flags when version wraps around */
       version = INIT_ATTS_VERSION;
-      for (j = nsAttsSize; j != 0; )
+      for (j = nsAttsSize; j != 0;)
         parser->m_nsAtts[--j].version = version;
     }
     parser->m_nsAttsVersion = --version;
@@ -3380,7 +3252,7 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
     /* expand prefixed names and check for duplicates */
     for (; i < attIndex; i += 2) {
       const XML_Char *s = appAtts[i];
-      if (s[-1] == 2) {  /* prefixed */
+      if (s[-1] == 2) { /* prefixed */
         ATTRIBUTE_ID *id;
         const BINDING *b;
         unsigned long uriHash;
@@ -3390,16 +3262,16 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
         copy_salt_to_sipkey(parser, &sip_key);
         sip24_init(&sip_state, &sip_key);
 
-        ((XML_Char *)s)[-1] = 0;  /* clear flag */
+        ((XML_Char *)s)[-1] = 0; /* clear flag */
         id = (ATTRIBUTE_ID *)lookup(parser, &dtd->attributeIds, s, 0);
-        if (!id || !id->prefix) {
+        if (! id || ! id->prefix) {
           /* This code is walking through the appAtts array, dealing
            * with (in this case) a prefixed attribute name.  To be in
            * the array, the attribute must have already been bound, so
            * has to have passed through the hash table lookup once
            * already.  That implies that an entry for it already
            * exists, so the lookup above will return a pointer to
-           * already allocated memory.  There is no opportunity for
+           * already allocated memory.  There is no opportunaity for
            * the allocator to fail, so the condition above cannot be
            * fulfilled.
            *
@@ -3410,12 +3282,12 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
           return XML_ERROR_NO_MEMORY; /* LCOV_EXCL_LINE */
         }
         b = id->prefix->binding;
-        if (!b)
+        if (! b)
           return XML_ERROR_UNBOUND_PREFIX;
 
         for (j = 0; j < b->uriLen; j++) {
           const XML_Char c = b->uri[j];
-          if (!poolAppendChar(&parser->m_tempPool, c))
+          if (! poolAppendChar(&parser->m_tempPool, c))
             return XML_ERROR_NO_MEMORY;
         }
 
@@ -3426,8 +3298,8 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
 
         sip24_update(&sip_state, s, keylen(s) * sizeof(XML_Char));
 
-        do {  /* copies null terminator */
-          if (!poolAppendChar(&parser->m_tempPool, *s))
+        do { /* copies null terminator */
+          if (! poolAppendChar(&parser->m_tempPool, *s))
             return XML_ERROR_NO_MEMORY;
         } while (*s++);
 
@@ -3438,28 +3310,29 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
           */
           unsigned char step = 0;
           unsigned long mask = nsAttsSize - 1;
-          j = uriHash & mask;  /* index into hash table */
+          j = uriHash & mask; /* index into hash table */
           while (parser->m_nsAtts[j].version == version) {
             /* for speed we compare stored hash values first */
             if (uriHash == parser->m_nsAtts[j].hash) {
               const XML_Char *s1 = poolStart(&parser->m_tempPool);
               const XML_Char *s2 = parser->m_nsAtts[j].uriName;
               /* s1 is null terminated, but not s2 */
-              for (; *s1 == *s2 && *s1 != 0; s1++, s2++);
+              for (; *s1 == *s2 && *s1 != 0; s1++, s2++)
+                ;
               if (*s1 == 0)
                 return XML_ERROR_DUPLICATE_ATTRIBUTE;
             }
-            if (!step)
+            if (! step)
               step = PROBE_STEP(uriHash, mask, parser->m_nsAttsPower);
             j < step ? (j += nsAttsSize - step) : (j -= step);
           }
         }
 
-        if (parser->m_ns_triplets) {  /* append namespace separator and prefix */
+        if (parser->m_ns_triplets) { /* append namespace separator and prefix */
           parser->m_tempPool.ptr[-1] = parser->m_namespaceSeparator;
           s = b->prefix->name;
           do {
-            if (!poolAppendChar(&parser->m_tempPool, *s))
+            if (! poolAppendChar(&parser->m_tempPool, *s))
               return XML_ERROR_NO_MEMORY;
           } while (*s++);
         }
@@ -3474,13 +3347,12 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
         parser->m_nsAtts[j].hash = uriHash;
         parser->m_nsAtts[j].uriName = s;
 
-        if (!--nPrefixes) {
+        if (! --nPrefixes) {
           i += 2;
           break;
         }
-      }
-      else  /* not prefixed */
-        ((XML_Char *)s)[-1] = 0;  /* clear flag */
+      } else                     /* not prefixed */
+        ((XML_Char *)s)[-1] = 0; /* clear flag */
     }
   }
   /* clear flags for the remaining attributes */
@@ -3489,40 +3361,38 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
   for (binding = *bindingsPtr; binding; binding = binding->nextTagBinding)
     binding->attId->name[-1] = 0;
 
-  if (!parser->m_ns)
+  if (! parser->m_ns)
     return XML_ERROR_NONE;
 
   /* expand the element type name */
   if (elementType->prefix) {
     binding = elementType->prefix->binding;
-    if (!binding)
+    if (! binding)
       return XML_ERROR_UNBOUND_PREFIX;
     localPart = tagNamePtr->str;
     while (*localPart++ != XML_T(ASCII_COLON))
       ;
-  }
-  else if (dtd->defaultPrefix.binding) {
+  } else if (dtd->defaultPrefix.binding) {
     binding = dtd->defaultPrefix.binding;
     localPart = tagNamePtr->str;
-  }
-  else
+  } else
     return XML_ERROR_NONE;
   prefixLen = 0;
   if (parser->m_ns_triplets && binding->prefix->name) {
     for (; binding->prefix->name[prefixLen++];)
-      ;  /* prefixLen includes null terminator */
+      ; /* prefixLen includes null terminator */
   }
   tagNamePtr->localPart = localPart;
   tagNamePtr->uriLen = binding->uriLen;
   tagNamePtr->prefix = binding->prefix->name;
   tagNamePtr->prefixLen = prefixLen;
   for (i = 0; localPart[i++];)
-    ;  /* i includes null terminator */
+    ; /* i includes null terminator */
   n = i + binding->uriLen + prefixLen;
   if (n > binding->uriAlloc) {
     TAG *p;
     uri = (XML_Char *)MALLOC(parser, (n + EXPAND_SPARE) * sizeof(XML_Char));
-    if (!uri)
+    if (! uri)
       return XML_ERROR_NO_MEMORY;
     binding->uriAlloc = n + EXPAND_SPARE;
     memcpy(uri, binding->uri, binding->uriLen * sizeof(XML_Char));
@@ -3538,7 +3408,7 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
   /* we always have a namespace separator between localPart and prefix */
   if (prefixLen) {
     uri += i - 1;
-    *uri = parser->m_namespaceSeparator;  /* replace null terminator */
+    *uri = parser->m_namespaceSeparator; /* replace null terminator */
     memcpy(uri + 1, binding->prefix->name, prefixLen * sizeof(XML_Char));
   }
   tagNamePtr->str = binding->uri;
@@ -3550,27 +3420,25 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
 */
 static enum XML_Error
 addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId,
-           const XML_Char *uri, BINDING **bindingsPtr)
-{
-  static const XML_Char xmlNamespace[] = {
-    ASCII_h, ASCII_t, ASCII_t, ASCII_p, ASCII_COLON, ASCII_SLASH, ASCII_SLASH,
-    ASCII_w, ASCII_w, ASCII_w, ASCII_PERIOD, ASCII_w, ASCII_3, ASCII_PERIOD,
-    ASCII_o, ASCII_r, ASCII_g, ASCII_SLASH, ASCII_X, ASCII_M, ASCII_L,
-    ASCII_SLASH, ASCII_1, ASCII_9, ASCII_9, ASCII_8, ASCII_SLASH,
-    ASCII_n, ASCII_a, ASCII_m, ASCII_e, ASCII_s, ASCII_p, ASCII_a, ASCII_c,
-    ASCII_e, '\0'
-  };
-  static const int xmlLen =
-    (int)sizeof(xmlNamespace)/sizeof(XML_Char) - 1;
-  static const XML_Char xmlnsNamespace[] = {
-    ASCII_h, ASCII_t, ASCII_t, ASCII_p, ASCII_COLON, ASCII_SLASH, ASCII_SLASH,
-    ASCII_w, ASCII_w, ASCII_w, ASCII_PERIOD, ASCII_w, ASCII_3, ASCII_PERIOD,
-    ASCII_o, ASCII_r, ASCII_g, ASCII_SLASH, ASCII_2, ASCII_0, ASCII_0,
-    ASCII_0, ASCII_SLASH, ASCII_x, ASCII_m, ASCII_l, ASCII_n, ASCII_s,
-    ASCII_SLASH, '\0'
-  };
-  static const int xmlnsLen =
-    (int)sizeof(xmlnsNamespace)/sizeof(XML_Char) - 1;
+           const XML_Char *uri, BINDING **bindingsPtr) {
+  static const XML_Char xmlNamespace[]
+      = {ASCII_h,      ASCII_t,     ASCII_t,     ASCII_p,      ASCII_COLON,
+         ASCII_SLASH,  ASCII_SLASH, ASCII_w,     ASCII_w,      ASCII_w,
+         ASCII_PERIOD, ASCII_w,     ASCII_3,     ASCII_PERIOD, ASCII_o,
+         ASCII_r,      ASCII_g,     ASCII_SLASH, ASCII_X,      ASCII_M,
+         ASCII_L,      ASCII_SLASH, ASCII_1,     ASCII_9,      ASCII_9,
+         ASCII_8,      ASCII_SLASH, ASCII_n,     ASCII_a,      ASCII_m,
+         ASCII_e,      ASCII_s,     ASCII_p,     ASCII_a,      ASCII_c,
+         ASCII_e,      '\0'};
+  static const int xmlLen = (int)sizeof(xmlNamespace) / sizeof(XML_Char) - 1;
+  static const XML_Char xmlnsNamespace[]
+      = {ASCII_h,     ASCII_t,      ASCII_t, ASCII_p, ASCII_COLON,  ASCII_SLASH,
+         ASCII_SLASH, ASCII_w,      ASCII_w, ASCII_w, ASCII_PERIOD, ASCII_w,
+         ASCII_3,     ASCII_PERIOD, ASCII_o, ASCII_r, ASCII_g,      ASCII_SLASH,
+         ASCII_2,     ASCII_0,      ASCII_0, ASCII_0, ASCII_SLASH,  ASCII_x,
+         ASCII_m,     ASCII_l,      ASCII_n, ASCII_s, ASCII_SLASH,  '\0'};
+  static const int xmlnsLen
+      = (int)sizeof(xmlnsNamespace) / sizeof(XML_Char) - 1;
 
   XML_Bool mustBeXML = XML_FALSE;
   XML_Bool isXML = XML_TRUE;
@@ -3583,14 +3451,11 @@ addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId,
   if (*uri == XML_T('\0') && prefix->name)
     return XML_ERROR_UNDECLARING_PREFIX;
 
-  if (prefix->name
-      && prefix->name[0] == XML_T(ASCII_x)
+  if (prefix->name && prefix->name[0] == XML_T(ASCII_x)
       && prefix->name[1] == XML_T(ASCII_m)
       && prefix->name[2] == XML_T(ASCII_l)) {
-
     /* Not allowed to bind xmlns */
-    if (prefix->name[3] == XML_T(ASCII_n)
-        && prefix->name[4] == XML_T(ASCII_s)
+    if (prefix->name[3] == XML_T(ASCII_n) && prefix->name[4] == XML_T(ASCII_s)
         && prefix->name[5] == XML_T('\0'))
       return XML_ERROR_RESERVED_PREFIX_XMLNS;
 
@@ -3602,7 +3467,7 @@ addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId,
     if (isXML && (len > xmlLen || uri[len] != xmlNamespace[len]))
       isXML = XML_FALSE;
 
-    if (!mustBeXML && isXMLNS
+    if (! mustBeXML && isXMLNS
         && (len > xmlnsLen || uri[len] != xmlnsNamespace[len]))
       isXMLNS = XML_FALSE;
   }
@@ -3621,21 +3486,21 @@ addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId,
   if (parser->m_freeBindingList) {
     b = parser->m_freeBindingList;
     if (len > b->uriAlloc) {
-      XML_Char *temp = (XML_Char *)REALLOC(parser, b->uri,
-                          sizeof(XML_Char) * (len + EXPAND_SPARE));
+      XML_Char *temp = (XML_Char *)REALLOC(
+          parser, b->uri, sizeof(XML_Char) * (len + EXPAND_SPARE));
       if (temp == NULL)
         return XML_ERROR_NO_MEMORY;
       b->uri = temp;
       b->uriAlloc = len + EXPAND_SPARE;
     }
     parser->m_freeBindingList = b->nextTagBinding;
-  }
-  else {
+  } else {
     b = (BINDING *)MALLOC(parser, sizeof(BINDING));
-    if (!b)
+    if (! b)
       return XML_ERROR_NO_MEMORY;
-    b->uri = (XML_Char *)MALLOC(parser, sizeof(XML_Char) * (len + EXPAND_SPARE));
-    if (!b->uri) {
+    b->uri
+        = (XML_Char *)MALLOC(parser, sizeof(XML_Char) * (len + EXPAND_SPARE));
+    if (! b->uri) {
       FREE(parser, b);
       return XML_ERROR_NO_MEMORY;
     }
@@ -3658,7 +3523,7 @@ addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId,
   /* if attId == NULL then we are not starting a namespace scope */
   if (attId && parser->m_startNamespaceDeclHandler)
     parser->m_startNamespaceDeclHandler(parser->m_handlerArg, prefix->name,
-                              prefix->binding ? uri : 0);
+                                        prefix->binding ? uri : 0);
   return XML_ERROR_NONE;
 }
 
@@ -3666,21 +3531,18 @@ addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId,
    the whole file is parsed with one call.
 */
 static enum XML_Error PTRCALL
-cdataSectionProcessor(XML_Parser parser,
-                      const char *start,
-                      const char *end,
-                      const char **endPtr)
-{
-  enum XML_Error result = doCdataSection(parser, parser->m_encoding, &start, end,
-                                         endPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer);
+cdataSectionProcessor(XML_Parser parser, const char *start, const char *end,
+                      const char **endPtr) {
+  enum XML_Error result
+      = doCdataSection(parser, parser->m_encoding, &start, end, endPtr,
+                       (XML_Bool)! parser->m_parsingStatus.finalBuffer);
   if (result != XML_ERROR_NONE)
     return result;
   if (start) {
-    if (parser->m_parentParser) {  /* we are parsing an external entity */
+    if (parser->m_parentParser) { /* we are parsing an external entity */
       parser->m_processor = externalEntityContentProcessor;
       return externalEntityContentProcessor(parser, start, end, endPtr);
-    }
-    else {
+    } else {
       parser->m_processor = contentProcessor;
       return contentProcessor(parser, start, end, endPtr);
     }
@@ -3692,13 +3554,8 @@ cdataSectionProcessor(XML_Parser parser,
    the section is not yet closed.
 */
 static enum XML_Error
-doCdataSection(XML_Parser parser,
-               const ENCODING *enc,
-               const char **startPtr,
-               const char *end,
-               const char **nextPtr,
-               XML_Bool haveMore)
-{
+doCdataSection(XML_Parser parser, const ENCODING *enc, const char **startPtr,
+               const char *end, const char **nextPtr, XML_Bool haveMore) {
   const char *s = *startPtr;
   const char **eventPP;
   const char **eventEndPP;
@@ -3706,8 +3563,7 @@ doCdataSection(XML_Parser parser,
     eventPP = &parser->m_eventPtr;
     *eventPP = s;
     eventEndPP = &parser->m_eventEndPtr;
-  }
-  else {
+  } else {
     eventPP = &(parser->m_openInternalEntities->internalEventPtr);
     eventEndPP = &(parser->m_openInternalEntities->internalEventEndPtr);
   }
@@ -3722,11 +3578,12 @@ doCdataSection(XML_Parser parser,
     case XML_TOK_CDATA_SECT_CLOSE:
       if (parser->m_endCdataSectionHandler)
         parser->m_endCdataSectionHandler(parser->m_handlerArg);
-/* BEGIN disabled code */
+      /* BEGIN disabled code */
       /* see comment under XML_TOK_CDATA_SECT_OPEN */
       else if (0 && parser->m_characterDataHandler)
-        parser->m_characterDataHandler(parser->m_handlerArg, parser->m_dataBuf, 0);
-/* END disabled code */
+        parser->m_characterDataHandler(parser->m_handlerArg, parser->m_dataBuf,
+                                       0);
+      /* END disabled code */
       else if (parser->m_defaultHandler)
         reportDefault(parser, enc, s, next);
       *startPtr = next;
@@ -3739,35 +3596,31 @@ doCdataSection(XML_Parser parser,
       if (parser->m_characterDataHandler) {
         XML_Char c = 0xA;
         parser->m_characterDataHandler(parser->m_handlerArg, &c, 1);
-      }
-      else if (parser->m_defaultHandler)
+      } else if (parser->m_defaultHandler)
         reportDefault(parser, enc, s, next);
       break;
-    case XML_TOK_DATA_CHARS:
-      {
-        XML_CharacterDataHandler charDataHandler = parser->m_characterDataHandler;
-        if (charDataHandler) {
-          if (MUST_CONVERT(enc, s)) {
-            for (;;) {
-              ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf;
-              const enum XML_Convert_Result convert_res = XmlConvert(enc, &s, next, &dataPtr, (ICHAR *)parser->m_dataBufEnd);
-              *eventEndPP = next;
-              charDataHandler(parser->m_handlerArg, parser->m_dataBuf,
-                              (int)(dataPtr - (ICHAR *)parser->m_dataBuf));
-              if ((convert_res == XML_CONVERT_COMPLETED) || (convert_res == XML_CONVERT_INPUT_INCOMPLETE))
-                break;
-              *eventPP = s;
-            }
+    case XML_TOK_DATA_CHARS: {
+      XML_CharacterDataHandler charDataHandler = parser->m_characterDataHandler;
+      if (charDataHandler) {
+        if (MUST_CONVERT(enc, s)) {
+          for (;;) {
+            ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf;
+            const enum XML_Convert_Result convert_res = XmlConvert(
+                enc, &s, next, &dataPtr, (ICHAR *)parser->m_dataBufEnd);
+            *eventEndPP = next;
+            charDataHandler(parser->m_handlerArg, parser->m_dataBuf,
+                            (int)(dataPtr - (ICHAR *)parser->m_dataBuf));
+            if ((convert_res == XML_CONVERT_COMPLETED)
+                || (convert_res == XML_CONVERT_INPUT_INCOMPLETE))
+              break;
+            *eventPP = s;
           }
-          else
-            charDataHandler(parser->m_handlerArg,
-                            (XML_Char *)s,
-                            (int)((XML_Char *)next - (XML_Char *)s));
-        }
-        else if (parser->m_defaultHandler)
-          reportDefault(parser, enc, s, next);
-      }
-      break;
+        } else
+          charDataHandler(parser->m_handlerArg, (XML_Char *)s,
+                          (int)((XML_Char *)next - (XML_Char *)s));
+      } else if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+    } break;
     case XML_TOK_INVALID:
       *eventPP = next;
       return XML_ERROR_INVALID_TOKEN;
@@ -3791,7 +3644,7 @@ doCdataSection(XML_Parser parser,
        * statistics.
        *
        * LCOV_EXCL_START
-      */
+       */
       *eventPP = next;
       return XML_ERROR_UNEXPECTED_STATE;
       /* LCOV_EXCL_STOP */
@@ -3804,7 +3657,7 @@ doCdataSection(XML_Parser parser,
       return XML_ERROR_NONE;
     case XML_FINISHED:
       return XML_ERROR_ABORTED;
-    default: ;
+    default:;
     }
   }
   /* not reached */
@@ -3816,13 +3669,11 @@ doCdataSection(XML_Parser parser,
    the whole file is parsed with one call.
 */
 static enum XML_Error PTRCALL
-ignoreSectionProcessor(XML_Parser parser,
-                       const char *start,
-                       const char *end,
-                       const char **endPtr)
-{
-  enum XML_Error result = doIgnoreSection(parser, parser->m_encoding, &start, end,
-                                          endPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer);
+ignoreSectionProcessor(XML_Parser parser, const char *start, const char *end,
+                       const char **endPtr) {
+  enum XML_Error result
+      = doIgnoreSection(parser, parser->m_encoding, &start, end, endPtr,
+                        (XML_Bool)! parser->m_parsingStatus.finalBuffer);
   if (result != XML_ERROR_NONE)
     return result;
   if (start) {
@@ -3836,13 +3687,8 @@ ignoreSectionProcessor(XML_Parser parser,
    if the section is not yet closed.
 */
 static enum XML_Error
-doIgnoreSection(XML_Parser parser,
-                const ENCODING *enc,
-                const char **startPtr,
-                const char *end,
-                const char **nextPtr,
-                XML_Bool haveMore)
-{
+doIgnoreSection(XML_Parser parser, const ENCODING *enc, const char **startPtr,
+                const char *end, const char **nextPtr, XML_Bool haveMore) {
   const char *next;
   int tok;
   const char *s = *startPtr;
@@ -3852,8 +3698,7 @@ doIgnoreSection(XML_Parser parser,
     eventPP = &parser->m_eventPtr;
     *eventPP = s;
     eventEndPP = &parser->m_eventEndPtr;
-  }
-  else {
+  } else {
     /* It's not entirely clear, but it seems the following two lines
      * of code cannot be executed.  The only occasions on which 'enc'
      * is not 'encoding' are when this function is called
@@ -3917,13 +3762,12 @@ doIgnoreSection(XML_Parser parser,
 #endif /* XML_DTD */
 
 static enum XML_Error
-initializeEncoding(XML_Parser parser)
-{
+initializeEncoding(XML_Parser parser) {
   const char *s;
 #ifdef XML_UNICODE
   char encodingBuf[128];
   /* See comments abount `protoclEncodingName` in parserInit() */
-  if (!parser->m_protocolEncodingName)
+  if (! parser->m_protocolEncodingName)
     s = NULL;
   else {
     int i;
@@ -3941,15 +3785,15 @@ initializeEncoding(XML_Parser parser)
 #else
   s = parser->m_protocolEncodingName;
 #endif
-  if ((parser->m_ns ? XmlInitEncodingNS : XmlInitEncoding)(&parser->m_initEncoding, &parser->m_encoding, s))
+  if ((parser->m_ns ? XmlInitEncodingNS : XmlInitEncoding)(
+          &parser->m_initEncoding, &parser->m_encoding, s))
     return XML_ERROR_NONE;
   return handleUnknownEncoding(parser, parser->m_protocolEncodingName);
 }
 
 static enum XML_Error
-processXmlDecl(XML_Parser parser, int isGeneralTextEntity,
-               const char *s, const char *next)
-{
+processXmlDecl(XML_Parser parser, int isGeneralTextEntity, const char *s,
+               const char *next) {
   const char *encodingName = NULL;
   const XML_Char *storedEncName = NULL;
   const ENCODING *newEncoding = NULL;
@@ -3957,52 +3801,41 @@ processXmlDecl(XML_Parser parser, int isGeneralTextEntity,
   const char *versionend;
   const XML_Char *storedversion = NULL;
   int standalone = -1;
-  if (!(parser->m_ns
-        ? XmlParseXmlDeclNS
-        : XmlParseXmlDecl)(isGeneralTextEntity,
-                           parser->m_encoding,
-                           s,
-                           next,
-                           &parser->m_eventPtr,
-                           &version,
-                           &versionend,
-                           &encodingName,
-                           &newEncoding,
-                           &standalone)) {
+  if (! (parser->m_ns ? XmlParseXmlDeclNS : XmlParseXmlDecl)(
+          isGeneralTextEntity, parser->m_encoding, s, next, &parser->m_eventPtr,
+          &version, &versionend, &encodingName, &newEncoding, &standalone)) {
     if (isGeneralTextEntity)
       return XML_ERROR_TEXT_DECL;
     else
       return XML_ERROR_XML_DECL;
   }
-  if (!isGeneralTextEntity && standalone == 1) {
+  if (! isGeneralTextEntity && standalone == 1) {
     parser->m_dtd->standalone = XML_TRUE;
 #ifdef XML_DTD
-    if (parser->m_paramEntityParsing == XML_PARAM_ENTITY_PARSING_UNLESS_STANDALONE)
+    if (parser->m_paramEntityParsing
+        == XML_PARAM_ENTITY_PARSING_UNLESS_STANDALONE)
       parser->m_paramEntityParsing = XML_PARAM_ENTITY_PARSING_NEVER;
 #endif /* XML_DTD */
   }
   if (parser->m_xmlDeclHandler) {
     if (encodingName != NULL) {
-      storedEncName = poolStoreString(&parser->m_temp2Pool,
-                                      parser->m_encoding,
-                                      encodingName,
-                                      encodingName
-                                      + XmlNameLength(parser->m_encoding, encodingName));
-      if (!storedEncName)
-              return XML_ERROR_NO_MEMORY;
+      storedEncName = poolStoreString(
+          &parser->m_temp2Pool, parser->m_encoding, encodingName,
+          encodingName + XmlNameLength(parser->m_encoding, encodingName));
+      if (! storedEncName)
+        return XML_ERROR_NO_MEMORY;
       poolFinish(&parser->m_temp2Pool);
     }
     if (version) {
-      storedversion = poolStoreString(&parser->m_temp2Pool,
-                                      parser->m_encoding,
-                                      version,
-                                      versionend - parser->m_encoding->minBytesPerChar);
-      if (!storedversion)
+      storedversion
+          = poolStoreString(&parser->m_temp2Pool, parser->m_encoding, version,
+                            versionend - parser->m_encoding->minBytesPerChar);
+      if (! storedversion)
         return XML_ERROR_NO_MEMORY;
     }
-    parser->m_xmlDeclHandler(parser->m_handlerArg, storedversion, storedEncName, standalone);
-  }
-  else if (parser->m_defaultHandler)
+    parser->m_xmlDeclHandler(parser->m_handlerArg, storedversion, storedEncName,
+                             standalone);
+  } else if (parser->m_defaultHandler)
     reportDefault(parser, parser->m_encoding, s, next);
   if (parser->m_protocolEncodingName == NULL) {
     if (newEncoding) {
@@ -4012,20 +3845,19 @@ processXmlDecl(XML_Parser parser, int isGeneralTextEntity,
        * this is UTF-16, is it the same endianness?
        */
       if (newEncoding->minBytesPerChar != parser->m_encoding->minBytesPerChar
-          || (newEncoding->minBytesPerChar == 2 &&
-              newEncoding != parser->m_encoding)) {
+          || (newEncoding->minBytesPerChar == 2
+              && newEncoding != parser->m_encoding)) {
         parser->m_eventPtr = encodingName;
         return XML_ERROR_INCORRECT_ENCODING;
       }
       parser->m_encoding = newEncoding;
-    }
-    else if (encodingName) {
+    } else if (encodingName) {
       enum XML_Error result;
-      if (!storedEncName) {
+      if (! storedEncName) {
         storedEncName = poolStoreString(
-          &parser->m_temp2Pool, parser->m_encoding, encodingName,
-          encodingName + XmlNameLength(parser->m_encoding, encodingName));
-        if (!storedEncName)
+            &parser->m_temp2Pool, parser->m_encoding, encodingName,
+            encodingName + XmlNameLength(parser->m_encoding, encodingName));
+        if (! storedEncName)
           return XML_ERROR_NO_MEMORY;
       }
       result = handleUnknownEncoding(parser, storedEncName);
@@ -4043,8 +3875,7 @@ processXmlDecl(XML_Parser parser, int isGeneralTextEntity,
 }
 
 static enum XML_Error
-handleUnknownEncoding(XML_Parser parser, const XML_Char *encodingName)
-{
+handleUnknownEncoding(XML_Parser parser, const XML_Char *encodingName) {
   if (parser->m_unknownEncodingHandler) {
     XML_Encoding info;
     int i;
@@ -4053,21 +3884,17 @@ handleUnknownEncoding(XML_Parser parser, const XML_Char *encodingName)
     info.convert = NULL;
     info.data = NULL;
     info.release = NULL;
-    if (parser->m_unknownEncodingHandler(parser->m_unknownEncodingHandlerData, encodingName,
-                               &info)) {
+    if (parser->m_unknownEncodingHandler(parser->m_unknownEncodingHandlerData,
+                                         encodingName, &info)) {
       ENCODING *enc;
       parser->m_unknownEncodingMem = MALLOC(parser, XmlSizeOfUnknownEncoding());
-      if (!parser->m_unknownEncodingMem) {
+      if (! parser->m_unknownEncodingMem) {
         if (info.release)
           info.release(info.data);
         return XML_ERROR_NO_MEMORY;
       }
-      enc = (parser->m_ns
-             ? XmlInitUnknownEncodingNS
-             : XmlInitUnknownEncoding)(parser->m_unknownEncodingMem,
-                                       info.map,
-                                       info.convert,
-                                       info.data);
+      enc = (parser->m_ns ? XmlInitUnknownEncodingNS : XmlInitUnknownEncoding)(
+          parser->m_unknownEncodingMem, info.map, info.convert, info.data);
       if (enc) {
         parser->m_unknownEncodingData = info.data;
         parser->m_unknownEncodingRelease = info.release;
@@ -4082,11 +3909,8 @@ handleUnknownEncoding(XML_Parser parser, const XML_Char *encodingName)
 }
 
 static enum XML_Error PTRCALL
-prologInitProcessor(XML_Parser parser,
-                    const char *s,
-                    const char *end,
-                    const char **nextPtr)
-{
+prologInitProcessor(XML_Parser parser, const char *s, const char *end,
+                    const char **nextPtr) {
   enum XML_Error result = initializeEncoding(parser);
   if (result != XML_ERROR_NONE)
     return result;
@@ -4097,11 +3921,8 @@ prologInitProcessor(XML_Parser parser,
 #ifdef XML_DTD
 
 static enum XML_Error PTRCALL
-externalParEntInitProcessor(XML_Parser parser,
-                            const char *s,
-                            const char *end,
-                            const char **nextPtr)
-{
+externalParEntInitProcessor(XML_Parser parser, const char *s, const char *end,
+                            const char **nextPtr) {
   enum XML_Error result = initializeEncoding(parser);
   if (result != XML_ERROR_NONE)
     return result;
@@ -4113,19 +3934,15 @@ externalParEntInitProcessor(XML_Parser parser,
   if (parser->m_prologState.inEntityValue) {
     parser->m_processor = entityValueInitProcessor;
     return entityValueInitProcessor(parser, s, end, nextPtr);
-  }
-  else {
+  } else {
     parser->m_processor = externalParEntProcessor;
     return externalParEntProcessor(parser, s, end, nextPtr);
   }
 }
 
 static enum XML_Error PTRCALL
-entityValueInitProcessor(XML_Parser parser,
-                         const char *s,
-                         const char *end,
-                         const char **nextPtr)
-{
+entityValueInitProcessor(XML_Parser parser, const char *s, const char *end,
+                         const char **nextPtr) {
   int tok;
   const char *start = s;
   const char *next = start;
@@ -4135,7 +3952,7 @@ entityValueInitProcessor(XML_Parser parser,
     tok = XmlPrologTok(parser->m_encoding, start, end, &next);
     parser->m_eventEndPtr = next;
     if (tok <= 0) {
-      if (!parser->m_parsingStatus.finalBuffer && tok != XML_TOK_INVALID) {
+      if (! parser->m_parsingStatus.finalBuffer && tok != XML_TOK_INVALID) {
         *nextPtr = s;
         return XML_ERROR_NONE;
       }
@@ -4146,22 +3963,21 @@ entityValueInitProcessor(XML_Parser parser,
         return XML_ERROR_UNCLOSED_TOKEN;
       case XML_TOK_PARTIAL_CHAR:
         return XML_ERROR_PARTIAL_CHAR;
-      case XML_TOK_NONE:   /* start == end */
+      case XML_TOK_NONE: /* start == end */
       default:
         break;
       }
       /* found end of entity value - can store it now */
       return storeEntityValue(parser, parser->m_encoding, s, end);
-    }
-    else if (tok == XML_TOK_XML_DECL) {
+    } else if (tok == XML_TOK_XML_DECL) {
       enum XML_Error result;
       result = processXmlDecl(parser, 0, start, next);
       if (result != XML_ERROR_NONE)
         return result;
-      /* At this point, m_parsingStatus.parsing cannot be XML_SUSPENDED.  For that
-       * to happen, a parameter entity parsing handler must have
-       * attempted to suspend the parser, which fails and raises an
-       * error.  The parser can be aborted, but can't be suspended.
+      /* At this point, m_parsingStatus.parsing cannot be XML_SUSPENDED.  For
+       * that to happen, a parameter entity parsing handler must have attempted
+       * to suspend the parser, which fails and raises an error.  The parser can
+       * be aborted, but can't be suspended.
        */
       if (parser->m_parsingStatus.parsing == XML_FINISHED)
         return XML_ERROR_ABORTED;
@@ -4177,7 +3993,8 @@ entityValueInitProcessor(XML_Parser parser,
        then, when this routine is entered the next time, XmlPrologTok will
        return XML_TOK_INVALID, since the BOM is still in the buffer
     */
-    else if (tok == XML_TOK_BOM && next == end && !parser->m_parsingStatus.finalBuffer) {
+    else if (tok == XML_TOK_BOM && next == end
+             && ! parser->m_parsingStatus.finalBuffer) {
       *nextPtr = next;
       return XML_ERROR_NONE;
     }
@@ -4195,17 +4012,14 @@ entityValueInitProcessor(XML_Parser parser,
 }
 
 static enum XML_Error PTRCALL
-externalParEntProcessor(XML_Parser parser,
-                        const char *s,
-                        const char *end,
-                        const char **nextPtr)
-{
+externalParEntProcessor(XML_Parser parser, const char *s, const char *end,
+                        const char **nextPtr) {
   const char *next = s;
   int tok;
 
   tok = XmlPrologTok(parser->m_encoding, s, end, &next);
   if (tok <= 0) {
-    if (!parser->m_parsingStatus.finalBuffer && tok != XML_TOK_INVALID) {
+    if (! parser->m_parsingStatus.finalBuffer && tok != XML_TOK_INVALID) {
       *nextPtr = s;
       return XML_ERROR_NONE;
     }
@@ -4216,7 +4030,7 @@ externalParEntProcessor(XML_Parser parser,
       return XML_ERROR_UNCLOSED_TOKEN;
     case XML_TOK_PARTIAL_CHAR:
       return XML_ERROR_PARTIAL_CHAR;
-    case XML_TOK_NONE:   /* start == end */
+    case XML_TOK_NONE: /* start == end */
     default:
       break;
     }
@@ -4231,16 +4045,13 @@ externalParEntProcessor(XML_Parser parser,
   }
 
   parser->m_processor = prologProcessor;
-  return doProlog(parser, parser->m_encoding, s, end, tok, next,
-                  nextPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer);
+  return doProlog(parser, parser->m_encoding, s, end, tok, next, nextPtr,
+                  (XML_Bool)! parser->m_parsingStatus.finalBuffer, XML_TRUE);
 }
 
 static enum XML_Error PTRCALL
-entityValueProcessor(XML_Parser parser,
-                     const char *s,
-                     const char *end,
-                     const char **nextPtr)
-{
+entityValueProcessor(XML_Parser parser, const char *s, const char *end,
+                     const char **nextPtr) {
   const char *start = s;
   const char *next = s;
   const ENCODING *enc = parser->m_encoding;
@@ -4249,7 +4060,7 @@ entityValueProcessor(XML_Parser parser,
   for (;;) {
     tok = XmlPrologTok(enc, start, end, &next);
     if (tok <= 0) {
-      if (!parser->m_parsingStatus.finalBuffer && tok != XML_TOK_INVALID) {
+      if (! parser->m_parsingStatus.finalBuffer && tok != XML_TOK_INVALID) {
         *nextPtr = s;
         return XML_ERROR_NONE;
       }
@@ -4260,7 +4071,7 @@ entityValueProcessor(XML_Parser parser,
         return XML_ERROR_UNCLOSED_TOKEN;
       case XML_TOK_PARTIAL_CHAR:
         return XML_ERROR_PARTIAL_CHAR;
-      case XML_TOK_NONE:   /* start == end */
+      case XML_TOK_NONE: /* start == end */
       default:
         break;
       }
@@ -4274,52 +4085,46 @@ entityValueProcessor(XML_Parser parser,
 #endif /* XML_DTD */
 
 static enum XML_Error PTRCALL
-prologProcessor(XML_Parser parser,
-                const char *s,
-                const char *end,
-                const char **nextPtr)
-{
+prologProcessor(XML_Parser parser, const char *s, const char *end,
+                const char **nextPtr) {
   const char *next = s;
   int tok = XmlPrologTok(parser->m_encoding, s, end, &next);
-  return doProlog(parser, parser->m_encoding, s, end, tok, next,
-                  nextPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer);
+  return doProlog(parser, parser->m_encoding, s, end, tok, next, nextPtr,
+                  (XML_Bool)! parser->m_parsingStatus.finalBuffer, XML_TRUE);
 }
 
 static enum XML_Error
-doProlog(XML_Parser parser,
-         const ENCODING *enc,
-         const char *s,
-         const char *end,
-         int tok,
-         const char *next,
-         const char **nextPtr,
-         XML_Bool haveMore)
-{
+doProlog(XML_Parser parser, const ENCODING *enc, const char *s, const char *end,
+         int tok, const char *next, const char **nextPtr, XML_Bool haveMore,
+         XML_Bool allowClosingDoctype) {
 #ifdef XML_DTD
-  static const XML_Char externalSubsetName[] = { ASCII_HASH , '\0' };
+  static const XML_Char externalSubsetName[] = {ASCII_HASH, '\0'};
 #endif /* XML_DTD */
-  static const XML_Char atypeCDATA[] =
-      { ASCII_C, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0' };
-  static const XML_Char atypeID[] = { ASCII_I, ASCII_D, '\0' };
-  static const XML_Char atypeIDREF[] =
-      { ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, '\0' };
-  static const XML_Char atypeIDREFS[] =
-      { ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, ASCII_S, '\0' };
-  static const XML_Char atypeENTITY[] =
-      { ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T, ASCII_Y, '\0' };
-  static const XML_Char atypeENTITIES[] = { ASCII_E, ASCII_N,
-      ASCII_T, ASCII_I, ASCII_T, ASCII_I, ASCII_E, ASCII_S, '\0' };
-  static const XML_Char atypeNMTOKEN[] = {
-      ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K, ASCII_E, ASCII_N, '\0' };
-  static const XML_Char atypeNMTOKENS[] = { ASCII_N, ASCII_M, ASCII_T,
-      ASCII_O, ASCII_K, ASCII_E, ASCII_N, ASCII_S, '\0' };
-  static const XML_Char notationPrefix[] = { ASCII_N, ASCII_O, ASCII_T,
-      ASCII_A, ASCII_T, ASCII_I, ASCII_O, ASCII_N, ASCII_LPAREN, '\0' };
-  static const XML_Char enumValueSep[] = { ASCII_PIPE, '\0' };
-  static const XML_Char enumValueStart[] = { ASCII_LPAREN, '\0' };
+  static const XML_Char atypeCDATA[]
+      = {ASCII_C, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0'};
+  static const XML_Char atypeID[] = {ASCII_I, ASCII_D, '\0'};
+  static const XML_Char atypeIDREF[]
+      = {ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, '\0'};
+  static const XML_Char atypeIDREFS[]
+      = {ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, ASCII_S, '\0'};
+  static const XML_Char atypeENTITY[]
+      = {ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T, ASCII_Y, '\0'};
+  static const XML_Char atypeENTITIES[]
+      = {ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T,
+         ASCII_I, ASCII_E, ASCII_S, '\0'};
+  static const XML_Char atypeNMTOKEN[]
+      = {ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K, ASCII_E, ASCII_N, '\0'};
+  static const XML_Char atypeNMTOKENS[]
+      = {ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K,
+         ASCII_E, ASCII_N, ASCII_S, '\0'};
+  static const XML_Char notationPrefix[]
+      = {ASCII_N, ASCII_O, ASCII_T, ASCII_A,      ASCII_T,
+         ASCII_I, ASCII_O, ASCII_N, ASCII_LPAREN, '\0'};
+  static const XML_Char enumValueSep[] = {ASCII_PIPE, '\0'};
+  static const XML_Char enumValueStart[] = {ASCII_LPAREN, '\0'};
 
   /* save one level of indirection */
-  DTD * const dtd = parser->m_dtd;
+  DTD *const dtd = parser->m_dtd;
 
   const char **eventPP;
   const char **eventEndPP;
@@ -4328,8 +4133,7 @@ doProlog(XML_Parser parser,
   if (enc == parser->m_encoding) {
     eventPP = &parser->m_eventPtr;
     eventEndPP = &parser->m_eventEndPtr;
-  }
-  else {
+  } else {
     eventPP = &(parser->m_openInternalEntities->internalEventPtr);
     eventEndPP = &(parser->m_openInternalEntities->internalEventEndPtr);
   }
@@ -4358,7 +4162,8 @@ doProlog(XML_Parser parser,
       case XML_TOK_NONE:
 #ifdef XML_DTD
         /* for internal PE NOT referenced between declarations */
-        if (enc != parser->m_encoding && !parser->m_openInternalEntities->betweenDecl) {
+        if (enc != parser->m_encoding
+            && ! parser->m_openInternalEntities->betweenDecl) {
           *nextPtr = s;
           return XML_ERROR_NONE;
         }
@@ -4383,19 +4188,18 @@ doProlog(XML_Parser parser,
     }
     role = XmlTokenRole(&parser->m_prologState, tok, s, next, enc);
     switch (role) {
-    case XML_ROLE_XML_DECL:
-      {
-        enum XML_Error result = processXmlDecl(parser, 0, s, next);
-        if (result != XML_ERROR_NONE)
-          return result;
-        enc = parser->m_encoding;
-        handleDefault = XML_FALSE;
-      }
-      break;
+    case XML_ROLE_XML_DECL: {
+      enum XML_Error result = processXmlDecl(parser, 0, s, next);
+      if (result != XML_ERROR_NONE)
+        return result;
+      enc = parser->m_encoding;
+      handleDefault = XML_FALSE;
+    } break;
     case XML_ROLE_DOCTYPE_NAME:
       if (parser->m_startDoctypeDeclHandler) {
-        parser->m_doctypeName = poolStoreString(&parser->m_tempPool, enc, s, next);
-        if (!parser->m_doctypeName)
+        parser->m_doctypeName
+            = poolStoreString(&parser->m_tempPool, enc, s, next);
+        if (! parser->m_doctypeName)
           return XML_ERROR_NO_MEMORY;
         poolFinish(&parser->m_tempPool);
         parser->m_doctypePubid = NULL;
@@ -4405,43 +4209,40 @@ doProlog(XML_Parser parser,
       break;
     case XML_ROLE_DOCTYPE_INTERNAL_SUBSET:
       if (parser->m_startDoctypeDeclHandler) {
-        parser->m_startDoctypeDeclHandler(parser->m_handlerArg, parser->m_doctypeName, parser->m_doctypeSysid,
-                                parser->m_doctypePubid, 1);
+        parser->m_startDoctypeDeclHandler(
+            parser->m_handlerArg, parser->m_doctypeName, parser->m_doctypeSysid,
+            parser->m_doctypePubid, 1);
         parser->m_doctypeName = NULL;
         poolClear(&parser->m_tempPool);
         handleDefault = XML_FALSE;
       }
       break;
 #ifdef XML_DTD
-    case XML_ROLE_TEXT_DECL:
-      {
-        enum XML_Error result = processXmlDecl(parser, 1, s, next);
-        if (result != XML_ERROR_NONE)
-          return result;
-        enc = parser->m_encoding;
-        handleDefault = XML_FALSE;
-      }
-      break;
+    case XML_ROLE_TEXT_DECL: {
+      enum XML_Error result = processXmlDecl(parser, 1, s, next);
+      if (result != XML_ERROR_NONE)
+        return result;
+      enc = parser->m_encoding;
+      handleDefault = XML_FALSE;
+    } break;
 #endif /* XML_DTD */
     case XML_ROLE_DOCTYPE_PUBLIC_ID:
 #ifdef XML_DTD
       parser->m_useForeignDTD = XML_FALSE;
-      parser->m_declEntity = (ENTITY *)lookup(parser,
-                                    &dtd->paramEntities,
-                                    externalSubsetName,
-                                    sizeof(ENTITY));
-      if (!parser->m_declEntity)
+      parser->m_declEntity = (ENTITY *)lookup(
+          parser, &dtd->paramEntities, externalSubsetName, sizeof(ENTITY));
+      if (! parser->m_declEntity)
         return XML_ERROR_NO_MEMORY;
 #endif /* XML_DTD */
       dtd->hasParamEntityRefs = XML_TRUE;
       if (parser->m_startDoctypeDeclHandler) {
         XML_Char *pubId;
-        if (!XmlIsPublicId(enc, s, next, eventPP))
+        if (! XmlIsPublicId(enc, s, next, eventPP))
           return XML_ERROR_PUBLICID;
         pubId = poolStoreString(&parser->m_tempPool, enc,
                                 s + enc->minBytesPerChar,
                                 next - enc->minBytesPerChar);
-        if (!pubId)
+        if (! pubId)
           return XML_ERROR_NO_MEMORY;
         normalizePublicId(pubId);
         poolFinish(&parser->m_tempPool);
@@ -4451,15 +4252,14 @@ doProlog(XML_Parser parser,
       }
       /* fall through */
     case XML_ROLE_ENTITY_PUBLIC_ID:
-      if (!XmlIsPublicId(enc, s, next, eventPP))
+      if (! XmlIsPublicId(enc, s, next, eventPP))
         return XML_ERROR_PUBLICID;
     alreadyChecked:
       if (dtd->keepProcessing && parser->m_declEntity) {
-        XML_Char *tem = poolStoreString(&dtd->pool,
-                                        enc,
-                                        s + enc->minBytesPerChar,
-                                        next - enc->minBytesPerChar);
-        if (!tem)
+        XML_Char *tem
+            = poolStoreString(&dtd->pool, enc, s + enc->minBytesPerChar,
+                              next - enc->minBytesPerChar);
+        if (! tem)
           return XML_ERROR_NO_MEMORY;
         normalizePublicId(tem);
         parser->m_declEntity->publicId = tem;
@@ -4472,9 +4272,15 @@ doProlog(XML_Parser parser,
       }
       break;
     case XML_ROLE_DOCTYPE_CLOSE:
+      if (allowClosingDoctype != XML_TRUE) {
+        /* Must not close doctype from within expanded parameter entities */
+        return XML_ERROR_INVALID_TOKEN;
+      }
+
       if (parser->m_doctypeName) {
-        parser->m_startDoctypeDeclHandler(parser->m_handlerArg, parser->m_doctypeName,
-                                parser->m_doctypeSysid, parser->m_doctypePubid, 0);
+        parser->m_startDoctypeDeclHandler(
+            parser->m_handlerArg, parser->m_doctypeName, parser->m_doctypeSysid,
+            parser->m_doctypePubid, 0);
         poolClear(&parser->m_tempPool);
         handleDefault = XML_FALSE;
       }
@@ -4486,12 +4292,11 @@ doProlog(XML_Parser parser,
       if (parser->m_doctypeSysid || parser->m_useForeignDTD) {
         XML_Bool hadParamEntityRefs = dtd->hasParamEntityRefs;
         dtd->hasParamEntityRefs = XML_TRUE;
-        if (parser->m_paramEntityParsing && parser->m_externalEntityRefHandler) {
-          ENTITY *entity = (ENTITY *)lookup(parser,
-                                            &dtd->paramEntities,
-                                            externalSubsetName,
-                                            sizeof(ENTITY));
-          if (!entity) {
+        if (parser->m_paramEntityParsing
+            && parser->m_externalEntityRefHandler) {
+          ENTITY *entity = (ENTITY *)lookup(parser, &dtd->paramEntities,
+                                            externalSubsetName, sizeof(ENTITY));
+          if (! entity) {
             /* The external subset name "#" will have already been
              * inserted into the hash table at the start of the
              * external entity parsing, so no allocation will happen
@@ -4502,22 +4307,19 @@ doProlog(XML_Parser parser,
           if (parser->m_useForeignDTD)
             entity->base = parser->m_curBase;
           dtd->paramEntityRead = XML_FALSE;
-          if (!parser->m_externalEntityRefHandler(parser->m_externalEntityRefHandlerArg,
-                                        0,
-                                        entity->base,
-                                        entity->systemId,
-                                        entity->publicId))
+          if (! parser->m_externalEntityRefHandler(
+                  parser->m_externalEntityRefHandlerArg, 0, entity->base,
+                  entity->systemId, entity->publicId))
             return XML_ERROR_EXTERNAL_ENTITY_HANDLING;
           if (dtd->paramEntityRead) {
-            if (!dtd->standalone &&
-                parser->m_notStandaloneHandler &&
-                !parser->m_notStandaloneHandler(parser->m_handlerArg))
+            if (! dtd->standalone && parser->m_notStandaloneHandler
+                && ! parser->m_notStandaloneHandler(parser->m_handlerArg))
               return XML_ERROR_NOT_STANDALONE;
           }
           /* if we didn't read the foreign DTD then this means that there
              is no external subset and we must reset dtd->hasParamEntityRefs
           */
-          else if (!parser->m_doctypeSysid)
+          else if (! parser->m_doctypeSysid)
             dtd->hasParamEntityRefs = hadParamEntityRefs;
           /* end of DTD - no need to update dtd->keepProcessing */
         }
@@ -4537,24 +4339,21 @@ doProlog(XML_Parser parser,
       if (parser->m_useForeignDTD) {
         XML_Bool hadParamEntityRefs = dtd->hasParamEntityRefs;
         dtd->hasParamEntityRefs = XML_TRUE;
-        if (parser->m_paramEntityParsing && parser->m_externalEntityRefHandler) {
+        if (parser->m_paramEntityParsing
+            && parser->m_externalEntityRefHandler) {
           ENTITY *entity = (ENTITY *)lookup(parser, &dtd->paramEntities,
-                                            externalSubsetName,
-                                            sizeof(ENTITY));
-          if (!entity)
+                                            externalSubsetName, sizeof(ENTITY));
+          if (! entity)
             return XML_ERROR_NO_MEMORY;
           entity->base = parser->m_curBase;
           dtd->paramEntityRead = XML_FALSE;
-          if (!parser->m_externalEntityRefHandler(parser->m_externalEntityRefHandlerArg,
-                                        0,
-                                        entity->base,
-                                        entity->systemId,
-                                        entity->publicId))
+          if (! parser->m_externalEntityRefHandler(
+                  parser->m_externalEntityRefHandlerArg, 0, entity->base,
+                  entity->systemId, entity->publicId))
             return XML_ERROR_EXTERNAL_ENTITY_HANDLING;
           if (dtd->paramEntityRead) {
-            if (!dtd->standalone &&
-                parser->m_notStandaloneHandler &&
-                !parser->m_notStandaloneHandler(parser->m_handlerArg))
+            if (! dtd->standalone && parser->m_notStandaloneHandler
+                && ! parser->m_notStandaloneHandler(parser->m_handlerArg))
               return XML_ERROR_NOT_STANDALONE;
           }
           /* if we didn't read the foreign DTD then this means that there
@@ -4570,12 +4369,12 @@ doProlog(XML_Parser parser,
       return contentProcessor(parser, s, end, nextPtr);
     case XML_ROLE_ATTLIST_ELEMENT_NAME:
       parser->m_declElementType = getElementType(parser, enc, s, next);
-      if (!parser->m_declElementType)
+      if (! parser->m_declElementType)
         return XML_ERROR_NO_MEMORY;
       goto checkAttListDeclHandler;
     case XML_ROLE_ATTRIBUTE_NAME:
       parser->m_declAttributeId = getAttributeId(parser, enc, s, next);
-      if (!parser->m_declAttributeId)
+      if (! parser->m_declAttributeId)
         return XML_ERROR_NO_MEMORY;
       parser->m_declAttributeIsCdata = XML_FALSE;
       parser->m_declAttributeType = NULL;
@@ -4616,15 +4415,13 @@ doProlog(XML_Parser parser,
         const XML_Char *prefix;
         if (parser->m_declAttributeType) {
           prefix = enumValueSep;
+        } else {
+          prefix = (role == XML_ROLE_ATTRIBUTE_NOTATION_VALUE ? notationPrefix
+                                                              : enumValueStart);
         }
-        else {
-          prefix = (role == XML_ROLE_ATTRIBUTE_NOTATION_VALUE
-                    ? notationPrefix
-                    : enumValueStart);
-        }
-        if (!poolAppendString(&parser->m_tempPool, prefix))
+        if (! poolAppendString(&parser->m_tempPool, prefix))
           return XML_ERROR_NO_MEMORY;
-        if (!poolAppend(&parser->m_tempPool, enc, s, next))
+        if (! poolAppend(&parser->m_tempPool, enc, s, next))
           return XML_ERROR_NO_MEMORY;
         parser->m_declAttributeType = parser->m_tempPool.start;
         handleDefault = XML_FALSE;
@@ -4633,25 +4430,27 @@ doProlog(XML_Parser parser,
     case XML_ROLE_IMPLIED_ATTRIBUTE_VALUE:
     case XML_ROLE_REQUIRED_ATTRIBUTE_VALUE:
       if (dtd->keepProcessing) {
-        if (!defineAttribute(parser->m_declElementType, parser->m_declAttributeId,
-                             parser->m_declAttributeIsCdata, parser->m_declAttributeIsId,
-                             0, parser))
+        if (! defineAttribute(parser->m_declElementType,
+                              parser->m_declAttributeId,
+                              parser->m_declAttributeIsCdata,
+                              parser->m_declAttributeIsId, 0, parser))
           return XML_ERROR_NO_MEMORY;
         if (parser->m_attlistDeclHandler && parser->m_declAttributeType) {
           if (*parser->m_declAttributeType == XML_T(ASCII_LPAREN)
               || (*parser->m_declAttributeType == XML_T(ASCII_N)
                   && parser->m_declAttributeType[1] == XML_T(ASCII_O))) {
             /* Enumerated or Notation type */
-            if (!poolAppendChar(&parser->m_tempPool, XML_T(ASCII_RPAREN))
-                || !poolAppendChar(&parser->m_tempPool, XML_T('\0')))
+            if (! poolAppendChar(&parser->m_tempPool, XML_T(ASCII_RPAREN))
+                || ! poolAppendChar(&parser->m_tempPool, XML_T('\0')))
               return XML_ERROR_NO_MEMORY;
             parser->m_declAttributeType = parser->m_tempPool.start;
             poolFinish(&parser->m_tempPool);
           }
           *eventEndPP = s;
-          parser->m_attlistDeclHandler(parser->m_handlerArg, parser->m_declElementType->name,
-                             parser->m_declAttributeId->name, parser->m_declAttributeType,
-                             0, role == XML_ROLE_REQUIRED_ATTRIBUTE_VALUE);
+          parser->m_attlistDeclHandler(
+              parser->m_handlerArg, parser->m_declElementType->name,
+              parser->m_declAttributeId->name, parser->m_declAttributeType, 0,
+              role == XML_ROLE_REQUIRED_ATTRIBUTE_VALUE);
           poolClear(&parser->m_tempPool);
           handleDefault = XML_FALSE;
         }
@@ -4661,35 +4460,34 @@ doProlog(XML_Parser parser,
     case XML_ROLE_FIXED_ATTRIBUTE_VALUE:
       if (dtd->keepProcessing) {
         const XML_Char *attVal;
-        enum XML_Error result =
-          storeAttributeValue(parser, enc, parser->m_declAttributeIsCdata,
-                              s + enc->minBytesPerChar,
-                              next - enc->minBytesPerChar,
-                              &dtd->pool);
+        enum XML_Error result = storeAttributeValue(
+            parser, enc, parser->m_declAttributeIsCdata,
+            s + enc->minBytesPerChar, next - enc->minBytesPerChar, &dtd->pool);
         if (result)
           return result;
         attVal = poolStart(&dtd->pool);
         poolFinish(&dtd->pool);
         /* ID attributes aren't allowed to have a default */
-        if (!defineAttribute(parser->m_declElementType, parser->m_declAttributeId,
-                             parser->m_declAttributeIsCdata, XML_FALSE, attVal, parser))
+        if (! defineAttribute(
+                parser->m_declElementType, parser->m_declAttributeId,
+                parser->m_declAttributeIsCdata, XML_FALSE, attVal, parser))
           return XML_ERROR_NO_MEMORY;
         if (parser->m_attlistDeclHandler && parser->m_declAttributeType) {
           if (*parser->m_declAttributeType == XML_T(ASCII_LPAREN)
               || (*parser->m_declAttributeType == XML_T(ASCII_N)
                   && parser->m_declAttributeType[1] == XML_T(ASCII_O))) {
             /* Enumerated or Notation type */
-            if (!poolAppendChar(&parser->m_tempPool, XML_T(ASCII_RPAREN))
-                || !poolAppendChar(&parser->m_tempPool, XML_T('\0')))
+            if (! poolAppendChar(&parser->m_tempPool, XML_T(ASCII_RPAREN))
+                || ! poolAppendChar(&parser->m_tempPool, XML_T('\0')))
               return XML_ERROR_NO_MEMORY;
             parser->m_declAttributeType = parser->m_tempPool.start;
             poolFinish(&parser->m_tempPool);
           }
           *eventEndPP = s;
-          parser->m_attlistDeclHandler(parser->m_handlerArg, parser->m_declElementType->name,
-                             parser->m_declAttributeId->name, parser->m_declAttributeType,
-                             attVal,
-                             role == XML_ROLE_FIXED_ATTRIBUTE_VALUE);
+          parser->m_attlistDeclHandler(
+              parser->m_handlerArg, parser->m_declElementType->name,
+              parser->m_declAttributeId->name, parser->m_declAttributeType,
+              attVal, role == XML_ROLE_FIXED_ATTRIBUTE_VALUE);
           poolClear(&parser->m_tempPool);
           handleDefault = XML_FALSE;
         }
@@ -4697,25 +4495,22 @@ doProlog(XML_Parser parser,
       break;
     case XML_ROLE_ENTITY_VALUE:
       if (dtd->keepProcessing) {
-        enum XML_Error result = storeEntityValue(parser, enc,
-                                            s + enc->minBytesPerChar,
-                                            next - enc->minBytesPerChar);
+        enum XML_Error result = storeEntityValue(
+            parser, enc, s + enc->minBytesPerChar, next - enc->minBytesPerChar);
         if (parser->m_declEntity) {
           parser->m_declEntity->textPtr = poolStart(&dtd->entityValuePool);
-          parser->m_declEntity->textLen = (int)(poolLength(&dtd->entityValuePool));
+          parser->m_declEntity->textLen
+              = (int)(poolLength(&dtd->entityValuePool));
           poolFinish(&dtd->entityValuePool);
           if (parser->m_entityDeclHandler) {
             *eventEndPP = s;
-            parser->m_entityDeclHandler(parser->m_handlerArg,
-                              parser->m_declEntity->name,
-                              parser->m_declEntity->is_param,
-                              parser->m_declEntity->textPtr,
-                              parser->m_declEntity->textLen,
-                              parser->m_curBase, 0, 0, 0);
+            parser->m_entityDeclHandler(
+                parser->m_handlerArg, parser->m_declEntity->name,
+                parser->m_declEntity->is_param, parser->m_declEntity->textPtr,
+                parser->m_declEntity->textLen, parser->m_curBase, 0, 0, 0);
             handleDefault = XML_FALSE;
           }
-        }
-        else
+        } else
           poolDiscard(&dtd->entityValuePool);
         if (result != XML_ERROR_NONE)
           return result;
@@ -4728,8 +4523,8 @@ doProlog(XML_Parser parser,
       dtd->hasParamEntityRefs = XML_TRUE;
       if (parser->m_startDoctypeDeclHandler) {
         parser->m_doctypeSysid = poolStoreString(&parser->m_tempPool, enc,
-                                       s + enc->minBytesPerChar,
-                                       next - enc->minBytesPerChar);
+                                                 s + enc->minBytesPerChar,
+                                                 next - enc->minBytesPerChar);
         if (parser->m_doctypeSysid == NULL)
           return XML_ERROR_NO_MEMORY;
         poolFinish(&parser->m_tempPool);
@@ -4741,22 +4536,20 @@ doProlog(XML_Parser parser,
            for the case where no parser->m_startDoctypeDeclHandler is set */
         parser->m_doctypeSysid = externalSubsetName;
 #endif /* XML_DTD */
-      if (!dtd->standalone
+      if (! dtd->standalone
 #ifdef XML_DTD
-          && !parser->m_paramEntityParsing
+          && ! parser->m_paramEntityParsing
 #endif /* XML_DTD */
           && parser->m_notStandaloneHandler
-          && !parser->m_notStandaloneHandler(parser->m_handlerArg))
+          && ! parser->m_notStandaloneHandler(parser->m_handlerArg))
         return XML_ERROR_NOT_STANDALONE;
 #ifndef XML_DTD
       break;
-#else /* XML_DTD */
-      if (!parser->m_declEntity) {
-        parser->m_declEntity = (ENTITY *)lookup(parser,
-                                      &dtd->paramEntities,
-                                      externalSubsetName,
-                                      sizeof(ENTITY));
-        if (!parser->m_declEntity)
+#else  /* XML_DTD */
+      if (! parser->m_declEntity) {
+        parser->m_declEntity = (ENTITY *)lookup(
+            parser, &dtd->paramEntities, externalSubsetName, sizeof(ENTITY));
+        if (! parser->m_declEntity)
           return XML_ERROR_NO_MEMORY;
         parser->m_declEntity->publicId = NULL;
       }
@@ -4764,10 +4557,10 @@ doProlog(XML_Parser parser,
       /* fall through */
     case XML_ROLE_ENTITY_SYSTEM_ID:
       if (dtd->keepProcessing && parser->m_declEntity) {
-        parser->m_declEntity->systemId = poolStoreString(&dtd->pool, enc,
-                                               s + enc->minBytesPerChar,
-                                               next - enc->minBytesPerChar);
-        if (!parser->m_declEntity->systemId)
+        parser->m_declEntity->systemId
+            = poolStoreString(&dtd->pool, enc, s + enc->minBytesPerChar,
+                              next - enc->minBytesPerChar);
+        if (! parser->m_declEntity->systemId)
           return XML_ERROR_NO_MEMORY;
         parser->m_declEntity->base = parser->m_curBase;
         poolFinish(&dtd->pool);
@@ -4779,115 +4572,103 @@ doProlog(XML_Parser parser,
       }
       break;
     case XML_ROLE_ENTITY_COMPLETE:
-      if (dtd->keepProcessing && parser->m_declEntity && parser->m_entityDeclHandler) {
+      if (dtd->keepProcessing && parser->m_declEntity
+          && parser->m_entityDeclHandler) {
         *eventEndPP = s;
-        parser->m_entityDeclHandler(parser->m_handlerArg,
-                          parser->m_declEntity->name,
-                          parser->m_declEntity->is_param,
-                          0,0,
-                          parser->m_declEntity->base,
-                          parser->m_declEntity->systemId,
-                          parser->m_declEntity->publicId,
-                          0);
+        parser->m_entityDeclHandler(
+            parser->m_handlerArg, parser->m_declEntity->name,
+            parser->m_declEntity->is_param, 0, 0, parser->m_declEntity->base,
+            parser->m_declEntity->systemId, parser->m_declEntity->publicId, 0);
         handleDefault = XML_FALSE;
       }
       break;
     case XML_ROLE_ENTITY_NOTATION_NAME:
       if (dtd->keepProcessing && parser->m_declEntity) {
-        parser->m_declEntity->notation = poolStoreString(&dtd->pool, enc, s, next);
-        if (!parser->m_declEntity->notation)
+        parser->m_declEntity->notation
+            = poolStoreString(&dtd->pool, enc, s, next);
+        if (! parser->m_declEntity->notation)
           return XML_ERROR_NO_MEMORY;
         poolFinish(&dtd->pool);
         if (parser->m_unparsedEntityDeclHandler) {
           *eventEndPP = s;
-          parser->m_unparsedEntityDeclHandler(parser->m_handlerArg,
-                                    parser->m_declEntity->name,
-                                    parser->m_declEntity->base,
-                                    parser->m_declEntity->systemId,
-                                    parser->m_declEntity->publicId,
-                                    parser->m_declEntity->notation);
+          parser->m_unparsedEntityDeclHandler(
+              parser->m_handlerArg, parser->m_declEntity->name,
+              parser->m_declEntity->base, parser->m_declEntity->systemId,
+              parser->m_declEntity->publicId, parser->m_declEntity->notation);
           handleDefault = XML_FALSE;
-        }
-        else if (parser->m_entityDeclHandler) {
+        } else if (parser->m_entityDeclHandler) {
           *eventEndPP = s;
-          parser->m_entityDeclHandler(parser->m_handlerArg,
-                            parser->m_declEntity->name,
-                            0,0,0,
-                            parser->m_declEntity->base,
-                            parser->m_declEntity->systemId,
-                            parser->m_declEntity->publicId,
-                            parser->m_declEntity->notation);
+          parser->m_entityDeclHandler(
+              parser->m_handlerArg, parser->m_declEntity->name, 0, 0, 0,
+              parser->m_declEntity->base, parser->m_declEntity->systemId,
+              parser->m_declEntity->publicId, parser->m_declEntity->notation);
           handleDefault = XML_FALSE;
         }
       }
       break;
-    case XML_ROLE_GENERAL_ENTITY_NAME:
-      {
-        if (XmlPredefinedEntityName(enc, s, next)) {
-          parser->m_declEntity = NULL;
-          break;
-        }
-        if (dtd->keepProcessing) {
-          const XML_Char *name = poolStoreString(&dtd->pool, enc, s, next);
-          if (!name)
-            return XML_ERROR_NO_MEMORY;
-          parser->m_declEntity = (ENTITY *)lookup(parser, &dtd->generalEntities, name,
-                                        sizeof(ENTITY));
-          if (!parser->m_declEntity)
-            return XML_ERROR_NO_MEMORY;
-          if (parser->m_declEntity->name != name) {
-            poolDiscard(&dtd->pool);
-            parser->m_declEntity = NULL;
-          }
-          else {
-            poolFinish(&dtd->pool);
-            parser->m_declEntity->publicId = NULL;
-            parser->m_declEntity->is_param = XML_FALSE;
-            /* if we have a parent parser or are reading an internal parameter
-               entity, then the entity declaration is not considered "internal"
-            */
-            parser->m_declEntity->is_internal = !(parser->m_parentParser || parser->m_openInternalEntities);
-            if (parser->m_entityDeclHandler)
-              handleDefault = XML_FALSE;
-          }
-        }
-        else {
+    case XML_ROLE_GENERAL_ENTITY_NAME: {
+      if (XmlPredefinedEntityName(enc, s, next)) {
+        parser->m_declEntity = NULL;
+        break;
+      }
+      if (dtd->keepProcessing) {
+        const XML_Char *name = poolStoreString(&dtd->pool, enc, s, next);
+        if (! name)
+          return XML_ERROR_NO_MEMORY;
+        parser->m_declEntity = (ENTITY *)lookup(parser, &dtd->generalEntities,
+                                                name, sizeof(ENTITY));
+        if (! parser->m_declEntity)
+          return XML_ERROR_NO_MEMORY;
+        if (parser->m_declEntity->name != name) {
           poolDiscard(&dtd->pool);
           parser->m_declEntity = NULL;
+        } else {
+          poolFinish(&dtd->pool);
+          parser->m_declEntity->publicId = NULL;
+          parser->m_declEntity->is_param = XML_FALSE;
+          /* if we have a parent parser or are reading an internal parameter
+             entity, then the entity declaration is not considered "internal"
+          */
+          parser->m_declEntity->is_internal
+              = ! (parser->m_parentParser || parser->m_openInternalEntities);
+          if (parser->m_entityDeclHandler)
+            handleDefault = XML_FALSE;
         }
+      } else {
+        poolDiscard(&dtd->pool);
+        parser->m_declEntity = NULL;
       }
-      break;
+    } break;
     case XML_ROLE_PARAM_ENTITY_NAME:
 #ifdef XML_DTD
       if (dtd->keepProcessing) {
         const XML_Char *name = poolStoreString(&dtd->pool, enc, s, next);
-        if (!name)
+        if (! name)
           return XML_ERROR_NO_MEMORY;
         parser->m_declEntity = (ENTITY *)lookup(parser, &dtd->paramEntities,
-                                           name, sizeof(ENTITY));
-        if (!parser->m_declEntity)
+                                                name, sizeof(ENTITY));
+        if (! parser->m_declEntity)
           return XML_ERROR_NO_MEMORY;
         if (parser->m_declEntity->name != name) {
           poolDiscard(&dtd->pool);
           parser->m_declEntity = NULL;
-        }
-        else {
+        } else {
           poolFinish(&dtd->pool);
           parser->m_declEntity->publicId = NULL;
           parser->m_declEntity->is_param = XML_TRUE;
           /* if we have a parent parser or are reading an internal parameter
              entity, then the entity declaration is not considered "internal"
           */
-          parser->m_declEntity->is_internal = !(parser->m_parentParser || parser->m_openInternalEntities);
+          parser->m_declEntity->is_internal
+              = ! (parser->m_parentParser || parser->m_openInternalEntities);
           if (parser->m_entityDeclHandler)
             handleDefault = XML_FALSE;
         }
-      }
-      else {
+      } else {
         poolDiscard(&dtd->pool);
         parser->m_declEntity = NULL;
       }
-#else /* not XML_DTD */
+#else  /* not XML_DTD */
       parser->m_declEntity = NULL;
 #endif /* XML_DTD */
       break;
@@ -4895,22 +4676,23 @@ doProlog(XML_Parser parser,
       parser->m_declNotationPublicId = NULL;
       parser->m_declNotationName = NULL;
       if (parser->m_notationDeclHandler) {
-        parser->m_declNotationName = poolStoreString(&parser->m_tempPool, enc, s, next);
-        if (!parser->m_declNotationName)
+        parser->m_declNotationName
+            = poolStoreString(&parser->m_tempPool, enc, s, next);
+        if (! parser->m_declNotationName)
           return XML_ERROR_NO_MEMORY;
         poolFinish(&parser->m_tempPool);
         handleDefault = XML_FALSE;
       }
       break;
     case XML_ROLE_NOTATION_PUBLIC_ID:
-      if (!XmlIsPublicId(enc, s, next, eventPP))
+      if (! XmlIsPublicId(enc, s, next, eventPP))
         return XML_ERROR_PUBLICID;
-      if (parser->m_declNotationName) {  /* means m_notationDeclHandler != NULL */
-        XML_Char *tem = poolStoreString(&parser->m_tempPool,
-                                        enc,
+      if (parser
+              ->m_declNotationName) { /* means m_notationDeclHandler != NULL */
+        XML_Char *tem = poolStoreString(&parser->m_tempPool, enc,
                                         s + enc->minBytesPerChar,
                                         next - enc->minBytesPerChar);
-        if (!tem)
+        if (! tem)
           return XML_ERROR_NO_MEMORY;
         normalizePublicId(tem);
         parser->m_declNotationPublicId = tem;
@@ -4920,18 +4702,15 @@ doProlog(XML_Parser parser,
       break;
     case XML_ROLE_NOTATION_SYSTEM_ID:
       if (parser->m_declNotationName && parser->m_notationDeclHandler) {
-        const XML_Char *systemId
-          = poolStoreString(&parser->m_tempPool, enc,
-                            s + enc->minBytesPerChar,
-                            next - enc->minBytesPerChar);
-        if (!systemId)
+        const XML_Char *systemId = poolStoreString(&parser->m_tempPool, enc,
+                                                   s + enc->minBytesPerChar,
+                                                   next - enc->minBytesPerChar);
+        if (! systemId)
           return XML_ERROR_NO_MEMORY;
         *eventEndPP = s;
-        parser->m_notationDeclHandler(parser->m_handlerArg,
-                            parser->m_declNotationName,
-                            parser->m_curBase,
-                            systemId,
-                            parser->m_declNotationPublicId);
+        parser->m_notationDeclHandler(
+            parser->m_handlerArg, parser->m_declNotationName, parser->m_curBase,
+            systemId, parser->m_declNotationPublicId);
         handleDefault = XML_FALSE;
       }
       poolClear(&parser->m_tempPool);
@@ -4939,11 +4718,9 @@ doProlog(XML_Parser parser,
     case XML_ROLE_NOTATION_NO_SYSTEM_ID:
       if (parser->m_declNotationPublicId && parser->m_notationDeclHandler) {
         *eventEndPP = s;
-        parser->m_notationDeclHandler(parser->m_handlerArg,
-                            parser->m_declNotationName,
-                            parser->m_curBase,
-                            0,
-                            parser->m_declNotationPublicId);
+        parser->m_notationDeclHandler(
+            parser->m_handlerArg, parser->m_declNotationName, parser->m_curBase,
+            0, parser->m_declNotationPublicId);
         handleDefault = XML_FALSE;
       }
       poolClear(&parser->m_tempPool);
@@ -4960,42 +4737,44 @@ doProlog(XML_Parser parser,
         return XML_ERROR_SYNTAX;
       }
 #ifdef XML_DTD
-    case XML_ROLE_IGNORE_SECT:
-      {
-        enum XML_Error result;
-        if (parser->m_defaultHandler)
-          reportDefault(parser, enc, s, next);
-        handleDefault = XML_FALSE;
-        result = doIgnoreSection(parser, enc, &next, end, nextPtr, haveMore);
-        if (result != XML_ERROR_NONE)
-          return result;
-        else if (!next) {
-          parser->m_processor = ignoreSectionProcessor;
-          return result;
-        }
+    case XML_ROLE_IGNORE_SECT: {
+      enum XML_Error result;
+      if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+      handleDefault = XML_FALSE;
+      result = doIgnoreSection(parser, enc, &next, end, nextPtr, haveMore);
+      if (result != XML_ERROR_NONE)
+        return result;
+      else if (! next) {
+        parser->m_processor = ignoreSectionProcessor;
+        return result;
       }
-      break;
+    } break;
 #endif /* XML_DTD */
     case XML_ROLE_GROUP_OPEN:
       if (parser->m_prologState.level >= parser->m_groupSize) {
         if (parser->m_groupSize) {
-          char *temp = (char *)REALLOC(parser, parser->m_groupConnector, parser->m_groupSize *= 2);
-          if (temp == NULL) {
-            parser->m_groupSize /= 2;
-            return XML_ERROR_NO_MEMORY;
+          {
+            char *const new_connector = (char *)REALLOC(
+                parser, parser->m_groupConnector, parser->m_groupSize *= 2);
+            if (new_connector == NULL) {
+              parser->m_groupSize /= 2;
+              return XML_ERROR_NO_MEMORY;
+            }
+            parser->m_groupConnector = new_connector;
           }
-          parser->m_groupConnector = temp;
+
           if (dtd->scaffIndex) {
-            int *temp = (int *)REALLOC(parser, dtd->scaffIndex,
-                          parser->m_groupSize * sizeof(int));
-            if (temp == NULL)
+            int *const new_scaff_index = (int *)REALLOC(
+                parser, dtd->scaffIndex, parser->m_groupSize * sizeof(int));
+            if (new_scaff_index == NULL)
               return XML_ERROR_NO_MEMORY;
-            dtd->scaffIndex = temp;
+            dtd->scaffIndex = new_scaff_index;
           }
-        }
-        else {
-          parser->m_groupConnector = (char *)MALLOC(parser, parser->m_groupSize = 32);
-          if (!parser->m_groupConnector) {
+        } else {
+          parser->m_groupConnector
+              = (char *)MALLOC(parser, parser->m_groupSize = 32);
+          if (! parser->m_groupConnector) {
             parser->m_groupSize = 0;
             return XML_ERROR_NO_MEMORY;
           }
@@ -5006,6 +4785,7 @@ doProlog(XML_Parser parser,
         int myindex = nextScaffoldPart(parser);
         if (myindex < 0)
           return XML_ERROR_NO_MEMORY;
+        assert(dtd->scaffIndex != NULL);
         dtd->scaffIndex[dtd->scaffLevel] = myindex;
         dtd->scaffLevel++;
         dtd->scaffold[myindex].type = XML_CTYPE_SEQ;
@@ -5024,10 +4804,9 @@ doProlog(XML_Parser parser,
       if (parser->m_groupConnector[parser->m_prologState.level] == ASCII_COMMA)
         return XML_ERROR_SYNTAX;
       if (dtd->in_eldecl
-          && !parser->m_groupConnector[parser->m_prologState.level]
+          && ! parser->m_groupConnector[parser->m_prologState.level]
           && (dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel - 1]].type
-              != XML_CTYPE_MIXED)
-          ) {
+              != XML_CTYPE_MIXED)) {
         dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel - 1]].type
             = XML_CTYPE_CHOICE;
         if (parser->m_elementDeclHandler)
@@ -5039,15 +4818,14 @@ doProlog(XML_Parser parser,
 #ifdef XML_DTD
     case XML_ROLE_INNER_PARAM_ENTITY_REF:
       dtd->hasParamEntityRefs = XML_TRUE;
-      if (!parser->m_paramEntityParsing)
+      if (! parser->m_paramEntityParsing)
         dtd->keepProcessing = dtd->standalone;
       else {
         const XML_Char *name;
         ENTITY *entity;
-        name = poolStoreString(&dtd->pool, enc,
-                                s + enc->minBytesPerChar,
-                                next - enc->minBytesPerChar);
-        if (!name)
+        name = poolStoreString(&dtd->pool, enc, s + enc->minBytesPerChar,
+                               next - enc->minBytesPerChar);
+        if (! name)
           return XML_ERROR_NO_MEMORY;
         entity = (ENTITY *)lookup(parser, &dtd->paramEntities, name, 0);
         poolDiscard(&dtd->pool);
@@ -5055,13 +4833,12 @@ doProlog(XML_Parser parser,
            if yes, check that the entity exists, and that it is internal,
            otherwise call the skipped entity handler
         */
-        if (parser->m_prologState.documentEntity &&
-            (dtd->standalone
-             ? !parser->m_openInternalEntities
-             : !dtd->hasParamEntityRefs)) {
-          if (!entity)
+        if (parser->m_prologState.documentEntity
+            && (dtd->standalone ? ! parser->m_openInternalEntities
+                                : ! dtd->hasParamEntityRefs)) {
+          if (! entity)
             return XML_ERROR_UNDEFINED_ENTITY;
-          else if (!entity->is_internal) {
+          else if (! entity->is_internal) {
             /* It's hard to exhaustively search the code to be sure,
              * but there doesn't seem to be a way of executing the
              * following line.  There are two cases:
@@ -5084,11 +4861,11 @@ doProlog(XML_Parser parser,
              */
             return XML_ERROR_ENTITY_DECLARED_IN_PE; /* LCOV_EXCL_LINE */
           }
-        }
-        else if (!entity) {
+        } else if (! entity) {
           dtd->keepProcessing = dtd->standalone;
           /* cannot report skipped entities in declarations */
-          if ((role == XML_ROLE_PARAM_ENTITY_REF) && parser->m_skippedEntityHandler) {
+          if ((role == XML_ROLE_PARAM_ENTITY_REF)
+              && parser->m_skippedEntityHandler) {
             parser->m_skippedEntityHandler(parser->m_handlerArg, name, 1);
             handleDefault = XML_FALSE;
           }
@@ -5098,8 +4875,8 @@ doProlog(XML_Parser parser,
           return XML_ERROR_RECURSIVE_ENTITY_REF;
         if (entity->textPtr) {
           enum XML_Error result;
-          XML_Bool betweenDecl =
-            (role == XML_ROLE_PARAM_ENTITY_REF ? XML_TRUE : XML_FALSE);
+          XML_Bool betweenDecl
+              = (role == XML_ROLE_PARAM_ENTITY_REF ? XML_TRUE : XML_FALSE);
           result = processInternalEntity(parser, entity, betweenDecl);
           if (result != XML_ERROR_NONE)
             return result;
@@ -5109,39 +4886,35 @@ doProlog(XML_Parser parser,
         if (parser->m_externalEntityRefHandler) {
           dtd->paramEntityRead = XML_FALSE;
           entity->open = XML_TRUE;
-          if (!parser->m_externalEntityRefHandler(parser->m_externalEntityRefHandlerArg,
-                                        0,
-                                        entity->base,
-                                        entity->systemId,
-                                        entity->publicId)) {
+          if (! parser->m_externalEntityRefHandler(
+                  parser->m_externalEntityRefHandlerArg, 0, entity->base,
+                  entity->systemId, entity->publicId)) {
             entity->open = XML_FALSE;
             return XML_ERROR_EXTERNAL_ENTITY_HANDLING;
           }
           entity->open = XML_FALSE;
           handleDefault = XML_FALSE;
-          if (!dtd->paramEntityRead) {
+          if (! dtd->paramEntityRead) {
             dtd->keepProcessing = dtd->standalone;
             break;
           }
-        }
-        else {
+        } else {
           dtd->keepProcessing = dtd->standalone;
           break;
         }
       }
 #endif /* XML_DTD */
-      if (!dtd->standalone &&
-          parser->m_notStandaloneHandler &&
-          !parser->m_notStandaloneHandler(parser->m_handlerArg))
+      if (! dtd->standalone && parser->m_notStandaloneHandler
+          && ! parser->m_notStandaloneHandler(parser->m_handlerArg))
         return XML_ERROR_NOT_STANDALONE;
       break;
 
-    /* Element declaration stuff */
+      /* Element declaration stuff */
 
     case XML_ROLE_ELEMENT_NAME:
       if (parser->m_elementDeclHandler) {
         parser->m_declElementType = getElementType(parser, enc, s, next);
-        if (!parser->m_declElementType)
+        if (! parser->m_declElementType)
           return XML_ERROR_NO_MEMORY;
         dtd->scaffLevel = 0;
         dtd->scaffCount = 0;
@@ -5154,18 +4927,19 @@ doProlog(XML_Parser parser,
     case XML_ROLE_CONTENT_EMPTY:
       if (dtd->in_eldecl) {
         if (parser->m_elementDeclHandler) {
-          XML_Content * content = (XML_Content *) MALLOC(parser, sizeof(XML_Content));
-          if (!content)
+          XML_Content *content
+              = (XML_Content *)MALLOC(parser, sizeof(XML_Content));
+          if (! content)
             return XML_ERROR_NO_MEMORY;
           content->quant = XML_CQUANT_NONE;
           content->name = NULL;
           content->numchildren = 0;
           content->children = NULL;
-          content->type = ((role == XML_ROLE_CONTENT_ANY) ?
-                           XML_CTYPE_ANY :
-                           XML_CTYPE_EMPTY);
+          content->type = ((role == XML_ROLE_CONTENT_ANY) ? XML_CTYPE_ANY
+                                                          : XML_CTYPE_EMPTY);
           *eventEndPP = s;
-          parser->m_elementDeclHandler(parser->m_handlerArg, parser->m_declElementType->name, content);
+          parser->m_elementDeclHandler(
+              parser->m_handlerArg, parser->m_declElementType->name, content);
           handleDefault = XML_FALSE;
         }
         dtd->in_eldecl = XML_FALSE;
@@ -5197,22 +4971,22 @@ doProlog(XML_Parser parser,
         ELEMENT_TYPE *el;
         const XML_Char *name;
         int nameLen;
-        const char *nxt = (quant == XML_CQUANT_NONE
-                           ? next
-                           : next - enc->minBytesPerChar);
+        const char *nxt
+            = (quant == XML_CQUANT_NONE ? next : next - enc->minBytesPerChar);
         int myindex = nextScaffoldPart(parser);
         if (myindex < 0)
           return XML_ERROR_NO_MEMORY;
         dtd->scaffold[myindex].type = XML_CTYPE_NAME;
         dtd->scaffold[myindex].quant = quant;
         el = getElementType(parser, enc, s, nxt);
-        if (!el)
+        if (! el)
           return XML_ERROR_NO_MEMORY;
         name = el->name;
         dtd->scaffold[myindex].name = name;
         nameLen = 0;
-        for (; name[nameLen++]; );
-        dtd->contentStringLen +=  nameLen;
+        for (; name[nameLen++];)
+          ;
+        dtd->contentStringLen += nameLen;
         if (parser->m_elementDeclHandler)
           handleDefault = XML_FALSE;
       }
@@ -5236,12 +5010,13 @@ doProlog(XML_Parser parser,
         dtd->scaffLevel--;
         dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel]].quant = quant;
         if (dtd->scaffLevel == 0) {
-          if (!handleDefault) {
+          if (! handleDefault) {
             XML_Content *model = build_model(parser);
-            if (!model)
+            if (! model)
               return XML_ERROR_NO_MEMORY;
             *eventEndPP = s;
-            parser->m_elementDeclHandler(parser->m_handlerArg, parser->m_declElementType->name, model);
+            parser->m_elementDeclHandler(
+                parser->m_handlerArg, parser->m_declElementType->name, model);
           }
           dtd->in_eldecl = XML_FALSE;
           dtd->contentStringLen = 0;
@@ -5251,12 +5026,12 @@ doProlog(XML_Parser parser,
       /* End element declaration stuff */
 
     case XML_ROLE_PI:
-      if (!reportProcessingInstruction(parser, enc, s, next))
+      if (! reportProcessingInstruction(parser, enc, s, next))
         return XML_ERROR_NO_MEMORY;
       handleDefault = XML_FALSE;
       break;
     case XML_ROLE_COMMENT:
-      if (!reportComment(parser, enc, s, next))
+      if (! reportComment(parser, enc, s, next))
         return XML_ERROR_NO_MEMORY;
       handleDefault = XML_FALSE;
       break;
@@ -5307,11 +5082,8 @@ doProlog(XML_Parser parser,
 }
 
 static enum XML_Error PTRCALL
-epilogProcessor(XML_Parser parser,
-                const char *s,
-                const char *end,
-                const char **nextPtr)
-{
+epilogProcessor(XML_Parser parser, const char *s, const char *end,
+                const char **nextPtr) {
   parser->m_processor = epilogProcessor;
   parser->m_eventPtr = s;
   for (;;) {
@@ -5336,24 +5108,24 @@ epilogProcessor(XML_Parser parser,
         reportDefault(parser, parser->m_encoding, s, next);
       break;
     case XML_TOK_PI:
-      if (!reportProcessingInstruction(parser, parser->m_encoding, s, next))
+      if (! reportProcessingInstruction(parser, parser->m_encoding, s, next))
         return XML_ERROR_NO_MEMORY;
       break;
     case XML_TOK_COMMENT:
-      if (!reportComment(parser, parser->m_encoding, s, next))
+      if (! reportComment(parser, parser->m_encoding, s, next))
         return XML_ERROR_NO_MEMORY;
       break;
     case XML_TOK_INVALID:
       parser->m_eventPtr = next;
       return XML_ERROR_INVALID_TOKEN;
     case XML_TOK_PARTIAL:
-      if (!parser->m_parsingStatus.finalBuffer) {
+      if (! parser->m_parsingStatus.finalBuffer) {
         *nextPtr = s;
         return XML_ERROR_NONE;
       }
       return XML_ERROR_UNCLOSED_TOKEN;
     case XML_TOK_PARTIAL_CHAR:
-      if (!parser->m_parsingStatus.finalBuffer) {
+      if (! parser->m_parsingStatus.finalBuffer) {
         *nextPtr = s;
         return XML_ERROR_NONE;
       }
@@ -5368,15 +5140,13 @@ epilogProcessor(XML_Parser parser,
       return XML_ERROR_NONE;
     case XML_FINISHED:
       return XML_ERROR_ABORTED;
-    default: ;
+    default:;
     }
   }
 }
 
 static enum XML_Error
-processInternalEntity(XML_Parser parser, ENTITY *entity,
-                      XML_Bool betweenDecl)
-{
+processInternalEntity(XML_Parser parser, ENTITY *entity, XML_Bool betweenDecl) {
   const char *textStart, *textEnd;
   const char *next;
   enum XML_Error result;
@@ -5385,10 +5155,10 @@ processInternalEntity(XML_Parser parser, ENTITY *entity,
   if (parser->m_freeInternalEntities) {
     openEntity = parser->m_freeInternalEntities;
     parser->m_freeInternalEntities = openEntity->next;
-  }
-  else {
-    openEntity = (OPEN_INTERNAL_ENTITY *)MALLOC(parser, sizeof(OPEN_INTERNAL_ENTITY));
-    if (!openEntity)
+  } else {
+    openEntity
+        = (OPEN_INTERNAL_ENTITY *)MALLOC(parser, sizeof(OPEN_INTERNAL_ENTITY));
+    if (! openEntity)
       return XML_ERROR_NO_MEMORY;
   }
   entity->open = XML_TRUE;
@@ -5407,21 +5177,20 @@ processInternalEntity(XML_Parser parser, ENTITY *entity,
 
 #ifdef XML_DTD
   if (entity->is_param) {
-    int tok = XmlPrologTok(parser->m_internalEncoding, textStart, textEnd, &next);
-    result = doProlog(parser, parser->m_internalEncoding, textStart, textEnd, tok,
-                      next, &next, XML_FALSE);
-  }
-  else
+    int tok
+        = XmlPrologTok(parser->m_internalEncoding, textStart, textEnd, &next);
+    result = doProlog(parser, parser->m_internalEncoding, textStart, textEnd,
+                      tok, next, &next, XML_FALSE, XML_FALSE);
+  } else
 #endif /* XML_DTD */
-    result = doContent(parser, parser->m_tagLevel, parser->m_internalEncoding, textStart,
-                       textEnd, &next, XML_FALSE);
+    result = doContent(parser, parser->m_tagLevel, parser->m_internalEncoding,
+                       textStart, textEnd, &next, XML_FALSE);
 
   if (result == XML_ERROR_NONE) {
     if (textEnd != next && parser->m_parsingStatus.parsing == XML_SUSPENDED) {
       entity->processed = (int)(next - textStart);
       parser->m_processor = internalEntityProcessor;
-    }
-    else {
+    } else {
       entity->open = XML_FALSE;
       parser->m_openInternalEntities = openEntity->next;
       /* put openEntity back in list of free instances */
@@ -5433,17 +5202,14 @@ processInternalEntity(XML_Parser parser, ENTITY *entity,
 }
 
 static enum XML_Error PTRCALL
-internalEntityProcessor(XML_Parser parser,
-                        const char *s,
-                        const char *end,
-                        const char **nextPtr)
-{
+internalEntityProcessor(XML_Parser parser, const char *s, const char *end,
+                        const char **nextPtr) {
   ENTITY *entity;
   const char *textStart, *textEnd;
   const char *next;
   enum XML_Error result;
   OPEN_INTERNAL_ENTITY *openEntity = parser->m_openInternalEntities;
-  if (!openEntity)
+  if (! openEntity)
     return XML_ERROR_UNEXPECTED_STATE;
 
   entity = openEntity->entity;
@@ -5454,22 +5220,23 @@ internalEntityProcessor(XML_Parser parser,
 
 #ifdef XML_DTD
   if (entity->is_param) {
-    int tok = XmlPrologTok(parser->m_internalEncoding, textStart, textEnd, &next);
-    result = doProlog(parser, parser->m_internalEncoding, textStart, textEnd, tok,
-                      next, &next, XML_FALSE);
-  }
-  else
+    int tok
+        = XmlPrologTok(parser->m_internalEncoding, textStart, textEnd, &next);
+    result = doProlog(parser, parser->m_internalEncoding, textStart, textEnd,
+                      tok, next, &next, XML_FALSE, XML_TRUE);
+  } else
 #endif /* XML_DTD */
-    result = doContent(parser, openEntity->startTagLevel, parser->m_internalEncoding,
-                       textStart, textEnd, &next, XML_FALSE);
+    result = doContent(parser, openEntity->startTagLevel,
+                       parser->m_internalEncoding, textStart, textEnd, &next,
+                       XML_FALSE);
 
   if (result != XML_ERROR_NONE)
     return result;
-  else if (textEnd != next && parser->m_parsingStatus.parsing == XML_SUSPENDED) {
+  else if (textEnd != next
+           && parser->m_parsingStatus.parsing == XML_SUSPENDED) {
     entity->processed = (int)(next - (char *)entity->textPtr);
     return result;
-  }
-  else {
+  } else {
     entity->open = XML_FALSE;
     parser->m_openInternalEntities = openEntity->next;
     /* put openEntity back in list of free instances */
@@ -5483,49 +5250,45 @@ internalEntityProcessor(XML_Parser parser,
     parser->m_processor = prologProcessor;
     tok = XmlPrologTok(parser->m_encoding, s, end, &next);
     return doProlog(parser, parser->m_encoding, s, end, tok, next, nextPtr,
-                    (XML_Bool)!parser->m_parsingStatus.finalBuffer);
-  }
-  else
+                    (XML_Bool)! parser->m_parsingStatus.finalBuffer, XML_TRUE);
+  } else
 #endif /* XML_DTD */
   {
     parser->m_processor = contentProcessor;
     /* see externalEntityContentProcessor vs contentProcessor */
-    return doContent(parser, parser->m_parentParser ? 1 : 0, parser->m_encoding, s, end,
-                     nextPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer);
+    return doContent(parser, parser->m_parentParser ? 1 : 0, parser->m_encoding,
+                     s, end, nextPtr,
+                     (XML_Bool)! parser->m_parsingStatus.finalBuffer);
   }
 }
 
 static enum XML_Error PTRCALL
-errorProcessor(XML_Parser parser,
-               const char *UNUSED_P(s),
-               const char *UNUSED_P(end),
-               const char **UNUSED_P(nextPtr))
-{
+errorProcessor(XML_Parser parser, const char *s, const char *end,
+               const char **nextPtr) {
+  UNUSED_P(s);
+  UNUSED_P(end);
+  UNUSED_P(nextPtr);
   return parser->m_errorCode;
 }
 
 static enum XML_Error
 storeAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata,
-                    const char *ptr, const char *end,
-                    STRING_POOL *pool)
-{
-  enum XML_Error result = appendAttributeValue(parser, enc, isCdata, ptr,
-                                               end, pool);
+                    const char *ptr, const char *end, STRING_POOL *pool) {
+  enum XML_Error result
+      = appendAttributeValue(parser, enc, isCdata, ptr, end, pool);
   if (result)
     return result;
-  if (!isCdata && poolLength(pool) && poolLastChar(pool) == 0x20)
+  if (! isCdata && poolLength(pool) && poolLastChar(pool) == 0x20)
     poolChop(pool);
-  if (!poolAppendChar(pool, XML_T('\0')))
+  if (! poolAppendChar(pool, XML_T('\0')))
     return XML_ERROR_NO_MEMORY;
   return XML_ERROR_NONE;
 }
 
 static enum XML_Error
 appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata,
-                     const char *ptr, const char *end,
-                     STRING_POOL *pool)
-{
-  DTD * const dtd = parser->m_dtd;  /* save one level of indirection */
+                     const char *ptr, const char *end, STRING_POOL *pool) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   for (;;) {
     const char *next;
     int tok = XmlAttributeValueTok(enc, ptr, end, &next);
@@ -5540,38 +5303,35 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata,
       if (enc == parser->m_encoding)
         parser->m_eventPtr = ptr;
       return XML_ERROR_INVALID_TOKEN;
-    case XML_TOK_CHAR_REF:
-      {
-        XML_Char buf[XML_ENCODE_MAX];
-        int i;
-        int n = XmlCharRefNumber(enc, ptr);
-        if (n < 0) {
-          if (enc == parser->m_encoding)
-            parser->m_eventPtr = ptr;
-          return XML_ERROR_BAD_CHAR_REF;
-        }
-        if (!isCdata
-            && n == 0x20 /* space */
-            && (poolLength(pool) == 0 || poolLastChar(pool) == 0x20))
-          break;
-        n = XmlEncode(n, (ICHAR *)buf);
-        /* The XmlEncode() functions can never return 0 here.  That
-         * error return happens if the code point passed in is either
-         * negative or greater than or equal to 0x110000.  The
-         * XmlCharRefNumber() functions will all return a number
-         * strictly less than 0x110000 or a negative value if an error
-         * occurred.  The negative value is intercepted above, so
-         * XmlEncode() is never passed a value it might return an
-         * error for.
-         */
-        for (i = 0; i < n; i++) {
-          if (!poolAppendChar(pool, buf[i]))
-            return XML_ERROR_NO_MEMORY;
-        }
+    case XML_TOK_CHAR_REF: {
+      XML_Char buf[XML_ENCODE_MAX];
+      int i;
+      int n = XmlCharRefNumber(enc, ptr);
+      if (n < 0) {
+        if (enc == parser->m_encoding)
+          parser->m_eventPtr = ptr;
+        return XML_ERROR_BAD_CHAR_REF;
       }
-      break;
+      if (! isCdata && n == 0x20 /* space */
+          && (poolLength(pool) == 0 || poolLastChar(pool) == 0x20))
+        break;
+      n = XmlEncode(n, (ICHAR *)buf);
+      /* The XmlEncode() functions can never return 0 here.  That
+       * error return happens if the code point passed in is either
+       * negative or greater than or equal to 0x110000.  The
+       * XmlCharRefNumber() functions will all return a number
+       * strictly less than 0x110000 or a negative value if an error
+       * occurred.  The negative value is intercepted above, so
+       * XmlEncode() is never passed a value it might return an
+       * error for.
+       */
+      for (i = 0; i < n; i++) {
+        if (! poolAppendChar(pool, buf[i]))
+          return XML_ERROR_NO_MEMORY;
+      }
+    } break;
     case XML_TOK_DATA_CHARS:
-      if (!poolAppend(pool, enc, ptr, next))
+      if (! poolAppend(pool, enc, ptr, next))
         return XML_ERROR_NO_MEMORY;
       break;
     case XML_TOK_TRAILING_CR:
@@ -5579,109 +5339,103 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata,
       /* fall through */
     case XML_TOK_ATTRIBUTE_VALUE_S:
     case XML_TOK_DATA_NEWLINE:
-      if (!isCdata && (poolLength(pool) == 0 || poolLastChar(pool) == 0x20))
+      if (! isCdata && (poolLength(pool) == 0 || poolLastChar(pool) == 0x20))
         break;
-      if (!poolAppendChar(pool, 0x20))
+      if (! poolAppendChar(pool, 0x20))
         return XML_ERROR_NO_MEMORY;
       break;
-    case XML_TOK_ENTITY_REF:
-      {
-        const XML_Char *name;
-        ENTITY *entity;
-        char checkEntityDecl;
-        XML_Char ch = (XML_Char) XmlPredefinedEntityName(enc,
-                                              ptr + enc->minBytesPerChar,
-                                              next - enc->minBytesPerChar);
-        if (ch) {
-          if (!poolAppendChar(pool, ch))
-                return XML_ERROR_NO_MEMORY;
-          break;
-        }
-        name = poolStoreString(&parser->m_temp2Pool, enc,
-                               ptr + enc->minBytesPerChar,
-                               next - enc->minBytesPerChar);
-        if (!name)
+    case XML_TOK_ENTITY_REF: {
+      const XML_Char *name;
+      ENTITY *entity;
+      char checkEntityDecl;
+      XML_Char ch = (XML_Char)XmlPredefinedEntityName(
+          enc, ptr + enc->minBytesPerChar, next - enc->minBytesPerChar);
+      if (ch) {
+        if (! poolAppendChar(pool, ch))
           return XML_ERROR_NO_MEMORY;
-        entity = (ENTITY *)lookup(parser, &dtd->generalEntities, name, 0);
-        poolDiscard(&parser->m_temp2Pool);
-        /* First, determine if a check for an existing declaration is needed;
-           if yes, check that the entity exists, and that it is internal.
-        */
-        if (pool == &dtd->pool)  /* are we called from prolog? */
-          checkEntityDecl =
+        break;
+      }
+      name = poolStoreString(&parser->m_temp2Pool, enc,
+                             ptr + enc->minBytesPerChar,
+                             next - enc->minBytesPerChar);
+      if (! name)
+        return XML_ERROR_NO_MEMORY;
+      entity = (ENTITY *)lookup(parser, &dtd->generalEntities, name, 0);
+      poolDiscard(&parser->m_temp2Pool);
+      /* First, determine if a check for an existing declaration is needed;
+         if yes, check that the entity exists, and that it is internal.
+      */
+      if (pool == &dtd->pool) /* are we called from prolog? */
+        checkEntityDecl =
 #ifdef XML_DTD
-              parser->m_prologState.documentEntity &&
+            parser->m_prologState.documentEntity &&
 #endif /* XML_DTD */
-              (dtd->standalone
-               ? !parser->m_openInternalEntities
-               : !dtd->hasParamEntityRefs);
-        else /* if (pool == &parser->m_tempPool): we are called from content */
-          checkEntityDecl = !dtd->hasParamEntityRefs || dtd->standalone;
-        if (checkEntityDecl) {
-          if (!entity)
-            return XML_ERROR_UNDEFINED_ENTITY;
-          else if (!entity->is_internal)
-            return XML_ERROR_ENTITY_DECLARED_IN_PE;
-        }
-        else if (!entity) {
-          /* Cannot report skipped entity here - see comments on
-             parser->m_skippedEntityHandler.
-          if (parser->m_skippedEntityHandler)
-            parser->m_skippedEntityHandler(parser->m_handlerArg, name, 0);
-          */
-          /* Cannot call the default handler because this would be
-             out of sync with the call to the startElementHandler.
-          if ((pool == &parser->m_tempPool) && parser->m_defaultHandler)
-            reportDefault(parser, enc, ptr, next);
-          */
-          break;
-        }
-        if (entity->open) {
-          if (enc == parser->m_encoding) {
-            /* It does not appear that this line can be executed.
-             *
-             * The "if (entity->open)" check catches recursive entity
-             * definitions.  In order to be called with an open
-             * entity, it must have gone through this code before and
-             * been through the recursive call to
-             * appendAttributeValue() some lines below.  That call
-             * sets the local encoding ("enc") to the parser's
-             * internal encoding (internal_utf8 or internal_utf16),
-             * which can never be the same as the principle encoding.
-             * It doesn't appear there is another code path that gets
-             * here with entity->open being TRUE.
-             *
-             * Since it is not certain that this logic is watertight,
-             * we keep the line and merely exclude it from coverage
-             * tests.
-             */
-            parser->m_eventPtr = ptr; /* LCOV_EXCL_LINE */
-          }
-          return XML_ERROR_RECURSIVE_ENTITY_REF;
-        }
-        if (entity->notation) {
-          if (enc == parser->m_encoding)
-            parser->m_eventPtr = ptr;
-          return XML_ERROR_BINARY_ENTITY_REF;
-        }
-        if (!entity->textPtr) {
-          if (enc == parser->m_encoding)
-            parser->m_eventPtr = ptr;
-          return XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF;
-        }
-        else {
-          enum XML_Error result;
-          const XML_Char *textEnd = entity->textPtr + entity->textLen;
-          entity->open = XML_TRUE;
-          result = appendAttributeValue(parser, parser->m_internalEncoding, isCdata,
-                                        (char *)entity->textPtr,
-                                        (char *)textEnd, pool);
-          entity->open = XML_FALSE;
-          if (result)
-            return result;
+            (dtd->standalone ? ! parser->m_openInternalEntities
+                             : ! dtd->hasParamEntityRefs);
+      else /* if (pool == &parser->m_tempPool): we are called from content */
+        checkEntityDecl = ! dtd->hasParamEntityRefs || dtd->standalone;
+      if (checkEntityDecl) {
+        if (! entity)
+          return XML_ERROR_UNDEFINED_ENTITY;
+        else if (! entity->is_internal)
+          return XML_ERROR_ENTITY_DECLARED_IN_PE;
+      } else if (! entity) {
+        /* Cannot report skipped entity here - see comments on
+           parser->m_skippedEntityHandler.
+        if (parser->m_skippedEntityHandler)
+          parser->m_skippedEntityHandler(parser->m_handlerArg, name, 0);
+        */
+        /* Cannot call the default handler because this would be
+           out of sync with the call to the startElementHandler.
+        if ((pool == &parser->m_tempPool) && parser->m_defaultHandler)
+          reportDefault(parser, enc, ptr, next);
+        */
+        break;
+      }
+      if (entity->open) {
+        if (enc == parser->m_encoding) {
+          /* It does not appear that this line can be executed.
+           *
+           * The "if (entity->open)" check catches recursive entity
+           * definitions.  In order to be called with an open
+           * entity, it must have gone through this code before and
+           * been through the recursive call to
+           * appendAttributeValue() some lines below.  That call
+           * sets the local encoding ("enc") to the parser's
+           * internal encoding (internal_utf8 or internal_utf16),
+           * which can never be the same as the principle encoding.
+           * It doesn't appear there is another code path that gets
+           * here with entity->open being TRUE.
+           *
+           * Since it is not certain that this logic is watertight,
+           * we keep the line and merely exclude it from coverage
+           * tests.
+           */
+          parser->m_eventPtr = ptr; /* LCOV_EXCL_LINE */
         }
+        return XML_ERROR_RECURSIVE_ENTITY_REF;
       }
-      break;
+      if (entity->notation) {
+        if (enc == parser->m_encoding)
+          parser->m_eventPtr = ptr;
+        return XML_ERROR_BINARY_ENTITY_REF;
+      }
+      if (! entity->textPtr) {
+        if (enc == parser->m_encoding)
+          parser->m_eventPtr = ptr;
+        return XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF;
+      } else {
+        enum XML_Error result;
+        const XML_Char *textEnd = entity->textPtr + entity->textLen;
+        entity->open = XML_TRUE;
+        result = appendAttributeValue(parser, parser->m_internalEncoding,
+                                      isCdata, (char *)entity->textPtr,
+                                      (char *)textEnd, pool);
+        entity->open = XML_FALSE;
+        if (result)
+          return result;
+      }
+    } break;
     default:
       /* The only token returned by XmlAttributeValueTok() that does
        * not have an explicit case here is XML_TOK_PARTIAL_CHAR.
@@ -5705,12 +5459,9 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata,
 }
 
 static enum XML_Error
-storeEntityValue(XML_Parser parser,
-                 const ENCODING *enc,
-                 const char *entityTextPtr,
-                 const char *entityTextEnd)
-{
-  DTD * const dtd = parser->m_dtd;  /* save one level of indirection */
+storeEntityValue(XML_Parser parser, const ENCODING *enc,
+                 const char *entityTextPtr, const char *entityTextEnd) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   STRING_POOL *pool = &(dtd->entityValuePool);
   enum XML_Error result = XML_ERROR_NONE;
 #ifdef XML_DTD
@@ -5720,8 +5471,8 @@ storeEntityValue(XML_Parser parser,
   /* never return Null for the value argument in EntityDeclHandler,
      since this would indicate an external entity; therefore we
      have to make sure that entityValuePool.start is not null */
-  if (!pool->blocks) {
-    if (!poolGrow(pool))
+  if (! pool->blocks) {
+    if (! poolGrow(pool))
       return XML_ERROR_NO_MEMORY;
   }
 
@@ -5737,13 +5488,13 @@ storeEntityValue(XML_Parser parser,
         name = poolStoreString(&parser->m_tempPool, enc,
                                entityTextPtr + enc->minBytesPerChar,
                                next - enc->minBytesPerChar);
-        if (!name) {
+        if (! name) {
           result = XML_ERROR_NO_MEMORY;
           goto endEntityValue;
         }
         entity = (ENTITY *)lookup(parser, &dtd->paramEntities, name, 0);
         poolDiscard(&parser->m_tempPool);
-        if (!entity) {
+        if (! entity) {
           /* not a well-formedness error - see XML 1.0: WFC Entity Declared */
           /* cannot report skipped entity here - see comments on
              parser->m_skippedEntityHandler
@@ -5763,29 +5514,23 @@ storeEntityValue(XML_Parser parser,
           if (parser->m_externalEntityRefHandler) {
             dtd->paramEntityRead = XML_FALSE;
             entity->open = XML_TRUE;
-            if (!parser->m_externalEntityRefHandler(parser->m_externalEntityRefHandlerArg,
-                                          0,
-                                          entity->base,
-                                          entity->systemId,
-                                          entity->publicId)) {
+            if (! parser->m_externalEntityRefHandler(
+                    parser->m_externalEntityRefHandlerArg, 0, entity->base,
+                    entity->systemId, entity->publicId)) {
               entity->open = XML_FALSE;
               result = XML_ERROR_EXTERNAL_ENTITY_HANDLING;
               goto endEntityValue;
             }
             entity->open = XML_FALSE;
-            if (!dtd->paramEntityRead)
+            if (! dtd->paramEntityRead)
               dtd->keepProcessing = dtd->standalone;
-          }
-          else
+          } else
             dtd->keepProcessing = dtd->standalone;
-        }
-        else {
+        } else {
           entity->open = XML_TRUE;
-          result = storeEntityValue(parser,
-                                    parser->m_internalEncoding,
-                                    (char *)entity->textPtr,
-                                    (char *)(entity->textPtr
-                                             + entity->textLen));
+          result = storeEntityValue(
+              parser, parser->m_internalEncoding, (char *)entity->textPtr,
+              (char *)(entity->textPtr + entity->textLen));
           entity->open = XML_FALSE;
           if (result)
             goto endEntityValue;
@@ -5803,7 +5548,7 @@ storeEntityValue(XML_Parser parser,
       goto endEntityValue;
     case XML_TOK_ENTITY_REF:
     case XML_TOK_DATA_CHARS:
-      if (!poolAppend(pool, enc, entityTextPtr, next)) {
+      if (! poolAppend(pool, enc, entityTextPtr, next)) {
         result = XML_ERROR_NO_MEMORY;
         goto endEntityValue;
       }
@@ -5812,42 +5557,40 @@ storeEntityValue(XML_Parser parser,
       next = entityTextPtr + enc->minBytesPerChar;
       /* fall through */
     case XML_TOK_DATA_NEWLINE:
-      if (pool->end == pool->ptr && !poolGrow(pool)) {
-              result = XML_ERROR_NO_MEMORY;
+      if (pool->end == pool->ptr && ! poolGrow(pool)) {
+        result = XML_ERROR_NO_MEMORY;
         goto endEntityValue;
       }
       *(pool->ptr)++ = 0xA;
       break;
-    case XML_TOK_CHAR_REF:
-      {
-        XML_Char buf[XML_ENCODE_MAX];
-        int i;
-        int n = XmlCharRefNumber(enc, entityTextPtr);
-        if (n < 0) {
-          if (enc == parser->m_encoding)
-            parser->m_eventPtr = entityTextPtr;
-          result = XML_ERROR_BAD_CHAR_REF;
+    case XML_TOK_CHAR_REF: {
+      XML_Char buf[XML_ENCODE_MAX];
+      int i;
+      int n = XmlCharRefNumber(enc, entityTextPtr);
+      if (n < 0) {
+        if (enc == parser->m_encoding)
+          parser->m_eventPtr = entityTextPtr;
+        result = XML_ERROR_BAD_CHAR_REF;
+        goto endEntityValue;
+      }
+      n = XmlEncode(n, (ICHAR *)buf);
+      /* The XmlEncode() functions can never return 0 here.  That
+       * error return happens if the code point passed in is either
+       * negative or greater than or equal to 0x110000.  The
+       * XmlCharRefNumber() functions will all return a number
+       * strictly less than 0x110000 or a negative value if an error
+       * occurred.  The negative value is intercepted above, so
+       * XmlEncode() is never passed a value it might return an
+       * error for.
+       */
+      for (i = 0; i < n; i++) {
+        if (pool->end == pool->ptr && ! poolGrow(pool)) {
+          result = XML_ERROR_NO_MEMORY;
           goto endEntityValue;
         }
-        n = XmlEncode(n, (ICHAR *)buf);
-        /* The XmlEncode() functions can never return 0 here.  That
-         * error return happens if the code point passed in is either
-         * negative or greater than or equal to 0x110000.  The
-         * XmlCharRefNumber() functions will all return a number
-         * strictly less than 0x110000 or a negative value if an error
-         * occurred.  The negative value is intercepted above, so
-         * XmlEncode() is never passed a value it might return an
-         * error for.
-         */
-        for (i = 0; i < n; i++) {
-          if (pool->end == pool->ptr && !poolGrow(pool)) {
-            result = XML_ERROR_NO_MEMORY;
-            goto endEntityValue;
-          }
-          *(pool->ptr)++ = buf[i];
-        }
+        *(pool->ptr)++ = buf[i];
       }
-      break;
+    } break;
     case XML_TOK_PARTIAL:
       if (enc == parser->m_encoding)
         parser->m_eventPtr = entityTextPtr;
@@ -5882,8 +5625,7 @@ storeEntityValue(XML_Parser parser,
 }
 
 static void FASTCALL
-normalizeLines(XML_Char *s)
-{
+normalizeLines(XML_Char *s) {
   XML_Char *p;
   for (;; s++) {
     if (*s == XML_T('\0'))
@@ -5897,8 +5639,7 @@ normalizeLines(XML_Char *s)
       *p++ = 0xA;
       if (*++s == 0xA)
         s++;
-    }
-    else
+    } else
       *p++ = *s++;
   } while (*s);
   *p = XML_T('\0');
@@ -5906,12 +5647,11 @@ normalizeLines(XML_Char *s)
 
 static int
 reportProcessingInstruction(XML_Parser parser, const ENCODING *enc,
-                            const char *start, const char *end)
-{
+                            const char *start, const char *end) {
   const XML_Char *target;
   XML_Char *data;
   const char *tem;
-  if (!parser->m_processingInstructionHandler) {
+  if (! parser->m_processingInstructionHandler) {
     if (parser->m_defaultHandler)
       reportDefault(parser, enc, start, end);
     return 1;
@@ -5919,13 +5659,12 @@ reportProcessingInstruction(XML_Parser parser, const ENCODING *enc,
   start += enc->minBytesPerChar * 2;
   tem = start + XmlNameLength(enc, start);
   target = poolStoreString(&parser->m_tempPool, enc, start, tem);
-  if (!target)
+  if (! target)
     return 0;
   poolFinish(&parser->m_tempPool);
-  data = poolStoreString(&parser->m_tempPool, enc,
-                        XmlSkipS(enc, tem),
-                        end - enc->minBytesPerChar*2);
-  if (!data)
+  data = poolStoreString(&parser->m_tempPool, enc, XmlSkipS(enc, tem),
+                         end - enc->minBytesPerChar * 2);
+  if (! data)
     return 0;
   normalizeLines(data);
   parser->m_processingInstructionHandler(parser->m_handlerArg, target, data);
@@ -5934,20 +5673,18 @@ reportProcessingInstruction(XML_Parser parser, const ENCODING *enc,
 }
 
 static int
-reportComment(XML_Parser parser, const ENCODING *enc,
-              const char *start, const char *end)
-{
+reportComment(XML_Parser parser, const ENCODING *enc, const char *start,
+              const char *end) {
   XML_Char *data;
-  if (!parser->m_commentHandler) {
+  if (! parser->m_commentHandler) {
     if (parser->m_defaultHandler)
       reportDefault(parser, enc, start, end);
     return 1;
   }
-  data = poolStoreString(&parser->m_tempPool,
-                         enc,
+  data = poolStoreString(&parser->m_tempPool, enc,
                          start + enc->minBytesPerChar * 4,
                          end - enc->minBytesPerChar * 3);
-  if (!data)
+  if (! data)
     return 0;
   normalizeLines(data);
   parser->m_commentHandler(parser->m_handlerArg, data);
@@ -5956,9 +5693,8 @@ reportComment(XML_Parser parser, const ENCODING *enc,
 }
 
 static void
-reportDefault(XML_Parser parser, const ENCODING *enc,
-              const char *s, const char *end)
-{
+reportDefault(XML_Parser parser, const ENCODING *enc, const char *s,
+              const char *end) {
   if (MUST_CONVERT(enc, s)) {
     enum XML_Convert_Result convert_res;
     const char **eventPP;
@@ -5966,8 +5702,7 @@ reportDefault(XML_Parser parser, const ENCODING *enc,
     if (enc == parser->m_encoding) {
       eventPP = &parser->m_eventPtr;
       eventEndPP = &parser->m_eventEndPtr;
-    }
-    else {
+    } else {
       /* To get here, two things must be true; the parser must be
        * using a character encoding that is not the same as the
        * encoding passed in, and the encoding passed in must need
@@ -5990,21 +5725,22 @@ reportDefault(XML_Parser parser, const ENCODING *enc,
     }
     do {
       ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf;
-      convert_res = XmlConvert(enc, &s, end, &dataPtr, (ICHAR *)parser->m_dataBufEnd);
+      convert_res
+          = XmlConvert(enc, &s, end, &dataPtr, (ICHAR *)parser->m_dataBufEnd);
       *eventEndPP = s;
-      parser->m_defaultHandler(parser->m_handlerArg, parser->m_dataBuf, (int)(dataPtr - (ICHAR *)parser->m_dataBuf));
+      parser->m_defaultHandler(parser->m_handlerArg, parser->m_dataBuf,
+                               (int)(dataPtr - (ICHAR *)parser->m_dataBuf));
       *eventPP = s;
-    } while ((convert_res != XML_CONVERT_COMPLETED) && (convert_res != XML_CONVERT_INPUT_INCOMPLETE));
-  }
-  else
-    parser->m_defaultHandler(parser->m_handlerArg, (XML_Char *)s, (int)((XML_Char *)end - (XML_Char *)s));
+    } while ((convert_res != XML_CONVERT_COMPLETED)
+             && (convert_res != XML_CONVERT_INPUT_INCOMPLETE));
+  } else
+    parser->m_defaultHandler(parser->m_handlerArg, (XML_Char *)s,
+                             (int)((XML_Char *)end - (XML_Char *)s));
 }
 
-
 static int
 defineAttribute(ELEMENT_TYPE *type, ATTRIBUTE_ID *attId, XML_Bool isCdata,
-                XML_Bool isId, const XML_Char *value, XML_Parser parser)
-{
+                XML_Bool isId, const XML_Char *value, XML_Parser parser) {
   DEFAULT_ATTRIBUTE *att;
   if (value || isId) {
     /* The handling of default attributes gets messed up if we have
@@ -6013,24 +5749,23 @@ defineAttribute(ELEMENT_TYPE *type, ATTRIBUTE_ID *attId, XML_Bool isCdata,
     for (i = 0; i < type->nDefaultAtts; i++)
       if (attId == type->defaultAtts[i].id)
         return 1;
-    if (isId && !type->idAtt && !attId->xmlns)
+    if (isId && ! type->idAtt && ! attId->xmlns)
       type->idAtt = attId;
   }
   if (type->nDefaultAtts == type->allocDefaultAtts) {
     if (type->allocDefaultAtts == 0) {
       type->allocDefaultAtts = 8;
-      type->defaultAtts = (DEFAULT_ATTRIBUTE *)MALLOC(parser, type->allocDefaultAtts
-                            * sizeof(DEFAULT_ATTRIBUTE));
-      if (!type->defaultAtts) {
+      type->defaultAtts = (DEFAULT_ATTRIBUTE *)MALLOC(
+          parser, type->allocDefaultAtts * sizeof(DEFAULT_ATTRIBUTE));
+      if (! type->defaultAtts) {
         type->allocDefaultAtts = 0;
         return 0;
       }
-    }
-    else {
+    } else {
       DEFAULT_ATTRIBUTE *temp;
       int count = type->allocDefaultAtts * 2;
-      temp = (DEFAULT_ATTRIBUTE *)
-        REALLOC(parser, type->defaultAtts, (count * sizeof(DEFAULT_ATTRIBUTE)));
+      temp = (DEFAULT_ATTRIBUTE *)REALLOC(parser, type->defaultAtts,
+                                          (count * sizeof(DEFAULT_ATTRIBUTE)));
       if (temp == NULL)
         return 0;
       type->allocDefaultAtts = count;
@@ -6041,30 +5776,29 @@ defineAttribute(ELEMENT_TYPE *type, ATTRIBUTE_ID *attId, XML_Bool isCdata,
   att->id = attId;
   att->value = value;
   att->isCdata = isCdata;
-  if (!isCdata)
+  if (! isCdata)
     attId->maybeTokenized = XML_TRUE;
   type->nDefaultAtts += 1;
   return 1;
 }
 
 static int
-setElementTypePrefix(XML_Parser parser, ELEMENT_TYPE *elementType)
-{
-  DTD * const dtd = parser->m_dtd;  /* save one level of indirection */
+setElementTypePrefix(XML_Parser parser, ELEMENT_TYPE *elementType) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   const XML_Char *name;
   for (name = elementType->name; *name; name++) {
     if (*name == XML_T(ASCII_COLON)) {
       PREFIX *prefix;
       const XML_Char *s;
       for (s = elementType->name; s != name; s++) {
-        if (!poolAppendChar(&dtd->pool, *s))
+        if (! poolAppendChar(&dtd->pool, *s))
           return 0;
       }
-      if (!poolAppendChar(&dtd->pool, XML_T('\0')))
+      if (! poolAppendChar(&dtd->pool, XML_T('\0')))
         return 0;
       prefix = (PREFIX *)lookup(parser, &dtd->prefixes, poolStart(&dtd->pool),
                                 sizeof(PREFIX));
-      if (!prefix)
+      if (! prefix)
         return 0;
       if (prefix->name == poolStart(&dtd->pool))
         poolFinish(&dtd->pool);
@@ -6078,55 +5812,53 @@ setElementTypePrefix(XML_Parser parser, ELEMENT_TYPE *elementType)
 }
 
 static ATTRIBUTE_ID *
-getAttributeId(XML_Parser parser, const ENCODING *enc,
-               const char *start, const char *end)
-{
-  DTD * const dtd = parser->m_dtd;  /* save one level of indirection */
+getAttributeId(XML_Parser parser, const ENCODING *enc, const char *start,
+               const char *end) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   ATTRIBUTE_ID *id;
   const XML_Char *name;
-  if (!poolAppendChar(&dtd->pool, XML_T('\0')))
+  if (! poolAppendChar(&dtd->pool, XML_T('\0')))
     return NULL;
   name = poolStoreString(&dtd->pool, enc, start, end);
-  if (!name)
+  if (! name)
     return NULL;
   /* skip quotation mark - its storage will be re-used (like in name[-1]) */
   ++name;
-  id = (ATTRIBUTE_ID *)lookup(parser, &dtd->attributeIds, name, sizeof(ATTRIBUTE_ID));
-  if (!id)
+  id = (ATTRIBUTE_ID *)lookup(parser, &dtd->attributeIds, name,
+                              sizeof(ATTRIBUTE_ID));
+  if (! id)
     return NULL;
   if (id->name != name)
     poolDiscard(&dtd->pool);
   else {
     poolFinish(&dtd->pool);
-    if (!parser->m_ns)
+    if (! parser->m_ns)
       ;
-    else if (name[0] == XML_T(ASCII_x)
-        && name[1] == XML_T(ASCII_m)
-        && name[2] == XML_T(ASCII_l)
-        && name[3] == XML_T(ASCII_n)
-        && name[4] == XML_T(ASCII_s)
-        && (name[5] == XML_T('\0') || name[5] == XML_T(ASCII_COLON))) {
+    else if (name[0] == XML_T(ASCII_x) && name[1] == XML_T(ASCII_m)
+             && name[2] == XML_T(ASCII_l) && name[3] == XML_T(ASCII_n)
+             && name[4] == XML_T(ASCII_s)
+             && (name[5] == XML_T('\0') || name[5] == XML_T(ASCII_COLON))) {
       if (name[5] == XML_T('\0'))
         id->prefix = &dtd->defaultPrefix;
       else
-        id->prefix = (PREFIX *)lookup(parser, &dtd->prefixes, name + 6, sizeof(PREFIX));
+        id->prefix = (PREFIX *)lookup(parser, &dtd->prefixes, name + 6,
+                                      sizeof(PREFIX));
       id->xmlns = XML_TRUE;
-    }
-    else {
+    } else {
       int i;
       for (i = 0; name[i]; i++) {
         /* attributes without prefix are *not* in the default namespace */
         if (name[i] == XML_T(ASCII_COLON)) {
           int j;
           for (j = 0; j < i; j++) {
-            if (!poolAppendChar(&dtd->pool, name[j]))
+            if (! poolAppendChar(&dtd->pool, name[j]))
               return NULL;
           }
-          if (!poolAppendChar(&dtd->pool, XML_T('\0')))
+          if (! poolAppendChar(&dtd->pool, XML_T('\0')))
             return NULL;
-          id->prefix = (PREFIX *)lookup(parser, &dtd->prefixes, poolStart(&dtd->pool),
-                                        sizeof(PREFIX));
-          if (!id->prefix)
+          id->prefix = (PREFIX *)lookup(parser, &dtd->prefixes,
+                                        poolStart(&dtd->pool), sizeof(PREFIX));
+          if (! id->prefix)
             return NULL;
           if (id->prefix->name == poolStart(&dtd->pool))
             poolFinish(&dtd->pool);
@@ -6143,22 +5875,22 @@ getAttributeId(XML_Parser parser, const ENCODING *enc,
 #define CONTEXT_SEP XML_T(ASCII_FF)
 
 static const XML_Char *
-getContext(XML_Parser parser)
-{
-  DTD * const dtd = parser->m_dtd;  /* save one level of indirection */
+getContext(XML_Parser parser) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   HASH_TABLE_ITER iter;
   XML_Bool needSep = XML_FALSE;
 
   if (dtd->defaultPrefix.binding) {
     int i;
     int len;
-    if (!poolAppendChar(&parser->m_tempPool, XML_T(ASCII_EQUALS)))
+    if (! poolAppendChar(&parser->m_tempPool, XML_T(ASCII_EQUALS)))
       return NULL;
     len = dtd->defaultPrefix.binding->uriLen;
     if (parser->m_namespaceSeparator)
       len--;
     for (i = 0; i < len; i++) {
-      if (!poolAppendChar(&parser->m_tempPool, dtd->defaultPrefix.binding->uri[i])) {
+      if (! poolAppendChar(&parser->m_tempPool,
+                           dtd->defaultPrefix.binding->uri[i])) {
         /* Because of memory caching, I don't believe this line can be
          * executed.
          *
@@ -6190,9 +5922,9 @@ getContext(XML_Parser parser)
     int len;
     const XML_Char *s;
     PREFIX *prefix = (PREFIX *)hashTableIterNext(&iter);
-    if (!prefix)
+    if (! prefix)
       break;
-    if (!prefix->binding) {
+    if (! prefix->binding) {
       /* This test appears to be (justifiable) paranoia.  There does
        * not seem to be a way of injecting a prefix without a binding
        * that doesn't get errored long before this function is called.
@@ -6201,98 +5933,96 @@ getContext(XML_Parser parser)
        */
       continue; /* LCOV_EXCL_LINE */
     }
-    if (needSep && !poolAppendChar(&parser->m_tempPool, CONTEXT_SEP))
+    if (needSep && ! poolAppendChar(&parser->m_tempPool, CONTEXT_SEP))
       return NULL;
     for (s = prefix->name; *s; s++)
-      if (!poolAppendChar(&parser->m_tempPool, *s))
+      if (! poolAppendChar(&parser->m_tempPool, *s))
         return NULL;
-    if (!poolAppendChar(&parser->m_tempPool, XML_T(ASCII_EQUALS)))
+    if (! poolAppendChar(&parser->m_tempPool, XML_T(ASCII_EQUALS)))
       return NULL;
     len = prefix->binding->uriLen;
     if (parser->m_namespaceSeparator)
       len--;
     for (i = 0; i < len; i++)
-      if (!poolAppendChar(&parser->m_tempPool, prefix->binding->uri[i]))
+      if (! poolAppendChar(&parser->m_tempPool, prefix->binding->uri[i]))
         return NULL;
     needSep = XML_TRUE;
   }
 
-
   hashTableIterInit(&iter, &(dtd->generalEntities));
   for (;;) {
     const XML_Char *s;
     ENTITY *e = (ENTITY *)hashTableIterNext(&iter);
-    if (!e)
+    if (! e)
       break;
-    if (!e->open)
+    if (! e->open)
       continue;
-    if (needSep && !poolAppendChar(&parser->m_tempPool, CONTEXT_SEP))
+    if (needSep && ! poolAppendChar(&parser->m_tempPool, CONTEXT_SEP))
       return NULL;
     for (s = e->name; *s; s++)
-      if (!poolAppendChar(&parser->m_tempPool, *s))
+      if (! poolAppendChar(&parser->m_tempPool, *s))
         return 0;
     needSep = XML_TRUE;
   }
 
-  if (!poolAppendChar(&parser->m_tempPool, XML_T('\0')))
+  if (! poolAppendChar(&parser->m_tempPool, XML_T('\0')))
     return NULL;
   return parser->m_tempPool.start;
 }
 
 static XML_Bool
-setContext(XML_Parser parser, const XML_Char *context)
-{
-  DTD * const dtd = parser->m_dtd;  /* save one level of indirection */
+setContext(XML_Parser parser, const XML_Char *context) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   const XML_Char *s = context;
 
   while (*context != XML_T('\0')) {
     if (*s == CONTEXT_SEP || *s == XML_T('\0')) {
       ENTITY *e;
-      if (!poolAppendChar(&parser->m_tempPool, XML_T('\0')))
+      if (! poolAppendChar(&parser->m_tempPool, XML_T('\0')))
         return XML_FALSE;
-      e = (ENTITY *)lookup(parser, &dtd->generalEntities, poolStart(&parser->m_tempPool), 0);
+      e = (ENTITY *)lookup(parser, &dtd->generalEntities,
+                           poolStart(&parser->m_tempPool), 0);
       if (e)
         e->open = XML_TRUE;
       if (*s != XML_T('\0'))
         s++;
       context = s;
       poolDiscard(&parser->m_tempPool);
-    }
-    else if (*s == XML_T(ASCII_EQUALS)) {
+    } else if (*s == XML_T(ASCII_EQUALS)) {
       PREFIX *prefix;
       if (poolLength(&parser->m_tempPool) == 0)
         prefix = &dtd->defaultPrefix;
       else {
-        if (!poolAppendChar(&parser->m_tempPool, XML_T('\0')))
+        if (! poolAppendChar(&parser->m_tempPool, XML_T('\0')))
           return XML_FALSE;
-        prefix = (PREFIX *)lookup(parser, &dtd->prefixes, poolStart(&parser->m_tempPool),
-                                  sizeof(PREFIX));
-        if (!prefix)
+        prefix
+            = (PREFIX *)lookup(parser, &dtd->prefixes,
+                               poolStart(&parser->m_tempPool), sizeof(PREFIX));
+        if (! prefix)
           return XML_FALSE;
         if (prefix->name == poolStart(&parser->m_tempPool)) {
           prefix->name = poolCopyString(&dtd->pool, prefix->name);
-          if (!prefix->name)
+          if (! prefix->name)
             return XML_FALSE;
         }
         poolDiscard(&parser->m_tempPool);
       }
-      for (context = s + 1;
-           *context != CONTEXT_SEP && *context != XML_T('\0');
+      for (context = s + 1; *context != CONTEXT_SEP && *context != XML_T('\0');
            context++)
-        if (!poolAppendChar(&parser->m_tempPool, *context))
+        if (! poolAppendChar(&parser->m_tempPool, *context))
           return XML_FALSE;
-      if (!poolAppendChar(&parser->m_tempPool, XML_T('\0')))
+      if (! poolAppendChar(&parser->m_tempPool, XML_T('\0')))
         return XML_FALSE;
       if (addBinding(parser, prefix, NULL, poolStart(&parser->m_tempPool),
-                     &parser->m_inheritedBindings) != XML_ERROR_NONE)
+                     &parser->m_inheritedBindings)
+          != XML_ERROR_NONE)
         return XML_FALSE;
       poolDiscard(&parser->m_tempPool);
       if (*context != XML_T('\0'))
         ++context;
       s = context;
-    }
-    else {
-      if (!poolAppendChar(&parser->m_tempPool, *s))
+    } else {
+      if (! poolAppendChar(&parser->m_tempPool, *s))
         return XML_FALSE;
       s++;
     }
@@ -6301,8 +6031,7 @@ setContext(XML_Parser parser, const XML_Char *context)
 }
 
 static void FASTCALL
-normalizePublicId(XML_Char *publicId)
-{
+normalizePublicId(XML_Char *publicId) {
   XML_Char *p = publicId;
   XML_Char *s;
   for (s = publicId; *s; s++) {
@@ -6323,8 +6052,7 @@ normalizePublicId(XML_Char *publicId)
 }
 
 static DTD *
-dtdCreate(const XML_Memory_Handling_Suite *ms)
-{
+dtdCreate(const XML_Memory_Handling_Suite *ms) {
   DTD *p = (DTD *)ms->malloc_fcn(sizeof(DTD));
   if (p == NULL)
     return p;
@@ -6356,13 +6084,12 @@ dtdCreate(const XML_Memory_Handling_Suite *ms)
 }
 
 static void
-dtdReset(DTD *p, const XML_Memory_Handling_Suite *ms)
-{
+dtdReset(DTD *p, const XML_Memory_Handling_Suite *ms) {
   HASH_TABLE_ITER iter;
   hashTableIterInit(&iter, &(p->elementTypes));
   for (;;) {
     ELEMENT_TYPE *e = (ELEMENT_TYPE *)hashTableIterNext(&iter);
-    if (!e)
+    if (! e)
       break;
     if (e->allocDefaultAtts != 0)
       ms->free_fcn(e->defaultAtts);
@@ -6398,13 +6125,12 @@ dtdReset(DTD *p, const XML_Memory_Handling_Suite *ms)
 }
 
 static void
-dtdDestroy(DTD *p, XML_Bool isDocEntity, const XML_Memory_Handling_Suite *ms)
-{
+dtdDestroy(DTD *p, XML_Bool isDocEntity, const XML_Memory_Handling_Suite *ms) {
   HASH_TABLE_ITER iter;
   hashTableIterInit(&iter, &(p->elementTypes));
   for (;;) {
     ELEMENT_TYPE *e = (ELEMENT_TYPE *)hashTableIterNext(&iter);
-    if (!e)
+    if (! e)
       break;
     if (e->allocDefaultAtts != 0)
       ms->free_fcn(e->defaultAtts);
@@ -6429,8 +6155,8 @@ dtdDestroy(DTD *p, XML_Bool isDocEntity, const XML_Memory_Handling_Suite *ms)
    The new DTD has already been initialized.
 */
 static int
-dtdCopy(XML_Parser oldParser, DTD *newDtd, const DTD *oldDtd, const XML_Memory_Handling_Suite *ms)
-{
+dtdCopy(XML_Parser oldParser, DTD *newDtd, const DTD *oldDtd,
+        const XML_Memory_Handling_Suite *ms) {
   HASH_TABLE_ITER iter;
 
   /* Copy the prefix table. */
@@ -6439,12 +6165,12 @@ dtdCopy(XML_Parser oldParser, DTD *newDtd, const DTD *oldDtd, const XML_Memory_H
   for (;;) {
     const XML_Char *name;
     const PREFIX *oldP = (PREFIX *)hashTableIterNext(&iter);
-    if (!oldP)
+    if (! oldP)
       break;
     name = poolCopyString(&(newDtd->pool), oldP->name);
-    if (!name)
+    if (! name)
       return 0;
-    if (!lookup(oldParser, &(newDtd->prefixes), name, sizeof(PREFIX)))
+    if (! lookup(oldParser, &(newDtd->prefixes), name, sizeof(PREFIX)))
       return 0;
   }
 
@@ -6457,18 +6183,18 @@ dtdCopy(XML_Parser oldParser, DTD *newDtd, const DTD *oldDtd, const XML_Memory_H
     const XML_Char *name;
     const ATTRIBUTE_ID *oldA = (ATTRIBUTE_ID *)hashTableIterNext(&iter);
 
-    if (!oldA)
+    if (! oldA)
       break;
     /* Remember to allocate the scratch byte before the name. */
-    if (!poolAppendChar(&(newDtd->pool), XML_T('\0')))
+    if (! poolAppendChar(&(newDtd->pool), XML_T('\0')))
       return 0;
     name = poolCopyString(&(newDtd->pool), oldA->name);
-    if (!name)
+    if (! name)
       return 0;
     ++name;
     newA = (ATTRIBUTE_ID *)lookup(oldParser, &(newDtd->attributeIds), name,
                                   sizeof(ATTRIBUTE_ID));
-    if (!newA)
+    if (! newA)
       return 0;
     newA->maybeTokenized = oldA->maybeTokenized;
     if (oldA->prefix) {
@@ -6490,57 +6216,52 @@ dtdCopy(XML_Parser oldParser, DTD *newDtd, const DTD *oldDtd, const XML_Memory_H
     ELEMENT_TYPE *newE;
     const XML_Char *name;
     const ELEMENT_TYPE *oldE = (ELEMENT_TYPE *)hashTableIterNext(&iter);
-    if (!oldE)
+    if (! oldE)
       break;
     name = poolCopyString(&(newDtd->pool), oldE->name);
-    if (!name)
+    if (! name)
       return 0;
     newE = (ELEMENT_TYPE *)lookup(oldParser, &(newDtd->elementTypes), name,
                                   sizeof(ELEMENT_TYPE));
-    if (!newE)
+    if (! newE)
       return 0;
     if (oldE->nDefaultAtts) {
-      newE->defaultAtts = (DEFAULT_ATTRIBUTE *)
-          ms->malloc_fcn(oldE->nDefaultAtts * sizeof(DEFAULT_ATTRIBUTE));
-      if (!newE->defaultAtts) {
+      newE->defaultAtts = (DEFAULT_ATTRIBUTE *)ms->malloc_fcn(
+          oldE->nDefaultAtts * sizeof(DEFAULT_ATTRIBUTE));
+      if (! newE->defaultAtts) {
         return 0;
       }
     }
     if (oldE->idAtt)
-      newE->idAtt = (ATTRIBUTE_ID *)
-          lookup(oldParser, &(newDtd->attributeIds), oldE->idAtt->name, 0);
+      newE->idAtt = (ATTRIBUTE_ID *)lookup(oldParser, &(newDtd->attributeIds),
+                                           oldE->idAtt->name, 0);
     newE->allocDefaultAtts = newE->nDefaultAtts = oldE->nDefaultAtts;
     if (oldE->prefix)
       newE->prefix = (PREFIX *)lookup(oldParser, &(newDtd->prefixes),
                                       oldE->prefix->name, 0);
     for (i = 0; i < newE->nDefaultAtts; i++) {
-      newE->defaultAtts[i].id = (ATTRIBUTE_ID *)
-          lookup(oldParser, &(newDtd->attributeIds), oldE->defaultAtts[i].id->name, 0);
+      newE->defaultAtts[i].id = (ATTRIBUTE_ID *)lookup(
+          oldParser, &(newDtd->attributeIds), oldE->defaultAtts[i].id->name, 0);
       newE->defaultAtts[i].isCdata = oldE->defaultAtts[i].isCdata;
       if (oldE->defaultAtts[i].value) {
         newE->defaultAtts[i].value
             = poolCopyString(&(newDtd->pool), oldE->defaultAtts[i].value);
-        if (!newE->defaultAtts[i].value)
+        if (! newE->defaultAtts[i].value)
           return 0;
-      }
-      else
+      } else
         newE->defaultAtts[i].value = NULL;
     }
   }
 
   /* Copy the entity tables. */
-  if (!copyEntityTable(oldParser,
-                       &(newDtd->generalEntities),
-                       &(newDtd->pool),
-                       &(oldDtd->generalEntities)))
-      return 0;
+  if (! copyEntityTable(oldParser, &(newDtd->generalEntities), &(newDtd->pool),
+                        &(oldDtd->generalEntities)))
+    return 0;
 
 #ifdef XML_DTD
-  if (!copyEntityTable(oldParser,
-                       &(newDtd->paramEntities),
-                       &(newDtd->pool),
-                       &(oldDtd->paramEntities)))
-      return 0;
+  if (! copyEntityTable(oldParser, &(newDtd->paramEntities), &(newDtd->pool),
+                        &(oldDtd->paramEntities)))
+    return 0;
   newDtd->paramEntityRead = oldDtd->paramEntityRead;
 #endif /* XML_DTD */
 
@@ -6557,14 +6278,11 @@ dtdCopy(XML_Parser oldParser, DTD *newDtd, const DTD *oldDtd, const XML_Memory_H
   newDtd->scaffIndex = oldDtd->scaffIndex;
 
   return 1;
-}  /* End dtdCopy */
+} /* End dtdCopy */
 
 static int
-copyEntityTable(XML_Parser oldParser,
-                HASH_TABLE *newTable,
-                STRING_POOL *newPool,
-                const HASH_TABLE *oldTable)
-{
+copyEntityTable(XML_Parser oldParser, HASH_TABLE *newTable,
+                STRING_POOL *newPool, const HASH_TABLE *oldTable) {
   HASH_TABLE_ITER iter;
   const XML_Char *cachedOldBase = NULL;
   const XML_Char *cachedNewBase = NULL;
@@ -6575,17 +6293,17 @@ copyEntityTable(XML_Parser oldParser,
     ENTITY *newE;
     const XML_Char *name;
     const ENTITY *oldE = (ENTITY *)hashTableIterNext(&iter);
-    if (!oldE)
+    if (! oldE)
       break;
     name = poolCopyString(newPool, oldE->name);
-    if (!name)
+    if (! name)
       return 0;
     newE = (ENTITY *)lookup(oldParser, newTable, name, sizeof(ENTITY));
-    if (!newE)
+    if (! newE)
       return 0;
     if (oldE->systemId) {
       const XML_Char *tem = poolCopyString(newPool, oldE->systemId);
-      if (!tem)
+      if (! tem)
         return 0;
       newE->systemId = tem;
       if (oldE->base) {
@@ -6594,29 +6312,28 @@ copyEntityTable(XML_Parser oldParser,
         else {
           cachedOldBase = oldE->base;
           tem = poolCopyString(newPool, cachedOldBase);
-          if (!tem)
+          if (! tem)
             return 0;
           cachedNewBase = newE->base = tem;
         }
       }
       if (oldE->publicId) {
         tem = poolCopyString(newPool, oldE->publicId);
-        if (!tem)
+        if (! tem)
           return 0;
         newE->publicId = tem;
       }
-    }
-    else {
-      const XML_Char *tem = poolCopyStringN(newPool, oldE->textPtr,
-                                            oldE->textLen);
-      if (!tem)
+    } else {
+      const XML_Char *tem
+          = poolCopyStringN(newPool, oldE->textPtr, oldE->textLen);
+      if (! tem)
         return 0;
       newE->textPtr = tem;
       newE->textLen = oldE->textLen;
     }
     if (oldE->notation) {
       const XML_Char *tem = poolCopyString(newPool, oldE->notation);
-      if (!tem)
+      if (! tem)
         return 0;
       newE->notation = tem;
     }
@@ -6629,8 +6346,7 @@ copyEntityTable(XML_Parser oldParser,
 #define INIT_POWER 6
 
 static XML_Bool FASTCALL
-keyeq(KEY s1, KEY s2)
-{
+keyeq(KEY s1, KEY s2) {
   for (; *s1 == *s2; s1++, s2++)
     if (*s1 == 0)
       return XML_TRUE;
@@ -6638,23 +6354,21 @@ keyeq(KEY s1, KEY s2)
 }
 
 static size_t
-keylen(KEY s)
-{
+keylen(KEY s) {
   size_t len = 0;
-  for (; *s; s++, len++);
+  for (; *s; s++, len++)
+    ;
   return len;
 }
 
 static void
-copy_salt_to_sipkey(XML_Parser parser, struct sipkey * key)
-{
+copy_salt_to_sipkey(XML_Parser parser, struct sipkey *key) {
   key->k[0] = 0;
   key->k[1] = get_hash_secret_salt(parser);
 }
 
 static unsigned long FASTCALL
-hash(XML_Parser parser, KEY s)
-{
+hash(XML_Parser parser, KEY s) {
   struct siphash state;
   struct sipkey key;
   (void)sip24_valid;
@@ -6665,26 +6379,24 @@ hash(XML_Parser parser, KEY s)
 }
 
 static NAMED *
-lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize)
-{
+lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize) {
   size_t i;
   if (table->size == 0) {
     size_t tsize;
-    if (!createSize)
+    if (! createSize)
       return NULL;
     table->power = INIT_POWER;
     /* table->size is a power of 2 */
     table->size = (size_t)1 << INIT_POWER;
     tsize = table->size * sizeof(NAMED *);
     table->v = (NAMED **)table->mem->malloc_fcn(tsize);
-    if (!table->v) {
+    if (! table->v) {
       table->size = 0;
       return NULL;
     }
     memset(table->v, 0, tsize);
     i = hash(parser, name) & ((unsigned long)table->size - 1);
-  }
-  else {
+  } else {
     unsigned long h = hash(parser, name);
     unsigned long mask = (unsigned long)table->size - 1;
     unsigned char step = 0;
@@ -6692,11 +6404,11 @@ lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize)
     while (table->v[i]) {
       if (keyeq(name, table->v[i]->name))
         return table->v[i];
-      if (!step)
+      if (! step)
         step = PROBE_STEP(h, mask, table->power);
       i < step ? (i += table->size - step) : (i -= step);
     }
-    if (!createSize)
+    if (! createSize)
       return NULL;
 
     /* check for overflow (table is half full) */
@@ -6706,7 +6418,7 @@ lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize)
       unsigned long newMask = (unsigned long)newSize - 1;
       size_t tsize = newSize * sizeof(NAMED *);
       NAMED **newV = (NAMED **)table->mem->malloc_fcn(tsize);
-      if (!newV)
+      if (! newV)
         return NULL;
       memset(newV, 0, tsize);
       for (i = 0; i < table->size; i++)
@@ -6715,7 +6427,7 @@ lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize)
           size_t j = newHash & newMask;
           step = 0;
           while (newV[j]) {
-            if (!step)
+            if (! step)
               step = PROBE_STEP(newHash, newMask, newPower);
             j < step ? (j += newSize - step) : (j -= step);
           }
@@ -6728,14 +6440,14 @@ lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize)
       i = h & newMask;
       step = 0;
       while (table->v[i]) {
-        if (!step)
+        if (! step)
           step = PROBE_STEP(h, newMask, newPower);
         i < step ? (i += newSize - step) : (i -= step);
       }
     }
   }
   table->v[i] = (NAMED *)table->mem->malloc_fcn(createSize);
-  if (!table->v[i])
+  if (! table->v[i])
     return NULL;
   memset(table->v[i], 0, createSize);
   table->v[i]->name = name;
@@ -6744,8 +6456,7 @@ lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize)
 }
 
 static void FASTCALL
-hashTableClear(HASH_TABLE *table)
-{
+hashTableClear(HASH_TABLE *table) {
   size_t i;
   for (i = 0; i < table->size; i++) {
     table->mem->free_fcn(table->v[i]);
@@ -6755,8 +6466,7 @@ hashTableClear(HASH_TABLE *table)
 }
 
 static void FASTCALL
-hashTableDestroy(HASH_TABLE *table)
-{
+hashTableDestroy(HASH_TABLE *table) {
   size_t i;
   for (i = 0; i < table->size; i++)
     table->mem->free_fcn(table->v[i]);
@@ -6764,8 +6474,7 @@ hashTableDestroy(HASH_TABLE *table)
 }
 
 static void FASTCALL
-hashTableInit(HASH_TABLE *p, const XML_Memory_Handling_Suite *ms)
-{
+hashTableInit(HASH_TABLE *p, const XML_Memory_Handling_Suite *ms) {
   p->power = 0;
   p->size = 0;
   p->used = 0;
@@ -6774,15 +6483,13 @@ hashTableInit(HASH_TABLE *p, const XML_Memory_Handling_Suite *ms)
 }
 
 static void FASTCALL
-hashTableIterInit(HASH_TABLE_ITER *iter, const HASH_TABLE *table)
-{
+hashTableIterInit(HASH_TABLE_ITER *iter, const HASH_TABLE *table) {
   iter->p = table->v;
   iter->end = iter->p + table->size;
 }
 
-static NAMED * FASTCALL
-hashTableIterNext(HASH_TABLE_ITER *iter)
-{
+static NAMED *FASTCALL
+hashTableIterNext(HASH_TABLE_ITER *iter) {
   while (iter->p != iter->end) {
     NAMED *tem = *(iter->p)++;
     if (tem)
@@ -6792,8 +6499,7 @@ hashTableIterNext(HASH_TABLE_ITER *iter)
 }
 
 static void FASTCALL
-poolInit(STRING_POOL *pool, const XML_Memory_Handling_Suite *ms)
-{
+poolInit(STRING_POOL *pool, const XML_Memory_Handling_Suite *ms) {
   pool->blocks = NULL;
   pool->freeBlocks = NULL;
   pool->start = NULL;
@@ -6803,9 +6509,8 @@ poolInit(STRING_POOL *pool, const XML_Memory_Handling_Suite *ms)
 }
 
 static void FASTCALL
-poolClear(STRING_POOL *pool)
-{
-  if (!pool->freeBlocks)
+poolClear(STRING_POOL *pool) {
+  if (! pool->freeBlocks)
     pool->freeBlocks = pool->blocks;
   else {
     BLOCK *p = pool->blocks;
@@ -6823,8 +6528,7 @@ poolClear(STRING_POOL *pool)
 }
 
 static void FASTCALL
-poolDestroy(STRING_POOL *pool)
-{
+poolDestroy(STRING_POOL *pool) {
   BLOCK *p = pool->blocks;
   while (p) {
     BLOCK *tem = p->next;
@@ -6840,26 +6544,26 @@ poolDestroy(STRING_POOL *pool)
 }
 
 static XML_Char *
-poolAppend(STRING_POOL *pool, const ENCODING *enc,
-           const char *ptr, const char *end)
-{
-  if (!pool->ptr && !poolGrow(pool))
+poolAppend(STRING_POOL *pool, const ENCODING *enc, const char *ptr,
+           const char *end) {
+  if (! pool->ptr && ! poolGrow(pool))
     return NULL;
   for (;;) {
-    const enum XML_Convert_Result convert_res = XmlConvert(enc, &ptr, end, (ICHAR **)&(pool->ptr), (ICHAR *)pool->end);
-    if ((convert_res == XML_CONVERT_COMPLETED) || (convert_res == XML_CONVERT_INPUT_INCOMPLETE))
+    const enum XML_Convert_Result convert_res = XmlConvert(
+        enc, &ptr, end, (ICHAR **)&(pool->ptr), (ICHAR *)pool->end);
+    if ((convert_res == XML_CONVERT_COMPLETED)
+        || (convert_res == XML_CONVERT_INPUT_INCOMPLETE))
       break;
-    if (!poolGrow(pool))
+    if (! poolGrow(pool))
       return NULL;
   }
   return pool->start;
 }
 
-static const XML_Char * FASTCALL
-poolCopyString(STRING_POOL *pool, const XML_Char *s)
-{
+static const XML_Char *FASTCALL
+poolCopyString(STRING_POOL *pool, const XML_Char *s) {
   do {
-    if (!poolAppendChar(pool, *s))
+    if (! poolAppendChar(pool, *s))
       return NULL;
   } while (*s++);
   s = pool->start;
@@ -6868,9 +6572,8 @@ poolCopyString(STRING_POOL *pool, const XML_Char *s)
 }
 
 static const XML_Char *
-poolCopyStringN(STRING_POOL *pool, const XML_Char *s, int n)
-{
-  if (!pool->ptr && !poolGrow(pool)) {
+poolCopyStringN(STRING_POOL *pool, const XML_Char *s, int n) {
+  if (! pool->ptr && ! poolGrow(pool)) {
     /* The following line is unreachable given the current usage of
      * poolCopyStringN().  Currently it is called from exactly one
      * place to copy the text of a simple general entity.  By that
@@ -6885,7 +6588,7 @@ poolCopyStringN(STRING_POOL *pool, const XML_Char *s, int n)
     return NULL; /* LCOV_EXCL_LINE */
   }
   for (; n > 0; --n, s++) {
-    if (!poolAppendChar(pool, *s))
+    if (! poolAppendChar(pool, *s))
       return NULL;
   }
   s = pool->start;
@@ -6893,11 +6596,10 @@ poolCopyStringN(STRING_POOL *pool, const XML_Char *s, int n)
   return s;
 }
 
-static const XML_Char * FASTCALL
-poolAppendString(STRING_POOL *pool, const XML_Char *s)
-{
+static const XML_Char *FASTCALL
+poolAppendString(STRING_POOL *pool, const XML_Char *s) {
   while (*s) {
-    if (!poolAppendChar(pool, *s))
+    if (! poolAppendChar(pool, *s))
       return NULL;
     s++;
   }
@@ -6905,20 +6607,18 @@ poolAppendString(STRING_POOL *pool, const XML_Char *s)
 }
 
 static XML_Char *
-poolStoreString(STRING_POOL *pool, const ENCODING *enc,
-                const char *ptr, const char *end)
-{
-  if (!poolAppend(pool, enc, ptr, end))
+poolStoreString(STRING_POOL *pool, const ENCODING *enc, const char *ptr,
+                const char *end) {
+  if (! poolAppend(pool, enc, ptr, end))
     return NULL;
-  if (pool->ptr == pool->end && !poolGrow(pool))
+  if (pool->ptr == pool->end && ! poolGrow(pool))
     return NULL;
   *(pool->ptr)++ = 0;
   return pool->start;
 }
 
 static size_t
-poolBytesToAllocateFor(int blockSize)
-{
+poolBytesToAllocateFor(int blockSize) {
   /* Unprotected math would be:
   ** return offsetof(BLOCK, s) + blockSize * sizeof(XML_Char);
   **
@@ -6926,7 +6626,7 @@ poolBytesToAllocateFor(int blockSize)
   ** For a + b * c we check b * c in isolation first, so that addition of a
   ** on top has no chance of making us accept a small non-negative number
   */
-  const size_t stretch = sizeof(XML_Char);  /* can be 4 bytes */
+  const size_t stretch = sizeof(XML_Char); /* can be 4 bytes */
 
   if (blockSize <= 0)
     return 0;
@@ -6936,8 +6636,8 @@ poolBytesToAllocateFor(int blockSize)
 
   {
     const int stretchedBlockSize = blockSize * (int)stretch;
-    const int bytesToAllocate = (int)(
-        offsetof(BLOCK, s) + (unsigned)stretchedBlockSize);
+    const int bytesToAllocate
+        = (int)(offsetof(BLOCK, s) + (unsigned)stretchedBlockSize);
     if (bytesToAllocate < 0)
       return 0;
 
@@ -6946,8 +6646,7 @@ poolBytesToAllocateFor(int blockSize)
 }
 
 static XML_Bool FASTCALL
-poolGrow(STRING_POOL *pool)
-{
+poolGrow(STRING_POOL *pool) {
   if (pool->freeBlocks) {
     if (pool->start == 0) {
       pool->blocks = pool->freeBlocks;
@@ -6973,7 +6672,7 @@ poolGrow(STRING_POOL *pool)
   }
   if (pool->blocks && pool->start == pool->blocks->s) {
     BLOCK *temp;
-    int blockSize = (int)((unsigned)(pool->end - pool->start)*2U);
+    int blockSize = (int)((unsigned)(pool->end - pool->start) * 2U);
     size_t bytesToAllocate;
 
     /* NOTE: Needs to be calculated prior to calling `realloc`
@@ -6994,8 +6693,8 @@ poolGrow(STRING_POOL *pool)
     if (bytesToAllocate == 0)
       return XML_FALSE;
 
-    temp = (BLOCK *)
-      pool->mem->realloc_fcn(pool->blocks, (unsigned)bytesToAllocate);
+    temp = (BLOCK *)pool->mem->realloc_fcn(pool->blocks,
+                                           (unsigned)bytesToAllocate);
     if (temp == NULL)
       return XML_FALSE;
     pool->blocks = temp;
@@ -7003,8 +6702,7 @@ poolGrow(STRING_POOL *pool)
     pool->ptr = pool->blocks->s + offsetInsideBlock;
     pool->start = pool->blocks->s;
     pool->end = pool->start + blockSize;
-  }
-  else {
+  } else {
     BLOCK *tem;
     int blockSize = (int)(pool->end - pool->start);
     size_t bytesToAllocate;
@@ -7019,7 +6717,7 @@ poolGrow(STRING_POOL *pool)
        * function).  Either way it isn't readily testable, so we
        * exclude it from the coverage statistics.
        */
-      return XML_FALSE;  /* LCOV_EXCL_LINE */
+      return XML_FALSE; /* LCOV_EXCL_LINE */
     }
 
     if (blockSize < INIT_BLOCK_SIZE)
@@ -7037,14 +6735,13 @@ poolGrow(STRING_POOL *pool)
       return XML_FALSE;
 
     tem = (BLOCK *)pool->mem->malloc_fcn(bytesToAllocate);
-    if (!tem)
+    if (! tem)
       return XML_FALSE;
     tem->size = blockSize;
     tem->next = pool->blocks;
     pool->blocks = tem;
     if (pool->ptr != pool->start)
-      memcpy(tem->s, pool->start,
-             (pool->ptr - pool->start) * sizeof(XML_Char));
+      memcpy(tem->s, pool->start, (pool->ptr - pool->start) * sizeof(XML_Char));
     pool->ptr = tem->s + (pool->ptr - pool->start);
     pool->start = tem->s;
     pool->end = tem->s + blockSize;
@@ -7053,15 +6750,14 @@ poolGrow(STRING_POOL *pool)
 }
 
 static int FASTCALL
-nextScaffoldPart(XML_Parser parser)
-{
-  DTD * const dtd = parser->m_dtd;  /* save one level of indirection */
-  CONTENT_SCAFFOLD * me;
+nextScaffoldPart(XML_Parser parser) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
+  CONTENT_SCAFFOLD *me;
   int next;
 
-  if (!dtd->scaffIndex) {
+  if (! dtd->scaffIndex) {
     dtd->scaffIndex = (int *)MALLOC(parser, parser->m_groupSize * sizeof(int));
-    if (!dtd->scaffIndex)
+    if (! dtd->scaffIndex)
       return -1;
     dtd->scaffIndex[0] = 0;
   }
@@ -7069,15 +6765,14 @@ nextScaffoldPart(XML_Parser parser)
   if (dtd->scaffCount >= dtd->scaffSize) {
     CONTENT_SCAFFOLD *temp;
     if (dtd->scaffold) {
-      temp = (CONTENT_SCAFFOLD *)
-        REALLOC(parser, dtd->scaffold, dtd->scaffSize * 2 * sizeof(CONTENT_SCAFFOLD));
+      temp = (CONTENT_SCAFFOLD *)REALLOC(
+          parser, dtd->scaffold, dtd->scaffSize * 2 * sizeof(CONTENT_SCAFFOLD));
       if (temp == NULL)
         return -1;
       dtd->scaffSize *= 2;
-    }
-    else {
+    } else {
       temp = (CONTENT_SCAFFOLD *)MALLOC(parser, INIT_SCAFFOLD_ELEMENTS
-                                        * sizeof(CONTENT_SCAFFOLD));
+                                                    * sizeof(CONTENT_SCAFFOLD));
       if (temp == NULL)
         return -1;
       dtd->scaffSize = INIT_SCAFFOLD_ELEMENTS;
@@ -7087,11 +6782,12 @@ nextScaffoldPart(XML_Parser parser)
   next = dtd->scaffCount++;
   me = &dtd->scaffold[next];
   if (dtd->scaffLevel) {
-    CONTENT_SCAFFOLD *parent = &dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel-1]];
+    CONTENT_SCAFFOLD *parent
+        = &dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel - 1]];
     if (parent->lastchild) {
       dtd->scaffold[parent->lastchild].nextsib = next;
     }
-    if (!parent->childcnt)
+    if (! parent->childcnt)
       parent->firstchild = next;
     parent->lastchild = next;
     parent->childcnt++;
@@ -7101,13 +6797,9 @@ nextScaffoldPart(XML_Parser parser)
 }
 
 static void
-build_node(XML_Parser parser,
-           int src_node,
-           XML_Content *dest,
-           XML_Content **contpos,
-           XML_Char **strpos)
-{
-  DTD * const dtd = parser->m_dtd;  /* save one level of indirection */
+build_node(XML_Parser parser, int src_node, XML_Content *dest,
+           XML_Content **contpos, XML_Char **strpos) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   dest->type = dtd->scaffold[src_node].type;
   dest->quant = dtd->scaffold[src_node].quant;
   if (dest->type == XML_CTYPE_NAME) {
@@ -7116,21 +6808,19 @@ build_node(XML_Parser parser,
     src = dtd->scaffold[src_node].name;
     for (;;) {
       *(*strpos)++ = *src;
-      if (!*src)
+      if (! *src)
         break;
       src++;
     }
     dest->numchildren = 0;
     dest->children = NULL;
-  }
-  else {
+  } else {
     unsigned int i;
     int cn;
     dest->numchildren = dtd->scaffold[src_node].childcnt;
     dest->children = *contpos;
     *contpos += dest->numchildren;
-    for (i = 0, cn = dtd->scaffold[src_node].firstchild;
-         i < dest->numchildren;
+    for (i = 0, cn = dtd->scaffold[src_node].firstchild; i < dest->numchildren;
          i++, cn = dtd->scaffold[cn].nextsib) {
       build_node(parser, cn, &(dest->children[i]), contpos, strpos);
     }
@@ -7139,20 +6829,19 @@ build_node(XML_Parser parser,
 }
 
 static XML_Content *
-build_model (XML_Parser parser)
-{
-  DTD * const dtd = parser->m_dtd;  /* save one level of indirection */
+build_model(XML_Parser parser) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   XML_Content *ret;
   XML_Content *cpos;
-  XML_Char * str;
+  XML_Char *str;
   int allocsize = (dtd->scaffCount * sizeof(XML_Content)
                    + (dtd->contentStringLen * sizeof(XML_Char)));
 
   ret = (XML_Content *)MALLOC(parser, allocsize);
-  if (!ret)
+  if (! ret)
     return NULL;
 
-  str =  (XML_Char *) (&ret[dtd->scaffCount]);
+  str = (XML_Char *)(&ret[dtd->scaffCount]);
   cpos = &ret[1];
 
   build_node(parser, 0, ret, &cpos, &str);
@@ -7160,49 +6849,45 @@ build_model (XML_Parser parser)
 }
 
 static ELEMENT_TYPE *
-getElementType(XML_Parser parser,
-               const ENCODING *enc,
-               const char *ptr,
-               const char *end)
-{
-  DTD * const dtd = parser->m_dtd;  /* save one level of indirection */
+getElementType(XML_Parser parser, const ENCODING *enc, const char *ptr,
+               const char *end) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   const XML_Char *name = poolStoreString(&dtd->pool, enc, ptr, end);
   ELEMENT_TYPE *ret;
 
-  if (!name)
+  if (! name)
     return NULL;
-  ret = (ELEMENT_TYPE *) lookup(parser, &dtd->elementTypes, name, sizeof(ELEMENT_TYPE));
-  if (!ret)
+  ret = (ELEMENT_TYPE *)lookup(parser, &dtd->elementTypes, name,
+                               sizeof(ELEMENT_TYPE));
+  if (! ret)
     return NULL;
   if (ret->name != name)
     poolDiscard(&dtd->pool);
   else {
     poolFinish(&dtd->pool);
-    if (!setElementTypePrefix(parser, ret))
+    if (! setElementTypePrefix(parser, ret))
       return NULL;
   }
   return ret;
 }
 
 static XML_Char *
-copyString(const XML_Char *s,
-           const XML_Memory_Handling_Suite *memsuite)
-{
-    int charsRequired = 0;
-    XML_Char *result;
-
-    /* First determine how long the string is */
-    while (s[charsRequired] != 0) {
-      charsRequired++;
-    }
-    /* Include the terminator */
+copyString(const XML_Char *s, const XML_Memory_Handling_Suite *memsuite) {
+  int charsRequired = 0;
+  XML_Char *result;
+
+  /* First determine how long the string is */
+  while (s[charsRequired] != 0) {
     charsRequired++;
+  }
+  /* Include the terminator */
+  charsRequired++;
 
-    /* Now allocate space for the copy */
-    result = memsuite->malloc_fcn(charsRequired * sizeof(XML_Char));
-    if (result == NULL)
-        return NULL;
-    /* Copy the original into place */
-    memcpy(result, s, charsRequired * sizeof(XML_Char));
-    return result;
+  /* Now allocate space for the copy */
+  result = memsuite->malloc_fcn(charsRequired * sizeof(XML_Char));
+  if (result == NULL)
+    return NULL;
+  /* Copy the original into place */
+  memcpy(result, s, charsRequired * sizeof(XML_Char));
+  return result;
 }
diff --git a/Modules/expat/xmlrole.c b/Modules/expat/xmlrole.c
index 708507d575be84..4d3e3e86e9e864 100644
--- a/Modules/expat/xmlrole.c
+++ b/Modules/expat/xmlrole.c
@@ -33,11 +33,11 @@
 #include 
 
 #ifdef _WIN32
-#include "winconfig.h"
+#  include "winconfig.h"
 #else
-#ifdef HAVE_EXPAT_CONFIG_H
-#include 
-#endif
+#  ifdef HAVE_EXPAT_CONFIG_H
+#    include 
+#  endif
 #endif /* ndef _WIN32 */
 
 #include "expat_external.h"
@@ -52,107 +52,88 @@
 
 */
 
-static const char KW_ANY[] = {
-    ASCII_A, ASCII_N, ASCII_Y, '\0' };
-static const char KW_ATTLIST[] = {
-    ASCII_A, ASCII_T, ASCII_T, ASCII_L, ASCII_I, ASCII_S, ASCII_T, '\0' };
-static const char KW_CDATA[] = {
-    ASCII_C, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0' };
-static const char KW_DOCTYPE[] = {
-    ASCII_D, ASCII_O, ASCII_C, ASCII_T, ASCII_Y, ASCII_P, ASCII_E, '\0' };
-static const char KW_ELEMENT[] = {
-    ASCII_E, ASCII_L, ASCII_E, ASCII_M, ASCII_E, ASCII_N, ASCII_T, '\0' };
-static const char KW_EMPTY[] = {
-    ASCII_E, ASCII_M, ASCII_P, ASCII_T, ASCII_Y, '\0' };
-static const char KW_ENTITIES[] = {
-    ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T, ASCII_I, ASCII_E, ASCII_S,
-    '\0' };
-static const char KW_ENTITY[] = {
-    ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T, ASCII_Y, '\0' };
-static const char KW_FIXED[] = {
-    ASCII_F, ASCII_I, ASCII_X, ASCII_E, ASCII_D, '\0' };
-static const char KW_ID[] = {
-    ASCII_I, ASCII_D, '\0' };
-static const char KW_IDREF[] = {
-    ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, '\0' };
-static const char KW_IDREFS[] = {
-    ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, ASCII_S, '\0' };
+static const char KW_ANY[] = {ASCII_A, ASCII_N, ASCII_Y, '\0'};
+static const char KW_ATTLIST[]
+    = {ASCII_A, ASCII_T, ASCII_T, ASCII_L, ASCII_I, ASCII_S, ASCII_T, '\0'};
+static const char KW_CDATA[]
+    = {ASCII_C, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0'};
+static const char KW_DOCTYPE[]
+    = {ASCII_D, ASCII_O, ASCII_C, ASCII_T, ASCII_Y, ASCII_P, ASCII_E, '\0'};
+static const char KW_ELEMENT[]
+    = {ASCII_E, ASCII_L, ASCII_E, ASCII_M, ASCII_E, ASCII_N, ASCII_T, '\0'};
+static const char KW_EMPTY[]
+    = {ASCII_E, ASCII_M, ASCII_P, ASCII_T, ASCII_Y, '\0'};
+static const char KW_ENTITIES[] = {ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T,
+                                   ASCII_I, ASCII_E, ASCII_S, '\0'};
+static const char KW_ENTITY[]
+    = {ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T, ASCII_Y, '\0'};
+static const char KW_FIXED[]
+    = {ASCII_F, ASCII_I, ASCII_X, ASCII_E, ASCII_D, '\0'};
+static const char KW_ID[] = {ASCII_I, ASCII_D, '\0'};
+static const char KW_IDREF[]
+    = {ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, '\0'};
+static const char KW_IDREFS[]
+    = {ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, ASCII_S, '\0'};
 #ifdef XML_DTD
-static const char KW_IGNORE[] = {
-    ASCII_I, ASCII_G, ASCII_N, ASCII_O, ASCII_R, ASCII_E, '\0' };
+static const char KW_IGNORE[]
+    = {ASCII_I, ASCII_G, ASCII_N, ASCII_O, ASCII_R, ASCII_E, '\0'};
 #endif
-static const char KW_IMPLIED[] = {
-    ASCII_I, ASCII_M, ASCII_P, ASCII_L, ASCII_I, ASCII_E, ASCII_D, '\0' };
+static const char KW_IMPLIED[]
+    = {ASCII_I, ASCII_M, ASCII_P, ASCII_L, ASCII_I, ASCII_E, ASCII_D, '\0'};
 #ifdef XML_DTD
-static const char KW_INCLUDE[] = {
-    ASCII_I, ASCII_N, ASCII_C, ASCII_L, ASCII_U, ASCII_D, ASCII_E, '\0' };
+static const char KW_INCLUDE[]
+    = {ASCII_I, ASCII_N, ASCII_C, ASCII_L, ASCII_U, ASCII_D, ASCII_E, '\0'};
 #endif
-static const char KW_NDATA[] = {
-    ASCII_N, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0' };
-static const char KW_NMTOKEN[] = {
-    ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K, ASCII_E, ASCII_N, '\0' };
-static const char KW_NMTOKENS[] = {
-    ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K, ASCII_E, ASCII_N, ASCII_S,
-    '\0' };
-static const char KW_NOTATION[] =
-    { ASCII_N, ASCII_O, ASCII_T, ASCII_A, ASCII_T, ASCII_I, ASCII_O, ASCII_N,
-      '\0' };
-static const char KW_PCDATA[] = {
-    ASCII_P, ASCII_C, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0' };
-static const char KW_PUBLIC[] = {
-    ASCII_P, ASCII_U, ASCII_B, ASCII_L, ASCII_I, ASCII_C, '\0' };
-static const char KW_REQUIRED[] = {
-    ASCII_R, ASCII_E, ASCII_Q, ASCII_U, ASCII_I, ASCII_R, ASCII_E, ASCII_D,
-    '\0' };
-static const char KW_SYSTEM[] = {
-    ASCII_S, ASCII_Y, ASCII_S, ASCII_T, ASCII_E, ASCII_M, '\0' };
+static const char KW_NDATA[]
+    = {ASCII_N, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0'};
+static const char KW_NMTOKEN[]
+    = {ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K, ASCII_E, ASCII_N, '\0'};
+static const char KW_NMTOKENS[] = {ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K,
+                                   ASCII_E, ASCII_N, ASCII_S, '\0'};
+static const char KW_NOTATION[] = {ASCII_N, ASCII_O, ASCII_T, ASCII_A, ASCII_T,
+                                   ASCII_I, ASCII_O, ASCII_N, '\0'};
+static const char KW_PCDATA[]
+    = {ASCII_P, ASCII_C, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0'};
+static const char KW_PUBLIC[]
+    = {ASCII_P, ASCII_U, ASCII_B, ASCII_L, ASCII_I, ASCII_C, '\0'};
+static const char KW_REQUIRED[] = {ASCII_R, ASCII_E, ASCII_Q, ASCII_U, ASCII_I,
+                                   ASCII_R, ASCII_E, ASCII_D, '\0'};
+static const char KW_SYSTEM[]
+    = {ASCII_S, ASCII_Y, ASCII_S, ASCII_T, ASCII_E, ASCII_M, '\0'};
 
 #ifndef MIN_BYTES_PER_CHAR
-#define MIN_BYTES_PER_CHAR(enc) ((enc)->minBytesPerChar)
+#  define MIN_BYTES_PER_CHAR(enc) ((enc)->minBytesPerChar)
 #endif
 
 #ifdef XML_DTD
-#define setTopLevel(state) \
-  ((state)->handler = ((state)->documentEntity \
-                       ? internalSubset \
-                       : externalSubset1))
+#  define setTopLevel(state)                                                   \
+    ((state)->handler                                                          \
+     = ((state)->documentEntity ? internalSubset : externalSubset1))
 #else /* not XML_DTD */
-#define setTopLevel(state) ((state)->handler = internalSubset)
+#  define setTopLevel(state) ((state)->handler = internalSubset)
 #endif /* not XML_DTD */
 
-typedef int PTRCALL PROLOG_HANDLER(PROLOG_STATE *state,
-                                   int tok,
-                                   const char *ptr,
-                                   const char *end,
+typedef int PTRCALL PROLOG_HANDLER(PROLOG_STATE *state, int tok,
+                                   const char *ptr, const char *end,
                                    const ENCODING *enc);
 
-static PROLOG_HANDLER
-  prolog0, prolog1, prolog2,
-  doctype0, doctype1, doctype2, doctype3, doctype4, doctype5,
-  internalSubset,
-  entity0, entity1, entity2, entity3, entity4, entity5, entity6,
-  entity7, entity8, entity9, entity10,
-  notation0, notation1, notation2, notation3, notation4,
-  attlist0, attlist1, attlist2, attlist3, attlist4, attlist5, attlist6,
-  attlist7, attlist8, attlist9,
-  element0, element1, element2, element3, element4, element5, element6,
-  element7,
+static PROLOG_HANDLER prolog0, prolog1, prolog2, doctype0, doctype1, doctype2,
+    doctype3, doctype4, doctype5, internalSubset, entity0, entity1, entity2,
+    entity3, entity4, entity5, entity6, entity7, entity8, entity9, entity10,
+    notation0, notation1, notation2, notation3, notation4, attlist0, attlist1,
+    attlist2, attlist3, attlist4, attlist5, attlist6, attlist7, attlist8,
+    attlist9, element0, element1, element2, element3, element4, element5,
+    element6, element7,
 #ifdef XML_DTD
-  externalSubset0, externalSubset1,
-  condSect0, condSect1, condSect2,
+    externalSubset0, externalSubset1, condSect0, condSect1, condSect2,
 #endif /* XML_DTD */
-  declClose,
-  error;
+    declClose, error;
 
 static int FASTCALL common(PROLOG_STATE *state, int tok);
 
 static int PTRCALL
-prolog0(PROLOG_STATE *state,
-        int tok,
-        const char *ptr,
-        const char *end,
-        const ENCODING *enc)
-{
+prolog0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     state->handler = prolog1;
@@ -169,10 +150,8 @@ prolog0(PROLOG_STATE *state,
   case XML_TOK_BOM:
     return XML_ROLE_NONE;
   case XML_TOK_DECL_OPEN:
-    if (!XmlNameMatchesAscii(enc,
-                             ptr + 2 * MIN_BYTES_PER_CHAR(enc),
-                             end,
-                             KW_DOCTYPE))
+    if (! XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), end,
+                              KW_DOCTYPE))
       break;
     state->handler = doctype0;
     return XML_ROLE_DOCTYPE_NONE;
@@ -184,12 +163,8 @@ prolog0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-prolog1(PROLOG_STATE *state,
-        int tok,
-        const char *ptr,
-        const char *end,
-        const ENCODING *enc)
-{
+prolog1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NONE;
@@ -207,10 +182,8 @@ prolog1(PROLOG_STATE *state,
      */
     return XML_ROLE_NONE; /* LCOV_EXCL_LINE */
   case XML_TOK_DECL_OPEN:
-    if (!XmlNameMatchesAscii(enc,
-                             ptr + 2 * MIN_BYTES_PER_CHAR(enc),
-                             end,
-                             KW_DOCTYPE))
+    if (! XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), end,
+                              KW_DOCTYPE))
       break;
     state->handler = doctype0;
     return XML_ROLE_DOCTYPE_NONE;
@@ -222,12 +195,11 @@ prolog1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-prolog2(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+prolog2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NONE;
@@ -243,12 +215,11 @@ prolog2(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-doctype0(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+doctype0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_DOCTYPE_NONE;
@@ -261,12 +232,8 @@ doctype0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-doctype1(PROLOG_STATE *state,
-         int tok,
-         const char *ptr,
-         const char *end,
-         const ENCODING *enc)
-{
+doctype1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_DOCTYPE_NONE;
@@ -291,12 +258,11 @@ doctype1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-doctype2(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+doctype2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_DOCTYPE_NONE;
@@ -308,12 +274,11 @@ doctype2(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-doctype3(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+doctype3(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_DOCTYPE_NONE;
@@ -325,12 +290,11 @@ doctype3(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-doctype4(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+doctype4(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_DOCTYPE_NONE;
@@ -345,12 +309,11 @@ doctype4(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-doctype5(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+doctype5(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_DOCTYPE_NONE;
@@ -362,40 +325,28 @@ doctype5(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-internalSubset(PROLOG_STATE *state,
-               int tok,
-               const char *ptr,
-               const char *end,
-               const ENCODING *enc)
-{
+internalSubset(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+               const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NONE;
   case XML_TOK_DECL_OPEN:
-    if (XmlNameMatchesAscii(enc,
-                            ptr + 2 * MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), end,
                             KW_ENTITY)) {
       state->handler = entity0;
       return XML_ROLE_ENTITY_NONE;
     }
-    if (XmlNameMatchesAscii(enc,
-                            ptr + 2 * MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), end,
                             KW_ATTLIST)) {
       state->handler = attlist0;
       return XML_ROLE_ATTLIST_NONE;
     }
-    if (XmlNameMatchesAscii(enc,
-                            ptr + 2 * MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), end,
                             KW_ELEMENT)) {
       state->handler = element0;
       return XML_ROLE_ELEMENT_NONE;
     }
-    if (XmlNameMatchesAscii(enc,
-                            ptr + 2 * MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), end,
                             KW_NOTATION)) {
       state->handler = notation0;
       return XML_ROLE_NOTATION_NONE;
@@ -419,12 +370,8 @@ internalSubset(PROLOG_STATE *state,
 #ifdef XML_DTD
 
 static int PTRCALL
-externalSubset0(PROLOG_STATE *state,
-                int tok,
-                const char *ptr,
-                const char *end,
-                const ENCODING *enc)
-{
+externalSubset0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+                const ENCODING *enc) {
   state->handler = externalSubset1;
   if (tok == XML_TOK_XML_DECL)
     return XML_ROLE_TEXT_DECL;
@@ -432,12 +379,8 @@ externalSubset0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-externalSubset1(PROLOG_STATE *state,
-                int tok,
-                const char *ptr,
-                const char *end,
-                const ENCODING *enc)
-{
+externalSubset1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+                const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_COND_SECT_OPEN:
     state->handler = condSect0;
@@ -464,12 +407,11 @@ externalSubset1(PROLOG_STATE *state,
 #endif /* XML_DTD */
 
 static int PTRCALL
-entity0(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -484,12 +426,11 @@ entity0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity1(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -501,12 +442,8 @@ entity1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity2(PROLOG_STATE *state,
-        int tok,
-        const char *ptr,
-        const char *end,
-        const ENCODING *enc)
-{
+entity2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -529,12 +466,11 @@ entity2(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity3(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity3(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -546,12 +482,11 @@ entity3(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity4(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity4(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -563,12 +498,8 @@ entity4(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity5(PROLOG_STATE *state,
-        int tok,
-        const char *ptr,
-        const char *end,
-        const ENCODING *enc)
-{
+entity5(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -586,12 +517,11 @@ entity5(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity6(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity6(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -604,12 +534,8 @@ entity6(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity7(PROLOG_STATE *state,
-        int tok,
-        const char *ptr,
-        const char *end,
-        const ENCODING *enc)
-{
+entity7(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -632,12 +558,11 @@ entity7(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity8(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity8(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -649,12 +574,11 @@ entity8(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity9(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity9(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -666,12 +590,11 @@ entity9(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity10(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+entity10(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -683,12 +606,11 @@ entity10(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-notation0(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+notation0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NOTATION_NONE;
@@ -700,12 +622,8 @@ notation0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-notation1(PROLOG_STATE *state,
-          int tok,
-          const char *ptr,
-          const char *end,
-          const ENCODING *enc)
-{
+notation1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NOTATION_NONE;
@@ -724,12 +642,11 @@ notation1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-notation2(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+notation2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NOTATION_NONE;
@@ -741,12 +658,11 @@ notation2(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-notation3(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+notation3(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NOTATION_NONE;
@@ -759,12 +675,11 @@ notation3(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-notation4(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+notation4(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NOTATION_NONE;
@@ -780,12 +695,11 @@ notation4(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist0(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -798,12 +712,11 @@ attlist0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist1(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -819,34 +732,23 @@ attlist1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist2(PROLOG_STATE *state,
-         int tok,
-         const char *ptr,
-         const char *end,
-         const ENCODING *enc)
-{
+attlist2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
-  case XML_TOK_NAME:
-    {
-      static const char * const types[] = {
-        KW_CDATA,
-        KW_ID,
-        KW_IDREF,
-        KW_IDREFS,
-        KW_ENTITY,
-        KW_ENTITIES,
-        KW_NMTOKEN,
-        KW_NMTOKENS,
-      };
-      int i;
-      for (i = 0; i < (int)(sizeof(types)/sizeof(types[0])); i++)
-        if (XmlNameMatchesAscii(enc, ptr, end, types[i])) {
-          state->handler = attlist8;
-          return XML_ROLE_ATTRIBUTE_TYPE_CDATA + i;
-        }
-    }
+  case XML_TOK_NAME: {
+    static const char *const types[] = {
+        KW_CDATA,  KW_ID,       KW_IDREF,   KW_IDREFS,
+        KW_ENTITY, KW_ENTITIES, KW_NMTOKEN, KW_NMTOKENS,
+    };
+    int i;
+    for (i = 0; i < (int)(sizeof(types) / sizeof(types[0])); i++)
+      if (XmlNameMatchesAscii(enc, ptr, end, types[i])) {
+        state->handler = attlist8;
+        return XML_ROLE_ATTRIBUTE_TYPE_CDATA + i;
+      }
+  }
     if (XmlNameMatchesAscii(enc, ptr, end, KW_NOTATION)) {
       state->handler = attlist5;
       return XML_ROLE_ATTLIST_NONE;
@@ -860,12 +762,11 @@ attlist2(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist3(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist3(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -879,12 +780,11 @@ attlist3(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist4(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist4(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -899,12 +799,11 @@ attlist4(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist5(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist5(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -916,12 +815,11 @@ attlist5(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist6(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist6(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -933,12 +831,11 @@ attlist6(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist7(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist7(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -954,33 +851,23 @@ attlist7(PROLOG_STATE *state,
 
 /* default value */
 static int PTRCALL
-attlist8(PROLOG_STATE *state,
-         int tok,
-         const char *ptr,
-         const char *end,
-         const ENCODING *enc)
-{
+attlist8(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
   case XML_TOK_POUND_NAME:
-    if (XmlNameMatchesAscii(enc,
-                            ptr + MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + MIN_BYTES_PER_CHAR(enc), end,
                             KW_IMPLIED)) {
       state->handler = attlist1;
       return XML_ROLE_IMPLIED_ATTRIBUTE_VALUE;
     }
-    if (XmlNameMatchesAscii(enc,
-                            ptr + MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + MIN_BYTES_PER_CHAR(enc), end,
                             KW_REQUIRED)) {
       state->handler = attlist1;
       return XML_ROLE_REQUIRED_ATTRIBUTE_VALUE;
     }
-    if (XmlNameMatchesAscii(enc,
-                            ptr + MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + MIN_BYTES_PER_CHAR(enc), end,
                             KW_FIXED)) {
       state->handler = attlist9;
       return XML_ROLE_ATTLIST_NONE;
@@ -994,12 +881,11 @@ attlist8(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist9(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist9(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -1011,12 +897,11 @@ attlist9(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element0(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+element0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1029,12 +914,8 @@ element0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element1(PROLOG_STATE *state,
-         int tok,
-         const char *ptr,
-         const char *end,
-         const ENCODING *enc)
-{
+element1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1059,19 +940,13 @@ element1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element2(PROLOG_STATE *state,
-         int tok,
-         const char *ptr,
-         const char *end,
-         const ENCODING *enc)
-{
+element2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
   case XML_TOK_POUND_NAME:
-    if (XmlNameMatchesAscii(enc,
-                            ptr + MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + MIN_BYTES_PER_CHAR(enc), end,
                             KW_PCDATA)) {
       state->handler = element3;
       return XML_ROLE_CONTENT_PCDATA;
@@ -1099,12 +974,11 @@ element2(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element3(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+element3(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1124,12 +998,11 @@ element3(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element4(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+element4(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1142,12 +1015,11 @@ element4(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element5(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+element5(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1163,12 +1035,11 @@ element5(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element6(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+element6(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1193,12 +1064,11 @@ element6(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element7(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+element7(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1243,12 +1113,8 @@ element7(PROLOG_STATE *state,
 #ifdef XML_DTD
 
 static int PTRCALL
-condSect0(PROLOG_STATE *state,
-          int tok,
-          const char *ptr,
-          const char *end,
-          const ENCODING *enc)
-{
+condSect0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NONE;
@@ -1267,12 +1133,11 @@ condSect0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-condSect1(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+condSect1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NONE;
@@ -1285,12 +1150,11 @@ condSect1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-condSect2(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+condSect2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NONE;
@@ -1304,12 +1168,11 @@ condSect2(PROLOG_STATE *state,
 #endif /* XML_DTD */
 
 static int PTRCALL
-declClose(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+declClose(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return state->role_none;
@@ -1341,21 +1204,21 @@ declClose(PROLOG_STATE *state,
  * LCOV_EXCL_START
  */
 static int PTRCALL
-error(PROLOG_STATE *UNUSED_P(state),
-      int UNUSED_P(tok),
-      const char *UNUSED_P(ptr),
-      const char *UNUSED_P(end),
-      const ENCODING *UNUSED_P(enc))
-{
+error(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+      const ENCODING *enc) {
+  UNUSED_P(state);
+  UNUSED_P(tok);
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   return XML_ROLE_NONE;
 }
 /* LCOV_EXCL_STOP */
 
 static int FASTCALL
-common(PROLOG_STATE *state, int tok)
-{
+common(PROLOG_STATE *state, int tok) {
 #ifdef XML_DTD
-  if (!state->documentEntity && tok == XML_TOK_PARAM_ENTITY_REF)
+  if (! state->documentEntity && tok == XML_TOK_PARAM_ENTITY_REF)
     return XML_ROLE_INNER_PARAM_ENTITY_REF;
 #endif
   state->handler = error;
@@ -1363,8 +1226,7 @@ common(PROLOG_STATE *state, int tok)
 }
 
 void
-XmlPrologStateInit(PROLOG_STATE *state)
-{
+XmlPrologStateInit(PROLOG_STATE *state) {
   state->handler = prolog0;
 #ifdef XML_DTD
   state->documentEntity = 1;
@@ -1376,8 +1238,7 @@ XmlPrologStateInit(PROLOG_STATE *state)
 #ifdef XML_DTD
 
 void
-XmlPrologStateInitExternalEntity(PROLOG_STATE *state)
-{
+XmlPrologStateInitExternalEntity(PROLOG_STATE *state) {
   state->handler = externalSubset0;
   state->documentEntity = 0;
   state->includeLevel = 0;
diff --git a/Modules/expat/xmlrole.h b/Modules/expat/xmlrole.h
index e5f048eab55c6e..036aba64fd29c6 100644
--- a/Modules/expat/xmlrole.h
+++ b/Modules/expat/xmlrole.h
@@ -36,7 +36,7 @@
 #ifdef __VMS
 /*      0        1         2         3      0        1         2         3
         1234567890123456789012345678901     1234567890123456789012345678901 */
-#define XmlPrologStateInitExternalEntity    XmlPrologStateInitExternalEnt
+#  define XmlPrologStateInitExternalEntity XmlPrologStateInitExternalEnt
 #endif
 
 #include "xmltok.h"
@@ -113,11 +113,8 @@ enum {
 };
 
 typedef struct prolog_state {
-  int (PTRCALL *handler) (struct prolog_state *state,
-                          int tok,
-                          const char *ptr,
-                          const char *end,
-                          const ENCODING *enc);
+  int(PTRCALL *handler)(struct prolog_state *state, int tok, const char *ptr,
+                        const char *end, const ENCODING *enc);
   unsigned level;
   int role_none;
 #ifdef XML_DTD
@@ -132,8 +129,8 @@ void XmlPrologStateInit(PROLOG_STATE *);
 void XmlPrologStateInitExternalEntity(PROLOG_STATE *);
 #endif /* XML_DTD */
 
-#define XmlTokenRole(state, tok, ptr, end, enc) \
- (((state)->handler)(state, tok, ptr, end, enc))
+#define XmlTokenRole(state, tok, ptr, end, enc)                                \
+  (((state)->handler)(state, tok, ptr, end, enc))
 
 #ifdef __cplusplus
 }
diff --git a/Modules/expat/xmltok.c b/Modules/expat/xmltok.c
index 6b415d83972ca6..11e9d1ccdad423 100644
--- a/Modules/expat/xmltok.c
+++ b/Modules/expat/xmltok.c
@@ -31,24 +31,23 @@
 */
 
 #include 
-#include   /* memcpy */
+#include  /* memcpy */
 
 #if defined(_MSC_VER) && (_MSC_VER <= 1700)
-  /* for vs2012/11.0/1700 and earlier Visual Studio compilers */
-# define bool   int
-# define false  0
-# define true   1
+/* for vs2012/11.0/1700 and earlier Visual Studio compilers */
+#  define bool int
+#  define false 0
+#  define true 1
 #else
-# include 
+#  include 
 #endif
 
-
 #ifdef _WIN32
-#include "winconfig.h"
+#  include "winconfig.h"
 #else
-#ifdef HAVE_EXPAT_CONFIG_H
-#include 
-#endif
+#  ifdef HAVE_EXPAT_CONFIG_H
+#    include 
+#  endif
 #endif /* ndef _WIN32 */
 
 #include "expat_external.h"
@@ -57,58 +56,49 @@
 #include "nametab.h"
 
 #ifdef XML_DTD
-#define IGNORE_SECTION_TOK_VTABLE , PREFIX(ignoreSectionTok)
+#  define IGNORE_SECTION_TOK_VTABLE , PREFIX(ignoreSectionTok)
 #else
-#define IGNORE_SECTION_TOK_VTABLE /* as nothing */
+#  define IGNORE_SECTION_TOK_VTABLE /* as nothing */
 #endif
 
-#define VTABLE1 \
-  { PREFIX(prologTok), PREFIX(contentTok), \
-    PREFIX(cdataSectionTok) IGNORE_SECTION_TOK_VTABLE }, \
-  { PREFIX(attributeValueTok), PREFIX(entityValueTok) }, \
-  PREFIX(nameMatchesAscii), \
-  PREFIX(nameLength), \
-  PREFIX(skipS), \
-  PREFIX(getAtts), \
-  PREFIX(charRefNumber), \
-  PREFIX(predefinedEntityName), \
-  PREFIX(updatePosition), \
-  PREFIX(isPublicId)
+#define VTABLE1                                                                \
+  {PREFIX(prologTok), PREFIX(contentTok),                                      \
+   PREFIX(cdataSectionTok) IGNORE_SECTION_TOK_VTABLE},                         \
+      {PREFIX(attributeValueTok), PREFIX(entityValueTok)},                     \
+      PREFIX(nameMatchesAscii), PREFIX(nameLength), PREFIX(skipS),             \
+      PREFIX(getAtts), PREFIX(charRefNumber), PREFIX(predefinedEntityName),    \
+      PREFIX(updatePosition), PREFIX(isPublicId)
 
 #define VTABLE VTABLE1, PREFIX(toUtf8), PREFIX(toUtf16)
 
-#define UCS2_GET_NAMING(pages, hi, lo) \
-   (namingBitmap[(pages[hi] << 3) + ((lo) >> 5)] & (1u << ((lo) & 0x1F)))
+#define UCS2_GET_NAMING(pages, hi, lo)                                         \
+  (namingBitmap[(pages[hi] << 3) + ((lo) >> 5)] & (1u << ((lo)&0x1F)))
 
 /* A 2 byte UTF-8 representation splits the characters 11 bits between
    the bottom 5 and 6 bits of the bytes.  We need 8 bits to index into
    pages, 3 bits to add to that index and 5 bits to generate the mask.
 */
-#define UTF8_GET_NAMING2(pages, byte) \
-    (namingBitmap[((pages)[(((byte)[0]) >> 2) & 7] << 3) \
-                      + ((((byte)[0]) & 3) << 1) \
-                      + ((((byte)[1]) >> 5) & 1)] \
-         & (1u << (((byte)[1]) & 0x1F)))
+#define UTF8_GET_NAMING2(pages, byte)                                          \
+  (namingBitmap[((pages)[(((byte)[0]) >> 2) & 7] << 3)                         \
+                + ((((byte)[0]) & 3) << 1) + ((((byte)[1]) >> 5) & 1)]         \
+   & (1u << (((byte)[1]) & 0x1F)))
 
 /* A 3 byte UTF-8 representation splits the characters 16 bits between
    the bottom 4, 6 and 6 bits of the bytes.  We need 8 bits to index
    into pages, 3 bits to add to that index and 5 bits to generate the
    mask.
 */
-#define UTF8_GET_NAMING3(pages, byte) \
-  (namingBitmap[((pages)[((((byte)[0]) & 0xF) << 4) \
-                             + ((((byte)[1]) >> 2) & 0xF)] \
-                       << 3) \
-                      + ((((byte)[1]) & 3) << 1) \
-                      + ((((byte)[2]) >> 5) & 1)] \
-         & (1u << (((byte)[2]) & 0x1F)))
-
-#define UTF8_GET_NAMING(pages, p, n) \
-  ((n) == 2 \
-  ? UTF8_GET_NAMING2(pages, (const unsigned char *)(p)) \
-  : ((n) == 3 \
-     ? UTF8_GET_NAMING3(pages, (const unsigned char *)(p)) \
-     : 0))
+#define UTF8_GET_NAMING3(pages, byte)                                          \
+  (namingBitmap                                                                \
+       [((pages)[((((byte)[0]) & 0xF) << 4) + ((((byte)[1]) >> 2) & 0xF)]      \
+         << 3)                                                                 \
+        + ((((byte)[1]) & 3) << 1) + ((((byte)[2]) >> 5) & 1)]                 \
+   & (1u << (((byte)[2]) & 0x1F)))
+
+#define UTF8_GET_NAMING(pages, p, n)                                           \
+  ((n) == 2                                                                    \
+       ? UTF8_GET_NAMING2(pages, (const unsigned char *)(p))                   \
+       : ((n) == 3 ? UTF8_GET_NAMING3(pages, (const unsigned char *)(p)) : 0))
 
 /* Detection of invalid UTF-8 sequences is based on Table 3.1B
    of Unicode 3.2: http://www.unicode.org/unicode/reports/tr28/
@@ -120,88 +110,76 @@
      (A & 0xC0) == 0xC0  means A > 0xBF
 */
 
-#define UTF8_INVALID2(p) \
+#define UTF8_INVALID2(p)                                                       \
   ((*p) < 0xC2 || ((p)[1] & 0x80) == 0 || ((p)[1] & 0xC0) == 0xC0)
 
-#define UTF8_INVALID3(p) \
-  (((p)[2] & 0x80) == 0 \
-  || \
-  ((*p) == 0xEF && (p)[1] == 0xBF \
-    ? \
-    (p)[2] > 0xBD \
-    : \
-    ((p)[2] & 0xC0) == 0xC0) \
-  || \
-  ((*p) == 0xE0 \
-    ? \
-    (p)[1] < 0xA0 || ((p)[1] & 0xC0) == 0xC0 \
-    : \
-    ((p)[1] & 0x80) == 0 \
-    || \
-    ((*p) == 0xED ? (p)[1] > 0x9F : ((p)[1] & 0xC0) == 0xC0)))
-
-#define UTF8_INVALID4(p) \
-  (((p)[3] & 0x80) == 0 || ((p)[3] & 0xC0) == 0xC0 \
-  || \
-  ((p)[2] & 0x80) == 0 || ((p)[2] & 0xC0) == 0xC0 \
-  || \
-  ((*p) == 0xF0 \
-    ? \
-    (p)[1] < 0x90 || ((p)[1] & 0xC0) == 0xC0 \
-    : \
-    ((p)[1] & 0x80) == 0 \
-    || \
-    ((*p) == 0xF4 ? (p)[1] > 0x8F : ((p)[1] & 0xC0) == 0xC0)))
+#define UTF8_INVALID3(p)                                                       \
+  (((p)[2] & 0x80) == 0                                                        \
+   || ((*p) == 0xEF && (p)[1] == 0xBF ? (p)[2] > 0xBD                          \
+                                      : ((p)[2] & 0xC0) == 0xC0)               \
+   || ((*p) == 0xE0                                                            \
+           ? (p)[1] < 0xA0 || ((p)[1] & 0xC0) == 0xC0                          \
+           : ((p)[1] & 0x80) == 0                                              \
+                 || ((*p) == 0xED ? (p)[1] > 0x9F : ((p)[1] & 0xC0) == 0xC0)))
+
+#define UTF8_INVALID4(p)                                                       \
+  (((p)[3] & 0x80) == 0 || ((p)[3] & 0xC0) == 0xC0 || ((p)[2] & 0x80) == 0     \
+   || ((p)[2] & 0xC0) == 0xC0                                                  \
+   || ((*p) == 0xF0                                                            \
+           ? (p)[1] < 0x90 || ((p)[1] & 0xC0) == 0xC0                          \
+           : ((p)[1] & 0x80) == 0                                              \
+                 || ((*p) == 0xF4 ? (p)[1] > 0x8F : ((p)[1] & 0xC0) == 0xC0)))
 
 static int PTRFASTCALL
-isNever(const ENCODING *UNUSED_P(enc), const char *UNUSED_P(p))
-{
+isNever(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  UNUSED_P(p);
   return 0;
 }
 
 static int PTRFASTCALL
-utf8_isName2(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isName2(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_GET_NAMING2(namePages, (const unsigned char *)p);
 }
 
 static int PTRFASTCALL
-utf8_isName3(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isName3(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_GET_NAMING3(namePages, (const unsigned char *)p);
 }
 
 #define utf8_isName4 isNever
 
 static int PTRFASTCALL
-utf8_isNmstrt2(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isNmstrt2(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_GET_NAMING2(nmstrtPages, (const unsigned char *)p);
 }
 
 static int PTRFASTCALL
-utf8_isNmstrt3(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isNmstrt3(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_GET_NAMING3(nmstrtPages, (const unsigned char *)p);
 }
 
 #define utf8_isNmstrt4 isNever
 
 static int PTRFASTCALL
-utf8_isInvalid2(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isInvalid2(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_INVALID2((const unsigned char *)p);
 }
 
 static int PTRFASTCALL
-utf8_isInvalid3(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isInvalid3(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_INVALID3((const unsigned char *)p);
 }
 
 static int PTRFASTCALL
-utf8_isInvalid4(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isInvalid4(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_INVALID4((const unsigned char *)p);
 }
 
@@ -209,61 +187,44 @@ struct normal_encoding {
   ENCODING enc;
   unsigned char type[256];
 #ifdef XML_MIN_SIZE
-  int (PTRFASTCALL *byteType)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isNameMin)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isNmstrtMin)(const ENCODING *, const char *);
-  int (PTRFASTCALL *byteToAscii)(const ENCODING *, const char *);
-  int (PTRCALL *charMatches)(const ENCODING *, const char *, int);
+  int(PTRFASTCALL *byteType)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isNameMin)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isNmstrtMin)(const ENCODING *, const char *);
+  int(PTRFASTCALL *byteToAscii)(const ENCODING *, const char *);
+  int(PTRCALL *charMatches)(const ENCODING *, const char *, int);
 #endif /* XML_MIN_SIZE */
-  int (PTRFASTCALL *isName2)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isName3)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isName4)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isNmstrt2)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isNmstrt3)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isNmstrt4)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isInvalid2)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isInvalid3)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isInvalid4)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isName2)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isName3)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isName4)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isNmstrt2)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isNmstrt3)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isNmstrt4)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isInvalid2)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isInvalid3)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isInvalid4)(const ENCODING *, const char *);
 };
 
-#define AS_NORMAL_ENCODING(enc)   ((const struct normal_encoding *) (enc))
+#define AS_NORMAL_ENCODING(enc) ((const struct normal_encoding *)(enc))
 
 #ifdef XML_MIN_SIZE
 
-#define STANDARD_VTABLE(E) \
- E ## byteType, \
- E ## isNameMin, \
- E ## isNmstrtMin, \
- E ## byteToAscii, \
- E ## charMatches,
+#  define STANDARD_VTABLE(E)                                                   \
+    E##byteType, E##isNameMin, E##isNmstrtMin, E##byteToAscii, E##charMatches,
 
 #else
 
-#define STANDARD_VTABLE(E) /* as nothing */
+#  define STANDARD_VTABLE(E) /* as nothing */
 
 #endif
 
-#define NORMAL_VTABLE(E) \
- E ## isName2, \
- E ## isName3, \
- E ## isName4, \
- E ## isNmstrt2, \
- E ## isNmstrt3, \
- E ## isNmstrt4, \
- E ## isInvalid2, \
- E ## isInvalid3, \
- E ## isInvalid4
-
-#define NULL_VTABLE \
- /* isName2 */ NULL, \
- /* isName3 */ NULL, \
- /* isName4 */ NULL, \
- /* isNmstrt2 */ NULL, \
- /* isNmstrt3 */ NULL, \
- /* isNmstrt4 */ NULL, \
- /* isInvalid2 */ NULL, \
- /* isInvalid3 */ NULL, \
- /* isInvalid4 */ NULL
+#define NORMAL_VTABLE(E)                                                       \
+  E##isName2, E##isName3, E##isName4, E##isNmstrt2, E##isNmstrt3,              \
+      E##isNmstrt4, E##isInvalid2, E##isInvalid3, E##isInvalid4
+
+#define NULL_VTABLE                                                            \
+  /* isName2 */ NULL, /* isName3 */ NULL, /* isName4 */ NULL,                  \
+      /* isNmstrt2 */ NULL, /* isNmstrt3 */ NULL, /* isNmstrt4 */ NULL,        \
+      /* isInvalid2 */ NULL, /* isInvalid3 */ NULL, /* isInvalid4 */ NULL
 
 static int FASTCALL checkCharRefNumber(int);
 
@@ -271,75 +232,70 @@ static int FASTCALL checkCharRefNumber(int);
 #include "ascii.h"
 
 #ifdef XML_MIN_SIZE
-#define sb_isNameMin isNever
-#define sb_isNmstrtMin isNever
+#  define sb_isNameMin isNever
+#  define sb_isNmstrtMin isNever
 #endif
 
 #ifdef XML_MIN_SIZE
-#define MINBPC(enc) ((enc)->minBytesPerChar)
+#  define MINBPC(enc) ((enc)->minBytesPerChar)
 #else
 /* minimum bytes per character */
-#define MINBPC(enc) 1
+#  define MINBPC(enc) 1
 #endif
 
-#define SB_BYTE_TYPE(enc, p) \
+#define SB_BYTE_TYPE(enc, p)                                                   \
   (((struct normal_encoding *)(enc))->type[(unsigned char)*(p)])
 
 #ifdef XML_MIN_SIZE
 static int PTRFASTCALL
-sb_byteType(const ENCODING *enc, const char *p)
-{
+sb_byteType(const ENCODING *enc, const char *p) {
   return SB_BYTE_TYPE(enc, p);
 }
-#define BYTE_TYPE(enc, p) \
- (AS_NORMAL_ENCODING(enc)->byteType(enc, p))
+#  define BYTE_TYPE(enc, p) (AS_NORMAL_ENCODING(enc)->byteType(enc, p))
 #else
-#define BYTE_TYPE(enc, p) SB_BYTE_TYPE(enc, p)
+#  define BYTE_TYPE(enc, p) SB_BYTE_TYPE(enc, p)
 #endif
 
 #ifdef XML_MIN_SIZE
-#define BYTE_TO_ASCII(enc, p) \
- (AS_NORMAL_ENCODING(enc)->byteToAscii(enc, p))
+#  define BYTE_TO_ASCII(enc, p) (AS_NORMAL_ENCODING(enc)->byteToAscii(enc, p))
 static int PTRFASTCALL
-sb_byteToAscii(const ENCODING *enc, const char *p)
-{
+sb_byteToAscii(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return *p;
 }
 #else
-#define BYTE_TO_ASCII(enc, p) (*(p))
+#  define BYTE_TO_ASCII(enc, p) (*(p))
 #endif
 
-#define IS_NAME_CHAR(enc, p, n) \
- (AS_NORMAL_ENCODING(enc)->isName ## n(enc, p))
-#define IS_NMSTRT_CHAR(enc, p, n) \
- (AS_NORMAL_ENCODING(enc)->isNmstrt ## n(enc, p))
-#define IS_INVALID_CHAR(enc, p, n) \
- (AS_NORMAL_ENCODING(enc)->isInvalid ## n(enc, p))
+#define IS_NAME_CHAR(enc, p, n) (AS_NORMAL_ENCODING(enc)->isName##n(enc, p))
+#define IS_NMSTRT_CHAR(enc, p, n) (AS_NORMAL_ENCODING(enc)->isNmstrt##n(enc, p))
+#define IS_INVALID_CHAR(enc, p, n)                                             \
+  (AS_NORMAL_ENCODING(enc)->isInvalid##n(enc, p))
 
 #ifdef XML_MIN_SIZE
-#define IS_NAME_CHAR_MINBPC(enc, p) \
- (AS_NORMAL_ENCODING(enc)->isNameMin(enc, p))
-#define IS_NMSTRT_CHAR_MINBPC(enc, p) \
- (AS_NORMAL_ENCODING(enc)->isNmstrtMin(enc, p))
+#  define IS_NAME_CHAR_MINBPC(enc, p)                                          \
+    (AS_NORMAL_ENCODING(enc)->isNameMin(enc, p))
+#  define IS_NMSTRT_CHAR_MINBPC(enc, p)                                        \
+    (AS_NORMAL_ENCODING(enc)->isNmstrtMin(enc, p))
 #else
-#define IS_NAME_CHAR_MINBPC(enc, p) (0)
-#define IS_NMSTRT_CHAR_MINBPC(enc, p) (0)
+#  define IS_NAME_CHAR_MINBPC(enc, p) (0)
+#  define IS_NMSTRT_CHAR_MINBPC(enc, p) (0)
 #endif
 
 #ifdef XML_MIN_SIZE
-#define CHAR_MATCHES(enc, p, c) \
- (AS_NORMAL_ENCODING(enc)->charMatches(enc, p, c))
+#  define CHAR_MATCHES(enc, p, c)                                              \
+    (AS_NORMAL_ENCODING(enc)->charMatches(enc, p, c))
 static int PTRCALL
-sb_charMatches(const ENCODING *enc, const char *p, int c)
-{
+sb_charMatches(const ENCODING *enc, const char *p, int c) {
+  UNUSED_P(enc);
   return *p == c;
 }
 #else
 /* c is an ASCII character */
-#define CHAR_MATCHES(enc, p, c) (*(p) == c)
+#  define CHAR_MATCHES(enc, p, c) (*(p) == c)
 #endif
 
-#define PREFIX(ident) normal_ ## ident
+#define PREFIX(ident) normal_##ident
 #define XML_TOK_IMPL_C
 #include "xmltok_impl.c"
 #undef XML_TOK_IMPL_C
@@ -354,42 +310,46 @@ sb_charMatches(const ENCODING *enc, const char *p, int c)
 #undef IS_NMSTRT_CHAR_MINBPC
 #undef IS_INVALID_CHAR
 
-enum {  /* UTF8_cvalN is value of masked first byte of N byte sequence */
-  UTF8_cval1 = 0x00,
-  UTF8_cval2 = 0xc0,
-  UTF8_cval3 = 0xe0,
-  UTF8_cval4 = 0xf0
+enum { /* UTF8_cvalN is value of masked first byte of N byte sequence */
+       UTF8_cval1 = 0x00,
+       UTF8_cval2 = 0xc0,
+       UTF8_cval3 = 0xe0,
+       UTF8_cval4 = 0xf0
 };
 
 void
-_INTERNAL_trim_to_complete_utf8_characters(const char * from, const char ** fromLimRef)
-{
-  const char * fromLim = *fromLimRef;
+_INTERNAL_trim_to_complete_utf8_characters(const char *from,
+                                           const char **fromLimRef) {
+  const char *fromLim = *fromLimRef;
   size_t walked = 0;
   for (; fromLim > from; fromLim--, walked++) {
     const unsigned char prev = (unsigned char)fromLim[-1];
-    if ((prev & 0xf8u) == 0xf0u) { /* 4-byte character, lead by 0b11110xxx byte */
+    if ((prev & 0xf8u)
+        == 0xf0u) { /* 4-byte character, lead by 0b11110xxx byte */
       if (walked + 1 >= 4) {
         fromLim += 4 - 1;
         break;
       } else {
         walked = 0;
       }
-    } else if ((prev & 0xf0u) == 0xe0u) { /* 3-byte character, lead by 0b1110xxxx byte */
+    } else if ((prev & 0xf0u)
+               == 0xe0u) { /* 3-byte character, lead by 0b1110xxxx byte */
       if (walked + 1 >= 3) {
         fromLim += 3 - 1;
         break;
       } else {
         walked = 0;
       }
-    } else if ((prev & 0xe0u) == 0xc0u) { /* 2-byte character, lead by 0b110xxxxx byte */
+    } else if ((prev & 0xe0u)
+               == 0xc0u) { /* 2-byte character, lead by 0b110xxxxx byte */
       if (walked + 1 >= 2) {
         fromLim += 2 - 1;
         break;
       } else {
         walked = 0;
       }
-    } else if ((prev & 0x80u) == 0x00u) { /* 1-byte character, matching 0b0xxxxxxx */
+    } else if ((prev & 0x80u)
+               == 0x00u) { /* 1-byte character, matching 0b0xxxxxxx */
       break;
     }
   }
@@ -397,16 +357,15 @@ _INTERNAL_trim_to_complete_utf8_characters(const char * from, const char ** from
 }
 
 static enum XML_Convert_Result PTRCALL
-utf8_toUtf8(const ENCODING *UNUSED_P(enc),
-            const char **fromP, const char *fromLim,
-            char **toP, const char *toLim)
-{
+utf8_toUtf8(const ENCODING *enc, const char **fromP, const char *fromLim,
+            char **toP, const char *toLim) {
   bool input_incomplete = false;
   bool output_exhausted = false;
 
   /* Avoid copying partial characters (due to limited space). */
   const ptrdiff_t bytesAvailable = fromLim - *fromP;
   const ptrdiff_t bytesStorable = toLim - *toP;
+  UNUSED_P(enc);
   if (bytesAvailable > bytesStorable) {
     fromLim = *fromP + bytesStorable;
     output_exhausted = true;
@@ -414,7 +373,7 @@ utf8_toUtf8(const ENCODING *UNUSED_P(enc),
 
   /* Avoid copying partial characters (from incomplete input). */
   {
-    const char * const fromLimBefore = fromLim;
+    const char *const fromLimBefore = fromLim;
     _INTERNAL_trim_to_complete_utf8_characters(*fromP, &fromLim);
     if (fromLim < fromLimBefore) {
       input_incomplete = true;
@@ -428,7 +387,7 @@ utf8_toUtf8(const ENCODING *UNUSED_P(enc),
     *toP += bytesToCopy;
   }
 
-  if (output_exhausted)  /* needs to go first */
+  if (output_exhausted) /* needs to go first */
     return XML_CONVERT_OUTPUT_EXHAUSTED;
   else if (input_incomplete)
     return XML_CONVERT_INPUT_INCOMPLETE;
@@ -437,10 +396,8 @@ utf8_toUtf8(const ENCODING *UNUSED_P(enc),
 }
 
 static enum XML_Convert_Result PTRCALL
-utf8_toUtf16(const ENCODING *enc,
-             const char **fromP, const char *fromLim,
-             unsigned short **toP, const unsigned short *toLim)
-{
+utf8_toUtf16(const ENCODING *enc, const char **fromP, const char *fromLim,
+             unsigned short **toP, const unsigned short *toLim) {
   enum XML_Convert_Result res = XML_CONVERT_COMPLETED;
   unsigned short *to = *toP;
   const char *from = *fromP;
@@ -459,30 +416,28 @@ utf8_toUtf16(const ENCODING *enc,
         res = XML_CONVERT_INPUT_INCOMPLETE;
         goto after;
       }
-      *to++ = (unsigned short)(((from[0] & 0xf) << 12)
-                               | ((from[1] & 0x3f) << 6) | (from[2] & 0x3f));
+      *to++ = (unsigned short)(((from[0] & 0xf) << 12) | ((from[1] & 0x3f) << 6)
+                               | (from[2] & 0x3f));
       from += 3;
       break;
-    case BT_LEAD4:
-      {
-        unsigned long n;
-        if (toLim - to < 2) {
-          res = XML_CONVERT_OUTPUT_EXHAUSTED;
-          goto after;
-        }
-        if (fromLim - from < 4) {
-          res = XML_CONVERT_INPUT_INCOMPLETE;
-          goto after;
-        }
-        n = ((from[0] & 0x7) << 18) | ((from[1] & 0x3f) << 12)
-            | ((from[2] & 0x3f) << 6) | (from[3] & 0x3f);
-        n -= 0x10000;
-        to[0] = (unsigned short)((n >> 10) | 0xD800);
-        to[1] = (unsigned short)((n & 0x3FF) | 0xDC00);
-        to += 2;
-        from += 4;
+    case BT_LEAD4: {
+      unsigned long n;
+      if (toLim - to < 2) {
+        res = XML_CONVERT_OUTPUT_EXHAUSTED;
+        goto after;
       }
-      break;
+      if (fromLim - from < 4) {
+        res = XML_CONVERT_INPUT_INCOMPLETE;
+        goto after;
+      }
+      n = ((from[0] & 0x7) << 18) | ((from[1] & 0x3f) << 12)
+          | ((from[2] & 0x3f) << 6) | (from[3] & 0x3f);
+      n -= 0x10000;
+      to[0] = (unsigned short)((n >> 10) | 0xD800);
+      to[1] = (unsigned short)((n & 0x3FF) | 0xDC00);
+      to += 2;
+      from += 4;
+    } break;
     default:
       *to++ = *from++;
       break;
@@ -497,56 +452,51 @@ utf8_toUtf16(const ENCODING *enc,
 }
 
 #ifdef XML_NS
-static const struct normal_encoding utf8_encoding_ns = {
-  { VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0 },
-  {
-#include "asciitab.h"
-#include "utf8tab.h"
-  },
-  STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)
-};
+static const struct normal_encoding utf8_encoding_ns
+    = {{VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0},
+       {
+#  include "asciitab.h"
+#  include "utf8tab.h"
+       },
+       STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)};
 #endif
 
-static const struct normal_encoding utf8_encoding = {
-  { VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0 },
-  {
+static const struct normal_encoding utf8_encoding
+    = {{VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0},
+       {
 #define BT_COLON BT_NMSTRT
 #include "asciitab.h"
 #undef BT_COLON
 #include "utf8tab.h"
-  },
-  STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)
-};
+       },
+       STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)};
 
 #ifdef XML_NS
 
-static const struct normal_encoding internal_utf8_encoding_ns = {
-  { VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0 },
-  {
-#include "iasciitab.h"
-#include "utf8tab.h"
-  },
-  STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)
-};
+static const struct normal_encoding internal_utf8_encoding_ns
+    = {{VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0},
+       {
+#  include "iasciitab.h"
+#  include "utf8tab.h"
+       },
+       STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)};
 
 #endif
 
-static const struct normal_encoding internal_utf8_encoding = {
-  { VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0 },
-  {
+static const struct normal_encoding internal_utf8_encoding
+    = {{VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0},
+       {
 #define BT_COLON BT_NMSTRT
 #include "iasciitab.h"
 #undef BT_COLON
 #include "utf8tab.h"
-  },
-  STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)
-};
+       },
+       STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)};
 
 static enum XML_Convert_Result PTRCALL
-latin1_toUtf8(const ENCODING *UNUSED_P(enc),
-              const char **fromP, const char *fromLim,
-              char **toP, const char *toLim)
-{
+latin1_toUtf8(const ENCODING *enc, const char **fromP, const char *fromLim,
+              char **toP, const char *toLim) {
+  UNUSED_P(enc);
   for (;;) {
     unsigned char c;
     if (*fromP == fromLim)
@@ -558,8 +508,7 @@ latin1_toUtf8(const ENCODING *UNUSED_P(enc),
       *(*toP)++ = (char)((c >> 6) | UTF8_cval2);
       *(*toP)++ = (char)((c & 0x3f) | 0x80);
       (*fromP)++;
-    }
-    else {
+    } else {
       if (*toP == toLim)
         return XML_CONVERT_OUTPUT_EXHAUSTED;
       *(*toP)++ = *(*fromP)++;
@@ -568,10 +517,9 @@ latin1_toUtf8(const ENCODING *UNUSED_P(enc),
 }
 
 static enum XML_Convert_Result PTRCALL
-latin1_toUtf16(const ENCODING *UNUSED_P(enc),
-               const char **fromP, const char *fromLim,
-               unsigned short **toP, const unsigned short *toLim)
-{
+latin1_toUtf16(const ENCODING *enc, const char **fromP, const char *fromLim,
+               unsigned short **toP, const unsigned short *toLim) {
+  UNUSED_P(enc);
   while (*fromP < fromLim && *toP < toLim)
     *(*toP)++ = (unsigned char)*(*fromP)++;
 
@@ -583,33 +531,30 @@ latin1_toUtf16(const ENCODING *UNUSED_P(enc),
 
 #ifdef XML_NS
 
-static const struct normal_encoding latin1_encoding_ns = {
-  { VTABLE1, latin1_toUtf8, latin1_toUtf16, 1, 0, 0 },
-  {
-#include "asciitab.h"
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(sb_) NULL_VTABLE
-};
+static const struct normal_encoding latin1_encoding_ns
+    = {{VTABLE1, latin1_toUtf8, latin1_toUtf16, 1, 0, 0},
+       {
+#  include "asciitab.h"
+#  include "latin1tab.h"
+       },
+       STANDARD_VTABLE(sb_) NULL_VTABLE};
 
 #endif
 
-static const struct normal_encoding latin1_encoding = {
-  { VTABLE1, latin1_toUtf8, latin1_toUtf16, 1, 0, 0 },
-  {
+static const struct normal_encoding latin1_encoding
+    = {{VTABLE1, latin1_toUtf8, latin1_toUtf16, 1, 0, 0},
+       {
 #define BT_COLON BT_NMSTRT
 #include "asciitab.h"
 #undef BT_COLON
 #include "latin1tab.h"
-  },
-  STANDARD_VTABLE(sb_) NULL_VTABLE
-};
+       },
+       STANDARD_VTABLE(sb_) NULL_VTABLE};
 
 static enum XML_Convert_Result PTRCALL
-ascii_toUtf8(const ENCODING *UNUSED_P(enc),
-             const char **fromP, const char *fromLim,
-             char **toP, const char *toLim)
-{
+ascii_toUtf8(const ENCODING *enc, const char **fromP, const char *fromLim,
+             char **toP, const char *toLim) {
+  UNUSED_P(enc);
   while (*fromP < fromLim && *toP < toLim)
     *(*toP)++ = *(*fromP)++;
 
@@ -621,40 +566,45 @@ ascii_toUtf8(const ENCODING *UNUSED_P(enc),
 
 #ifdef XML_NS
 
-static const struct normal_encoding ascii_encoding_ns = {
-  { VTABLE1, ascii_toUtf8, latin1_toUtf16, 1, 1, 0 },
-  {
-#include "asciitab.h"
-/* BT_NONXML == 0 */
-  },
-  STANDARD_VTABLE(sb_) NULL_VTABLE
-};
+static const struct normal_encoding ascii_encoding_ns
+    = {{VTABLE1, ascii_toUtf8, latin1_toUtf16, 1, 1, 0},
+       {
+#  include "asciitab.h"
+           /* BT_NONXML == 0 */
+       },
+       STANDARD_VTABLE(sb_) NULL_VTABLE};
 
 #endif
 
-static const struct normal_encoding ascii_encoding = {
-  { VTABLE1, ascii_toUtf8, latin1_toUtf16, 1, 1, 0 },
-  {
+static const struct normal_encoding ascii_encoding
+    = {{VTABLE1, ascii_toUtf8, latin1_toUtf16, 1, 1, 0},
+       {
 #define BT_COLON BT_NMSTRT
 #include "asciitab.h"
 #undef BT_COLON
-/* BT_NONXML == 0 */
-  },
-  STANDARD_VTABLE(sb_) NULL_VTABLE
-};
+           /* BT_NONXML == 0 */
+       },
+       STANDARD_VTABLE(sb_) NULL_VTABLE};
 
 static int PTRFASTCALL
-unicode_byte_type(char hi, char lo)
-{
+unicode_byte_type(char hi, char lo) {
   switch ((unsigned char)hi) {
-  case 0xD8: case 0xD9: case 0xDA: case 0xDB:
+  /* 0xD800–0xDBFF first 16-bit code unit or high surrogate (W1) */
+  case 0xD8:
+  case 0xD9:
+  case 0xDA:
+  case 0xDB:
     return BT_LEAD4;
-  case 0xDC: case 0xDD: case 0xDE: case 0xDF:
+  /* 0xDC00–0xDFFF second 16-bit code unit or low surrogate (W2) */
+  case 0xDC:
+  case 0xDD:
+  case 0xDE:
+  case 0xDF:
     return BT_TRAIL;
   case 0xFF:
     switch ((unsigned char)lo) {
-    case 0xFF:
-    case 0xFE:
+    case 0xFF: /* noncharacter-FFFF */
+    case 0xFE: /* noncharacter-FFFE */
       return BT_NONXML;
     }
     break;
@@ -662,102 +612,105 @@ unicode_byte_type(char hi, char lo)
   return BT_NONASCII;
 }
 
-#define DEFINE_UTF16_TO_UTF8(E) \
-static enum XML_Convert_Result  PTRCALL \
-E ## toUtf8(const ENCODING *UNUSED_P(enc), \
-            const char **fromP, const char *fromLim, \
-            char **toP, const char *toLim) \
-{ \
-  const char *from = *fromP; \
-  fromLim = from + (((fromLim - from) >> 1) << 1);  /* shrink to even */ \
-  for (; from < fromLim; from += 2) { \
-    int plane; \
-    unsigned char lo2; \
-    unsigned char lo = GET_LO(from); \
-    unsigned char hi = GET_HI(from); \
-    switch (hi) { \
-    case 0: \
-      if (lo < 0x80) { \
-        if (*toP == toLim) { \
-          *fromP = from; \
-          return XML_CONVERT_OUTPUT_EXHAUSTED; \
-        } \
-        *(*toP)++ = lo; \
-        break; \
-      } \
-      /* fall through */ \
-    case 0x1: case 0x2: case 0x3: \
-    case 0x4: case 0x5: case 0x6: case 0x7: \
-      if (toLim -  *toP < 2) { \
-        *fromP = from; \
-        return XML_CONVERT_OUTPUT_EXHAUSTED; \
-      } \
-      *(*toP)++ = ((lo >> 6) | (hi << 2) |  UTF8_cval2); \
-      *(*toP)++ = ((lo & 0x3f) | 0x80); \
-      break; \
-    default: \
-      if (toLim -  *toP < 3)  { \
-        *fromP = from; \
-        return XML_CONVERT_OUTPUT_EXHAUSTED; \
-      } \
-      /* 16 bits divided 4, 6, 6 amongst 3 bytes */ \
-      *(*toP)++ = ((hi >> 4) | UTF8_cval3); \
-      *(*toP)++ = (((hi & 0xf) << 2) | (lo >> 6) | 0x80); \
-      *(*toP)++ = ((lo & 0x3f) | 0x80); \
-      break; \
-    case 0xD8: case 0xD9: case 0xDA: case 0xDB: \
-      if (toLim -  *toP < 4) { \
-        *fromP = from; \
-        return XML_CONVERT_OUTPUT_EXHAUSTED; \
-      } \
-      if (fromLim - from < 4) { \
-        *fromP = from; \
-        return XML_CONVERT_INPUT_INCOMPLETE; \
-      } \
-      plane = (((hi & 0x3) << 2) | ((lo >> 6) & 0x3)) + 1; \
-      *(*toP)++ = ((plane >> 2) | UTF8_cval4); \
-      *(*toP)++ = (((lo >> 2) & 0xF) | ((plane & 0x3) << 4) | 0x80); \
-      from += 2; \
-      lo2 = GET_LO(from); \
-      *(*toP)++ = (((lo & 0x3) << 4) \
-                   | ((GET_HI(from) & 0x3) << 2) \
-                   | (lo2 >> 6) \
-                   | 0x80); \
-      *(*toP)++ = ((lo2 & 0x3f) | 0x80); \
-      break; \
-    } \
-  } \
-  *fromP = from; \
-  if (from < fromLim) \
-    return XML_CONVERT_INPUT_INCOMPLETE; \
-  else \
-    return XML_CONVERT_COMPLETED; \
-}
+#define DEFINE_UTF16_TO_UTF8(E)                                                \
+  static enum XML_Convert_Result PTRCALL E##toUtf8(                            \
+      const ENCODING *enc, const char **fromP, const char *fromLim,            \
+      char **toP, const char *toLim) {                                         \
+    const char *from = *fromP;                                                 \
+    UNUSED_P(enc);                                                             \
+    fromLim = from + (((fromLim - from) >> 1) << 1); /* shrink to even */      \
+    for (; from < fromLim; from += 2) {                                        \
+      int plane;                                                               \
+      unsigned char lo2;                                                       \
+      unsigned char lo = GET_LO(from);                                         \
+      unsigned char hi = GET_HI(from);                                         \
+      switch (hi) {                                                            \
+      case 0:                                                                  \
+        if (lo < 0x80) {                                                       \
+          if (*toP == toLim) {                                                 \
+            *fromP = from;                                                     \
+            return XML_CONVERT_OUTPUT_EXHAUSTED;                               \
+          }                                                                    \
+          *(*toP)++ = lo;                                                      \
+          break;                                                               \
+        }                                                                      \
+        /* fall through */                                                     \
+      case 0x1:                                                                \
+      case 0x2:                                                                \
+      case 0x3:                                                                \
+      case 0x4:                                                                \
+      case 0x5:                                                                \
+      case 0x6:                                                                \
+      case 0x7:                                                                \
+        if (toLim - *toP < 2) {                                                \
+          *fromP = from;                                                       \
+          return XML_CONVERT_OUTPUT_EXHAUSTED;                                 \
+        }                                                                      \
+        *(*toP)++ = ((lo >> 6) | (hi << 2) | UTF8_cval2);                      \
+        *(*toP)++ = ((lo & 0x3f) | 0x80);                                      \
+        break;                                                                 \
+      default:                                                                 \
+        if (toLim - *toP < 3) {                                                \
+          *fromP = from;                                                       \
+          return XML_CONVERT_OUTPUT_EXHAUSTED;                                 \
+        }                                                                      \
+        /* 16 bits divided 4, 6, 6 amongst 3 bytes */                          \
+        *(*toP)++ = ((hi >> 4) | UTF8_cval3);                                  \
+        *(*toP)++ = (((hi & 0xf) << 2) | (lo >> 6) | 0x80);                    \
+        *(*toP)++ = ((lo & 0x3f) | 0x80);                                      \
+        break;                                                                 \
+      case 0xD8:                                                               \
+      case 0xD9:                                                               \
+      case 0xDA:                                                               \
+      case 0xDB:                                                               \
+        if (toLim - *toP < 4) {                                                \
+          *fromP = from;                                                       \
+          return XML_CONVERT_OUTPUT_EXHAUSTED;                                 \
+        }                                                                      \
+        if (fromLim - from < 4) {                                              \
+          *fromP = from;                                                       \
+          return XML_CONVERT_INPUT_INCOMPLETE;                                 \
+        }                                                                      \
+        plane = (((hi & 0x3) << 2) | ((lo >> 6) & 0x3)) + 1;                   \
+        *(*toP)++ = (char)((plane >> 2) | UTF8_cval4);                         \
+        *(*toP)++ = (((lo >> 2) & 0xF) | ((plane & 0x3) << 4) | 0x80);         \
+        from += 2;                                                             \
+        lo2 = GET_LO(from);                                                    \
+        *(*toP)++ = (((lo & 0x3) << 4) | ((GET_HI(from) & 0x3) << 2)           \
+                     | (lo2 >> 6) | 0x80);                                     \
+        *(*toP)++ = ((lo2 & 0x3f) | 0x80);                                     \
+        break;                                                                 \
+      }                                                                        \
+    }                                                                          \
+    *fromP = from;                                                             \
+    if (from < fromLim)                                                        \
+      return XML_CONVERT_INPUT_INCOMPLETE;                                     \
+    else                                                                       \
+      return XML_CONVERT_COMPLETED;                                            \
+  }
 
-#define DEFINE_UTF16_TO_UTF16(E) \
-static enum XML_Convert_Result  PTRCALL \
-E ## toUtf16(const ENCODING *UNUSED_P(enc), \
-             const char **fromP, const char *fromLim, \
-             unsigned short **toP, const unsigned short *toLim) \
-{ \
-  enum XML_Convert_Result res = XML_CONVERT_COMPLETED; \
-  fromLim = *fromP + (((fromLim - *fromP) >> 1) << 1);  /* shrink to even */ \
-  /* Avoid copying first half only of surrogate */ \
-  if (fromLim - *fromP > ((toLim - *toP) << 1) \
-      && (GET_HI(fromLim - 2) & 0xF8) == 0xD8) { \
-    fromLim -= 2; \
-    res = XML_CONVERT_INPUT_INCOMPLETE; \
-  } \
-  for (; *fromP < fromLim && *toP < toLim; *fromP += 2) \
-    *(*toP)++ = (GET_HI(*fromP) << 8) | GET_LO(*fromP); \
-  if ((*toP == toLim) && (*fromP < fromLim)) \
-    return XML_CONVERT_OUTPUT_EXHAUSTED; \
-  else \
-    return res; \
-}
+#define DEFINE_UTF16_TO_UTF16(E)                                               \
+  static enum XML_Convert_Result PTRCALL E##toUtf16(                           \
+      const ENCODING *enc, const char **fromP, const char *fromLim,            \
+      unsigned short **toP, const unsigned short *toLim) {                     \
+    enum XML_Convert_Result res = XML_CONVERT_COMPLETED;                       \
+    UNUSED_P(enc);                                                             \
+    fromLim = *fromP + (((fromLim - *fromP) >> 1) << 1); /* shrink to even */  \
+    /* Avoid copying first half only of surrogate */                           \
+    if (fromLim - *fromP > ((toLim - *toP) << 1)                               \
+        && (GET_HI(fromLim - 2) & 0xF8) == 0xD8) {                             \
+      fromLim -= 2;                                                            \
+      res = XML_CONVERT_INPUT_INCOMPLETE;                                      \
+    }                                                                          \
+    for (; *fromP < fromLim && *toP < toLim; *fromP += 2)                      \
+      *(*toP)++ = (GET_HI(*fromP) << 8) | GET_LO(*fromP);                      \
+    if ((*toP == toLim) && (*fromP < fromLim))                                 \
+      return XML_CONVERT_OUTPUT_EXHAUSTED;                                     \
+    else                                                                       \
+      return res;                                                              \
+  }
 
-#define SET2(ptr, ch) \
-  (((ptr)[0] = ((ch) & 0xff)), ((ptr)[1] = ((ch) >> 8)))
+#define SET2(ptr, ch) (((ptr)[0] = ((ch)&0xff)), ((ptr)[1] = ((ch) >> 8)))
 #define GET_LO(ptr) ((unsigned char)(ptr)[0])
 #define GET_HI(ptr) ((unsigned char)(ptr)[1])
 
@@ -768,8 +721,7 @@ DEFINE_UTF16_TO_UTF16(little2_)
 #undef GET_LO
 #undef GET_HI
 
-#define SET2(ptr, ch) \
-  (((ptr)[0] = ((ch) >> 8)), ((ptr)[1] = ((ch) & 0xFF)))
+#define SET2(ptr, ch) (((ptr)[0] = ((ch) >> 8)), ((ptr)[1] = ((ch)&0xFF)))
 #define GET_LO(ptr) ((unsigned char)(ptr)[1])
 #define GET_HI(ptr) ((unsigned char)(ptr)[0])
 
@@ -780,292 +732,279 @@ DEFINE_UTF16_TO_UTF16(big2_)
 #undef GET_LO
 #undef GET_HI
 
-#define LITTLE2_BYTE_TYPE(enc, p) \
- ((p)[1] == 0 \
-  ? ((struct normal_encoding *)(enc))->type[(unsigned char)*(p)] \
-  : unicode_byte_type((p)[1], (p)[0]))
-#define LITTLE2_BYTE_TO_ASCII(enc, p) ((p)[1] == 0 ? (p)[0] : -1)
-#define LITTLE2_CHAR_MATCHES(enc, p, c) ((p)[1] == 0 && (p)[0] == c)
-#define LITTLE2_IS_NAME_CHAR_MINBPC(enc, p) \
+#define LITTLE2_BYTE_TYPE(enc, p)                                              \
+  ((p)[1] == 0 ? ((struct normal_encoding *)(enc))->type[(unsigned char)*(p)]  \
+               : unicode_byte_type((p)[1], (p)[0]))
+#define LITTLE2_BYTE_TO_ASCII(p) ((p)[1] == 0 ? (p)[0] : -1)
+#define LITTLE2_CHAR_MATCHES(p, c) ((p)[1] == 0 && (p)[0] == c)
+#define LITTLE2_IS_NAME_CHAR_MINBPC(p)                                         \
   UCS2_GET_NAMING(namePages, (unsigned char)p[1], (unsigned char)p[0])
-#define LITTLE2_IS_NMSTRT_CHAR_MINBPC(enc, p) \
+#define LITTLE2_IS_NMSTRT_CHAR_MINBPC(p)                                       \
   UCS2_GET_NAMING(nmstrtPages, (unsigned char)p[1], (unsigned char)p[0])
 
 #ifdef XML_MIN_SIZE
 
 static int PTRFASTCALL
-little2_byteType(const ENCODING *enc, const char *p)
-{
+little2_byteType(const ENCODING *enc, const char *p) {
   return LITTLE2_BYTE_TYPE(enc, p);
 }
 
 static int PTRFASTCALL
-little2_byteToAscii(const ENCODING *enc, const char *p)
-{
-  return LITTLE2_BYTE_TO_ASCII(enc, p);
+little2_byteToAscii(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  return LITTLE2_BYTE_TO_ASCII(p);
 }
 
 static int PTRCALL
-little2_charMatches(const ENCODING *enc, const char *p, int c)
-{
-  return LITTLE2_CHAR_MATCHES(enc, p, c);
+little2_charMatches(const ENCODING *enc, const char *p, int c) {
+  UNUSED_P(enc);
+  return LITTLE2_CHAR_MATCHES(p, c);
 }
 
 static int PTRFASTCALL
-little2_isNameMin(const ENCODING *enc, const char *p)
-{
-  return LITTLE2_IS_NAME_CHAR_MINBPC(enc, p);
+little2_isNameMin(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  return LITTLE2_IS_NAME_CHAR_MINBPC(p);
 }
 
 static int PTRFASTCALL
-little2_isNmstrtMin(const ENCODING *enc, const char *p)
-{
-  return LITTLE2_IS_NMSTRT_CHAR_MINBPC(enc, p);
+little2_isNmstrtMin(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  return LITTLE2_IS_NMSTRT_CHAR_MINBPC(p);
 }
 
-#undef VTABLE
-#define VTABLE VTABLE1, little2_toUtf8, little2_toUtf16
+#  undef VTABLE
+#  define VTABLE VTABLE1, little2_toUtf8, little2_toUtf16
 
 #else /* not XML_MIN_SIZE */
 
-#undef PREFIX
-#define PREFIX(ident) little2_ ## ident
-#define MINBPC(enc) 2
+#  undef PREFIX
+#  define PREFIX(ident) little2_##ident
+#  define MINBPC(enc) 2
 /* CHAR_MATCHES is guaranteed to have MINBPC bytes available. */
-#define BYTE_TYPE(enc, p) LITTLE2_BYTE_TYPE(enc, p)
-#define BYTE_TO_ASCII(enc, p) LITTLE2_BYTE_TO_ASCII(enc, p)
-#define CHAR_MATCHES(enc, p, c) LITTLE2_CHAR_MATCHES(enc, p, c)
-#define IS_NAME_CHAR(enc, p, n) 0
-#define IS_NAME_CHAR_MINBPC(enc, p) LITTLE2_IS_NAME_CHAR_MINBPC(enc, p)
-#define IS_NMSTRT_CHAR(enc, p, n) (0)
-#define IS_NMSTRT_CHAR_MINBPC(enc, p) LITTLE2_IS_NMSTRT_CHAR_MINBPC(enc, p)
-
-#define XML_TOK_IMPL_C
-#include "xmltok_impl.c"
-#undef XML_TOK_IMPL_C
-
-#undef MINBPC
-#undef BYTE_TYPE
-#undef BYTE_TO_ASCII
-#undef CHAR_MATCHES
-#undef IS_NAME_CHAR
-#undef IS_NAME_CHAR_MINBPC
-#undef IS_NMSTRT_CHAR
-#undef IS_NMSTRT_CHAR_MINBPC
-#undef IS_INVALID_CHAR
+#  define BYTE_TYPE(enc, p) LITTLE2_BYTE_TYPE(enc, p)
+#  define BYTE_TO_ASCII(enc, p) LITTLE2_BYTE_TO_ASCII(p)
+#  define CHAR_MATCHES(enc, p, c) LITTLE2_CHAR_MATCHES(p, c)
+#  define IS_NAME_CHAR(enc, p, n) 0
+#  define IS_NAME_CHAR_MINBPC(enc, p) LITTLE2_IS_NAME_CHAR_MINBPC(p)
+#  define IS_NMSTRT_CHAR(enc, p, n) (0)
+#  define IS_NMSTRT_CHAR_MINBPC(enc, p) LITTLE2_IS_NMSTRT_CHAR_MINBPC(p)
+
+#  define XML_TOK_IMPL_C
+#  include "xmltok_impl.c"
+#  undef XML_TOK_IMPL_C
+
+#  undef MINBPC
+#  undef BYTE_TYPE
+#  undef BYTE_TO_ASCII
+#  undef CHAR_MATCHES
+#  undef IS_NAME_CHAR
+#  undef IS_NAME_CHAR_MINBPC
+#  undef IS_NMSTRT_CHAR
+#  undef IS_NMSTRT_CHAR_MINBPC
+#  undef IS_INVALID_CHAR
 
 #endif /* not XML_MIN_SIZE */
 
 #ifdef XML_NS
 
-static const struct normal_encoding little2_encoding_ns = {
-  { VTABLE, 2, 0,
-#if BYTEORDER == 1234
-    1
-#else
-    0
-#endif
-  },
-  {
-#include "asciitab.h"
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(little2_) NULL_VTABLE
-};
+static const struct normal_encoding little2_encoding_ns
+    = {{VTABLE, 2, 0,
+#  if BYTEORDER == 1234
+        1
+#  else
+        0
+#  endif
+       },
+       {
+#  include "asciitab.h"
+#  include "latin1tab.h"
+       },
+       STANDARD_VTABLE(little2_) NULL_VTABLE};
 
 #endif
 
-static const struct normal_encoding little2_encoding = {
-  { VTABLE, 2, 0,
+static const struct normal_encoding little2_encoding
+    = {{VTABLE, 2, 0,
 #if BYTEORDER == 1234
-    1
+        1
 #else
-    0
+        0
 #endif
-  },
-  {
+       },
+       {
 #define BT_COLON BT_NMSTRT
 #include "asciitab.h"
 #undef BT_COLON
 #include "latin1tab.h"
-  },
-  STANDARD_VTABLE(little2_) NULL_VTABLE
-};
+       },
+       STANDARD_VTABLE(little2_) NULL_VTABLE};
 
 #if BYTEORDER != 4321
 
-#ifdef XML_NS
+#  ifdef XML_NS
 
-static const struct normal_encoding internal_little2_encoding_ns = {
-  { VTABLE, 2, 0, 1 },
-  {
-#include "iasciitab.h"
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(little2_) NULL_VTABLE
-};
+static const struct normal_encoding internal_little2_encoding_ns
+    = {{VTABLE, 2, 0, 1},
+       {
+#    include "iasciitab.h"
+#    include "latin1tab.h"
+       },
+       STANDARD_VTABLE(little2_) NULL_VTABLE};
 
-#endif
+#  endif
 
-static const struct normal_encoding internal_little2_encoding = {
-  { VTABLE, 2, 0, 1 },
-  {
-#define BT_COLON BT_NMSTRT
-#include "iasciitab.h"
-#undef BT_COLON
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(little2_) NULL_VTABLE
-};
+static const struct normal_encoding internal_little2_encoding
+    = {{VTABLE, 2, 0, 1},
+       {
+#  define BT_COLON BT_NMSTRT
+#  include "iasciitab.h"
+#  undef BT_COLON
+#  include "latin1tab.h"
+       },
+       STANDARD_VTABLE(little2_) NULL_VTABLE};
 
 #endif
 
-
-#define BIG2_BYTE_TYPE(enc, p) \
- ((p)[0] == 0 \
-  ? ((struct normal_encoding *)(enc))->type[(unsigned char)(p)[1]] \
-  : unicode_byte_type((p)[0], (p)[1]))
-#define BIG2_BYTE_TO_ASCII(enc, p) ((p)[0] == 0 ? (p)[1] : -1)
-#define BIG2_CHAR_MATCHES(enc, p, c) ((p)[0] == 0 && (p)[1] == c)
-#define BIG2_IS_NAME_CHAR_MINBPC(enc, p) \
+#define BIG2_BYTE_TYPE(enc, p)                                                 \
+  ((p)[0] == 0                                                                 \
+       ? ((struct normal_encoding *)(enc))->type[(unsigned char)(p)[1]]        \
+       : unicode_byte_type((p)[0], (p)[1]))
+#define BIG2_BYTE_TO_ASCII(p) ((p)[0] == 0 ? (p)[1] : -1)
+#define BIG2_CHAR_MATCHES(p, c) ((p)[0] == 0 && (p)[1] == c)
+#define BIG2_IS_NAME_CHAR_MINBPC(p)                                            \
   UCS2_GET_NAMING(namePages, (unsigned char)p[0], (unsigned char)p[1])
-#define BIG2_IS_NMSTRT_CHAR_MINBPC(enc, p) \
+#define BIG2_IS_NMSTRT_CHAR_MINBPC(p)                                          \
   UCS2_GET_NAMING(nmstrtPages, (unsigned char)p[0], (unsigned char)p[1])
 
 #ifdef XML_MIN_SIZE
 
 static int PTRFASTCALL
-big2_byteType(const ENCODING *enc, const char *p)
-{
+big2_byteType(const ENCODING *enc, const char *p) {
   return BIG2_BYTE_TYPE(enc, p);
 }
 
 static int PTRFASTCALL
-big2_byteToAscii(const ENCODING *enc, const char *p)
-{
-  return BIG2_BYTE_TO_ASCII(enc, p);
+big2_byteToAscii(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  return BIG2_BYTE_TO_ASCII(p);
 }
 
 static int PTRCALL
-big2_charMatches(const ENCODING *enc, const char *p, int c)
-{
-  return BIG2_CHAR_MATCHES(enc, p, c);
+big2_charMatches(const ENCODING *enc, const char *p, int c) {
+  UNUSED_P(enc);
+  return BIG2_CHAR_MATCHES(p, c);
 }
 
 static int PTRFASTCALL
-big2_isNameMin(const ENCODING *enc, const char *p)
-{
-  return BIG2_IS_NAME_CHAR_MINBPC(enc, p);
+big2_isNameMin(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  return BIG2_IS_NAME_CHAR_MINBPC(p);
 }
 
 static int PTRFASTCALL
-big2_isNmstrtMin(const ENCODING *enc, const char *p)
-{
-  return BIG2_IS_NMSTRT_CHAR_MINBPC(enc, p);
+big2_isNmstrtMin(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  return BIG2_IS_NMSTRT_CHAR_MINBPC(p);
 }
 
-#undef VTABLE
-#define VTABLE VTABLE1, big2_toUtf8, big2_toUtf16
+#  undef VTABLE
+#  define VTABLE VTABLE1, big2_toUtf8, big2_toUtf16
 
 #else /* not XML_MIN_SIZE */
 
-#undef PREFIX
-#define PREFIX(ident) big2_ ## ident
-#define MINBPC(enc) 2
+#  undef PREFIX
+#  define PREFIX(ident) big2_##ident
+#  define MINBPC(enc) 2
 /* CHAR_MATCHES is guaranteed to have MINBPC bytes available. */
-#define BYTE_TYPE(enc, p) BIG2_BYTE_TYPE(enc, p)
-#define BYTE_TO_ASCII(enc, p) BIG2_BYTE_TO_ASCII(enc, p)
-#define CHAR_MATCHES(enc, p, c) BIG2_CHAR_MATCHES(enc, p, c)
-#define IS_NAME_CHAR(enc, p, n) 0
-#define IS_NAME_CHAR_MINBPC(enc, p) BIG2_IS_NAME_CHAR_MINBPC(enc, p)
-#define IS_NMSTRT_CHAR(enc, p, n) (0)
-#define IS_NMSTRT_CHAR_MINBPC(enc, p) BIG2_IS_NMSTRT_CHAR_MINBPC(enc, p)
-
-#define XML_TOK_IMPL_C
-#include "xmltok_impl.c"
-#undef XML_TOK_IMPL_C
-
-#undef MINBPC
-#undef BYTE_TYPE
-#undef BYTE_TO_ASCII
-#undef CHAR_MATCHES
-#undef IS_NAME_CHAR
-#undef IS_NAME_CHAR_MINBPC
-#undef IS_NMSTRT_CHAR
-#undef IS_NMSTRT_CHAR_MINBPC
-#undef IS_INVALID_CHAR
+#  define BYTE_TYPE(enc, p) BIG2_BYTE_TYPE(enc, p)
+#  define BYTE_TO_ASCII(enc, p) BIG2_BYTE_TO_ASCII(p)
+#  define CHAR_MATCHES(enc, p, c) BIG2_CHAR_MATCHES(p, c)
+#  define IS_NAME_CHAR(enc, p, n) 0
+#  define IS_NAME_CHAR_MINBPC(enc, p) BIG2_IS_NAME_CHAR_MINBPC(p)
+#  define IS_NMSTRT_CHAR(enc, p, n) (0)
+#  define IS_NMSTRT_CHAR_MINBPC(enc, p) BIG2_IS_NMSTRT_CHAR_MINBPC(p)
+
+#  define XML_TOK_IMPL_C
+#  include "xmltok_impl.c"
+#  undef XML_TOK_IMPL_C
+
+#  undef MINBPC
+#  undef BYTE_TYPE
+#  undef BYTE_TO_ASCII
+#  undef CHAR_MATCHES
+#  undef IS_NAME_CHAR
+#  undef IS_NAME_CHAR_MINBPC
+#  undef IS_NMSTRT_CHAR
+#  undef IS_NMSTRT_CHAR_MINBPC
+#  undef IS_INVALID_CHAR
 
 #endif /* not XML_MIN_SIZE */
 
 #ifdef XML_NS
 
-static const struct normal_encoding big2_encoding_ns = {
-  { VTABLE, 2, 0,
-#if BYTEORDER == 4321
-  1
-#else
-  0
-#endif
-  },
-  {
-#include "asciitab.h"
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(big2_) NULL_VTABLE
-};
+static const struct normal_encoding big2_encoding_ns
+    = {{VTABLE, 2, 0,
+#  if BYTEORDER == 4321
+        1
+#  else
+        0
+#  endif
+       },
+       {
+#  include "asciitab.h"
+#  include "latin1tab.h"
+       },
+       STANDARD_VTABLE(big2_) NULL_VTABLE};
 
 #endif
 
-static const struct normal_encoding big2_encoding = {
-  { VTABLE, 2, 0,
+static const struct normal_encoding big2_encoding
+    = {{VTABLE, 2, 0,
 #if BYTEORDER == 4321
-  1
+        1
 #else
-  0
+        0
 #endif
-  },
-  {
+       },
+       {
 #define BT_COLON BT_NMSTRT
 #include "asciitab.h"
 #undef BT_COLON
 #include "latin1tab.h"
-  },
-  STANDARD_VTABLE(big2_) NULL_VTABLE
-};
+       },
+       STANDARD_VTABLE(big2_) NULL_VTABLE};
 
 #if BYTEORDER != 1234
 
-#ifdef XML_NS
+#  ifdef XML_NS
 
-static const struct normal_encoding internal_big2_encoding_ns = {
-  { VTABLE, 2, 0, 1 },
-  {
-#include "iasciitab.h"
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(big2_) NULL_VTABLE
-};
+static const struct normal_encoding internal_big2_encoding_ns
+    = {{VTABLE, 2, 0, 1},
+       {
+#    include "iasciitab.h"
+#    include "latin1tab.h"
+       },
+       STANDARD_VTABLE(big2_) NULL_VTABLE};
 
-#endif
+#  endif
 
-static const struct normal_encoding internal_big2_encoding = {
-  { VTABLE, 2, 0, 1 },
-  {
-#define BT_COLON BT_NMSTRT
-#include "iasciitab.h"
-#undef BT_COLON
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(big2_) NULL_VTABLE
-};
+static const struct normal_encoding internal_big2_encoding
+    = {{VTABLE, 2, 0, 1},
+       {
+#  define BT_COLON BT_NMSTRT
+#  include "iasciitab.h"
+#  undef BT_COLON
+#  include "latin1tab.h"
+       },
+       STANDARD_VTABLE(big2_) NULL_VTABLE};
 
 #endif
 
 #undef PREFIX
 
 static int FASTCALL
-streqci(const char *s1, const char *s2)
-{
+streqci(const char *s1, const char *s2) {
   for (;;) {
     char c1 = *s1++;
     char c2 = *s2++;
@@ -1079,22 +1018,21 @@ streqci(const char *s1, const char *s2)
       c2 += ASCII_A - ASCII_a; /* LCOV_EXCL_LINE */
     if (c1 != c2)
       return 0;
-    if (!c1)
+    if (! c1)
       break;
   }
   return 1;
 }
 
 static void PTRCALL
-initUpdatePosition(const ENCODING *UNUSED_P(enc), const char *ptr,
-                   const char *end, POSITION *pos)
-{
+initUpdatePosition(const ENCODING *enc, const char *ptr, const char *end,
+                   POSITION *pos) {
+  UNUSED_P(enc);
   normal_updatePosition(&utf8_encoding.enc, ptr, end, pos);
 }
 
 static int
-toAscii(const ENCODING *enc, const char *ptr, const char *end)
-{
+toAscii(const ENCODING *enc, const char *ptr, const char *end) {
   char buf[1];
   char *p = buf;
   XmlUtf8Convert(enc, &ptr, end, &p, p + 1);
@@ -1105,8 +1043,7 @@ toAscii(const ENCODING *enc, const char *ptr, const char *end)
 }
 
 static int FASTCALL
-isSpace(int c)
-{
+isSpace(int c) {
   switch (c) {
   case 0x20:
   case 0xD:
@@ -1121,21 +1058,16 @@ isSpace(int c)
    followed by name=val.
 */
 static int
-parsePseudoAttribute(const ENCODING *enc,
-                     const char *ptr,
-                     const char *end,
-                     const char **namePtr,
-                     const char **nameEndPtr,
-                     const char **valPtr,
-                     const char **nextTokPtr)
-{
+parsePseudoAttribute(const ENCODING *enc, const char *ptr, const char *end,
+                     const char **namePtr, const char **nameEndPtr,
+                     const char **valPtr, const char **nextTokPtr) {
   int c;
   char open;
   if (ptr == end) {
     *namePtr = NULL;
     return 1;
   }
-  if (!isSpace(toAscii(enc, ptr, end))) {
+  if (! isSpace(toAscii(enc, ptr, end))) {
     *nextTokPtr = ptr;
     return 0;
   }
@@ -1191,12 +1123,9 @@ parsePseudoAttribute(const ENCODING *enc,
     c = toAscii(enc, ptr, end);
     if (c == open)
       break;
-    if (!(ASCII_a <= c && c <= ASCII_z)
-        && !(ASCII_A <= c && c <= ASCII_Z)
-        && !(ASCII_0 <= c && c <= ASCII_9)
-        && c != ASCII_PERIOD
-        && c != ASCII_MINUS
-        && c != ASCII_UNDERSCORE) {
+    if (! (ASCII_a <= c && c <= ASCII_z) && ! (ASCII_A <= c && c <= ASCII_Z)
+        && ! (ASCII_0 <= c && c <= ASCII_9) && c != ASCII_PERIOD
+        && c != ASCII_MINUS && c != ASCII_UNDERSCORE) {
       *nextTokPtr = ptr;
       return 0;
     }
@@ -1205,68 +1134,52 @@ parsePseudoAttribute(const ENCODING *enc,
   return 1;
 }
 
-static const char KW_version[] = {
-  ASCII_v, ASCII_e, ASCII_r, ASCII_s, ASCII_i, ASCII_o, ASCII_n, '\0'
-};
+static const char KW_version[]
+    = {ASCII_v, ASCII_e, ASCII_r, ASCII_s, ASCII_i, ASCII_o, ASCII_n, '\0'};
 
-static const char KW_encoding[] = {
-  ASCII_e, ASCII_n, ASCII_c, ASCII_o, ASCII_d, ASCII_i, ASCII_n, ASCII_g, '\0'
-};
+static const char KW_encoding[] = {ASCII_e, ASCII_n, ASCII_c, ASCII_o, ASCII_d,
+                                   ASCII_i, ASCII_n, ASCII_g, '\0'};
 
-static const char KW_standalone[] = {
-  ASCII_s, ASCII_t, ASCII_a, ASCII_n, ASCII_d, ASCII_a, ASCII_l, ASCII_o,
-  ASCII_n, ASCII_e, '\0'
-};
+static const char KW_standalone[]
+    = {ASCII_s, ASCII_t, ASCII_a, ASCII_n, ASCII_d, ASCII_a,
+       ASCII_l, ASCII_o, ASCII_n, ASCII_e, '\0'};
 
-static const char KW_yes[] = {
-  ASCII_y, ASCII_e, ASCII_s,  '\0'
-};
+static const char KW_yes[] = {ASCII_y, ASCII_e, ASCII_s, '\0'};
 
-static const char KW_no[] = {
-  ASCII_n, ASCII_o,  '\0'
-};
+static const char KW_no[] = {ASCII_n, ASCII_o, '\0'};
 
 static int
-doParseXmlDecl(const ENCODING *(*encodingFinder)(const ENCODING *,
-                                                 const char *,
+doParseXmlDecl(const ENCODING *(*encodingFinder)(const ENCODING *, const char *,
                                                  const char *),
-               int isGeneralTextEntity,
-               const ENCODING *enc,
-               const char *ptr,
-               const char *end,
-               const char **badPtr,
-               const char **versionPtr,
-               const char **versionEndPtr,
-               const char **encodingName,
-               const ENCODING **encoding,
-               int *standalone)
-{
+               int isGeneralTextEntity, const ENCODING *enc, const char *ptr,
+               const char *end, const char **badPtr, const char **versionPtr,
+               const char **versionEndPtr, const char **encodingName,
+               const ENCODING **encoding, int *standalone) {
   const char *val = NULL;
   const char *name = NULL;
   const char *nameEnd = NULL;
   ptr += 5 * enc->minBytesPerChar;
   end -= 2 * enc->minBytesPerChar;
-  if (!parsePseudoAttribute(enc, ptr, end, &name, &nameEnd, &val, &ptr)
-      || !name) {
+  if (! parsePseudoAttribute(enc, ptr, end, &name, &nameEnd, &val, &ptr)
+      || ! name) {
     *badPtr = ptr;
     return 0;
   }
-  if (!XmlNameMatchesAscii(enc, name, nameEnd, KW_version)) {
-    if (!isGeneralTextEntity) {
+  if (! XmlNameMatchesAscii(enc, name, nameEnd, KW_version)) {
+    if (! isGeneralTextEntity) {
       *badPtr = name;
       return 0;
     }
-  }
-  else {
+  } else {
     if (versionPtr)
       *versionPtr = val;
     if (versionEndPtr)
       *versionEndPtr = ptr;
-    if (!parsePseudoAttribute(enc, ptr, end, &name, &nameEnd, &val, &ptr)) {
+    if (! parsePseudoAttribute(enc, ptr, end, &name, &nameEnd, &val, &ptr)) {
       *badPtr = ptr;
       return 0;
     }
-    if (!name) {
+    if (! name) {
       if (isGeneralTextEntity) {
         /* a TextDecl must have an EncodingDecl */
         *badPtr = ptr;
@@ -1277,7 +1190,7 @@ doParseXmlDecl(const ENCODING *(*encodingFinder)(const ENCODING *,
   }
   if (XmlNameMatchesAscii(enc, name, nameEnd, KW_encoding)) {
     int c = toAscii(enc, val, end);
-    if (!(ASCII_a <= c && c <= ASCII_z) && !(ASCII_A <= c && c <= ASCII_Z)) {
+    if (! (ASCII_a <= c && c <= ASCII_z) && ! (ASCII_A <= c && c <= ASCII_Z)) {
       *badPtr = val;
       return 0;
     }
@@ -1285,14 +1198,14 @@ doParseXmlDecl(const ENCODING *(*encodingFinder)(const ENCODING *,
       *encodingName = val;
     if (encoding)
       *encoding = encodingFinder(enc, val, ptr - enc->minBytesPerChar);
-    if (!parsePseudoAttribute(enc, ptr, end, &name, &nameEnd, &val, &ptr)) {
+    if (! parsePseudoAttribute(enc, ptr, end, &name, &nameEnd, &val, &ptr)) {
       *badPtr = ptr;
       return 0;
     }
-    if (!name)
+    if (! name)
       return 1;
   }
-  if (!XmlNameMatchesAscii(enc, name, nameEnd, KW_standalone)
+  if (! XmlNameMatchesAscii(enc, name, nameEnd, KW_standalone)
       || isGeneralTextEntity) {
     *badPtr = name;
     return 0;
@@ -1300,12 +1213,10 @@ doParseXmlDecl(const ENCODING *(*encodingFinder)(const ENCODING *,
   if (XmlNameMatchesAscii(enc, val, ptr - enc->minBytesPerChar, KW_yes)) {
     if (standalone)
       *standalone = 1;
-  }
-  else if (XmlNameMatchesAscii(enc, val, ptr - enc->minBytesPerChar, KW_no)) {
+  } else if (XmlNameMatchesAscii(enc, val, ptr - enc->minBytesPerChar, KW_no)) {
     if (standalone)
       *standalone = 0;
-  }
-  else {
+  } else {
     *badPtr = val;
     return 0;
   }
@@ -1319,11 +1230,16 @@ doParseXmlDecl(const ENCODING *(*encodingFinder)(const ENCODING *,
 }
 
 static int FASTCALL
-checkCharRefNumber(int result)
-{
+checkCharRefNumber(int result) {
   switch (result >> 8) {
-  case 0xD8: case 0xD9: case 0xDA: case 0xDB:
-  case 0xDC: case 0xDD: case 0xDE: case 0xDF:
+  case 0xD8:
+  case 0xD9:
+  case 0xDA:
+  case 0xDB:
+  case 0xDC:
+  case 0xDD:
+  case 0xDE:
+  case 0xDF:
     return -1;
   case 0:
     if (latin1_encoding.type[result] == BT_NONXML)
@@ -1338,8 +1254,7 @@ checkCharRefNumber(int result)
 }
 
 int FASTCALL
-XmlUtf8Encode(int c, char *buf)
-{
+XmlUtf8Encode(int c, char *buf) {
   enum {
     /* minN is minimum legal resulting value for N byte sequence */
     min2 = 0x80,
@@ -1375,8 +1290,7 @@ XmlUtf8Encode(int c, char *buf)
 }
 
 int FASTCALL
-XmlUtf16Encode(int charNum, unsigned short *buf)
-{
+XmlUtf16Encode(int charNum, unsigned short *buf) {
   if (charNum < 0)
     return 0;
   if (charNum < 0x10000) {
@@ -1400,17 +1314,15 @@ struct unknown_encoding {
   char utf8[256][4];
 };
 
-#define AS_UNKNOWN_ENCODING(enc)  ((const struct unknown_encoding *) (enc))
+#define AS_UNKNOWN_ENCODING(enc) ((const struct unknown_encoding *)(enc))
 
 int
-XmlSizeOfUnknownEncoding(void)
-{
+XmlSizeOfUnknownEncoding(void) {
   return sizeof(struct unknown_encoding);
 }
 
 static int PTRFASTCALL
-unknown_isName(const ENCODING *enc, const char *p)
-{
+unknown_isName(const ENCODING *enc, const char *p) {
   const struct unknown_encoding *uenc = AS_UNKNOWN_ENCODING(enc);
   int c = uenc->convert(uenc->userData, p);
   if (c & ~0xFFFF)
@@ -1419,8 +1331,7 @@ unknown_isName(const ENCODING *enc, const char *p)
 }
 
 static int PTRFASTCALL
-unknown_isNmstrt(const ENCODING *enc, const char *p)
-{
+unknown_isNmstrt(const ENCODING *enc, const char *p) {
   const struct unknown_encoding *uenc = AS_UNKNOWN_ENCODING(enc);
   int c = uenc->convert(uenc->userData, p);
   if (c & ~0xFFFF)
@@ -1429,18 +1340,15 @@ unknown_isNmstrt(const ENCODING *enc, const char *p)
 }
 
 static int PTRFASTCALL
-unknown_isInvalid(const ENCODING *enc, const char *p)
-{
+unknown_isInvalid(const ENCODING *enc, const char *p) {
   const struct unknown_encoding *uenc = AS_UNKNOWN_ENCODING(enc);
   int c = uenc->convert(uenc->userData, p);
   return (c & ~0xFFFF) || checkCharRefNumber(c) < 0;
 }
 
 static enum XML_Convert_Result PTRCALL
-unknown_toUtf8(const ENCODING *enc,
-               const char **fromP, const char *fromLim,
-               char **toP, const char *toLim)
-{
+unknown_toUtf8(const ENCODING *enc, const char **fromP, const char *fromLim,
+               char **toP, const char *toLim) {
   const struct unknown_encoding *uenc = AS_UNKNOWN_ENCODING(enc);
   char buf[XML_UTF8_ENCODE_MAX];
   for (;;) {
@@ -1458,8 +1366,7 @@ unknown_toUtf8(const ENCODING *enc,
       utf8 = buf;
       *fromP += (AS_NORMAL_ENCODING(enc)->type[(unsigned char)**fromP]
                  - (BT_LEAD2 - 2));
-    }
-    else {
+    } else {
       if (n > toLim - *toP)
         return XML_CONVERT_OUTPUT_EXHAUSTED;
       (*fromP)++;
@@ -1470,20 +1377,16 @@ unknown_toUtf8(const ENCODING *enc,
 }
 
 static enum XML_Convert_Result PTRCALL
-unknown_toUtf16(const ENCODING *enc,
-                const char **fromP, const char *fromLim,
-                unsigned short **toP, const unsigned short *toLim)
-{
+unknown_toUtf16(const ENCODING *enc, const char **fromP, const char *fromLim,
+                unsigned short **toP, const unsigned short *toLim) {
   const struct unknown_encoding *uenc = AS_UNKNOWN_ENCODING(enc);
   while (*fromP < fromLim && *toP < toLim) {
     unsigned short c = uenc->utf16[(unsigned char)**fromP];
     if (c == 0) {
-      c = (unsigned short)
-          uenc->convert(uenc->userData, *fromP);
+      c = (unsigned short)uenc->convert(uenc->userData, *fromP);
       *fromP += (AS_NORMAL_ENCODING(enc)->type[(unsigned char)**fromP]
                  - (BT_LEAD2 - 2));
-    }
-    else
+    } else
       (*fromP)++;
     *(*toP)++ = c;
   }
@@ -1495,19 +1398,14 @@ unknown_toUtf16(const ENCODING *enc,
 }
 
 ENCODING *
-XmlInitUnknownEncoding(void *mem,
-                       int *table,
-                       CONVERTER convert,
-                       void *userData)
-{
+XmlInitUnknownEncoding(void *mem, int *table, CONVERTER convert,
+                       void *userData) {
   int i;
   struct unknown_encoding *e = (struct unknown_encoding *)mem;
-  for (i = 0; i < (int)sizeof(struct normal_encoding); i++)
-    ((char *)mem)[i] = ((char *)&latin1_encoding)[i];
+  memcpy(mem, &latin1_encoding, sizeof(struct normal_encoding));
   for (i = 0; i < 128; i++)
     if (latin1_encoding.type[i] != BT_OTHER
-        && latin1_encoding.type[i] != BT_NONXML
-        && table[i] != i)
+        && latin1_encoding.type[i] != BT_NONXML && table[i] != i)
       return 0;
   for (i = 0; i < 256; i++) {
     int c = table[i];
@@ -1517,35 +1415,30 @@ XmlInitUnknownEncoding(void *mem,
       e->utf16[i] = 0xFFFF;
       e->utf8[i][0] = 1;
       e->utf8[i][1] = 0;
-    }
-    else if (c < 0) {
+    } else if (c < 0) {
       if (c < -4)
         return 0;
       /* Multi-byte sequences need a converter function */
-      if (!convert)
+      if (! convert)
         return 0;
       e->normal.type[i] = (unsigned char)(BT_LEAD2 - (c + 2));
       e->utf8[i][0] = 0;
       e->utf16[i] = 0;
-    }
-    else if (c < 0x80) {
+    } else if (c < 0x80) {
       if (latin1_encoding.type[c] != BT_OTHER
-          && latin1_encoding.type[c] != BT_NONXML
-          && c != i)
+          && latin1_encoding.type[c] != BT_NONXML && c != i)
         return 0;
       e->normal.type[i] = latin1_encoding.type[c];
       e->utf8[i][0] = 1;
       e->utf8[i][1] = (char)c;
       e->utf16[i] = (unsigned short)(c == 0 ? 0xFFFF : c);
-    }
-    else if (checkCharRefNumber(c) < 0) {
+    } else if (checkCharRefNumber(c) < 0) {
       e->normal.type[i] = BT_NONXML;
       /* This shouldn't really get used. */
       e->utf16[i] = 0xFFFF;
       e->utf8[i][0] = 1;
       e->utf8[i][1] = 0;
-    }
-    else {
+    } else {
       if (c > 0xFFFF)
         return 0;
       if (UCS2_GET_NAMING(nmstrtPages, c >> 8, c & 0xff))
@@ -1590,44 +1483,32 @@ enum {
   NO_ENC
 };
 
-static const char KW_ISO_8859_1[] = {
-  ASCII_I, ASCII_S, ASCII_O, ASCII_MINUS, ASCII_8, ASCII_8, ASCII_5, ASCII_9,
-  ASCII_MINUS, ASCII_1, '\0'
-};
-static const char KW_US_ASCII[] = {
-  ASCII_U, ASCII_S, ASCII_MINUS, ASCII_A, ASCII_S, ASCII_C, ASCII_I, ASCII_I,
-  '\0'
-};
-static const char KW_UTF_8[] =  {
-  ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_8, '\0'
-};
-static const char KW_UTF_16[] = {
-  ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_1, ASCII_6, '\0'
-};
-static const char KW_UTF_16BE[] = {
-  ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_1, ASCII_6, ASCII_B, ASCII_E,
-  '\0'
-};
-static const char KW_UTF_16LE[] = {
-  ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_1, ASCII_6, ASCII_L, ASCII_E,
-  '\0'
-};
+static const char KW_ISO_8859_1[]
+    = {ASCII_I, ASCII_S, ASCII_O,     ASCII_MINUS, ASCII_8, ASCII_8,
+       ASCII_5, ASCII_9, ASCII_MINUS, ASCII_1,     '\0'};
+static const char KW_US_ASCII[]
+    = {ASCII_U, ASCII_S, ASCII_MINUS, ASCII_A, ASCII_S,
+       ASCII_C, ASCII_I, ASCII_I,     '\0'};
+static const char KW_UTF_8[]
+    = {ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_8, '\0'};
+static const char KW_UTF_16[]
+    = {ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_1, ASCII_6, '\0'};
+static const char KW_UTF_16BE[]
+    = {ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_1,
+       ASCII_6, ASCII_B, ASCII_E, '\0'};
+static const char KW_UTF_16LE[]
+    = {ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_1,
+       ASCII_6, ASCII_L, ASCII_E, '\0'};
 
 static int FASTCALL
-getEncodingIndex(const char *name)
-{
-  static const char * const encodingNames[] = {
-    KW_ISO_8859_1,
-    KW_US_ASCII,
-    KW_UTF_8,
-    KW_UTF_16,
-    KW_UTF_16BE,
-    KW_UTF_16LE,
+getEncodingIndex(const char *name) {
+  static const char *const encodingNames[] = {
+      KW_ISO_8859_1, KW_US_ASCII, KW_UTF_8, KW_UTF_16, KW_UTF_16BE, KW_UTF_16LE,
   };
   int i;
   if (name == NULL)
     return NO_ENC;
-  for (i = 0; i < (int)(sizeof(encodingNames)/sizeof(encodingNames[0])); i++)
+  for (i = 0; i < (int)(sizeof(encodingNames) / sizeof(encodingNames[0])); i++)
     if (streqci(name, encodingNames[i]))
       return i;
   return UNKNOWN_ENC;
@@ -1647,15 +1528,9 @@ getEncodingIndex(const char *name)
    XML_PROLOG_STATE otherwise.
 */
 
-
 static int
-initScan(const ENCODING * const *encodingTable,
-         const INIT_ENCODING *enc,
-         int state,
-         const char *ptr,
-         const char *end,
-         const char **nextTokPtr)
-{
+initScan(const ENCODING *const *encodingTable, const INIT_ENCODING *enc,
+         int state, const char *ptr, const char *end, const char **nextTokPtr) {
   const ENCODING **encPtr;
 
   if (ptr >= end)
@@ -1680,20 +1555,17 @@ initScan(const ENCODING * const *encodingTable,
     case 0xFE:
     case 0xFF:
     case 0xEF: /* possibly first byte of UTF-8 BOM */
-      if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC
-          && state == XML_CONTENT_STATE)
+      if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC && state == XML_CONTENT_STATE)
         break;
       /* fall through */
     case 0x00:
     case 0x3C:
       return XML_TOK_PARTIAL;
     }
-  }
-  else {
+  } else {
     switch (((unsigned char)ptr[0] << 8) | (unsigned char)ptr[1]) {
     case 0xFEFF:
-      if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC
-          && state == XML_CONTENT_STATE)
+      if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC && state == XML_CONTENT_STATE)
         break;
       *nextTokPtr = ptr + 2;
       *encPtr = encodingTable[UTF_16BE_ENC];
@@ -1707,8 +1579,7 @@ initScan(const ENCODING * const *encodingTable,
       *encPtr = encodingTable[UTF_16LE_ENC];
       return XmlTok(*encPtr, state, ptr, end, nextTokPtr);
     case 0xFFFE:
-      if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC
-          && state == XML_CONTENT_STATE)
+      if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC && state == XML_CONTENT_STATE)
         break;
       *nextTokPtr = ptr + 2;
       *encPtr = encodingTable[UTF_16LE_ENC];
@@ -1723,8 +1594,8 @@ initScan(const ENCODING * const *encodingTable,
       */
       if (state == XML_CONTENT_STATE) {
         int e = INIT_ENC_INDEX(enc);
-        if (e == ISO_8859_1_ENC || e == UTF_16BE_ENC
-            || e == UTF_16LE_ENC || e == UTF_16_ENC)
+        if (e == ISO_8859_1_ENC || e == UTF_16BE_ENC || e == UTF_16LE_ENC
+            || e == UTF_16_ENC)
           break;
       }
       if (ptr + 2 == end)
@@ -1747,8 +1618,7 @@ initScan(const ENCODING * const *encodingTable,
           break;
         *encPtr = encodingTable[UTF_16BE_ENC];
         return XmlTok(*encPtr, state, ptr, end, nextTokPtr);
-      }
-      else if (ptr[1] == '\0') {
+      } else if (ptr[1] == '\0') {
         /* We could recover here in the case:
             - parsing an external entity
             - second byte is 0
@@ -1770,7 +1640,6 @@ initScan(const ENCODING * const *encodingTable,
   return XmlTok(*encPtr, state, ptr, end, nextTokPtr);
 }
 
-
 #define NS(x) x
 #define ns(x) x
 #define XML_TOK_NS_C
@@ -1781,22 +1650,19 @@ initScan(const ENCODING * const *encodingTable,
 
 #ifdef XML_NS
 
-#define NS(x) x ## NS
-#define ns(x) x ## _ns
+#  define NS(x) x##NS
+#  define ns(x) x##_ns
 
-#define XML_TOK_NS_C
-#include "xmltok_ns.c"
-#undef XML_TOK_NS_C
+#  define XML_TOK_NS_C
+#  include "xmltok_ns.c"
+#  undef XML_TOK_NS_C
 
-#undef NS
-#undef ns
+#  undef NS
+#  undef ns
 
 ENCODING *
-XmlInitUnknownEncodingNS(void *mem,
-                         int *table,
-                         CONVERTER convert,
-                         void *userData)
-{
+XmlInitUnknownEncodingNS(void *mem, int *table, CONVERTER convert,
+                         void *userData) {
   ENCODING *enc = XmlInitUnknownEncoding(mem, table, convert, userData);
   if (enc)
     ((struct normal_encoding *)enc)->type[ASCII_COLON] = BT_COLON;
diff --git a/Modules/expat/xmltok.h b/Modules/expat/xmltok.h
index 50926f38ab323e..2adbf5307befae 100644
--- a/Modules/expat/xmltok.h
+++ b/Modules/expat/xmltok.h
@@ -38,16 +38,18 @@ extern "C" {
 #endif
 
 /* The following token may be returned by XmlContentTok */
-#define XML_TOK_TRAILING_RSQB -5 /* ] or ]] at the end of the scan; might be
-                                    start of illegal ]]> sequence */
+#define XML_TOK_TRAILING_RSQB                                                  \
+  -5 /* ] or ]] at the end of the scan; might be                               \
+        start of illegal ]]> sequence */
 /* The following tokens may be returned by both XmlPrologTok and
    XmlContentTok.
 */
-#define XML_TOK_NONE -4          /* The string to be scanned is empty */
-#define XML_TOK_TRAILING_CR -3   /* A CR at the end of the scan;
-                                    might be part of CRLF sequence */
-#define XML_TOK_PARTIAL_CHAR -2  /* only part of a multibyte sequence */
-#define XML_TOK_PARTIAL -1       /* only part of a token */
+#define XML_TOK_NONE -4 /* The string to be scanned is empty */
+#define XML_TOK_TRAILING_CR                                                    \
+  -3                            /* A CR at the end of the scan;                \
+                                   might be part of CRLF sequence */
+#define XML_TOK_PARTIAL_CHAR -2 /* only part of a multibyte sequence */
+#define XML_TOK_PARTIAL -1      /* only part of a token */
 #define XML_TOK_INVALID 0
 
 /* The following tokens are returned by XmlContentTok; some are also
@@ -62,24 +64,24 @@ extern "C" {
 #define XML_TOK_DATA_NEWLINE 7
 #define XML_TOK_CDATA_SECT_OPEN 8
 #define XML_TOK_ENTITY_REF 9
-#define XML_TOK_CHAR_REF 10               /* numeric character reference */
+#define XML_TOK_CHAR_REF 10 /* numeric character reference */
 
 /* The following tokens may be returned by both XmlPrologTok and
    XmlContentTok.
 */
-#define XML_TOK_PI 11                     /* processing instruction */
-#define XML_TOK_XML_DECL 12               /* XML decl or text decl */
+#define XML_TOK_PI 11       /* processing instruction */
+#define XML_TOK_XML_DECL 12 /* XML decl or text decl */
 #define XML_TOK_COMMENT 13
-#define XML_TOK_BOM 14                    /* Byte order mark */
+#define XML_TOK_BOM 14 /* Byte order mark */
 
 /* The following tokens are returned only by XmlPrologTok */
 #define XML_TOK_PROLOG_S 15
-#define XML_TOK_DECL_OPEN 16              /*  */
+#define XML_TOK_DECL_OPEN 16  /*  */
 #define XML_TOK_NAME 18
 #define XML_TOK_NMTOKEN 19
-#define XML_TOK_POUND_NAME 20             /* #name */
-#define XML_TOK_OR 21                     /* | */
+#define XML_TOK_POUND_NAME 20 /* #name */
+#define XML_TOK_OR 21         /* | */
 #define XML_TOK_PERCENT 22
 #define XML_TOK_OPEN_PAREN 23
 #define XML_TOK_CLOSE_PAREN 24
@@ -90,14 +92,14 @@ extern "C" {
 #define XML_TOK_INSTANCE_START 29
 
 /* The following occur only in element type declarations */
-#define XML_TOK_NAME_QUESTION 30          /* name? */
-#define XML_TOK_NAME_ASTERISK 31          /* name* */
-#define XML_TOK_NAME_PLUS 32              /* name+ */
-#define XML_TOK_COND_SECT_OPEN 33         /*  */
-#define XML_TOK_CLOSE_PAREN_QUESTION 35   /* )? */
-#define XML_TOK_CLOSE_PAREN_ASTERISK 36   /* )* */
-#define XML_TOK_CLOSE_PAREN_PLUS 37       /* )+ */
+#define XML_TOK_NAME_QUESTION 30        /* name? */
+#define XML_TOK_NAME_ASTERISK 31        /* name* */
+#define XML_TOK_NAME_PLUS 32            /* name+ */
+#define XML_TOK_COND_SECT_OPEN 33       /*  */
+#define XML_TOK_CLOSE_PAREN_QUESTION 35 /* )? */
+#define XML_TOK_CLOSE_PAREN_ASTERISK 36 /* )* */
+#define XML_TOK_CLOSE_PAREN_PLUS 37     /* )+ */
 #define XML_TOK_COMMA 38
 
 /* The following token is returned only by XmlAttributeValueTok */
@@ -112,20 +114,20 @@ extern "C" {
 #define XML_TOK_PREFIXED_NAME 41
 
 #ifdef XML_DTD
-#define XML_TOK_IGNORE_SECT 42
+#  define XML_TOK_IGNORE_SECT 42
 #endif /* XML_DTD */
 
 #ifdef XML_DTD
-#define XML_N_STATES 4
+#  define XML_N_STATES 4
 #else /* not XML_DTD */
-#define XML_N_STATES 3
+#  define XML_N_STATES 3
 #endif /* not XML_DTD */
 
 #define XML_PROLOG_STATE 0
 #define XML_CONTENT_STATE 1
 #define XML_CDATA_SECTION_STATE 2
 #ifdef XML_DTD
-#define XML_IGNORE_SECTION_STATE 3
+#  define XML_IGNORE_SECTION_STATE 3
 #endif /* XML_DTD */
 
 #define XML_N_LITERAL_TYPES 2
@@ -153,52 +155,41 @@ typedef struct {
 struct encoding;
 typedef struct encoding ENCODING;
 
-typedef int (PTRCALL *SCANNER)(const ENCODING *,
-                               const char *,
-                               const char *,
-                               const char **);
+typedef int(PTRCALL *SCANNER)(const ENCODING *, const char *, const char *,
+                              const char **);
 
 enum XML_Convert_Result {
   XML_CONVERT_COMPLETED = 0,
   XML_CONVERT_INPUT_INCOMPLETE = 1,
-  XML_CONVERT_OUTPUT_EXHAUSTED = 2  /* and therefore potentially input remaining as well */
+  XML_CONVERT_OUTPUT_EXHAUSTED
+  = 2 /* and therefore potentially input remaining as well */
 };
 
 struct encoding {
   SCANNER scanners[XML_N_STATES];
   SCANNER literalScanners[XML_N_LITERAL_TYPES];
-  int (PTRCALL *nameMatchesAscii)(const ENCODING *,
-                                  const char *,
-                                  const char *,
-                                  const char *);
-  int (PTRFASTCALL *nameLength)(const ENCODING *, const char *);
+  int(PTRCALL *nameMatchesAscii)(const ENCODING *, const char *, const char *,
+                                 const char *);
+  int(PTRFASTCALL *nameLength)(const ENCODING *, const char *);
   const char *(PTRFASTCALL *skipS)(const ENCODING *, const char *);
-  int (PTRCALL *getAtts)(const ENCODING *enc,
-                         const char *ptr,
-                         int attsMax,
-                         ATTRIBUTE *atts);
-  int (PTRFASTCALL *charRefNumber)(const ENCODING *enc, const char *ptr);
-  int (PTRCALL *predefinedEntityName)(const ENCODING *,
-                                      const char *,
-                                      const char *);
-  void (PTRCALL *updatePosition)(const ENCODING *,
-                                 const char *ptr,
-                                 const char *end,
-                                 POSITION *);
-  int (PTRCALL *isPublicId)(const ENCODING *enc,
-                            const char *ptr,
-                            const char *end,
-                            const char **badPtr);
-  enum XML_Convert_Result (PTRCALL *utf8Convert)(const ENCODING *enc,
-                              const char **fromP,
-                              const char *fromLim,
-                              char **toP,
-                              const char *toLim);
-  enum XML_Convert_Result (PTRCALL *utf16Convert)(const ENCODING *enc,
-                               const char **fromP,
-                               const char *fromLim,
-                               unsigned short **toP,
-                               const unsigned short *toLim);
+  int(PTRCALL *getAtts)(const ENCODING *enc, const char *ptr, int attsMax,
+                        ATTRIBUTE *atts);
+  int(PTRFASTCALL *charRefNumber)(const ENCODING *enc, const char *ptr);
+  int(PTRCALL *predefinedEntityName)(const ENCODING *, const char *,
+                                     const char *);
+  void(PTRCALL *updatePosition)(const ENCODING *, const char *ptr,
+                                const char *end, POSITION *);
+  int(PTRCALL *isPublicId)(const ENCODING *enc, const char *ptr,
+                           const char *end, const char **badPtr);
+  enum XML_Convert_Result(PTRCALL *utf8Convert)(const ENCODING *enc,
+                                                const char **fromP,
+                                                const char *fromLim, char **toP,
+                                                const char *toLim);
+  enum XML_Convert_Result(PTRCALL *utf16Convert)(const ENCODING *enc,
+                                                 const char **fromP,
+                                                 const char *fromLim,
+                                                 unsigned short **toP,
+                                                 const unsigned short *toLim);
   int minBytesPerChar;
   char isUtf8;
   char isUtf16;
@@ -225,66 +216,62 @@ struct encoding {
    the prolog outside literals, comments and processing instructions.
 */
 
-
-#define XmlTok(enc, state, ptr, end, nextTokPtr) \
+#define XmlTok(enc, state, ptr, end, nextTokPtr)                               \
   (((enc)->scanners[state])(enc, ptr, end, nextTokPtr))
 
-#define XmlPrologTok(enc, ptr, end, nextTokPtr) \
-   XmlTok(enc, XML_PROLOG_STATE, ptr, end, nextTokPtr)
+#define XmlPrologTok(enc, ptr, end, nextTokPtr)                                \
+  XmlTok(enc, XML_PROLOG_STATE, ptr, end, nextTokPtr)
 
-#define XmlContentTok(enc, ptr, end, nextTokPtr) \
-   XmlTok(enc, XML_CONTENT_STATE, ptr, end, nextTokPtr)
+#define XmlContentTok(enc, ptr, end, nextTokPtr)                               \
+  XmlTok(enc, XML_CONTENT_STATE, ptr, end, nextTokPtr)
 
-#define XmlCdataSectionTok(enc, ptr, end, nextTokPtr) \
-   XmlTok(enc, XML_CDATA_SECTION_STATE, ptr, end, nextTokPtr)
+#define XmlCdataSectionTok(enc, ptr, end, nextTokPtr)                          \
+  XmlTok(enc, XML_CDATA_SECTION_STATE, ptr, end, nextTokPtr)
 
 #ifdef XML_DTD
 
-#define XmlIgnoreSectionTok(enc, ptr, end, nextTokPtr) \
-   XmlTok(enc, XML_IGNORE_SECTION_STATE, ptr, end, nextTokPtr)
+#  define XmlIgnoreSectionTok(enc, ptr, end, nextTokPtr)                       \
+    XmlTok(enc, XML_IGNORE_SECTION_STATE, ptr, end, nextTokPtr)
 
 #endif /* XML_DTD */
 
 /* This is used for performing a 2nd-level tokenization on the content
    of a literal that has already been returned by XmlTok.
 */
-#define XmlLiteralTok(enc, literalType, ptr, end, nextTokPtr) \
+#define XmlLiteralTok(enc, literalType, ptr, end, nextTokPtr)                  \
   (((enc)->literalScanners[literalType])(enc, ptr, end, nextTokPtr))
 
-#define XmlAttributeValueTok(enc, ptr, end, nextTokPtr) \
-   XmlLiteralTok(enc, XML_ATTRIBUTE_VALUE_LITERAL, ptr, end, nextTokPtr)
+#define XmlAttributeValueTok(enc, ptr, end, nextTokPtr)                        \
+  XmlLiteralTok(enc, XML_ATTRIBUTE_VALUE_LITERAL, ptr, end, nextTokPtr)
 
-#define XmlEntityValueTok(enc, ptr, end, nextTokPtr) \
-   XmlLiteralTok(enc, XML_ENTITY_VALUE_LITERAL, ptr, end, nextTokPtr)
+#define XmlEntityValueTok(enc, ptr, end, nextTokPtr)                           \
+  XmlLiteralTok(enc, XML_ENTITY_VALUE_LITERAL, ptr, end, nextTokPtr)
 
-#define XmlNameMatchesAscii(enc, ptr1, end1, ptr2) \
+#define XmlNameMatchesAscii(enc, ptr1, end1, ptr2)                             \
   (((enc)->nameMatchesAscii)(enc, ptr1, end1, ptr2))
 
-#define XmlNameLength(enc, ptr) \
-  (((enc)->nameLength)(enc, ptr))
+#define XmlNameLength(enc, ptr) (((enc)->nameLength)(enc, ptr))
 
-#define XmlSkipS(enc, ptr) \
-  (((enc)->skipS)(enc, ptr))
+#define XmlSkipS(enc, ptr) (((enc)->skipS)(enc, ptr))
 
-#define XmlGetAttributes(enc, ptr, attsMax, atts) \
+#define XmlGetAttributes(enc, ptr, attsMax, atts)                              \
   (((enc)->getAtts)(enc, ptr, attsMax, atts))
 
-#define XmlCharRefNumber(enc, ptr) \
-  (((enc)->charRefNumber)(enc, ptr))
+#define XmlCharRefNumber(enc, ptr) (((enc)->charRefNumber)(enc, ptr))
 
-#define XmlPredefinedEntityName(enc, ptr, end) \
+#define XmlPredefinedEntityName(enc, ptr, end)                                 \
   (((enc)->predefinedEntityName)(enc, ptr, end))
 
-#define XmlUpdatePosition(enc, ptr, end, pos) \
+#define XmlUpdatePosition(enc, ptr, end, pos)                                  \
   (((enc)->updatePosition)(enc, ptr, end, pos))
 
-#define XmlIsPublicId(enc, ptr, end, badPtr) \
+#define XmlIsPublicId(enc, ptr, end, badPtr)                                   \
   (((enc)->isPublicId)(enc, ptr, end, badPtr))
 
-#define XmlUtf8Convert(enc, fromP, fromLim, toP, toLim) \
+#define XmlUtf8Convert(enc, fromP, fromLim, toP, toLim)                        \
   (((enc)->utf8Convert)(enc, fromP, fromLim, toP, toLim))
 
-#define XmlUtf16Convert(enc, fromP, fromLim, toP, toLim) \
+#define XmlUtf16Convert(enc, fromP, fromLim, toP, toLim)                       \
   (((enc)->utf16Convert)(enc, fromP, fromLim, toP, toLim))
 
 typedef struct {
@@ -292,16 +279,11 @@ typedef struct {
   const ENCODING **encPtr;
 } INIT_ENCODING;
 
-int XmlParseXmlDecl(int isGeneralTextEntity,
-                    const ENCODING *enc,
-                    const char *ptr,
-                    const char *end,
-                    const char **badPtr,
-                    const char **versionPtr,
-                    const char **versionEndPtr,
+int XmlParseXmlDecl(int isGeneralTextEntity, const ENCODING *enc,
+                    const char *ptr, const char *end, const char **badPtr,
+                    const char **versionPtr, const char **versionEndPtr,
                     const char **encodingNamePtr,
-                    const ENCODING **namedEncodingPtr,
-                    int *standalonePtr);
+                    const ENCODING **namedEncodingPtr, int *standalonePtr);
 
 int XmlInitEncoding(INIT_ENCODING *, const ENCODING **, const char *name);
 const ENCODING *XmlGetUtf8InternalEncoding(void);
@@ -310,34 +292,22 @@ int FASTCALL XmlUtf8Encode(int charNumber, char *buf);
 int FASTCALL XmlUtf16Encode(int charNumber, unsigned short *buf);
 int XmlSizeOfUnknownEncoding(void);
 
+typedef int(XMLCALL *CONVERTER)(void *userData, const char *p);
 
-typedef int (XMLCALL *CONVERTER) (void *userData, const char *p);
-
-ENCODING *
-XmlInitUnknownEncoding(void *mem,
-                       int *table,
-                       CONVERTER convert,
-                       void *userData);
+ENCODING *XmlInitUnknownEncoding(void *mem, int *table, CONVERTER convert,
+                                 void *userData);
 
-int XmlParseXmlDeclNS(int isGeneralTextEntity,
-                      const ENCODING *enc,
-                      const char *ptr,
-                      const char *end,
-                      const char **badPtr,
-                      const char **versionPtr,
-                      const char **versionEndPtr,
+int XmlParseXmlDeclNS(int isGeneralTextEntity, const ENCODING *enc,
+                      const char *ptr, const char *end, const char **badPtr,
+                      const char **versionPtr, const char **versionEndPtr,
                       const char **encodingNamePtr,
-                      const ENCODING **namedEncodingPtr,
-                      int *standalonePtr);
+                      const ENCODING **namedEncodingPtr, int *standalonePtr);
 
 int XmlInitEncodingNS(INIT_ENCODING *, const ENCODING **, const char *name);
 const ENCODING *XmlGetUtf8InternalEncodingNS(void);
 const ENCODING *XmlGetUtf16InternalEncodingNS(void);
-ENCODING *
-XmlInitUnknownEncodingNS(void *mem,
-                         int *table,
-                         CONVERTER convert,
-                         void *userData);
+ENCODING *XmlInitUnknownEncodingNS(void *mem, int *table, CONVERTER convert,
+                                   void *userData);
 #ifdef __cplusplus
 }
 #endif
diff --git a/Modules/expat/xmltok_impl.c b/Modules/expat/xmltok_impl.c
index 4d9ae7dc3896b8..c209221cd79d13 100644
--- a/Modules/expat/xmltok_impl.c
+++ b/Modules/expat/xmltok_impl.c
@@ -32,130 +32,124 @@
 
 #ifdef XML_TOK_IMPL_C
 
-#ifndef IS_INVALID_CHAR
-#define IS_INVALID_CHAR(enc, ptr, n) (0)
-#endif
-
-#define INVALID_LEAD_CASE(n, ptr, nextTokPtr) \
-    case BT_LEAD ## n: \
-      if (end - ptr < n) \
-        return XML_TOK_PARTIAL_CHAR; \
-      if (IS_INVALID_CHAR(enc, ptr, n)) { \
-        *(nextTokPtr) = (ptr); \
-        return XML_TOK_INVALID; \
-      } \
-      ptr += n; \
-      break;
+#  ifndef IS_INVALID_CHAR
+#    define IS_INVALID_CHAR(enc, ptr, n) (0)
+#  endif
+
+#  define INVALID_LEAD_CASE(n, ptr, nextTokPtr)                                \
+  case BT_LEAD##n:                                                             \
+    if (end - ptr < n)                                                         \
+      return XML_TOK_PARTIAL_CHAR;                                             \
+    if (IS_INVALID_CHAR(enc, ptr, n)) {                                        \
+      *(nextTokPtr) = (ptr);                                                   \
+      return XML_TOK_INVALID;                                                  \
+    }                                                                          \
+    ptr += n;                                                                  \
+    break;
 
-#define INVALID_CASES(ptr, nextTokPtr) \
-  INVALID_LEAD_CASE(2, ptr, nextTokPtr) \
-  INVALID_LEAD_CASE(3, ptr, nextTokPtr) \
-  INVALID_LEAD_CASE(4, ptr, nextTokPtr) \
-  case BT_NONXML: \
-  case BT_MALFORM: \
-  case BT_TRAIL: \
-    *(nextTokPtr) = (ptr); \
+#  define INVALID_CASES(ptr, nextTokPtr)                                       \
+    INVALID_LEAD_CASE(2, ptr, nextTokPtr)                                      \
+    INVALID_LEAD_CASE(3, ptr, nextTokPtr)                                      \
+    INVALID_LEAD_CASE(4, ptr, nextTokPtr)                                      \
+  case BT_NONXML:                                                              \
+  case BT_MALFORM:                                                             \
+  case BT_TRAIL:                                                               \
+    *(nextTokPtr) = (ptr);                                                     \
     return XML_TOK_INVALID;
 
-#define CHECK_NAME_CASE(n, enc, ptr, end, nextTokPtr) \
-   case BT_LEAD ## n: \
-     if (end - ptr < n) \
-       return XML_TOK_PARTIAL_CHAR; \
-     if (!IS_NAME_CHAR(enc, ptr, n)) { \
-       *nextTokPtr = ptr; \
-       return XML_TOK_INVALID; \
-     } \
-     ptr += n; \
-     break;
-
-#define CHECK_NAME_CASES(enc, ptr, end, nextTokPtr) \
-  case BT_NONASCII: \
-    if (!IS_NAME_CHAR_MINBPC(enc, ptr)) { \
-      *nextTokPtr = ptr; \
-      return XML_TOK_INVALID; \
-    } \
-    /* fall through */ \
-  case BT_NMSTRT: \
-  case BT_HEX: \
-  case BT_DIGIT: \
-  case BT_NAME: \
-  case BT_MINUS: \
-    ptr += MINBPC(enc); \
-    break; \
-  CHECK_NAME_CASE(2, enc, ptr, end, nextTokPtr) \
-  CHECK_NAME_CASE(3, enc, ptr, end, nextTokPtr) \
-  CHECK_NAME_CASE(4, enc, ptr, end, nextTokPtr)
-
-#define CHECK_NMSTRT_CASE(n, enc, ptr, end, nextTokPtr) \
-   case BT_LEAD ## n: \
-     if (end - ptr < n) \
-       return XML_TOK_PARTIAL_CHAR; \
-     if (!IS_NMSTRT_CHAR(enc, ptr, n)) { \
-       *nextTokPtr = ptr; \
-       return XML_TOK_INVALID; \
-     } \
-     ptr += n; \
-     break;
-
-#define CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr) \
-  case BT_NONASCII: \
-    if (!IS_NMSTRT_CHAR_MINBPC(enc, ptr)) { \
-      *nextTokPtr = ptr; \
-      return XML_TOK_INVALID; \
-    } \
-    /* fall through */ \
-  case BT_NMSTRT: \
-  case BT_HEX: \
-    ptr += MINBPC(enc); \
-    break; \
-  CHECK_NMSTRT_CASE(2, enc, ptr, end, nextTokPtr) \
-  CHECK_NMSTRT_CASE(3, enc, ptr, end, nextTokPtr) \
-  CHECK_NMSTRT_CASE(4, enc, ptr, end, nextTokPtr)
-
-#ifndef PREFIX
-#define PREFIX(ident) ident
-#endif
-
-
-#define HAS_CHARS(enc, ptr, end, count) \
-    (end - ptr >= count * MINBPC(enc))
+#  define CHECK_NAME_CASE(n, enc, ptr, end, nextTokPtr)                        \
+  case BT_LEAD##n:                                                             \
+    if (end - ptr < n)                                                         \
+      return XML_TOK_PARTIAL_CHAR;                                             \
+    if (! IS_NAME_CHAR(enc, ptr, n)) {                                         \
+      *nextTokPtr = ptr;                                                       \
+      return XML_TOK_INVALID;                                                  \
+    }                                                                          \
+    ptr += n;                                                                  \
+    break;
 
-#define HAS_CHAR(enc, ptr, end) \
-    HAS_CHARS(enc, ptr, end, 1)
+#  define CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)                          \
+  case BT_NONASCII:                                                            \
+    if (! IS_NAME_CHAR_MINBPC(enc, ptr)) {                                     \
+      *nextTokPtr = ptr;                                                       \
+      return XML_TOK_INVALID;                                                  \
+    }                                                                          \
+    /* fall through */                                                         \
+  case BT_NMSTRT:                                                              \
+  case BT_HEX:                                                                 \
+  case BT_DIGIT:                                                               \
+  case BT_NAME:                                                                \
+  case BT_MINUS:                                                               \
+    ptr += MINBPC(enc);                                                        \
+    break;                                                                     \
+    CHECK_NAME_CASE(2, enc, ptr, end, nextTokPtr)                              \
+    CHECK_NAME_CASE(3, enc, ptr, end, nextTokPtr)                              \
+    CHECK_NAME_CASE(4, enc, ptr, end, nextTokPtr)
+
+#  define CHECK_NMSTRT_CASE(n, enc, ptr, end, nextTokPtr)                      \
+  case BT_LEAD##n:                                                             \
+    if (end - ptr < n)                                                         \
+      return XML_TOK_PARTIAL_CHAR;                                             \
+    if (! IS_NMSTRT_CHAR(enc, ptr, n)) {                                       \
+      *nextTokPtr = ptr;                                                       \
+      return XML_TOK_INVALID;                                                  \
+    }                                                                          \
+    ptr += n;                                                                  \
+    break;
 
-#define REQUIRE_CHARS(enc, ptr, end, count) \
-    { \
-      if (! HAS_CHARS(enc, ptr, end, count)) { \
-        return XML_TOK_PARTIAL; \
-      } \
+#  define CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)                        \
+  case BT_NONASCII:                                                            \
+    if (! IS_NMSTRT_CHAR_MINBPC(enc, ptr)) {                                   \
+      *nextTokPtr = ptr;                                                       \
+      return XML_TOK_INVALID;                                                  \
+    }                                                                          \
+    /* fall through */                                                         \
+  case BT_NMSTRT:                                                              \
+  case BT_HEX:                                                                 \
+    ptr += MINBPC(enc);                                                        \
+    break;                                                                     \
+    CHECK_NMSTRT_CASE(2, enc, ptr, end, nextTokPtr)                            \
+    CHECK_NMSTRT_CASE(3, enc, ptr, end, nextTokPtr)                            \
+    CHECK_NMSTRT_CASE(4, enc, ptr, end, nextTokPtr)
+
+#  ifndef PREFIX
+#    define PREFIX(ident) ident
+#  endif
+
+#  define HAS_CHARS(enc, ptr, end, count) (end - ptr >= count * MINBPC(enc))
+
+#  define HAS_CHAR(enc, ptr, end) HAS_CHARS(enc, ptr, end, 1)
+
+#  define REQUIRE_CHARS(enc, ptr, end, count)                                  \
+    {                                                                          \
+      if (! HAS_CHARS(enc, ptr, end, count)) {                                 \
+        return XML_TOK_PARTIAL;                                                \
+      }                                                                        \
     }
 
-#define REQUIRE_CHAR(enc, ptr, end) \
-    REQUIRE_CHARS(enc, ptr, end, 1)
-
+#  define REQUIRE_CHAR(enc, ptr, end) REQUIRE_CHARS(enc, ptr, end, 1)
 
 /* ptr points to character following " */
       switch (BYTE_TYPE(enc, ptr + MINBPC(enc))) {
-      case BT_S: case BT_CR: case BT_LF: case BT_PERCNT:
+      case BT_S:
+      case BT_CR:
+      case BT_LF:
+      case BT_PERCNT:
         *nextTokPtr = ptr;
         return XML_TOK_INVALID;
       }
       /* fall through */
-    case BT_S: case BT_CR: case BT_LF:
+    case BT_S:
+    case BT_CR:
+    case BT_LF:
       *nextTokPtr = ptr;
       return XML_TOK_DECL_OPEN;
     case BT_NMSTRT:
@@ -220,12 +218,12 @@ PREFIX(scanDecl)(const ENCODING *enc, const char *ptr,
 }
 
 static int PTRCALL
-PREFIX(checkPiTarget)(const ENCODING *UNUSED_P(enc), const char *ptr,
-                      const char *end, int *tokPtr)
-{
+PREFIX(checkPiTarget)(const ENCODING *enc, const char *ptr, const char *end,
+                      int *tokPtr) {
   int upper = 0;
+  UNUSED_P(enc);
   *tokPtr = XML_TOK_PI;
-  if (end - ptr != MINBPC(enc)*3)
+  if (end - ptr != MINBPC(enc) * 3)
     return 1;
   switch (BYTE_TO_ASCII(enc, ptr)) {
   case ASCII_x:
@@ -265,30 +263,31 @@ PREFIX(checkPiTarget)(const ENCODING *UNUSED_P(enc), const char *ptr,
 /* ptr points to character following "= end)
     return XML_TOK_NONE;
   if (MINBPC(enc) > 1) {
@@ -361,11 +359,11 @@ PREFIX(cdataSectionTok)(const ENCODING *enc, const char *ptr,
   case BT_RSQB:
     ptr += MINBPC(enc);
     REQUIRE_CHAR(enc, ptr, end);
-    if (!CHAR_MATCHES(enc, ptr, ASCII_RSQB))
+    if (! CHAR_MATCHES(enc, ptr, ASCII_RSQB))
       break;
     ptr += MINBPC(enc);
     REQUIRE_CHAR(enc, ptr, end);
-    if (!CHAR_MATCHES(enc, ptr, ASCII_GT)) {
+    if (! CHAR_MATCHES(enc, ptr, ASCII_GT)) {
       ptr -= MINBPC(enc);
       break;
     }
@@ -381,23 +379,25 @@ PREFIX(cdataSectionTok)(const ENCODING *enc, const char *ptr,
   case BT_LF:
     *nextTokPtr = ptr + MINBPC(enc);
     return XML_TOK_DATA_NEWLINE;
-  INVALID_CASES(ptr, nextTokPtr)
+    INVALID_CASES(ptr, nextTokPtr)
   default:
     ptr += MINBPC(enc);
     break;
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: \
-      if (end - ptr < n || IS_INVALID_CHAR(enc, ptr, n)) { \
-        *nextTokPtr = ptr; \
-        return XML_TOK_DATA_CHARS; \
-      } \
-      ptr += n; \
-      break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    if (end - ptr < n || IS_INVALID_CHAR(enc, ptr, n)) {                       \
+      *nextTokPtr = ptr;                                                       \
+      return XML_TOK_DATA_CHARS;                                               \
+    }                                                                          \
+    ptr += n;                                                                  \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_NONXML:
     case BT_MALFORM:
     case BT_TRAIL:
@@ -418,23 +418,26 @@ PREFIX(cdataSectionTok)(const ENCODING *enc, const char *ptr,
 /* ptr points to character following "= end)
     return XML_TOK_NONE;
   if (MINBPC(enc) > 1) {
@@ -842,48 +843,50 @@ PREFIX(contentTok)(const ENCODING *enc, const char *ptr, const char *end,
     ptr += MINBPC(enc);
     if (! HAS_CHAR(enc, ptr, end))
       return XML_TOK_TRAILING_RSQB;
-    if (!CHAR_MATCHES(enc, ptr, ASCII_RSQB))
+    if (! CHAR_MATCHES(enc, ptr, ASCII_RSQB))
       break;
     ptr += MINBPC(enc);
     if (! HAS_CHAR(enc, ptr, end))
       return XML_TOK_TRAILING_RSQB;
-    if (!CHAR_MATCHES(enc, ptr, ASCII_GT)) {
+    if (! CHAR_MATCHES(enc, ptr, ASCII_GT)) {
       ptr -= MINBPC(enc);
       break;
     }
     *nextTokPtr = ptr;
     return XML_TOK_INVALID;
-  INVALID_CASES(ptr, nextTokPtr)
+    INVALID_CASES(ptr, nextTokPtr)
   default:
     ptr += MINBPC(enc);
     break;
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: \
-      if (end - ptr < n || IS_INVALID_CHAR(enc, ptr, n)) { \
-        *nextTokPtr = ptr; \
-        return XML_TOK_DATA_CHARS; \
-      } \
-      ptr += n; \
-      break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    if (end - ptr < n || IS_INVALID_CHAR(enc, ptr, n)) {                       \
+      *nextTokPtr = ptr;                                                       \
+      return XML_TOK_DATA_CHARS;                                               \
+    }                                                                          \
+    ptr += n;                                                                  \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_RSQB:
       if (HAS_CHARS(enc, ptr, end, 2)) {
-         if (!CHAR_MATCHES(enc, ptr + MINBPC(enc), ASCII_RSQB)) {
-           ptr += MINBPC(enc);
-           break;
-         }
-         if (HAS_CHARS(enc, ptr, end, 3)) {
-           if (!CHAR_MATCHES(enc, ptr + 2*MINBPC(enc), ASCII_GT)) {
-             ptr += MINBPC(enc);
-             break;
-           }
-           *nextTokPtr = ptr + 2*MINBPC(enc);
-           return XML_TOK_INVALID;
-         }
+        if (! CHAR_MATCHES(enc, ptr + MINBPC(enc), ASCII_RSQB)) {
+          ptr += MINBPC(enc);
+          break;
+        }
+        if (HAS_CHARS(enc, ptr, end, 3)) {
+          if (! CHAR_MATCHES(enc, ptr + 2 * MINBPC(enc), ASCII_GT)) {
+            ptr += MINBPC(enc);
+            break;
+          }
+          *nextTokPtr = ptr + 2 * MINBPC(enc);
+          return XML_TOK_INVALID;
+        }
       }
       /* fall through */
     case BT_AMP:
@@ -908,12 +911,14 @@ PREFIX(contentTok)(const ENCODING *enc, const char *ptr, const char *end,
 
 static int PTRCALL
 PREFIX(scanPercent)(const ENCODING *enc, const char *ptr, const char *end,
-                    const char **nextTokPtr)
-{
+                    const char **nextTokPtr) {
   REQUIRE_CHAR(enc, ptr, end);
   switch (BYTE_TYPE(enc, ptr)) {
-  CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
-  case BT_S: case BT_LF: case BT_CR: case BT_PERCNT:
+    CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
+  case BT_S:
+  case BT_LF:
+  case BT_CR:
+  case BT_PERCNT:
     *nextTokPtr = ptr;
     return XML_TOK_PERCENT;
   default:
@@ -922,7 +927,7 @@ PREFIX(scanPercent)(const ENCODING *enc, const char *ptr, const char *end,
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-    CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
+      CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
     case BT_SEMI:
       *nextTokPtr = ptr + MINBPC(enc);
       return XML_TOK_PARAM_ENTITY_REF;
@@ -936,20 +941,24 @@ PREFIX(scanPercent)(const ENCODING *enc, const char *ptr, const char *end,
 
 static int PTRCALL
 PREFIX(scanPoundName)(const ENCODING *enc, const char *ptr, const char *end,
-                      const char **nextTokPtr)
-{
+                      const char **nextTokPtr) {
   REQUIRE_CHAR(enc, ptr, end);
   switch (BYTE_TYPE(enc, ptr)) {
-  CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
+    CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
   default:
     *nextTokPtr = ptr;
     return XML_TOK_INVALID;
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-    CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
-    case BT_CR: case BT_LF: case BT_S:
-    case BT_RPAR: case BT_GT: case BT_PERCNT: case BT_VERBAR:
+      CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
+    case BT_CR:
+    case BT_LF:
+    case BT_S:
+    case BT_RPAR:
+    case BT_GT:
+    case BT_PERCNT:
+    case BT_VERBAR:
       *nextTokPtr = ptr;
       return XML_TOK_POUND_NAME;
     default:
@@ -961,14 +970,12 @@ PREFIX(scanPoundName)(const ENCODING *enc, const char *ptr, const char *end,
 }
 
 static int PTRCALL
-PREFIX(scanLit)(int open, const ENCODING *enc,
-                const char *ptr, const char *end,
-                const char **nextTokPtr)
-{
+PREFIX(scanLit)(int open, const ENCODING *enc, const char *ptr, const char *end,
+                const char **nextTokPtr) {
   while (HAS_CHAR(enc, ptr, end)) {
     int t = BYTE_TYPE(enc, ptr);
     switch (t) {
-    INVALID_CASES(ptr, nextTokPtr)
+      INVALID_CASES(ptr, nextTokPtr)
     case BT_QUOT:
     case BT_APOS:
       ptr += MINBPC(enc);
@@ -978,8 +985,12 @@ PREFIX(scanLit)(int open, const ENCODING *enc,
         return -XML_TOK_LITERAL;
       *nextTokPtr = ptr;
       switch (BYTE_TYPE(enc, ptr)) {
-      case BT_S: case BT_CR: case BT_LF:
-      case BT_GT: case BT_PERCNT: case BT_LSQB:
+      case BT_S:
+      case BT_CR:
+      case BT_LF:
+      case BT_GT:
+      case BT_PERCNT:
+      case BT_LSQB:
         return XML_TOK_LITERAL;
       default:
         return XML_TOK_INVALID;
@@ -994,8 +1005,7 @@ PREFIX(scanLit)(int open, const ENCODING *enc,
 
 static int PTRCALL
 PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
-                  const char **nextTokPtr)
-{
+                  const char **nextTokPtr) {
   int tok;
   if (ptr >= end)
     return XML_TOK_NONE;
@@ -1013,27 +1023,26 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
     return PREFIX(scanLit)(BT_QUOT, enc, ptr + MINBPC(enc), end, nextTokPtr);
   case BT_APOS:
     return PREFIX(scanLit)(BT_APOS, enc, ptr + MINBPC(enc), end, nextTokPtr);
-  case BT_LT:
-    {
-      ptr += MINBPC(enc);
-      REQUIRE_CHAR(enc, ptr, end);
-      switch (BYTE_TYPE(enc, ptr)) {
-      case BT_EXCL:
-        return PREFIX(scanDecl)(enc, ptr + MINBPC(enc), end, nextTokPtr);
-      case BT_QUEST:
-        return PREFIX(scanPi)(enc, ptr + MINBPC(enc), end, nextTokPtr);
-      case BT_NMSTRT:
-      case BT_HEX:
-      case BT_NONASCII:
-      case BT_LEAD2:
-      case BT_LEAD3:
-      case BT_LEAD4:
-        *nextTokPtr = ptr - MINBPC(enc);
-        return XML_TOK_INSTANCE_START;
-      }
-      *nextTokPtr = ptr;
-      return XML_TOK_INVALID;
+  case BT_LT: {
+    ptr += MINBPC(enc);
+    REQUIRE_CHAR(enc, ptr, end);
+    switch (BYTE_TYPE(enc, ptr)) {
+    case BT_EXCL:
+      return PREFIX(scanDecl)(enc, ptr + MINBPC(enc), end, nextTokPtr);
+    case BT_QUEST:
+      return PREFIX(scanPi)(enc, ptr + MINBPC(enc), end, nextTokPtr);
+    case BT_NMSTRT:
+    case BT_HEX:
+    case BT_NONASCII:
+    case BT_LEAD2:
+    case BT_LEAD3:
+    case BT_LEAD4:
+      *nextTokPtr = ptr - MINBPC(enc);
+      return XML_TOK_INSTANCE_START;
     }
+    *nextTokPtr = ptr;
+    return XML_TOK_INVALID;
+  }
   case BT_CR:
     if (ptr + MINBPC(enc) == end) {
       *nextTokPtr = end;
@@ -1041,13 +1050,15 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
       return -XML_TOK_PROLOG_S;
     }
     /* fall through */
-  case BT_S: case BT_LF:
+  case BT_S:
+  case BT_LF:
     for (;;) {
       ptr += MINBPC(enc);
       if (! HAS_CHAR(enc, ptr, end))
         break;
       switch (BYTE_TYPE(enc, ptr)) {
-      case BT_S: case BT_LF:
+      case BT_S:
+      case BT_LF:
         break;
       case BT_CR:
         /* don't split CR/LF pair */
@@ -1076,7 +1087,7 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
     if (CHAR_MATCHES(enc, ptr, ASCII_RSQB)) {
       REQUIRE_CHARS(enc, ptr, end, 2);
       if (CHAR_MATCHES(enc, ptr + MINBPC(enc), ASCII_GT)) {
-        *nextTokPtr = ptr + 2*MINBPC(enc);
+        *nextTokPtr = ptr + 2 * MINBPC(enc);
         return XML_TOK_COND_SECT_CLOSE;
       }
     }
@@ -1099,8 +1110,12 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
     case BT_PLUS:
       *nextTokPtr = ptr + MINBPC(enc);
       return XML_TOK_CLOSE_PAREN_PLUS;
-    case BT_CR: case BT_LF: case BT_S:
-    case BT_GT: case BT_COMMA: case BT_VERBAR:
+    case BT_CR:
+    case BT_LF:
+    case BT_S:
+    case BT_GT:
+    case BT_COMMA:
+    case BT_VERBAR:
     case BT_RPAR:
       *nextTokPtr = ptr;
       return XML_TOK_CLOSE_PAREN;
@@ -1115,24 +1130,26 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
     return XML_TOK_DECL_CLOSE;
   case BT_NUM:
     return PREFIX(scanPoundName)(enc, ptr + MINBPC(enc), end, nextTokPtr);
-#define LEAD_CASE(n) \
-  case BT_LEAD ## n: \
-    if (end - ptr < n) \
-      return XML_TOK_PARTIAL_CHAR; \
-    if (IS_NMSTRT_CHAR(enc, ptr, n)) { \
-      ptr += n; \
-      tok = XML_TOK_NAME; \
-      break; \
-    } \
-    if (IS_NAME_CHAR(enc, ptr, n)) { \
-      ptr += n; \
-      tok = XML_TOK_NMTOKEN; \
-      break; \
-    } \
-    *nextTokPtr = ptr; \
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    if (end - ptr < n)                                                         \
+      return XML_TOK_PARTIAL_CHAR;                                             \
+    if (IS_NMSTRT_CHAR(enc, ptr, n)) {                                         \
+      ptr += n;                                                                \
+      tok = XML_TOK_NAME;                                                      \
+      break;                                                                   \
+    }                                                                          \
+    if (IS_NAME_CHAR(enc, ptr, n)) {                                           \
+      ptr += n;                                                                \
+      tok = XML_TOK_NMTOKEN;                                                   \
+      break;                                                                   \
+    }                                                                          \
+    *nextTokPtr = ptr;                                                         \
     return XML_TOK_INVALID;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+    LEAD_CASE(2)
+    LEAD_CASE(3)
+    LEAD_CASE(4)
+#  undef LEAD_CASE
   case BT_NMSTRT:
   case BT_HEX:
     tok = XML_TOK_NAME;
@@ -1141,9 +1158,9 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
   case BT_DIGIT:
   case BT_NAME:
   case BT_MINUS:
-#ifdef XML_NS
+#  ifdef XML_NS
   case BT_COLON:
-#endif
+#  endif
     tok = XML_TOK_NMTOKEN;
     ptr += MINBPC(enc);
     break;
@@ -1165,13 +1182,19 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-    CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
-    case BT_GT: case BT_RPAR: case BT_COMMA:
-    case BT_VERBAR: case BT_LSQB: case BT_PERCNT:
-    case BT_S: case BT_CR: case BT_LF:
+      CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
+    case BT_GT:
+    case BT_RPAR:
+    case BT_COMMA:
+    case BT_VERBAR:
+    case BT_LSQB:
+    case BT_PERCNT:
+    case BT_S:
+    case BT_CR:
+    case BT_LF:
       *nextTokPtr = ptr;
       return tok;
-#ifdef XML_NS
+#  ifdef XML_NS
     case BT_COLON:
       ptr += MINBPC(enc);
       switch (tok) {
@@ -1179,7 +1202,7 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
         REQUIRE_CHAR(enc, ptr, end);
         tok = XML_TOK_PREFIXED_NAME;
         switch (BYTE_TYPE(enc, ptr)) {
-        CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
+          CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
         default:
           tok = XML_TOK_NMTOKEN;
           break;
@@ -1190,23 +1213,23 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
         break;
       }
       break;
-#endif
+#  endif
     case BT_PLUS:
-      if (tok == XML_TOK_NMTOKEN)  {
+      if (tok == XML_TOK_NMTOKEN) {
         *nextTokPtr = ptr;
         return XML_TOK_INVALID;
       }
       *nextTokPtr = ptr + MINBPC(enc);
       return XML_TOK_NAME_PLUS;
     case BT_AST:
-      if (tok == XML_TOK_NMTOKEN)  {
+      if (tok == XML_TOK_NMTOKEN) {
         *nextTokPtr = ptr;
         return XML_TOK_INVALID;
       }
       *nextTokPtr = ptr + MINBPC(enc);
       return XML_TOK_NAME_ASTERISK;
     case BT_QUEST:
-      if (tok == XML_TOK_NMTOKEN)  {
+      if (tok == XML_TOK_NMTOKEN) {
         *nextTokPtr = ptr;
         return XML_TOK_INVALID;
       }
@@ -1221,9 +1244,8 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
 }
 
 static int PTRCALL
-PREFIX(attributeValueTok)(const ENCODING *enc, const char *ptr,
-                          const char *end, const char **nextTokPtr)
-{
+PREFIX(attributeValueTok)(const ENCODING *enc, const char *ptr, const char *end,
+                          const char **nextTokPtr) {
   const char *start;
   if (ptr >= end)
     return XML_TOK_NONE;
@@ -1238,10 +1260,14 @@ PREFIX(attributeValueTok)(const ENCODING *enc, const char *ptr,
   start = ptr;
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: ptr += n; break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    ptr += n;                                                                  \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_AMP:
       if (ptr == start)
         return PREFIX(scanRef)(enc, ptr + MINBPC(enc), end, nextTokPtr);
@@ -1287,9 +1313,8 @@ PREFIX(attributeValueTok)(const ENCODING *enc, const char *ptr,
 }
 
 static int PTRCALL
-PREFIX(entityValueTok)(const ENCODING *enc, const char *ptr,
-                       const char *end, const char **nextTokPtr)
-{
+PREFIX(entityValueTok)(const ENCODING *enc, const char *ptr, const char *end,
+                       const char **nextTokPtr) {
   const char *start;
   if (ptr >= end)
     return XML_TOK_NONE;
@@ -1304,10 +1329,14 @@ PREFIX(entityValueTok)(const ENCODING *enc, const char *ptr,
   start = ptr;
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: ptr += n; break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    ptr += n;                                                                  \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_AMP:
       if (ptr == start)
         return PREFIX(scanRef)(enc, ptr + MINBPC(enc), end, nextTokPtr);
@@ -1315,8 +1344,7 @@ PREFIX(entityValueTok)(const ENCODING *enc, const char *ptr,
       return XML_TOK_DATA_CHARS;
     case BT_PERCNT:
       if (ptr == start) {
-        int tok =  PREFIX(scanPercent)(enc, ptr + MINBPC(enc),
-                                       end, nextTokPtr);
+        int tok = PREFIX(scanPercent)(enc, ptr + MINBPC(enc), end, nextTokPtr);
         return (tok == XML_TOK_PERCENT) ? XML_TOK_INVALID : tok;
       }
       *nextTokPtr = ptr;
@@ -1349,12 +1377,11 @@ PREFIX(entityValueTok)(const ENCODING *enc, const char *ptr,
   return XML_TOK_DATA_CHARS;
 }
 
-#ifdef XML_DTD
+#  ifdef XML_DTD
 
 static int PTRCALL
-PREFIX(ignoreSectionTok)(const ENCODING *enc, const char *ptr,
-                         const char *end, const char **nextTokPtr)
-{
+PREFIX(ignoreSectionTok)(const ENCODING *enc, const char *ptr, const char *end,
+                         const char **nextTokPtr) {
   int level = 0;
   if (MINBPC(enc) > 1) {
     size_t n = end - ptr;
@@ -1365,7 +1392,7 @@ PREFIX(ignoreSectionTok)(const ENCODING *enc, const char *ptr,
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-    INVALID_CASES(ptr, nextTokPtr)
+      INVALID_CASES(ptr, nextTokPtr)
     case BT_LT:
       ptr += MINBPC(enc);
       REQUIRE_CHAR(enc, ptr, end);
@@ -1402,12 +1429,11 @@ PREFIX(ignoreSectionTok)(const ENCODING *enc, const char *ptr,
   return XML_TOK_PARTIAL;
 }
 
-#endif /* XML_DTD */
+#  endif /* XML_DTD */
 
 static int PTRCALL
 PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end,
-                   const char **badPtr)
-{
+                   const char **badPtr) {
   ptr += MINBPC(enc);
   end -= MINBPC(enc);
   for (; HAS_CHAR(enc, ptr, end); ptr += MINBPC(enc)) {
@@ -1430,9 +1456,9 @@ PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end,
     case BT_AST:
     case BT_PERCNT:
     case BT_NUM:
-#ifdef XML_NS
+#  ifdef XML_NS
     case BT_COLON:
-#endif
+#  endif
       break;
     case BT_S:
       if (CHAR_MATCHES(enc, ptr, ASCII_TAB)) {
@@ -1442,7 +1468,7 @@ PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end,
       break;
     case BT_NAME:
     case BT_NMSTRT:
-      if (!(BYTE_TO_ASCII(enc, ptr) & ~0x7f))
+      if (! (BYTE_TO_ASCII(enc, ptr) & ~0x7f))
         break;
       /* fall through */
     default:
@@ -1466,9 +1492,8 @@ PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end,
 */
 
 static int PTRCALL
-PREFIX(getAtts)(const ENCODING *enc, const char *ptr,
-                int attsMax, ATTRIBUTE *atts)
-{
+PREFIX(getAtts)(const ENCODING *enc, const char *ptr, int attsMax,
+                ATTRIBUTE *atts) {
   enum { other, inName, inValue } state = inName;
   int nAtts = 0;
   int open = 0; /* defined when state == inValue;
@@ -1476,32 +1501,35 @@ PREFIX(getAtts)(const ENCODING *enc, const char *ptr,
 
   for (ptr += MINBPC(enc);; ptr += MINBPC(enc)) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define START_NAME \
-      if (state == other) { \
-        if (nAtts < attsMax) { \
-          atts[nAtts].name = ptr; \
-          atts[nAtts].normalized = 1; \
-        } \
-        state = inName; \
-      }
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: START_NAME ptr += (n - MINBPC(enc)); break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define START_NAME                                                           \
+    if (state == other) {                                                      \
+      if (nAtts < attsMax) {                                                   \
+        atts[nAtts].name = ptr;                                                \
+        atts[nAtts].normalized = 1;                                            \
+      }                                                                        \
+      state = inName;                                                          \
+    }
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    START_NAME ptr += (n - MINBPC(enc));                                       \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_NONASCII:
     case BT_NMSTRT:
     case BT_HEX:
       START_NAME
       break;
-#undef START_NAME
+#  undef START_NAME
     case BT_QUOT:
       if (state != inValue) {
         if (nAtts < attsMax)
           atts[nAtts].valuePtr = ptr + MINBPC(enc);
         state = inValue;
         open = BT_QUOT;
-      }
-      else if (open == BT_QUOT) {
+      } else if (open == BT_QUOT) {
         state = other;
         if (nAtts < attsMax)
           atts[nAtts].valueEnd = ptr;
@@ -1514,8 +1542,7 @@ PREFIX(getAtts)(const ENCODING *enc, const char *ptr,
           atts[nAtts].valuePtr = ptr + MINBPC(enc);
         state = inValue;
         open = BT_APOS;
-      }
-      else if (open == BT_APOS) {
+      } else if (open == BT_APOS) {
         state = other;
         if (nAtts < attsMax)
           atts[nAtts].valueEnd = ptr;
@@ -1529,16 +1556,15 @@ PREFIX(getAtts)(const ENCODING *enc, const char *ptr,
     case BT_S:
       if (state == inName)
         state = other;
-      else if (state == inValue
-               && nAtts < attsMax
-               && atts[nAtts].normalized
+      else if (state == inValue && nAtts < attsMax && atts[nAtts].normalized
                && (ptr == atts[nAtts].valuePtr
                    || BYTE_TO_ASCII(enc, ptr) != ASCII_SPACE
                    || BYTE_TO_ASCII(enc, ptr + MINBPC(enc)) == ASCII_SPACE
                    || BYTE_TYPE(enc, ptr + MINBPC(enc)) == open))
         atts[nAtts].normalized = 0;
       break;
-    case BT_CR: case BT_LF:
+    case BT_CR:
+    case BT_LF:
       /* This case ensures that the first attribute name is counted
          Apart from that we could just change state on the quote. */
       if (state == inName)
@@ -1559,29 +1585,44 @@ PREFIX(getAtts)(const ENCODING *enc, const char *ptr,
 }
 
 static int PTRFASTCALL
-PREFIX(charRefNumber)(const ENCODING *UNUSED_P(enc), const char *ptr)
-{
+PREFIX(charRefNumber)(const ENCODING *enc, const char *ptr) {
   int result = 0;
   /* skip &# */
-  ptr += 2*MINBPC(enc);
+  UNUSED_P(enc);
+  ptr += 2 * MINBPC(enc);
   if (CHAR_MATCHES(enc, ptr, ASCII_x)) {
-    for (ptr += MINBPC(enc);
-         !CHAR_MATCHES(enc, ptr, ASCII_SEMI);
+    for (ptr += MINBPC(enc); ! CHAR_MATCHES(enc, ptr, ASCII_SEMI);
          ptr += MINBPC(enc)) {
       int c = BYTE_TO_ASCII(enc, ptr);
       switch (c) {
-      case ASCII_0: case ASCII_1: case ASCII_2: case ASCII_3: case ASCII_4:
-      case ASCII_5: case ASCII_6: case ASCII_7: case ASCII_8: case ASCII_9:
+      case ASCII_0:
+      case ASCII_1:
+      case ASCII_2:
+      case ASCII_3:
+      case ASCII_4:
+      case ASCII_5:
+      case ASCII_6:
+      case ASCII_7:
+      case ASCII_8:
+      case ASCII_9:
         result <<= 4;
         result |= (c - ASCII_0);
         break;
-      case ASCII_A: case ASCII_B: case ASCII_C:
-      case ASCII_D: case ASCII_E: case ASCII_F:
+      case ASCII_A:
+      case ASCII_B:
+      case ASCII_C:
+      case ASCII_D:
+      case ASCII_E:
+      case ASCII_F:
         result <<= 4;
         result += 10 + (c - ASCII_A);
         break;
-      case ASCII_a: case ASCII_b: case ASCII_c:
-      case ASCII_d: case ASCII_e: case ASCII_f:
+      case ASCII_a:
+      case ASCII_b:
+      case ASCII_c:
+      case ASCII_d:
+      case ASCII_e:
+      case ASCII_f:
         result <<= 4;
         result += 10 + (c - ASCII_a);
         break;
@@ -1589,9 +1630,8 @@ PREFIX(charRefNumber)(const ENCODING *UNUSED_P(enc), const char *ptr)
       if (result >= 0x110000)
         return -1;
     }
-  }
-  else {
-    for (; !CHAR_MATCHES(enc, ptr, ASCII_SEMI); ptr += MINBPC(enc)) {
+  } else {
+    for (; ! CHAR_MATCHES(enc, ptr, ASCII_SEMI); ptr += MINBPC(enc)) {
       int c = BYTE_TO_ASCII(enc, ptr);
       result *= 10;
       result += (c - ASCII_0);
@@ -1603,10 +1643,10 @@ PREFIX(charRefNumber)(const ENCODING *UNUSED_P(enc), const char *ptr)
 }
 
 static int PTRCALL
-PREFIX(predefinedEntityName)(const ENCODING *UNUSED_P(enc), const char *ptr,
-                             const char *end)
-{
-  switch ((end - ptr)/MINBPC(enc)) {
+PREFIX(predefinedEntityName)(const ENCODING *enc, const char *ptr,
+                             const char *end) {
+  UNUSED_P(enc);
+  switch ((end - ptr) / MINBPC(enc)) {
   case 2:
     if (CHAR_MATCHES(enc, ptr + MINBPC(enc), ASCII_t)) {
       switch (BYTE_TO_ASCII(enc, ptr)) {
@@ -1657,9 +1697,9 @@ PREFIX(predefinedEntityName)(const ENCODING *UNUSED_P(enc), const char *ptr,
 }
 
 static int PTRCALL
-PREFIX(nameMatchesAscii)(const ENCODING *UNUSED_P(enc), const char *ptr1,
-                         const char *end1, const char *ptr2)
-{
+PREFIX(nameMatchesAscii)(const ENCODING *enc, const char *ptr1,
+                         const char *end1, const char *ptr2) {
+  UNUSED_P(enc);
   for (; *ptr2; ptr1 += MINBPC(enc), ptr2++) {
     if (end1 - ptr1 < MINBPC(enc)) {
       /* This line cannot be executed.  The incoming data has already
@@ -1669,27 +1709,30 @@ PREFIX(nameMatchesAscii)(const ENCODING *UNUSED_P(enc), const char *ptr1,
        */
       return 0; /* LCOV_EXCL_LINE */
     }
-    if (!CHAR_MATCHES(enc, ptr1, *ptr2))
+    if (! CHAR_MATCHES(enc, ptr1, *ptr2))
       return 0;
   }
   return ptr1 == end1;
 }
 
 static int PTRFASTCALL
-PREFIX(nameLength)(const ENCODING *enc, const char *ptr)
-{
+PREFIX(nameLength)(const ENCODING *enc, const char *ptr) {
   const char *start = ptr;
   for (;;) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: ptr += n; break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    ptr += n;                                                                  \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_NONASCII:
     case BT_NMSTRT:
-#ifdef XML_NS
+#  ifdef XML_NS
     case BT_COLON:
-#endif
+#  endif
     case BT_HEX:
     case BT_DIGIT:
     case BT_NAME:
@@ -1702,9 +1745,8 @@ PREFIX(nameLength)(const ENCODING *enc, const char *ptr)
   }
 }
 
-static const char * PTRFASTCALL
-PREFIX(skipS)(const ENCODING *enc, const char *ptr)
-{
+static const char *PTRFASTCALL
+PREFIX(skipS)(const ENCODING *enc, const char *ptr) {
   for (;;) {
     switch (BYTE_TYPE(enc, ptr)) {
     case BT_LF:
@@ -1719,19 +1761,18 @@ PREFIX(skipS)(const ENCODING *enc, const char *ptr)
 }
 
 static void PTRCALL
-PREFIX(updatePosition)(const ENCODING *enc,
-                       const char *ptr,
-                       const char *end,
-                       POSITION *pos)
-{
+PREFIX(updatePosition)(const ENCODING *enc, const char *ptr, const char *end,
+                       POSITION *pos) {
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: \
-      ptr += n; \
-      break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    ptr += n;                                                                  \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_LF:
       pos->columnNumber = (XML_Size)-1;
       pos->lineNumber++;
@@ -1752,12 +1793,12 @@ PREFIX(updatePosition)(const ENCODING *enc,
   }
 }
 
-#undef DO_LEAD_CASE
-#undef MULTIBYTE_CASES
-#undef INVALID_CASES
-#undef CHECK_NAME_CASE
-#undef CHECK_NAME_CASES
-#undef CHECK_NMSTRT_CASE
-#undef CHECK_NMSTRT_CASES
+#  undef DO_LEAD_CASE
+#  undef MULTIBYTE_CASES
+#  undef INVALID_CASES
+#  undef CHECK_NAME_CASE
+#  undef CHECK_NAME_CASES
+#  undef CHECK_NMSTRT_CASE
+#  undef CHECK_NMSTRT_CASES
 
 #endif /* XML_TOK_IMPL_C */
diff --git a/Modules/expat/xmltok_impl.h b/Modules/expat/xmltok_impl.h
index a6420f48eedc04..e925dbc7e2c833 100644
--- a/Modules/expat/xmltok_impl.h
+++ b/Modules/expat/xmltok_impl.h
@@ -31,43 +31,43 @@
 */
 
 enum {
-  BT_NONXML,
-  BT_MALFORM,
-  BT_LT,
-  BT_AMP,
-  BT_RSQB,
-  BT_LEAD2,
-  BT_LEAD3,
-  BT_LEAD4,
-  BT_TRAIL,
-  BT_CR,
-  BT_LF,
-  BT_GT,
-  BT_QUOT,
-  BT_APOS,
-  BT_EQUALS,
-  BT_QUEST,
-  BT_EXCL,
-  BT_SOL,
-  BT_SEMI,
-  BT_NUM,
-  BT_LSQB,
-  BT_S,
-  BT_NMSTRT,
-  BT_COLON,
-  BT_HEX,
-  BT_DIGIT,
-  BT_NAME,
-  BT_MINUS,
-  BT_OTHER, /* known not to be a name or name start character */
+  BT_NONXML,   /* e.g. noncharacter-FFFF */
+  BT_MALFORM,  /* illegal, with regard to encoding */
+  BT_LT,       /* less than = "<" */
+  BT_AMP,      /* ampersand = "&" */
+  BT_RSQB,     /* right square bracket = "[" */
+  BT_LEAD2,    /* lead byte of a 2-byte UTF-8 character */
+  BT_LEAD3,    /* lead byte of a 3-byte UTF-8 character */
+  BT_LEAD4,    /* lead byte of a 4-byte UTF-8 character */
+  BT_TRAIL,    /* trailing unit, e.g. second 16-bit unit of a 4-byte char. */
+  BT_CR,       /* carriage return = "\r" */
+  BT_LF,       /* line feed = "\n" */
+  BT_GT,       /* greater than = ">" */
+  BT_QUOT,     /* quotation character = "\"" */
+  BT_APOS,     /* aposthrophe = "'" */
+  BT_EQUALS,   /* equal sign = "=" */
+  BT_QUEST,    /* question mark = "?" */
+  BT_EXCL,     /* exclamation mark = "!" */
+  BT_SOL,      /* solidus, slash = "/" */
+  BT_SEMI,     /* semicolon = ";" */
+  BT_NUM,      /* number sign = "#" */
+  BT_LSQB,     /* left square bracket = "[" */
+  BT_S,        /* white space, e.g. "\t", " "[, "\r"] */
+  BT_NMSTRT,   /* non-hex name start letter = "G".."Z" + "g".."z" + "_" */
+  BT_COLON,    /* colon = ":" */
+  BT_HEX,      /* hex letter = "A".."F" + "a".."f" */
+  BT_DIGIT,    /* digit = "0".."9" */
+  BT_NAME,     /* dot and middle dot = "." + chr(0xb7) */
+  BT_MINUS,    /* minus = "-" */
+  BT_OTHER,    /* known not to be a name or name start character */
   BT_NONASCII, /* might be a name or name start character */
-  BT_PERCNT,
-  BT_LPAR,
-  BT_RPAR,
-  BT_AST,
-  BT_PLUS,
-  BT_COMMA,
-  BT_VERBAR
+  BT_PERCNT,   /* percent sign = "%" */
+  BT_LPAR,     /* left parenthesis = "(" */
+  BT_RPAR,     /* right parenthesis = "(" */
+  BT_AST,      /* asterisk = "*" */
+  BT_PLUS,     /* plus sign = "+" */
+  BT_COMMA,    /* comma = "," */
+  BT_VERBAR    /* vertical bar = "|" */
 };
 
 #include 
diff --git a/Modules/expat/xmltok_ns.c b/Modules/expat/xmltok_ns.c
index 23d31e8e424916..919c74e9f97fe8 100644
--- a/Modules/expat/xmltok_ns.c
+++ b/Modules/expat/xmltok_ns.c
@@ -33,56 +33,47 @@
 #ifdef XML_TOK_NS_C
 
 const ENCODING *
-NS(XmlGetUtf8InternalEncoding)(void)
-{
+NS(XmlGetUtf8InternalEncoding)(void) {
   return &ns(internal_utf8_encoding).enc;
 }
 
 const ENCODING *
-NS(XmlGetUtf16InternalEncoding)(void)
-{
-#if BYTEORDER == 1234
+NS(XmlGetUtf16InternalEncoding)(void) {
+#  if BYTEORDER == 1234
   return &ns(internal_little2_encoding).enc;
-#elif BYTEORDER == 4321
+#  elif BYTEORDER == 4321
   return &ns(internal_big2_encoding).enc;
-#else
+#  else
   const short n = 1;
-  return (*(const char *)&n
-          ? &ns(internal_little2_encoding).enc
-          : &ns(internal_big2_encoding).enc);
-#endif
+  return (*(const char *)&n ? &ns(internal_little2_encoding).enc
+                            : &ns(internal_big2_encoding).enc);
+#  endif
 }
 
-static const ENCODING * const NS(encodings)[] = {
-  &ns(latin1_encoding).enc,
-  &ns(ascii_encoding).enc,
-  &ns(utf8_encoding).enc,
-  &ns(big2_encoding).enc,
-  &ns(big2_encoding).enc,
-  &ns(little2_encoding).enc,
-  &ns(utf8_encoding).enc /* NO_ENC */
+static const ENCODING *const NS(encodings)[] = {
+    &ns(latin1_encoding).enc, &ns(ascii_encoding).enc,
+    &ns(utf8_encoding).enc,   &ns(big2_encoding).enc,
+    &ns(big2_encoding).enc,   &ns(little2_encoding).enc,
+    &ns(utf8_encoding).enc /* NO_ENC */
 };
 
 static int PTRCALL
 NS(initScanProlog)(const ENCODING *enc, const char *ptr, const char *end,
-                   const char **nextTokPtr)
-{
-  return initScan(NS(encodings), (const INIT_ENCODING *)enc,
-                  XML_PROLOG_STATE, ptr, end, nextTokPtr);
+                   const char **nextTokPtr) {
+  return initScan(NS(encodings), (const INIT_ENCODING *)enc, XML_PROLOG_STATE,
+                  ptr, end, nextTokPtr);
 }
 
 static int PTRCALL
 NS(initScanContent)(const ENCODING *enc, const char *ptr, const char *end,
-                    const char **nextTokPtr)
-{
-  return initScan(NS(encodings), (const INIT_ENCODING *)enc,
-                  XML_CONTENT_STATE, ptr, end, nextTokPtr);
+                    const char **nextTokPtr) {
+  return initScan(NS(encodings), (const INIT_ENCODING *)enc, XML_CONTENT_STATE,
+                  ptr, end, nextTokPtr);
 }
 
 int
 NS(XmlInitEncoding)(INIT_ENCODING *p, const ENCODING **encPtr,
-                    const char *name)
-{
+                    const char *name) {
   int i = getEncodingIndex(name);
   if (i == UNKNOWN_ENC)
     return 0;
@@ -96,9 +87,8 @@ NS(XmlInitEncoding)(INIT_ENCODING *p, const ENCODING **encPtr,
 }
 
 static const ENCODING *
-NS(findEncoding)(const ENCODING *enc, const char *ptr, const char *end)
-{
-#define ENCODING_MAX 128
+NS(findEncoding)(const ENCODING *enc, const char *ptr, const char *end) {
+#  define ENCODING_MAX 128
   char buf[ENCODING_MAX];
   char *p = buf;
   int i;
@@ -115,28 +105,14 @@ NS(findEncoding)(const ENCODING *enc, const char *ptr, const char *end)
 }
 
 int
-NS(XmlParseXmlDecl)(int isGeneralTextEntity,
-                    const ENCODING *enc,
-                    const char *ptr,
-                    const char *end,
-                    const char **badPtr,
-                    const char **versionPtr,
-                    const char **versionEndPtr,
-                    const char **encodingName,
-                    const ENCODING **encoding,
-                    int *standalone)
-{
-  return doParseXmlDecl(NS(findEncoding),
-                        isGeneralTextEntity,
-                        enc,
-                        ptr,
-                        end,
-                        badPtr,
-                        versionPtr,
-                        versionEndPtr,
-                        encodingName,
-                        encoding,
-                        standalone);
+NS(XmlParseXmlDecl)(int isGeneralTextEntity, const ENCODING *enc,
+                    const char *ptr, const char *end, const char **badPtr,
+                    const char **versionPtr, const char **versionEndPtr,
+                    const char **encodingName, const ENCODING **encoding,
+                    int *standalone) {
+  return doParseXmlDecl(NS(findEncoding), isGeneralTextEntity, enc, ptr, end,
+                        badPtr, versionPtr, versionEndPtr, encodingName,
+                        encoding, standalone);
 }
 
 #endif /* XML_TOK_NS_C */
diff --git a/PCbuild/_elementtree.vcxproj b/PCbuild/_elementtree.vcxproj
index 33a017327189ed..4a125b243b780d 100644
--- a/PCbuild/_elementtree.vcxproj
+++ b/PCbuild/_elementtree.vcxproj
@@ -116,7 +116,6 @@
   
   
     
-    
     
     
     
diff --git a/PCbuild/_elementtree.vcxproj.filters b/PCbuild/_elementtree.vcxproj.filters
index 4597ee521b3331..6acdf35846ab14 100644
--- a/PCbuild/_elementtree.vcxproj.filters
+++ b/PCbuild/_elementtree.vcxproj.filters
@@ -33,9 +33,6 @@
     
       Header Files
     
-    
-      Header Files
-    
     
       Header Files
     
diff --git a/PCbuild/pyexpat.vcxproj b/PCbuild/pyexpat.vcxproj
index 28a11a82936ee2..b2d9f5d57d4975 100644
--- a/PCbuild/pyexpat.vcxproj
+++ b/PCbuild/pyexpat.vcxproj
@@ -100,7 +100,6 @@
   
   
     
-    
     
     
     
diff --git a/PCbuild/pyexpat.vcxproj.filters b/PCbuild/pyexpat.vcxproj.filters
index cb02847980c634..f8d46026c9c284 100644
--- a/PCbuild/pyexpat.vcxproj.filters
+++ b/PCbuild/pyexpat.vcxproj.filters
@@ -20,9 +20,6 @@
     
       Source Files
     
-    
-      Source Files
-    
     
       Source Files
     

From b3c35fea13838426525f87366261efa50e5c0864 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Thu, 26 Sep 2019 00:12:39 -0700
Subject: [PATCH 0742/2163] Doc: Use the `with` statement in the first example
 of the ftplib doc. (GH-16271) (GH-16412)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

(cherry picked from commit 5d326abf2cb4891b78d9319a81bffb3974b5b745)

Co-authored-by: Stéphane Wirtel 
---
 Doc/library/ftplib.rst | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Doc/library/ftplib.rst b/Doc/library/ftplib.rst
index 7423413d209ae0..db0212367340d9 100644
--- a/Doc/library/ftplib.rst
+++ b/Doc/library/ftplib.rst
@@ -33,7 +33,8 @@ Here's a sample session using the :mod:`ftplib` module::
    drwxr-sr-x    4 1176     1176         4096 Nov 17  2008 project
    drwxr-xr-x    3 1176     1176         4096 Oct 10  2012 tools
    '226 Directory send OK.'
-   >>> ftp.retrbinary('RETR README', open('README', 'wb').write)
+   >>> with open('README', 'wb') as fp:
+   >>>     ftp.retrbinary('RETR README', fp.write)
    '226 Transfer complete.'
    >>> ftp.quit()
 

From 68040edb79895c577e2526ad5f30b1b161b2c32b Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Thu, 26 Sep 2019 02:16:38 -0700
Subject: [PATCH 0743/2163] bpo-38130: Fix error in explaining when an
 exception is re-raised (GH-16016) (GH-16415)

Co-Authored-By: Ashwin Ramaswami 
(cherry picked from commit 1ad7be2f16cc9955f271f57a5089602bb41eee85)

Co-authored-by: Mohammad Dehghan 
---
 Doc/tutorial/errors.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Doc/tutorial/errors.rst b/Doc/tutorial/errors.rst
index 9585f0673aec55..4bc7184d1078c5 100644
--- a/Doc/tutorial/errors.rst
+++ b/Doc/tutorial/errors.rst
@@ -343,7 +343,7 @@ example::
 
 If a :keyword:`finally` clause is present, the :keyword:`finally` clause will execute as the last task before the :keyword:`try` statement completes. The :keyword:`finally` clause runs whether or not the :keyword:`try` statement produces an exception. The following points discuss more complex cases when an exception occurs:
 
-* If an exception occurs during execution of the :keyword:`!try` clause, the exception may be handled by an :keyword:`except` clause. In all cases, the exception is re-raised after the :keyword:`!finally` clause has been executed.
+* If an exception occurs during execution of the :keyword:`!try` clause, the exception may be handled by an :keyword:`except` clause. If the exception is not handled by an :keyword:`except` clause, the exception is re-raised after the :keyword:`!finally` clause has been executed.
 
 * An exception could occur during execution of an :keyword:`!except` or :keyword:`!else` clause. Again, the exception is re-raised after the :keyword:`!finally` clause has been executed.
 

From 96c8475362acb41decd1d7db9243f328973e5de7 Mon Sep 17 00:00:00 2001
From: Victor Stinner 
Date: Thu, 26 Sep 2019 16:17:34 +0200
Subject: [PATCH 0744/2163] [3.8] bpo-38234: Backport init path config changes
 from master (GH-16423)

* bpo-38234: Py_SetPath() uses the program full path (GH-16357)

Py_SetPath() now sets sys.executable to the program full path
(Py_GetProgramFullPath()), rather than to the program name
(Py_GetProgramName()).

Fix also memory leaks in pathconfig_set_from_config().

(cherry picked from commit 1ce152a42eaa917d7763bce93f1e1ca72530d7ca)

* bpo-38234: Add tests for Python init path config (GH-16358)


(cherry picked from commit bb6bf7d342b4503a6227fd209fac934905b6a1aa)

* bpo-38234: test_embed: test pyvenv.cfg and pybuilddir.txt (GH-16366)

Add test_init_pybuilddir() and test_init_pyvenv_cfg() to test_embed
to test pyvenv.cfg and pybuilddir.txt configuration files.

Fix sysconfig._generate_posix_vars(): pybuilddir.txt uses UTF-8
encoding, not ASCII.

(cherry picked from commit 52ad33abbfb6637d74932617c7013bae0ccf6e32)

* bpo-38234: Cleanup getpath.c (GH-16367)

* search_for_prefix() directly calls reduce() if found is greater
  than 0.
* Add calculate_pybuilddir() subfunction.
* search_for_prefix(): add path string buffer for readability.
* Fix some error handling code paths: release resources on error.
* calculate_read_pyenv(): rename tmpbuffer to filename.
* test.pythoninfo now also logs windows.dll_path

(cherry picked from commit 221fd84703c545408bbb4a6e0b58459651331f5c)

* bpo-38234: Fix test_embed pathconfig tests (GH-16390)

bpo-38234: On macOS and FreeBSD, the temporary directory can be
symbolic link. For example, /tmp can be a symbolic link to /var/tmp.
Call realpath() to resolve all symbolic links.

(cherry picked from commit 00508a7407d7d300b487532e2271534b20e378a7)

* bpo-38234: Add test_init_setpath_config() to test_embed (GH-16402)

* Add test_embed.test_init_setpath_config(): test Py_SetPath()
  with PyConfig.
* test_init_setpath() and test_init_setpythonhome() no longer call
  Py_SetProgramName(), but use the default program name.
* _PyPathConfig: isolated, site_import  and base_executable
  fields are now only available on Windows.
* If executable is set explicitly in the configuration, ignore
  calculated base_executable: _PyConfig_InitPathConfig() copies
  executable to base_executable.
* Complete path config documentation.

(cherry picked from commit 8bf39b606ef7b02c0279a80789f3c4824b0da5e9)

* bpo-38234: Complete init config documentation (GH-16404)


(cherry picked from commit 88feaecd46a8f427e30ef7ad8cfcddfe392a2402)

* bpo-38234: Fix test_embed.test_init_setpath_config() on FreeBSD (GH-16406)

Explicitly preinitializes with a Python preconfiguration to avoid
Py_SetPath() implicit preinitialization with a compat
preconfiguration.

Fix also test_init_setpath() and test_init_setpythonhome() on macOS:
use self.test_exe as the executable (and base_executable), rather
than shutil.which('python3').

(cherry picked from commit 49d99f01e6e51acec5ca57a02e857f0796bc418b)

* bpo-38234: Py_Initialize() sets global path configuration (GH-16421)

* Py_InitializeFromConfig() now writes PyConfig path configuration to
  the global path configuration (_Py_path_config).
* Add test_embed.test_get_pathconfig().
* Fix typo in _PyWideStringList_Join().

(cherry picked from commit 12f2f177fc483723406d7917194e7f655a20631b)
---
 Doc/c-api/init.rst                            |   8 +-
 Doc/c-api/init_config.rst                     |  79 ++--
 Doc/whatsnew/3.8.rst                          |   5 +
 Include/internal/pycore_pathconfig.h          |  17 +-
 Lib/sysconfig.py                              |   2 +-
 Lib/test/pythoninfo.py                        | 126 +++++--
 Lib/test/test_embed.py                        | 342 +++++++++++++++++-
 .../2019-09-24-17-09-48.bpo-38234.d0bhEA.rst  |   3 +
 Modules/getpath.c                             | 195 ++++++----
 PC/getpathp.c                                 |  36 +-
 Programs/_testembed.c                         | 113 +++++-
 Python/pathconfig.c                           |  73 +++-
 Python/pylifecycle.c                          |   4 +-
 13 files changed, 795 insertions(+), 208 deletions(-)
 create mode 100644 Misc/NEWS.d/next/C API/2019-09-24-17-09-48.bpo-38234.d0bhEA.rst

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 0f8ff3b0dde8ff..0b7a84d031532e 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -472,8 +472,8 @@ Process-wide parameters
    dependent delimiter character, which is ``':'`` on Unix and Mac OS X, ``';'``
    on Windows.
 
-   This also causes :data:`sys.executable` to be set only to the raw program
-   name (see :c:func:`Py_SetProgramName`) and for :data:`sys.prefix` and
+   This also causes :data:`sys.executable` to be set to the program
+   full path (see :c:func:`Py_GetProgramFullPath`) and for :data:`sys.prefix` and
    :data:`sys.exec_prefix` to be empty.  It is up to the caller to modify these
    if required after calling :c:func:`Py_Initialize`.
 
@@ -483,6 +483,10 @@ Process-wide parameters
    The path argument is copied internally, so the caller may free it after the
    call completes.
 
+   .. versionchanged:: 3.8
+      The program full path is now used for :data:`sys.executable`, instead
+      of the program name.
+
 
 .. c:function:: const char* Py_GetVersion()
 
diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst
index bc24fa0813172d..0c3c725c841ac1 100644
--- a/Doc/c-api/init_config.rst
+++ b/Doc/c-api/init_config.rst
@@ -241,6 +241,7 @@ PyPreConfig
       locale to decide if it should be coerced.
 
    .. c:member:: int coerce_c_locale_warn
+
       If non-zero, emit a warning if the C locale is coerced.
 
    .. c:member:: int dev_mode
@@ -300,7 +301,7 @@ For :ref:`Python Configuration `
 (:c:func:`PyPreConfig_InitPythonConfig`), if Python is initialized with
 command line arguments, the command line arguments must also be passed to
 preinitialize Python, since they have an effect on the pre-configuration
-like encodings. For example, the :option:`-X` ``utf8`` command line option
+like encodings. For example, the :option:`-X utf8 <-X>` command line option
 enables the UTF-8 Mode.
 
 ``PyMem_SetAllocator()`` can be called after :c:func:`Py_PreInitialize` and
@@ -464,7 +465,7 @@ PyConfig
 
    .. c:member:: int dev_mode
 
-      Development mode: see :option:`-X` ``dev``.
+      Development mode: see :option:`-X dev <-X>`.
 
    .. c:member:: int dump_refs
 
@@ -482,7 +483,7 @@ PyConfig
 
    .. c:member:: int faulthandler
 
-      If non-zero, call :func:`faulthandler.enable`.
+      If non-zero, call :func:`faulthandler.enable` at startup.
 
    .. c:member:: wchar_t* filesystem_encoding
 
@@ -504,6 +505,9 @@ PyConfig
 
       Python home directory.
 
+      Initialized from :envvar:`PYTHONHOME` environment variable value by
+      default.
+
    .. c:member:: int import_time
 
       If non-zero, profile import time.
@@ -561,7 +565,7 @@ PyConfig
 
       :data:`sys.path`. If :c:member:`~PyConfig.module_search_paths_set` is
       equal to 0, the :c:member:`~PyConfig.module_search_paths` is overridden
-      by the function computing the :ref:`Path Configuration
+      by the function calculating the :ref:`Path Configuration
       `.
 
    .. c:member:: int optimization_level
@@ -586,9 +590,9 @@ PyConfig
 
    .. c:member:: int pathconfig_warnings
 
-      If equal to 0, suppress warnings when computing the path configuration
-      (Unix only, Windows does not log any warning). Otherwise, warnings are
-      written into ``stderr``.
+      If equal to 0, suppress warnings when calculating the :ref:`Path
+      Configuration ` (Unix only, Windows does not log any
+      warning). Otherwise, warnings are written into ``stderr``.
 
    .. c:member:: wchar_t* prefix
 
@@ -596,39 +600,46 @@ PyConfig
 
    .. c:member:: wchar_t* program_name
 
-      Program name.
+      Program name. Used to initialize :c:member:`~PyConfig.executable`, and in
+      early error messages.
 
    .. c:member:: wchar_t* pycache_prefix
 
-      ``.pyc`` cache prefix.
+      :data:`sys.pycache_prefix`: ``.pyc`` cache prefix.
+
+      If NULL, :data:`sys.pycache_prefix` is set to ``None``.
 
    .. c:member:: int quiet
 
       Quiet mode. For example, don't display the copyright and version messages
-      even in interactive mode.
+      in interactive mode.
 
    .. c:member:: wchar_t* run_command
 
-      ``python3 -c COMMAND`` argument.
+      ``python3 -c COMMAND`` argument. Used by :c:func:`Py_RunMain`.
 
    .. c:member:: wchar_t* run_filename
 
-      ``python3 FILENAME`` argument.
+      ``python3 FILENAME`` argument. Used by :c:func:`Py_RunMain`.
 
    .. c:member:: wchar_t* run_module
 
-      ``python3 -m MODULE`` argument.
+      ``python3 -m MODULE`` argument. Used by :c:func:`Py_RunMain`.
 
    .. c:member:: int show_alloc_count
 
       Show allocation counts at exit?
 
+      Set to 1 by :option:`-X showalloccount <-X>` command line option.
+
       Need a special Python build with ``COUNT_ALLOCS`` macro defined.
 
    .. c:member:: int show_ref_count
 
       Show total reference count at exit?
 
+      Set to 1 by :option:`-X showrefcount <-X>` command line option.
+
       Need a debug build of Python (``Py_REF_DEBUG`` macro must be defined).
 
    .. c:member:: int site_import
@@ -647,7 +658,7 @@ PyConfig
 
    .. c:member:: int tracemalloc
 
-      If non-zero, call :func:`tracemalloc.start`.
+      If non-zero, call :func:`tracemalloc.start` at startup.
 
    .. c:member:: int use_environment
 
@@ -669,6 +680,9 @@ PyConfig
 
       If non-zero, write ``.pyc`` files.
 
+      :data:`sys.dont_write_bytecode` is initialized to the inverted value of
+      :c:member:`~PyConfig.write_bytecode`.
+
    .. c:member:: PyWideStringList xoptions
 
       :data:`sys._xoptions`.
@@ -694,8 +708,8 @@ Function to initialize Python:
 The caller is responsible to handle exceptions (error or exit) using
 :c:func:`PyStatus_Exception` and :c:func:`Py_ExitStatusException`.
 
-``PyImport_FrozenModules``, ``PyImport_AppendInittab()`` or
-``PyImport_ExtendInittab()`` is used: they must be set or called after Python
+If ``PyImport_FrozenModules``, ``PyImport_AppendInittab()`` or
+``PyImport_ExtendInittab()`` are used, they must be set or called after Python
 preinitialization and before the Python initialization.
 
 Example setting the program name::
@@ -760,7 +774,7 @@ configuration, and then override some parameters::
 
         /* Append our custom search path to sys.path */
         status = PyWideStringList_Append(&config.module_search_paths,
-                                      L"/path/to/more/modules");
+                                         L"/path/to/more/modules");
         if (PyStatus_Exception(status)) {
             goto done;
         }
@@ -791,9 +805,9 @@ isolate Python from the system. For example, to embed Python into an
 application.
 
 This configuration ignores global configuration variables, environments
-variables and command line arguments (:c:member:`PyConfig.argv` is not parsed).
-The C standard streams (ex: ``stdout``) and the LC_CTYPE locale are left
-unchanged by default.
+variables, command line arguments (:c:member:`PyConfig.argv` is not parsed)
+and user site directory. The C standard streams (ex: ``stdout``) and the
+LC_CTYPE locale are left unchanged. Signal handlers are not installed.
 
 Configuration files are still used with this configuration. Set the
 :ref:`Path Configuration ` ("output fields") to ignore these
@@ -864,29 +878,38 @@ Path Configuration
 
 :c:type:`PyConfig` contains multiple fields for the path configuration:
 
-* Path configuration input fields:
+* Path configuration inputs:
 
   * :c:member:`PyConfig.home`
   * :c:member:`PyConfig.pathconfig_warnings`
   * :c:member:`PyConfig.program_name`
   * :c:member:`PyConfig.pythonpath_env`
+  * current working directory: to get absolute paths
+  * ``PATH`` environment variable to get the program full path
+    (from :c:member:`PyConfig.program_name`)
+  * ``__PYVENV_LAUNCHER__`` environment variable
+  * (Windows only) Application paths in the registry under
+    "Software\Python\PythonCore\X.Y\PythonPath" of HKEY_CURRENT_USER and
+    HKEY_LOCAL_MACHINE (where X.Y is the Python version).
 
 * Path configuration output fields:
 
+  * :c:member:`PyConfig.base_exec_prefix`
   * :c:member:`PyConfig.base_executable`
+  * :c:member:`PyConfig.base_prefix`
   * :c:member:`PyConfig.exec_prefix`
   * :c:member:`PyConfig.executable`
-  * :c:member:`PyConfig.prefix`
   * :c:member:`PyConfig.module_search_paths_set`,
     :c:member:`PyConfig.module_search_paths`
+  * :c:member:`PyConfig.prefix`
 
-If at least one "output field" is not set, Python computes the path
+If at least one "output field" is not set, Python calculates the path
 configuration to fill unset fields. If
 :c:member:`~PyConfig.module_search_paths_set` is equal to 0,
 :c:member:`~PyConfig.module_search_paths` is overridden and
 :c:member:`~PyConfig.module_search_paths_set` is set to 1.
 
-It is possible to completely ignore the function computing the default
+It is possible to completely ignore the function calculating the default
 path configuration by setting explicitly all path configuration output
 fields listed above. A string is considered as set even if it is non-empty.
 ``module_search_paths`` is considered as set if
@@ -894,7 +917,7 @@ fields listed above. A string is considered as set even if it is non-empty.
 configuration input fields are ignored as well.
 
 Set :c:member:`~PyConfig.pathconfig_warnings` to 0 to suppress warnings when
-computing the path configuration (Unix only, Windows does not log any warning).
+calculating the path configuration (Unix only, Windows does not log any warning).
 
 If :c:member:`~PyConfig.base_prefix` or :c:member:`~PyConfig.base_exec_prefix`
 fields are not set, they inherit their value from :c:member:`~PyConfig.prefix`
@@ -961,7 +984,7 @@ initialization, the core feature of the :pep:`432`:
   * Builtin exceptions;
   * Builtin and frozen modules;
   * The :mod:`sys` module is only partially initialized
-    (ex: :data:`sys.path` doesn't exist yet);
+    (ex: :data:`sys.path` doesn't exist yet).
 
 * "Main" initialization phase, Python is fully initialized:
 
@@ -987,9 +1010,9 @@ No module is imported during the "Core" phase and the ``importlib`` module is
 not configured: the :ref:`Path Configuration ` is only
 applied during the "Main" phase. It may allow to customize Python in Python to
 override or tune the :ref:`Path Configuration `, maybe
-install a custom sys.meta_path importer or an import hook, etc.
+install a custom :data:`sys.meta_path` importer or an import hook, etc.
 
-It may become possible to compute the :ref:`Path Configuration
+It may become possible to calculatin the :ref:`Path Configuration
 ` in Python, after the Core phase and before the Main phase,
 which is one of the :pep:`432` motivation.
 
diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index c2455f487b14be..0995cb3b91196e 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -1347,6 +1347,11 @@ Build and C API Changes
   parameter for indicating the number of positional-only arguments.
   (Contributed by Pablo Galindo in :issue:`37221`.)
 
+* :c:func:`Py_SetPath` now sets :data:`sys.executable` to the program full
+  path (:c:func:`Py_GetProgramFullPath`) rather than to the program name
+  (:c:func:`Py_GetProgramName`).
+  (Contributed by Victor Stinner in :issue:`38234`.)
+
 
 Deprecated
 ==========
diff --git a/Include/internal/pycore_pathconfig.h b/Include/internal/pycore_pathconfig.h
index 61b3790fe1f4ac..ce75ccee835a20 100644
--- a/Include/internal/pycore_pathconfig.h
+++ b/Include/internal/pycore_pathconfig.h
@@ -19,6 +19,7 @@ typedef struct _PyPathConfig {
     wchar_t *program_name;
     /* Set by Py_SetPythonHome() or PYTHONHOME environment variable */
     wchar_t *home;
+#ifdef MS_WINDOWS
     /* isolated and site_import are used to set Py_IsolatedFlag and
        Py_NoSiteFlag flags on Windows in read_pth_file(). These fields
        are ignored when their value are equal to -1 (unset). */
@@ -26,12 +27,18 @@ typedef struct _PyPathConfig {
     int site_import;
     /* Set when a venv is detected */
     wchar_t *base_executable;
+#endif
 } _PyPathConfig;
 
-#define _PyPathConfig_INIT \
-    {.module_search_path = NULL, \
-     .isolated = -1, \
-     .site_import = -1}
+#ifdef MS_WINDOWS
+#  define _PyPathConfig_INIT \
+      {.module_search_path = NULL, \
+       .isolated = -1, \
+       .site_import = -1}
+#else
+#  define _PyPathConfig_INIT \
+      {.module_search_path = NULL}
+#endif
 /* Note: _PyPathConfig_INIT sets other fields to 0/NULL */
 
 PyAPI_DATA(_PyPathConfig) _Py_path_config;
@@ -59,7 +66,7 @@ extern int _Py_FindEnvConfigValue(
 extern wchar_t* _Py_GetDLLPath(void);
 #endif
 
-extern PyStatus _PyPathConfig_Init(void);
+extern PyStatus _PyConfig_WritePathConfig(const PyConfig *config);
 extern void _Py_DumpPathConfig(PyThreadState *tstate);
 
 #ifdef __cplusplus
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
index e76e6927cb1ff3..b9e2fafbc084a6 100644
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -412,7 +412,7 @@ def _generate_posix_vars():
         pprint.pprint(vars, stream=f)
 
     # Create file used for sys.path fixup -- see Modules/getpath.c
-    with open('pybuilddir.txt', 'w', encoding='ascii') as f:
+    with open('pybuilddir.txt', 'w', encoding='utf8') as f:
         f.write(pybuilddir)
 
 def _init_posix(vars):
diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py
index e9edf675b9105a..d2fa6c59365961 100644
--- a/Lib/test/pythoninfo.py
+++ b/Lib/test/pythoninfo.py
@@ -161,6 +161,25 @@ def collect_builtins(info_add):
     info_add('builtins.float.double_format', float.__getformat__("double"))
 
 
+def collect_urandom(info_add):
+    import os
+
+    if hasattr(os, 'getrandom'):
+        # PEP 524: Check if system urandom is initialized
+        try:
+            try:
+                os.getrandom(1, os.GRND_NONBLOCK)
+                state = 'ready (initialized)'
+            except BlockingIOError as exc:
+                state = 'not seeded yet (%s)' % exc
+            info_add('os.getrandom', state)
+        except OSError as exc:
+            # Python was compiled on a more recent Linux version
+            # than the current Linux kernel: ignore OSError(ENOSYS)
+            if exc.errno != errno.ENOSYS:
+                raise
+
+
 def collect_os(info_add):
     import os
 
@@ -180,16 +199,16 @@ def format_attr(attr, value):
     )
     copy_attributes(info_add, os, 'os.%s', attributes, formatter=format_attr)
 
-    call_func(info_add, 'os.cwd', os, 'getcwd')
+    call_func(info_add, 'os.getcwd', os, 'getcwd')
 
-    call_func(info_add, 'os.uid', os, 'getuid')
-    call_func(info_add, 'os.gid', os, 'getgid')
+    call_func(info_add, 'os.getuid', os, 'getuid')
+    call_func(info_add, 'os.getgid', os, 'getgid')
     call_func(info_add, 'os.uname', os, 'uname')
 
     def format_groups(groups):
         return ', '.join(map(str, groups))
 
-    call_func(info_add, 'os.groups', os, 'getgroups', formatter=format_groups)
+    call_func(info_add, 'os.getgroups', os, 'getgroups', formatter=format_groups)
 
     if hasattr(os, 'getlogin'):
         try:
@@ -202,7 +221,7 @@ def format_groups(groups):
             info_add("os.login", login)
 
     call_func(info_add, 'os.cpu_count', os, 'cpu_count')
-    call_func(info_add, 'os.loadavg', os, 'getloadavg')
+    call_func(info_add, 'os.getloadavg', os, 'getloadavg')
 
     # Environment variables used by the stdlib and tests. Don't log the full
     # environment: filter to list to not leak sensitive information.
@@ -286,20 +305,32 @@ def format_groups(groups):
         os.umask(mask)
         info_add("os.umask", '%03o' % mask)
 
-    if hasattr(os, 'getrandom'):
-        # PEP 524: Check if system urandom is initialized
-        try:
-            try:
-                os.getrandom(1, os.GRND_NONBLOCK)
-                state = 'ready (initialized)'
-            except BlockingIOError as exc:
-                state = 'not seeded yet (%s)' % exc
-            info_add('os.getrandom', state)
-        except OSError as exc:
-            # Python was compiled on a more recent Linux version
-            # than the current Linux kernel: ignore OSError(ENOSYS)
-            if exc.errno != errno.ENOSYS:
-                raise
+
+def collect_pwd(info_add):
+    try:
+        import pwd
+    except ImportError:
+        return
+    import os
+
+    uid = os.getuid()
+    try:
+        entry = pwd.getpwuid(uid)
+    except KeyError:
+        entry = None
+
+    info_add('pwd.getpwuid(%s)'% uid,
+             entry if entry is not None else '')
+
+    if entry is None:
+        # there is nothing interesting to read if the current user identifier
+        # is not the password database
+        return
+
+    if hasattr(os, 'getgrouplist'):
+        groups = os.getgrouplist(entry.pw_name, entry.pw_gid)
+        groups = ', '.join(map(str, groups))
+        info_add('os.getgrouplist', groups)
 
 
 def collect_readline(info_add):
@@ -620,36 +651,71 @@ def collect_subprocess(info_add):
     copy_attributes(info_add, subprocess, 'subprocess.%s', ('_USE_POSIX_SPAWN',))
 
 
+def collect_windows(info_add):
+    try:
+        import ctypes
+    except ImportError:
+        return
+
+    if not hasattr(ctypes, 'WinDLL'):
+        return
+
+    ntdll = ctypes.WinDLL('ntdll')
+    BOOLEAN = ctypes.c_ubyte
+
+    try:
+        RtlAreLongPathsEnabled = ntdll.RtlAreLongPathsEnabled
+    except AttributeError:
+        res = ''
+    else:
+        RtlAreLongPathsEnabled.restype = BOOLEAN
+        RtlAreLongPathsEnabled.argtypes = ()
+        res = bool(RtlAreLongPathsEnabled())
+    info_add('windows.RtlAreLongPathsEnabled', res)
+
+    try:
+        import _winapi
+        dll_path = _winapi.GetModuleFileName(sys.dllhandle)
+        info_add('windows.dll_path', dll_path)
+    except (ImportError, AttributeError):
+        pass
+
+
 def collect_info(info):
     error = False
     info_add = info.add
 
     for collect_func in (
-        # collect_os() should be the first, to check the getrandom() status
-        collect_os,
+        # collect_urandom() must be the first, to check the getrandom() status.
+        # Other functions may block on os.urandom() indirectly and so change
+        # its state.
+        collect_urandom,
 
         collect_builtins,
+        collect_cc,
+        collect_datetime,
+        collect_decimal,
+        collect_expat,
         collect_gdb,
+        collect_gdbm,
+        collect_get_config,
         collect_locale,
+        collect_os,
         collect_platform,
+        collect_pwd,
         collect_readline,
+        collect_resource,
         collect_socket,
         collect_sqlite,
         collect_ssl,
+        collect_subprocess,
         collect_sys,
         collect_sysconfig,
+        collect_testcapi,
         collect_time,
-        collect_datetime,
         collect_tkinter,
+        collect_windows,
         collect_zlib,
-        collect_expat,
-        collect_decimal,
-        collect_testcapi,
-        collect_resource,
-        collect_cc,
-        collect_gdbm,
-        collect_get_config,
-        collect_subprocess,
 
         # Collecting from tests should be last as they have side effects.
         collect_test_socket,
diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py
index e02acbc6befec6..ed2b96fbfc1fab 100644
--- a/Lib/test/test_embed.py
+++ b/Lib/test/test_embed.py
@@ -3,15 +3,19 @@
 import unittest
 
 from collections import namedtuple
+import contextlib
 import json
 import os
 import re
+import shutil
 import subprocess
 import sys
+import tempfile
 import textwrap
 
 
 MS_WINDOWS = (os.name == 'nt')
+MACOS = (sys.platform == 'darwin')
 
 PYMEM_ALLOCATOR_NOT_SET = 0
 PYMEM_ALLOCATOR_DEBUG = 2
@@ -25,6 +29,12 @@
 API_ISOLATED = 3
 
 
+def debug_build(program):
+    program = os.path.basename(program)
+    name = os.path.splitext(program)[0]
+    return name.endswith("_d")
+
+
 def remove_python_envvars():
     env = dict(os.environ)
     # Remove PYTHON* environment variables to get deterministic environment
@@ -40,7 +50,7 @@ def setUp(self):
         basepath = os.path.dirname(os.path.dirname(os.path.dirname(here)))
         exename = "_testembed"
         if MS_WINDOWS:
-            ext = ("_d" if "_d" in sys.executable else "") + ".exe"
+            ext = ("_d" if debug_build(sys.executable) else "") + ".exe"
             exename += ext
             exepath = os.path.dirname(sys.executable)
         else:
@@ -58,7 +68,8 @@ def tearDown(self):
         os.chdir(self.oldcwd)
 
     def run_embedded_interpreter(self, *args, env=None,
-                                 timeout=None, returncode=0, input=None):
+                                 timeout=None, returncode=0, input=None,
+                                 cwd=None):
         """Runs a test in the embedded interpreter"""
         cmd = [self.test_exe]
         cmd.extend(args)
@@ -72,7 +83,8 @@ def run_embedded_interpreter(self, *args, env=None,
                              stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE,
                              universal_newlines=True,
-                             env=env)
+                             env=env,
+                             cwd=cwd)
         try:
             (out, err) = p.communicate(input=input, timeout=timeout)
         except:
@@ -460,6 +472,11 @@ class InitConfigTests(EmbeddingTestsMixin, unittest.TestCase):
 
     EXPECTED_CONFIG = None
 
+    @classmethod
+    def tearDownClass(cls):
+        # clear cache
+        cls.EXPECTED_CONFIG = None
+
     def main_xoptions(self, xoptions_list):
         xoptions = {}
         for opt in xoptions_list:
@@ -470,7 +487,8 @@ def main_xoptions(self, xoptions_list):
                 xoptions[opt] = True
         return xoptions
 
-    def _get_expected_config(self, env):
+    def _get_expected_config_impl(self):
+        env = remove_python_envvars()
         code = textwrap.dedent('''
             import json
             import sys
@@ -489,23 +507,37 @@ def _get_expected_config(self, env):
         args = [sys.executable, '-S', '-c', code]
         proc = subprocess.run(args, env=env,
                               stdout=subprocess.PIPE,
-                              stderr=subprocess.STDOUT)
+                              stderr=subprocess.PIPE)
         if proc.returncode:
             raise Exception(f"failed to get the default config: "
                             f"stdout={proc.stdout!r} stderr={proc.stderr!r}")
         stdout = proc.stdout.decode('utf-8')
+        # ignore stderr
         try:
             return json.loads(stdout)
         except json.JSONDecodeError:
             self.fail(f"fail to decode stdout: {stdout!r}")
 
+    def _get_expected_config(self):
+        cls = InitConfigTests
+        if cls.EXPECTED_CONFIG is None:
+            cls.EXPECTED_CONFIG = self._get_expected_config_impl()
+
+        # get a copy
+        configs = {}
+        for config_key, config_value in cls.EXPECTED_CONFIG.items():
+            config = {}
+            for key, value in config_value.items():
+                if isinstance(value, list):
+                    value = value.copy()
+                config[key] = value
+            configs[config_key] = config
+        return configs
+
     def get_expected_config(self, expected_preconfig, expected, env, api,
                             modify_path_cb=None):
         cls = self.__class__
-        if cls.EXPECTED_CONFIG is None:
-            cls.EXPECTED_CONFIG = self._get_expected_config(env)
-        configs = {key: dict(value)
-                   for key, value in self.EXPECTED_CONFIG.items()}
+        configs = self._get_expected_config()
 
         pre_config = configs['pre_config']
         for key, value in expected_preconfig.items():
@@ -553,9 +585,10 @@ def get_expected_config(self, expected_preconfig, expected, env, api,
             if value is self.GET_DEFAULT_CONFIG:
                 expected[key] = config[key]
 
-        prepend_path = expected['pythonpath_env']
-        if prepend_path is not None:
-            expected['module_search_paths'] = [prepend_path, *expected['module_search_paths']]
+        pythonpath_env = expected['pythonpath_env']
+        if pythonpath_env is not None:
+            paths = pythonpath_env.split(os.path.pathsep)
+            expected['module_search_paths'] = [*paths, *expected['module_search_paths']]
         if modify_path_cb is not None:
             expected['module_search_paths'] = expected['module_search_paths'].copy()
             modify_path_cb(expected['module_search_paths'])
@@ -603,13 +636,19 @@ def check_global_config(self, configs):
         self.assertEqual(configs['global_config'], expected)
 
     def check_all_configs(self, testname, expected_config=None,
-                     expected_preconfig=None, modify_path_cb=None, stderr=None,
-                     *, api):
-        env = remove_python_envvars()
-
-        if api == API_ISOLATED:
+                          expected_preconfig=None, modify_path_cb=None,
+                          stderr=None, *, api, preconfig_api=None,
+                          env=None, ignore_stderr=False, cwd=None):
+        new_env = remove_python_envvars()
+        if env is not None:
+            new_env.update(env)
+        env = new_env
+
+        if preconfig_api is None:
+            preconfig_api = api
+        if preconfig_api == API_ISOLATED:
             default_preconfig = self.PRE_CONFIG_ISOLATED
-        elif api == API_PYTHON:
+        elif preconfig_api == API_PYTHON:
             default_preconfig = self.PRE_CONFIG_PYTHON
         else:
             default_preconfig = self.PRE_CONFIG_COMPAT
@@ -631,10 +670,11 @@ def check_all_configs(self, testname, expected_config=None,
                                  expected_config, env,
                                  api, modify_path_cb)
 
-        out, err = self.run_embedded_interpreter(testname, env=env)
+        out, err = self.run_embedded_interpreter(testname,
+                                                 env=env, cwd=cwd)
         if stderr is None and not expected_config['verbose']:
             stderr = ""
-        if stderr is not None:
+        if stderr is not None and not ignore_stderr:
             self.assertEqual(err.rstrip(), stderr)
         try:
             configs = json.loads(out)
@@ -965,6 +1005,268 @@ def test_init_dont_parse_argv(self):
         self.check_all_configs("test_init_dont_parse_argv", config, pre_config,
                                api=API_PYTHON)
 
+    def default_program_name(self, config):
+        if MS_WINDOWS:
+            program_name = 'python'
+            executable = self.test_exe
+        else:
+            program_name = 'python3'
+            if MACOS:
+                executable = self.test_exe
+            else:
+                executable = shutil.which(program_name) or ''
+        config.update({
+            'program_name': program_name,
+            'base_executable': executable,
+            'executable': executable,
+        })
+
+    def test_init_setpath(self):
+        # Test Py_SetPath()
+        config = self._get_expected_config()
+        paths = config['config']['module_search_paths']
+
+        config = {
+            'module_search_paths': paths,
+            'prefix': '',
+            'base_prefix': '',
+            'exec_prefix': '',
+            'base_exec_prefix': '',
+        }
+        self.default_program_name(config)
+        env = {'TESTPATH': os.path.pathsep.join(paths)}
+        self.check_all_configs("test_init_setpath", config,
+                               api=API_COMPAT, env=env,
+                               ignore_stderr=True)
+
+    def test_init_setpath_config(self):
+        # Test Py_SetPath() with PyConfig
+        config = self._get_expected_config()
+        paths = config['config']['module_search_paths']
+
+        config = {
+            # set by Py_SetPath()
+            'module_search_paths': paths,
+            'prefix': '',
+            'base_prefix': '',
+            'exec_prefix': '',
+            'base_exec_prefix': '',
+            # overriden by PyConfig
+            'program_name': 'conf_program_name',
+            'base_executable': 'conf_executable',
+            'executable': 'conf_executable',
+        }
+        env = {'TESTPATH': os.path.pathsep.join(paths)}
+        self.check_all_configs("test_init_setpath_config", config,
+                               api=API_PYTHON, env=env, ignore_stderr=True)
+
+    def module_search_paths(self, prefix=None, exec_prefix=None):
+        config = self._get_expected_config()
+        if prefix is None:
+            prefix = config['config']['prefix']
+        if exec_prefix is None:
+            exec_prefix = config['config']['prefix']
+        if MS_WINDOWS:
+            return config['config']['module_search_paths']
+        else:
+            ver = sys.version_info
+            return [
+                os.path.join(prefix, 'lib',
+                             f'python{ver.major}{ver.minor}.zip'),
+                os.path.join(prefix, 'lib',
+                             f'python{ver.major}.{ver.minor}'),
+                os.path.join(exec_prefix, 'lib',
+                             f'python{ver.major}.{ver.minor}', 'lib-dynload'),
+            ]
+
+    @contextlib.contextmanager
+    def tmpdir_with_python(self):
+        # Temporary directory with a copy of the Python program
+        with tempfile.TemporaryDirectory() as tmpdir:
+            # bpo-38234: On macOS and FreeBSD, the temporary directory
+            # can be symbolic link. For example, /tmp can be a symbolic link
+            # to /var/tmp. Call realpath() to resolve all symbolic links.
+            tmpdir = os.path.realpath(tmpdir)
+
+            if MS_WINDOWS:
+                # Copy pythonXY.dll (or pythonXY_d.dll)
+                ver = sys.version_info
+                dll = f'python{ver.major}{ver.minor}'
+                if debug_build(sys.executable):
+                    dll += '_d'
+                dll += '.dll'
+                dll = os.path.join(os.path.dirname(self.test_exe), dll)
+                dll_copy = os.path.join(tmpdir, os.path.basename(dll))
+                shutil.copyfile(dll, dll_copy)
+
+            # Copy Python program
+            exec_copy = os.path.join(tmpdir, os.path.basename(self.test_exe))
+            shutil.copyfile(self.test_exe, exec_copy)
+            shutil.copystat(self.test_exe, exec_copy)
+            self.test_exe = exec_copy
+
+            yield tmpdir
+
+    def test_init_setpythonhome(self):
+        # Test Py_SetPythonHome(home) with PYTHONPATH env var
+        config = self._get_expected_config()
+        paths = config['config']['module_search_paths']
+        paths_str = os.path.pathsep.join(paths)
+
+        for path in paths:
+            if not os.path.isdir(path):
+                continue
+            if os.path.exists(os.path.join(path, 'os.py')):
+                home = os.path.dirname(path)
+                break
+        else:
+            self.fail(f"Unable to find home in {paths!r}")
+
+        prefix = exec_prefix = home
+        ver = sys.version_info
+        expected_paths = self.module_search_paths(prefix=home, exec_prefix=home)
+
+        config = {
+            'home': home,
+            'module_search_paths': expected_paths,
+            'prefix': prefix,
+            'base_prefix': prefix,
+            'exec_prefix': exec_prefix,
+            'base_exec_prefix': exec_prefix,
+            'pythonpath_env': paths_str,
+        }
+        self.default_program_name(config)
+        env = {'TESTHOME': home, 'PYTHONPATH': paths_str}
+        self.check_all_configs("test_init_setpythonhome", config,
+                               api=API_COMPAT, env=env)
+
+    def copy_paths_by_env(self, config):
+        all_configs = self._get_expected_config()
+        paths = all_configs['config']['module_search_paths']
+        paths_str = os.path.pathsep.join(paths)
+        config['pythonpath_env'] = paths_str
+        env = {'PYTHONPATH': paths_str}
+        return env
+
+    @unittest.skipIf(MS_WINDOWS, 'Windows does not use pybuilddir.txt')
+    def test_init_pybuilddir(self):
+        # Test path configuration with pybuilddir.txt configuration file
+
+        with self.tmpdir_with_python() as tmpdir:
+            # pybuilddir.txt is a sub-directory relative to the current
+            # directory (tmpdir)
+            subdir = 'libdir'
+            libdir = os.path.join(tmpdir, subdir)
+            os.mkdir(libdir)
+
+            filename = os.path.join(tmpdir, 'pybuilddir.txt')
+            with open(filename, "w", encoding="utf8") as fp:
+                fp.write(subdir)
+
+            module_search_paths = self.module_search_paths()
+            module_search_paths[-1] = libdir
+
+            executable = self.test_exe
+            config = {
+                'base_executable': executable,
+                'executable': executable,
+                'module_search_paths': module_search_paths,
+            }
+            env = self.copy_paths_by_env(config)
+            self.check_all_configs("test_init_compat_config", config,
+                                   api=API_COMPAT, env=env,
+                                   ignore_stderr=True, cwd=tmpdir)
+
+    def test_init_pyvenv_cfg(self):
+        # Test path configuration with pyvenv.cfg configuration file
+
+        with self.tmpdir_with_python() as tmpdir, \
+             tempfile.TemporaryDirectory() as pyvenv_home:
+            ver = sys.version_info
+
+            if not MS_WINDOWS:
+                lib_dynload = os.path.join(pyvenv_home,
+                                           'lib',
+                                           f'python{ver.major}.{ver.minor}',
+                                           'lib-dynload')
+                os.makedirs(lib_dynload)
+            else:
+                lib_dynload = os.path.join(pyvenv_home, 'lib')
+                os.makedirs(lib_dynload)
+                # getpathp.c uses Lib\os.py as the LANDMARK
+                shutil.copyfile(os.__file__, os.path.join(lib_dynload, 'os.py'))
+
+            filename = os.path.join(tmpdir, 'pyvenv.cfg')
+            with open(filename, "w", encoding="utf8") as fp:
+                print("home = %s" % pyvenv_home, file=fp)
+                print("include-system-site-packages = false", file=fp)
+
+            paths = self.module_search_paths()
+            if not MS_WINDOWS:
+                paths[-1] = lib_dynload
+            else:
+                for index, path in enumerate(paths):
+                    if index == 0:
+                        paths[index] = os.path.join(tmpdir, os.path.basename(path))
+                    else:
+                        paths[index] = os.path.join(pyvenv_home, os.path.basename(path))
+                paths[-1] = pyvenv_home
+
+            executable = self.test_exe
+            exec_prefix = pyvenv_home
+            config = {
+                'base_exec_prefix': exec_prefix,
+                'exec_prefix': exec_prefix,
+                'base_executable': executable,
+                'executable': executable,
+                'module_search_paths': paths,
+            }
+            if MS_WINDOWS:
+                config['base_prefix'] = pyvenv_home
+                config['prefix'] = pyvenv_home
+            env = self.copy_paths_by_env(config)
+            self.check_all_configs("test_init_compat_config", config,
+                                   api=API_COMPAT, env=env,
+                                   ignore_stderr=True, cwd=tmpdir)
+
+    def test_global_pathconfig(self):
+        # Test C API functions getting the path configuration:
+        #
+        # - Py_GetExecPrefix()
+        # - Py_GetPath()
+        # - Py_GetPrefix()
+        # - Py_GetProgramFullPath()
+        # - Py_GetProgramName()
+        # - Py_GetPythonHome()
+        #
+        # The global path configuration (_Py_path_config) must be a copy
+        # of the path configuration of PyInterpreter.config (PyConfig).
+        ctypes = support.import_module('ctypes')
+        _testinternalcapi = support.import_module('_testinternalcapi')
+
+        def get_func(name):
+            func = getattr(ctypes.pythonapi, name)
+            func.argtypes = ()
+            func.restype = ctypes.c_wchar_p
+            return func
+
+        Py_GetPath = get_func('Py_GetPath')
+        Py_GetPrefix = get_func('Py_GetPrefix')
+        Py_GetExecPrefix = get_func('Py_GetExecPrefix')
+        Py_GetProgramName = get_func('Py_GetProgramName')
+        Py_GetProgramFullPath = get_func('Py_GetProgramFullPath')
+        Py_GetPythonHome = get_func('Py_GetPythonHome')
+
+        config = _testinternalcapi.get_configs()['config']
+
+        self.assertEqual(Py_GetPath().split(os.path.pathsep),
+                         config['module_search_paths'])
+        self.assertEqual(Py_GetPrefix(), config['prefix'])
+        self.assertEqual(Py_GetExecPrefix(), config['exec_prefix'])
+        self.assertEqual(Py_GetProgramName(), config['program_name'])
+        self.assertEqual(Py_GetProgramFullPath(), config['executable'])
+        self.assertEqual(Py_GetPythonHome(), config['home'])
+
 
 class AuditingTests(EmbeddingTestsMixin, unittest.TestCase):
     def test_open_code_hook(self):
diff --git a/Misc/NEWS.d/next/C API/2019-09-24-17-09-48.bpo-38234.d0bhEA.rst b/Misc/NEWS.d/next/C API/2019-09-24-17-09-48.bpo-38234.d0bhEA.rst
new file mode 100644
index 00000000000000..ba4cc312e69281
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2019-09-24-17-09-48.bpo-38234.d0bhEA.rst	
@@ -0,0 +1,3 @@
+:c:func:`Py_SetPath` now sets :data:`sys.executable` to the program full
+path (:c:func:`Py_GetProgramFullPath`) rather than to the program name
+(:c:func:`Py_GetProgramName`).
diff --git a/Modules/getpath.c b/Modules/getpath.c
index de32c3d51702b7..b727f66953b460 100644
--- a/Modules/getpath.c
+++ b/Modules/getpath.c
@@ -370,12 +370,15 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
                   const wchar_t *argv0_path,
                   wchar_t *prefix, size_t prefix_len, int *found)
 {
+    wchar_t path[MAXPATHLEN+1];
+    memset(path, 0, sizeof(path));
+    size_t path_len = Py_ARRAY_LENGTH(path);
+
     PyStatus status;
-    size_t n;
-    wchar_t *vpath;
 
     /* If PYTHONHOME is set, we believe it unconditionally */
     if (pathconfig->home) {
+        /* Path:  /  */
         if (safe_wcscpy(prefix, pathconfig->home, prefix_len) < 0) {
             return PATHLEN_ERR();
         }
@@ -387,27 +390,25 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }
-        status = joinpath(prefix, LANDMARK, prefix_len);
-        if (_PyStatus_EXCEPTION(status)) {
-            return status;
-        }
         *found = 1;
         return _PyStatus_OK();
     }
 
     /* Check to see if argv[0] is in the build directory */
-    if (safe_wcscpy(prefix, argv0_path, prefix_len) < 0) {
+    if (safe_wcscpy(path, argv0_path, path_len) < 0) {
         return PATHLEN_ERR();
     }
-    status = joinpath(prefix, L"Modules/Setup.local", prefix_len);
+    status = joinpath(path, L"Modules/Setup.local", path_len);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
 
-    if (isfile(prefix)) {
-        /* Check VPATH to see if argv0_path is in the build directory. */
-        vpath = Py_DecodeLocale(VPATH, NULL);
+    if (isfile(path)) {
+        /* Check VPATH to see if argv0_path is in the build directory.
+           VPATH can be empty. */
+        wchar_t *vpath = Py_DecodeLocale(VPATH, NULL);
         if (vpath != NULL) {
+            /* Path:  /  / Lib / LANDMARK */
             if (safe_wcscpy(prefix, argv0_path, prefix_len) < 0) {
                 return PATHLEN_ERR();
             }
@@ -428,6 +429,7 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
 
             if (ismodule(prefix, prefix_len)) {
                 *found = -1;
+                reduce(prefix);
                 return _PyStatus_OK();
             }
         }
@@ -440,7 +442,8 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
     }
 
     do {
-        n = wcslen(prefix);
+        /* Path:  /  / LANDMARK */
+        size_t n = wcslen(prefix);
         status = joinpath(prefix, calculate->lib_python, prefix_len);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
@@ -452,13 +455,15 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
 
         if (ismodule(prefix, prefix_len)) {
             *found = 1;
+            reduce(prefix);
             return _PyStatus_OK();
         }
         prefix[n] = L'\0';
         reduce(prefix);
     } while (prefix[0]);
 
-    /* Look at configure's PREFIX */
+    /* Look at configure's PREFIX.
+       Path:  /  / LANDMARK */
     if (safe_wcscpy(prefix, calculate->prefix, prefix_len) < 0) {
         return PATHLEN_ERR();
     }
@@ -473,6 +478,7 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
 
     if (ismodule(prefix, prefix_len)) {
         *found = 1;
+        reduce(prefix);
         return _PyStatus_OK();
     }
 
@@ -509,9 +515,6 @@ calculate_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
             return status;
         }
     }
-    else {
-        reduce(prefix);
-    }
     return _PyStatus_OK();
 }
 
@@ -546,6 +549,67 @@ calculate_set_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
 }
 
 
+static PyStatus
+calculate_pybuilddir(const wchar_t *argv0_path,
+                     wchar_t *exec_prefix, size_t exec_prefix_len,
+                     int *found)
+{
+    PyStatus status;
+
+    wchar_t filename[MAXPATHLEN+1];
+    memset(filename, 0, sizeof(filename));
+    size_t filename_len = Py_ARRAY_LENGTH(filename);
+
+    /* Check to see if argv[0] is in the build directory. "pybuilddir.txt"
+       is written by setup.py and contains the relative path to the location
+       of shared library modules.
+
+       Filename:  / "pybuilddir.txt" */
+    if (safe_wcscpy(filename, argv0_path, filename_len) < 0) {
+        return PATHLEN_ERR();
+    }
+    status = joinpath(filename, L"pybuilddir.txt", filename_len);
+    if (_PyStatus_EXCEPTION(status)) {
+        return status;
+    }
+
+    if (!isfile(filename)) {
+        return _PyStatus_OK();
+    }
+
+    FILE *fp = _Py_wfopen(filename, L"rb");
+    if (fp == NULL) {
+        errno = 0;
+        return _PyStatus_OK();
+    }
+
+    char buf[MAXPATHLEN + 1];
+    size_t n = fread(buf, 1, Py_ARRAY_LENGTH(buf) - 1, fp);
+    buf[n] = '\0';
+    fclose(fp);
+
+    size_t dec_len;
+    wchar_t *pybuilddir = _Py_DecodeUTF8_surrogateescape(buf, n, &dec_len);
+    if (!pybuilddir) {
+        return DECODE_LOCALE_ERR("pybuilddir.txt", dec_len);
+    }
+
+    /* Path:  /  */
+    if (safe_wcscpy(exec_prefix, argv0_path, exec_prefix_len) < 0) {
+        PyMem_RawFree(pybuilddir);
+        return PATHLEN_ERR();
+    }
+    status = joinpath(exec_prefix, pybuilddir, exec_prefix_len);
+    PyMem_RawFree(pybuilddir);
+    if (_PyStatus_EXCEPTION(status)) {
+        return status;
+    }
+
+    *found = -1;
+    return _PyStatus_OK();
+}
+
+
 /* search_for_exec_prefix requires that argv0_path be no more than
    MAXPATHLEN bytes long.
 */
@@ -556,10 +620,10 @@ search_for_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
                        int *found)
 {
     PyStatus status;
-    size_t n;
 
     /* If PYTHONHOME is set, we believe it unconditionally */
     if (pathconfig->home) {
+        /* Path:  /  / "lib-dynload" */
         wchar_t *delim = wcschr(pathconfig->home, DELIM);
         if (delim) {
             if (safe_wcscpy(exec_prefix, delim+1, exec_prefix_len) < 0) {
@@ -583,47 +647,15 @@ search_for_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
         return _PyStatus_OK();
     }
 
-    /* Check to see if argv[0] is in the build directory. "pybuilddir.txt"
-       is written by setup.py and contains the relative path to the location
-       of shared library modules. */
-    if (safe_wcscpy(exec_prefix, argv0_path, exec_prefix_len) < 0) {
-        return PATHLEN_ERR();
-    }
-    status = joinpath(exec_prefix, L"pybuilddir.txt", exec_prefix_len);
+    /* Check for pybuilddir.txt */
+    assert(*found == 0);
+    status = calculate_pybuilddir(argv0_path, exec_prefix, exec_prefix_len,
+                                  found);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
-
-    if (isfile(exec_prefix)) {
-        FILE *f = _Py_wfopen(exec_prefix, L"rb");
-        if (f == NULL) {
-            errno = 0;
-        }
-        else {
-            char buf[MAXPATHLEN + 1];
-            n = fread(buf, 1, Py_ARRAY_LENGTH(buf) - 1, f);
-            buf[n] = '\0';
-            fclose(f);
-
-            wchar_t *pybuilddir;
-            size_t dec_len;
-            pybuilddir = _Py_DecodeUTF8_surrogateescape(buf, n, &dec_len);
-            if (!pybuilddir) {
-                return DECODE_LOCALE_ERR("pybuilddir.txt", dec_len);
-            }
-
-            if (safe_wcscpy(exec_prefix, argv0_path, exec_prefix_len) < 0) {
-                return PATHLEN_ERR();
-            }
-            status = joinpath(exec_prefix, pybuilddir, exec_prefix_len);
-            PyMem_RawFree(pybuilddir );
-            if (_PyStatus_EXCEPTION(status)) {
-                return status;
-            }
-
-            *found = -1;
-            return _PyStatus_OK();
-        }
+    if (*found) {
+        return _PyStatus_OK();
     }
 
     /* Search from argv0_path, until root is found */
@@ -633,7 +665,8 @@ search_for_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
     }
 
     do {
-        n = wcslen(exec_prefix);
+        /* Path:  /  / "lib-dynload" */
+        size_t n = wcslen(exec_prefix);
         status = joinpath(exec_prefix, calculate->lib_python, exec_prefix_len);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
@@ -650,7 +683,9 @@ search_for_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
         reduce(exec_prefix);
     } while (exec_prefix[0]);
 
-    /* Look at configure's EXEC_PREFIX */
+    /* Look at configure's EXEC_PREFIX.
+
+       Path:  /  / "lib-dynload" */
     if (safe_wcscpy(exec_prefix, calculate->exec_prefix, exec_prefix_len) < 0) {
         return PATHLEN_ERR();
     }
@@ -962,43 +997,49 @@ calculate_read_pyenv(PyCalculatePath *calculate,
                      wchar_t *argv0_path, size_t argv0_path_len)
 {
     PyStatus status;
-    wchar_t tmpbuffer[MAXPATHLEN+1];
-    const size_t buflen = Py_ARRAY_LENGTH(tmpbuffer);
-    wchar_t *env_cfg = L"pyvenv.cfg";
+    const wchar_t *env_cfg = L"pyvenv.cfg";
     FILE *env_file;
 
-    if (safe_wcscpy(tmpbuffer, argv0_path, buflen) < 0) {
+    wchar_t filename[MAXPATHLEN+1];
+    const size_t filename_len = Py_ARRAY_LENGTH(filename);
+    memset(filename, 0, sizeof(filename));
+
+    /* Filename:  / "pyvenv.cfg" */
+    if (safe_wcscpy(filename, argv0_path, filename_len) < 0) {
         return PATHLEN_ERR();
     }
 
-    status = joinpath(tmpbuffer, env_cfg, buflen);
+    status = joinpath(filename, env_cfg, filename_len);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
-    env_file = _Py_wfopen(tmpbuffer, L"r");
+    env_file = _Py_wfopen(filename, L"r");
     if (env_file == NULL) {
         errno = 0;
 
-        reduce(tmpbuffer);
-        reduce(tmpbuffer);
-        status = joinpath(tmpbuffer, env_cfg, buflen);
+        /* Filename:  / "pyvenv.cfg" */
+        reduce(filename);
+        reduce(filename);
+        status = joinpath(filename, env_cfg, filename_len);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }
 
-        env_file = _Py_wfopen(tmpbuffer, L"r");
+        env_file = _Py_wfopen(filename, L"r");
         if (env_file == NULL) {
             errno = 0;
+            return _PyStatus_OK();
         }
     }
 
-    if (env_file == NULL) {
-        return _PyStatus_OK();
-    }
-
     /* Look for a 'home' variable and set argv0_path to it, if found */
-    if (_Py_FindEnvConfigValue(env_file, L"home", tmpbuffer, buflen)) {
-        if (safe_wcscpy(argv0_path, tmpbuffer, argv0_path_len) < 0) {
+    wchar_t home[MAXPATHLEN+1];
+    memset(home, 0, sizeof(home));
+
+    if (_Py_FindEnvConfigValue(env_file, L"home",
+                               home, Py_ARRAY_LENGTH(home))) {
+        if (safe_wcscpy(argv0_path, home, argv0_path_len) < 0) {
+            fclose(env_file);
             return PATHLEN_ERR();
         }
     }
@@ -1012,12 +1053,12 @@ calculate_zip_path(PyCalculatePath *calculate, const wchar_t *prefix,
                    wchar_t *zip_path, size_t zip_path_len)
 {
     PyStatus status;
-    if (safe_wcscpy(zip_path, prefix, zip_path_len) < 0) {
-        return PATHLEN_ERR();
-    }
 
     if (calculate->prefix_found > 0) {
         /* Use the reduced prefix returned by Py_GetPrefix() */
+        if (safe_wcscpy(zip_path, prefix, zip_path_len) < 0) {
+            return PATHLEN_ERR();
+        }
         reduce(zip_path);
         reduce(zip_path);
     }
@@ -1200,6 +1241,8 @@ calculate_path(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
         return status;
     }
 
+    /* If a pyvenv.cfg configure file is found,
+       argv0_path is overriden with its 'home' variable. */
     status = calculate_read_pyenv(calculate,
                                   argv0_path, Py_ARRAY_LENGTH(argv0_path));
     if (_PyStatus_EXCEPTION(status)) {
diff --git a/PC/getpathp.c b/PC/getpathp.c
index c4c0636dddeb1f..8bac592aefdb06 100644
--- a/PC/getpathp.c
+++ b/PC/getpathp.c
@@ -757,34 +757,34 @@ static void
 calculate_pyvenv_file(PyCalculatePath *calculate,
                       wchar_t *argv0_path, size_t argv0_path_len)
 {
-    wchar_t envbuffer[MAXPATHLEN+1];
+    wchar_t filename[MAXPATHLEN+1];
     const wchar_t *env_cfg = L"pyvenv.cfg";
 
-    wcscpy_s(envbuffer, MAXPATHLEN+1, argv0_path);
-    join(envbuffer, env_cfg);
+    /* Filename:  / "pyvenv.cfg" */
+    wcscpy_s(filename, MAXPATHLEN+1, argv0_path);
+    join(filename, env_cfg);
 
-    FILE *env_file = _Py_wfopen(envbuffer, L"r");
+    FILE *env_file = _Py_wfopen(filename, L"r");
     if (env_file == NULL) {
         errno = 0;
 
-        reduce(envbuffer);
-        reduce(envbuffer);
-        join(envbuffer, env_cfg);
+        /* Filename:  / "pyvenv.cfg" */
+        reduce(filename);
+        reduce(filename);
+        join(filename, env_cfg);
 
-        env_file = _Py_wfopen(envbuffer, L"r");
+        env_file = _Py_wfopen(filename, L"r");
         if (env_file == NULL) {
             errno = 0;
+            return;
         }
     }
 
-    if (env_file == NULL) {
-        return;
-    }
-
     /* Look for a 'home' variable and set argv0_path to it, if found */
-    wchar_t tmpbuffer[MAXPATHLEN+1];
-    if (_Py_FindEnvConfigValue(env_file, L"home", tmpbuffer, MAXPATHLEN)) {
-        wcscpy_s(argv0_path, argv0_path_len, tmpbuffer);
+    wchar_t home[MAXPATHLEN+1];
+    if (_Py_FindEnvConfigValue(env_file, L"home",
+                               home, Py_ARRAY_LENGTH(home))) {
+        wcscpy_s(argv0_path, argv0_path_len, home);
     }
     fclose(env_file);
 }
@@ -1099,11 +1099,11 @@ calculate_free(PyCalculatePath *calculate)
    - __PYVENV_LAUNCHER__ environment variable
    - GetModuleFileNameW(NULL): fully qualified path of the executable file of
      the current process
-   - .pth configuration file
+   - ._pth configuration file
    - pyvenv.cfg configuration file
    - Registry key "Software\Python\PythonCore\X.Y\PythonPath"
-     of HKEY_LOCAL_MACHINE and HKEY_CURRENT_USER where X.Y is the Python
-     version (major.minor).
+     of HKEY_CURRENT_USER and HKEY_LOCAL_MACHINE where X.Y is the Python
+     version.
 
    Outputs, 'pathconfig' fields:
 
diff --git a/Programs/_testembed.c b/Programs/_testembed.c
index c3ccc0ec325bb8..83c266b885af18 100644
--- a/Programs/_testembed.c
+++ b/Programs/_testembed.c
@@ -20,11 +20,12 @@
  * Executed via 'EmbeddingTests' in Lib/test/test_capi.py
  *********************************************************/
 
+/* Use path starting with "./" avoids a search along the PATH */
+#define PROGRAM_NAME L"./_testembed"
+
 static void _testembed_Py_Initialize(void)
 {
-    /* HACK: the "./" at front avoids a search along the PATH in
-       Modules/getpath.c */
-    Py_SetProgramName(L"./_testembed");
+    Py_SetProgramName(PROGRAM_NAME);
     Py_Initialize();
 }
 
@@ -363,8 +364,7 @@ config_set_wide_string_list(PyConfig *config, PyWideStringList *list,
 
 static void config_set_program_name(PyConfig *config)
 {
-    /* Use path starting with "./" avoids a search along the PATH */
-    const wchar_t *program_name = L"./_testembed";
+    const wchar_t *program_name = PROGRAM_NAME;
     config_set_string(config, &config->program_name, program_name);
 }
 
@@ -1263,7 +1263,7 @@ static int _audit_hook_run(const char *eventName, PyObject *args, void *userData
 static int test_audit_run_command(void)
 {
     AuditRunCommandTest test = {"cpython.run_command"};
-    wchar_t *argv[] = {L"./_testembed", L"-c", L"pass"};
+    wchar_t *argv[] = {PROGRAM_NAME, L"-c", L"pass"};
 
     Py_IgnoreEnvironmentFlag = 0;
     PySys_AddAuditHook(_audit_hook_run, (void*)&test);
@@ -1274,7 +1274,7 @@ static int test_audit_run_command(void)
 static int test_audit_run_file(void)
 {
     AuditRunCommandTest test = {"cpython.run_file"};
-    wchar_t *argv[] = {L"./_testembed", L"filename.py"};
+    wchar_t *argv[] = {PROGRAM_NAME, L"filename.py"};
 
     Py_IgnoreEnvironmentFlag = 0;
     PySys_AddAuditHook(_audit_hook_run, (void*)&test);
@@ -1312,21 +1312,21 @@ static int run_audit_run_test(int argc, wchar_t **argv, void *test)
 static int test_audit_run_interactivehook(void)
 {
     AuditRunCommandTest test = {"cpython.run_interactivehook", 10};
-    wchar_t *argv[] = {L"./_testembed"};
+    wchar_t *argv[] = {PROGRAM_NAME};
     return run_audit_run_test(Py_ARRAY_LENGTH(argv), argv, &test);
 }
 
 static int test_audit_run_startup(void)
 {
     AuditRunCommandTest test = {"cpython.run_startup", 10};
-    wchar_t *argv[] = {L"./_testembed"};
+    wchar_t *argv[] = {PROGRAM_NAME};
     return run_audit_run_test(Py_ARRAY_LENGTH(argv), argv, &test);
 }
 
 static int test_audit_run_stdin(void)
 {
     AuditRunCommandTest test = {"cpython.run_stdin"};
-    wchar_t *argv[] = {L"./_testembed"};
+    wchar_t *argv[] = {PROGRAM_NAME};
     return run_audit_run_test(Py_ARRAY_LENGTH(argv), argv, &test);
 }
 
@@ -1423,6 +1423,95 @@ static int test_init_sys_add(void)
 }
 
 
+static int test_init_setpath(void)
+{
+    char *env = getenv("TESTPATH");
+    if (!env) {
+        fprintf(stderr, "missing TESTPATH env var\n");
+        return 1;
+    }
+    wchar_t *path = Py_DecodeLocale(env, NULL);
+    if (path == NULL) {
+        fprintf(stderr, "failed to decode TESTPATH\n");
+        return 1;
+    }
+    Py_SetPath(path);
+    PyMem_RawFree(path);
+    putenv("TESTPATH=");
+
+    Py_Initialize();
+    dump_config();
+    Py_Finalize();
+    return 0;
+}
+
+
+static int test_init_setpath_config(void)
+{
+    PyStatus status;
+    PyPreConfig preconfig;
+    PyPreConfig_InitPythonConfig(&preconfig);
+
+    /* Explicitly preinitializes with Python preconfiguration to avoid
+      Py_SetPath() implicit preinitialization with compat preconfiguration. */
+    status = Py_PreInitialize(&preconfig);
+    if (PyStatus_Exception(status)) {
+        Py_ExitStatusException(status);
+    }
+
+    char *env = getenv("TESTPATH");
+    if (!env) {
+        fprintf(stderr, "missing TESTPATH env var\n");
+        return 1;
+    }
+    wchar_t *path = Py_DecodeLocale(env, NULL);
+    if (path == NULL) {
+        fprintf(stderr, "failed to decode TESTPATH\n");
+        return 1;
+    }
+    Py_SetPath(path);
+    PyMem_RawFree(path);
+    putenv("TESTPATH=");
+
+    PyConfig config;
+
+    status = PyConfig_InitPythonConfig(&config);
+    if (PyStatus_Exception(status)) {
+        Py_ExitStatusException(status);
+    }
+    config_set_string(&config, &config.program_name, L"conf_program_name");
+    config_set_string(&config, &config.executable, L"conf_executable");
+    init_from_config_clear(&config);
+
+    dump_config();
+    Py_Finalize();
+    return 0;
+}
+
+
+static int test_init_setpythonhome(void)
+{
+    char *env = getenv("TESTHOME");
+    if (!env) {
+        fprintf(stderr, "missing TESTHOME env var\n");
+        return 1;
+    }
+    wchar_t *home = Py_DecodeLocale(env, NULL);
+    if (home == NULL) {
+        fprintf(stderr, "failed to decode TESTHOME\n");
+        return 1;
+    }
+    Py_SetPythonHome(home);
+    PyMem_RawFree(home);
+    putenv("TESTHOME=");
+
+    Py_Initialize();
+    dump_config();
+    Py_Finalize();
+    return 0;
+}
+
+
 static void configure_init_main(PyConfig *config)
 {
     wchar_t* argv[] = {
@@ -1559,7 +1648,11 @@ static struct TestCase TestCases[] = {
     {"test_init_run_main", test_init_run_main},
     {"test_init_main", test_init_main},
     {"test_init_sys_add", test_init_sys_add},
+    {"test_init_setpath", test_init_setpath},
+    {"test_init_setpath_config", test_init_setpath_config},
+    {"test_init_setpythonhome", test_init_setpythonhome},
     {"test_run_main", test_run_main},
+
     {"test_open_code_hook", test_open_code_hook},
     {"test_audit", test_audit},
     {"test_audit_subinterpreter", test_audit_subinterpreter},
diff --git a/Python/pathconfig.c b/Python/pathconfig.c
index 09533100b46385..d5b8b1acff8a20 100644
--- a/Python/pathconfig.c
+++ b/Python/pathconfig.c
@@ -23,6 +23,7 @@ wchar_t *_Py_dll_path = NULL;
 static int
 copy_wstr(wchar_t **dst, const wchar_t *src)
 {
+    assert(*dst == NULL);
     if (src != NULL) {
         *dst = _PyMem_RawWcsdup(src);
         if (*dst == NULL) {
@@ -57,7 +58,10 @@ pathconfig_clear(_PyPathConfig *config)
     CLEAR(config->module_search_path);
     CLEAR(config->program_name);
     CLEAR(config->home);
+#ifdef MS_WINDOWS
     CLEAR(config->base_executable);
+#endif
+
 #undef CLEAR
 
     PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &old_alloc);
@@ -82,9 +86,11 @@ pathconfig_copy(_PyPathConfig *config, const _PyPathConfig *config2)
     COPY_ATTR(module_search_path);
     COPY_ATTR(program_name);
     COPY_ATTR(home);
+#ifdef MS_WINDOWS
     config->isolated = config2->isolated;
     config->site_import = config2->site_import;
     COPY_ATTR(base_executable);
+#endif
 
 #undef COPY_ATTR
 
@@ -127,7 +133,7 @@ _PyWideStringList_Join(const PyWideStringList *list, wchar_t sep)
     for (Py_ssize_t i=0; i < list->length; i++) {
         wchar_t *path = list->items[i];
         if (i != 0) {
-            *str++ = SEP;
+            *str++ = sep;
         }
         len = wcslen(path);
         memcpy(str, path, len * sizeof(wchar_t));
@@ -139,11 +145,11 @@ _PyWideStringList_Join(const PyWideStringList *list, wchar_t sep)
 }
 
 
+#ifdef MS_WINDOWS
 /* Initialize _Py_dll_path on Windows. Do nothing on other platforms. */
-PyStatus
-_PyPathConfig_Init(void)
+static PyStatus
+_PyPathConfig_InitDLLPath(void)
 {
-#ifdef MS_WINDOWS
     if (_Py_dll_path == NULL) {
         /* Already set: nothing to do */
         return _PyStatus_OK();
@@ -159,9 +165,9 @@ _PyPathConfig_Init(void)
     if (_Py_dll_path == NULL) {
         return _PyStatus_NO_MEMORY();
     }
-#endif
     return _PyStatus_OK();
 }
+#endif
 
 
 static PyStatus
@@ -172,6 +178,7 @@ pathconfig_set_from_config(_PyPathConfig *pathconfig, const PyConfig *config)
     _PyMem_SetDefaultAllocator(PYMEM_DOMAIN_RAW, &old_alloc);
 
     if (config->module_search_paths_set) {
+        PyMem_RawFree(pathconfig->module_search_path);
         pathconfig->module_search_path = _PyWideStringList_Join(&config->module_search_paths, DELIM);
         if (pathconfig->module_search_path == NULL) {
             goto no_memory;
@@ -180,17 +187,21 @@ pathconfig_set_from_config(_PyPathConfig *pathconfig, const PyConfig *config)
 
 #define COPY_CONFIG(PATH_ATTR, CONFIG_ATTR) \
         if (config->CONFIG_ATTR) { \
+            PyMem_RawFree(pathconfig->PATH_ATTR); \
+            pathconfig->PATH_ATTR = NULL; \
             if (copy_wstr(&pathconfig->PATH_ATTR, config->CONFIG_ATTR) < 0) { \
                 goto no_memory; \
             } \
         }
 
-    COPY_CONFIG(base_executable, base_executable);
     COPY_CONFIG(program_full_path, executable);
     COPY_CONFIG(prefix, prefix);
     COPY_CONFIG(exec_prefix, exec_prefix);
     COPY_CONFIG(program_name, program_name);
     COPY_CONFIG(home, home);
+#ifdef MS_WINDOWS
+    COPY_CONFIG(base_executable, base_executable);
+#endif
 
 #undef COPY_CONFIG
 
@@ -206,6 +217,20 @@ pathconfig_set_from_config(_PyPathConfig *pathconfig, const PyConfig *config)
 }
 
 
+PyStatus
+_PyConfig_WritePathConfig(const PyConfig *config)
+{
+#ifdef MS_WINDOWS
+    PyStatus status = _PyPathConfig_InitDLLPath();
+    if (_PyStatus_EXCEPTION(status)) {
+        return status;
+    }
+#endif
+
+    return pathconfig_set_from_config(&_Py_path_config, config);
+}
+
+
 static PyStatus
 config_init_module_search_paths(PyConfig *config, _PyPathConfig *pathconfig)
 {
@@ -326,18 +351,32 @@ config_calculate_pathconfig(PyConfig *config)
             } \
         }
 
+#ifdef MS_WINDOWS
+    if (config->executable != NULL && config->base_executable == NULL) {
+        /* If executable is set explicitly in the configuration,
+           ignore calculated base_executable: _PyConfig_InitPathConfig()
+           will copy executable to base_executable */
+    }
+    else {
+        COPY_ATTR(base_executable, base_executable);
+    }
+#endif
+
     COPY_ATTR(program_full_path, executable);
     COPY_ATTR(prefix, prefix);
     COPY_ATTR(exec_prefix, exec_prefix);
-    COPY_ATTR(base_executable, base_executable);
+
 #undef COPY_ATTR
 
+#ifdef MS_WINDOWS
+    /* If a ._pth file is found: isolated and site_import are overriden */
     if (pathconfig.isolated != -1) {
         config->isolated = pathconfig.isolated;
     }
     if (pathconfig.site_import != -1) {
         config->site_import = pathconfig.site_import;
     }
+#endif
 
     status = _PyStatus_OK();
     goto done;
@@ -356,9 +395,9 @@ _PyConfig_InitPathConfig(PyConfig *config)
 {
     /* Do we need to calculate the path? */
     if (!config->module_search_paths_set
-        || (config->executable == NULL)
-        || (config->prefix == NULL)
-        || (config->exec_prefix == NULL))
+        || config->executable == NULL
+        || config->prefix == NULL
+        || config->exec_prefix == NULL)
     {
         PyStatus status = config_calculate_pathconfig(config);
         if (_PyStatus_EXCEPTION(status)) {
@@ -416,11 +455,12 @@ pathconfig_global_init(void)
 {
     PyStatus status;
 
-    /* Initialize _Py_dll_path if needed */
-    status = _PyPathConfig_Init();
+#ifdef MS_WINDOWS
+    status = _PyPathConfig_InitDLLPath();
     if (_PyStatus_EXCEPTION(status)) {
         Py_ExitStatusException(status);
     }
+#endif
 
     if (_Py_path_config.module_search_path == NULL) {
         status = pathconfig_global_read(&_Py_path_config);
@@ -438,7 +478,9 @@ pathconfig_global_init(void)
     assert(_Py_path_config.module_search_path != NULL);
     assert(_Py_path_config.program_name != NULL);
     /* home can be NULL */
+#ifdef MS_WINDOWS
     assert(_Py_path_config.base_executable != NULL);
+#endif
 }
 
 
@@ -455,16 +497,15 @@ Py_SetPath(const wchar_t *path)
     PyMemAllocatorEx old_alloc;
     _PyMem_SetDefaultAllocator(PYMEM_DOMAIN_RAW, &old_alloc);
 
-    /* Getting the program name calls pathconfig_global_init() */
-    wchar_t *program_name = _PyMem_RawWcsdup(Py_GetProgramName());
+    /* Getting the program full path calls pathconfig_global_init() */
+    wchar_t *program_full_path = _PyMem_RawWcsdup(Py_GetProgramFullPath());
 
     PyMem_RawFree(_Py_path_config.program_full_path);
     PyMem_RawFree(_Py_path_config.prefix);
     PyMem_RawFree(_Py_path_config.exec_prefix);
     PyMem_RawFree(_Py_path_config.module_search_path);
 
-    /* Copy program_name to program_full_path  */
-    _Py_path_config.program_full_path = program_name;
+    _Py_path_config.program_full_path = program_full_path;
     _Py_path_config.prefix = _PyMem_RawWcsdup(L"");
     _Py_path_config.exec_prefix = _PyMem_RawWcsdup(L"");
     _Py_path_config.module_search_path = _PyMem_RawWcsdup(path);
diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c
index 4d7873fd522481..42a062e108651d 100644
--- a/Python/pylifecycle.c
+++ b/Python/pylifecycle.c
@@ -472,7 +472,7 @@ pyinit_core_reconfigure(_PyRuntimeState *runtime,
     config = &interp->config;
 
     if (config->_install_importlib) {
-        status = _PyPathConfig_Init();
+        status = _PyConfig_WritePathConfig(config);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }
@@ -641,7 +641,7 @@ pycore_init_import_warnings(PyInterpreterState *interp, PyObject *sysmod)
     }
 
     if (config->_install_importlib) {
-        status = _PyPathConfig_Init();
+        status = _PyConfig_WritePathConfig(config);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }

From c9893400652f38804aed0be8d8f70feda9465c47 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Thu, 26 Sep 2019 08:13:39 -0700
Subject: [PATCH 0745/2163] bpo-38239: Fix test_gdb for Link Time Optimization
 (LTO) (GH-16422)

(cherry picked from commit 64b4a3a2deabcd4103fac2759a311fe94159b4d1)

Co-authored-by: Victor Stinner 
---
 Lib/test/test_gdb.py                                  | 11 +++++++++--
 .../Tests/2019-09-26-15-48-36.bpo-38239.MfoVzY.rst    |  1 +
 2 files changed, 10 insertions(+), 2 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Tests/2019-09-26-15-48-36.bpo-38239.MfoVzY.rst

diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py
index e07d3273a4552d..e1060330550e34 100644
--- a/Lib/test/test_gdb.py
+++ b/Lib/test/test_gdb.py
@@ -255,8 +255,15 @@ def get_gdb_repr(self, source,
         # gdb can insert additional '\n' and space characters in various places
         # in its output, depending on the width of the terminal it's connected
         # to (using its "wrap_here" function)
-        m = re.match(r'.*#0\s+builtin_id\s+\(self\=.*,\s+v=\s*(.*?)?\)\s+at\s+\S*Python/bltinmodule.c.*',
-                     gdb_output, re.DOTALL)
+        m = re.search(
+            # Match '#0 builtin_id(self=..., v=...)'
+            r'#0\s+builtin_id\s+\(self\=.*,\s+v=\s*(.*?)?\)'
+            # Match ' at Python/bltinmodule.c'.
+            # bpo-38239: builtin_id() is defined in Python/bltinmodule.c,
+            # but accept any "Directory\file.c" to support Link Time
+            # Optimization (LTO).
+            r'\s+at\s+\S*[A-Za-z]+/[A-Za-z0-9_-]+\.c',
+            gdb_output, re.DOTALL)
         if not m:
             self.fail('Unexpected gdb output: %r\n%s' % (gdb_output, gdb_output))
         return m.group(1), gdb_output
diff --git a/Misc/NEWS.d/next/Tests/2019-09-26-15-48-36.bpo-38239.MfoVzY.rst b/Misc/NEWS.d/next/Tests/2019-09-26-15-48-36.bpo-38239.MfoVzY.rst
new file mode 100644
index 00000000000000..f79da29fa18288
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-09-26-15-48-36.bpo-38239.MfoVzY.rst
@@ -0,0 +1 @@
+Fix test_gdb for Link Time Optimization (LTO) builds.

From 1931132db33dd002cef0b19b8eaa219c1757797e Mon Sep 17 00:00:00 2001
From: Christian Heimes 
Date: Thu, 26 Sep 2019 22:53:09 +0200
Subject: [PATCH 0746/2163] [3.8] bpo-38275: Skip ssl tests for disabled
 versions (GH-16386) (GH-16425)

test_ssl now handles disabled TLS/SSL versions better. OpenSSL's crypto
policy and run-time settings are recognized and tests for disabled versions
are skipped.

Signed-off-by: Christian Heimes 

https://bugs.python.org/issue38275
(cherry picked from commit df6ac7e2b82d921a6e9ff5571b40c6dbcf635581)
---
 Lib/test/test_ssl.py                          | 196 +++++++++++++-----
 .../2019-09-25-14-40-57.bpo-38275.-kdveI.rst  |   4 +
 2 files changed, 149 insertions(+), 51 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Tests/2019-09-25-14-40-57.bpo-38275.-kdveI.rst

diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
index 9f3365aa864cf3..419506f4d3c072 100644
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -19,6 +19,7 @@
 import weakref
 import platform
 import sysconfig
+import functools
 try:
     import ctypes
 except ImportError:
@@ -143,6 +144,87 @@ def data_file(*name):
 OP_ENABLE_MIDDLEBOX_COMPAT = getattr(ssl, "OP_ENABLE_MIDDLEBOX_COMPAT", 0)
 
 
+def has_tls_protocol(protocol):
+    """Check if a TLS protocol is available and enabled
+
+    :param protocol: enum ssl._SSLMethod member or name
+    :return: bool
+    """
+    if isinstance(protocol, str):
+        assert protocol.startswith('PROTOCOL_')
+        protocol = getattr(ssl, protocol, None)
+        if protocol is None:
+            return False
+    if protocol in {
+        ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLS_SERVER,
+        ssl.PROTOCOL_TLS_CLIENT
+    }:
+        # auto-negotiate protocols are always available
+        return True
+    name = protocol.name
+    return has_tls_version(name[len('PROTOCOL_'):])
+
+
+@functools.lru_cache
+def has_tls_version(version):
+    """Check if a TLS/SSL version is enabled
+
+    :param version: TLS version name or ssl.TLSVersion member
+    :return: bool
+    """
+    if version == "SSLv2":
+        # never supported and not even in TLSVersion enum
+        return False
+
+    if isinstance(version, str):
+        version = ssl.TLSVersion.__members__[version]
+
+    # check compile time flags like ssl.HAS_TLSv1_2
+    if not getattr(ssl, f'HAS_{version.name}'):
+        return False
+
+    # check runtime and dynamic crypto policy settings. A TLS version may
+    # be compiled in but disabled by a policy or config option.
+    ctx = ssl.SSLContext()
+    if (
+            hasattr(ctx, 'minimum_version') and
+            ctx.minimum_version != ssl.TLSVersion.MINIMUM_SUPPORTED and
+            version < ctx.minimum_version
+    ):
+        return False
+    if (
+        hasattr(ctx, 'maximum_version') and
+        ctx.maximum_version != ssl.TLSVersion.MAXIMUM_SUPPORTED and
+        version > ctx.maximum_version
+    ):
+        return False
+
+    return True
+
+
+def requires_tls_version(version):
+    """Decorator to skip tests when a required TLS version is not available
+
+    :param version: TLS version name or ssl.TLSVersion member
+    :return:
+    """
+    def decorator(func):
+        @functools.wraps(func)
+        def wrapper(*args, **kw):
+            if not has_tls_version(version):
+                raise unittest.SkipTest(f"{version} is not available.")
+            else:
+                return func(*args, **kw)
+        return wrapper
+    return decorator
+
+
+requires_minimum_version = unittest.skipUnless(
+    hasattr(ssl.SSLContext, 'minimum_version'),
+    "required OpenSSL >= 1.1.0g"
+)
+
+
 def handle_error(prefix):
     exc_format = ' '.join(traceback.format_exception(*sys.exc_info()))
     if support.verbose:
@@ -1104,20 +1186,23 @@ def test_hostname_checks_common_name(self):
             with self.assertRaises(AttributeError):
                 ctx.hostname_checks_common_name = True
 
-    @unittest.skipUnless(hasattr(ssl.SSLContext, 'minimum_version'),
-                         "required OpenSSL 1.1.0g")
+    @requires_minimum_version
     @unittest.skipIf(IS_LIBRESSL, "see bpo-34001")
     def test_min_max_version(self):
         ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
         # OpenSSL default is MINIMUM_SUPPORTED, however some vendors like
         # Fedora override the setting to TLS 1.0.
+        minimum_range = {
+            # stock OpenSSL
+            ssl.TLSVersion.MINIMUM_SUPPORTED,
+            # Fedora 29 uses TLS 1.0 by default
+            ssl.TLSVersion.TLSv1,
+            # RHEL 8 uses TLS 1.2 by default
+            ssl.TLSVersion.TLSv1_2
+        }
+
         self.assertIn(
-            ctx.minimum_version,
-            {ssl.TLSVersion.MINIMUM_SUPPORTED,
-             # Fedora 29 uses TLS 1.0 by default
-             ssl.TLSVersion.TLSv1,
-             # RHEL 8 uses TLS 1.2 by default
-             ssl.TLSVersion.TLSv1_2}
+            ctx.minimum_version, minimum_range
         )
         self.assertEqual(
             ctx.maximum_version, ssl.TLSVersion.MAXIMUM_SUPPORTED
@@ -1163,8 +1248,8 @@ def test_min_max_version(self):
 
         ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1_1)
 
-        self.assertEqual(
-            ctx.minimum_version, ssl.TLSVersion.MINIMUM_SUPPORTED
+        self.assertIn(
+            ctx.minimum_version, minimum_range
         )
         self.assertEqual(
             ctx.maximum_version, ssl.TLSVersion.MAXIMUM_SUPPORTED
@@ -2717,6 +2802,8 @@ def test_echo(self):
         for protocol in PROTOCOLS:
             if protocol in {ssl.PROTOCOL_TLS_CLIENT, ssl.PROTOCOL_TLS_SERVER}:
                 continue
+            if not has_tls_protocol(protocol):
+                continue
             with self.subTest(protocol=ssl._PROTOCOL_NAMES[protocol]):
                 context = ssl.SSLContext(protocol)
                 context.load_cert_chain(CERTFILE)
@@ -3008,7 +3095,7 @@ def test_wrong_cert_tls12(self):
             else:
                 self.fail("Use of invalid cert should have failed!")
 
-    @unittest.skipUnless(ssl.HAS_TLSv1_3, "Test needs TLS 1.3")
+    @requires_tls_version('TLSv1_3')
     def test_wrong_cert_tls13(self):
         client_context, server_context, hostname = testing_context()
         # load client cert that is not signed by trusted CA
@@ -3103,8 +3190,7 @@ def test_ssl_cert_verify_error(self):
                     self.assertIn(msg, repr(e))
                     self.assertIn('certificate verify failed', repr(e))
 
-    @unittest.skipUnless(hasattr(ssl, 'PROTOCOL_SSLv2'),
-                         "OpenSSL is compiled without SSLv2 support")
+    @requires_tls_version('SSLv2')
     def test_protocol_sslv2(self):
         """Connecting to an SSLv2 server with various client options"""
         if support.verbose:
@@ -3113,7 +3199,7 @@ def test_protocol_sslv2(self):
         try_protocol_combo(ssl.PROTOCOL_SSLv2, ssl.PROTOCOL_SSLv2, True, ssl.CERT_OPTIONAL)
         try_protocol_combo(ssl.PROTOCOL_SSLv2, ssl.PROTOCOL_SSLv2, True, ssl.CERT_REQUIRED)
         try_protocol_combo(ssl.PROTOCOL_SSLv2, ssl.PROTOCOL_TLS, False)
-        if hasattr(ssl, 'PROTOCOL_SSLv3'):
+        if has_tls_version('SSLv3'):
             try_protocol_combo(ssl.PROTOCOL_SSLv2, ssl.PROTOCOL_SSLv3, False)
         try_protocol_combo(ssl.PROTOCOL_SSLv2, ssl.PROTOCOL_TLSv1, False)
         # SSLv23 client with specific SSL options
@@ -3130,7 +3216,7 @@ def test_PROTOCOL_TLS(self):
         """Connecting to an SSLv23 server with various client options"""
         if support.verbose:
             sys.stdout.write("\n")
-        if hasattr(ssl, 'PROTOCOL_SSLv2'):
+        if has_tls_version('SSLv2'):
             try:
                 try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_SSLv2, True)
             except OSError as x:
@@ -3139,34 +3225,36 @@ def test_PROTOCOL_TLS(self):
                     sys.stdout.write(
                         " SSL2 client to SSL23 server test unexpectedly failed:\n %s\n"
                         % str(x))
-        if hasattr(ssl, 'PROTOCOL_SSLv3'):
+        if has_tls_version('SSLv3'):
             try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_SSLv3, False)
         try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLS, True)
-        try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLSv1, 'TLSv1')
+        if has_tls_version('TLSv1'):
+            try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLSv1, 'TLSv1')
 
-        if hasattr(ssl, 'PROTOCOL_SSLv3'):
+        if has_tls_version('SSLv3'):
             try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_SSLv3, False, ssl.CERT_OPTIONAL)
         try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLS, True, ssl.CERT_OPTIONAL)
-        try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_OPTIONAL)
+        if has_tls_version('TLSv1'):
+            try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_OPTIONAL)
 
-        if hasattr(ssl, 'PROTOCOL_SSLv3'):
+        if has_tls_version('SSLv3'):
             try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_SSLv3, False, ssl.CERT_REQUIRED)
         try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLS, True, ssl.CERT_REQUIRED)
-        try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_REQUIRED)
+        if has_tls_version('TLSv1'):
+            try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_REQUIRED)
 
         # Server with specific SSL options
-        if hasattr(ssl, 'PROTOCOL_SSLv3'):
+        if has_tls_version('SSLv3'):
             try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_SSLv3, False,
                            server_options=ssl.OP_NO_SSLv3)
         # Will choose TLSv1
         try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLS, True,
                            server_options=ssl.OP_NO_SSLv2 | ssl.OP_NO_SSLv3)
-        try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLSv1, False,
-                           server_options=ssl.OP_NO_TLSv1)
-
+        if has_tls_version('TLSv1'):
+            try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLSv1, False,
+                               server_options=ssl.OP_NO_TLSv1)
 
-    @unittest.skipUnless(hasattr(ssl, 'PROTOCOL_SSLv3'),
-                         "OpenSSL is compiled without SSLv3 support")
+    @requires_tls_version('SSLv3')
     def test_protocol_sslv3(self):
         """Connecting to an SSLv3 server with various client options"""
         if support.verbose:
@@ -3174,7 +3262,7 @@ def test_protocol_sslv3(self):
         try_protocol_combo(ssl.PROTOCOL_SSLv3, ssl.PROTOCOL_SSLv3, 'SSLv3')
         try_protocol_combo(ssl.PROTOCOL_SSLv3, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_OPTIONAL)
         try_protocol_combo(ssl.PROTOCOL_SSLv3, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_REQUIRED)
-        if hasattr(ssl, 'PROTOCOL_SSLv2'):
+        if has_tls_version('SSLv2'):
             try_protocol_combo(ssl.PROTOCOL_SSLv3, ssl.PROTOCOL_SSLv2, False)
         try_protocol_combo(ssl.PROTOCOL_SSLv3, ssl.PROTOCOL_TLS, False,
                            client_options=ssl.OP_NO_SSLv3)
@@ -3184,6 +3272,7 @@ def test_protocol_sslv3(self):
             try_protocol_combo(ssl.PROTOCOL_SSLv3, ssl.PROTOCOL_TLS,
                                False, client_options=ssl.OP_NO_SSLv2)
 
+    @requires_tls_version('TLSv1')
     def test_protocol_tlsv1(self):
         """Connecting to a TLSv1 server with various client options"""
         if support.verbose:
@@ -3191,34 +3280,32 @@ def test_protocol_tlsv1(self):
         try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_TLSv1, 'TLSv1')
         try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_OPTIONAL)
         try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_REQUIRED)
-        if hasattr(ssl, 'PROTOCOL_SSLv2'):
+        if has_tls_version('SSLv2'):
             try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv2, False)
-        if hasattr(ssl, 'PROTOCOL_SSLv3'):
+        if has_tls_version('SSLv3'):
             try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv3, False)
         try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_TLS, False,
                            client_options=ssl.OP_NO_TLSv1)
 
-    @unittest.skipUnless(hasattr(ssl, "PROTOCOL_TLSv1_1"),
-                         "TLS version 1.1 not supported.")
+    @requires_tls_version('TLSv1_1')
     def test_protocol_tlsv1_1(self):
         """Connecting to a TLSv1.1 server with various client options.
            Testing against older TLS versions."""
         if support.verbose:
             sys.stdout.write("\n")
         try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_TLSv1_1, 'TLSv1.1')
-        if hasattr(ssl, 'PROTOCOL_SSLv2'):
+        if has_tls_version('SSLv2'):
             try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv2, False)
-        if hasattr(ssl, 'PROTOCOL_SSLv3'):
+        if has_tls_version('SSLv3'):
             try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv3, False)
         try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_TLS, False,
                            client_options=ssl.OP_NO_TLSv1_1)
 
         try_protocol_combo(ssl.PROTOCOL_TLS, ssl.PROTOCOL_TLSv1_1, 'TLSv1.1')
-        try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_TLSv1, False)
-        try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_TLSv1_1, False)
+        try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_TLSv1_2, False)
+        try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_TLSv1_1, False)
 
-    @unittest.skipUnless(hasattr(ssl, "PROTOCOL_TLSv1_2"),
-                         "TLS version 1.2 not supported.")
+    @requires_tls_version('TLSv1_2')
     def test_protocol_tlsv1_2(self):
         """Connecting to a TLSv1.2 server with various client options.
            Testing against older TLS versions."""
@@ -3227,9 +3314,9 @@ def test_protocol_tlsv1_2(self):
         try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_TLSv1_2, 'TLSv1.2',
                            server_options=ssl.OP_NO_SSLv3|ssl.OP_NO_SSLv2,
                            client_options=ssl.OP_NO_SSLv3|ssl.OP_NO_SSLv2,)
-        if hasattr(ssl, 'PROTOCOL_SSLv2'):
+        if has_tls_version('SSLv2'):
             try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv2, False)
-        if hasattr(ssl, 'PROTOCOL_SSLv3'):
+        if has_tls_version('SSLv3'):
             try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv3, False)
         try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_TLS, False,
                            client_options=ssl.OP_NO_TLSv1_2)
@@ -3672,7 +3759,7 @@ def test_version_basic(self):
                 self.assertIs(s.version(), None)
                 self.assertIs(s._sslobj, None)
                 s.connect((HOST, server.port))
-                if IS_OPENSSL_1_1_1 and ssl.HAS_TLSv1_3:
+                if IS_OPENSSL_1_1_1 and has_tls_version('TLSv1_3'):
                     self.assertEqual(s.version(), 'TLSv1.3')
                 elif ssl.OPENSSL_VERSION_INFO >= (1, 0, 2):
                     self.assertEqual(s.version(), 'TLSv1.2')
@@ -3681,8 +3768,7 @@ def test_version_basic(self):
             self.assertIs(s._sslobj, None)
             self.assertIs(s.version(), None)
 
-    @unittest.skipUnless(ssl.HAS_TLSv1_3,
-                         "test requires TLSv1.3 enabled OpenSSL")
+    @requires_tls_version('TLSv1_3')
     def test_tls1_3(self):
         context = ssl.SSLContext(ssl.PROTOCOL_TLS)
         context.load_cert_chain(CERTFILE)
@@ -3699,9 +3785,9 @@ def test_tls1_3(self):
                 })
                 self.assertEqual(s.version(), 'TLSv1.3')
 
-    @unittest.skipUnless(hasattr(ssl.SSLContext, 'minimum_version'),
-                         "required OpenSSL 1.1.0g")
-    def test_min_max_version(self):
+    @requires_minimum_version
+    @requires_tls_version('TLSv1_2')
+    def test_min_max_version_tlsv1_2(self):
         client_context, server_context, hostname = testing_context()
         # client TLSv1.0 to 1.2
         client_context.minimum_version = ssl.TLSVersion.TLSv1
@@ -3716,7 +3802,13 @@ def test_min_max_version(self):
                 s.connect((HOST, server.port))
                 self.assertEqual(s.version(), 'TLSv1.2')
 
+    @requires_minimum_version
+    @requires_tls_version('TLSv1_1')
+    def test_min_max_version_tlsv1_1(self):
+        client_context, server_context, hostname = testing_context()
         # client 1.0 to 1.2, server 1.0 to 1.1
+        client_context.minimum_version = ssl.TLSVersion.TLSv1
+        client_context.maximum_version = ssl.TLSVersion.TLSv1_2
         server_context.minimum_version = ssl.TLSVersion.TLSv1
         server_context.maximum_version = ssl.TLSVersion.TLSv1_1
 
@@ -3726,6 +3818,10 @@ def test_min_max_version(self):
                 s.connect((HOST, server.port))
                 self.assertEqual(s.version(), 'TLSv1.1')
 
+    @requires_minimum_version
+    @requires_tls_version('TLSv1_2')
+    def test_min_max_version_mismatch(self):
+        client_context, server_context, hostname = testing_context()
         # client 1.0, server 1.2 (mismatch)
         server_context.maximum_version = ssl.TLSVersion.TLSv1_2
         server_context.minimum_version = ssl.TLSVersion.TLSv1_2
@@ -3738,10 +3834,8 @@ def test_min_max_version(self):
                     s.connect((HOST, server.port))
                 self.assertIn("alert", str(e.exception))
 
-
-    @unittest.skipUnless(hasattr(ssl.SSLContext, 'minimum_version'),
-                         "required OpenSSL 1.1.0g")
-    @unittest.skipUnless(ssl.HAS_SSLv3, "requires SSLv3 support")
+    @requires_minimum_version
+    @requires_tls_version('SSLv3')
     def test_min_max_version_sslv3(self):
         client_context, server_context, hostname = testing_context()
         server_context.minimum_version = ssl.TLSVersion.SSLv3
@@ -4264,7 +4358,7 @@ def test_session_handling(self):
                                  'Session refers to a different SSLContext.')
 
 
-@unittest.skipUnless(ssl.HAS_TLSv1_3, "Test needs TLS 1.3")
+@unittest.skipUnless(has_tls_version('TLSv1_3'), "Test needs TLS 1.3")
 class TestPostHandshakeAuth(unittest.TestCase):
     def test_pha_setter(self):
         protocols = [
diff --git a/Misc/NEWS.d/next/Tests/2019-09-25-14-40-57.bpo-38275.-kdveI.rst b/Misc/NEWS.d/next/Tests/2019-09-25-14-40-57.bpo-38275.-kdveI.rst
new file mode 100644
index 00000000000000..893c0f137aeaff
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-09-25-14-40-57.bpo-38275.-kdveI.rst
@@ -0,0 +1,4 @@
+test_ssl now handles disabled TLS/SSL versions better. OpenSSL's crypto
+policy and run-time settings are recognized and tests for disabled versions
+are skipped. Tests also accept more TLS minimum_versions for platforms that
+override OpenSSL's default with strict settings.

From b2c2a0c02c6056dc73311a8a8a295249c454913a Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Fri, 27 Sep 2019 01:13:38 -0700
Subject: [PATCH 0747/2163] cleanup ababstractproperty in typing.py (GH-16432)

(cherry picked from commit 6ce03ec627680ce0829a5b3067fab7faed21b533)

Co-authored-by: HongWeipeng <961365124@qq.com>
---
 Lib/typing.py | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)

diff --git a/Lib/typing.py b/Lib/typing.py
index 048db3175e61d4..825be10812deef 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -17,7 +17,7 @@
 * Wrapper submodules for re and io related types.
 """
 
-from abc import abstractmethod, abstractproperty, ABCMeta
+from abc import abstractmethod, ABCMeta
 import collections
 import collections.abc
 import contextlib
@@ -1823,11 +1823,13 @@ class IO(Generic[AnyStr]):
 
     __slots__ = ()
 
-    @abstractproperty
+    @property
+    @abstractmethod
     def mode(self) -> str:
         pass
 
-    @abstractproperty
+    @property
+    @abstractmethod
     def name(self) -> str:
         pass
 
@@ -1923,23 +1925,28 @@ class TextIO(IO[str]):
 
     __slots__ = ()
 
-    @abstractproperty
+    @property
+    @abstractmethod
     def buffer(self) -> BinaryIO:
         pass
 
-    @abstractproperty
+    @property
+    @abstractmethod
     def encoding(self) -> str:
         pass
 
-    @abstractproperty
+    @property
+    @abstractmethod
     def errors(self) -> Optional[str]:
         pass
 
-    @abstractproperty
+    @property
+    @abstractmethod
     def line_buffering(self) -> bool:
         pass
 
-    @abstractproperty
+    @property
+    @abstractmethod
     def newlines(self) -> Any:
         pass
 

From 14ddca726a8cd337d0461934374b5e6bf65bf812 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Fri, 27 Sep 2019 04:18:24 -0700
Subject: [PATCH 0748/2163] bpo-38206: Clarify tp_dealloc requirements for heap
 allocated types. (GH-16248)

As mentioned in the bpo ticket, this mistake came up on two reviews:
- https://github.com/python/cpython/pull/16127GH-pullrequestreview-288312751
- https://github.com/python/cpython/pull/16071GH-pullrequestreview-287819525

Would be nice to have it documented in a more permanent place than 3.8's whatsnew entry.

https://bugs.python.org/issue38206

Automerge-Triggered-By: @encukou
(cherry picked from commit 5faff977adbe089e1f91a5916ccb2160a22dd292)

Co-authored-by: Ammar Askar 
---
 Doc/c-api/type.rst    |  3 ++-
 Doc/c-api/typeobj.rst | 24 ++++++++++++++++++++----
 2 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst
index 6416951ac5a94c..239355a802413e 100644
--- a/Doc/c-api/type.rst
+++ b/Doc/c-api/type.rst
@@ -118,7 +118,8 @@ The following functions and structs are used to create
 
 .. c:function:: PyObject* PyType_FromSpecWithBases(PyType_Spec *spec, PyObject *bases)
 
-   Creates and returns a heap type object from the *spec*.
+   Creates and returns a heap type object from the *spec*
+   (:const:`Py_TPFLAGS_HEAPTYPE`).
 
    If *bases* is a tuple, the created heap type contains all types contained
    in it as base types.
diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst
index 7c7a79129ccca2..bb767500743098 100644
--- a/Doc/c-api/typeobj.rst
+++ b/Doc/c-api/typeobj.rst
@@ -654,9 +654,9 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    the instance is still in existence, but there are no references to it.  The
    destructor function should free all references which the instance owns, free all
    memory buffers owned by the instance (using the freeing function corresponding
-   to the allocation function used to allocate the buffer), and finally (as its
-   last action) call the type's :c:member:`~PyTypeObject.tp_free` function.  If the type is not
-   subtypable (doesn't have the :const:`Py_TPFLAGS_BASETYPE` flag bit set), it is
+   to the allocation function used to allocate the buffer), and call the type's
+   :c:member:`~PyTypeObject.tp_free` function.  If the type is not subtypable
+   (doesn't have the :const:`Py_TPFLAGS_BASETYPE` flag bit set), it is
    permissible to call the object deallocator directly instead of via
    :c:member:`~PyTypeObject.tp_free`.  The object deallocator should be the one used to allocate the
    instance; this is normally :c:func:`PyObject_Del` if the instance was allocated
@@ -664,6 +664,21 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    :c:func:`PyObject_GC_Del` if the instance was allocated using
    :c:func:`PyObject_GC_New` or :c:func:`PyObject_GC_NewVar`.
 
+   Finally, if the type is heap allocated (:const:`Py_TPFLAGS_HEAPTYPE`), the
+   deallocator should decrement the reference count for its type object after
+   calling the type deallocator. In order to avoid dangling pointers, the
+   recommended way to achieve this is:
+
+   .. code-block:: c
+
+     static void foo_dealloc(foo_object *self) {
+         PyTypeObject *tp = Py_TYPE(self);
+         // free references and buffers here
+         tp->tp_free(self);
+         Py_DECREF(tp);
+     }
+
+
    **Inheritance:**
 
    This field is inherited by subtypes.
@@ -1021,7 +1036,8 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    .. data:: Py_TPFLAGS_HEAPTYPE
 
-      This bit is set when the type object itself is allocated on the heap.  In this
+      This bit is set when the type object itself is allocated on the heap, for
+      example, types created dynamically using :c:func:`PyType_FromSpec`.  In this
       case, the :attr:`ob_type` field of its instances is considered a reference to
       the type, and the type object is INCREF'ed when a new instance is created, and
       DECREF'ed when an instance is destroyed (this does not apply to instances of

From 6447b9f9bd27e1f6b04cef674dd3a7ab27bf4f28 Mon Sep 17 00:00:00 2001
From: "Miss Islington (bot)"
 <31488909+miss-islington@users.noreply.github.com>
Date: Fri, 27 Sep 2019 13:19:41 -0700
Subject: [PATCH 0749/2163] bpo-38243, xmlrpc.server: Escape the server_title
 (GH-16373)

Escape the server title of xmlrpc.server.DocXMLRPCServer
when rendering the document page as HTML.
(cherry picked from commit e8650a4f8c7fb76f570d4ca9c1fbe44e91c8dfaa)

Co-authored-by: Dong-hee Na 
---
 Lib/test/test_docxmlrpc.py                       | 16 ++++++++++++++++
 Lib/xmlrpc/server.py                             |  3 ++-
 .../2019-09-25-13-21-09.bpo-38243.1pfz24.rst     |  3 +++
 3 files changed, 21 insertions(+), 1 deletion(-)
 create mode 100644 Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst

diff --git a/Lib/test/test_docxmlrpc.py b/Lib/test/test_docxmlrpc.py
index 116e626740df85..7d3e30cbee964a 100644
--- a/Lib/test/test_docxmlrpc.py
+++ b/Lib/test/test_docxmlrpc.py
@@ -1,5 +1,6 @@
 from xmlrpc.server import DocXMLRPCServer
 import http.client
+import re
 import sys
 import threading
 import unittest
@@ -192,6 +193,21 @@ def test_annotations(self):
              b'method_annotation(x: bytes)
'), response.read()) + def test_server_title_escape(self): + # bpo-38243: Ensure that the server title and documentation + # are escaped for HTML. + self.serv.set_server_title('test_title - + @@ -62,7 +62,7 @@

Navigation

next |
  • - previous |
  • Navigation
  • - 3.9.0a0 Documentation » + 3.9.0a1 Documentation »
  • @@ -240,7 +240,8 @@

    Edit menu (Shell and Editor)Table of Contents

    Previous topic

    -

    tkinter.scrolledtext — Scrolled Text Widget

    +

    tkinter.tix — Extension widgets for Tk

    Next topic

    Other Graphical User Interface Packages

    @@ -919,7 +920,7 @@

    Navigation

    next |
  • - previous |
  • Navigation
  • - 3.9.0a0 Documentation » + 3.9.0a1 Documentation »
  • @@ -959,11 +960,11 @@

    Navigation



    - Last updated on Sep 01, 2019. + Last updated on Nov 24, 2019. Found a bug?
    - Created using Sphinx 2.1.2. + Created using Sphinx 2.1.1. diff --git a/Lib/idlelib/idle_test/mock_idle.py b/Lib/idlelib/idle_test/mock_idle.py index f279a52fd511f0..71fa480ce4d05c 100644 --- a/Lib/idlelib/idle_test/mock_idle.py +++ b/Lib/idlelib/idle_test/mock_idle.py @@ -40,8 +40,9 @@ def __call__(self, *args, **kwds): class Editor: '''Minimally imitate editor.EditorWindow class. ''' - def __init__(self, flist=None, filename=None, key=None, root=None): - self.text = Text() + def __init__(self, flist=None, filename=None, key=None, root=None, + text=None): # Allow real Text with mock Editor. + self.text = text or Text() self.undo = UndoDelegator() def get_selection_indices(self): diff --git a/Lib/idlelib/idle_test/test_format.py b/Lib/idlelib/idle_test/test_format.py index 20b5970f4982da..a79bb515089e7b 100644 --- a/Lib/idlelib/idle_test/test_format.py +++ b/Lib/idlelib/idle_test/test_format.py @@ -611,37 +611,33 @@ def test_change_indentwidth(self, askinteger): class RstripTest(unittest.TestCase): - def test_rstrip_line(self): - editor = MockEditor() - text = editor.text - do_rstrip = ft.Rstrip(editor).do_rstrip - eq = self.assertEqual + @classmethod + def setUpClass(cls): + requires('gui') + cls.root = Tk() + cls.root.withdraw() + cls.text = Text(cls.root) + cls.editor = MockEditor(text=cls.text) + cls.do_rstrip = ft.Rstrip(cls.editor).do_rstrip + + @classmethod + def tearDownClass(cls): + del cls.text, cls.do_rstrip, cls.editor + cls.root.update_idletasks() + cls.root.destroy() + del cls.root - do_rstrip() - eq(text.get('1.0', 'insert'), '') - text.insert('1.0', ' ') - do_rstrip() - eq(text.get('1.0', 'insert'), '') - text.insert('1.0', ' \n') - do_rstrip() - eq(text.get('1.0', 'insert'), '\n') - - def test_rstrip_multiple(self): - editor = MockEditor() - # Comment above, uncomment 3 below to test with real Editor & Text. - #from idlelib.editor import EditorWindow as Editor - #from tkinter import Tk - #editor = Editor(root=Tk()) - text = editor.text - do_rstrip = ft.Rstrip(editor).do_rstrip + def tearDown(self): + self.text.delete('1.0', 'end-1c') + def test_rstrip_lines(self): original = ( "Line with an ending tab \n" "Line ending in 5 spaces \n" "Linewithnospaces\n" " indented line\n" " indented line with trailing space \n" - " ") + " \n") stripped = ( "Line with an ending tab\n" "Line ending in 5 spaces\n" @@ -649,9 +645,23 @@ def test_rstrip_multiple(self): " indented line\n" " indented line with trailing space\n") - text.insert('1.0', original) - do_rstrip() - self.assertEqual(text.get('1.0', 'insert'), stripped) + self.text.insert('1.0', original) + self.do_rstrip() + self.assertEqual(self.text.get('1.0', 'insert'), stripped) + + def test_rstrip_end(self): + text = self.text + for code in ('', '\n', '\n\n\n'): + with self.subTest(code=code): + text.insert('1.0', code) + self.do_rstrip() + self.assertEqual(text.get('1.0','end-1c'), '') + for code in ('a\n', 'a\n\n', 'a\n\n\n'): + with self.subTest(code=code): + text.delete('1.0', 'end-1c') + text.insert('1.0', code) + self.do_rstrip() + self.assertEqual(text.get('1.0','end-1c'), 'a\n') if __name__ == '__main__': diff --git a/Misc/NEWS.d/next/IDLE/2019-11-23-21-50-57.bpo-38862.KQ9A0m.rst b/Misc/NEWS.d/next/IDLE/2019-11-23-21-50-57.bpo-38862.KQ9A0m.rst new file mode 100644 index 00000000000000..14bab9e854bdce --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2019-11-23-21-50-57.bpo-38862.KQ9A0m.rst @@ -0,0 +1,2 @@ +'Strip Trailing Whitespace' on the Format menu removes extra newlines +at the end of non-shell files. From 65c92c5870944b972a879031abd4c20c4f0d7981 Mon Sep 17 00:00:00 2001 From: "Bruno P. Kinoshita" Date: Wed, 27 Nov 2019 17:49:37 +1300 Subject: [PATCH 0963/2163] [3.8] bpo-38688, shutil.copytree: consume iterator and create list of entries to prevent infinite recursion (GH-17397) (cherry picked from commit 9bbcbc9f6dfe1368fe7330b117707f828e6a2c18) Co-authored-by: Bruno P. Kinoshita --- Lib/shutil.py | 13 +++++++------ Lib/test/test_shutil.py | 12 ++++++++++++ .../2019-11-22-10-45-03.bpo-38668.iKx23z.rst | 5 +++++ 3 files changed, 24 insertions(+), 6 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-11-22-10-45-03.bpo-38668.iKx23z.rst diff --git a/Lib/shutil.py b/Lib/shutil.py index 1e89256cc34443..755ce392e6d9ae 100644 --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -442,7 +442,7 @@ def _ignore_patterns(path, names): def _copytree(entries, src, dst, symlinks, ignore, copy_function, ignore_dangling_symlinks, dirs_exist_ok=False): if ignore is not None: - ignored_names = ignore(src, set(os.listdir(src))) + ignored_names = ignore(src, {x.name for x in entries}) else: ignored_names = set() @@ -543,11 +543,12 @@ def copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, """ sys.audit("shutil.copytree", src, dst) - with os.scandir(src) as entries: - return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks, - ignore=ignore, copy_function=copy_function, - ignore_dangling_symlinks=ignore_dangling_symlinks, - dirs_exist_ok=dirs_exist_ok) + with os.scandir(src) as itr: + entries = list(itr) + return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks, + ignore=ignore, copy_function=copy_function, + ignore_dangling_symlinks=ignore_dangling_symlinks, + dirs_exist_ok=dirs_exist_ok) if hasattr(os.stat_result, 'st_file_attributes'): # Special handling for directory junctions to make them behave like diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py index b98e7dc798abe7..286e333a8aa9cd 100644 --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -1605,6 +1605,18 @@ def test_copytree_return_value(self): rv = shutil.copytree(src_dir, dst_dir) self.assertEqual(['foo'], os.listdir(rv)) + def test_copytree_subdirectory(self): + # copytree where dst is a subdirectory of src, see Issue 38688 + base_dir = self.mkdtemp() + self.addCleanup(shutil.rmtree, base_dir, ignore_errors=True) + src_dir = os.path.join(base_dir, "t", "pg") + dst_dir = os.path.join(src_dir, "somevendor", "1.0") + os.makedirs(src_dir) + src = os.path.join(src_dir, 'pol') + write_file(src, 'pol') + rv = shutil.copytree(src_dir, dst_dir) + self.assertEqual(['pol'], os.listdir(rv)) + class TestWhich(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Library/2019-11-22-10-45-03.bpo-38668.iKx23z.rst b/Misc/NEWS.d/next/Library/2019-11-22-10-45-03.bpo-38668.iKx23z.rst new file mode 100644 index 00000000000000..28b82ab1619e35 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-11-22-10-45-03.bpo-38668.iKx23z.rst @@ -0,0 +1,5 @@ +Calling func:`shutil.copytree` to copy a directory tree from one directory +to another subdirectory resulted in an endless loop and a RecursionError. A +fix was added to consume an iterator and create the list of the entries to +be copied, avoiding the recursion for newly created directories. Patch by +Bruno P. Kinoshita. From 0f9c9d53283420a570850aa92869d032b40d4fba Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 27 Nov 2019 00:53:52 -0800 Subject: [PATCH 0964/2163] bpo-38524: document implicit and explicit calling of descriptors' __set_name__ (GH-17364) (cherry picked from commit 1bddf890e595a865414645c6041733043c4081f8) Co-authored-by: Florian Dahlitz --- Doc/reference/datamodel.rst | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 1703be6e77ad94..ae24adeacb1677 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1655,8 +1655,19 @@ class' :attr:`~object.__dict__`. Called at the time the owning class *owner* is created. The descriptor has been assigned to *name*. - .. versionadded:: 3.6 + .. note:: + + ``__set_name__`` is only called implicitly as part of the ``type`` constructor, so + it will need to be called explicitly with the appropriate parameters when a + descriptor is added to a class after initial creation:: + descr = custom_descriptor() + cls.attr = descr + descr.__set_name__(cls, 'attr') + + See :ref:`class-object-creation` for more details. + + .. versionadded:: 3.6 The attribute :attr:`__objclass__` is interpreted by the :mod:`inspect` module as specifying the class where this object was defined (setting this From d21b8e82dda3caf0989bb39bc0e0ce2bfef97081 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 27 Nov 2019 21:23:14 -0800 Subject: [PATCH 0965/2163] bpo-26730: Fix SpooledTemporaryFile data corruption (GH-17400) SpooledTemporaryFile.rollback() might cause data corruption when it is in text mode. Co-Authored-By: Serhiy Storchaka (cherry picked from commit ea9835c5d154ab6a54eed627958473b6768b28cc) Co-authored-by: Inada Naoki --- Doc/library/tempfile.rst | 4 +-- Lib/tempfile.py | 15 ++++++----- Lib/test/test_tempfile.py | 25 +++++++++++-------- .../2019-11-27-16-30-02.bpo-26730.56cdBn.rst | 2 ++ 4 files changed, 27 insertions(+), 19 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-11-27-16-30-02.bpo-26730.56cdBn.rst diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index fff7a7a03eb81d..a59817c1039210 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -105,8 +105,8 @@ The module defines the following user-callable items: causes the file to roll over to an on-disk file regardless of its size. The returned object is a file-like object whose :attr:`_file` attribute - is either an :class:`io.BytesIO` or :class:`io.StringIO` object (depending on - whether binary or text *mode* was specified) or a true file + is either an :class:`io.BytesIO` or :class:`io.TextIOWrapper` object + (depending on whether binary or text *mode* was specified) or a true file object, depending on whether :func:`rollover` has been called. This file-like object can be used in a :keyword:`with` statement, just like a normal file. diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 45709cb06cf4e6..62875540f8b920 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -633,10 +633,9 @@ def __init__(self, max_size=0, mode='w+b', buffering=-1, if 'b' in mode: self._file = _io.BytesIO() else: - # Setting newline="\n" avoids newline translation; - # this is important because otherwise on Windows we'd - # get double newline translation upon rollover(). - self._file = _io.StringIO(newline="\n") + self._file = _io.TextIOWrapper(_io.BytesIO(), + encoding=encoding, errors=errors, + newline=newline) self._max_size = max_size self._rolled = False self._TemporaryFileArgs = {'mode': mode, 'buffering': buffering, @@ -656,8 +655,12 @@ def rollover(self): newfile = self._file = TemporaryFile(**self._TemporaryFileArgs) del self._TemporaryFileArgs - newfile.write(file.getvalue()) - newfile.seek(file.tell(), 0) + pos = file.tell() + if hasattr(newfile, 'buffer'): + newfile.buffer.write(file.detach().getvalue()) + else: + newfile.write(file.getvalue()) + newfile.seek(pos, 0) self._rolled = True diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index f995f6c9bfaf00..232c5dae10fdf4 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -1114,7 +1114,8 @@ def test_properties(self): def test_text_mode(self): # Creating a SpooledTemporaryFile with a text mode should produce # a file object reading and writing (Unicode) text strings. - f = tempfile.SpooledTemporaryFile(mode='w+', max_size=10) + f = tempfile.SpooledTemporaryFile(mode='w+', max_size=10, + encoding="utf-8") f.write("abc\n") f.seek(0) self.assertEqual(f.read(), "abc\n") @@ -1124,9 +1125,9 @@ def test_text_mode(self): self.assertFalse(f._rolled) self.assertEqual(f.mode, 'w+') self.assertIsNone(f.name) - self.assertIsNone(f.newlines) - self.assertIsNone(f.encoding) - self.assertIsNone(f.errors) + self.assertEqual(f.newlines, os.linesep) + self.assertEqual(f.encoding, "utf-8") + self.assertEqual(f.errors, "strict") f.write("xyzzy\n") f.seek(0) @@ -1139,8 +1140,8 @@ def test_text_mode(self): self.assertEqual(f.mode, 'w+') self.assertIsNotNone(f.name) self.assertEqual(f.newlines, os.linesep) - self.assertIsNotNone(f.encoding) - self.assertIsNotNone(f.errors) + self.assertEqual(f.encoding, "utf-8") + self.assertEqual(f.errors, "strict") def test_text_newline_and_encoding(self): f = tempfile.SpooledTemporaryFile(mode='w+', max_size=10, @@ -1152,13 +1153,15 @@ def test_text_newline_and_encoding(self): self.assertFalse(f._rolled) self.assertEqual(f.mode, 'w+') self.assertIsNone(f.name) - self.assertIsNone(f.newlines) - self.assertIsNone(f.encoding) - self.assertIsNone(f.errors) + self.assertIsNotNone(f.newlines) + self.assertEqual(f.encoding, "utf-8") + self.assertEqual(f.errors, "ignore") - f.write("\u039B" * 20 + "\r\n") + f.write("\u039C" * 10 + "\r\n") + f.write("\u039D" * 20) f.seek(0) - self.assertEqual(f.read(), "\u039B\r\n" + ("\u039B" * 20) + "\r\n") + self.assertEqual(f.read(), + "\u039B\r\n" + ("\u039C" * 10) + "\r\n" + ("\u039D" * 20)) self.assertTrue(f._rolled) self.assertEqual(f.mode, 'w+') self.assertIsNotNone(f.name) diff --git a/Misc/NEWS.d/next/Library/2019-11-27-16-30-02.bpo-26730.56cdBn.rst b/Misc/NEWS.d/next/Library/2019-11-27-16-30-02.bpo-26730.56cdBn.rst new file mode 100644 index 00000000000000..a92b90a4956053 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-11-27-16-30-02.bpo-26730.56cdBn.rst @@ -0,0 +1,2 @@ +Fix ``SpooledTemporaryFile.rollover()`` might corrupt the file when it is in +text mode. Patch by Serhiy Storchaka. From c0db88f6abbace79644b2aca2290bf41b1a37174 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 27 Nov 2019 21:29:02 -0800 Subject: [PATCH 0966/2163] bpo-38524: clarify example a bit and improve formatting (GH-17406) (cherry picked from commit 02519f75d15b063914a11351da30178ca4ceb54b) Co-authored-by: Tal Einat --- Doc/reference/datamodel.rst | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index ae24adeacb1677..46d50ad600ff52 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1657,13 +1657,16 @@ class' :attr:`~object.__dict__`. .. note:: - ``__set_name__`` is only called implicitly as part of the ``type`` constructor, so - it will need to be called explicitly with the appropriate parameters when a - descriptor is added to a class after initial creation:: + :meth:`__set_name__` is only called implicitly as part of the + :class:`type` constructor, so it will need to be called explicitly with + the appropriate parameters when a descriptor is added to a class after + initial creation:: + class A: + pass descr = custom_descriptor() - cls.attr = descr - descr.__set_name__(cls, 'attr') + A.attr = descr + descr.__set_name__(A, 'attr') See :ref:`class-object-creation` for more details. From 18d8edbbb6626ac9cdf1152a720811beb2230b33 Mon Sep 17 00:00:00 2001 From: Tzu-ping Chung Date: Thu, 28 Nov 2019 23:44:08 +0800 Subject: [PATCH 0967/2163] bpo-38928: Remove upgrade_dependencies() from venv doc (GH-17410) --- Doc/library/venv.rst | 8 -------- 1 file changed, 8 deletions(-) diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst index 3483ef36a2c33e..31a3f41e4724ce 100644 --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@ -186,14 +186,6 @@ creation according to their needs, the :class:`EnvBuilder` class. Installs activation scripts appropriate to the platform into the virtual environment. - .. method:: upgrade_dependencies(context) - - Upgrades the core venv dependency packages (currently ``pip`` and - ``setuptools``) in the environment. This is done by shelling out to the - ``pip`` executable in the environment. - - .. versionadded:: 3.8 - .. method:: post_setup(context) A placeholder method which can be overridden in third party From b74a6f14b94d36fb72b1344663e81776bf450847 Mon Sep 17 00:00:00 2001 From: Steve Dower Date: Thu, 28 Nov 2019 08:46:23 -0800 Subject: [PATCH 0968/2163] bpo-38920: Add audit hooks for when sys.excepthook and sys.unraisablehook are invoked (GH-17392) Also fixes some potential segfaults in unraisable hook handling. --- Doc/library/sys.rst | 23 +++++- Lib/test/audit-tests.py | 37 ++++++++++ Lib/test/test_audit.py | 56 +++++++++++---- .../2019-11-26-09-16-47.bpo-38920.Vx__sT.rst | 2 + Python/errors.c | 72 +++++++++++-------- Python/pythonrun.c | 8 +++ Python/sysmodule.c | 4 +- 7 files changed, 154 insertions(+), 48 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-11-26-09-16-47.bpo-38920.Vx__sT.rst diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 8d0de479e9cf5c..1cf19b8ad6c025 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -36,13 +36,18 @@ always available. .. audit-event:: sys.addaudithook "" sys.addaudithook Raise an auditing event ``sys.addaudithook`` with no arguments. If any - existing hooks raise an exception derived from :class:`Exception`, the + existing hooks raise an exception derived from :class:`RuntimeError`, the new hook will not be added and the exception suppressed. As a result, callers cannot assume that their hook has been added unless they control all existing hooks. .. versionadded:: 3.8 + .. versionchanged:: 3.8.1 + + Exceptions derived from :class:`Exception` but not :class:`RuntimeError` + are no longer suppressed. + .. impl-detail:: When tracing is enabled (see :func:`settrace`), Python hooks are only @@ -308,6 +313,15 @@ always available. before the program exits. The handling of such top-level exceptions can be customized by assigning another three-argument function to ``sys.excepthook``. + .. audit-event:: sys.excepthook hook,type,value,traceback sys.excepthook + + Raise an auditing event ``sys.excepthook`` with arguments ``hook``, + ``type``, ``value``, ``traceback`` when an uncaught exception occurs. + If no hook has been set, ``hook`` may be ``None``. If any hook raises + an exception derived from :class:`RuntimeError` the call to the hook will + be suppressed. Otherwise, the audit hook exception will be reported as + unraisable and ``sys.excepthook`` will be called. + .. seealso:: The :func:`sys.unraisablehook` function handles unraisable exceptions @@ -1563,6 +1577,13 @@ always available. See also :func:`excepthook` which handles uncaught exceptions. + .. audit-event:: sys.unraisablehook hook,unraisable sys.unraisablehook + + Raise an auditing event ``sys.unraisablehook`` with arguments + ``hook``, ``unraisable`` when an exception that cannot be handled occurs. + The ``unraisable`` object is the same as what will be passed to the hook. + If no hook has been set, ``hook`` may be ``None``. + .. versionadded:: 3.8 .. data:: version diff --git a/Lib/test/audit-tests.py b/Lib/test/audit-tests.py index ddeff22030a4b2..ed08612c0417fd 100644 --- a/Lib/test/audit-tests.py +++ b/Lib/test/audit-tests.py @@ -263,13 +263,50 @@ def trace(frame, event, *args): def test_mmap(): import mmap + with TestHook() as hook: mmap.mmap(-1, 8) assertEqual(hook.seen[0][1][:2], (-1, 8)) +def test_excepthook(): + def excepthook(exc_type, exc_value, exc_tb): + if exc_type is not RuntimeError: + sys.__excepthook__(exc_type, exc_value, exc_tb) + + def hook(event, args): + if event == "sys.excepthook": + if not isinstance(args[2], args[1]): + raise TypeError(f"Expected isinstance({args[2]!r}, " f"{args[1]!r})") + if args[0] != excepthook: + raise ValueError(f"Expected {args[0]} == {excepthook}") + print(event, repr(args[2])) + + sys.addaudithook(hook) + sys.excepthook = excepthook + raise RuntimeError("fatal-error") + + +def test_unraisablehook(): + from _testcapi import write_unraisable_exc + + def unraisablehook(hookargs): + pass + + def hook(event, args): + if event == "sys.unraisablehook": + if args[0] != unraisablehook: + raise ValueError(f"Expected {args[0]} == {unraisablehook}") + print(event, repr(args[1].exc_value), args[1].err_msg) + + sys.addaudithook(hook) + sys.unraisablehook = unraisablehook + write_unraisable_exc(RuntimeError("nonfatal-error"), "for audit hook test", None) + + if __name__ == "__main__": from test.libregrtest.setup import suppress_msvcrt_asserts + suppress_msvcrt_asserts(False) test = sys.argv[1] diff --git a/Lib/test/test_audit.py b/Lib/test/test_audit.py index 41f9fae1022318..31a08559273eec 100644 --- a/Lib/test/test_audit.py +++ b/Lib/test/test_audit.py @@ -24,7 +24,23 @@ def do_test(self, *args): sys.stdout.writelines(p.stdout) sys.stderr.writelines(p.stderr) if p.returncode: - self.fail(''.join(p.stderr)) + self.fail("".join(p.stderr)) + + def run_python(self, *args): + events = [] + with subprocess.Popen( + [sys.executable, "-X utf8", AUDIT_TESTS_PY, *args], + encoding="utf-8", + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) as p: + p.wait() + sys.stderr.writelines(p.stderr) + return ( + p.returncode, + [line.strip().partition(" ") for line in p.stdout], + "".join(p.stderr), + ) def test_basic(self): self.do_test("test_basic") @@ -36,19 +52,11 @@ def test_block_add_hook_baseexception(self): self.do_test("test_block_add_hook_baseexception") def test_finalize_hooks(self): - events = [] - with subprocess.Popen( - [sys.executable, "-X utf8", AUDIT_TESTS_PY, "test_finalize_hooks"], - encoding="utf-8", - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - ) as p: - p.wait() - for line in p.stdout: - events.append(line.strip().partition(" ")) - sys.stderr.writelines(p.stderr) - if p.returncode: - self.fail(''.join(p.stderr)) + returncode, events, stderr = self.run_python("test_finalize_hooks") + if stderr: + print(stderr, file=sys.stderr) + if returncode: + self.fail(stderr) firstId = events[0][2] self.assertSequenceEqual( @@ -76,6 +84,26 @@ def test_cantrace(self): def test_mmap(self): self.do_test("test_mmap") + def test_excepthook(self): + returncode, events, stderr = self.run_python("test_excepthook") + if not returncode: + self.fail(f"Expected fatal exception\n{stderr}") + + self.assertSequenceEqual( + [("sys.excepthook", " ", "RuntimeError('fatal-error')")], events + ) + + def test_unraisablehook(self): + returncode, events, stderr = self.run_python("test_unraisablehook") + if returncode: + self.fail(stderr) + + self.assertEqual(events[0][0], "sys.unraisablehook") + self.assertEqual( + events[0][2], + "RuntimeError('nonfatal-error') Exception ignored for audit hook test", + ) + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-11-26-09-16-47.bpo-38920.Vx__sT.rst b/Misc/NEWS.d/next/Core and Builtins/2019-11-26-09-16-47.bpo-38920.Vx__sT.rst new file mode 100644 index 00000000000000..2e9e443dd999bd --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2019-11-26-09-16-47.bpo-38920.Vx__sT.rst @@ -0,0 +1,2 @@ +Add audit hooks for when :func:`sys.excepthook` and +:func:`sys.unraisablehook` are invoked diff --git a/Python/errors.c b/Python/errors.c index 8a94afdd8c4101..197d9779b390c7 100644 --- a/Python/errors.c +++ b/Python/errors.c @@ -1367,44 +1367,54 @@ _PyErr_WriteUnraisableMsg(const char *err_msg_str, PyObject *obj) } } + PyObject *hook_args = make_unraisable_hook_args( + tstate, exc_type, exc_value, exc_tb, err_msg, obj); + if (hook_args == NULL) { + err_msg_str = ("Exception ignored on building " + "sys.unraisablehook arguments"); + goto error; + } + _Py_IDENTIFIER(unraisablehook); PyObject *hook = _PySys_GetObjectId(&PyId_unraisablehook); - if (hook != NULL && hook != Py_None) { - PyObject *hook_args; - - hook_args = make_unraisable_hook_args(tstate, exc_type, exc_value, - exc_tb, err_msg, obj); - if (hook_args != NULL) { - PyObject *args[1] = {hook_args}; - PyObject *res = _PyObject_FastCall(hook, args, 1); - Py_DECREF(hook_args); - if (res != NULL) { - Py_DECREF(res); - goto done; - } - - err_msg_str = "Exception ignored in sys.unraisablehook"; - } - else { - err_msg_str = ("Exception ignored on building " - "sys.unraisablehook arguments"); - } + if (hook == NULL) { + Py_DECREF(hook_args); + goto default_hook; + } - Py_XDECREF(err_msg); - err_msg = PyUnicode_FromString(err_msg_str); - if (err_msg == NULL) { - PyErr_Clear(); - } + if (PySys_Audit("sys.unraisablehook", "OO", hook, hook_args) < 0) { + Py_DECREF(hook_args); + err_msg_str = "Exception ignored in audit hook"; + obj = NULL; + goto error; + } - /* sys.unraisablehook failed: log its error using default hook */ - Py_XDECREF(exc_type); - Py_XDECREF(exc_value); - Py_XDECREF(exc_tb); - _PyErr_Fetch(tstate, &exc_type, &exc_value, &exc_tb); + if (hook == Py_None) { + Py_DECREF(hook_args); + goto default_hook; + } - obj = hook; + PyObject *args[1] = {hook_args}; + PyObject *res = _PyObject_FastCall(hook, args, 1); + Py_DECREF(hook_args); + if (res != NULL) { + Py_DECREF(res); + goto done; } + /* sys.unraisablehook failed: log its error using default hook */ + obj = hook; + err_msg_str = NULL; + +error: + /* err_msg_str and obj have been updated and we have a new exception */ + Py_XSETREF(err_msg, PyUnicode_FromString(err_msg_str ? + err_msg_str : "Exception ignored in sys.unraisablehook")); + Py_XDECREF(exc_type); + Py_XDECREF(exc_value); + Py_XDECREF(exc_tb); + _PyErr_Fetch(tstate, &exc_type, &exc_value, &exc_tb); + default_hook: /* Call the default unraisable hook (ignore failure) */ (void)write_unraisable_exc(tstate, exc_type, exc_value, exc_tb, diff --git a/Python/pythonrun.c b/Python/pythonrun.c index c9afa8f8baaf85..a7da143077a7a7 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -695,6 +695,14 @@ _PyErr_PrintEx(PyThreadState *tstate, int set_sys_last_vars) } } hook = _PySys_GetObjectId(&PyId_excepthook); + if (PySys_Audit("sys.excepthook", "OOOO", hook ? hook : Py_None, + exception, v, tb) < 0) { + if (PyErr_ExceptionMatches(PyExc_RuntimeError)) { + PyErr_Clear(); + goto done; + } + _PyErr_WriteUnraisableMsg("in audit hook", NULL); + } if (hook) { PyObject* stack[3]; PyObject *result; diff --git a/Python/sysmodule.c b/Python/sysmodule.c index 5b0fb813b4566b..1255665d024c3d 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -293,8 +293,8 @@ PySys_AddAuditHook(Py_AuditHookFunction hook, void *userData) /* Cannot invoke hooks until we are initialized */ if (Py_IsInitialized()) { if (PySys_Audit("sys.addaudithook", NULL) < 0) { - if (PyErr_ExceptionMatches(PyExc_Exception)) { - /* We do not report errors derived from Exception */ + if (PyErr_ExceptionMatches(PyExc_RuntimeError)) { + /* We do not report errors derived from RuntimeError */ PyErr_Clear(); return 0; } From 305189ecdc8322c22879a04564cad5989f937462 Mon Sep 17 00:00:00 2001 From: "Jules Lasne (jlasne)" Date: Fri, 29 Nov 2019 05:47:45 +0100 Subject: [PATCH 0969/2163] [3.8] Added missing coma after end of list in subprocess.rst (GH-17389) (cherry picked from commit f25875af425a3480e557aaedf49c3bb867bcbd5d) --- Doc/library/subprocess.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 954e0fec11828a..ea12cd133a6ffc 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -596,7 +596,7 @@ functions. Popen and the other functions in this module that use it raise an :ref:`auditing event ` ``subprocess.Popen`` with arguments - ``executable``, ``args``, ``cwd``, ``env``. The value for ``args`` + ``executable``, ``args``, ``cwd``, and ``env``. The value for ``args`` may be a single string or a list of strings, depending on platform. .. versionchanged:: 3.2 From 5f234538ab5625a81476981ab8772b46b67bf66e Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 30 Nov 2019 21:52:39 -0800 Subject: [PATCH 0970/2163] Fix typos (GH-17423) (cherry picked from commit 575d0b46d122292ca6e0576a91265d7abf7cbc3d) Co-authored-by: Ofek Lev --- Doc/library/datetime.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index b1e1b25691d8f1..b49eab44aef9e3 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -881,7 +881,7 @@ Other constructors, all class methods: Because naive ``datetime`` objects are treated by many ``datetime`` methods as local times, it is preferred to use aware datetimes to represent times in UTC. As such, the recommended way to create an object representing the - current time in UTC by calling ``datetime.now(timezone.utc)``. + current time in UTC is by calling ``datetime.now(timezone.utc)``. .. classmethod:: datetime.fromtimestamp(timestamp, tz=None) @@ -942,7 +942,7 @@ Other constructors, all class methods: Because naive ``datetime`` objects are treated by many ``datetime`` methods as local times, it is preferred to use aware datetimes to represent times in UTC. As such, the recommended way to create an object representing a - specific timestamp in UTC by calling + specific timestamp in UTC is by calling ``datetime.fromtimestamp(timestamp, tz=timezone.utc)``. .. versionchanged:: 3.3 From d300c0e845cdb898b0950642ed75dd0ae883ad12 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 1 Dec 2019 12:14:26 -0800 Subject: [PATCH 0971/2163] document threading.Lock.locked() (GH-17427) (cherry picked from commit fdafa1d0ed0a8930b52ee81e57c931cc4d5c2388) Co-authored-by: idomic --- Doc/library/threading.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index fd815ef01716d8..93ea4bda7cd7aa 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -488,6 +488,10 @@ All methods are executed atomically. There is no return value. + .. method:: locked() + Return true if the lock is acquired. + + .. _rlock-objects: From 4f1eaf028058cc357030dfaa5e611c90662539f0 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 1 Dec 2019 15:24:17 -0800 Subject: [PATCH 0972/2163] bpo-38449: Add URL delimiters test cases (GH-16729) * bpo-38449: Add tricky test cases * bpo-38449: Reflect codereview (cherry picked from commit 2fe4c48917c2d1b40cf063c6ed22ae2e71f4cb62) Co-authored-by: Dong-hee Na --- Lib/test/test_mimetypes.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/Lib/test/test_mimetypes.py b/Lib/test/test_mimetypes.py index bfd5eeedaa77b4..a5a06b189dec4f 100644 --- a/Lib/test/test_mimetypes.py +++ b/Lib/test/test_mimetypes.py @@ -51,6 +51,21 @@ def test_non_standard_types(self): eq(self.db.guess_type('foo.xul', strict=False), ('text/xul', None)) eq(self.db.guess_extension('image/jpg', strict=False), '.jpg') + def test_filename_with_url_delimiters(self): + # bpo-38449: URL delimiters cases should be handled also. + # They would have different mime types if interpreted as URL as + # compared to when interpreted as filename because of the semicolon. + eq = self.assertEqual + gzip_expected = ('application/x-tar', 'gzip') + eq(self.db.guess_type(";1.tar.gz"), gzip_expected) + eq(self.db.guess_type("?1.tar.gz"), gzip_expected) + eq(self.db.guess_type("#1.tar.gz"), gzip_expected) + eq(self.db.guess_type("#1#.tar.gz"), gzip_expected) + eq(self.db.guess_type(";1#.tar.gz"), gzip_expected) + eq(self.db.guess_type(";&1=123;?.tar.gz"), gzip_expected) + eq(self.db.guess_type("?k1=v1&k2=v2.tar.gz"), gzip_expected) + eq(self.db.guess_type(r" \"\`;b&b&c |.tar.gz"), gzip_expected) + def test_guess_all_types(self): eq = self.assertEqual unless = self.assertTrue From 9e728806d03fff8fa9e75159e567b2b4e040971b Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 2 Dec 2019 08:34:44 -0800 Subject: [PATCH 0973/2163] bpo-38815: Accept TLSv3 default in min max test (GH-NNNN) (GH-17437) Make ssl tests less strict and also accept TLSv3 as the default maximum version. This change unbreaks test_min_max_version on Fedora 32. https://bugs.python.org/issue38815 (cherry picked from commit 34864d1cffdbfc620f8517dab9a68ae9a37b8c53) Co-authored-by: torsava --- Lib/test/test_ssl.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py index 419506f4d3c072..539cb7751db89d 100644 --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1200,12 +1200,18 @@ def test_min_max_version(self): # RHEL 8 uses TLS 1.2 by default ssl.TLSVersion.TLSv1_2 } + maximum_range = { + # stock OpenSSL + ssl.TLSVersion.MAXIMUM_SUPPORTED, + # Fedora 32 uses TLS 1.3 by default + ssl.TLSVersion.TLSv1_3 + } self.assertIn( ctx.minimum_version, minimum_range ) - self.assertEqual( - ctx.maximum_version, ssl.TLSVersion.MAXIMUM_SUPPORTED + self.assertIn( + ctx.maximum_version, maximum_range ) ctx.minimum_version = ssl.TLSVersion.TLSv1_1 From 8859fc629474ab1ca7eb2e67aec538097c327e58 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 2 Dec 2019 14:44:44 -0800 Subject: [PATCH 0974/2163] bpo-38945: UU Encoding: Don't let newline in filename corrupt the output format (GH-17418) (cherry picked from commit a62ad4730c9b575f140f24074656c0257c86a09a) Co-authored-by: Matthew Rollings <1211162+stealthcopter@users.noreply.github.com> --- Lib/encodings/uu_codec.py | 4 ++++ Lib/test/test_uu.py | 9 +++++++++ Lib/uu.py | 7 +++++++ .../Security/2019-12-01-22-44-40.bpo-38945.ztmNXc.rst | 1 + 4 files changed, 21 insertions(+) create mode 100644 Misc/NEWS.d/next/Security/2019-12-01-22-44-40.bpo-38945.ztmNXc.rst diff --git a/Lib/encodings/uu_codec.py b/Lib/encodings/uu_codec.py index 2a5728fb5b74ad..4e58c62fe9ef0f 100644 --- a/Lib/encodings/uu_codec.py +++ b/Lib/encodings/uu_codec.py @@ -20,6 +20,10 @@ def uu_encode(input, errors='strict', filename='', mode=0o666): read = infile.read write = outfile.write + # Remove newline chars from filename + filename = filename.replace('\n','\\n') + filename = filename.replace('\r','\\r') + # Encode write(('begin %o %s\n' % (mode & 0o777, filename)).encode('ascii')) chunk = read(45) diff --git a/Lib/test/test_uu.py b/Lib/test/test_uu.py index c9f05e5b760d92..c8709f7a0d6660 100644 --- a/Lib/test/test_uu.py +++ b/Lib/test/test_uu.py @@ -136,6 +136,15 @@ def test_garbage_padding(self): decoded = codecs.decode(encodedtext, "uu_codec") self.assertEqual(decoded, plaintext) + def test_newlines_escaped(self): + # Test newlines are escaped with uu.encode + inp = io.BytesIO(plaintext) + out = io.BytesIO() + filename = "test.txt\n\roverflow.txt" + safefilename = b"test.txt\\n\\roverflow.txt" + uu.encode(inp, out, filename) + self.assertIn(safefilename, out.getvalue()) + class UUStdIOTest(unittest.TestCase): def setUp(self): diff --git a/Lib/uu.py b/Lib/uu.py index 9b1e5e607207f7..9f1f37f1a64101 100755 --- a/Lib/uu.py +++ b/Lib/uu.py @@ -73,6 +73,13 @@ def encode(in_file, out_file, name=None, mode=None, *, backtick=False): name = '-' if mode is None: mode = 0o666 + + # + # Remove newline chars from name + # + name = name.replace('\n','\\n') + name = name.replace('\r','\\r') + # # Write the data # diff --git a/Misc/NEWS.d/next/Security/2019-12-01-22-44-40.bpo-38945.ztmNXc.rst b/Misc/NEWS.d/next/Security/2019-12-01-22-44-40.bpo-38945.ztmNXc.rst new file mode 100644 index 00000000000000..1bf6ed567b2412 --- /dev/null +++ b/Misc/NEWS.d/next/Security/2019-12-01-22-44-40.bpo-38945.ztmNXc.rst @@ -0,0 +1 @@ +Newline characters have been escaped when performing uu encoding to prevent them from overflowing into to content section of the encoded file. This prevents malicious or accidental modification of data during the decoding process. \ No newline at end of file From baf07395eaa77e515ddfa1d3f42785d50b4d2889 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 3 Dec 2019 15:37:40 -0800 Subject: [PATCH 0975/2163] bpo-27873: Update docstring for multiprocessing.Pool.map (GH-17436) Update docstring for `multiprocessing.Pool.map` to mention `pool.starmap()`. Prev PR: https://github.com/python/cpython/pull/17367 @aeros https://bugs.python.org/issue27873 (cherry picked from commit eb48a451e3844185b9a8751c9badffbddc89689d) Co-authored-by: An Long --- Doc/library/multiprocessing.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index d8182feab963a1..3c7b5cc1262097 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -2169,7 +2169,8 @@ with the :class:`Pool` class. .. method:: map(func, iterable[, chunksize]) A parallel equivalent of the :func:`map` built-in function (it supports only - one *iterable* argument though). It blocks until the result is ready. + one *iterable* argument though, for multiple iterables see :meth:`starmap`). + It blocks until the result is ready. This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. The (approximate) size of these From a75cad440ab50d823af5f06e51dfed3a319f1e8c Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 4 Dec 2019 01:57:55 -0800 Subject: [PATCH 0976/2163] bpo-33684: json.tool: Use utf-8 for infile and outfile. (GH-17460) (cherry picked from commit 808769f3a4cbdc47cf1a5708dd61b1787bb192d4) Co-authored-by: Inada Naoki --- Lib/json/tool.py | 6 ++++-- Lib/test/test_json/test_tool.py | 21 ++++++++++++++++--- .../2019-12-04-15-28-40.bpo-33684.QeSmQP.rst | 2 ++ 3 files changed, 24 insertions(+), 5 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-04-15-28-40.bpo-33684.QeSmQP.rst diff --git a/Lib/json/tool.py b/Lib/json/tool.py index b3ef9923e31469..8db9ea40ad1b05 100644 --- a/Lib/json/tool.py +++ b/Lib/json/tool.py @@ -20,10 +20,12 @@ def main(): description = ('A simple command line interface for json module ' 'to validate and pretty-print JSON objects.') parser = argparse.ArgumentParser(prog=prog, description=description) - parser.add_argument('infile', nargs='?', type=argparse.FileType(), + parser.add_argument('infile', nargs='?', + type=argparse.FileType(encoding="utf-8"), help='a JSON file to be validated or pretty-printed', default=sys.stdin) - parser.add_argument('outfile', nargs='?', type=argparse.FileType('w'), + parser.add_argument('outfile', nargs='?', + type=argparse.FileType('w', encoding="utf-8"), help='write the output of infile to outfile', default=sys.stdout) parser.add_argument('--sort-keys', action='store_true', default=False, diff --git a/Lib/test/test_json/test_tool.py b/Lib/test/test_json/test_tool.py index 1e95bc79e5d189..f362f1b13a253e 100644 --- a/Lib/test/test_json/test_tool.py +++ b/Lib/test/test_json/test_tool.py @@ -89,11 +89,11 @@ def test_stdin_stdout(self): self.assertEqual(out.splitlines(), self.expect.encode().splitlines()) self.assertEqual(err, b'') - def _create_infile(self): + def _create_infile(self, data=None): infile = support.TESTFN - with open(infile, "w") as fp: + with open(infile, "w", encoding="utf-8") as fp: self.addCleanup(os.remove, infile) - fp.write(self.data) + fp.write(data or self.data) return infile def test_infile_stdout(self): @@ -103,6 +103,21 @@ def test_infile_stdout(self): self.assertEqual(out.splitlines(), self.expect.encode().splitlines()) self.assertEqual(err, b'') + def test_non_ascii_infile(self): + data = '{"msg": "\u3053\u3093\u306b\u3061\u306f"}' + expect = textwrap.dedent('''\ + { + "msg": "\\u3053\\u3093\\u306b\\u3061\\u306f" + } + ''').encode() + + infile = self._create_infile(data) + rc, out, err = assert_python_ok('-m', 'json.tool', infile) + + self.assertEqual(rc, 0) + self.assertEqual(out.splitlines(), expect.splitlines()) + self.assertEqual(err, b'') + def test_infile_outfile(self): infile = self._create_infile() outfile = support.TESTFN + '.out' diff --git a/Misc/NEWS.d/next/Library/2019-12-04-15-28-40.bpo-33684.QeSmQP.rst b/Misc/NEWS.d/next/Library/2019-12-04-15-28-40.bpo-33684.QeSmQP.rst new file mode 100644 index 00000000000000..107f9bb008330f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-04-15-28-40.bpo-33684.QeSmQP.rst @@ -0,0 +1,2 @@ +Fix ``json.tool`` failed to read a JSON file with non-ASCII characters when +locale encoding is not UTF-8. From 68669ef7883ea6338ca441e50f4f9d975f54d017 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 4 Dec 2019 08:21:16 -0800 Subject: [PATCH 0977/2163] bpo-38634: Allow non-apple build to cope with libedit (GH-16986) The readline module now detects if Python is linked to libedit at runtime on all platforms. Previously, the check was only done on macOS. If Python is used as a library by a binary linking to libedit, the linker resolves the rl_initialize symbol required by the readline module against libedit instead of libreadline, which leads to a segfault. Take advantage of the existing supporting code to have readline module being compatible with both situations. (cherry picked from commit 7105319ada2e663659020cbe9fdf7ff38f421ab2) Co-authored-by: serge-sans-paille --- .../2019-12-04-15-56-28.bpo-38634.pq0ZWa.rst | 2 ++ Modules/readline.c | 23 ++++--------------- 2 files changed, 6 insertions(+), 19 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-04-15-56-28.bpo-38634.pq0ZWa.rst diff --git a/Misc/NEWS.d/next/Library/2019-12-04-15-56-28.bpo-38634.pq0ZWa.rst b/Misc/NEWS.d/next/Library/2019-12-04-15-56-28.bpo-38634.pq0ZWa.rst new file mode 100644 index 00000000000000..d60c3172c2e510 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-04-15-56-28.bpo-38634.pq0ZWa.rst @@ -0,0 +1,2 @@ +The :mod:`readline` module now detects if Python is linked to libedit at runtime +on all platforms. Previously, the check was only done on macOS. diff --git a/Modules/readline.c b/Modules/readline.c index 57335fe911bff2..081657fb236991 100644 --- a/Modules/readline.c +++ b/Modules/readline.c @@ -45,14 +45,14 @@ extern char **completion_matches(char *, CPFunction *); #endif #endif -#ifdef __APPLE__ /* * It is possible to link the readline module to the readline * emulation library of editline/libedit. * - * On OSX this emulation library is not 100% API compatible - * with the "real" readline and cannot be detected at compile-time, - * hence we use a runtime check to detect if we're using libedit + * This emulation library is not 100% API compatible with the "real" readline + * and cannot be detected at compile-time, + * hence we use a runtime check to detect if the Python readlinke module is + * linked to libedit. * * Currently there is one known API incompatibility: * - 'get_history' has a 1-based index with GNU readline, and a 0-based @@ -64,7 +64,6 @@ static int using_libedit_emulation = 0; static const char libedit_version_tag[] = "EditLine wrapper"; static int libedit_history_start = 0; -#endif /* __APPLE__ */ #ifdef HAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK static void @@ -693,7 +692,6 @@ get_history_item(PyObject *self, PyObject *args) if (!PyArg_ParseTuple(args, "i:get_history_item", &idx)) return NULL; -#ifdef __APPLE__ if (using_libedit_emulation) { /* Older versions of libedit's readline emulation * use 0-based indexes, while readline and newer @@ -713,7 +711,6 @@ get_history_item(PyObject *self, PyObject *args) Py_RETURN_NONE; } } -#endif /* __APPLE__ */ if ((hist_ent = history_get(idx))) return decode(hist_ent->line); else { @@ -1080,7 +1077,6 @@ setup_readline(readlinestate *mod_state) /* The name must be defined before initialization */ rl_readline_name = "python"; -#ifdef __APPLE__ /* the libedit readline emulation resets key bindings etc * when calling rl_initialize. So call it upfront */ @@ -1097,7 +1093,6 @@ setup_readline(readlinestate *mod_state) libedit_history_start = 1; } clear_history(); -#endif /* __APPLE__ */ using_history(); @@ -1126,9 +1121,7 @@ setup_readline(readlinestate *mod_state) mod_state->begidx = PyLong_FromLong(0L); mod_state->endidx = PyLong_FromLong(0L); -#ifdef __APPLE__ if (!using_libedit_emulation) -#endif { if (!isatty(STDOUT_FILENO)) { /* Issue #19884: stdout is not a terminal. Disable meta modifier @@ -1148,11 +1141,9 @@ setup_readline(readlinestate *mod_state) * XXX: A bug in the readline-2.2 library causes a memory leak * inside this function. Nothing we can do about it. */ -#ifdef __APPLE__ if (using_libedit_emulation) rl_read_init_file(NULL); else -#endif /* __APPLE__ */ rl_initialize(); RESTORE_LOCALE(saved_locale) @@ -1281,12 +1272,10 @@ call_readline(FILE *sys_stdin, FILE *sys_stdout, const char *prompt) int length = _py_get_history_length(); if (length > 0) { HIST_ENTRY *hist_ent; -#ifdef __APPLE__ if (using_libedit_emulation) { /* handle older 0-based or newer 1-based indexing */ hist_ent = history_get(length + libedit_history_start - 1); } else -#endif /* __APPLE__ */ hist_ent = history_get(length); line = hist_ent ? hist_ent->line : ""; } else @@ -1314,10 +1303,8 @@ call_readline(FILE *sys_stdin, FILE *sys_stdout, const char *prompt) PyDoc_STRVAR(doc_module, "Importing this module enables command line editing using GNU readline."); -#ifdef __APPLE__ PyDoc_STRVAR(doc_module_le, "Importing this module enables command line editing using libedit readline."); -#endif /* __APPLE__ */ static struct PyModuleDef readlinemodule = { PyModuleDef_HEAD_INIT, @@ -1338,7 +1325,6 @@ PyInit_readline(void) PyObject *m; readlinestate *mod_state; -#ifdef __APPLE__ if (strncmp(rl_library_version, libedit_version_tag, strlen(libedit_version_tag)) == 0) { using_libedit_emulation = 1; } @@ -1346,7 +1332,6 @@ PyInit_readline(void) if (using_libedit_emulation) readlinemodule.m_doc = doc_module_le; -#endif /* __APPLE__ */ m = PyModule_Create(&readlinemodule); From f4a21d3b239bf4f4e4e2a8a5936b9b040645b246 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 4 Dec 2019 12:30:31 -0800 Subject: [PATCH 0978/2163] bpo-38965: Fix faulthandler._stack_overflow() on GCC 10 (GH-17467) Use the "volatile" keyword to prevent tail call optimization on any compiler, rather than relying on compiler specific pragma. (cherry picked from commit 8b787964e0a647caa0558b7c29ae501470d727d9) Co-authored-by: Victor Stinner --- .../2019-12-04-17-08-55.bpo-38965.yqax3m.rst | 3 +++ Modules/faulthandler.c | 16 ++++++---------- 2 files changed, 9 insertions(+), 10 deletions(-) create mode 100644 Misc/NEWS.d/next/Tests/2019-12-04-17-08-55.bpo-38965.yqax3m.rst diff --git a/Misc/NEWS.d/next/Tests/2019-12-04-17-08-55.bpo-38965.yqax3m.rst b/Misc/NEWS.d/next/Tests/2019-12-04-17-08-55.bpo-38965.yqax3m.rst new file mode 100644 index 00000000000000..517a1371eacd9a --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2019-12-04-17-08-55.bpo-38965.yqax3m.rst @@ -0,0 +1,3 @@ +Fix test_faulthandler on GCC 10. Use the "volatile" keyword in +``faulthandler._stack_overflow()`` to prevent tail call optimization on any +compiler, rather than relying on compiler specific pragma. diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c index 251d8e08a208e1..230cde4934fd43 100644 --- a/Modules/faulthandler.c +++ b/Modules/faulthandler.c @@ -1097,18 +1097,14 @@ faulthandler_fatal_error_py(PyObject *self, PyObject *args) #if defined(HAVE_SIGALTSTACK) && defined(HAVE_SIGACTION) #define FAULTHANDLER_STACK_OVERFLOW -#ifdef __INTEL_COMPILER - /* Issue #23654: Turn off ICC's tail call optimization for the - * stack_overflow generator. ICC turns the recursive tail call into - * a loop. */ -# pragma intel optimization_level 0 -#endif -static -uintptr_t +static uintptr_t stack_overflow(uintptr_t min_sp, uintptr_t max_sp, size_t *depth) { - /* allocate 4096 bytes on the stack at each call */ - unsigned char buffer[4096]; + /* Allocate (at least) 4096 bytes on the stack at each call. + + bpo-23654, bpo-38965: use volatile keyword to prevent tail call + optimization. */ + volatile unsigned char buffer[4096]; uintptr_t sp = (uintptr_t)&buffer; *depth += 1; if (sp < min_sp || max_sp < sp) From cfdaf92221da3c264d0da9c588994fefe4073196 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 5 Dec 2019 08:51:30 -0800 Subject: [PATCH 0979/2163] [3.8] bpo-38270: Fix indentation of test_hmac assertions (GH-17446) (GH-17450) Since https://github.com/python/cpython/commit/c64a1a61e6fc542cada40eb069a239317e1af36e two assertions were indented and thus ignored when running test_hmac. This PR fixes it. As the change is quite trivial I didn't add a NEWS entry. https://bugs.python.org/issue38270 (cherry picked from commit 894331838b256412c95d54051ec46a1cb96f52e7) Co-authored-by: stratakis https://bugs.python.org/issue38270 Automerge-Triggered-By: @tiran --- Lib/test/test_hmac.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_hmac.py b/Lib/test/test_hmac.py index ea00367c8027a6..23c108f6e3c27c 100644 --- a/Lib/test/test_hmac.py +++ b/Lib/test/test_hmac.py @@ -367,7 +367,7 @@ def test_with_bytearray(self): digestmod="sha256") except Exception: self.fail("Constructor call with bytearray arguments raised exception.") - self.assertEqual(h.hexdigest(), self.expected) + self.assertEqual(h.hexdigest(), self.expected) @requires_hashdigest('sha256') def test_with_memoryview_msg(self): @@ -375,7 +375,7 @@ def test_with_memoryview_msg(self): h = hmac.HMAC(b"key", memoryview(b"hash this!"), digestmod="sha256") except Exception: self.fail("Constructor call with memoryview msg raised exception.") - self.assertEqual(h.hexdigest(), self.expected) + self.assertEqual(h.hexdigest(), self.expected) @requires_hashdigest('sha256') def test_withmodule(self): From e21aa61e96f8343200e765d119ebe778873a6bf1 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 5 Dec 2019 09:42:01 -0800 Subject: [PATCH 0980/2163] bpo-38698: Prevent UnboundLocalError to pop up in parse_message_id (GH-17277) parse_message_id() was improperly using a token defined inside an exception handler, which was raising `UnboundLocalError` on parsing an invalid value. https://bugs.python.org/issue38698 (cherry picked from commit bb815499af855b1759c02535f8d7a9d0358e74e8) Co-authored-by: Claudiu Popa --- Lib/email/_header_value_parser.py | 3 ++- Lib/test/test_email/test__header_value_parser.py | 6 ++++++ .../next/Library/2019-12-02-10-35-19.bpo-38698.WZnAPQ.rst | 5 +++++ 3 files changed, 13 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-02-10-35-19.bpo-38698.WZnAPQ.rst diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py index 1668b4a14e9b91..abdef8189ca6fb 100644 --- a/Lib/email/_header_value_parser.py +++ b/Lib/email/_header_value_parser.py @@ -2113,7 +2113,8 @@ def parse_message_id(value): except errors.HeaderParseError: message_id.defects.append(errors.InvalidHeaderDefect( "Expected msg-id but found {!r}".format(value))) - message_id.append(token) + else: + message_id.append(token) return message_id # diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py index e442c44a2a74d0..2f63a3b3e05246 100644 --- a/Lib/test/test_email/test__header_value_parser.py +++ b/Lib/test/test_email/test__header_value_parser.py @@ -2638,6 +2638,12 @@ def test_get_msg_id_no_id_right_part(self): ) self.assertEqual(msg_id.token_type, 'msg-id') + def test_get_msg_id_invalid_expected_msg_id_not_found(self): + text = "Message-Id: 935-XPB-567:0:86089:180874:0:45327:9:90305:17843586-40@example.com" + msg_id = parser.parse_message_id(text) + self.assertDefectsEqual(msg_id.all_defects, + [errors.InvalidHeaderDefect]) + def test_get_msg_id_no_angle_start(self): with self.assertRaises(errors.HeaderParseError): parser.get_msg_id("msgwithnoankle") diff --git a/Misc/NEWS.d/next/Library/2019-12-02-10-35-19.bpo-38698.WZnAPQ.rst b/Misc/NEWS.d/next/Library/2019-12-02-10-35-19.bpo-38698.WZnAPQ.rst new file mode 100644 index 00000000000000..e606acb5dcf573 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-02-10-35-19.bpo-38698.WZnAPQ.rst @@ -0,0 +1,5 @@ +Prevent UnboundLocalError to pop up in parse_message_id + +parse_message_id() was improperly using a token defined inside an exception +handler, which was raising `UnboundLocalError` on parsing an invalid value. +Patch by Claudiu Popa. From 681285d052977e3a3a82ef665e788946fca1ac59 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 6 Dec 2019 06:59:49 -0800 Subject: [PATCH 0981/2163] bpo-36820: Break unnecessary cycle in socket.py, codeop.py and dyld.py (GH-13135) Break cycle generated when saving an exception in socket.py, codeop.py and dyld.py as they keep alive not only the exception but user objects through the ``__traceback__`` attribute. https://bugs.python.org/issue36820 Automerge-Triggered-By: @pablogsal (cherry picked from commit b64334cb93d0ddbb551c8cd712942bab2fc72772) Co-authored-by: Mario Corchero --- Lib/codeop.py | 11 +++++++---- Lib/ctypes/macholib/dyld.py | 2 ++ Lib/socket.py | 6 +++++- .../Library/2019-05-06-15-34-17.bpo-36820.Eh5mIB.rst | 3 +++ 4 files changed, 17 insertions(+), 5 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-05-06-15-34-17.bpo-36820.Eh5mIB.rst diff --git a/Lib/codeop.py b/Lib/codeop.py index fb759da42ad1a2..0fa677f609b74b 100644 --- a/Lib/codeop.py +++ b/Lib/codeop.py @@ -93,10 +93,13 @@ def _maybe_compile(compiler, source, filename, symbol): except SyntaxError as e: err2 = e - if code: - return code - if not code1 and repr(err1) == repr(err2): - raise err1 + try: + if code: + return code + if not code1 and repr(err1) == repr(err2): + raise err1 + finally: + err1 = err2 = None def _compile(source, filename, symbol): return compile(source, filename, symbol, PyCF_DONT_IMPLY_DEDENT) diff --git a/Lib/ctypes/macholib/dyld.py b/Lib/ctypes/macholib/dyld.py index c158e672f0511a..9d86b058765a3e 100644 --- a/Lib/ctypes/macholib/dyld.py +++ b/Lib/ctypes/macholib/dyld.py @@ -149,6 +149,8 @@ def framework_find(fn, executable_path=None, env=None): return dyld_find(fn, executable_path=executable_path, env=env) except ValueError: raise error + finally: + error = None def test_dyld_find(): env = {} diff --git a/Lib/socket.py b/Lib/socket.py index 813f4ef5c3e1f5..5b17906ef479a1 100644 --- a/Lib/socket.py +++ b/Lib/socket.py @@ -804,7 +804,11 @@ def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT, sock.close() if err is not None: - raise err + try: + raise err + finally: + # Break explicitly a reference cycle + err = None else: raise error("getaddrinfo returns an empty list") diff --git a/Misc/NEWS.d/next/Library/2019-05-06-15-34-17.bpo-36820.Eh5mIB.rst b/Misc/NEWS.d/next/Library/2019-05-06-15-34-17.bpo-36820.Eh5mIB.rst new file mode 100644 index 00000000000000..82f6635c81582d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-05-06-15-34-17.bpo-36820.Eh5mIB.rst @@ -0,0 +1,3 @@ +Break cycle generated when saving an exception in socket.py, codeop.py and +dyld.py as they keep alive not only the exception but user objects through +the ``__traceback__`` attribute. Patch by Mario Corchero. From c9f480d2ccda9de46584cabe086f0acfa45f2faf Mon Sep 17 00:00:00 2001 From: Steve Dower Date: Fri, 6 Dec 2019 09:40:39 -0800 Subject: [PATCH 0982/2163] bpo-33125: Add support for building and releasing Windows ARM64 packages (GH-17480) Note that the support is not actually enabled yet, and so we won't be publishing these packages. However, for those who want to build it themselves (even by reusing the Azure Pipelines definition), it's now relatively easy to enable. --- .azure-pipelines/ci.yml | 2 +- .azure-pipelines/pr.yml | 5 ++- .../windows-release/build-steps.yml | 3 +- .../windows-release/layout-command.yml | 10 ++++- .../windows-release/msi-steps.yml | 1 + .../windows-release/stage-build.yml | 18 ++++++-- .../windows-release/stage-layout-embed.yml | 5 +++ .../windows-release/stage-layout-full.yml | 17 ++++++-- .../windows-release/stage-layout-msix.yml | 19 +++++--- .../windows-release/stage-layout-nuget.yml | 12 ++++-- .../windows-release/stage-pack-msix.yml | 10 +++++ .../windows-release/stage-pack-nuget.yml | 2 + .../stage-publish-nugetorg.yml | 5 +++ .../stage-publish-pythonorg.yml | 4 ++ .../windows-release/stage-sign.yml | 2 + .azure-pipelines/windows-steps.yml | 4 +- .../2019-11-14-08-57-50.bpo-33125.EN5MWS.rst | 1 + PC/layout/main.py | 24 +++++++++-- PC/layout/support/appxmanifest.py | 43 +++++++++++++++---- PC/layout/support/constants.py | 32 ++++++++++---- PC/layout/support/nuspec.py | 40 +++++++++-------- PC/layout/support/props.py | 22 +++++----- PCbuild/build.bat | 2 +- PCbuild/find_msbuild.bat | 2 +- PCbuild/pcbuild.proj | 2 +- PCbuild/prepare_libffi.bat | 7 ++- PCbuild/python_uwp.vcxproj | 32 ++++++++++++++ PCbuild/pythonw_uwp.vcxproj | 32 ++++++++++++++ 28 files changed, 285 insertions(+), 73 deletions(-) create mode 100644 Misc/NEWS.d/next/Windows/2019-11-14-08-57-50.bpo-33125.EN5MWS.rst diff --git a/.azure-pipelines/ci.yml b/.azure-pipelines/ci.yml index fe10e45b679226..4c2f115cd9b600 100644 --- a/.azure-pipelines/ci.yml +++ b/.azure-pipelines/ci.yml @@ -145,7 +145,7 @@ jobs: buildOpt: '-p x64' testRunTitle: '$(Build.SourceBranchName)-win64' testRunPlatform: win64 - maxParallel: 2 + maxParallel: 4 steps: - template: ./windows-steps.yml diff --git a/.azure-pipelines/pr.yml b/.azure-pipelines/pr.yml index 09209fc0c954d0..73d4f55b864500 100644 --- a/.azure-pipelines/pr.yml +++ b/.azure-pipelines/pr.yml @@ -145,7 +145,10 @@ jobs: buildOpt: '-p x64' testRunTitle: '$(System.PullRequest.TargetBranch)-win64' testRunPlatform: win64 - maxParallel: 2 + winarm64: + arch: arm64 + buildOpt: '-p arm64' + maxParallel: 4 steps: - template: ./windows-steps.yml diff --git a/.azure-pipelines/windows-release/build-steps.yml b/.azure-pipelines/windows-release/build-steps.yml index d4563cd0d722c1..e2b6683f46e3c3 100644 --- a/.azure-pipelines/windows-release/build-steps.yml +++ b/.azure-pipelines/windows-release/build-steps.yml @@ -43,7 +43,7 @@ steps: - powershell: | $env:SigningCertificate = $null - .\python.bat PC\layout -vv -t "$(Build.BinariesDirectory)\catalog" --catalog "${env:CAT}.cdf" --preset-default + python PC\layout -vv -b "$(Build.BinariesDirectory)\bin" -t "$(Build.BinariesDirectory)\catalog" --catalog "${env:CAT}.cdf" --preset-default --arch $(Arch) makecat "${env:CAT}.cdf" del "${env:CAT}.cdf" if (-not (Test-Path "${env:CAT}.cat")) { @@ -52,6 +52,7 @@ steps: displayName: 'Generate catalog' env: CAT: $(Build.BinariesDirectory)\bin\$(Arch)\python + PYTHON_HEXVERSION: $(VersionHex) - task: PublishPipelineArtifact@0 displayName: 'Publish binaries' diff --git a/.azure-pipelines/windows-release/layout-command.yml b/.azure-pipelines/windows-release/layout-command.yml index 2dcd6ed26ca3a8..406ccd859faa6a 100644 --- a/.azure-pipelines/windows-release/layout-command.yml +++ b/.azure-pipelines/windows-release/layout-command.yml @@ -1,12 +1,20 @@ steps: +- task: DownloadPipelineArtifact@1 + displayName: 'Download artifact: bin_$(HostArch)' + condition: and(succeeded(), variables['HostArch']) + inputs: + artifactName: bin_$(HostArch) + targetPath: $(Build.BinariesDirectory)\bin_$(HostArch) + - powershell: > Write-Host ( '##vso[task.setvariable variable=LayoutCmd]& - "{0}\bin\python.exe" + "$(Python)" "{1}\PC\layout" -vv --source "{1}" --build "{0}\bin" + --arch "$(Name)" --temp "{0}\layout-temp" --include-cat "{0}\bin\python.cat" --doc-build "{0}\doc"' diff --git a/.azure-pipelines/windows-release/msi-steps.yml b/.azure-pipelines/windows-release/msi-steps.yml index f7bff162f8e027..a460eb1bac8fe5 100644 --- a/.azure-pipelines/windows-release/msi-steps.yml +++ b/.azure-pipelines/windows-release/msi-steps.yml @@ -54,6 +54,7 @@ steps: - powershell: | copy $(Build.BinariesDirectory)\amd64\Activate.ps1 Lib\venv\scripts\common\Activate.ps1 -Force displayName: 'Copy signed files into sources' + condition: and(succeeded(), variables['SigningCertificate']) - script: | call Tools\msi\get_externals.bat diff --git a/.azure-pipelines/windows-release/stage-build.yml b/.azure-pipelines/windows-release/stage-build.yml index c98576ef9705c0..60d72b282d1e51 100644 --- a/.azure-pipelines/windows-release/stage-build.yml +++ b/.azure-pipelines/windows-release/stage-build.yml @@ -16,14 +16,16 @@ jobs: env: BUILDDIR: $(Build.BinariesDirectory)\Doc - #- powershell: iwr "https://www.python.org/ftp/python/3.7.3/python373.chm" -OutFile "$(Build.BinariesDirectory)\python390a0.chm" - # displayName: 'Cheat at building CHM docs' - - script: Doc\make.bat htmlhelp displayName: 'Build CHM docs' env: BUILDDIR: $(Build.BinariesDirectory)\Doc + #- powershell: | + # mkdir -Force "$(Build.BinariesDirectory)\Doc\htmlhelp" + # iwr "https://www.python.org/ftp/python/3.8.0/python380.chm" -OutFile "$(Build.BinariesDirectory)\Doc\htmlhelp\python390a0.chm" + # displayName: 'Cheat at building CHM docs' + - task: CopyFiles@2 displayName: 'Assemble artifact: Doc' inputs: @@ -65,6 +67,16 @@ jobs: Arch: amd64 Platform: x64 Configuration: Debug + arm64: + Name: arm64 + Arch: arm64 + Platform: ARM64 + Configuration: Release + arm64_d: + Name: arm64_d + Arch: arm64 + Platform: ARM64 + Configuration: Debug steps: - template: ./build-steps.yml diff --git a/.azure-pipelines/windows-release/stage-layout-embed.yml b/.azure-pipelines/windows-release/stage-layout-embed.yml index 09857ff676b355..3306e1cbc49d98 100644 --- a/.azure-pipelines/windows-release/stage-layout-embed.yml +++ b/.azure-pipelines/windows-release/stage-layout-embed.yml @@ -19,6 +19,11 @@ jobs: Name: amd64 Python: $(Build.BinariesDirectory)\bin\python.exe PYTHONHOME: $(Build.SourcesDirectory) + arm64: + Name: arm64 + HostArch: amd64 + Python: $(Build.BinariesDirectory)\bin_amd64\python.exe + PYTHONHOME: $(Build.SourcesDirectory) steps: - template: ./checkout.yml diff --git a/.azure-pipelines/windows-release/stage-layout-full.yml b/.azure-pipelines/windows-release/stage-layout-full.yml index 12c347239013c1..78bc1b3975e93d 100644 --- a/.azure-pipelines/windows-release/stage-layout-full.yml +++ b/.azure-pipelines/windows-release/stage-layout-full.yml @@ -13,11 +13,18 @@ jobs: matrix: win32: Name: win32 - Python: $(Build.BinariesDirectory)\bin_$(Name)\python.exe + Python: $(Build.BinariesDirectory)\bin\python.exe PYTHONHOME: $(Build.SourcesDirectory) + TclLibrary: $(Build.BinariesDirectory)\tcltk_lib\tcl8 amd64: Name: amd64 - Python: $(Build.BinariesDirectory)\bin_$(Name)\python.exe + Python: $(Build.BinariesDirectory)\bin\python.exe + PYTHONHOME: $(Build.SourcesDirectory) + TclLibrary: $(Build.BinariesDirectory)\tcltk_lib\tcl8 + arm64: + Name: arm64 + HostArch: amd64 + Python: $(Build.BinariesDirectory)\bin_amd64\python.exe PYTHONHOME: $(Build.SourcesDirectory) steps: @@ -43,13 +50,15 @@ jobs: - task: DownloadPipelineArtifact@1 displayName: 'Download artifact: tcltk_lib_$(Name)' + condition: and(succeeded(), variables['TclLibrary']) inputs: artifactName: tcltk_lib_$(Name) targetPath: $(Build.BinariesDirectory)\tcltk_lib - powershell: | - copy $(Build.BinariesDirectory)\bin\Activate.ps1 Lib\venv\scripts\common\Activate.ps1 -Force + copy "$(Build.BinariesDirectory)\bin\Activate.ps1" Lib\venv\scripts\common\Activate.ps1 -Force displayName: 'Copy signed files into sources' + condition: and(succeeded(), variables['SigningCertificate']) - template: ./layout-command.yml @@ -57,7 +66,7 @@ jobs: $(LayoutCmd) --copy "$(Build.ArtifactStagingDirectory)\layout" --preset-default displayName: 'Generate full layout' env: - TCL_LIBRARY: $(Build.BinariesDirectory)\tcltk_lib\tcl8 + TCL_LIBRARY: $(TclLibrary) - task: PublishPipelineArtifact@0 displayName: 'Publish Artifact: layout_full_$(Name)' diff --git a/.azure-pipelines/windows-release/stage-layout-msix.yml b/.azure-pipelines/windows-release/stage-layout-msix.yml index ba86392f3ec699..60a5c9ea5435c8 100644 --- a/.azure-pipelines/windows-release/stage-layout-msix.yml +++ b/.azure-pipelines/windows-release/stage-layout-msix.yml @@ -12,11 +12,18 @@ jobs: matrix: #win32: # Name: win32 - # Python: $(Build.BinariesDirectory)\bin_$(Name)\python.exe + # Python: $(Build.BinariesDirectory)\bin\python.exe # PYTHONHOME: $(Build.SourcesDirectory) + # TclLibrary: $(Build.BinariesDirectory)\tcltk_lib\tcl8 amd64: Name: amd64 - Python: $(Build.BinariesDirectory)\bin_$(Name)\python.exe + Python: $(Build.BinariesDirectory)\bin\python.exe + PYTHONHOME: $(Build.SourcesDirectory) + TclLibrary: $(Build.BinariesDirectory)\tcltk_lib\tcl8 + arm64: + Name: arm64 + HostArch: amd64 + Python: $(Build.BinariesDirectory)\bin_amd64\python.exe PYTHONHOME: $(Build.SourcesDirectory) steps: @@ -36,13 +43,15 @@ jobs: - task: DownloadPipelineArtifact@1 displayName: 'Download artifact: tcltk_lib_$(Name)' + condition: and(succeeded(), variables['TclLibrary']) inputs: artifactName: tcltk_lib_$(Name) targetPath: $(Build.BinariesDirectory)\tcltk_lib - powershell: | - copy $(Build.BinariesDirectory)\bin\Activate.ps1 Lib\venv\scripts\common\Activate.ps1 -Force + copy "$(Build.BinariesDirectory)\bin\Activate.ps1" Lib\venv\scripts\common\Activate.ps1 -Force displayName: 'Copy signed files into sources' + condition: and(succeeded(), variables['SigningCertificate']) - template: ./layout-command.yml @@ -51,7 +60,7 @@ jobs: $(LayoutCmd) --copy "$(Build.ArtifactStagingDirectory)\appx-store" --preset-appx --precompile displayName: 'Generate store APPX layout' env: - TCL_LIBRARY: $(Build.BinariesDirectory)\tcltk_lib\tcl8 + TCL_LIBRARY: $(TclLibrary) - task: PublishPipelineArtifact@0 displayName: 'Publish Artifact: layout_appxstore_$(Name)' @@ -79,7 +88,7 @@ jobs: $(LayoutCmd) --copy "$(Build.ArtifactStagingDirectory)\appx" --preset-appx --precompile --include-symbols --include-tests displayName: 'Generate sideloading APPX layout' env: - TCL_LIBRARY: $(Build.BinariesDirectory)\tcltk_lib\tcl8 + TCL_LIBRARY: $(TclLibrary) - task: PublishPipelineArtifact@0 displayName: 'Publish Artifact: layout_appx_$(Name)' diff --git a/.azure-pipelines/windows-release/stage-layout-nuget.yml b/.azure-pipelines/windows-release/stage-layout-nuget.yml index 7954c4547f50ab..7e20f89530349c 100644 --- a/.azure-pipelines/windows-release/stage-layout-nuget.yml +++ b/.azure-pipelines/windows-release/stage-layout-nuget.yml @@ -13,11 +13,16 @@ jobs: matrix: win32: Name: win32 - Python: $(Build.BinariesDirectory)\bin_$(Name)\python.exe + Python: $(Build.BinariesDirectory)\bin\python.exe PYTHONHOME: $(Build.SourcesDirectory) amd64: Name: amd64 - Python: $(Build.BinariesDirectory)\bin_$(Name)\python.exe + Python: $(Build.BinariesDirectory)\bin\python.exe + PYTHONHOME: $(Build.SourcesDirectory) + arm64: + Name: arm64 + HostArch: amd64 + Python: $(Build.BinariesDirectory)\bin_amd64\python.exe PYTHONHOME: $(Build.SourcesDirectory) steps: @@ -32,14 +37,13 @@ jobs: - powershell: | copy $(Build.BinariesDirectory)\bin\Activate.ps1 Lib\venv\scripts\common\Activate.ps1 -Force displayName: 'Copy signed files into sources' + condition: and(succeeded(), variables['SigningCertificate']) - template: ./layout-command.yml - powershell: | $(LayoutCmd) --copy "$(Build.ArtifactStagingDirectory)\nuget" --preset-nuget displayName: 'Generate nuget layout' - env: - TCL_LIBRARY: $(Build.BinariesDirectory)\bin_$(Name)\tcl\tcl8 - task: PublishPipelineArtifact@0 displayName: 'Publish Artifact: layout_nuget_$(Name)' diff --git a/.azure-pipelines/windows-release/stage-pack-msix.yml b/.azure-pipelines/windows-release/stage-pack-msix.yml index eebc63fb8809bc..f17ba9628e21b0 100644 --- a/.azure-pipelines/windows-release/stage-pack-msix.yml +++ b/.azure-pipelines/windows-release/stage-pack-msix.yml @@ -20,6 +20,16 @@ jobs: Artifact: appxstore Suffix: -store Upload: true + arm64: + Name: arm64 + Artifact: appx + Suffix: + ShouldSign: true + arm64_store: + Name: arm64 + Artifact: appxstore + Suffix: -store + Upload: true steps: - template: ./checkout.yml diff --git a/.azure-pipelines/windows-release/stage-pack-nuget.yml b/.azure-pipelines/windows-release/stage-pack-nuget.yml index f59bbe9b39a8d7..34619fc5fdc318 100644 --- a/.azure-pipelines/windows-release/stage-pack-nuget.yml +++ b/.azure-pipelines/windows-release/stage-pack-nuget.yml @@ -15,6 +15,8 @@ jobs: Name: amd64 win32: Name: win32 + arm64: + Name: arm64 steps: - checkout: none diff --git a/.azure-pipelines/windows-release/stage-publish-nugetorg.yml b/.azure-pipelines/windows-release/stage-publish-nugetorg.yml index 570cdb3ec57f0d..b78bd493a0fd1b 100644 --- a/.azure-pipelines/windows-release/stage-publish-nugetorg.yml +++ b/.azure-pipelines/windows-release/stage-publish-nugetorg.yml @@ -31,6 +31,11 @@ jobs: buildVersionToDownload: specific buildId: $(BuildToPublish) + - powershell: 'gci pythonarm*.nupkg | %{ Write-Host "Not publishing: $($_.Name)"; gi $_ } | del' + displayName: 'Prevent publishing ARM/ARM64 packages' + workingDirectory: '$(Build.BinariesDirectory)\nuget' + condition: and(succeeded(), not(variables['PublishArmPackages'])) + - task: NuGetCommand@2 displayName: Push packages condition: and(succeeded(), eq(variables['SigningCertificate'], variables['__RealSigningCertificate'])) diff --git a/.azure-pipelines/windows-release/stage-publish-pythonorg.yml b/.azure-pipelines/windows-release/stage-publish-pythonorg.yml index 2dd354a8c276f6..8c95f1b950cd75 100644 --- a/.azure-pipelines/windows-release/stage-publish-pythonorg.yml +++ b/.azure-pipelines/windows-release/stage-publish-pythonorg.yml @@ -39,6 +39,10 @@ jobs: artifactName: embed downloadPath: $(Build.BinariesDirectory) + - powershell: 'gci *embed-arm*.zip | %{ Write-Host "Not publishing: $($_.Name)"; gi $_ } | del' + displayName: 'Prevent publishing ARM/ARM64 packages' + workingDirectory: '$(Build.BinariesDirectory)\embed' + condition: and(succeeded(), not(variables['PublishArmPackages'])) - task: DownloadPipelineArtifact@1 displayName: 'Download artifact from $(BuildToPublish): Doc' diff --git a/.azure-pipelines/windows-release/stage-sign.yml b/.azure-pipelines/windows-release/stage-sign.yml index 2307c6c9c8f97d..a0adc0581229d9 100644 --- a/.azure-pipelines/windows-release/stage-sign.yml +++ b/.azure-pipelines/windows-release/stage-sign.yml @@ -19,6 +19,8 @@ jobs: Name: win32 amd64: Name: amd64 + arm64: + Name: arm64 steps: - template: ./checkout.yml diff --git a/.azure-pipelines/windows-steps.yml b/.azure-pipelines/windows-steps.yml index 794a23a5d77e86..f502c40637c310 100644 --- a/.azure-pipelines/windows-steps.yml +++ b/.azure-pipelines/windows-steps.yml @@ -19,9 +19,11 @@ steps: - script: python.bat -m test.pythoninfo displayName: 'Display build info' + condition: and(succeeded(), variables['testRunPlatform']) - script: PCbuild\rt.bat -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0 --junit-xml="$(Build.BinariesDirectory)\test-results.xml" --tempdir="$(Build.BinariesDirectory)\test" displayName: 'Tests' + condition: and(succeeded(), variables['testRunPlatform']) env: PREFIX: $(Py_OutDir)\$(arch) @@ -32,4 +34,4 @@ steps: mergeTestResults: true testRunTitle: $(testRunTitle) platform: $(testRunPlatform) - condition: succeededOrFailed() + condition: and(succeededOrFailed(), variables['testRunPlatform']) diff --git a/Misc/NEWS.d/next/Windows/2019-11-14-08-57-50.bpo-33125.EN5MWS.rst b/Misc/NEWS.d/next/Windows/2019-11-14-08-57-50.bpo-33125.EN5MWS.rst new file mode 100644 index 00000000000000..0bc98c1a2fae77 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2019-11-14-08-57-50.bpo-33125.EN5MWS.rst @@ -0,0 +1 @@ +Add support for building and releasing Windows ARM64 packages. diff --git a/PC/layout/main.py b/PC/layout/main.py index 3ca49d08e8f981..305cb517d0f71d 100644 --- a/PC/layout/main.py +++ b/PC/layout/main.py @@ -285,14 +285,13 @@ def _compile_one_py(src, dest, name, optimize, checked=True): log_warning("Failed to compile {}", src) return None + # name argument added to address bpo-37641 def _py_temp_compile(src, name, ns, dest_dir=None, checked=True): if not ns.precompile or src not in PY_FILES or src.parent in DATA_DIRS: return None dest = (dest_dir or ns.temp) / (src.stem + ".pyc") - return _compile_one_py( - src, dest, name, optimize=2, checked=checked - ) + return _compile_one_py(src, dest, name, optimize=2, checked=checked) def _write_to_zip(zf, dest, src, ns, checked=True): @@ -496,6 +495,13 @@ def main(): parser.add_argument( "-b", "--build", metavar="dir", help="Specify the build directory", type=Path ) + parser.add_argument( + "--arch", + metavar="architecture", + help="Specify the target architecture", + type=str, + default=None, + ) parser.add_argument( "--doc-build", metavar="dir", @@ -587,6 +593,8 @@ def main(): ns.doc_build = (Path.cwd() / ns.doc_build).resolve() if ns.include_cat and not ns.include_cat.is_absolute(): ns.include_cat = (Path.cwd() / ns.include_cat).resolve() + if not ns.arch: + ns.arch = "amd64" if sys.maxsize > 2 ** 32 else "win32" if ns.copy and not ns.copy.is_absolute(): ns.copy = (Path.cwd() / ns.copy).resolve() @@ -602,6 +610,7 @@ def main(): Source: {ns.source} Build: {ns.build} Temp: {ns.temp} +Arch: {ns.arch} Copy to: {ns.copy} Zip to: {ns.zip} @@ -609,6 +618,15 @@ def main(): ns=ns, ) + if ns.arch not in ("win32", "amd64", "arm32", "arm64"): + log_error("--arch is not a valid value (win32, amd64, arm32, arm64)") + return 4 + if ns.arch in ("arm32", "arm64"): + for n in ("include_idle", "include_tcltk"): + if getattr(ns, n): + log_warning(f"Disabling --{n.replace('_', '-')} on unsupported platform") + setattr(ns, n, False) + if ns.include_idle and not ns.include_tcltk: log_warning("Assuming --include-tcltk to support --include-idle") ns.include_tcltk = True diff --git a/PC/layout/support/appxmanifest.py b/PC/layout/support/appxmanifest.py index de5813a2536aee..9e008f793cf550 100644 --- a/PC/layout/support/appxmanifest.py +++ b/PC/layout/support/appxmanifest.py @@ -28,7 +28,14 @@ ), DisplayName="Python {}".format(VER_DOT), Description="The Python {} runtime and console.".format(VER_DOT), - ProcessorArchitecture="x64" if IS_X64 else "x86", +) + +APPX_PLATFORM_DATA = dict( + _keys=("ProcessorArchitecture",), + win32=("x86",), + amd64=("x64",), + arm32=("arm",), + arm64=("arm64",), ) PYTHON_VE_DATA = dict( @@ -65,7 +72,7 @@ BackgroundColor="transparent", ) -PY_PNG = '_resources/py.png' +PY_PNG = "_resources/py.png" APPXMANIFEST_NS = { "": "http://schemas.microsoft.com/appx/manifest/foundation/windows10", @@ -147,18 +154,22 @@ SCCD_FILENAME = "PC/classicAppCompat.sccd" +SPECIAL_LOOKUP = object() + REGISTRY = { "HKCU\\Software\\Python\\PythonCore": { VER_DOT: { "DisplayName": APPX_DATA["DisplayName"], "SupportUrl": "https://www.python.org/", - "SysArchitecture": "64bit" if IS_X64 else "32bit", + "SysArchitecture": SPECIAL_LOOKUP, "SysVersion": VER_DOT, "Version": "{}.{}.{}".format(VER_MAJOR, VER_MINOR, VER_MICRO), "InstallPath": { "": "[{AppVPackageRoot}]", "ExecutablePath": "[{{AppVPackageRoot}}]\\python{}.exe".format(VER_DOT), - "WindowedExecutablePath": "[{{AppVPackageRoot}}]\\pythonw{}.exe".format(VER_DOT), + "WindowedExecutablePath": "[{{AppVPackageRoot}}]\\pythonw{}.exe".format( + VER_DOT + ), }, "Help": { "Main Python Documentation": { @@ -338,6 +349,16 @@ def _get_registry_entries(ns, root="", d=None): if len(fullkey.parts) > 1: yield str(fullkey), None, None yield from _get_registry_entries(ns, fullkey, value) + elif value is SPECIAL_LOOKUP: + if key == "SysArchitecture": + return { + "win32": "32bit", + "amd64": "64bit", + "arm32": "32bit", + "arm64": "64bit", + }[ns.arch] + else: + raise ValueError(f"Key '{key}' unhandled for special lookup") elif len(r.parts) > 1: yield str(r), key, value @@ -376,14 +397,18 @@ def get_appxmanifest(ns): NS = APPXMANIFEST_NS QN = ET.QName + data = dict(APPX_DATA) + for k, v in zip(APPX_PLATFORM_DATA["_keys"], APPX_PLATFORM_DATA[ns.arch]): + data[k] = v + node = xml.find("m:Identity", NS) for k in node.keys(): - value = APPX_DATA.get(k) + value = data.get(k) if value: node.set(k, value) for node in xml.find("m:Properties", NS): - value = APPX_DATA.get(node.tag.rpartition("}")[2]) + value = data.get(node.tag.rpartition("}")[2]) if value: node.text = value @@ -405,7 +430,7 @@ def get_appxmanifest(ns): ["python", "python{}".format(VER_MAJOR), "python{}".format(VER_DOT)], PYTHON_VE_DATA, "console", - ("python.file", [".py"], '"%1"', 'Python File', PY_PNG), + ("python.file", [".py"], '"%1"', "Python File", PY_PNG), ) add_application( @@ -416,7 +441,7 @@ def get_appxmanifest(ns): ["pythonw", "pythonw{}".format(VER_MAJOR), "pythonw{}".format(VER_DOT)], PYTHONW_VE_DATA, "windows", - ("python.windowedfile", [".pyw"], '"%1"', 'Python File (no console)', PY_PNG), + ("python.windowedfile", [".pyw"], '"%1"', "Python File (no console)", PY_PNG), ) if ns.include_pip and ns.include_launchers: @@ -428,7 +453,7 @@ def get_appxmanifest(ns): ["pip", "pip{}".format(VER_MAJOR), "pip{}".format(VER_DOT)], PIP_VE_DATA, "console", - ("python.wheel", [".whl"], 'install "%1"', 'Python Wheel'), + ("python.wheel", [".whl"], 'install "%1"', "Python Wheel"), ) if ns.include_idle and ns.include_launchers: diff --git a/PC/layout/support/constants.py b/PC/layout/support/constants.py index d76fa3bbf3b476..a8647631e9b4cd 100644 --- a/PC/layout/support/constants.py +++ b/PC/layout/support/constants.py @@ -5,15 +5,31 @@ __author__ = "Steve Dower " __version__ = "3.8" +import os +import re import struct import sys -VER_MAJOR, VER_MINOR, VER_MICRO, VER_FIELD4 = struct.pack(">i", sys.hexversion) + +def _unpack_hexversion(): + try: + hexversion = int(os.getenv("PYTHON_HEXVERSION"), 16) + except (TypeError, ValueError): + hexversion = sys.hexversion + return struct.pack(">i", sys.hexversion) + + +def _get_suffix(field4): + name = {0xA0: "a", 0xB0: "b", 0xC0: "c"}.get(field4 & 0xF0, "") + if name: + serial = field4 & 0x0F + return f"{name}{serial}" + return "" + + +VER_MAJOR, VER_MINOR, VER_MICRO, VER_FIELD4 = _unpack_hexversion() +VER_SUFFIX = _get_suffix(VER_FIELD4) VER_FIELD3 = VER_MICRO << 8 | VER_FIELD4 -VER_NAME = {"alpha": "a", "beta": "b", "candidate": "rc"}.get( - sys.version_info.releaselevel, "" -) -VER_SERIAL = sys.version_info.serial if VER_NAME else "" VER_DOT = "{}.{}".format(VER_MAJOR, VER_MINOR) PYTHON_DLL_NAME = "python{}{}.dll".format(VER_MAJOR, VER_MINOR) @@ -21,8 +37,6 @@ PYTHON_ZIP_NAME = "python{}{}.zip".format(VER_MAJOR, VER_MINOR) PYTHON_PTH_NAME = "python{}{}._pth".format(VER_MAJOR, VER_MINOR) -PYTHON_CHM_NAME = "python{}{}{}{}{}.chm".format( - VER_MAJOR, VER_MINOR, VER_MICRO, VER_NAME, VER_SERIAL +PYTHON_CHM_NAME = "python{}{}{}{}.chm".format( + VER_MAJOR, VER_MINOR, VER_MICRO, VER_SUFFIX ) - -IS_X64 = sys.maxsize > 2 ** 32 diff --git a/PC/layout/support/nuspec.py b/PC/layout/support/nuspec.py index ba26ff337e91e6..b85095c555fe0a 100644 --- a/PC/layout/support/nuspec.py +++ b/PC/layout/support/nuspec.py @@ -13,25 +13,21 @@ NUSPEC_DATA = { "PYTHON_TAG": VER_DOT, "PYTHON_VERSION": os.getenv("PYTHON_NUSPEC_VERSION"), - "PYTHON_BITNESS": "64-bit" if IS_X64 else "32-bit", - "PACKAGENAME": os.getenv("PYTHON_NUSPEC_PACKAGENAME"), - "PACKAGETITLE": os.getenv("PYTHON_NUSPEC_PACKAGETITLE"), "FILELIST": r' ', } -if not NUSPEC_DATA["PYTHON_VERSION"]: - if VER_NAME: - NUSPEC_DATA["PYTHON_VERSION"] = "{}.{}-{}{}".format( - VER_DOT, VER_MICRO, VER_NAME, VER_SERIAL - ) - else: - NUSPEC_DATA["PYTHON_VERSION"] = "{}.{}".format(VER_DOT, VER_MICRO) - -if not NUSPEC_DATA["PACKAGETITLE"]: - NUSPEC_DATA["PACKAGETITLE"] = "Python" if IS_X64 else "Python (32-bit)" +NUSPEC_PLATFORM_DATA = dict( + _keys=("PYTHON_BITNESS", "PACKAGENAME", "PACKAGETITLE"), + win32=("32-bit", "pythonx86", "Python (32-bit)"), + amd64=("64-bit", "python", "Python"), + arm32=("ARM", "pythonarm", "Python (ARM)"), + arm64=("ARM64", "pythonarm64", "Python (ARM64)"), +) -if not NUSPEC_DATA["PACKAGENAME"]: - NUSPEC_DATA["PACKAGENAME"] = "python" if IS_X64 else "pythonx86" +if not NUSPEC_DATA["PYTHON_VERSION"]: + NUSPEC_DATA["PYTHON_VERSION"] = "{}.{}{}{}".format( + VER_DOT, VER_MICRO, "-" if VER_SUFFIX else "", VER_SUFFIX + ) FILELIST_WITH_PROPS = r""" """ @@ -56,11 +52,21 @@ """ +def _get_nuspec_data_overrides(ns): + for k, v in zip(NUSPEC_PLATFORM_DATA["_keys"], NUSPEC_PLATFORM_DATA[ns.arch]): + ev = os.getenv("PYTHON_NUSPEC_" + k) + if ev: + yield k, ev + yield k, v + + def get_nuspec_layout(ns): if ns.include_all or ns.include_nuspec: - data = NUSPEC_DATA + data = dict(NUSPEC_DATA) + for k, v in _get_nuspec_data_overrides(ns): + if not data.get(k): + data[k] = v if ns.include_all or ns.include_props: - data = dict(data) data["FILELIST"] = FILELIST_WITH_PROPS nuspec = NUSPEC_TEMPLATE.format_map(data) yield "python.nuspec", ("python.nuspec", nuspec.encode("utf-8")) diff --git a/PC/layout/support/props.py b/PC/layout/support/props.py index 4d3b06195f6ed2..b1560b5244762f 100644 --- a/PC/layout/support/props.py +++ b/PC/layout/support/props.py @@ -18,15 +18,9 @@ } if not PROPS_DATA["PYTHON_VERSION"]: - if VER_NAME: - PROPS_DATA["PYTHON_VERSION"] = "{}.{}-{}{}".format( - VER_DOT, VER_MICRO, VER_NAME, VER_SERIAL - ) - else: - PROPS_DATA["PYTHON_VERSION"] = "{}.{}".format(VER_DOT, VER_MICRO) - -if not PROPS_DATA["PYTHON_PLATFORM"]: - PROPS_DATA["PYTHON_PLATFORM"] = "x64" if IS_X64 else "Win32" + PROPS_DATA["PYTHON_VERSION"] = "{}.{}{}{}".format( + VER_DOT, VER_MICRO, "-" if VER_SUFFIX else "", VER_SUFFIX + ) PROPS_DATA["PYTHON_TARGET"] = "_GetPythonRuntimeFilesDependsOn{}{}_{}".format( VER_MAJOR, VER_MINOR, PROPS_DATA["PYTHON_PLATFORM"] @@ -94,5 +88,13 @@ def get_props_layout(ns): if ns.include_all or ns.include_props: # TODO: Filter contents of props file according to included/excluded items - props = PROPS_TEMPLATE.format_map(PROPS_DATA) + d = dict(PROPS_DATA) + if not d.get("PYTHON_PLATFORM"): + d["PYTHON_PLATFORM"] = { + "win32": "Win32", + "amd64": "X64", + "arm32": "ARM", + "arm64": "ARM64", + }[ns.arch] + props = PROPS_TEMPLATE.format_map(d) yield "python.props", ("python.props", props.encode("utf-8")) diff --git a/PCbuild/build.bat b/PCbuild/build.bat index bce599329e73ab..623409c24ec04a 100644 --- a/PCbuild/build.bat +++ b/PCbuild/build.bat @@ -57,7 +57,7 @@ set conf=Release set target=Build set dir=%~dp0 set parallel=/m -set verbose=/nologo /v:m +set verbose=/nologo /v:m /clp:summary set kill= set do_pgo= set pgo_job=-m test --pgo diff --git a/PCbuild/find_msbuild.bat b/PCbuild/find_msbuild.bat index a2810f09c45e21..bc9d00c22c8268 100644 --- a/PCbuild/find_msbuild.bat +++ b/PCbuild/find_msbuild.bat @@ -32,7 +32,7 @@ @rem VS 2017 and later provide vswhere.exe, which can be used @if not exist "%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" goto :skip_vswhere @set _Py_MSBuild_Root= -@for /F "tokens=*" %%i in ('"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -property installationPath -latest') DO @(set _Py_MSBuild_Root=%%i\MSBuild) +@for /F "tokens=*" %%i in ('"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -property installationPath -latest -prerelease') DO @(set _Py_MSBuild_Root=%%i\MSBuild) @if not defined _Py_MSBuild_Root goto :skip_vswhere @for %%j in (Current 15.0) DO @if exist "%_Py_MSBuild_Root%\%%j\Bin\msbuild.exe" (set MSBUILD="%_Py_MSBuild_Root%\%%j\Bin\msbuild.exe") @set _Py_MSBuild_Root= diff --git a/PCbuild/pcbuild.proj b/PCbuild/pcbuild.proj index 35f173ff864ea4..22a9eed18d42bb 100644 --- a/PCbuild/pcbuild.proj +++ b/PCbuild/pcbuild.proj @@ -60,7 +60,7 @@ - + diff --git a/PCbuild/prepare_libffi.bat b/PCbuild/prepare_libffi.bat index c65a5f7010bd06..f41ba83379af96 100644 --- a/PCbuild/prepare_libffi.bat +++ b/PCbuild/prepare_libffi.bat @@ -93,7 +93,10 @@ echo LIBFFI_SOURCE: %LIBFFI_SOURCE% echo MSVCC : %MSVCC% echo. -if not exist Makefile.in (%SH% -lc "(cd $LIBFFI_SOURCE; ./autogen.sh;)") +if not exist Makefile.in ( + %SH% -lc "(cd $LIBFFI_SOURCE; ./autogen.sh;)" + if errorlevel 1 exit /B 1 +) if "%BUILD_X64%"=="1" call :BuildOne x64 x86_64-w64-cygwin x86_64-w64-cygwin if "%BUILD_X86%"=="1" call :BuildOne x86 i686-pc-cygwin i686-pc-cygwin @@ -158,11 +161,13 @@ echo ================================================================ echo Configure the build to generate fficonfig.h and ffi.h echo ================================================================ %SH% -lc "(cd $OLDPWD; ./configure CC='%MSVCC% %ASSEMBLER% %BUILD_PDB%' CXX='%MSVCC% %ASSEMBLER% %BUILD_PDB%' LD='link' CPP='cl -nologo -EP' CXXCPP='cl -nologo -EP' CPPFLAGS='-DFFI_BUILDING_DLL' %BUILD_NOOPT% NM='dumpbin -symbols' STRIP=':' --build=$BUILD --host=$HOST;)" +if errorlevel 1 exit /B %ERRORLEVEL% echo ================================================================ echo Building libffi echo ================================================================ %SH% -lc "(cd $OLDPWD; export PATH=/usr/bin:$PATH; cp src/%SRC_ARCHITECTURE%/ffitarget.h include; make; find .;)" +if errorlevel 1 exit /B %ERRORLEVEL% REM Tests are not needed to produce artifacts if "%LIBFFI_TEST%" EQU "1" ( diff --git a/PCbuild/python_uwp.vcxproj b/PCbuild/python_uwp.vcxproj index 14e138cbed3e00..5ff120a0da331a 100644 --- a/PCbuild/python_uwp.vcxproj +++ b/PCbuild/python_uwp.vcxproj @@ -1,6 +1,14 @@  + + Debug + ARM + + + Debug + ARM64 + Debug Win32 @@ -9,6 +17,14 @@ Debug x64 + + PGInstrument + ARM + + + PGInstrument + ARM64 + PGInstrument Win32 @@ -17,6 +33,14 @@ PGInstrument x64 + + PGUpdate + ARM + + + PGUpdate + ARM64 + PGUpdate Win32 @@ -25,6 +49,14 @@ PGUpdate x64 + + Release + ARM + + + Release + ARM64 + Release Win32 diff --git a/PCbuild/pythonw_uwp.vcxproj b/PCbuild/pythonw_uwp.vcxproj index e2c01710498eec..828d0d1ccac217 100644 --- a/PCbuild/pythonw_uwp.vcxproj +++ b/PCbuild/pythonw_uwp.vcxproj @@ -1,6 +1,14 @@  + + Debug + ARM + + + Debug + ARM64 + Debug Win32 @@ -9,6 +17,14 @@ Debug x64 + + PGInstrument + ARM + + + PGInstrument + ARM64 + PGInstrument Win32 @@ -17,6 +33,14 @@ PGInstrument x64 + + PGUpdate + ARM + + + PGUpdate + ARM64 + PGUpdate Win32 @@ -25,6 +49,14 @@ PGUpdate x64 + + Release + ARM + + + Release + ARM64 + Release Win32 From 836cf31a3cf468ed9598a220b8e194b366287bfe Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 6 Dec 2019 11:32:33 -0800 Subject: [PATCH 0983/2163] bpo-37931: Fix crash on OSX re-initializing os.environ (GH-15428) On most platforms, the `environ` symbol is accessible everywhere. In a dylib on OSX, it's not easily accessible, you need to find it with _NSGetEnviron. The code was caching the *value* of environ. But a setenv() can change the value, leaving garbage at the old value. Fix: don't cache the value of environ, just read it every time. (cherry picked from commit 723f71abf7ab0a7be394f9f7b2daa9ecdf6fb1eb) Co-authored-by: Benoit Hudson --- Misc/ACKS | 1 + .../macOS/2019-08-23-12-14-34.bpo-37931.goYgQj.rst | 3 +++ Modules/posixmodule.c | 10 +++++----- 3 files changed, 9 insertions(+), 5 deletions(-) create mode 100644 Misc/NEWS.d/next/macOS/2019-08-23-12-14-34.bpo-37931.goYgQj.rst diff --git a/Misc/ACKS b/Misc/ACKS index e52ae984f15714..62c5928c508ff9 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -725,6 +725,7 @@ Miro Hrončok Chiu-Hsiang Hsu Chih-Hao Huang Christian Hudon +Benoît Hudson Lawrence Hudson Michael Hudson Jim Hugunin diff --git a/Misc/NEWS.d/next/macOS/2019-08-23-12-14-34.bpo-37931.goYgQj.rst b/Misc/NEWS.d/next/macOS/2019-08-23-12-14-34.bpo-37931.goYgQj.rst new file mode 100644 index 00000000000000..45b54e89cb89e6 --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2019-08-23-12-14-34.bpo-37931.goYgQj.rst @@ -0,0 +1,3 @@ +Fixed a crash on OSX dynamic builds that occurred when re-initializing the +posix module after a Py_Finalize if the environment had changed since the +previous `import posix`. Patch by Benoît Hudson. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 7c823434e67fe0..850769fd95eef7 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -1372,7 +1372,6 @@ win32_get_reparse_tag(HANDLE reparse_point_handle, ULONG *reparse_tag) ** man environ(7). */ #include -static char **environ; #elif !defined(_MSC_VER) && (!defined(__WATCOMC__) || defined(__QNX__) || defined(__VXWORKS__)) extern char **environ; #endif /* !_MSC_VER */ @@ -1390,15 +1389,16 @@ convertenviron(void) d = PyDict_New(); if (d == NULL) return NULL; -#if defined(WITH_NEXT_FRAMEWORK) || (defined(__APPLE__) && defined(Py_ENABLE_SHARED)) - if (environ == NULL) - environ = *_NSGetEnviron(); -#endif #ifdef MS_WINDOWS /* _wenviron must be initialized in this way if the program is started through main() instead of wmain(). */ _wgetenv(L""); e = _wenviron; +#elif defined(WITH_NEXT_FRAMEWORK) || (defined(__APPLE__) && defined(Py_ENABLE_SHARED)) + /* environ is not accessible as an extern in a shared object on OSX; use + _NSGetEnviron to resolve it. The value changes if you add environment + variables between calls to Py_Initialize, so don't cache the value. */ + e = *_NSGetEnviron(); #else e = environ; #endif From 7fde4f446a3dcfed780a38fbfcd7c0b4d9d73b93 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 7 Dec 2019 03:39:57 -0800 Subject: [PATCH 0984/2163] bpo-38529: Fix asyncio stream warning (GH-17474) (cherry picked from commit 7ddcd0caa4c2e6b43265df144f59c5aa508a94f2) Co-authored-by: Andrew Svetlov --- Lib/asyncio/streams.py | 19 +------ Lib/test/test_asyncio/test_streams.py | 53 ------------------- .../2019-12-05-16-13-25.bpo-38529.yvQgx3.rst | 2 + 3 files changed, 3 insertions(+), 71 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-05-16-13-25.bpo-38529.yvQgx3.rst diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py index 795530e6f69efe..3c80bb88925905 100644 --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -214,8 +214,7 @@ class StreamReaderProtocol(FlowControlMixin, protocols.Protocol): def __init__(self, stream_reader, client_connected_cb=None, loop=None): super().__init__(loop=loop) if stream_reader is not None: - self._stream_reader_wr = weakref.ref(stream_reader, - self._on_reader_gc) + self._stream_reader_wr = weakref.ref(stream_reader) self._source_traceback = stream_reader._source_traceback else: self._stream_reader_wr = None @@ -231,22 +230,6 @@ def __init__(self, stream_reader, client_connected_cb=None, loop=None): self._over_ssl = False self._closed = self._loop.create_future() - def _on_reader_gc(self, wr): - transport = self._transport - if transport is not None: - # connection_made was called - context = { - 'message': ('An open stream object is being garbage ' - 'collected; call "stream.close()" explicitly.') - } - if self._source_traceback: - context['source_traceback'] = self._source_traceback - self._loop.call_exception_handler(context) - transport.abort() - else: - self._reject_connection = True - self._stream_reader_wr = None - @property def _stream_reader(self): if self._stream_reader_wr is None: diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py index b9413ab35fc50a..12bd536911dbb2 100644 --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -924,59 +924,6 @@ def test_wait_closed_on_close_with_unread_data(self): wr.close() self.loop.run_until_complete(wr.wait_closed()) - def test_del_stream_before_sock_closing(self): - messages = [] - self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) - - with test_utils.run_test_server() as httpd: - with self.assertWarns(DeprecationWarning): - rd, wr = self.loop.run_until_complete( - asyncio.open_connection(*httpd.address, loop=self.loop)) - sock = wr.get_extra_info('socket') - self.assertNotEqual(sock.fileno(), -1) - - wr.write(b'GET / HTTP/1.0\r\n\r\n') - f = rd.readline() - data = self.loop.run_until_complete(f) - self.assertEqual(data, b'HTTP/1.0 200 OK\r\n') - - # drop refs to reader/writer - del rd - del wr - gc.collect() - # make a chance to close the socket - test_utils.run_briefly(self.loop) - - self.assertEqual(1, len(messages)) - self.assertEqual(sock.fileno(), -1) - - self.assertEqual(1, len(messages)) - self.assertEqual('An open stream object is being garbage ' - 'collected; call "stream.close()" explicitly.', - messages[0]['message']) - - def test_del_stream_before_connection_made(self): - messages = [] - self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) - - with test_utils.run_test_server() as httpd: - rd = asyncio.StreamReader(loop=self.loop) - pr = asyncio.StreamReaderProtocol(rd, loop=self.loop) - del rd - gc.collect() - tr, _ = self.loop.run_until_complete( - self.loop.create_connection( - lambda: pr, *httpd.address)) - - sock = tr.get_extra_info('socket') - self.assertEqual(sock.fileno(), -1) - - self.assertEqual(1, len(messages)) - self.assertEqual('An open stream was garbage collected prior to ' - 'establishing network connection; ' - 'call "stream.close()" explicitly.', - messages[0]['message']) - def test_async_writer_api(self): async def inner(httpd): rd, wr = await asyncio.open_connection(*httpd.address) diff --git a/Misc/NEWS.d/next/Library/2019-12-05-16-13-25.bpo-38529.yvQgx3.rst b/Misc/NEWS.d/next/Library/2019-12-05-16-13-25.bpo-38529.yvQgx3.rst new file mode 100644 index 00000000000000..c688926b4a49b0 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-05-16-13-25.bpo-38529.yvQgx3.rst @@ -0,0 +1,2 @@ +Drop too noisy asyncio warning about deletion of a stream without explicit +``.close()`` call. From ce0a2a86207dacc1945c6d756527226d8feed0e0 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 7 Dec 2019 03:41:41 -0800 Subject: [PATCH 0985/2163] Make repr of C accelerated TaskWakeupMethWrapper the same as of pure Python version (GH-17484) (cherry picked from commit 969ae7aca809a8dacafee04c261110eea0ac1945) Co-authored-by: Andrew Svetlov --- .../2019-12-06-15-11-42.bpo-38986.bg6iZt.rst | 2 ++ Modules/_asynciomodule.c | 18 +++++++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-06-15-11-42.bpo-38986.bg6iZt.rst diff --git a/Misc/NEWS.d/next/Library/2019-12-06-15-11-42.bpo-38986.bg6iZt.rst b/Misc/NEWS.d/next/Library/2019-12-06-15-11-42.bpo-38986.bg6iZt.rst new file mode 100644 index 00000000000000..777535299be17a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-06-15-11-42.bpo-38986.bg6iZt.rst @@ -0,0 +1,2 @@ +Make repr of C accelerated TaskWakeupMethWrapper the same as of pure Python +version. diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c index d5f845ef0bed11..f4efa2120a9c85 100644 --- a/Modules/_asynciomodule.c +++ b/Modules/_asynciomodule.c @@ -1811,6 +1811,21 @@ TaskWakeupMethWrapper_dealloc(TaskWakeupMethWrapper *o) Py_TYPE(o)->tp_free(o); } +static PyObject * +TaskWakeupMethWrapper_get___self__(TaskWakeupMethWrapper *o, void *Py_UNUSED(ignored)) +{ + if (o->ww_task) { + Py_INCREF(o->ww_task); + return (PyObject*)o->ww_task; + } + Py_RETURN_NONE; +} + +static PyGetSetDef TaskWakeupMethWrapper_getsetlist[] = { + {"__self__", (getter)TaskWakeupMethWrapper_get___self__, NULL, NULL}, + {NULL} /* Sentinel */ +}; + static PyTypeObject TaskWakeupMethWrapper_Type = { PyVarObject_HEAD_INIT(NULL, 0) "TaskWakeupMethWrapper", @@ -1822,6 +1837,7 @@ static PyTypeObject TaskWakeupMethWrapper_Type = { .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, .tp_traverse = (traverseproc)TaskWakeupMethWrapper_traverse, .tp_clear = (inquiry)TaskWakeupMethWrapper_clear, + .tp_getset = TaskWakeupMethWrapper_getsetlist, }; static PyObject * @@ -3258,7 +3274,7 @@ module_init(void) } if (module_initialized != 0) { return 0; - } + } else { module_initialized = 1; } From 930cef2770b641f49e69b67840daaa53b65cd0e0 Mon Sep 17 00:00:00 2001 From: Andrew Svetlov Date: Sat, 7 Dec 2019 14:44:20 +0200 Subject: [PATCH 0986/2163] [3.8] bpo-37404: Raising value error if an SSLSocket is passed to asyncio functions (GH-16457) (#17496) https://bugs.python.org/issue37404 (cherry picked from commit 892f9e0777f262d366d4747a54c33a1c15a49da6) Co-authored-by: idomic --- Lib/asyncio/selector_events.py | 10 ++++++++++ .../Build/2019-12-01-21-45-24.bpo-37404.cNsA7S.rst | 2 ++ 2 files changed, 12 insertions(+) create mode 100644 Misc/NEWS.d/next/Build/2019-12-01-21-45-24.bpo-37404.cNsA7S.rst diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py index 00e3244bfb294c..e1abf5118619cc 100644 --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -348,6 +348,8 @@ async def sock_recv(self, sock, n): The maximum amount of data to be received at once is specified by nbytes. """ + if isinstance(sock, ssl.SSLSocket): + raise TypeError("Socket cannot be of type SSLSocket") if self._debug and sock.gettimeout() != 0: raise ValueError("the socket must be non-blocking") try: @@ -386,6 +388,8 @@ async def sock_recv_into(self, sock, buf): The received data is written into *buf* (a writable buffer). The return value is the number of bytes written. """ + if isinstance(sock, ssl.SSLSocket): + raise TypeError("Socket cannot be of type SSLSocket") if self._debug and sock.gettimeout() != 0: raise ValueError("the socket must be non-blocking") try: @@ -425,6 +429,8 @@ async def sock_sendall(self, sock, data): raised, and there is no way to determine how much data, if any, was successfully processed by the receiving end of the connection. """ + if isinstance(sock, ssl.SSLSocket): + raise TypeError("Socket cannot be of type SSLSocket") if self._debug and sock.gettimeout() != 0: raise ValueError("the socket must be non-blocking") try: @@ -472,6 +478,8 @@ async def sock_connect(self, sock, address): This method is a coroutine. """ + if isinstance(sock, ssl.SSLSocket): + raise TypeError("Socket cannot be of type SSLSocket") if self._debug and sock.gettimeout() != 0: raise ValueError("the socket must be non-blocking") @@ -533,6 +541,8 @@ async def sock_accept(self, sock): object usable to send and receive data on the connection, and address is the address bound to the socket on the other end of the connection. """ + if isinstance(sock, ssl.SSLSocket): + raise TypeError("Socket cannot be of type SSLSocket") if self._debug and sock.gettimeout() != 0: raise ValueError("the socket must be non-blocking") fut = self.create_future() diff --git a/Misc/NEWS.d/next/Build/2019-12-01-21-45-24.bpo-37404.cNsA7S.rst b/Misc/NEWS.d/next/Build/2019-12-01-21-45-24.bpo-37404.cNsA7S.rst new file mode 100644 index 00000000000000..067fc9d3f189dc --- /dev/null +++ b/Misc/NEWS.d/next/Build/2019-12-01-21-45-24.bpo-37404.cNsA7S.rst @@ -0,0 +1,2 @@ +:mod:`asyncio` now raises :exc:`TyperError` when calling incompatible methods +with an :class:`ssl.SSLSocket` socket. Patch by Ido Michael. From 9d3cacd5901f8fbbc4f8b78fc35abad01a0e6546 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 7 Dec 2019 09:20:27 -0800 Subject: [PATCH 0987/2163] [3.8] bpo-38820: OpenSSL 3.0.0 compatibility. (GH-17190) (GH-17499) test_openssl_version now accepts version 3.0.0. getpeercert() no longer returns IPv6 addresses with a trailing new line. Signed-off-by: Christian Heimes https://bugs.python.org/issue38820 (cherry picked from commit 2b7de6696bf2f924cd2cd9ff0a539c8aa37c6244) Co-authored-by: Christian Heimes https://bugs.python.org/issue38820 Automerge-Triggered-By: @tiran --- Doc/library/ssl.rst | 3 ++ Lib/test/test_ssl.py | 12 ++--- .../2019-11-16-16-09-07.bpo-38820.ivhUSV.rst | 2 + Modules/_ssl.c | 49 ++++++++++++++++++- 4 files changed, 59 insertions(+), 7 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-11-16-16-09-07.bpo-38820.ivhUSV.rst diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index f8fa25df6c2419..bbb4c412d9303e 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1256,6 +1256,9 @@ SSL sockets also have the following additional methods and attributes: The returned dictionary includes additional X509v3 extension items such as ``crlDistributionPoints``, ``caIssuers`` and ``OCSP`` URIs. + .. versionchanged:: 3.8.1 + IPv6 address strings no longer have a trailing new line. + .. method:: SSLSocket.cipher() Returns a three-value tuple containing the name of the cipher being used, the diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py index 539cb7751db89d..0bc0a8c4522d52 100644 --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -485,7 +485,7 @@ def test_parse_cert_CVE_2013_4238(self): ('email', 'null@python.org\x00user@example.org'), ('URI', 'http://null.python.org\x00http://example.org'), ('IP Address', '192.0.2.1'), - ('IP Address', '2001:DB8:0:0:0:0:0:1\n')) + ('IP Address', '2001:DB8:0:0:0:0:0:1')) else: # OpenSSL 0.9.7 doesn't support IPv6 addresses in subjectAltName san = (('DNS', 'altnull.python.org\x00example.com'), @@ -512,7 +512,7 @@ def test_parse_all_sans(self): (('commonName', 'dirname example'),))), ('URI', 'https://www.python.org/'), ('IP Address', '127.0.0.1'), - ('IP Address', '0:0:0:0:0:0:0:1\n'), + ('IP Address', '0:0:0:0:0:0:0:1'), ('Registered ID', '1.2.3.4.5') ) ) @@ -539,11 +539,11 @@ def test_openssl_version(self): # Some sanity checks follow # >= 0.9 self.assertGreaterEqual(n, 0x900000) - # < 3.0 - self.assertLess(n, 0x30000000) + # < 4.0 + self.assertLess(n, 0x40000000) major, minor, fix, patch, status = t - self.assertGreaterEqual(major, 0) - self.assertLess(major, 3) + self.assertGreaterEqual(major, 1) + self.assertLess(major, 4) self.assertGreaterEqual(minor, 0) self.assertLess(minor, 256) self.assertGreaterEqual(fix, 0) diff --git a/Misc/NEWS.d/next/Library/2019-11-16-16-09-07.bpo-38820.ivhUSV.rst b/Misc/NEWS.d/next/Library/2019-11-16-16-09-07.bpo-38820.ivhUSV.rst new file mode 100644 index 00000000000000..2c6a6e853c25f6 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-11-16-16-09-07.bpo-38820.ivhUSV.rst @@ -0,0 +1,2 @@ +Make Python compatible with OpenSSL 3.0.0. :func:`ssl.SSLSocket.getpeercert` +no longer returns IPv6 addresses with a trailing new line. diff --git a/Modules/_ssl.c b/Modules/_ssl.c index 6f1f9c881530f0..43b236c2121203 100644 --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -1410,6 +1410,54 @@ _get_peer_alt_names (X509 *certificate) { PyTuple_SET_ITEM(t, 1, v); break; + case GEN_IPADD: + /* OpenSSL < 3.0.0 adds a trailing \n to IPv6. 3.0.0 removed + * the trailing newline. Remove it in all versions + */ + t = PyTuple_New(2); + if (t == NULL) + goto fail; + + v = PyUnicode_FromString("IP Address"); + if (v == NULL) { + Py_DECREF(t); + goto fail; + } + PyTuple_SET_ITEM(t, 0, v); + + if (name->d.ip->length == 4) { + unsigned char *p = name->d.ip->data; + v = PyUnicode_FromFormat( + "%d.%d.%d.%d", + p[0], p[1], p[2], p[3] + ); + } else if (name->d.ip->length == 16) { + /* PyUnicode_FromFormat() does not support %X */ + unsigned char *p = name->d.ip->data; + len = sprintf( + buf, + "%X:%X:%X:%X:%X:%X:%X:%X", + p[0] << 8 | p[1], + p[2] << 8 | p[3], + p[4] << 8 | p[5], + p[6] << 8 | p[7], + p[8] << 8 | p[9], + p[10] << 8 | p[11], + p[12] << 8 | p[13], + p[14] << 8 | p[15] + ); + v = PyUnicode_FromStringAndSize(buf, len); + } else { + v = PyUnicode_FromString(""); + } + + if (v == NULL) { + Py_DECREF(t); + goto fail; + } + PyTuple_SET_ITEM(t, 1, v); + break; + default: /* for everything else, we use the OpenSSL print form */ switch (gntype) { @@ -1417,7 +1465,6 @@ _get_peer_alt_names (X509 *certificate) { case GEN_OTHERNAME: case GEN_X400: case GEN_EDIPARTY: - case GEN_IPADD: case GEN_RID: break; default: From 960fca1a5887a277fd6031cf4c4b6fb31b08ebf5 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 8 Dec 2019 04:49:07 -0800 Subject: [PATCH 0988/2163] bpo-38979: fix ContextVar "__class_getitem__" method (GH-17497) now contextvars.ContextVar "__class_getitem__" method returns ContextVar class, not None. https://bugs.python.org/issue38979 Automerge-Triggered-By: @asvetlov (cherry picked from commit 28c91631c24e53713ad0e8a2bbae716373f5e53d) Co-authored-by: AMIR <31338382+amiremohamadi@users.noreply.github.com> --- Lib/test/test_context.py | 7 ++++--- .../next/Library/2019-12-07-16-32-42.bpo-38979.q0sIHy.rst | 1 + Python/context.c | 7 ++++--- 3 files changed, 9 insertions(+), 6 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-07-16-32-42.bpo-38979.q0sIHy.rst diff --git a/Lib/test/test_context.py b/Lib/test/test_context.py index efd7319a23ae02..b9e991a4000929 100644 --- a/Lib/test/test_context.py +++ b/Lib/test/test_context.py @@ -38,9 +38,6 @@ def test_context_var_new_1(self): self.assertNotEqual(hash(c), hash('aaa')) - def test_context_var_new_2(self): - self.assertIsNone(contextvars.ContextVar[int]) - @isolated_context def test_context_var_repr_1(self): c = contextvars.ContextVar('a') @@ -361,6 +358,10 @@ def sub(num): tp.shutdown() self.assertEqual(results, list(range(10))) + def test_contextvar_getitem(self): + clss = contextvars.ContextVar + self.assertEqual(clss[str], clss) + # HAMT Tests diff --git a/Misc/NEWS.d/next/Library/2019-12-07-16-32-42.bpo-38979.q0sIHy.rst b/Misc/NEWS.d/next/Library/2019-12-07-16-32-42.bpo-38979.q0sIHy.rst new file mode 100644 index 00000000000000..6a91a12e4930a9 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-07-16-32-42.bpo-38979.q0sIHy.rst @@ -0,0 +1 @@ +Return class from ``ContextVar.__class_getitem__`` to simplify subclassing. diff --git a/Python/context.c b/Python/context.c index f48c376b4ffaa9..5c30e47f35dd78 100644 --- a/Python/context.c +++ b/Python/context.c @@ -1010,9 +1010,10 @@ _contextvars_ContextVar_reset(PyContextVar *self, PyObject *token) static PyObject * -contextvar_cls_getitem(PyObject *self, PyObject *args) +contextvar_cls_getitem(PyObject *self, PyObject *arg) { - Py_RETURN_NONE; + Py_INCREF(self); + return self; } static PyMemberDef PyContextVar_members[] = { @@ -1025,7 +1026,7 @@ static PyMethodDef PyContextVar_methods[] = { _CONTEXTVARS_CONTEXTVAR_SET_METHODDEF _CONTEXTVARS_CONTEXTVAR_RESET_METHODDEF {"__class_getitem__", contextvar_cls_getitem, - METH_VARARGS | METH_STATIC, NULL}, + METH_O | METH_CLASS, NULL}, {NULL, NULL} }; From f66f4a09d0b6817fe6a86a567fd506aa223f1563 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 8 Dec 2019 18:11:31 -0800 Subject: [PATCH 0989/2163] bpo-38698: Add a new InvalidMessageID token to email header parser. (GH-17503) This adds a new InvalidMessageID token to the email header parser which can be used to represent invalid message-id headers in the parse tree. (cherry picked from commit 68157da8b42b26408af5d157d2dba4fcf29c6320) Co-authored-by: Abhilash Raj --- Lib/email/_header_value_parser.py | 20 ++++++++-- .../test_email/test__header_value_parser.py | 40 +++++++++++++++++-- .../2019-12-07-21-49-50.bpo-38698.HxoSym.rst | 3 ++ 3 files changed, 56 insertions(+), 7 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-07-21-49-50.bpo-38698.HxoSym.rst diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py index abdef8189ca6fb..cb013225ec60cb 100644 --- a/Lib/email/_header_value_parser.py +++ b/Lib/email/_header_value_parser.py @@ -850,10 +850,15 @@ def fold(self, policy): # message-id tokens may not be folded. return str(self) + policy.linesep + class MessageID(MsgID): token_type = 'message-id' +class InvalidMessageID(MessageID): + token_type = 'invalid-message-id' + + class Header(TokenList): token_type = 'header' @@ -2110,11 +2115,18 @@ def parse_message_id(value): message_id = MessageID() try: token, value = get_msg_id(value) - except errors.HeaderParseError: - message_id.defects.append(errors.InvalidHeaderDefect( - "Expected msg-id but found {!r}".format(value))) - else: message_id.append(token) + except errors.HeaderParseError as ex: + token = get_unstructured(value) + message_id = InvalidMessageID(token) + message_id.defects.append( + errors.InvalidHeaderDefect("Invalid msg-id: {!r}".format(ex))) + else: + # Value after parsing a valid msg_id should be None. + if value: + message_id.defects.append(errors.InvalidHeaderDefect( + "Unexpected {!r}".format(value))) + return message_id # diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py index 2f63a3b3e05246..7c9f9877d7be2a 100644 --- a/Lib/test/test_email/test__header_value_parser.py +++ b/Lib/test/test_email/test__header_value_parser.py @@ -2639,10 +2639,44 @@ def test_get_msg_id_no_id_right_part(self): self.assertEqual(msg_id.token_type, 'msg-id') def test_get_msg_id_invalid_expected_msg_id_not_found(self): - text = "Message-Id: 935-XPB-567:0:86089:180874:0:45327:9:90305:17843586-40@example.com" + text = "935-XPB-567:0:45327:9:90305:17843586-40@example.com" msg_id = parser.parse_message_id(text) - self.assertDefectsEqual(msg_id.all_defects, - [errors.InvalidHeaderDefect]) + self.assertDefectsEqual( + msg_id.all_defects, + [errors.InvalidHeaderDefect]) + + def test_parse_invalid_message_id(self): + message_id = self._test_parse_x( + parser.parse_message_id, + "935-XPB-567:0:45327:9:90305:17843586-40@example.com", + "935-XPB-567:0:45327:9:90305:17843586-40@example.com", + "935-XPB-567:0:45327:9:90305:17843586-40@example.com", + [errors.InvalidHeaderDefect], + ) + self.assertEqual(message_id.token_type, 'invalid-message-id') + + def test_parse_valid_message_id(self): + message_id = self._test_parse_x( + parser.parse_message_id, + "", + "", + "", + [], + ) + self.assertEqual(message_id.token_type, 'message-id') + + def test_parse_message_id_with_remaining(self): + message_id = self._test_parse_x( + parser.parse_message_id, + "thensomething", + "", + "", + [errors.InvalidHeaderDefect], + [], + ) + self.assertEqual(message_id.token_type, 'message-id') + self.assertEqual(str(message_id.all_defects[0]), + "Unexpected 'thensomething'") def test_get_msg_id_no_angle_start(self): with self.assertRaises(errors.HeaderParseError): diff --git a/Misc/NEWS.d/next/Library/2019-12-07-21-49-50.bpo-38698.HxoSym.rst b/Misc/NEWS.d/next/Library/2019-12-07-21-49-50.bpo-38698.HxoSym.rst new file mode 100644 index 00000000000000..b930dea0fa7bc7 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-07-21-49-50.bpo-38698.HxoSym.rst @@ -0,0 +1,3 @@ +Add a new ``InvalidMessageID`` token to email parser to represent invalid +Message-ID headers. Also, add defects when there is remaining value after +parsing the header. From 2abd3a8f580e6c7b1ce88b2ae9f9a783f4aea5d3 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 8 Dec 2019 18:12:50 -0800 Subject: [PATCH 0990/2163] bpo-38708: email: Fix a potential IndexError when parsing Message-ID (GH-17504) Fix a potential IndexError when passing an empty value to the message-id parser. Instead, HeaderParseError should be raised. (cherry picked from commit 3ae4ea1931361dd2743e464790e739d9285501bf) Co-authored-by: Abhilash Raj --- Lib/email/_header_value_parser.py | 2 +- Lib/test/test_email/test__header_value_parser.py | 6 ++++++ .../next/Library/2019-12-07-22-25-39.bpo-38708.rZTUfk.rst | 1 + 3 files changed, 8 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-07-22-25-39.bpo-38708.rZTUfk.rst diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py index cb013225ec60cb..9c55ef7fb453be 100644 --- a/Lib/email/_header_value_parser.py +++ b/Lib/email/_header_value_parser.py @@ -2047,7 +2047,7 @@ def get_msg_id(value): no-fold-literal = "[" *dtext "]" """ msg_id = MsgID() - if value[0] in CFWS_LEADER: + if value and value[0] in CFWS_LEADER: token, value = get_cfws(value) msg_id.append(token) if not value or value[0] != '<': diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py index 7c9f9877d7be2a..4b5b44258b228f 100644 --- a/Lib/test/test_email/test__header_value_parser.py +++ b/Lib/test/test_email/test__header_value_parser.py @@ -2583,6 +2583,11 @@ def test_invalid_content_transfer_encoding(self): # get_msg_id + def test_get_msg_id_empty(self): + # bpo-38708: Test that HeaderParseError is raised and not IndexError. + with self.assertRaises(errors.HeaderParseError): + parser.get_msg_id('') + def test_get_msg_id_valid(self): msg_id = self._test_get_x( parser.get_msg_id, @@ -2694,6 +2699,7 @@ def test_get_msg_id_no_angle_end(self): self.assertEqual(msg_id.token_type, 'msg-id') + @parameterize class Test_parse_mime_parameters(TestParserMixin, TestEmailBase): diff --git a/Misc/NEWS.d/next/Library/2019-12-07-22-25-39.bpo-38708.rZTUfk.rst b/Misc/NEWS.d/next/Library/2019-12-07-22-25-39.bpo-38708.rZTUfk.rst new file mode 100644 index 00000000000000..23a0a46d1fea14 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-07-22-25-39.bpo-38708.rZTUfk.rst @@ -0,0 +1 @@ +Fix a potential IndexError in email parser when parsing an empty msg-id. From 184a3812b81e2f7d4bc6453bf7ceabe8ac590202 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 8 Dec 2019 20:56:19 -0800 Subject: [PATCH 0991/2163] bpo-38673: dont switch to ps2 if the line starts with comment or whitespace (GH-17421) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit https://bugs.python.org/issue38673 (cherry picked from commit 109fc2792a490ee5cd8a423e17d415fbdedec5c8) Co-authored-by: Batuhan Taşkaya <47358913+isidentical@users.noreply.github.com> --- .../2019-12-01-00-17-44.bpo-38673.K_Tze-.rst | 1 + Parser/tokenizer.c | 6 ++++++ 2 files changed, 7 insertions(+) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-12-01-00-17-44.bpo-38673.K_Tze-.rst diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-12-01-00-17-44.bpo-38673.K_Tze-.rst b/Misc/NEWS.d/next/Core and Builtins/2019-12-01-00-17-44.bpo-38673.K_Tze-.rst new file mode 100644 index 00000000000000..8f8cf88e5e2103 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2019-12-01-00-17-44.bpo-38673.K_Tze-.rst @@ -0,0 +1 @@ +In REPL mode, don't switch to PS2 if the line starts with comment or whitespace. Based on work by Batuhan Taşkaya. diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 5763e47c4b00b3..f84093dae5b62e 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -1148,6 +1148,12 @@ tok_get(struct tok_state *tok, char **p_start, char **p_end) if (col == 0 && c == '\n' && tok->prompt != NULL) { blankline = 0; /* Let it through */ } + else if (tok->prompt != NULL && tok->lineno == 1) { + /* In interactive mode, if the first line contains + only spaces and/or a comment, let it through. */ + blankline = 0; + col = altcol = 0; + } else { blankline = 1; /* Ignore completely */ } From 4594565b56e9c99d2d3fb7549041bbca5ecba8e2 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 8 Dec 2019 22:59:04 -0800 Subject: [PATCH 0992/2163] bpo-38669: patch.object now raises a helpful error (GH17510) This means a clearer message is now shown when patch.object is called with two string arguments, rather than a class and a string argument. (cherry picked from commit cd90a52983db34896a6335a572d55bdda274778f) Co-authored-by: Elena Oat --- Lib/unittest/mock.py | 4 ++++ Lib/unittest/test/testmock/testpatch.py | 4 ++++ .../next/Tests/2019-11-04-02-54-16.bpo-38669.pazXZ8.rst | 1 + 3 files changed, 9 insertions(+) create mode 100644 Misc/NEWS.d/next/Tests/2019-11-04-02-54-16.bpo-38669.pazXZ8.rst diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py index d6e30679456a50..e92ccf168dbbde 100644 --- a/Lib/unittest/mock.py +++ b/Lib/unittest/mock.py @@ -1587,6 +1587,10 @@ def _patch_object( When used as a class decorator `patch.object` honours `patch.TEST_PREFIX` for choosing which methods to wrap. """ + if type(target) is str: + raise TypeError( + f"{target!r} must be the actual object to be patched, not a str" + ) getter = lambda: target return _patch( getter, attribute, new, spec, create, diff --git a/Lib/unittest/test/testmock/testpatch.py b/Lib/unittest/test/testmock/testpatch.py index 0632d95e58fec0..e065a2c35fbeec 100644 --- a/Lib/unittest/test/testmock/testpatch.py +++ b/Lib/unittest/test/testmock/testpatch.py @@ -105,6 +105,10 @@ def test(): self.assertEqual(Something.attribute, sentinel.Original, "patch not restored") + def test_patchobject_with_string_as_target(self): + msg = "'Something' must be the actual object to be patched, not a str" + with self.assertRaisesRegex(TypeError, msg): + patch.object('Something', 'do_something') def test_patchobject_with_none(self): class Something(object): diff --git a/Misc/NEWS.d/next/Tests/2019-11-04-02-54-16.bpo-38669.pazXZ8.rst b/Misc/NEWS.d/next/Tests/2019-11-04-02-54-16.bpo-38669.pazXZ8.rst new file mode 100644 index 00000000000000..5060ecf2dc5a46 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2019-11-04-02-54-16.bpo-38669.pazXZ8.rst @@ -0,0 +1 @@ +Raise :exc:`TypeError` when passing target as a string with :meth:`unittest.mock.patch.object`. \ No newline at end of file From d08fd298dc8d5631f6c504d01ee4f9cfb47db79d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 9 Dec 2019 03:15:23 -0800 Subject: [PATCH 0993/2163] bpo-38547: Fix test_pty if the process is the session leader (GH-17519) Fix test_pty: if the process is the session leader, closing the master file descriptor raises a SIGHUP signal: simply ignore SIGHUP when running the tests. (cherry picked from commit a1838ec2592e5082c75c77888f2a7a3eb21133e5) Co-authored-by: Victor Stinner --- Lib/test/test_pty.py | 19 ++++++++++++++++--- .../2019-12-09-11-32-34.bpo-38547.Juw54e.rst | 3 +++ 2 files changed, 19 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Tests/2019-12-09-11-32-34.bpo-38547.Juw54e.rst diff --git a/Lib/test/test_pty.py b/Lib/test/test_pty.py index 3b448569a2ffcb..ce85f575a08308 100644 --- a/Lib/test/test_pty.py +++ b/Lib/test/test_pty.py @@ -66,16 +66,27 @@ def _readline(fd): # XXX(nnorwitz): these tests leak fds when there is an error. class PtyTest(unittest.TestCase): def setUp(self): - # isatty() and close() can hang on some platforms. Set an alarm - # before running the test to make sure we don't hang forever. old_alarm = signal.signal(signal.SIGALRM, self.handle_sig) self.addCleanup(signal.signal, signal.SIGALRM, old_alarm) + + old_sighup = signal.signal(signal.SIGHUP, self.handle_sighup) + self.addCleanup(signal.signal, signal.SIGHUP, old_alarm) + + # isatty() and close() can hang on some platforms. Set an alarm + # before running the test to make sure we don't hang forever. self.addCleanup(signal.alarm, 0) signal.alarm(10) def handle_sig(self, sig, frame): self.fail("isatty hung") + @staticmethod + def handle_sighup(sig, frame): + # if the process is the session leader, os.close(master_fd) + # of "master_fd, slave_name = pty.master_open()" raises SIGHUP + # signal: just ignore the signal. + pass + def test_basic(self): try: debug("Calling master_open()") @@ -122,9 +133,11 @@ def test_basic(self): self.assertEqual(b'For my pet fish, Eric.\n', normalize_output(s2)) os.close(slave_fd) + # closing master_fd can raise a SIGHUP if the process is + # the session leader: we installed a SIGHUP signal handler + # to ignore this signal. os.close(master_fd) - def test_fork(self): debug("calling pty.fork()") pid, master_fd = pty.fork() diff --git a/Misc/NEWS.d/next/Tests/2019-12-09-11-32-34.bpo-38547.Juw54e.rst b/Misc/NEWS.d/next/Tests/2019-12-09-11-32-34.bpo-38547.Juw54e.rst new file mode 100644 index 00000000000000..10f3cc08511ddb --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2019-12-09-11-32-34.bpo-38547.Juw54e.rst @@ -0,0 +1,3 @@ +Fix test_pty: if the process is the session leader, closing the master file +descriptor raises a SIGHUP signal: simply ignore SIGHUP when running the +tests. From 0381ea79ac2da03179c8512c581cac588b69cff9 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 9 Dec 2019 14:46:33 +0100 Subject: [PATCH 0994/2163] bpo-38916: Document array.array deprecation (GH-17523) array.array: Document that tostring() and fromstring() deprecated aliases will be removed in Python 3.9. --- Doc/library/array.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Doc/library/array.rst b/Doc/library/array.rst index 2ae2a071262a17..6809c512b3f020 100644 --- a/Doc/library/array.rst +++ b/Doc/library/array.rst @@ -173,6 +173,8 @@ The following data items and methods are also supported: Deprecated alias for :meth:`frombytes`. + .. deprecated-removed:: 3.2 3.9 + .. method:: array.fromunicode(s) @@ -235,6 +237,8 @@ The following data items and methods are also supported: Deprecated alias for :meth:`tobytes`. + .. deprecated-removed:: 3.2 3.9 + .. method:: array.tounicode() From b22183f2738e0e6c23f8c5fb5b232768c184ec96 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 9 Dec 2019 06:19:48 -0800 Subject: [PATCH 0995/2163] bpo-39006: Fix asyncio when the ssl module is missing (GH-17524) Fix asyncio when the ssl module is missing: only check for ssl.SSLSocket instance if the ssl module is available. (cherry picked from commit 82b4950b5e92bec343a436b3f9c116400b66e1b9) Co-authored-by: Victor Stinner --- Lib/asyncio/selector_events.py | 20 +++++++++---------- .../2019-12-09-14-40-09.bpo-39006.v4VsPg.rst | 2 ++ 2 files changed, 12 insertions(+), 10 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-09-14-40-09.bpo-39006.v4VsPg.rst diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py index e1abf5118619cc..a05cbb6bdd69dc 100644 --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -40,6 +40,11 @@ def _test_selector_event(selector, fd, event): return bool(key.events & event) +def _check_ssl_socket(sock): + if ssl is not None and isinstance(sock, ssl.SSLSocket): + raise TypeError("Socket cannot be of type SSLSocket") + + class BaseSelectorEventLoop(base_events.BaseEventLoop): """Selector event loop. @@ -348,8 +353,7 @@ async def sock_recv(self, sock, n): The maximum amount of data to be received at once is specified by nbytes. """ - if isinstance(sock, ssl.SSLSocket): - raise TypeError("Socket cannot be of type SSLSocket") + _check_ssl_socket(sock) if self._debug and sock.gettimeout() != 0: raise ValueError("the socket must be non-blocking") try: @@ -388,8 +392,7 @@ async def sock_recv_into(self, sock, buf): The received data is written into *buf* (a writable buffer). The return value is the number of bytes written. """ - if isinstance(sock, ssl.SSLSocket): - raise TypeError("Socket cannot be of type SSLSocket") + _check_ssl_socket(sock) if self._debug and sock.gettimeout() != 0: raise ValueError("the socket must be non-blocking") try: @@ -429,8 +432,7 @@ async def sock_sendall(self, sock, data): raised, and there is no way to determine how much data, if any, was successfully processed by the receiving end of the connection. """ - if isinstance(sock, ssl.SSLSocket): - raise TypeError("Socket cannot be of type SSLSocket") + _check_ssl_socket(sock) if self._debug and sock.gettimeout() != 0: raise ValueError("the socket must be non-blocking") try: @@ -478,8 +480,7 @@ async def sock_connect(self, sock, address): This method is a coroutine. """ - if isinstance(sock, ssl.SSLSocket): - raise TypeError("Socket cannot be of type SSLSocket") + _check_ssl_socket(sock) if self._debug and sock.gettimeout() != 0: raise ValueError("the socket must be non-blocking") @@ -541,8 +542,7 @@ async def sock_accept(self, sock): object usable to send and receive data on the connection, and address is the address bound to the socket on the other end of the connection. """ - if isinstance(sock, ssl.SSLSocket): - raise TypeError("Socket cannot be of type SSLSocket") + _check_ssl_socket(sock) if self._debug and sock.gettimeout() != 0: raise ValueError("the socket must be non-blocking") fut = self.create_future() diff --git a/Misc/NEWS.d/next/Library/2019-12-09-14-40-09.bpo-39006.v4VsPg.rst b/Misc/NEWS.d/next/Library/2019-12-09-14-40-09.bpo-39006.v4VsPg.rst new file mode 100644 index 00000000000000..8402845a5a0475 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-09-14-40-09.bpo-39006.v4VsPg.rst @@ -0,0 +1,2 @@ +Fix asyncio when the ssl module is missing: only check for ssl.SSLSocket +instance if the ssl module is available. From 79c29742a8ba96fc933efe34e55e90537b3eb780 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 9 Dec 2019 06:39:54 -0800 Subject: [PATCH 0996/2163] bpo-37228: Fix loop.create_datagram_endpoint()'s usage of SO_REUSEADDR (GH-17311) (#17529) (cherry picked from commit ab513a38c98695f271e448fe2cb7c5e39eeaaaaf) Co-authored-by: Kyle Stanley --- Doc/library/asyncio-eventloop.rst | 24 ++++++++--- Lib/asyncio/base_events.py | 25 +++++++---- Lib/test/test_asyncio/test_base_events.py | 41 +++++++++++-------- .../2019-11-21-21-36-54.bpo-37228.yBZnFG.rst | 6 +++ 4 files changed, 67 insertions(+), 29 deletions(-) create mode 100644 Misc/NEWS.d/next/Security/2019-11-21-21-36-54.bpo-37228.yBZnFG.rst diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 1f4f0b6e75b9e7..ecd69fd669f1a5 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -461,6 +461,21 @@ Opening network connections reuse_address=None, reuse_port=None, \ allow_broadcast=None, sock=None) + .. note:: + The parameter *reuse_address* is no longer supported, as using + :py:data:`~sockets.SO_REUSEADDR` poses a significant security concern for + UDP. Explicitly passing ``reuse_address=True`` will raise an exception. + + When multiple processes with differing UIDs assign sockets to an + indentical UDP socket address with ``SO_REUSEADDR``, incoming packets can + become randomly distributed among the sockets. + + For supported platforms, *reuse_port* can be used as a replacement for + similar functionality. With *reuse_port*, + :py:data:`~sockets.SO_REUSEPORT` is used instead, which specifically + prevents processes with differing UIDs from assigning sockets to the same + socket address. + Create a datagram connection. The socket family can be either :py:data:`~socket.AF_INET`, @@ -489,11 +504,6 @@ Opening network connections resolution. If given, these should all be integers from the corresponding :mod:`socket` module constants. - * *reuse_address* tells the kernel to reuse a local socket in - ``TIME_WAIT`` state, without waiting for its natural timeout to - expire. If not specified will automatically be set to ``True`` on - Unix. - * *reuse_port* tells the kernel to allow this endpoint to be bound to the same port as other existing endpoints are bound to, so long as they all set this flag when being created. This option is not supported on Windows @@ -515,6 +525,10 @@ Opening network connections The *family*, *proto*, *flags*, *reuse_address*, *reuse_port, *allow_broadcast*, and *sock* parameters were added. + .. versionchanged:: 3.8.1 + The *reuse_address* parameter is no longer supported due to security + concerns. + .. versionchanged:: 3.8 Added support for Windows. diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py index 14b80bdda9c039..bfd40115bed38a 100644 --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -66,6 +66,10 @@ # Maximum timeout passed to select to avoid OS limitations MAXIMUM_SELECT_TIMEOUT = 24 * 3600 +# Used for deprecation and removal of `loop.create_datagram_endpoint()`'s +# *reuse_address* parameter +_unset = object() + def _format_handle(handle): cb = handle._callback @@ -1201,7 +1205,7 @@ async def start_tls(self, transport, protocol, sslcontext, *, async def create_datagram_endpoint(self, protocol_factory, local_addr=None, remote_addr=None, *, family=0, proto=0, flags=0, - reuse_address=None, reuse_port=None, + reuse_address=_unset, reuse_port=None, allow_broadcast=None, sock=None): """Create datagram connection.""" if sock is not None: @@ -1210,7 +1214,7 @@ async def create_datagram_endpoint(self, protocol_factory, f'A UDP Socket was expected, got {sock!r}') if (local_addr or remote_addr or family or proto or flags or - reuse_address or reuse_port or allow_broadcast): + reuse_port or allow_broadcast): # show the problematic kwargs in exception msg opts = dict(local_addr=local_addr, remote_addr=remote_addr, family=family, proto=proto, flags=flags, @@ -1277,8 +1281,18 @@ async def create_datagram_endpoint(self, protocol_factory, exceptions = [] - if reuse_address is None: - reuse_address = os.name == 'posix' and sys.platform != 'cygwin' + # bpo-37228 + if reuse_address is not _unset: + if reuse_address: + raise ValueError("Passing `reuse_address=True` is no " + "longer supported, as the usage of " + "SO_REUSEPORT in UDP poses a significant " + "security concern.") + else: + warnings.warn("The *reuse_address* parameter has been " + "deprecated as of 3.5.10 and is scheduled " + "for removal in 3.11.", DeprecationWarning, + stacklevel=2) for ((family, proto), (local_address, remote_address)) in addr_pairs_info: @@ -1287,9 +1301,6 @@ async def create_datagram_endpoint(self, protocol_factory, try: sock = socket.socket( family=family, type=socket.SOCK_DGRAM, proto=proto) - if reuse_address: - sock.setsockopt( - socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) if reuse_port: _set_reuseport(sock) if allow_broadcast: diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py index 92fdaf17e20110..3a23059911211f 100644 --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -1732,10 +1732,6 @@ class FakeSock: MyDatagramProto, flags=1, sock=FakeSock()) self.assertRaises(ValueError, self.loop.run_until_complete, fut) - fut = self.loop.create_datagram_endpoint( - MyDatagramProto, reuse_address=True, sock=FakeSock()) - self.assertRaises(ValueError, self.loop.run_until_complete, fut) - fut = self.loop.create_datagram_endpoint( MyDatagramProto, reuse_port=True, sock=FakeSock()) self.assertRaises(ValueError, self.loop.run_until_complete, fut) @@ -1746,7 +1742,6 @@ class FakeSock: def test_create_datagram_endpoint_sockopts(self): # Socket options should not be applied unless asked for. - # SO_REUSEADDR defaults to on for UNIX. # SO_REUSEPORT is not available on all platforms. coro = self.loop.create_datagram_endpoint( @@ -1755,18 +1750,8 @@ def test_create_datagram_endpoint_sockopts(self): transport, protocol = self.loop.run_until_complete(coro) sock = transport.get_extra_info('socket') - reuse_address_default_on = ( - os.name == 'posix' and sys.platform != 'cygwin') reuseport_supported = hasattr(socket, 'SO_REUSEPORT') - if reuse_address_default_on: - self.assertTrue( - sock.getsockopt( - socket.SOL_SOCKET, socket.SO_REUSEADDR)) - else: - self.assertFalse( - sock.getsockopt( - socket.SOL_SOCKET, socket.SO_REUSEADDR)) if reuseport_supported: self.assertFalse( sock.getsockopt( @@ -1782,13 +1767,12 @@ def test_create_datagram_endpoint_sockopts(self): coro = self.loop.create_datagram_endpoint( lambda: MyDatagramProto(create_future=True, loop=self.loop), local_addr=('127.0.0.1', 0), - reuse_address=True, reuse_port=reuseport_supported, allow_broadcast=True) transport, protocol = self.loop.run_until_complete(coro) sock = transport.get_extra_info('socket') - self.assertTrue( + self.assertFalse( sock.getsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR)) if reuseport_supported: @@ -1803,6 +1787,29 @@ def test_create_datagram_endpoint_sockopts(self): self.loop.run_until_complete(protocol.done) self.assertEqual('CLOSED', protocol.state) + def test_create_datagram_endpoint_reuse_address_error(self): + # bpo-37228: Ensure that explicit passing of `reuse_address=True` + # raises an error, as it is not safe to use SO_REUSEADDR when using UDP + + coro = self.loop.create_datagram_endpoint( + lambda: MyDatagramProto(create_future=True, loop=self.loop), + local_addr=('127.0.0.1', 0), + reuse_address=True) + + with self.assertRaises(ValueError): + self.loop.run_until_complete(coro) + + def test_create_datagram_endpoint_reuse_address_warning(self): + # bpo-37228: Deprecate *reuse_address* parameter + + coro = self.loop.create_datagram_endpoint( + lambda: MyDatagramProto(create_future=True, loop=self.loop), + local_addr=('127.0.0.1', 0), + reuse_address=False) + + with self.assertWarns(DeprecationWarning): + self.loop.run_until_complete(coro) + @patch_socket def test_create_datagram_endpoint_nosoreuseport(self, m_socket): del m_socket.SO_REUSEPORT diff --git a/Misc/NEWS.d/next/Security/2019-11-21-21-36-54.bpo-37228.yBZnFG.rst b/Misc/NEWS.d/next/Security/2019-11-21-21-36-54.bpo-37228.yBZnFG.rst new file mode 100644 index 00000000000000..0fafb63402e468 --- /dev/null +++ b/Misc/NEWS.d/next/Security/2019-11-21-21-36-54.bpo-37228.yBZnFG.rst @@ -0,0 +1,6 @@ +Due to significant security concerns, the *reuse_address* parameter of +:meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is +because of the behavior of ``SO_REUSEADDR`` in UDP. For more details, see the +documentation for ``loop.create_datagram_endpoint()``. +(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in +:issue:`37228`.) From 0d57db27f2c563b6433a220b646b50bdeff8a1f2 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 9 Dec 2019 08:07:51 -0800 Subject: [PATCH 0997/2163] bpo-34776: Fix dataclasses to support __future__ "annotations" mode (GH-9518) (#17531) (cherry picked from commit d219cc4180e7589807ebbef7421879f095e72a98) Co-authored-by: Yury Selivanov --- Lib/dataclasses.py | 87 +++++++++++-------- Lib/test/dataclass_textanno.py | 12 +++ Lib/test/test_dataclasses.py | 12 +++ .../2018-09-23-14-24-37.bpo-34776.1SrQe3.rst | 1 + 4 files changed, 78 insertions(+), 34 deletions(-) create mode 100644 Lib/test/dataclass_textanno.py create mode 100644 Misc/NEWS.d/next/Library/2018-09-23-14-24-37.bpo-34776.1SrQe3.rst diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py index 391f32e11d52a1..74f79294e81bd8 100644 --- a/Lib/dataclasses.py +++ b/Lib/dataclasses.py @@ -379,23 +379,24 @@ def _create_fn(name, args, body, *, globals=None, locals=None, # worries about external callers. if locals is None: locals = {} - # __builtins__ may be the "builtins" module or - # the value of its "__dict__", - # so make sure "__builtins__" is the module. - if globals is not None and '__builtins__' not in globals: - globals['__builtins__'] = builtins + if 'BUILTINS' not in locals: + locals['BUILTINS'] = builtins return_annotation = '' if return_type is not MISSING: locals['_return_type'] = return_type return_annotation = '->_return_type' args = ','.join(args) - body = '\n'.join(f' {b}' for b in body) + body = '\n'.join(f' {b}' for b in body) # Compute the text of the entire function. - txt = f'def {name}({args}){return_annotation}:\n{body}' + txt = f' def {name}({args}){return_annotation}:\n{body}' - exec(txt, globals, locals) - return locals[name] + local_vars = ', '.join(locals.keys()) + txt = f"def __create_fn__({local_vars}):\n{txt}\n return {name}" + + ns = {} + exec(txt, globals, ns) + return ns['__create_fn__'](**locals) def _field_assign(frozen, name, value, self_name): @@ -406,7 +407,7 @@ def _field_assign(frozen, name, value, self_name): # self_name is what "self" is called in this function: don't # hard-code "self", since that might be a field name. if frozen: - return f'__builtins__.object.__setattr__({self_name},{name!r},{value})' + return f'BUILTINS.object.__setattr__({self_name},{name!r},{value})' return f'{self_name}.{name}={value}' @@ -483,7 +484,7 @@ def _init_param(f): return f'{f.name}:_type_{f.name}{default}' -def _init_fn(fields, frozen, has_post_init, self_name): +def _init_fn(fields, frozen, has_post_init, self_name, globals): # fields contains both real fields and InitVar pseudo-fields. # Make sure we don't have fields without defaults following fields @@ -501,12 +502,15 @@ def _init_fn(fields, frozen, has_post_init, self_name): raise TypeError(f'non-default argument {f.name!r} ' 'follows default argument') - globals = {'MISSING': MISSING, - '_HAS_DEFAULT_FACTORY': _HAS_DEFAULT_FACTORY} + locals = {f'_type_{f.name}': f.type for f in fields} + locals.update({ + 'MISSING': MISSING, + '_HAS_DEFAULT_FACTORY': _HAS_DEFAULT_FACTORY, + }) body_lines = [] for f in fields: - line = _field_init(f, frozen, globals, self_name) + line = _field_init(f, frozen, locals, self_name) # line is None means that this field doesn't require # initialization (it's a pseudo-field). Just skip it. if line: @@ -522,7 +526,6 @@ def _init_fn(fields, frozen, has_post_init, self_name): if not body_lines: body_lines = ['pass'] - locals = {f'_type_{f.name}': f.type for f in fields} return _create_fn('__init__', [self_name] + [_init_param(f) for f in fields if f.init], body_lines, @@ -531,20 +534,19 @@ def _init_fn(fields, frozen, has_post_init, self_name): return_type=None) -def _repr_fn(fields): +def _repr_fn(fields, globals): fn = _create_fn('__repr__', ('self',), ['return self.__class__.__qualname__ + f"(' + ', '.join([f"{f.name}={{self.{f.name}!r}}" for f in fields]) + - ')"']) + ')"'], + globals=globals) return _recursive_repr(fn) -def _frozen_get_del_attr(cls, fields): - # XXX: globals is modified on the first call to _create_fn, then - # the modified version is used in the second call. Is this okay? - globals = {'cls': cls, +def _frozen_get_del_attr(cls, fields, globals): + locals = {'cls': cls, 'FrozenInstanceError': FrozenInstanceError} if fields: fields_str = '(' + ','.join(repr(f.name) for f in fields) + ',)' @@ -556,17 +558,19 @@ def _frozen_get_del_attr(cls, fields): (f'if type(self) is cls or name in {fields_str}:', ' raise FrozenInstanceError(f"cannot assign to field {name!r}")', f'super(cls, self).__setattr__(name, value)'), + locals=locals, globals=globals), _create_fn('__delattr__', ('self', 'name'), (f'if type(self) is cls or name in {fields_str}:', ' raise FrozenInstanceError(f"cannot delete field {name!r}")', f'super(cls, self).__delattr__(name)'), + locals=locals, globals=globals), ) -def _cmp_fn(name, op, self_tuple, other_tuple): +def _cmp_fn(name, op, self_tuple, other_tuple, globals): # Create a comparison function. If the fields in the object are # named 'x' and 'y', then self_tuple is the string # '(self.x,self.y)' and other_tuple is the string @@ -576,14 +580,16 @@ def _cmp_fn(name, op, self_tuple, other_tuple): ('self', 'other'), [ 'if other.__class__ is self.__class__:', f' return {self_tuple}{op}{other_tuple}', - 'return NotImplemented']) + 'return NotImplemented'], + globals=globals) -def _hash_fn(fields): +def _hash_fn(fields, globals): self_tuple = _tuple_str('self', fields) return _create_fn('__hash__', ('self',), - [f'return hash({self_tuple})']) + [f'return hash({self_tuple})'], + globals=globals) def _is_classvar(a_type, typing): @@ -756,14 +762,14 @@ def _set_new_attribute(cls, name, value): # take. The common case is to do nothing, so instead of providing a # function that is a no-op, use None to signify that. -def _hash_set_none(cls, fields): +def _hash_set_none(cls, fields, globals): return None -def _hash_add(cls, fields): +def _hash_add(cls, fields, globals): flds = [f for f in fields if (f.compare if f.hash is None else f.hash)] - return _hash_fn(flds) + return _hash_fn(flds, globals) -def _hash_exception(cls, fields): +def _hash_exception(cls, fields, globals): # Raise an exception. raise TypeError(f'Cannot overwrite attribute __hash__ ' f'in class {cls.__name__}') @@ -805,6 +811,16 @@ def _process_class(cls, init, repr, eq, order, unsafe_hash, frozen): # is defined by the base class, which is found first. fields = {} + if cls.__module__ in sys.modules: + globals = sys.modules[cls.__module__].__dict__ + else: + # Theoretically this can happen if someone writes + # a custom string to cls.__module__. In which case + # such dataclass won't be fully introspectable + # (w.r.t. typing.get_type_hints) but will still function + # correctly. + globals = {} + setattr(cls, _PARAMS, _DataclassParams(init, repr, eq, order, unsafe_hash, frozen)) @@ -914,6 +930,7 @@ def _process_class(cls, init, repr, eq, order, unsafe_hash, frozen): # if possible. '__dataclass_self__' if 'self' in fields else 'self', + globals, )) # Get the fields as a list, and include only real fields. This is @@ -922,7 +939,7 @@ def _process_class(cls, init, repr, eq, order, unsafe_hash, frozen): if repr: flds = [f for f in field_list if f.repr] - _set_new_attribute(cls, '__repr__', _repr_fn(flds)) + _set_new_attribute(cls, '__repr__', _repr_fn(flds, globals)) if eq: # Create _eq__ method. There's no need for a __ne__ method, @@ -932,7 +949,8 @@ def _process_class(cls, init, repr, eq, order, unsafe_hash, frozen): other_tuple = _tuple_str('other', flds) _set_new_attribute(cls, '__eq__', _cmp_fn('__eq__', '==', - self_tuple, other_tuple)) + self_tuple, other_tuple, + globals=globals)) if order: # Create and set the ordering methods. @@ -945,13 +963,14 @@ def _process_class(cls, init, repr, eq, order, unsafe_hash, frozen): ('__ge__', '>='), ]: if _set_new_attribute(cls, name, - _cmp_fn(name, op, self_tuple, other_tuple)): + _cmp_fn(name, op, self_tuple, other_tuple, + globals=globals)): raise TypeError(f'Cannot overwrite attribute {name} ' f'in class {cls.__name__}. Consider using ' 'functools.total_ordering') if frozen: - for fn in _frozen_get_del_attr(cls, field_list): + for fn in _frozen_get_del_attr(cls, field_list, globals): if _set_new_attribute(cls, fn.__name__, fn): raise TypeError(f'Cannot overwrite attribute {fn.__name__} ' f'in class {cls.__name__}') @@ -964,7 +983,7 @@ def _process_class(cls, init, repr, eq, order, unsafe_hash, frozen): if hash_action: # No need to call _set_new_attribute here, since by the time # we're here the overwriting is unconditional. - cls.__hash__ = hash_action(cls, field_list) + cls.__hash__ = hash_action(cls, field_list, globals) if not getattr(cls, '__doc__'): # Create a class doc-string. diff --git a/Lib/test/dataclass_textanno.py b/Lib/test/dataclass_textanno.py new file mode 100644 index 00000000000000..3eb6c943d4c434 --- /dev/null +++ b/Lib/test/dataclass_textanno.py @@ -0,0 +1,12 @@ +from __future__ import annotations + +import dataclasses + + +class Foo: + pass + + +@dataclasses.dataclass +class Bar: + foo: Foo diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py index b018133cf23025..2f3c531fc208ad 100755 --- a/Lib/test/test_dataclasses.py +++ b/Lib/test/test_dataclasses.py @@ -10,6 +10,7 @@ import unittest from unittest.mock import Mock from typing import ClassVar, Any, List, Union, Tuple, Dict, Generic, TypeVar, Optional +from typing import get_type_hints from collections import deque, OrderedDict, namedtuple from functools import total_ordering @@ -2926,6 +2927,17 @@ def test_classvar_module_level_import(self): # won't exist on the instance. self.assertNotIn('not_iv4', c.__dict__) + def test_text_annotations(self): + from test import dataclass_textanno + + self.assertEqual( + get_type_hints(dataclass_textanno.Bar), + {'foo': dataclass_textanno.Foo}) + self.assertEqual( + get_type_hints(dataclass_textanno.Bar.__init__), + {'foo': dataclass_textanno.Foo, + 'return': type(None)}) + class TestMakeDataclass(unittest.TestCase): def test_simple(self): diff --git a/Misc/NEWS.d/next/Library/2018-09-23-14-24-37.bpo-34776.1SrQe3.rst b/Misc/NEWS.d/next/Library/2018-09-23-14-24-37.bpo-34776.1SrQe3.rst new file mode 100644 index 00000000000000..815a4876e0b4aa --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-23-14-24-37.bpo-34776.1SrQe3.rst @@ -0,0 +1 @@ +Fix dataclasses to support forward references in type annotations From c5a2a974d3608810dc0b06b46876542e1216f7e6 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 9 Dec 2019 09:02:22 -0800 Subject: [PATCH 0998/2163] Fix APPX registry key generation (GH-17489) (cherry picked from commit e89e159b18cc9f32a0a4a818d080eb6a63d888a7) Co-authored-by: Steve Dower --- PC/layout/support/appxmanifest.py | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/PC/layout/support/appxmanifest.py b/PC/layout/support/appxmanifest.py index 9e008f793cf550..9a7439d027126d 100644 --- a/PC/layout/support/appxmanifest.py +++ b/PC/layout/support/appxmanifest.py @@ -339,7 +339,17 @@ def _get_registry_entries(ns, root="", d=None): for key, value in d.items(): if key == "_condition": continue - elif isinstance(value, dict): + if value is SPECIAL_LOOKUP: + if key == "SysArchitecture": + value = { + "win32": "32bit", + "amd64": "64bit", + "arm32": "32bit", + "arm64": "64bit", + }[ns.arch] + else: + raise ValueError(f"Key '{key}' unhandled for special lookup") + if isinstance(value, dict): cond = value.get("_condition") if cond and not cond(ns): continue @@ -349,16 +359,6 @@ def _get_registry_entries(ns, root="", d=None): if len(fullkey.parts) > 1: yield str(fullkey), None, None yield from _get_registry_entries(ns, fullkey, value) - elif value is SPECIAL_LOOKUP: - if key == "SysArchitecture": - return { - "win32": "32bit", - "amd64": "64bit", - "arm32": "32bit", - "arm64": "64bit", - }[ns.arch] - else: - raise ValueError(f"Key '{key}' unhandled for special lookup") elif len(r.parts) > 1: yield str(r), key, value From 3c5feaffde1944052830c896ae39c54e76a2e063 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 9 Dec 2019 09:32:34 -0800 Subject: [PATCH 0999/2163] bpo-38992: avoid fsum test failure from constant-folding (GH-17513) (GH-17530) * Issue 38992: avoid fsum test failure * Add NEWS entry (cherry picked from commit bba873e633f0f1e88ea12fb935cbd58faa77f976) Co-authored-by: Mark Dickinson --- Lib/test/test_math.py | 8 +++++++- .../next/Tests/2019-12-08-15-11-06.bpo-38992.cVoHOZ.rst | 1 + 2 files changed, 8 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Tests/2019-12-08-15-11-06.bpo-38992.cVoHOZ.rst diff --git a/Lib/test/test_math.py b/Lib/test/test_math.py index 6edc3e1a547a49..ac978247fc917f 100644 --- a/Lib/test/test_math.py +++ b/Lib/test/test_math.py @@ -654,7 +654,6 @@ def msum(iterable): float.fromhex('0x1.df11f45f4e61ap+2')), ([(-1.)**n/n for n in range(1, 1001)], float.fromhex('-0x1.62a2af1bd3624p-1')), - ([1.7**(i+1)-1.7**i for i in range(1000)] + [-1.7**1000], -1.0), ([1e16, 1., 1e-16], 10000000000000002.0), ([1e16-2., 1.-2.**-53, -(1e16-2.), -(1.-2.**-53)], 0.0), # exercise code for resizing partials array @@ -663,6 +662,13 @@ def msum(iterable): float.fromhex('0x1.5555555555555p+970')), ] + # Telescoping sum, with exact differences (due to Sterbenz) + terms = [1.7**i for i in range(1001)] + test_values.append(( + [terms[i+1] - terms[i] for i in range(1000)] + [-terms[1000]], + -terms[0] + )) + for i, (vals, expected) in enumerate(test_values): try: actual = math.fsum(vals) diff --git a/Misc/NEWS.d/next/Tests/2019-12-08-15-11-06.bpo-38992.cVoHOZ.rst b/Misc/NEWS.d/next/Tests/2019-12-08-15-11-06.bpo-38992.cVoHOZ.rst new file mode 100644 index 00000000000000..815ae0f65c8732 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2019-12-08-15-11-06.bpo-38992.cVoHOZ.rst @@ -0,0 +1 @@ +Fix a test for :func:`math.fsum` that was failing due to constant folding. From b00a2b5b76afded9fccaad61cfca1cc0baafdbdc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Mon, 9 Dec 2019 18:47:55 +0100 Subject: [PATCH 1000/2163] Python 3.8.1rc1 --- Include/patchlevel.h | 8 +- Lib/pydoc_data/topics.py | 229 ++++-- Misc/NEWS.d/3.8.1rc1.rst | 778 ++++++++++++++++++ .../2019-10-11-15-32-58.bpo-37415.D9RXrq.rst | 2 - .../2019-11-04-14-30-37.bpo-38684.aed593.rst | 1 - .../2019-11-15-09-25-44.bpo-38809.9jwta6.rst | 2 - .../2019-12-01-21-45-24.bpo-37404.cNsA7S.rst | 2 - .../2019-10-07-17-15-09.bpo-36389.hFX_jD.rst | 3 - .../2019-10-21-09-24-03.bpo-38540.314N_T.rst | 3 - .../2019-11-04-21-10-47.bpo-37633.oOGVdo.rst | 1 - .../2019-07-13-18-01-13.bpo-35409.ozbcsR.rst | 2 - .../2019-10-20-00-36-18.bpo-38525.Vty1cA.rst | 2 - .../2019-10-20-12-43-48.bpo-38535.ESMkVN.rst | 2 - .../2019-10-30-11-25-25.bpo-38640.4sAFh5.rst | 3 - .../2019-11-08-00-36-10.bpo-38707.SZL036.rst | 1 - .../2019-11-22-22-18-50.bpo-38892.LS586s.rst | 1 - .../2019-11-26-09-16-47.bpo-38920.Vx__sT.rst | 2 - .../2019-11-26-12-20-34.bpo-38922.i6ja-i.rst | 2 - .../2019-12-01-00-17-44.bpo-38673.K_Tze-.rst | 1 - .../2019-10-26-13-19-07.bpo-38592.Y96BYO.rst | 1 - .../2019-11-12-15-31-09.bpo-38778.PHhTlv.rst | 1 - .../2019-11-15-09-22-28.bpo-38351.xwhlse.rst | 1 - .../2019-11-15-11-39-13.bpo-38816.vUaSVL.rst | 3 - .../2019-10-26-18-16-24.bpo-38598.6kH9FY.rst | 1 - .../2019-10-28-04-48-03.bpo-4630.upgjiV.rst | 3 - .../2019-10-30-22-11-16.bpo-38636.hUhDeB.rst | 3 - .../2019-11-09-23-55-59.bpo-26353.duYZiF.rst | 2 - .../2019-11-23-21-50-57.bpo-38862.KQ9A0m.rst | 2 - .../2017-12-26-14-32-23.bpo-27657.6BhyVK.rst | 2 - .../2018-04-24-13-18-48.bpo-33348.XaJDei.rst | 2 - .../2018-09-23-14-24-37.bpo-34776.1SrQe3.rst | 1 - .../2019-05-06-15-34-17.bpo-36820.Eh5mIB.rst | 3 - .../2019-07-09-05-44-39.bpo-36993.4javqu.rst | 2 - .../2019-10-09-18-16-51.bpo-38422.aiM5bq.rst | 1 - .../2019-10-15-09-47-40.bpo-33604.J12cWT.rst | 3 - .../2019-10-15-11-37-57.bpo-38478.A87OPO.rst | 3 - .../2019-10-18-13-57-31.bpo-38521.U-7aaM.rst | 1 - .../2019-10-20-12-04-48.bpo-31202.NfdIus.rst | 2 - .../2019-10-23-16-25-12.bpo-34679.Bnw8o3.rst | 2 - .../2019-10-27-00-08-49.bpo-38334.pfLLmc.rst | 1 - .../2019-11-06-15-26-15.bpo-38686.HNFBce.rst | 1 - .../2019-11-11-21-43-06.bpo-27805.D3zl1_.rst | 2 - .../2019-11-12-15-46-28.bpo-38723.gcdMFn.rst | 1 - .../2019-11-13-16-17-43.bpo-38785.NEOEfk.rst | 2 - .../2019-11-15-09-30-29.bpo-38807.PsmRog.rst | 1 - .../2019-11-16-16-09-07.bpo-38820.ivhUSV.rst | 2 - .../2019-11-16-23-26-25.bpo-38821.-albNN.rst | 1 - .../2019-11-19-16-28-25.bpo-38857.YPUkU9.rst | 4 - .../2019-11-19-16-30-46.bpo-38859.AZUzL8.rst | 3 - .../2019-11-21-11-39-17.bpo-37838.lRFcEC.rst | 1 - .../2019-11-22-10-45-03.bpo-38668.iKx23z.rst | 5 - .../2019-11-27-16-30-02.bpo-26730.56cdBn.rst | 2 - .../2019-12-02-10-35-19.bpo-38698.WZnAPQ.rst | 5 - .../2019-12-04-15-28-40.bpo-33684.QeSmQP.rst | 2 - .../2019-12-04-15-56-28.bpo-38634.pq0ZWa.rst | 2 - .../2019-12-05-16-13-25.bpo-38529.yvQgx3.rst | 2 - .../2019-12-06-15-11-42.bpo-38986.bg6iZt.rst | 2 - .../2019-12-07-16-32-42.bpo-38979.q0sIHy.rst | 1 - .../2019-12-07-21-49-50.bpo-38698.HxoSym.rst | 3 - .../2019-12-07-22-25-39.bpo-38708.rZTUfk.rst | 1 - .../2019-12-09-14-40-09.bpo-39006.v4VsPg.rst | 2 - .../2019-10-08-19-29-55.bpo-38418.QL7s0-.rst | 1 - .../2019-11-14-16-13-23.bpo-38622.3DYkfb.rst | 1 - .../2019-11-15-00-54-42.bpo-38804.vjbM8V.rst | 1 - .../2019-11-18-16-17-56.bpo-38722.x3mECW.rst | 2 - .../2019-11-21-21-36-54.bpo-37228.yBZnFG.rst | 6 - .../2019-12-01-22-44-40.bpo-38945.ztmNXc.rst | 1 - .../2019-10-16-01-36-15.bpo-35998.G305Bf.rst | 5 - .../2019-11-04-02-54-16.bpo-38669.pazXZ8.rst | 1 - .../2019-11-20-16-08-19.bpo-38841.5F5Lbw.rst | 2 - .../2019-11-21-09-11-06.bpo-38875.wSZJal.rst | 1 - .../2019-12-04-17-08-55.bpo-38965.yqax3m.rst | 3 - .../2019-12-08-15-11-06.bpo-38992.cVoHOZ.rst | 1 - .../2019-12-09-11-32-34.bpo-38547.Juw54e.rst | 3 - .../2019-10-16-09-49-09.bpo-38492.Te1LxC.rst | 1 - .../2019-10-28-05-01-29.bpo-38519.dCkY66.rst | 2 - .../2019-10-28-10-32-43.bpo-38453.NwwatW.rst | 1 - .../2019-10-28-10-48-16.bpo-38589.V69Q1a.rst | 1 - .../2019-11-14-08-57-50.bpo-33125.EN5MWS.rst | 1 - .../2019-08-23-12-14-34.bpo-37931.goYgQj.rst | 3 - README.rst | 4 +- 81 files changed, 931 insertions(+), 240 deletions(-) create mode 100644 Misc/NEWS.d/3.8.1rc1.rst delete mode 100644 Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst delete mode 100644 Misc/NEWS.d/next/Build/2019-11-04-14-30-37.bpo-38684.aed593.rst delete mode 100644 Misc/NEWS.d/next/Build/2019-11-15-09-25-44.bpo-38809.9jwta6.rst delete mode 100644 Misc/NEWS.d/next/Build/2019-12-01-21-45-24.bpo-37404.cNsA7S.rst delete mode 100644 Misc/NEWS.d/next/C API/2019-10-07-17-15-09.bpo-36389.hFX_jD.rst delete mode 100644 Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst delete mode 100644 Misc/NEWS.d/next/C API/2019-11-04-21-10-47.bpo-37633.oOGVdo.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-07-13-18-01-13.bpo-35409.ozbcsR.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-10-20-00-36-18.bpo-38525.Vty1cA.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-10-30-11-25-25.bpo-38640.4sAFh5.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-11-08-00-36-10.bpo-38707.SZL036.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-11-22-22-18-50.bpo-38892.LS586s.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-11-26-09-16-47.bpo-38920.Vx__sT.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-11-26-12-20-34.bpo-38922.i6ja-i.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-12-01-00-17-44.bpo-38673.K_Tze-.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2019-11-12-15-31-09.bpo-38778.PHhTlv.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2019-11-15-09-22-28.bpo-38351.xwhlse.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2019-11-15-11-39-13.bpo-38816.vUaSVL.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2019-10-28-04-48-03.bpo-4630.upgjiV.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2019-10-30-22-11-16.bpo-38636.hUhDeB.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2019-11-09-23-55-59.bpo-26353.duYZiF.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2019-11-23-21-50-57.bpo-38862.KQ9A0m.rst delete mode 100644 Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst delete mode 100644 Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst delete mode 100644 Misc/NEWS.d/next/Library/2018-09-23-14-24-37.bpo-34776.1SrQe3.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-05-06-15-34-17.bpo-36820.Eh5mIB.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-10-09-18-16-51.bpo-38422.aiM5bq.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-10-15-09-47-40.bpo-33604.J12cWT.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-10-15-11-37-57.bpo-38478.A87OPO.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-10-18-13-57-31.bpo-38521.U-7aaM.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-10-23-16-25-12.bpo-34679.Bnw8o3.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-06-15-26-15.bpo-38686.HNFBce.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-11-21-43-06.bpo-27805.D3zl1_.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-12-15-46-28.bpo-38723.gcdMFn.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-13-16-17-43.bpo-38785.NEOEfk.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-15-09-30-29.bpo-38807.PsmRog.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-16-16-09-07.bpo-38820.ivhUSV.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-16-23-26-25.bpo-38821.-albNN.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-19-16-28-25.bpo-38857.YPUkU9.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-19-16-30-46.bpo-38859.AZUzL8.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-21-11-39-17.bpo-37838.lRFcEC.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-22-10-45-03.bpo-38668.iKx23z.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-27-16-30-02.bpo-26730.56cdBn.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-02-10-35-19.bpo-38698.WZnAPQ.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-04-15-28-40.bpo-33684.QeSmQP.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-04-15-56-28.bpo-38634.pq0ZWa.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-05-16-13-25.bpo-38529.yvQgx3.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-06-15-11-42.bpo-38986.bg6iZt.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-07-16-32-42.bpo-38979.q0sIHy.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-07-21-49-50.bpo-38698.HxoSym.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-07-22-25-39.bpo-38708.rZTUfk.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-09-14-40-09.bpo-39006.v4VsPg.rst delete mode 100644 Misc/NEWS.d/next/Security/2019-10-08-19-29-55.bpo-38418.QL7s0-.rst delete mode 100644 Misc/NEWS.d/next/Security/2019-11-14-16-13-23.bpo-38622.3DYkfb.rst delete mode 100644 Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst delete mode 100644 Misc/NEWS.d/next/Security/2019-11-18-16-17-56.bpo-38722.x3mECW.rst delete mode 100644 Misc/NEWS.d/next/Security/2019-11-21-21-36-54.bpo-37228.yBZnFG.rst delete mode 100644 Misc/NEWS.d/next/Security/2019-12-01-22-44-40.bpo-38945.ztmNXc.rst delete mode 100644 Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst delete mode 100644 Misc/NEWS.d/next/Tests/2019-11-04-02-54-16.bpo-38669.pazXZ8.rst delete mode 100644 Misc/NEWS.d/next/Tests/2019-11-20-16-08-19.bpo-38841.5F5Lbw.rst delete mode 100644 Misc/NEWS.d/next/Tests/2019-11-21-09-11-06.bpo-38875.wSZJal.rst delete mode 100644 Misc/NEWS.d/next/Tests/2019-12-04-17-08-55.bpo-38965.yqax3m.rst delete mode 100644 Misc/NEWS.d/next/Tests/2019-12-08-15-11-06.bpo-38992.cVoHOZ.rst delete mode 100644 Misc/NEWS.d/next/Tests/2019-12-09-11-32-34.bpo-38547.Juw54e.rst delete mode 100644 Misc/NEWS.d/next/Windows/2019-10-16-09-49-09.bpo-38492.Te1LxC.rst delete mode 100644 Misc/NEWS.d/next/Windows/2019-10-28-05-01-29.bpo-38519.dCkY66.rst delete mode 100644 Misc/NEWS.d/next/Windows/2019-10-28-10-32-43.bpo-38453.NwwatW.rst delete mode 100644 Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst delete mode 100644 Misc/NEWS.d/next/Windows/2019-11-14-08-57-50.bpo-33125.EN5MWS.rst delete mode 100644 Misc/NEWS.d/next/macOS/2019-08-23-12-14-34.bpo-37931.goYgQj.rst diff --git a/Include/patchlevel.h b/Include/patchlevel.h index df8bd3448b1517..b3a2ecb9e7526c 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -18,12 +18,12 @@ /*--start constants--*/ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 8 -#define PY_MICRO_VERSION 0 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL -#define PY_RELEASE_SERIAL 0 +#define PY_MICRO_VERSION 1 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA +#define PY_RELEASE_SERIAL 1 /* Version as a string */ -#define PY_VERSION "3.8.0+" +#define PY_VERSION "3.8.1rc1" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index 327a421bbb6bc9..a11be7ab0f836a 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Mon Oct 14 14:33:10 2019 +# Autogenerated by Sphinx on Mon Dec 9 18:44:17 2019 topics = {'assert': 'The "assert" statement\n' '**********************\n' '\n' @@ -744,10 +744,11 @@ 'returned.\n' '\n' 'The "__dir__" function should accept no arguments, and ' - 'return a list\n' - 'of strings that represents the names accessible on ' - 'module. If present,\n' - 'this function overrides the standard "dir()" search on a ' + 'return a\n' + 'sequence of strings that represents the names accessible ' + 'on module. If\n' + 'present, this function overrides the standard "dir()" ' + 'search on a\n' 'module.\n' '\n' 'For a more fine grained customization of the module ' @@ -861,6 +862,22 @@ 'created. The\n' ' descriptor has been assigned to *name*.\n' '\n' + ' Note: "__set_name__()" is only called implicitly as ' + 'part of the\n' + ' "type" constructor, so it will need to be called ' + 'explicitly with\n' + ' the appropriate parameters when a descriptor is ' + 'added to a class\n' + ' after initial creation:\n' + '\n' + ' class A:\n' + ' pass\n' + ' descr = custom_descriptor()\n' + ' A.attr = descr\n' + " descr.__set_name__(A, 'attr')\n" + '\n' + ' See Creating the class object for more details.\n' + '\n' ' New in version 3.6.\n' '\n' 'The attribute "__objclass__" is interpreted by the ' @@ -1089,7 +1106,13 @@ 'attributes created by\n' ' slots (the other bases must have empty slot layouts) - ' 'violations\n' - ' raise "TypeError".\n', + ' raise "TypeError".\n' + '\n' + '* If an iterator is used for *__slots__* then a ' + 'descriptor is\n' + ' created for each of the iterator’s values. However, ' + 'the *__slots__*\n' + ' attribute will be an empty iterator.\n', 'attribute-references': 'Attribute references\n' '********************\n' '\n' @@ -4216,6 +4239,17 @@ ' Quit from the debugger. The program being executed is ' 'aborted.\n' '\n' + 'debug code\n' + '\n' + ' Enter a recursive debugger that steps through the code ' + 'argument\n' + ' (which is an arbitrary expression or statement to be executed ' + 'in\n' + ' the current environment).\n' + '\n' + 'retval\n' + 'Print the return value for the last return of a function.\n' + '\n' '-[ Footnotes ]-\n' '\n' '[1] Whether a frame is considered to originate in a certain ' @@ -6227,6 +6261,10 @@ 'that\n' 'determine dynamically the modules to be loaded.\n' '\n' + 'Raises an auditing event "import" with arguments "module", ' + '"filename",\n' + '"sys.path", "sys.meta_path", "sys.path_hooks".\n' + '\n' '\n' 'Future statements\n' '=================\n' @@ -7450,9 +7488,9 @@ 'to allow\n' 'efficient iteration through the container; for mappings, ' '"__iter__()"\n' - 'should be the same as "keys()"; for sequences, it should ' - 'iterate\n' - 'through the values.\n' + 'should iterate through the object’s keys; for sequences, ' + 'it should\n' + 'iterate through the values.\n' '\n' 'object.__len__(self)\n' '\n' @@ -7604,12 +7642,12 @@ '\n' 'The membership test operators ("in" and "not in") are ' 'normally\n' - 'implemented as an iteration through a sequence. However, ' + 'implemented as an iteration through a container. However, ' 'container\n' 'objects can supply the following special method with a ' 'more efficient\n' 'implementation, which also does not require the object be ' - 'a sequence.\n' + 'iterable.\n' '\n' 'object.__contains__(self, item)\n' '\n' @@ -8391,10 +8429,11 @@ 'returned.\n' '\n' 'The "__dir__" function should accept no arguments, and ' - 'return a list\n' - 'of strings that represents the names accessible on module. ' - 'If present,\n' - 'this function overrides the standard "dir()" search on a ' + 'return a\n' + 'sequence of strings that represents the names accessible on ' + 'module. If\n' + 'present, this function overrides the standard "dir()" search ' + 'on a\n' 'module.\n' '\n' 'For a more fine grained customization of the module behavior ' @@ -8508,6 +8547,22 @@ 'The\n' ' descriptor has been assigned to *name*.\n' '\n' + ' Note: "__set_name__()" is only called implicitly as part ' + 'of the\n' + ' "type" constructor, so it will need to be called ' + 'explicitly with\n' + ' the appropriate parameters when a descriptor is added ' + 'to a class\n' + ' after initial creation:\n' + '\n' + ' class A:\n' + ' pass\n' + ' descr = custom_descriptor()\n' + ' A.attr = descr\n' + " descr.__set_name__(A, 'attr')\n" + '\n' + ' See Creating the class object for more details.\n' + '\n' ' New in version 3.6.\n' '\n' 'The attribute "__objclass__" is interpreted by the "inspect" ' @@ -8734,6 +8789,12 @@ 'violations\n' ' raise "TypeError".\n' '\n' + '* If an iterator is used for *__slots__* then a descriptor ' + 'is\n' + ' created for each of the iterator’s values. However, the ' + '*__slots__*\n' + ' attribute will be an empty iterator.\n' + '\n' '\n' 'Customizing class creation\n' '==========================\n' @@ -9179,9 +9240,9 @@ 'allow\n' 'efficient iteration through the container; for mappings, ' '"__iter__()"\n' - 'should be the same as "keys()"; for sequences, it should ' - 'iterate\n' - 'through the values.\n' + 'should iterate through the object’s keys; for sequences, it ' + 'should\n' + 'iterate through the values.\n' '\n' 'object.__len__(self)\n' '\n' @@ -9332,12 +9393,12 @@ '\n' 'The membership test operators ("in" and "not in") are ' 'normally\n' - 'implemented as an iteration through a sequence. However, ' + 'implemented as an iteration through a container. However, ' 'container\n' 'objects can supply the following special method with a more ' 'efficient\n' - 'implementation, which also does not require the object be a ' - 'sequence.\n' + 'implementation, which also does not require the object be ' + 'iterable.\n' '\n' 'object.__contains__(self, item)\n' '\n' @@ -9941,20 +10002,20 @@ '\n' 'str.isalnum()\n' '\n' - ' Return true if all characters in the string are ' + ' Return "True" if all characters in the string are ' 'alphanumeric and\n' - ' there is at least one character, false otherwise. A ' - 'character "c"\n' - ' is alphanumeric if one of the following returns ' + ' there is at least one character, "False" otherwise. A ' + 'character\n' + ' "c" is alphanumeric if one of the following returns ' '"True":\n' ' "c.isalpha()", "c.isdecimal()", "c.isdigit()", or ' '"c.isnumeric()".\n' '\n' 'str.isalpha()\n' '\n' - ' Return true if all characters in the string are ' + ' Return "True" if all characters in the string are ' 'alphabetic and\n' - ' there is at least one character, false otherwise. ' + ' there is at least one character, "False" otherwise. ' 'Alphabetic\n' ' characters are those characters defined in the Unicode ' 'character\n' @@ -9968,45 +10029,46 @@ '\n' 'str.isascii()\n' '\n' - ' Return true if the string is empty or all characters in ' - 'the string\n' - ' are ASCII, false otherwise. ASCII characters have code ' - 'points in\n' - ' the range U+0000-U+007F.\n' + ' Return "True" if the string is empty or all characters ' + 'in the\n' + ' string are ASCII, "False" otherwise. ASCII characters ' + 'have code\n' + ' points in the range U+0000-U+007F.\n' '\n' ' New in version 3.7.\n' '\n' 'str.isdecimal()\n' '\n' - ' Return true if all characters in the string are decimal ' - 'characters\n' - ' and there is at least one character, false otherwise. ' - 'Decimal\n' - ' characters are those that can be used to form numbers ' - 'in base 10,\n' - ' e.g. U+0660, ARABIC-INDIC DIGIT ZERO. Formally a ' - 'decimal character\n' - ' is a character in the Unicode General Category “Nd”.\n' + ' Return "True" if all characters in the string are ' + 'decimal\n' + ' characters and there is at least one character, "False" ' + 'otherwise.\n' + ' Decimal characters are those that can be used to form ' + 'numbers in\n' + ' base 10, e.g. U+0660, ARABIC-INDIC DIGIT ZERO. ' + 'Formally a decimal\n' + ' character is a character in the Unicode General ' + 'Category “Nd”.\n' '\n' 'str.isdigit()\n' '\n' - ' Return true if all characters in the string are digits ' - 'and there is\n' - ' at least one character, false otherwise. Digits ' - 'include decimal\n' - ' characters and digits that need special handling, such ' - 'as the\n' - ' compatibility superscript digits. This covers digits ' - 'which cannot\n' - ' be used to form numbers in base 10, like the Kharosthi ' - 'numbers.\n' - ' Formally, a digit is a character that has the property ' - 'value\n' - ' Numeric_Type=Digit or Numeric_Type=Decimal.\n' + ' Return "True" if all characters in the string are ' + 'digits and there\n' + ' is at least one character, "False" otherwise. Digits ' + 'include\n' + ' decimal characters and digits that need special ' + 'handling, such as\n' + ' the compatibility superscript digits. This covers ' + 'digits which\n' + ' cannot be used to form numbers in base 10, like the ' + 'Kharosthi\n' + ' numbers. Formally, a digit is a character that has the ' + 'property\n' + ' value Numeric_Type=Digit or Numeric_Type=Decimal.\n' '\n' 'str.isidentifier()\n' '\n' - ' Return true if the string is a valid identifier ' + ' Return "True" if the string is a valid identifier ' 'according to the\n' ' language definition, section Identifiers and keywords.\n' '\n' @@ -10025,32 +10087,33 @@ '\n' 'str.islower()\n' '\n' - ' Return true if all cased characters [4] in the string ' - 'are lowercase\n' - ' and there is at least one cased character, false ' - 'otherwise.\n' + ' Return "True" if all cased characters [4] in the string ' + 'are\n' + ' lowercase and there is at least one cased character, ' + '"False"\n' + ' otherwise.\n' '\n' 'str.isnumeric()\n' '\n' - ' Return true if all characters in the string are numeric ' - 'characters,\n' - ' and there is at least one character, false otherwise. ' - 'Numeric\n' - ' characters include digit characters, and all characters ' - 'that have\n' - ' the Unicode numeric value property, e.g. U+2155, VULGAR ' - 'FRACTION\n' - ' ONE FIFTH. Formally, numeric characters are those with ' - 'the\n' - ' property value Numeric_Type=Digit, Numeric_Type=Decimal ' - 'or\n' + ' Return "True" if all characters in the string are ' + 'numeric\n' + ' characters, and there is at least one character, ' + '"False" otherwise.\n' + ' Numeric characters include digit characters, and all ' + 'characters\n' + ' that have the Unicode numeric value property, e.g. ' + 'U+2155, VULGAR\n' + ' FRACTION ONE FIFTH. Formally, numeric characters are ' + 'those with\n' + ' the property value Numeric_Type=Digit, ' + 'Numeric_Type=Decimal or\n' ' Numeric_Type=Numeric.\n' '\n' 'str.isprintable()\n' '\n' - ' Return true if all characters in the string are ' + ' Return "True" if all characters in the string are ' 'printable or the\n' - ' string is empty, false otherwise. Nonprintable ' + ' string is empty, "False" otherwise. Nonprintable ' 'characters are\n' ' those characters defined in the Unicode character ' 'database as\n' @@ -10066,9 +10129,10 @@ '\n' 'str.isspace()\n' '\n' - ' Return true if there are only whitespace characters in ' - 'the string\n' - ' and there is at least one character, false otherwise.\n' + ' Return "True" if there are only whitespace characters ' + 'in the string\n' + ' and there is at least one character, "False" ' + 'otherwise.\n' '\n' ' A character is *whitespace* if in the Unicode character ' 'database\n' @@ -10080,20 +10144,21 @@ '\n' 'str.istitle()\n' '\n' - ' Return true if the string is a titlecased string and ' + ' Return "True" if the string is a titlecased string and ' 'there is at\n' ' least one character, for example uppercase characters ' 'may only\n' ' follow uncased characters and lowercase characters only ' 'cased ones.\n' - ' Return false otherwise.\n' + ' Return "False" otherwise.\n' '\n' 'str.isupper()\n' '\n' - ' Return true if all cased characters [4] in the string ' - 'are uppercase\n' - ' and there is at least one cased character, false ' - 'otherwise.\n' + ' Return "True" if all cased characters [4] in the string ' + 'are\n' + ' uppercase and there is at least one cased character, ' + '"False"\n' + ' otherwise.\n' '\n' 'str.join(iterable)\n' '\n' diff --git a/Misc/NEWS.d/3.8.1rc1.rst b/Misc/NEWS.d/3.8.1rc1.rst new file mode 100644 index 00000000000000..958a8b7b800251 --- /dev/null +++ b/Misc/NEWS.d/3.8.1rc1.rst @@ -0,0 +1,778 @@ +.. bpo: 38945 +.. date: 2019-12-01-22-44-40 +.. nonce: ztmNXc +.. release date: 2019-12-09 +.. section: Security + +Newline characters have been escaped when performing uu encoding to prevent +them from overflowing into to content section of the encoded file. This +prevents malicious or accidental modification of data during the decoding +process. + +.. + +.. bpo: 37228 +.. date: 2019-11-21-21-36-54 +.. nonce: yBZnFG +.. section: Security + +Due to significant security concerns, the *reuse_address* parameter of +:meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This +is because of the behavior of ``SO_REUSEADDR`` in UDP. For more details, see +the documentation for ``loop.create_datagram_endpoint()``. (Contributed by +Kyle Stanley, Antoine Pitrou, and Yury Selivanov in :issue:`37228`.) + +.. + +.. bpo: 38722 +.. date: 2019-11-18-16-17-56 +.. nonce: x3mECW +.. section: Security + +:mod:`runpy` now uses :meth:`io.open_code` to open code files. Patch by +Jason Killen. + +.. + +.. bpo: 38804 +.. date: 2019-11-15-00-54-42 +.. nonce: vjbM8V +.. section: Security + +Fixes a ReDoS vulnerability in :mod:`http.cookiejar`. Patch by Ben Caller. + +.. + +.. bpo: 38622 +.. date: 2019-11-14-16-13-23 +.. nonce: 3DYkfb +.. section: Security + +Add additional audit events for the :mod:`ctypes` module. + +.. + +.. bpo: 38418 +.. date: 2019-10-08-19-29-55 +.. nonce: QL7s0- +.. section: Security + +Fixes audit event for :func:`os.system` to be named ``os.system``. + +.. + +.. bpo: 38673 +.. date: 2019-12-01-00-17-44 +.. nonce: K_Tze- +.. section: Core and Builtins + +In REPL mode, don't switch to PS2 if the line starts with comment or +whitespace. Based on work by Batuhan Taşkaya. + +.. + +.. bpo: 38922 +.. date: 2019-11-26-12-20-34 +.. nonce: i6ja-i +.. section: Core and Builtins + +Calling ``replace`` on a code object now raises the ``code.__new__`` audit +event. + +.. + +.. bpo: 38920 +.. date: 2019-11-26-09-16-47 +.. nonce: Vx__sT +.. section: Core and Builtins + +Add audit hooks for when :func:`sys.excepthook` and +:func:`sys.unraisablehook` are invoked + +.. + +.. bpo: 38892 +.. date: 2019-11-22-22-18-50 +.. nonce: LS586s +.. section: Core and Builtins + +Improve documentation for audit events table and functions. + +.. + +.. bpo: 38707 +.. date: 2019-11-08-00-36-10 +.. nonce: SZL036 +.. section: Core and Builtins + +``MainThread.native_id`` is now correctly reset in child processes spawned +using :class:`multiprocessing.Process`, instead of retaining the parent's +value. + +.. + +.. bpo: 38640 +.. date: 2019-10-30-11-25-25 +.. nonce: 4sAFh5 +.. section: Core and Builtins + +Fixed a bug in the compiler that was causing to raise in the presence of +break statements and continue statements inside always false while loops. +Patch by Pablo Galindo. + +.. + +.. bpo: 38535 +.. date: 2019-10-20-12-43-48 +.. nonce: ESMkVN +.. section: Core and Builtins + +Fixed line numbers and column offsets for AST nodes for calls without +arguments in decorators. + +.. + +.. bpo: 38525 +.. date: 2019-10-20-00-36-18 +.. nonce: Vty1cA +.. section: Core and Builtins + +Fix a segmentation fault when using reverse iterators of empty ``dict`` +objects. Patch by Dong-hee Na and Inada Naoki. + +.. + +.. bpo: 35409 +.. date: 2019-07-13-18-01-13 +.. nonce: ozbcsR +.. section: Core and Builtins + +Ignore GeneratorExit exceptions when throwing an exception into the aclose +coroutine of an asynchronous generator. + +.. + +.. bpo: 39006 +.. date: 2019-12-09-14-40-09 +.. nonce: v4VsPg +.. section: Library + +Fix asyncio when the ssl module is missing: only check for ssl.SSLSocket +instance if the ssl module is available. + +.. + +.. bpo: 38708 +.. date: 2019-12-07-22-25-39 +.. nonce: rZTUfk +.. section: Library + +Fix a potential IndexError in email parser when parsing an empty msg-id. + +.. + +.. bpo: 38698 +.. date: 2019-12-07-21-49-50 +.. nonce: HxoSym +.. section: Library + +Add a new ``InvalidMessageID`` token to email parser to represent invalid +Message-ID headers. Also, add defects when there is remaining value after +parsing the header. + +.. + +.. bpo: 38979 +.. date: 2019-12-07-16-32-42 +.. nonce: q0sIHy +.. section: Library + +Return class from ``ContextVar.__class_getitem__`` to simplify subclassing. + +.. + +.. bpo: 38986 +.. date: 2019-12-06-15-11-42 +.. nonce: bg6iZt +.. section: Library + +Make repr of C accelerated TaskWakeupMethWrapper the same as of pure Python +version. + +.. + +.. bpo: 38529 +.. date: 2019-12-05-16-13-25 +.. nonce: yvQgx3 +.. section: Library + +Drop too noisy asyncio warning about deletion of a stream without explicit +``.close()`` call. + +.. + +.. bpo: 38634 +.. date: 2019-12-04-15-56-28 +.. nonce: pq0ZWa +.. section: Library + +The :mod:`readline` module now detects if Python is linked to libedit at +runtime on all platforms. Previously, the check was only done on macOS. + +.. + +.. bpo: 33684 +.. date: 2019-12-04-15-28-40 +.. nonce: QeSmQP +.. section: Library + +Fix ``json.tool`` failed to read a JSON file with non-ASCII characters when +locale encoding is not UTF-8. + +.. + +.. bpo: 38698 +.. date: 2019-12-02-10-35-19 +.. nonce: WZnAPQ +.. section: Library + +Prevent UnboundLocalError to pop up in parse_message_id + +parse_message_id() was improperly using a token defined inside an exception +handler, which was raising `UnboundLocalError` on parsing an invalid value. +Patch by Claudiu Popa. + +.. + +.. bpo: 26730 +.. date: 2019-11-27-16-30-02 +.. nonce: 56cdBn +.. section: Library + +Fix ``SpooledTemporaryFile.rollover()`` might corrupt the file when it is in +text mode. Patch by Serhiy Storchaka. + +.. + +.. bpo: 38668 +.. date: 2019-11-22-10-45-03 +.. nonce: iKx23z +.. section: Library + +Calling func:`shutil.copytree` to copy a directory tree from one directory +to another subdirectory resulted in an endless loop and a RecursionError. A +fix was added to consume an iterator and create the list of the entries to +be copied, avoiding the recursion for newly created directories. Patch by +Bruno P. Kinoshita. + +.. + +.. bpo: 37838 +.. date: 2019-11-21-11-39-17 +.. nonce: lRFcEC +.. section: Library + +:meth:`typing.get_type_hints` properly handles functions decorated with +:meth:`functools.wraps`. + +.. + +.. bpo: 38859 +.. date: 2019-11-19-16-30-46 +.. nonce: AZUzL8 +.. section: Library + +AsyncMock now returns StopAsyncIteration on the exaustion of a side_effects +iterable. Since PEP-479 its Impossible to raise a StopIteration exception +from a coroutine. + +.. + +.. bpo: 38857 +.. date: 2019-11-19-16-28-25 +.. nonce: YPUkU9 +.. section: Library + +AsyncMock fix for return values that are awaitable types. This also covers +side_effect iterable values that happend to be awaitable, and wraps +callables that return an awaitable type. Before these awaitables were being +awaited instead of being returned as is. + +.. + +.. bpo: 38821 +.. date: 2019-11-16-23-26-25 +.. nonce: -albNN +.. section: Library + +Fix unhandled exceptions in :mod:`argparse` when internationalizing error +messages for arguments with ``nargs`` set to special (non-integer) values. +Patch by Federico Bond. + +.. + +.. bpo: 38820 +.. date: 2019-11-16-16-09-07 +.. nonce: ivhUSV +.. section: Library + +Make Python compatible with OpenSSL 3.0.0. :func:`ssl.SSLSocket.getpeercert` +no longer returns IPv6 addresses with a trailing new line. + +.. + +.. bpo: 38807 +.. date: 2019-11-15-09-30-29 +.. nonce: PsmRog +.. section: Library + +Update :exc:`TypeError` messages for :meth:`os.path.join` to include +:class:`os.PathLike` objects as acceptable input types. + +.. + +.. bpo: 38785 +.. date: 2019-11-13-16-17-43 +.. nonce: NEOEfk +.. section: Library + +Prevent asyncio from crashing if parent ``__init__`` is not called from a +constructor of object derived from ``asyncio.Future``. + +.. + +.. bpo: 38723 +.. date: 2019-11-12-15-46-28 +.. nonce: gcdMFn +.. section: Library + +:mod:`pdb` now uses :meth:`io.open_code` to trigger auditing events. + +.. + +.. bpo: 27805 +.. date: 2019-11-11-21-43-06 +.. nonce: D3zl1_ +.. section: Library + +Allow opening pipes and other non-seekable files in append mode with +:func:`open`. + +.. + +.. bpo: 38686 +.. date: 2019-11-06-15-26-15 +.. nonce: HNFBce +.. section: Library + +Added support for multiple ``qop`` values in +:class:`urllib.request.AbstractDigestAuthHandler`. + +.. + +.. bpo: 38334 +.. date: 2019-10-27-00-08-49 +.. nonce: pfLLmc +.. section: Library + +Fixed seeking backward on an encrypted :class:`zipfile.ZipExtFile`. + +.. + +.. bpo: 34679 +.. date: 2019-10-23-16-25-12 +.. nonce: Bnw8o3 +.. section: Library + +asynci.ProactorEventLoop.close() now only calls signal.set_wakeup_fd() in +the main thread. + +.. + +.. bpo: 31202 +.. date: 2019-10-20-12-04-48 +.. nonce: NfdIus +.. section: Library + +The case the result of :func:`pathlib.WindowsPath.glob` matches now the case +of the pattern for literal parts. + +.. + +.. bpo: 38521 +.. date: 2019-10-18-13-57-31 +.. nonce: U-7aaM +.. section: Library + +Fixed erroneous equality comparison in statistics.NormalDist(). + +.. + +.. bpo: 38478 +.. date: 2019-10-15-11-37-57 +.. nonce: A87OPO +.. section: Library + +Fixed a bug in :meth:`inspect.signature.bind` that was causing it to fail +when handling a keyword argument with same name as positional-only +parameter. Patch by Pablo Galindo. + +.. + +.. bpo: 33604 +.. date: 2019-10-15-09-47-40 +.. nonce: J12cWT +.. section: Library + +Fixed `hmac.new` and `hmac.HMAC` to raise TypeError instead of ValueError +when the digestmod parameter, now required in 3.8, is omitted. Also +clarified the hmac module documentation and docstrings. + +.. + +.. bpo: 38422 +.. date: 2019-10-09-18-16-51 +.. nonce: aiM5bq +.. section: Library + +Clarify docstrings of pathlib suffix(es) + +.. + +.. bpo: 36993 +.. date: 2019-07-09-05-44-39 +.. nonce: 4javqu +.. section: Library + +Improve error reporting for corrupt zip files with bad zip64 extra data. +Patch by Daniel Hillier. + +.. + +.. bpo: 36820 +.. date: 2019-05-06-15-34-17 +.. nonce: Eh5mIB +.. section: Library + +Break cycle generated when saving an exception in socket.py, codeop.py and +dyld.py as they keep alive not only the exception but user objects through +the ``__traceback__`` attribute. Patch by Mario Corchero. + +.. + +.. bpo: 34776 +.. date: 2018-09-23-14-24-37 +.. nonce: 1SrQe3 +.. section: Library + +Fix dataclasses to support forward references in type annotations + +.. + +.. bpo: 33348 +.. date: 2018-04-24-13-18-48 +.. nonce: XaJDei +.. section: Library + +lib2to3 now recognizes expressions after ``*`` and `**` like in ``f(*[] or +[])``. + +.. + +.. bpo: 27657 +.. date: 2017-12-26-14-32-23 +.. nonce: 6BhyVK +.. section: Library + +Fix urllib.parse.urlparse() with numeric paths. A string like "path:80" is +no longer parsed as a path but as a scheme ("path") and a path ("80"). + +.. + +.. bpo: 38816 +.. date: 2019-11-15-11-39-13 +.. nonce: vUaSVL +.. section: Documentation + +Provides more details about the interaction between :c:func:`fork` and +CPython's runtime, focusing just on the C-API. This includes cautions about +where :c:func:`fork` should and shouldn't be called. + +.. + +.. bpo: 38351 +.. date: 2019-11-15-09-22-28 +.. nonce: xwhlse +.. section: Documentation + +Modernize :mod:`email` examples from %-formatting to f-strings. + +.. + +.. bpo: 38778 +.. date: 2019-11-12-15-31-09 +.. nonce: PHhTlv +.. section: Documentation + +Document the fact that :exc:`RuntimeError` is raised if :meth:`os.fork` is +called in a subinterpreter. + +.. + +.. bpo: 38592 +.. date: 2019-10-26-13-19-07 +.. nonce: Y96BYO +.. section: Documentation + +Add Brazilian Portuguese to the language switcher at Python Documentation +website. + +.. + +.. bpo: 38547 +.. date: 2019-12-09-11-32-34 +.. nonce: Juw54e +.. section: Tests + +Fix test_pty: if the process is the session leader, closing the master file +descriptor raises a SIGHUP signal: simply ignore SIGHUP when running the +tests. + +.. + +.. bpo: 38992 +.. date: 2019-12-08-15-11-06 +.. nonce: cVoHOZ +.. section: Tests + +Fix a test for :func:`math.fsum` that was failing due to constant folding. + +.. + +.. bpo: 38965 +.. date: 2019-12-04-17-08-55 +.. nonce: yqax3m +.. section: Tests + +Fix test_faulthandler on GCC 10. Use the "volatile" keyword in +``faulthandler._stack_overflow()`` to prevent tail call optimization on any +compiler, rather than relying on compiler specific pragma. + +.. + +.. bpo: 38875 +.. date: 2019-11-21-09-11-06 +.. nonce: wSZJal +.. section: Tests + +test_capi: trashcan tests now require the test "cpu" resource. + +.. + +.. bpo: 38841 +.. date: 2019-11-20-16-08-19 +.. nonce: 5F5Lbw +.. section: Tests + +Skip asyncio test_create_datagram_endpoint_existing_sock_unix on platforms +lacking a functional bind() for named unix domain sockets. + +.. + +.. bpo: 38669 +.. date: 2019-11-04-02-54-16 +.. nonce: pazXZ8 +.. section: Tests + +Raise :exc:`TypeError` when passing target as a string with +:meth:`unittest.mock.patch.object`. + +.. + +.. bpo: 35998 +.. date: 2019-10-16-01-36-15 +.. nonce: G305Bf +.. section: Tests + +Fix a race condition in test_asyncio.test_start_tls_server_1(). Previously, +there was a race condition between the test main() function which replaces +the protocol and the test ServerProto protocol which sends ANSWER once it +gets HELLO. Now, only the test main() function is responsible to send data, +ServerProto no longer sends data. + +.. + +.. bpo: 37404 +.. date: 2019-12-01-21-45-24 +.. nonce: cNsA7S +.. section: Build + +:mod:`asyncio` now raises :exc:`TyperError` when calling incompatible +methods with an :class:`ssl.SSLSocket` socket. Patch by Ido Michael. + +.. + +.. bpo: 38809 +.. date: 2019-11-15-09-25-44 +.. nonce: 9jwta6 +.. section: Build + +On Windows, build scripts will now recognize and use python.exe from an +active virtual env. + +.. + +.. bpo: 38684 +.. date: 2019-11-04-14-30-37 +.. nonce: aed593 +.. section: Build + +Fix _hashlib build when Blake2 is disabled, but OpenSSL supports it. + +.. + +.. bpo: 37415 +.. date: 2019-10-11-15-32-58 +.. nonce: D9RXrq +.. section: Build + +Fix stdatomic.h header check for ICC compiler: the ICC implementation lacks +atomic_uintptr_t type which is needed by Python. + +.. + +.. bpo: 33125 +.. date: 2019-11-14-08-57-50 +.. nonce: EN5MWS +.. section: Windows + +Add support for building and releasing Windows ARM64 packages. + +.. + +.. bpo: 38589 +.. date: 2019-10-28-10-48-16 +.. nonce: V69Q1a +.. section: Windows + +Fixes HTML Help shortcut when Windows is not installed to C drive + +.. + +.. bpo: 38453 +.. date: 2019-10-28-10-32-43 +.. nonce: NwwatW +.. section: Windows + +Ensure ntpath.realpath() correctly resolves relative paths. + +.. + +.. bpo: 38519 +.. date: 2019-10-28-05-01-29 +.. nonce: dCkY66 +.. section: Windows + +Restores the internal C headers that were missing from the nuget.org and +Microsoft Store packages. + +.. + +.. bpo: 38492 +.. date: 2019-10-16-09-49-09 +.. nonce: Te1LxC +.. section: Windows + +Remove ``pythonw.exe`` dependency on the Microsoft C++ runtime. + +.. + +.. bpo: 37931 +.. date: 2019-08-23-12-14-34 +.. nonce: goYgQj +.. section: macOS + +Fixed a crash on OSX dynamic builds that occurred when re-initializing the +posix module after a Py_Finalize if the environment had changed since the +previous `import posix`. Patch by Benoît Hudson. + +.. + +.. bpo: 38862 +.. date: 2019-11-23-21-50-57 +.. nonce: KQ9A0m +.. section: IDLE + +'Strip Trailing Whitespace' on the Format menu removes extra newlines at the +end of non-shell files. + +.. + +.. bpo: 26353 +.. date: 2019-11-09-23-55-59 +.. nonce: duYZiF +.. section: IDLE + +Stop adding newline when saving an IDLE shell window. + +.. + +.. bpo: 38636 +.. date: 2019-10-30-22-11-16 +.. nonce: hUhDeB +.. section: IDLE + +Fix IDLE Format menu tab toggle and file indent width. These functions +(default shortcuts Alt-T and Alt-U) were mistakenly disabled in 3.7.5 and +3.8.0. + +.. + +.. bpo: 4630 +.. date: 2019-10-28-04-48-03 +.. nonce: upgjiV +.. section: IDLE + +Add an option to toggle IDLE's cursor blink for shell, editor, and output +windows. See Settings, General, Window Preferences, Cursor Blink. Patch by +Zachary Spytz. + +.. + +.. bpo: 38598 +.. date: 2019-10-26-18-16-24 +.. nonce: 6kH9FY +.. section: IDLE + +Do not try to compile IDLE shell or output windows + +.. + +.. bpo: 37633 +.. date: 2019-11-04-21-10-47 +.. nonce: oOGVdo +.. section: C API + +Reëxport some function compatibility wrappers for macros in ``pythonrun.h``. + +.. + +.. bpo: 38540 +.. date: 2019-10-21-09-24-03 +.. nonce: 314N_T +.. section: C API + +Fixed possible leak in :c:func:`PyArg_Parse` and similar functions for +format units ``"es#"`` and ``"et#"`` when the macro +:c:macro:`PY_SSIZE_T_CLEAN` is not defined. + +.. + +.. bpo: 36389 +.. date: 2019-10-07-17-15-09 +.. nonce: hFX_jD +.. section: C API + +The ``_PyObject_CheckConsistency()`` function is now also available in +release mode. For example, it can be used to debug a crash in the +``visit_decref()`` function of the GC. diff --git a/Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst b/Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst deleted file mode 100644 index 98f4a3bf4f5457..00000000000000 --- a/Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix stdatomic.h header check for ICC compiler: the ICC implementation lacks -atomic_uintptr_t type which is needed by Python. diff --git a/Misc/NEWS.d/next/Build/2019-11-04-14-30-37.bpo-38684.aed593.rst b/Misc/NEWS.d/next/Build/2019-11-04-14-30-37.bpo-38684.aed593.rst deleted file mode 100644 index c715ff97041275..00000000000000 --- a/Misc/NEWS.d/next/Build/2019-11-04-14-30-37.bpo-38684.aed593.rst +++ /dev/null @@ -1 +0,0 @@ -Fix _hashlib build when Blake2 is disabled, but OpenSSL supports it. diff --git a/Misc/NEWS.d/next/Build/2019-11-15-09-25-44.bpo-38809.9jwta6.rst b/Misc/NEWS.d/next/Build/2019-11-15-09-25-44.bpo-38809.9jwta6.rst deleted file mode 100644 index 10eaf844e8aaf0..00000000000000 --- a/Misc/NEWS.d/next/Build/2019-11-15-09-25-44.bpo-38809.9jwta6.rst +++ /dev/null @@ -1,2 +0,0 @@ -On Windows, build scripts will now recognize and use python.exe from an -active virtual env. diff --git a/Misc/NEWS.d/next/Build/2019-12-01-21-45-24.bpo-37404.cNsA7S.rst b/Misc/NEWS.d/next/Build/2019-12-01-21-45-24.bpo-37404.cNsA7S.rst deleted file mode 100644 index 067fc9d3f189dc..00000000000000 --- a/Misc/NEWS.d/next/Build/2019-12-01-21-45-24.bpo-37404.cNsA7S.rst +++ /dev/null @@ -1,2 +0,0 @@ -:mod:`asyncio` now raises :exc:`TyperError` when calling incompatible methods -with an :class:`ssl.SSLSocket` socket. Patch by Ido Michael. diff --git a/Misc/NEWS.d/next/C API/2019-10-07-17-15-09.bpo-36389.hFX_jD.rst b/Misc/NEWS.d/next/C API/2019-10-07-17-15-09.bpo-36389.hFX_jD.rst deleted file mode 100644 index 6c42882cdbaa60..00000000000000 --- a/Misc/NEWS.d/next/C API/2019-10-07-17-15-09.bpo-36389.hFX_jD.rst +++ /dev/null @@ -1,3 +0,0 @@ -The ``_PyObject_CheckConsistency()`` function is now also available in release -mode. For example, it can be used to debug a crash in the ``visit_decref()`` -function of the GC. diff --git a/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst b/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst deleted file mode 100644 index 1d73ad8fe96e68..00000000000000 --- a/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fixed possible leak in :c:func:`PyArg_Parse` and similar functions for -format units ``"es#"`` and ``"et#"`` when the macro -:c:macro:`PY_SSIZE_T_CLEAN` is not defined. diff --git a/Misc/NEWS.d/next/C API/2019-11-04-21-10-47.bpo-37633.oOGVdo.rst b/Misc/NEWS.d/next/C API/2019-11-04-21-10-47.bpo-37633.oOGVdo.rst deleted file mode 100644 index fdf6abbf1c649b..00000000000000 --- a/Misc/NEWS.d/next/C API/2019-11-04-21-10-47.bpo-37633.oOGVdo.rst +++ /dev/null @@ -1 +0,0 @@ -Reëxport some function compatibility wrappers for macros in ``pythonrun.h``. diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-07-13-18-01-13.bpo-35409.ozbcsR.rst b/Misc/NEWS.d/next/Core and Builtins/2019-07-13-18-01-13.bpo-35409.ozbcsR.rst deleted file mode 100644 index 0f35a91ec74e6f..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-07-13-18-01-13.bpo-35409.ozbcsR.rst +++ /dev/null @@ -1,2 +0,0 @@ -Ignore GeneratorExit exceptions when throwing an exception into the aclose -coroutine of an asynchronous generator. diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-20-00-36-18.bpo-38525.Vty1cA.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-00-36-18.bpo-38525.Vty1cA.rst deleted file mode 100644 index c74d143762253d..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-10-20-00-36-18.bpo-38525.Vty1cA.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix a segmentation fault when using reverse iterators of empty ``dict`` objects. -Patch by Dong-hee Na and Inada Naoki. diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst deleted file mode 100644 index 7671fd06474ba5..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed line numbers and column offsets for AST nodes for calls without -arguments in decorators. diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-30-11-25-25.bpo-38640.4sAFh5.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-30-11-25-25.bpo-38640.4sAFh5.rst deleted file mode 100644 index d99db3cce49a66..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-10-30-11-25-25.bpo-38640.4sAFh5.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fixed a bug in the compiler that was causing to raise in the presence of -break statements and continue statements inside always false while loops. -Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-11-08-00-36-10.bpo-38707.SZL036.rst b/Misc/NEWS.d/next/Core and Builtins/2019-11-08-00-36-10.bpo-38707.SZL036.rst deleted file mode 100644 index 4ef9ed81931b72..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-11-08-00-36-10.bpo-38707.SZL036.rst +++ /dev/null @@ -1 +0,0 @@ -``MainThread.native_id`` is now correctly reset in child processes spawned using :class:`multiprocessing.Process`, instead of retaining the parent's value. diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-11-22-22-18-50.bpo-38892.LS586s.rst b/Misc/NEWS.d/next/Core and Builtins/2019-11-22-22-18-50.bpo-38892.LS586s.rst deleted file mode 100644 index 5df67dcbfeac9a..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-11-22-22-18-50.bpo-38892.LS586s.rst +++ /dev/null @@ -1 +0,0 @@ -Improve documentation for audit events table and functions. diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-11-26-09-16-47.bpo-38920.Vx__sT.rst b/Misc/NEWS.d/next/Core and Builtins/2019-11-26-09-16-47.bpo-38920.Vx__sT.rst deleted file mode 100644 index 2e9e443dd999bd..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-11-26-09-16-47.bpo-38920.Vx__sT.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add audit hooks for when :func:`sys.excepthook` and -:func:`sys.unraisablehook` are invoked diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-11-26-12-20-34.bpo-38922.i6ja-i.rst b/Misc/NEWS.d/next/Core and Builtins/2019-11-26-12-20-34.bpo-38922.i6ja-i.rst deleted file mode 100644 index a7af652e5a5bdd..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-11-26-12-20-34.bpo-38922.i6ja-i.rst +++ /dev/null @@ -1,2 +0,0 @@ -Calling ``replace`` on a code object now raises the ``code.__new__`` audit -event. diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-12-01-00-17-44.bpo-38673.K_Tze-.rst b/Misc/NEWS.d/next/Core and Builtins/2019-12-01-00-17-44.bpo-38673.K_Tze-.rst deleted file mode 100644 index 8f8cf88e5e2103..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-12-01-00-17-44.bpo-38673.K_Tze-.rst +++ /dev/null @@ -1 +0,0 @@ -In REPL mode, don't switch to PS2 if the line starts with comment or whitespace. Based on work by Batuhan Taşkaya. diff --git a/Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst b/Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst deleted file mode 100644 index 3752d48a7cb4f8..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst +++ /dev/null @@ -1 +0,0 @@ -Add Brazilian Portuguese to the language switcher at Python Documentation website. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Documentation/2019-11-12-15-31-09.bpo-38778.PHhTlv.rst b/Misc/NEWS.d/next/Documentation/2019-11-12-15-31-09.bpo-38778.PHhTlv.rst deleted file mode 100644 index 053e1d294b70f8..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2019-11-12-15-31-09.bpo-38778.PHhTlv.rst +++ /dev/null @@ -1 +0,0 @@ -Document the fact that :exc:`RuntimeError` is raised if :meth:`os.fork` is called in a subinterpreter. diff --git a/Misc/NEWS.d/next/Documentation/2019-11-15-09-22-28.bpo-38351.xwhlse.rst b/Misc/NEWS.d/next/Documentation/2019-11-15-09-22-28.bpo-38351.xwhlse.rst deleted file mode 100644 index 8e0dc9eb4ca2f4..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2019-11-15-09-22-28.bpo-38351.xwhlse.rst +++ /dev/null @@ -1 +0,0 @@ -Modernize :mod:`email` examples from %-formatting to f-strings. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Documentation/2019-11-15-11-39-13.bpo-38816.vUaSVL.rst b/Misc/NEWS.d/next/Documentation/2019-11-15-11-39-13.bpo-38816.vUaSVL.rst deleted file mode 100644 index 49accbc70663a8..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2019-11-15-11-39-13.bpo-38816.vUaSVL.rst +++ /dev/null @@ -1,3 +0,0 @@ -Provides more details about the interaction between :c:func:`fork` and -CPython's runtime, focusing just on the C-API. This includes cautions -about where :c:func:`fork` should and shouldn't be called. diff --git a/Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst b/Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst deleted file mode 100644 index 5d04e4a79b6221..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst +++ /dev/null @@ -1 +0,0 @@ -Do not try to compile IDLE shell or output windows diff --git a/Misc/NEWS.d/next/IDLE/2019-10-28-04-48-03.bpo-4630.upgjiV.rst b/Misc/NEWS.d/next/IDLE/2019-10-28-04-48-03.bpo-4630.upgjiV.rst deleted file mode 100644 index 759b35b77fb8d0..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2019-10-28-04-48-03.bpo-4630.upgjiV.rst +++ /dev/null @@ -1,3 +0,0 @@ -Add an option to toggle IDLE's cursor blink for shell, editor, and output -windows. See Settings, General, Window Preferences, Cursor Blink. -Patch by Zachary Spytz. diff --git a/Misc/NEWS.d/next/IDLE/2019-10-30-22-11-16.bpo-38636.hUhDeB.rst b/Misc/NEWS.d/next/IDLE/2019-10-30-22-11-16.bpo-38636.hUhDeB.rst deleted file mode 100644 index 4262dbea6d8673..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2019-10-30-22-11-16.bpo-38636.hUhDeB.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix IDLE Format menu tab toggle and file indent width. These functions -(default shortcuts Alt-T and Alt-U) were mistakenly disabled in 3.7.5 -and 3.8.0. diff --git a/Misc/NEWS.d/next/IDLE/2019-11-09-23-55-59.bpo-26353.duYZiF.rst b/Misc/NEWS.d/next/IDLE/2019-11-09-23-55-59.bpo-26353.duYZiF.rst deleted file mode 100644 index fd0a2a36821008..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2019-11-09-23-55-59.bpo-26353.duYZiF.rst +++ /dev/null @@ -1,2 +0,0 @@ -Stop adding newline when saving an IDLE shell window. - diff --git a/Misc/NEWS.d/next/IDLE/2019-11-23-21-50-57.bpo-38862.KQ9A0m.rst b/Misc/NEWS.d/next/IDLE/2019-11-23-21-50-57.bpo-38862.KQ9A0m.rst deleted file mode 100644 index 14bab9e854bdce..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2019-11-23-21-50-57.bpo-38862.KQ9A0m.rst +++ /dev/null @@ -1,2 +0,0 @@ -'Strip Trailing Whitespace' on the Format menu removes extra newlines -at the end of non-shell files. diff --git a/Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst b/Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst deleted file mode 100644 index 77746c0ce630ff..00000000000000 --- a/Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix urllib.parse.urlparse() with numeric paths. A string like "path:80" is -no longer parsed as a path but as a scheme ("path") and a path ("80"). diff --git a/Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst b/Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst deleted file mode 100644 index f95a73fb57ed41..00000000000000 --- a/Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst +++ /dev/null @@ -1,2 +0,0 @@ -lib2to3 now recognizes expressions after ``*`` and `**` like in ``f(*[] or -[])``. diff --git a/Misc/NEWS.d/next/Library/2018-09-23-14-24-37.bpo-34776.1SrQe3.rst b/Misc/NEWS.d/next/Library/2018-09-23-14-24-37.bpo-34776.1SrQe3.rst deleted file mode 100644 index 815a4876e0b4aa..00000000000000 --- a/Misc/NEWS.d/next/Library/2018-09-23-14-24-37.bpo-34776.1SrQe3.rst +++ /dev/null @@ -1 +0,0 @@ -Fix dataclasses to support forward references in type annotations diff --git a/Misc/NEWS.d/next/Library/2019-05-06-15-34-17.bpo-36820.Eh5mIB.rst b/Misc/NEWS.d/next/Library/2019-05-06-15-34-17.bpo-36820.Eh5mIB.rst deleted file mode 100644 index 82f6635c81582d..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-05-06-15-34-17.bpo-36820.Eh5mIB.rst +++ /dev/null @@ -1,3 +0,0 @@ -Break cycle generated when saving an exception in socket.py, codeop.py and -dyld.py as they keep alive not only the exception but user objects through -the ``__traceback__`` attribute. Patch by Mario Corchero. diff --git a/Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst b/Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst deleted file mode 100644 index 009e07b92d22d3..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst +++ /dev/null @@ -1,2 +0,0 @@ -Improve error reporting for corrupt zip files with bad zip64 extra data. Patch -by Daniel Hillier. diff --git a/Misc/NEWS.d/next/Library/2019-10-09-18-16-51.bpo-38422.aiM5bq.rst b/Misc/NEWS.d/next/Library/2019-10-09-18-16-51.bpo-38422.aiM5bq.rst deleted file mode 100644 index 0958fe265db560..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-10-09-18-16-51.bpo-38422.aiM5bq.rst +++ /dev/null @@ -1 +0,0 @@ -Clarify docstrings of pathlib suffix(es) diff --git a/Misc/NEWS.d/next/Library/2019-10-15-09-47-40.bpo-33604.J12cWT.rst b/Misc/NEWS.d/next/Library/2019-10-15-09-47-40.bpo-33604.J12cWT.rst deleted file mode 100644 index fbd73003cfcb64..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-10-15-09-47-40.bpo-33604.J12cWT.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fixed `hmac.new` and `hmac.HMAC` to raise TypeError instead of ValueError -when the digestmod parameter, now required in 3.8, is omitted. Also -clarified the hmac module documentation and docstrings. diff --git a/Misc/NEWS.d/next/Library/2019-10-15-11-37-57.bpo-38478.A87OPO.rst b/Misc/NEWS.d/next/Library/2019-10-15-11-37-57.bpo-38478.A87OPO.rst deleted file mode 100644 index b19fa23639d886..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-10-15-11-37-57.bpo-38478.A87OPO.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fixed a bug in :meth:`inspect.signature.bind` that was causing it to fail -when handling a keyword argument with same name as positional-only parameter. -Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Library/2019-10-18-13-57-31.bpo-38521.U-7aaM.rst b/Misc/NEWS.d/next/Library/2019-10-18-13-57-31.bpo-38521.U-7aaM.rst deleted file mode 100644 index 9335bdc9545c4d..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-10-18-13-57-31.bpo-38521.U-7aaM.rst +++ /dev/null @@ -1 +0,0 @@ -Fixed erroneous equality comparison in statistics.NormalDist(). diff --git a/Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst b/Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst deleted file mode 100644 index 8edb09d61317b3..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst +++ /dev/null @@ -1,2 +0,0 @@ -The case the result of :func:`pathlib.WindowsPath.glob` matches now the case -of the pattern for literal parts. diff --git a/Misc/NEWS.d/next/Library/2019-10-23-16-25-12.bpo-34679.Bnw8o3.rst b/Misc/NEWS.d/next/Library/2019-10-23-16-25-12.bpo-34679.Bnw8o3.rst deleted file mode 100644 index 34334db6032b5b..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-10-23-16-25-12.bpo-34679.Bnw8o3.rst +++ /dev/null @@ -1,2 +0,0 @@ -asynci.ProactorEventLoop.close() now only calls signal.set_wakeup_fd() in the -main thread. diff --git a/Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst b/Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst deleted file mode 100644 index 0d05d3f6e6c09d..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst +++ /dev/null @@ -1 +0,0 @@ -Fixed seeking backward on an encrypted :class:`zipfile.ZipExtFile`. diff --git a/Misc/NEWS.d/next/Library/2019-11-06-15-26-15.bpo-38686.HNFBce.rst b/Misc/NEWS.d/next/Library/2019-11-06-15-26-15.bpo-38686.HNFBce.rst deleted file mode 100644 index 7a419ff1e3338e..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-06-15-26-15.bpo-38686.HNFBce.rst +++ /dev/null @@ -1 +0,0 @@ -Added support for multiple ``qop`` values in :class:`urllib.request.AbstractDigestAuthHandler`. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2019-11-11-21-43-06.bpo-27805.D3zl1_.rst b/Misc/NEWS.d/next/Library/2019-11-11-21-43-06.bpo-27805.D3zl1_.rst deleted file mode 100644 index 37be6a5d0b22bc..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-11-21-43-06.bpo-27805.D3zl1_.rst +++ /dev/null @@ -1,2 +0,0 @@ -Allow opening pipes and other non-seekable files in append mode with -:func:`open`. diff --git a/Misc/NEWS.d/next/Library/2019-11-12-15-46-28.bpo-38723.gcdMFn.rst b/Misc/NEWS.d/next/Library/2019-11-12-15-46-28.bpo-38723.gcdMFn.rst deleted file mode 100644 index c84bb8589d3034..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-12-15-46-28.bpo-38723.gcdMFn.rst +++ /dev/null @@ -1 +0,0 @@ -:mod:`pdb` now uses :meth:`io.open_code` to trigger auditing events. diff --git a/Misc/NEWS.d/next/Library/2019-11-13-16-17-43.bpo-38785.NEOEfk.rst b/Misc/NEWS.d/next/Library/2019-11-13-16-17-43.bpo-38785.NEOEfk.rst deleted file mode 100644 index 49e99379980814..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-13-16-17-43.bpo-38785.NEOEfk.rst +++ /dev/null @@ -1,2 +0,0 @@ -Prevent asyncio from crashing if parent ``__init__`` is not called from a -constructor of object derived from ``asyncio.Future``. diff --git a/Misc/NEWS.d/next/Library/2019-11-15-09-30-29.bpo-38807.PsmRog.rst b/Misc/NEWS.d/next/Library/2019-11-15-09-30-29.bpo-38807.PsmRog.rst deleted file mode 100644 index 2bd7e3deb19ed2..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-15-09-30-29.bpo-38807.PsmRog.rst +++ /dev/null @@ -1 +0,0 @@ -Update :exc:`TypeError` messages for :meth:`os.path.join` to include :class:`os.PathLike` objects as acceptable input types. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2019-11-16-16-09-07.bpo-38820.ivhUSV.rst b/Misc/NEWS.d/next/Library/2019-11-16-16-09-07.bpo-38820.ivhUSV.rst deleted file mode 100644 index 2c6a6e853c25f6..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-16-16-09-07.bpo-38820.ivhUSV.rst +++ /dev/null @@ -1,2 +0,0 @@ -Make Python compatible with OpenSSL 3.0.0. :func:`ssl.SSLSocket.getpeercert` -no longer returns IPv6 addresses with a trailing new line. diff --git a/Misc/NEWS.d/next/Library/2019-11-16-23-26-25.bpo-38821.-albNN.rst b/Misc/NEWS.d/next/Library/2019-11-16-23-26-25.bpo-38821.-albNN.rst deleted file mode 100644 index 2e7a22f661ac69..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-16-23-26-25.bpo-38821.-albNN.rst +++ /dev/null @@ -1 +0,0 @@ -Fix unhandled exceptions in :mod:`argparse` when internationalizing error messages for arguments with ``nargs`` set to special (non-integer) values. Patch by Federico Bond. diff --git a/Misc/NEWS.d/next/Library/2019-11-19-16-28-25.bpo-38857.YPUkU9.rst b/Misc/NEWS.d/next/Library/2019-11-19-16-28-25.bpo-38857.YPUkU9.rst deleted file mode 100644 index f28df2811fb787..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-19-16-28-25.bpo-38857.YPUkU9.rst +++ /dev/null @@ -1,4 +0,0 @@ -AsyncMock fix for return values that are awaitable types. This also covers -side_effect iterable values that happend to be awaitable, and wraps -callables that return an awaitable type. Before these awaitables were being -awaited instead of being returned as is. diff --git a/Misc/NEWS.d/next/Library/2019-11-19-16-30-46.bpo-38859.AZUzL8.rst b/Misc/NEWS.d/next/Library/2019-11-19-16-30-46.bpo-38859.AZUzL8.rst deleted file mode 100644 index c059539a1de602..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-19-16-30-46.bpo-38859.AZUzL8.rst +++ /dev/null @@ -1,3 +0,0 @@ -AsyncMock now returns StopAsyncIteration on the exaustion of a side_effects -iterable. Since PEP-479 its Impossible to raise a StopIteration exception -from a coroutine. diff --git a/Misc/NEWS.d/next/Library/2019-11-21-11-39-17.bpo-37838.lRFcEC.rst b/Misc/NEWS.d/next/Library/2019-11-21-11-39-17.bpo-37838.lRFcEC.rst deleted file mode 100644 index 96d804addeb609..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-21-11-39-17.bpo-37838.lRFcEC.rst +++ /dev/null @@ -1 +0,0 @@ -:meth:`typing.get_type_hints` properly handles functions decorated with :meth:`functools.wraps`. diff --git a/Misc/NEWS.d/next/Library/2019-11-22-10-45-03.bpo-38668.iKx23z.rst b/Misc/NEWS.d/next/Library/2019-11-22-10-45-03.bpo-38668.iKx23z.rst deleted file mode 100644 index 28b82ab1619e35..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-22-10-45-03.bpo-38668.iKx23z.rst +++ /dev/null @@ -1,5 +0,0 @@ -Calling func:`shutil.copytree` to copy a directory tree from one directory -to another subdirectory resulted in an endless loop and a RecursionError. A -fix was added to consume an iterator and create the list of the entries to -be copied, avoiding the recursion for newly created directories. Patch by -Bruno P. Kinoshita. diff --git a/Misc/NEWS.d/next/Library/2019-11-27-16-30-02.bpo-26730.56cdBn.rst b/Misc/NEWS.d/next/Library/2019-11-27-16-30-02.bpo-26730.56cdBn.rst deleted file mode 100644 index a92b90a4956053..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-27-16-30-02.bpo-26730.56cdBn.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix ``SpooledTemporaryFile.rollover()`` might corrupt the file when it is in -text mode. Patch by Serhiy Storchaka. diff --git a/Misc/NEWS.d/next/Library/2019-12-02-10-35-19.bpo-38698.WZnAPQ.rst b/Misc/NEWS.d/next/Library/2019-12-02-10-35-19.bpo-38698.WZnAPQ.rst deleted file mode 100644 index e606acb5dcf573..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-02-10-35-19.bpo-38698.WZnAPQ.rst +++ /dev/null @@ -1,5 +0,0 @@ -Prevent UnboundLocalError to pop up in parse_message_id - -parse_message_id() was improperly using a token defined inside an exception -handler, which was raising `UnboundLocalError` on parsing an invalid value. -Patch by Claudiu Popa. diff --git a/Misc/NEWS.d/next/Library/2019-12-04-15-28-40.bpo-33684.QeSmQP.rst b/Misc/NEWS.d/next/Library/2019-12-04-15-28-40.bpo-33684.QeSmQP.rst deleted file mode 100644 index 107f9bb008330f..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-04-15-28-40.bpo-33684.QeSmQP.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix ``json.tool`` failed to read a JSON file with non-ASCII characters when -locale encoding is not UTF-8. diff --git a/Misc/NEWS.d/next/Library/2019-12-04-15-56-28.bpo-38634.pq0ZWa.rst b/Misc/NEWS.d/next/Library/2019-12-04-15-56-28.bpo-38634.pq0ZWa.rst deleted file mode 100644 index d60c3172c2e510..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-04-15-56-28.bpo-38634.pq0ZWa.rst +++ /dev/null @@ -1,2 +0,0 @@ -The :mod:`readline` module now detects if Python is linked to libedit at runtime -on all platforms. Previously, the check was only done on macOS. diff --git a/Misc/NEWS.d/next/Library/2019-12-05-16-13-25.bpo-38529.yvQgx3.rst b/Misc/NEWS.d/next/Library/2019-12-05-16-13-25.bpo-38529.yvQgx3.rst deleted file mode 100644 index c688926b4a49b0..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-05-16-13-25.bpo-38529.yvQgx3.rst +++ /dev/null @@ -1,2 +0,0 @@ -Drop too noisy asyncio warning about deletion of a stream without explicit -``.close()`` call. diff --git a/Misc/NEWS.d/next/Library/2019-12-06-15-11-42.bpo-38986.bg6iZt.rst b/Misc/NEWS.d/next/Library/2019-12-06-15-11-42.bpo-38986.bg6iZt.rst deleted file mode 100644 index 777535299be17a..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-06-15-11-42.bpo-38986.bg6iZt.rst +++ /dev/null @@ -1,2 +0,0 @@ -Make repr of C accelerated TaskWakeupMethWrapper the same as of pure Python -version. diff --git a/Misc/NEWS.d/next/Library/2019-12-07-16-32-42.bpo-38979.q0sIHy.rst b/Misc/NEWS.d/next/Library/2019-12-07-16-32-42.bpo-38979.q0sIHy.rst deleted file mode 100644 index 6a91a12e4930a9..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-07-16-32-42.bpo-38979.q0sIHy.rst +++ /dev/null @@ -1 +0,0 @@ -Return class from ``ContextVar.__class_getitem__`` to simplify subclassing. diff --git a/Misc/NEWS.d/next/Library/2019-12-07-21-49-50.bpo-38698.HxoSym.rst b/Misc/NEWS.d/next/Library/2019-12-07-21-49-50.bpo-38698.HxoSym.rst deleted file mode 100644 index b930dea0fa7bc7..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-07-21-49-50.bpo-38698.HxoSym.rst +++ /dev/null @@ -1,3 +0,0 @@ -Add a new ``InvalidMessageID`` token to email parser to represent invalid -Message-ID headers. Also, add defects when there is remaining value after -parsing the header. diff --git a/Misc/NEWS.d/next/Library/2019-12-07-22-25-39.bpo-38708.rZTUfk.rst b/Misc/NEWS.d/next/Library/2019-12-07-22-25-39.bpo-38708.rZTUfk.rst deleted file mode 100644 index 23a0a46d1fea14..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-07-22-25-39.bpo-38708.rZTUfk.rst +++ /dev/null @@ -1 +0,0 @@ -Fix a potential IndexError in email parser when parsing an empty msg-id. diff --git a/Misc/NEWS.d/next/Library/2019-12-09-14-40-09.bpo-39006.v4VsPg.rst b/Misc/NEWS.d/next/Library/2019-12-09-14-40-09.bpo-39006.v4VsPg.rst deleted file mode 100644 index 8402845a5a0475..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-09-14-40-09.bpo-39006.v4VsPg.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix asyncio when the ssl module is missing: only check for ssl.SSLSocket -instance if the ssl module is available. diff --git a/Misc/NEWS.d/next/Security/2019-10-08-19-29-55.bpo-38418.QL7s0-.rst b/Misc/NEWS.d/next/Security/2019-10-08-19-29-55.bpo-38418.QL7s0-.rst deleted file mode 100644 index b42a00047b96c0..00000000000000 --- a/Misc/NEWS.d/next/Security/2019-10-08-19-29-55.bpo-38418.QL7s0-.rst +++ /dev/null @@ -1 +0,0 @@ -Fixes audit event for :func:`os.system` to be named ``os.system``. diff --git a/Misc/NEWS.d/next/Security/2019-11-14-16-13-23.bpo-38622.3DYkfb.rst b/Misc/NEWS.d/next/Security/2019-11-14-16-13-23.bpo-38622.3DYkfb.rst deleted file mode 100644 index 0373c14d0a1cd6..00000000000000 --- a/Misc/NEWS.d/next/Security/2019-11-14-16-13-23.bpo-38622.3DYkfb.rst +++ /dev/null @@ -1 +0,0 @@ -Add additional audit events for the :mod:`ctypes` module. diff --git a/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst b/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst deleted file mode 100644 index 1f45142d9f7437..00000000000000 --- a/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst +++ /dev/null @@ -1 +0,0 @@ -Fixes a ReDoS vulnerability in :mod:`http.cookiejar`. Patch by Ben Caller. diff --git a/Misc/NEWS.d/next/Security/2019-11-18-16-17-56.bpo-38722.x3mECW.rst b/Misc/NEWS.d/next/Security/2019-11-18-16-17-56.bpo-38722.x3mECW.rst deleted file mode 100644 index 0277d3e5689a2a..00000000000000 --- a/Misc/NEWS.d/next/Security/2019-11-18-16-17-56.bpo-38722.x3mECW.rst +++ /dev/null @@ -1,2 +0,0 @@ -:mod:`runpy` now uses :meth:`io.open_code` to open code files. -Patch by Jason Killen. diff --git a/Misc/NEWS.d/next/Security/2019-11-21-21-36-54.bpo-37228.yBZnFG.rst b/Misc/NEWS.d/next/Security/2019-11-21-21-36-54.bpo-37228.yBZnFG.rst deleted file mode 100644 index 0fafb63402e468..00000000000000 --- a/Misc/NEWS.d/next/Security/2019-11-21-21-36-54.bpo-37228.yBZnFG.rst +++ /dev/null @@ -1,6 +0,0 @@ -Due to significant security concerns, the *reuse_address* parameter of -:meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is -because of the behavior of ``SO_REUSEADDR`` in UDP. For more details, see the -documentation for ``loop.create_datagram_endpoint()``. -(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in -:issue:`37228`.) diff --git a/Misc/NEWS.d/next/Security/2019-12-01-22-44-40.bpo-38945.ztmNXc.rst b/Misc/NEWS.d/next/Security/2019-12-01-22-44-40.bpo-38945.ztmNXc.rst deleted file mode 100644 index 1bf6ed567b2412..00000000000000 --- a/Misc/NEWS.d/next/Security/2019-12-01-22-44-40.bpo-38945.ztmNXc.rst +++ /dev/null @@ -1 +0,0 @@ -Newline characters have been escaped when performing uu encoding to prevent them from overflowing into to content section of the encoded file. This prevents malicious or accidental modification of data during the decoding process. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst b/Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst deleted file mode 100644 index 43d3942fd0772e..00000000000000 --- a/Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst +++ /dev/null @@ -1,5 +0,0 @@ -Fix a race condition in test_asyncio.test_start_tls_server_1(). Previously, -there was a race condition between the test main() function which replaces the -protocol and the test ServerProto protocol which sends ANSWER once it gets -HELLO. Now, only the test main() function is responsible to send data, -ServerProto no longer sends data. diff --git a/Misc/NEWS.d/next/Tests/2019-11-04-02-54-16.bpo-38669.pazXZ8.rst b/Misc/NEWS.d/next/Tests/2019-11-04-02-54-16.bpo-38669.pazXZ8.rst deleted file mode 100644 index 5060ecf2dc5a46..00000000000000 --- a/Misc/NEWS.d/next/Tests/2019-11-04-02-54-16.bpo-38669.pazXZ8.rst +++ /dev/null @@ -1 +0,0 @@ -Raise :exc:`TypeError` when passing target as a string with :meth:`unittest.mock.patch.object`. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Tests/2019-11-20-16-08-19.bpo-38841.5F5Lbw.rst b/Misc/NEWS.d/next/Tests/2019-11-20-16-08-19.bpo-38841.5F5Lbw.rst deleted file mode 100644 index 1f4ae424bbcdee..00000000000000 --- a/Misc/NEWS.d/next/Tests/2019-11-20-16-08-19.bpo-38841.5F5Lbw.rst +++ /dev/null @@ -1,2 +0,0 @@ -Skip asyncio test_create_datagram_endpoint_existing_sock_unix on platforms -lacking a functional bind() for named unix domain sockets. diff --git a/Misc/NEWS.d/next/Tests/2019-11-21-09-11-06.bpo-38875.wSZJal.rst b/Misc/NEWS.d/next/Tests/2019-11-21-09-11-06.bpo-38875.wSZJal.rst deleted file mode 100644 index 3f6c86d32262cb..00000000000000 --- a/Misc/NEWS.d/next/Tests/2019-11-21-09-11-06.bpo-38875.wSZJal.rst +++ /dev/null @@ -1 +0,0 @@ -test_capi: trashcan tests now require the test "cpu" resource. diff --git a/Misc/NEWS.d/next/Tests/2019-12-04-17-08-55.bpo-38965.yqax3m.rst b/Misc/NEWS.d/next/Tests/2019-12-04-17-08-55.bpo-38965.yqax3m.rst deleted file mode 100644 index 517a1371eacd9a..00000000000000 --- a/Misc/NEWS.d/next/Tests/2019-12-04-17-08-55.bpo-38965.yqax3m.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix test_faulthandler on GCC 10. Use the "volatile" keyword in -``faulthandler._stack_overflow()`` to prevent tail call optimization on any -compiler, rather than relying on compiler specific pragma. diff --git a/Misc/NEWS.d/next/Tests/2019-12-08-15-11-06.bpo-38992.cVoHOZ.rst b/Misc/NEWS.d/next/Tests/2019-12-08-15-11-06.bpo-38992.cVoHOZ.rst deleted file mode 100644 index 815ae0f65c8732..00000000000000 --- a/Misc/NEWS.d/next/Tests/2019-12-08-15-11-06.bpo-38992.cVoHOZ.rst +++ /dev/null @@ -1 +0,0 @@ -Fix a test for :func:`math.fsum` that was failing due to constant folding. diff --git a/Misc/NEWS.d/next/Tests/2019-12-09-11-32-34.bpo-38547.Juw54e.rst b/Misc/NEWS.d/next/Tests/2019-12-09-11-32-34.bpo-38547.Juw54e.rst deleted file mode 100644 index 10f3cc08511ddb..00000000000000 --- a/Misc/NEWS.d/next/Tests/2019-12-09-11-32-34.bpo-38547.Juw54e.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix test_pty: if the process is the session leader, closing the master file -descriptor raises a SIGHUP signal: simply ignore SIGHUP when running the -tests. diff --git a/Misc/NEWS.d/next/Windows/2019-10-16-09-49-09.bpo-38492.Te1LxC.rst b/Misc/NEWS.d/next/Windows/2019-10-16-09-49-09.bpo-38492.Te1LxC.rst deleted file mode 100644 index 41fe695413f977..00000000000000 --- a/Misc/NEWS.d/next/Windows/2019-10-16-09-49-09.bpo-38492.Te1LxC.rst +++ /dev/null @@ -1 +0,0 @@ -Remove ``pythonw.exe`` dependency on the Microsoft C++ runtime. diff --git a/Misc/NEWS.d/next/Windows/2019-10-28-05-01-29.bpo-38519.dCkY66.rst b/Misc/NEWS.d/next/Windows/2019-10-28-05-01-29.bpo-38519.dCkY66.rst deleted file mode 100644 index 56d8bb2b34c101..00000000000000 --- a/Misc/NEWS.d/next/Windows/2019-10-28-05-01-29.bpo-38519.dCkY66.rst +++ /dev/null @@ -1,2 +0,0 @@ -Restores the internal C headers that were missing from the nuget.org and -Microsoft Store packages. diff --git a/Misc/NEWS.d/next/Windows/2019-10-28-10-32-43.bpo-38453.NwwatW.rst b/Misc/NEWS.d/next/Windows/2019-10-28-10-32-43.bpo-38453.NwwatW.rst deleted file mode 100644 index deacb03c6f01db..00000000000000 --- a/Misc/NEWS.d/next/Windows/2019-10-28-10-32-43.bpo-38453.NwwatW.rst +++ /dev/null @@ -1 +0,0 @@ -Ensure ntpath.realpath() correctly resolves relative paths. diff --git a/Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst b/Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst deleted file mode 100644 index e864290723e077..00000000000000 --- a/Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst +++ /dev/null @@ -1 +0,0 @@ -Fixes HTML Help shortcut when Windows is not installed to C drive diff --git a/Misc/NEWS.d/next/Windows/2019-11-14-08-57-50.bpo-33125.EN5MWS.rst b/Misc/NEWS.d/next/Windows/2019-11-14-08-57-50.bpo-33125.EN5MWS.rst deleted file mode 100644 index 0bc98c1a2fae77..00000000000000 --- a/Misc/NEWS.d/next/Windows/2019-11-14-08-57-50.bpo-33125.EN5MWS.rst +++ /dev/null @@ -1 +0,0 @@ -Add support for building and releasing Windows ARM64 packages. diff --git a/Misc/NEWS.d/next/macOS/2019-08-23-12-14-34.bpo-37931.goYgQj.rst b/Misc/NEWS.d/next/macOS/2019-08-23-12-14-34.bpo-37931.goYgQj.rst deleted file mode 100644 index 45b54e89cb89e6..00000000000000 --- a/Misc/NEWS.d/next/macOS/2019-08-23-12-14-34.bpo-37931.goYgQj.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fixed a crash on OSX dynamic builds that occurred when re-initializing the -posix module after a Py_Finalize if the environment had changed since the -previous `import posix`. Patch by Benoît Hudson. diff --git a/README.rst b/README.rst index 76d70b3e2d6d13..4064b511a87a16 100644 --- a/README.rst +++ b/README.rst @@ -1,5 +1,5 @@ -This is Python version 3.8.0 -============================ +This is Python version 3.8.1rc1 +=============================== .. image:: https://travis-ci.org/python/cpython.svg?branch=3.8 :alt: CPython build status on Travis CI From c93d68bbb986ee0879f5627223e0bd2bb91f63dd Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 9 Dec 2019 11:22:30 -0800 Subject: [PATCH 1001/2163] bpo-39008: Require Py_ssize_t for PySys_Audit formats rather than raise a deprecation warning (GH-17540) (cherry picked from commit b8cbe74c3498c617f0e73fd0cdc5c07f2c532092) Co-authored-by: Steve Dower --- Doc/c-api/sys.rst | 8 ++++++++ .../2019-12-09-10-38-51.bpo-39008.Rrp6f1.rst | 3 +++ Python/sysmodule.c | 2 +- 3 files changed, 12 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-12-09-10-38-51.bpo-39008.Rrp6f1.rst diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index eccb8a67e82a5d..c851ff66487d5c 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -320,10 +320,18 @@ accessible to C code. They all work with the current interpreter thread's arguments to this function will be consumed, using it may cause reference leaks.) + Note that ``#`` format characters should always be treated as + ``Py_ssize_t``, regardless of whether ``PY_SSIZE_T_CLEAN`` was defined. + :func:`sys.audit` performs the same function from Python code. .. versionadded:: 3.8 + .. versionchanged:: 3.8.2 + + Require ``Py_ssize_t`` for ``#`` format characters. Previously, an + unavoidable deprecation warning was raised. + .. c:function:: int PySys_AddAuditHook(Py_AuditHookFunction hook, void *userData) diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-12-09-10-38-51.bpo-39008.Rrp6f1.rst b/Misc/NEWS.d/next/Core and Builtins/2019-12-09-10-38-51.bpo-39008.Rrp6f1.rst new file mode 100644 index 00000000000000..35237ce2714a01 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2019-12-09-10-38-51.bpo-39008.Rrp6f1.rst @@ -0,0 +1,3 @@ +:c:func:`PySys_Audit` now requires ``Py_ssize_t`` to be used for size +arguments in the format string, regardless of whethen ``PY_SSIZE_T_CLEAN`` +was defined at include time. diff --git a/Python/sysmodule.c b/Python/sysmodule.c index 1255665d024c3d..da4b6e1a7806b0 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -159,7 +159,7 @@ PySys_Audit(const char *event, const char *argFormat, ...) if (argFormat && argFormat[0]) { va_list args; va_start(args, argFormat); - eventArgs = Py_VaBuildValue(argFormat, args); + eventArgs = _Py_VaBuildValue_SizeT(argFormat, args); va_end(args); if (eventArgs && !PyTuple_Check(eventArgs)) { PyObject *argTuple = PyTuple_Pack(1, eventArgs); From 0ac9aaeb9727aa60bef918b619116a729d3ea56d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 9 Dec 2019 11:36:25 -0800 Subject: [PATCH 1002/2163] bpo-39007: Add auditing events to functions in winreg (GH-17541) Also allows winreg.CloseKey() to accept same types as other functions. (cherry picked from commit ee17e3735634c5fe15a43f897707de8011618627) Co-authored-by: Steve Dower --- Doc/library/winreg.rst | 49 ++++++ Lib/test/audit-tests.py | 23 +++ Lib/test/test_audit.py | 14 ++ .../2019-12-09-10-40-34.bpo-39007.vtarxo.rst | 1 + PC/winreg.c | 158 +++++++++++++++--- 5 files changed, 218 insertions(+), 27 deletions(-) create mode 100644 Misc/NEWS.d/next/Windows/2019-12-09-10-40-34.bpo-39007.vtarxo.rst diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst index 5e810680b2643f..dccb7db27e90cc 100644 --- a/Doc/library/winreg.rst +++ b/Doc/library/winreg.rst @@ -53,6 +53,8 @@ This module offers the following functions: The return value is the handle of the opened key. If the function fails, an :exc:`OSError` exception is raised. + .. audit-event:: winreg.ConnectRegistry computer_name,key winreg.ConnectRegistry + .. versionchanged:: 3.3 See :ref:`above `. @@ -75,6 +77,10 @@ This module offers the following functions: The return value is the handle of the opened key. If the function fails, an :exc:`OSError` exception is raised. + .. audit-event:: winreg.CreateKey key,sub_key,access winreg.CreateKey + + .. audit-event:: winreg.OpenKey/result key winreg.CreateKey + .. versionchanged:: 3.3 See :ref:`above `. @@ -103,6 +109,10 @@ This module offers the following functions: The return value is the handle of the opened key. If the function fails, an :exc:`OSError` exception is raised. + .. audit-event:: winreg.CreateKey key,sub_key,access winreg.CreateKeyEx + + .. audit-event:: winreg.OpenKey/result key winreg.CreateKeyEx + .. versionadded:: 3.2 .. versionchanged:: 3.3 @@ -124,6 +134,8 @@ This module offers the following functions: If the method succeeds, the entire key, including all of its values, is removed. If the method fails, an :exc:`OSError` exception is raised. + .. audit-event:: winreg.DeleteKey key,sub_key,access winreg.DeleteKey + .. versionchanged:: 3.3 See :ref:`above `. @@ -158,6 +170,8 @@ This module offers the following functions: On unsupported Windows versions, :exc:`NotImplementedError` is raised. + .. audit-event:: winreg.DeleteKey key,sub_key,access winreg.DeleteKeyEx + .. versionadded:: 3.2 .. versionchanged:: 3.3 @@ -173,6 +187,8 @@ This module offers the following functions: *value* is a string that identifies the value to remove. + .. audit-event:: winreg.DeleteValue key,value winreg.DeleteValue + .. function:: EnumKey(key, index) @@ -187,6 +203,8 @@ This module offers the following functions: typically called repeatedly until an :exc:`OSError` exception is raised, indicating, no more values are available. + .. audit-event:: winreg.EnumKey key,index winreg.EnumKey + .. versionchanged:: 3.3 See :ref:`above `. @@ -220,6 +238,8 @@ This module offers the following functions: | | :meth:`SetValueEx`) | +-------+--------------------------------------------+ + .. audit-event:: winreg.EnumValue key,index winreg.EnumValue + .. versionchanged:: 3.3 See :ref:`above `. @@ -235,6 +255,8 @@ This module offers the following functions: >>> ExpandEnvironmentStrings('%windir%') 'C:\\Windows' + .. audit-event:: winreg.ExpandEnvironmentStrings str winreg.ExpandEnvironmentStrings + .. function:: FlushKey(key) @@ -279,6 +301,8 @@ This module offers the following functions: If *key* is a handle returned by :func:`ConnectRegistry`, then the path specified in *file_name* is relative to the remote computer. + .. audit-event:: winreg.LoadKey key,sub_key,file_name winreg.LoadKey + .. function:: OpenKey(key, sub_key, reserved=0, access=KEY_READ) OpenKeyEx(key, sub_key, reserved=0, access=KEY_READ) @@ -300,6 +324,10 @@ This module offers the following functions: If the function fails, :exc:`OSError` is raised. + .. audit-event:: winreg.OpenKey key,sub_key,access winreg.OpenKey + + .. audit-event:: winreg.OpenKey/result key winreg.OpenKey + .. versionchanged:: 3.2 Allow the use of named arguments. @@ -330,6 +358,8 @@ This module offers the following functions: | | nanoseconds since Jan 1, 1601. | +-------+---------------------------------------------+ + .. audit-event:: winreg.QueryInfoKey key winreg.QueryInfoKey + .. function:: QueryValue(key, sub_key) @@ -347,6 +377,8 @@ This module offers the following functions: underlying API call doesn't return the type, so always use :func:`QueryValueEx` if possible. + .. audit-event:: winreg.QueryValue key,sub_key,value_name winreg.QueryValue + .. function:: QueryValueEx(key, value_name) @@ -370,6 +402,8 @@ This module offers the following functions: | | :meth:`SetValueEx`) | +-------+-----------------------------------------+ + .. audit-event:: winreg.QueryValue key,sub_key,value_name winreg.QueryValueEx + .. function:: SaveKey(key, file_name) @@ -393,6 +427,8 @@ This module offers the following functions: This function passes ``NULL`` for *security_attributes* to the API. + .. audit-event:: winreg.SaveKey key,file_name winreg.SaveKey + .. function:: SetValue(key, sub_key, type, value) @@ -419,6 +455,8 @@ This module offers the following functions: The key identified by the *key* parameter must have been opened with :const:`KEY_SET_VALUE` access. + .. audit-event:: winreg.SetValue key,sub_key,type,value winreg.SetValue + .. function:: SetValueEx(key, value_name, reserved, type, value) @@ -447,6 +485,8 @@ This module offers the following functions: bytes) should be stored as files with the filenames stored in the configuration registry. This helps the registry perform efficiently. + .. audit-event:: winreg.SetValue key,sub_key,type,value winreg.SetValueEx + .. function:: DisableReflectionKey(key) @@ -463,6 +503,8 @@ This module offers the following functions: effect. Disabling reflection for a key does not affect reflection of any subkeys. + .. audit-event:: winreg.DisableReflectionKey key winreg.DisableReflectionKey + .. function:: EnableReflectionKey(key) @@ -476,6 +518,8 @@ This module offers the following functions: Restoring reflection for a key does not affect reflection of any subkeys. + .. audit-event:: winreg.EnableReflectionKey key winreg.EnableReflectionKey + .. function:: QueryReflectionKey(key) @@ -489,6 +533,8 @@ This module offers the following functions: Will generally raise :exc:`NotImplementedError` if executed on a 32-bit operating system. + .. audit-event:: winreg.QueryReflectionKey key winreg.QueryReflectionKey + .. _constants: @@ -741,6 +787,9 @@ integer handle, and also disconnect the Windows handle from the handle object. handle is not closed. You would call this function when you need the underlying Win32 handle to exist beyond the lifetime of the handle object. + .. audit-event:: winreg.PyHKEY.Detach key winreg.PyHKEY.Detach + + .. method:: PyHKEY.__enter__() PyHKEY.__exit__(\*exc_info) diff --git a/Lib/test/audit-tests.py b/Lib/test/audit-tests.py index ed08612c0417fd..33f320992bb23f 100644 --- a/Lib/test/audit-tests.py +++ b/Lib/test/audit-tests.py @@ -304,6 +304,29 @@ def hook(event, args): write_unraisable_exc(RuntimeError("nonfatal-error"), "for audit hook test", None) +def test_winreg(): + from winreg import OpenKey, EnumKey, CloseKey, HKEY_LOCAL_MACHINE + + def hook(event, args): + if not event.startswith("winreg."): + return + print(event, *args) + + sys.addaudithook(hook) + + k = OpenKey(HKEY_LOCAL_MACHINE, "Software") + EnumKey(k, 0) + try: + EnumKey(k, 10000) + except OSError: + pass + else: + raise RuntimeError("Expected EnumKey(HKLM, 10000) to fail") + + kv = k.Detach() + CloseKey(kv) + + if __name__ == "__main__": from test.libregrtest.setup import suppress_msvcrt_asserts diff --git a/Lib/test/test_audit.py b/Lib/test/test_audit.py index 31a08559273eec..73dd5c5b7db30c 100644 --- a/Lib/test/test_audit.py +++ b/Lib/test/test_audit.py @@ -104,6 +104,20 @@ def test_unraisablehook(self): "RuntimeError('nonfatal-error') Exception ignored for audit hook test", ) + def test_winreg(self): + support.import_module("winreg") + returncode, events, stderr = self.run_python("test_winreg") + if returncode: + self.fail(stderr) + + self.assertEqual(events[0][0], "winreg.OpenKey") + self.assertEqual(events[1][0], "winreg.OpenKey/result") + expected = events[1][2] + self.assertTrue(expected) + self.assertSequenceEqual(["winreg.EnumKey", " ", f"{expected} 0"], events[2]) + self.assertSequenceEqual(["winreg.EnumKey", " ", f"{expected} 10000"], events[3]) + self.assertSequenceEqual(["winreg.PyHKEY.Detach", " ", expected], events[4]) + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Windows/2019-12-09-10-40-34.bpo-39007.vtarxo.rst b/Misc/NEWS.d/next/Windows/2019-12-09-10-40-34.bpo-39007.vtarxo.rst new file mode 100644 index 00000000000000..f2f72f9dad3f6e --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2019-12-09-10-40-34.bpo-39007.vtarxo.rst @@ -0,0 +1 @@ +Add auditing events to functions in :mod:`winreg`. diff --git a/PC/winreg.c b/PC/winreg.c index 72a7c380beefe1..5dff7deadf767a 100644 --- a/PC/winreg.c +++ b/PC/winreg.c @@ -293,6 +293,9 @@ winreg_HKEYType_Detach_impl(PyHKEYObject *self) /*[clinic end generated code: output=dda5a9e1a01ae78f input=dd2cc09e6c6ba833]*/ { void* ret; + if (PySys_Audit("winreg.PyHKEY.Detach", "n", (Py_ssize_t)self->hkey) < 0) { + return NULL; + } ret = (void*)self->hkey; self->hkey = 0; return PyLong_FromVoidPtr(ret); @@ -397,15 +400,15 @@ BOOL PyHKEY_Close(PyObject *ob_handle) { LONG rc; - PyHKEYObject *key; + HKEY key; - if (!PyHKEY_Check(ob_handle)) { - PyErr_SetString(PyExc_TypeError, "bad operand type"); + if (!PyHKEY_AsHKEY(ob_handle, &key, TRUE)) { return FALSE; } - key = (PyHKEYObject *)ob_handle; - rc = key->hkey ? RegCloseKey((HKEY)key->hkey) : ERROR_SUCCESS; - key->hkey = 0; + if (PyHKEY_Check(ob_handle)) { + ((PyHKEYObject*)ob_handle)->hkey = 0; + } + rc = key ? RegCloseKey(key) : ERROR_SUCCESS; if (rc != ERROR_SUCCESS) PyErr_SetFromWindowsErrWithFunction(rc, "RegCloseKey"); return rc == ERROR_SUCCESS; @@ -841,6 +844,10 @@ winreg_ConnectRegistry_impl(PyObject *module, { HKEY retKey; long rc; + if (PySys_Audit("winreg.ConnectRegistry", "un", + computer_name, (Py_ssize_t)key) < 0) { + return NULL; + } Py_BEGIN_ALLOW_THREADS rc = RegConnectRegistryW(computer_name, key, &retKey); Py_END_ALLOW_THREADS @@ -878,11 +885,20 @@ winreg_CreateKey_impl(PyObject *module, HKEY key, const Py_UNICODE *sub_key) HKEY retKey; long rc; + if (PySys_Audit("winreg.CreateKey", "nun", + (Py_ssize_t)key, sub_key, + (Py_ssize_t)KEY_WRITE) < 0) { + return NULL; + } rc = RegCreateKeyW(key, sub_key, &retKey); if (rc != ERROR_SUCCESS) { PyErr_SetFromWindowsErrWithFunction(rc, "CreateKey"); return NULL; } + if (PySys_Audit("winreg.OpenKey/result", "n", + (Py_ssize_t)retKey) < 0) { + return NULL; + } return retKey; } @@ -919,12 +935,21 @@ winreg_CreateKeyEx_impl(PyObject *module, HKEY key, HKEY retKey; long rc; + if (PySys_Audit("winreg.CreateKey", "nun", + (Py_ssize_t)key, sub_key, + (Py_ssize_t)access) < 0) { + return NULL; + } rc = RegCreateKeyExW(key, sub_key, reserved, NULL, 0, access, NULL, &retKey, NULL); if (rc != ERROR_SUCCESS) { PyErr_SetFromWindowsErrWithFunction(rc, "CreateKeyEx"); return NULL; } + if (PySys_Audit("winreg.OpenKey/result", "n", + (Py_ssize_t)retKey) < 0) { + return NULL; + } return retKey; } @@ -951,6 +976,11 @@ winreg_DeleteKey_impl(PyObject *module, HKEY key, const Py_UNICODE *sub_key) /*[clinic end generated code: output=d2652a84f70e0862 input=b31d225b935e4211]*/ { long rc; + if (PySys_Audit("winreg.DeleteKey", "nun", + (Py_ssize_t)key, sub_key, + (Py_ssize_t)0) < 0) { + return NULL; + } rc = RegDeleteKeyW(key, sub_key ); if (rc != ERROR_SUCCESS) return PyErr_SetFromWindowsErrWithFunction(rc, "RegDeleteKey"); @@ -992,13 +1022,17 @@ winreg_DeleteKeyEx_impl(PyObject *module, HKEY key, RDKEFunc pfn = NULL; long rc; + if (PySys_Audit("winreg.DeleteKey", "nun", + (Py_ssize_t)key, sub_key, + (Py_ssize_t)access) < 0) { + return NULL; + } /* Only available on 64bit platforms, so we must load it dynamically. */ Py_BEGIN_ALLOW_THREADS hMod = GetModuleHandleW(L"advapi32.dll"); if (hMod) - pfn = (RDKEFunc)GetProcAddress(hMod, - "RegDeleteKeyExW"); + pfn = (RDKEFunc)GetProcAddress(hMod, "RegDeleteKeyExW"); Py_END_ALLOW_THREADS if (!pfn) { PyErr_SetString(PyExc_NotImplementedError, @@ -1031,6 +1065,10 @@ winreg_DeleteValue_impl(PyObject *module, HKEY key, const Py_UNICODE *value) /*[clinic end generated code: output=56fa9d21f3a54371 input=a78d3407a4197b21]*/ { long rc; + if (PySys_Audit("winreg.DeleteValue", "nu", + (Py_ssize_t)key, value) < 0) { + return NULL; + } Py_BEGIN_ALLOW_THREADS rc = RegDeleteValueW(key, value); Py_END_ALLOW_THREADS @@ -1063,6 +1101,10 @@ winreg_EnumKey_impl(PyObject *module, HKEY key, int index) long rc; PyObject *retStr; + if (PySys_Audit("winreg.EnumKey", "ni", + (Py_ssize_t)key, index) < 0) { + return NULL; + } /* The Windows docs claim that the max key name length is 255 * characters, plus a terminating nul character. However, * empirical testing demonstrates that it is possible to @@ -1121,6 +1163,10 @@ winreg_EnumValue_impl(PyObject *module, HKEY key, int index) PyObject *obData; PyObject *retVal; + if (PySys_Audit("winreg.EnumValue", "ni", + (Py_ssize_t)key, index) < 0) { + return NULL; + } if ((rc = RegQueryInfoKeyW(key, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &retValueSize, &retDataSize, NULL, NULL)) @@ -1204,6 +1250,11 @@ winreg_ExpandEnvironmentStrings_impl(PyObject *module, DWORD rc; PyObject *o; + if (PySys_Audit("winreg.ExpandEnvironmentStrings", "u", + string) < 0) { + return NULL; + } + retValueSize = ExpandEnvironmentStringsW(string, retValue, 0); if (retValueSize == 0) { return PyErr_SetFromWindowsErrWithFunction(retValueSize, @@ -1295,6 +1346,10 @@ winreg_LoadKey_impl(PyObject *module, HKEY key, const Py_UNICODE *sub_key, { long rc; + if (PySys_Audit("winreg.LoadKey", "nuu", + (Py_ssize_t)key, sub_key, file_name) < 0) { + return NULL; + } Py_BEGIN_ALLOW_THREADS rc = RegLoadKeyW(key, sub_key, file_name ); Py_END_ALLOW_THREADS @@ -1330,6 +1385,11 @@ winreg_OpenKey_impl(PyObject *module, HKEY key, const Py_UNICODE *sub_key, HKEY retKey; long rc; + if (PySys_Audit("winreg.OpenKey", "nun", + (Py_ssize_t)key, sub_key, + (Py_ssize_t)access) < 0) { + return NULL; + } Py_BEGIN_ALLOW_THREADS rc = RegOpenKeyExW(key, sub_key, reserved, access, &retKey); Py_END_ALLOW_THREADS @@ -1337,6 +1397,10 @@ winreg_OpenKey_impl(PyObject *module, HKEY key, const Py_UNICODE *sub_key, PyErr_SetFromWindowsErrWithFunction(rc, "RegOpenKeyEx"); return NULL; } + if (PySys_Audit("winreg.OpenKey/result", "n", + (Py_ssize_t)retKey) < 0) { + return NULL; + } return retKey; } @@ -1377,25 +1441,30 @@ static PyObject * winreg_QueryInfoKey_impl(PyObject *module, HKEY key) /*[clinic end generated code: output=dc657b8356a4f438 input=c3593802390cde1f]*/ { - long rc; - DWORD nSubKeys, nValues; - FILETIME ft; - LARGE_INTEGER li; - PyObject *l; - PyObject *ret; - - if ((rc = RegQueryInfoKey(key, NULL, NULL, 0, &nSubKeys, NULL, NULL, - &nValues, NULL, NULL, NULL, &ft)) - != ERROR_SUCCESS) - return PyErr_SetFromWindowsErrWithFunction(rc, "RegQueryInfoKey"); - li.LowPart = ft.dwLowDateTime; - li.HighPart = ft.dwHighDateTime; - l = PyLong_FromLongLong(li.QuadPart); - if (l == NULL) - return NULL; - ret = Py_BuildValue("iiO", nSubKeys, nValues, l); - Py_DECREF(l); - return ret; + long rc; + DWORD nSubKeys, nValues; + FILETIME ft; + LARGE_INTEGER li; + PyObject *l; + PyObject *ret; + + if (PySys_Audit("winreg.QueryInfoKey", "n", (Py_ssize_t)key) < 0) { + return NULL; + } + if ((rc = RegQueryInfoKey(key, NULL, NULL, 0, &nSubKeys, NULL, NULL, + &nValues, NULL, NULL, NULL, &ft)) + != ERROR_SUCCESS) { + return PyErr_SetFromWindowsErrWithFunction(rc, "RegQueryInfoKey"); + } + li.LowPart = ft.dwLowDateTime; + li.HighPart = ft.dwHighDateTime; + l = PyLong_FromLongLong(li.QuadPart); + if (l == NULL) { + return NULL; + } + ret = Py_BuildValue("iiO", nSubKeys, nValues, l); + Py_DECREF(l); + return ret; } /*[clinic input] @@ -1430,6 +1499,10 @@ winreg_QueryValue_impl(PyObject *module, HKEY key, const Py_UNICODE *sub_key) DWORD retSize = 0; wchar_t *tmp; + if (PySys_Audit("winreg.QueryValue", "nuu", + (Py_ssize_t)key, sub_key, NULL) < 0) { + return NULL; + } rc = RegQueryValueW(key, sub_key, NULL, &retSize); if (rc == ERROR_MORE_DATA) retSize = 256; @@ -1497,6 +1570,10 @@ winreg_QueryValueEx_impl(PyObject *module, HKEY key, const Py_UNICODE *name) PyObject *obData; PyObject *result; + if (PySys_Audit("winreg.QueryValue", "nuu", + (Py_ssize_t)key, NULL, name) < 0) { + return NULL; + } rc = RegQueryValueExW(key, name, NULL, NULL, NULL, &bufSize); if (rc == ERROR_MORE_DATA) bufSize = 256; @@ -1570,6 +1647,10 @@ winreg_SaveKey_impl(PyObject *module, HKEY key, const Py_UNICODE *file_name) if (!PyWinObject_AsSECURITY_ATTRIBUTES(obSA, &pSA, TRUE)) return NULL; */ + if (PySys_Audit("winreg.SaveKey", "nu", + (Py_ssize_t)key, file_name) < 0) { + return NULL; + } Py_BEGIN_ALLOW_THREADS rc = RegSaveKeyW(key, file_name, pSA ); Py_END_ALLOW_THREADS @@ -1622,6 +1703,12 @@ winreg_SetValue_impl(PyObject *module, HKEY key, const Py_UNICODE *sub_key, return NULL; } + if (PySys_Audit("winreg.SetValue", "nunu#", + (Py_ssize_t)key, sub_key, (Py_ssize_t)type, + value, value_length) < 0) { + return NULL; + } + Py_BEGIN_ALLOW_THREADS rc = RegSetValueW(key, sub_key, REG_SZ, value, (DWORD)(value_length + 1)); Py_END_ALLOW_THREADS @@ -1692,6 +1779,11 @@ winreg_SetValueEx_impl(PyObject *module, HKEY key, "Could not convert the data to the specified type."); return NULL; } + if (PySys_Audit("winreg.SetValue", "nunO", + (Py_ssize_t)key, value_name, (Py_ssize_t)type, + value) < 0) { + return NULL; + } Py_BEGIN_ALLOW_THREADS rc = RegSetValueExW(key, value_name, 0, type, data, len); Py_END_ALLOW_THREADS @@ -1727,6 +1819,10 @@ winreg_DisableReflectionKey_impl(PyObject *module, HKEY key) RDRKFunc pfn = NULL; LONG rc; + if (PySys_Audit("winreg.DisableReflectionKey", "n", (Py_ssize_t)key) < 0) { + return NULL; + } + /* Only available on 64bit platforms, so we must load it dynamically.*/ Py_BEGIN_ALLOW_THREADS @@ -1772,6 +1868,10 @@ winreg_EnableReflectionKey_impl(PyObject *module, HKEY key) RERKFunc pfn = NULL; LONG rc; + if (PySys_Audit("winreg.EnableReflectionKey", "n", (Py_ssize_t)key) < 0) { + return NULL; + } + /* Only available on 64bit platforms, so we must load it dynamically.*/ Py_BEGIN_ALLOW_THREADS @@ -1816,6 +1916,10 @@ winreg_QueryReflectionKey_impl(PyObject *module, HKEY key) BOOL result; LONG rc; + if (PySys_Audit("winreg.QueryReflectionKey", "n", (Py_ssize_t)key) < 0) { + return NULL; + } + /* Only available on 64bit platforms, so we must load it dynamically.*/ Py_BEGIN_ALLOW_THREADS From 2b2c7bf231aadc0dd985e02549b1a8b93334047a Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 9 Dec 2019 15:38:33 -0800 Subject: [PATCH 1003/2163] bpo-38944: Escape key now closes IDLE completion windows. (GH-17419) (cherry picked from commit 232689b40d8fcbbac27c8705607ff482ea5b46f8) Co-authored-by: JohnnyNajera <58344607+JohnnyNajera@users.noreply.github.com> --- Lib/idlelib/NEWS.txt | 3 +++ Lib/idlelib/autocomplete_w.py | 2 +- Misc/NEWS.d/next/IDLE/2019-11-30-12-10-36.bpo-38944._3xjKG.rst | 1 + 3 files changed, 5 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/IDLE/2019-11-30-12-10-36.bpo-38944._3xjKG.rst diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index ed5cd4dc03b31b..900ec5b33d30de 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,9 @@ Released on 2019-12-16? ====================================== +bpo-38944: Excape key now closes IDLE completion windows. Patch by +Johnny Najera. + bpo-38862: 'Strip Trailing Whitespace' on the Format menu removes extra newlines at the end of non-shell files. diff --git a/Lib/idlelib/autocomplete_w.py b/Lib/idlelib/autocomplete_w.py index 5035e067392e5f..f20b6330997886 100644 --- a/Lib/idlelib/autocomplete_w.py +++ b/Lib/idlelib/autocomplete_w.py @@ -17,7 +17,7 @@ # before the default specific IDLE function KEYPRESS_SEQUENCES = ("", "", "", "", "", "", "", "", - "", "") + "", "", "") KEYRELEASE_VIRTUAL_EVENT_NAME = "<>" KEYRELEASE_SEQUENCE = "" LISTUPDATE_SEQUENCE = "" diff --git a/Misc/NEWS.d/next/IDLE/2019-11-30-12-10-36.bpo-38944._3xjKG.rst b/Misc/NEWS.d/next/IDLE/2019-11-30-12-10-36.bpo-38944._3xjKG.rst new file mode 100644 index 00000000000000..38084fafd2f561 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2019-11-30-12-10-36.bpo-38944._3xjKG.rst @@ -0,0 +1 @@ +Excape key now closes IDLE completion windows. Patch by Johnny Najera. From 34d5d5e096ee804e94199bf242469cdf9bbc3316 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 9 Dec 2019 16:48:20 -0800 Subject: [PATCH 1004/2163] bpo-38943: Fix IDLE autocomplete window not always appearing (GH-17416) This has happened on some versions of Ubuntu. (cherry picked from commit bbc4162bafe018f07bab0b624b37974cc33daad9) Co-authored-by: JohnnyNajera <58344607+JohnnyNajera@users.noreply.github.com> --- Lib/idlelib/NEWS.txt | 3 +++ Lib/idlelib/autocomplete_w.py | 1 + Misc/NEWS.d/next/IDLE/2019-11-29-23-44-11.bpo-38943.8pUKKs.rst | 2 ++ 3 files changed, 6 insertions(+) create mode 100644 Misc/NEWS.d/next/IDLE/2019-11-29-23-44-11.bpo-38943.8pUKKs.rst diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 900ec5b33d30de..e829bc910ea877 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,9 @@ Released on 2019-12-16? ====================================== +bpo-38943: Fix autocomplete windows not always appearing on some +systems. Patch by Johnny Najera. + bpo-38944: Excape key now closes IDLE completion windows. Patch by Johnny Najera. diff --git a/Lib/idlelib/autocomplete_w.py b/Lib/idlelib/autocomplete_w.py index f20b6330997886..0643c092c6e548 100644 --- a/Lib/idlelib/autocomplete_w.py +++ b/Lib/idlelib/autocomplete_w.py @@ -257,6 +257,7 @@ def winconfig_event(self, event): # place acw above current line new_y -= acw_height acw.wm_geometry("+%d+%d" % (new_x, new_y)) + acw.update_idletasks() if platform.system().startswith('Windows'): # See issue 15786. When on Windows platform, Tk will misbehave diff --git a/Misc/NEWS.d/next/IDLE/2019-11-29-23-44-11.bpo-38943.8pUKKs.rst b/Misc/NEWS.d/next/IDLE/2019-11-29-23-44-11.bpo-38943.8pUKKs.rst new file mode 100644 index 00000000000000..5c9323e2467876 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2019-11-29-23-44-11.bpo-38943.8pUKKs.rst @@ -0,0 +1,2 @@ +Fix IDLE autocomplete windows not always appearing on some systems. +Patch by Johnny Najera. From 859767d58ea8e34313f00fc102a810efff285941 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 9 Dec 2019 17:35:25 -0800 Subject: [PATCH 1005/2163] bpo-39002: Fix simple typo: tranlation -> translation (GH-17517) (GH-17538) (cherry picked from commit c18b805ac6a2d22176240ca93982fa1fb6559ec7) Co-authored-by: Tim Gates --- Lib/test/test_statistics.py | 4 ++-- Misc/ACKS | 1 + .../Documentation/2019-12-09-10-12-00.bpo-39002.AfgvfO.rst | 1 + 3 files changed, 4 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Documentation/2019-12-09-10-12-00.bpo-39002.AfgvfO.rst diff --git a/Lib/test/test_statistics.py b/Lib/test/test_statistics.py index bebd9b5d6f5026..a9a427bc8d972a 100644 --- a/Lib/test/test_statistics.py +++ b/Lib/test/test_statistics.py @@ -2192,7 +2192,7 @@ def test_specific_cases(self): quantiles(padded_data, n=n, method='inclusive'), (n, data), ) - # Invariant under tranlation and scaling + # Invariant under translation and scaling def f(x): return 3.5 * x - 1234.675 exp = list(map(f, expected)) @@ -2232,7 +2232,7 @@ def test_specific_cases_inclusive(self): result = quantiles(map(datatype, data), n=n, method="inclusive") self.assertTrue(all(type(x) == datatype) for x in result) self.assertEqual(result, list(map(datatype, expected))) - # Invariant under tranlation and scaling + # Invariant under translation and scaling def f(x): return 3.5 * x - 1234.675 exp = list(map(f, expected)) diff --git a/Misc/ACKS b/Misc/ACKS index 62c5928c508ff9..a17db456576318 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1885,3 +1885,4 @@ Batuhan Taskaya Aleksandr Balezin Robert Leenders Ngalim Siregar +Tim Gates diff --git a/Misc/NEWS.d/next/Documentation/2019-12-09-10-12-00.bpo-39002.AfgvfO.rst b/Misc/NEWS.d/next/Documentation/2019-12-09-10-12-00.bpo-39002.AfgvfO.rst new file mode 100644 index 00000000000000..a6dfa22eb7a169 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2019-12-09-10-12-00.bpo-39002.AfgvfO.rst @@ -0,0 +1 @@ +Fix simple typo in Lib/test/test_statistics.py. From 4ce626eeff8e3fce3cf3d9692a9c6e3ed013442a Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 9 Dec 2019 21:28:10 -0800 Subject: [PATCH 1006/2163] Fix Windows release builds (GH-17550) (cherry picked from commit abdeb57a212556b4cd3568cca7d316d71a5b8cf0) Co-authored-by: Steve Dower --- .azure-pipelines/windows-release/build-steps.yml | 2 +- .azure-pipelines/windows-release/stage-build.yml | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/.azure-pipelines/windows-release/build-steps.yml b/.azure-pipelines/windows-release/build-steps.yml index e2b6683f46e3c3..5ca2016d65f9e1 100644 --- a/.azure-pipelines/windows-release/build-steps.yml +++ b/.azure-pipelines/windows-release/build-steps.yml @@ -43,7 +43,7 @@ steps: - powershell: | $env:SigningCertificate = $null - python PC\layout -vv -b "$(Build.BinariesDirectory)\bin" -t "$(Build.BinariesDirectory)\catalog" --catalog "${env:CAT}.cdf" --preset-default --arch $(Arch) + $(_HostPython) PC\layout -vv -b "$(Build.BinariesDirectory)\bin" -t "$(Build.BinariesDirectory)\catalog" --catalog "${env:CAT}.cdf" --preset-default --arch $(Arch) makecat "${env:CAT}.cdf" del "${env:CAT}.cdf" if (-not (Test-Path "${env:CAT}.cat")) { diff --git a/.azure-pipelines/windows-release/stage-build.yml b/.azure-pipelines/windows-release/stage-build.yml index 60d72b282d1e51..9391a91e30b5e6 100644 --- a/.azure-pipelines/windows-release/stage-build.yml +++ b/.azure-pipelines/windows-release/stage-build.yml @@ -57,26 +57,31 @@ jobs: Arch: win32 Platform: x86 Configuration: Release + _HostPython: .\python win32_d: Name: win32_d Arch: win32 Platform: x86 Configuration: Debug + _HostPython: .\python amd64_d: Name: amd64_d Arch: amd64 Platform: x64 Configuration: Debug + _HostPython: .\python arm64: Name: arm64 Arch: arm64 Platform: ARM64 Configuration: Release + _HostPython: python arm64_d: Name: arm64_d Arch: arm64 Platform: ARM64 Configuration: Debug + _HostPython: python steps: - template: ./build-steps.yml @@ -98,6 +103,7 @@ jobs: Arch: amd64 Platform: x64 Configuration: Release + _HostPython: .\python steps: - template: ./build-steps.yml @@ -123,6 +129,7 @@ jobs: Arch: amd64 Platform: x64 Configuration: Release + _HostPython: .\python steps: - template: ./build-steps.yml From 7c543f4023e1ba25c53ba3b39ba9bc77c9d9d0b1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Tue, 10 Dec 2019 09:13:16 +0100 Subject: [PATCH 1007/2163] Post 3.8.1rc1 --- Include/patchlevel.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h index b3a2ecb9e7526c..cbffcd13a858e4 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 1 /* Version as a string */ -#define PY_VERSION "3.8.1rc1" +#define PY_VERSION "3.8.1rc1+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. From 3ac65a7161ddf492050302e0c7d67d1d81cb4930 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Date: Tue, 10 Dec 2019 16:11:33 +0000 Subject: [PATCH 1008/2163] [3.8] Minor fixes to the NEWS entries (GH-17557) Automerge-Triggered-By: @pablogsal --- Misc/NEWS.d/3.8.1rc1.rst | 2 +- .../next/Documentation/2019-12-09-10-12-00.bpo-39002.AfgvfO.rst | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) delete mode 100644 Misc/NEWS.d/next/Documentation/2019-12-09-10-12-00.bpo-39002.AfgvfO.rst diff --git a/Misc/NEWS.d/3.8.1rc1.rst b/Misc/NEWS.d/3.8.1rc1.rst index 958a8b7b800251..351996433c4cff 100644 --- a/Misc/NEWS.d/3.8.1rc1.rst +++ b/Misc/NEWS.d/3.8.1rc1.rst @@ -753,7 +753,7 @@ Do not try to compile IDLE shell or output windows .. nonce: oOGVdo .. section: C API -Reëxport some function compatibility wrappers for macros in ``pythonrun.h``. +Re-export some function compatibility wrappers for macros in ``pythonrun.h``. .. diff --git a/Misc/NEWS.d/next/Documentation/2019-12-09-10-12-00.bpo-39002.AfgvfO.rst b/Misc/NEWS.d/next/Documentation/2019-12-09-10-12-00.bpo-39002.AfgvfO.rst deleted file mode 100644 index a6dfa22eb7a169..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2019-12-09-10-12-00.bpo-39002.AfgvfO.rst +++ /dev/null @@ -1 +0,0 @@ -Fix simple typo in Lib/test/test_statistics.py. From 00e2fe4f214eadd5714751968e158c78a8a3f04b Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 10 Dec 2019 15:40:02 -0800 Subject: [PATCH 1009/2163] bpo-39012: Fix RC version suffix for nuget release files (GH-17564) (cherry picked from commit d0802d07d2c864b95480a9b24c7cc050e19189d5) Co-authored-by: Steve Dower --- PC/layout/support/constants.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PC/layout/support/constants.py b/PC/layout/support/constants.py index a8647631e9b4cd..6cf0fe1d34c4ac 100644 --- a/PC/layout/support/constants.py +++ b/PC/layout/support/constants.py @@ -20,7 +20,7 @@ def _unpack_hexversion(): def _get_suffix(field4): - name = {0xA0: "a", 0xB0: "b", 0xC0: "c"}.get(field4 & 0xF0, "") + name = {0xA0: "a", 0xB0: "b", 0xC0: "rc"}.get(field4 & 0xF0, "") if name: serial = field4 & 0x0F return f"{name}{serial}" From b738237d6792acba85b1f6e6c8993a812c7fd815 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 10 Dec 2019 17:47:06 -0800 Subject: [PATCH 1010/2163] bpo-39022, bpo-38594: Sync with importlib_metadata 1.3 (GH-17568) (GH-17569) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * bpo-39022, bpo-38594: Sync with importlib_metadata 1.3 including improved docs for custom finders and better serialization support in EntryPoints. * 📜🤖 Added by blurb_it. * Correct module reference (cherry picked from commit b7a0109cd2bafaa21a4d50aad307e901c68f9156) Co-authored-by: Jason R. Coombs --- Doc/library/importlib.metadata.rst | 12 +++---- Lib/importlib/metadata.py | 23 ++++++++++++- Lib/test/test_importlib/test_main.py | 33 +++++++++++++++++++ .../2019-12-10-23-34-48.bpo-39022.QDtIxI.rst | 1 + 4 files changed, 62 insertions(+), 7 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-10-23-34-48.bpo-39022.QDtIxI.rst diff --git a/Doc/library/importlib.metadata.rst b/Doc/library/importlib.metadata.rst index 4a4a8f7dfa0f87..dc6b66ca384d04 100644 --- a/Doc/library/importlib.metadata.rst +++ b/Doc/library/importlib.metadata.rst @@ -216,9 +216,9 @@ system `finders`_. To find a distribution package's metadata, ``importlib.metadata`` queries the list of `meta path finders`_ on `sys.meta_path`_. -By default ``importlib.metadata`` installs a finder for distribution packages -found on the file system. This finder doesn't actually find any *packages*, -but it can find the packages' metadata. +The default ``PathFinder`` for Python includes a hook that calls into +``importlib.metadata.MetadataPathFinder`` for finding distributions +loaded from typical file-system-based paths. The abstract class :py:class:`importlib.abc.MetaPathFinder` defines the interface expected of finders by Python's import system. @@ -239,9 +239,9 @@ properties indicating the path to search and names to match and may supply other relevant context. What this means in practice is that to support finding distribution package -metadata in locations other than the file system, you should derive from -``Distribution`` and implement the ``load_metadata()`` method. Then from -your finder, return instances of this derived ``Distribution`` in the +metadata in locations other than the file system, subclass +``Distribution`` and implement the abstract methods. Then from +a custom finder, return instances of this derived ``Distribution`` in the ``find_distributions()`` method. diff --git a/Lib/importlib/metadata.py b/Lib/importlib/metadata.py index 8cb45ec1ef3a29..53f9fb59346684 100644 --- a/Lib/importlib/metadata.py +++ b/Lib/importlib/metadata.py @@ -37,7 +37,8 @@ class PackageNotFoundError(ModuleNotFoundError): """The package was not found.""" -class EntryPoint(collections.namedtuple('EntryPointBase', 'name value group')): +class EntryPoint( + collections.namedtuple('EntryPointBase', 'name value group')): """An entry point as defined by Python packaging conventions. See `the packaging docs on entry points @@ -107,6 +108,12 @@ def __iter__(self): """ return iter((self.name, self)) + def __reduce__(self): + return ( + self.__class__, + (self.name, self.value, self.group), + ) + class PackagePath(pathlib.PurePosixPath): """A reference to a path in a package""" @@ -334,10 +341,21 @@ class DistributionFinder(MetaPathFinder): """ class Context: + """ + Keyword arguments presented by the caller to + ``distributions()`` or ``Distribution.discover()`` + to narrow the scope of a search for distributions + in all DistributionFinders. + + Each DistributionFinder may expect any parameters + and should attempt to honor the canonical + parameters defined below when appropriate. + """ name = None """ Specific name for which a distribution finder should match. + A name of ``None`` matches all distributions. """ def __init__(self, **kwargs): @@ -347,6 +365,9 @@ def __init__(self, **kwargs): def path(self): """ The path that a distribution finder should search. + + Typically refers to Python package paths and defaults + to ``sys.path``. """ return vars(self).get('path', sys.path) diff --git a/Lib/test/test_importlib/test_main.py b/Lib/test/test_importlib/test_main.py index 4d5b1273d9d103..c5f1dbbae325ed 100644 --- a/Lib/test/test_importlib/test_main.py +++ b/Lib/test/test_importlib/test_main.py @@ -1,6 +1,8 @@ # coding: utf-8 import re +import json +import pickle import textwrap import unittest import importlib.metadata @@ -181,3 +183,34 @@ def test_egg(self): with self.add_sys_path(egg): with self.assertRaises(PackageNotFoundError): version('foo') + + +class TestEntryPoints(unittest.TestCase): + def __init__(self, *args): + super(TestEntryPoints, self).__init__(*args) + self.ep = importlib.metadata.EntryPoint('name', 'value', 'group') + + def test_entry_point_pickleable(self): + revived = pickle.loads(pickle.dumps(self.ep)) + assert revived == self.ep + + def test_immutable(self): + """EntryPoints should be immutable""" + with self.assertRaises(AttributeError): + self.ep.name = 'badactor' + + def test_repr(self): + assert 'EntryPoint' in repr(self.ep) + assert 'name=' in repr(self.ep) + assert "'name'" in repr(self.ep) + + def test_hashable(self): + """EntryPoints should be hashable""" + hash(self.ep) + + def test_json_dump(self): + """ + json should not expect to be able to dump an EntryPoint + """ + with self.assertRaises(Exception): + json.dumps(self.ep) diff --git a/Misc/NEWS.d/next/Library/2019-12-10-23-34-48.bpo-39022.QDtIxI.rst b/Misc/NEWS.d/next/Library/2019-12-10-23-34-48.bpo-39022.QDtIxI.rst new file mode 100644 index 00000000000000..4af21be6074269 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-10-23-34-48.bpo-39022.QDtIxI.rst @@ -0,0 +1 @@ +Update importliib.metadata to include improvements from importlib_metadata 1.3 including better serialization of EntryPoints and improved documentation for custom finders. \ No newline at end of file From 3b18b17efcee6f968cf85c543254b3611311e9f4 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 13 Dec 2019 08:21:54 -0800 Subject: [PATCH 1011/2163] bpo-39031: Include elif keyword when producing lineno/col-offset info for if_stmt (GH-17582) (GH-17589) When parsing an "elif" node, lineno and col_offset of the node now point to the "elif" keyword and not to its condition, making it consistent with the "if" node. https://bugs.python.org/issue39031 Automerge-Triggered-By: @pablogsal (cherry picked from commit 025a602af7ee284d8db6955c26016f3f27d35536) Co-authored-by: Lysandros Nikolaou --- Lib/test/test_ast.py | 9 +++++++++ .../2019-12-12-21-05-43.bpo-39031.imlCYZ.rst | 2 ++ Python/ast.c | 4 ++-- 3 files changed, 13 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-12-12-21-05-43.bpo-39031.imlCYZ.rst diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py index 266075094bcbae..7df577985de177 100644 --- a/Lib/test/test_ast.py +++ b/Lib/test/test_ast.py @@ -68,6 +68,8 @@ def to_tuple(t): "while v:pass", # If "if v:pass", + # If-Elif + "if a:\n pass\nelif b:\n pass", # With "with x as y: pass", "with x as y, z as q: pass", @@ -799,6 +801,12 @@ def test_multi_line_docstring_col_offset_and_lineno_issue16806(self): self.assertEqual(node.body[2].col_offset, 0) self.assertEqual(node.body[2].lineno, 13) + def test_elif_stmt_start_position(self): + node = ast.parse('if a:\n pass\nelif b:\n pass\n') + elif_stmt = node.body[0].orelse[0] + self.assertEqual(elif_stmt.lineno, 3) + self.assertEqual(elif_stmt.col_offset, 0) + def test_literal_eval(self): self.assertEqual(ast.literal_eval('[1, 2, 3]'), [1, 2, 3]) self.assertEqual(ast.literal_eval('{"foo": 42}'), {"foo": 42}) @@ -1781,6 +1789,7 @@ def main(): ('Module', [('For', (1, 0), ('Name', (1, 4), 'v', ('Store',)), ('Name', (1, 9), 'v', ('Load',)), [('Pass', (1, 11))], [], None)], []), ('Module', [('While', (1, 0), ('Name', (1, 6), 'v', ('Load',)), [('Pass', (1, 8))], [])], []), ('Module', [('If', (1, 0), ('Name', (1, 3), 'v', ('Load',)), [('Pass', (1, 5))], [])], []), +('Module', [('If', (1, 0), ('Name', (1, 3), 'a', ('Load',)), [('Pass', (2, 2))], [('If', (3, 0), ('Name', (3, 5), 'b', ('Load',)), [('Pass', (4, 2))], [])])], []), ('Module', [('With', (1, 0), [('withitem', ('Name', (1, 5), 'x', ('Load',)), ('Name', (1, 10), 'y', ('Store',)))], [('Pass', (1, 13))], None)], []), ('Module', [('With', (1, 0), [('withitem', ('Name', (1, 5), 'x', ('Load',)), ('Name', (1, 10), 'y', ('Store',))), ('withitem', ('Name', (1, 13), 'z', ('Load',)), ('Name', (1, 18), 'q', ('Store',)))], [('Pass', (1, 21))], None)], []), ('Module', [('Raise', (1, 0), ('Call', (1, 6), ('Name', (1, 6), 'Exception', ('Load',)), [('Constant', (1, 16), 'string', None)], []), None)], []), diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-12-12-21-05-43.bpo-39031.imlCYZ.rst b/Misc/NEWS.d/next/Core and Builtins/2019-12-12-21-05-43.bpo-39031.imlCYZ.rst new file mode 100644 index 00000000000000..738902ce907ad3 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2019-12-12-21-05-43.bpo-39031.imlCYZ.rst @@ -0,0 +1,2 @@ +When parsing an "elif" node, lineno and col_offset of the node now point to the "elif" keyword and not to its condition, making it consistent with the "if" node. +Patch by Lysandros Nikolaou. diff --git a/Python/ast.c b/Python/ast.c index 2031b88359363a..15d20279b16e30 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -4079,8 +4079,8 @@ ast_for_if_stmt(struct compiling *c, const node *n) } asdl_seq_SET(newobj, 0, If(expression, suite_seq, orelse, - LINENO(CHILD(n, off)), - CHILD(n, off)->n_col_offset, + LINENO(CHILD(n, off - 1)), + CHILD(n, off - 1)->n_col_offset, end_lineno, end_col_offset, c->c_arena)); orelse = newobj; } From aa74a53ad61134911ac7904f24fd2630aeaa8ac8 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 13 Dec 2019 15:30:41 -0800 Subject: [PATCH 1012/2163] bpo-36406: Handle namespace packages in doctest (GH-12520) (GH-17591) (cherry picked from commit 8289e27393395ee903bd096d42e07c112d7f15c6) Co-authored-by: Xtreak --- Lib/doctest.py | 3 ++- Lib/test/test_doctest.py | 6 +++++- .../next/Library/2019-03-24-12-12-27.bpo-36406.mCEkOl.rst | 1 + 3 files changed, 8 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-03-24-12-12-27.bpo-36406.mCEkOl.rst diff --git a/Lib/doctest.py b/Lib/doctest.py index bf4889f59e0da4..dcbcfe52e90f6b 100644 --- a/Lib/doctest.py +++ b/Lib/doctest.py @@ -1059,7 +1059,8 @@ def _get_test(self, obj, name, module, globs, source_lines): if module is None: filename = None else: - filename = getattr(module, '__file__', module.__name__) + # __file__ can be None for namespace packages. + filename = getattr(module, '__file__', None) or module.__name__ if filename[-4:] == ".pyc": filename = filename[:-1] return self._parser.get_doctest(docstring, globs, name, diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py index 5ea18f52c4fcf2..e37d842e37ad0a 100644 --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -701,8 +701,12 @@ def test_empty_namespace_package(self): finally: support.forget(pkg_name) sys.path.pop() - assert doctest.DocTestFinder().find(mod) == [] + include_empty_finder = doctest.DocTestFinder(exclude_empty=False) + exclude_empty_finder = doctest.DocTestFinder(exclude_empty=True) + + self.assertEqual(len(include_empty_finder.find(mod)), 1) + self.assertEqual(len(exclude_empty_finder.find(mod)), 0) def test_DocTestParser(): r""" Unit tests for the `DocTestParser` class. diff --git a/Misc/NEWS.d/next/Library/2019-03-24-12-12-27.bpo-36406.mCEkOl.rst b/Misc/NEWS.d/next/Library/2019-03-24-12-12-27.bpo-36406.mCEkOl.rst new file mode 100644 index 00000000000000..3d81eb50418b04 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-03-24-12-12-27.bpo-36406.mCEkOl.rst @@ -0,0 +1 @@ +Handle namespace packages in :mod:`doctest`. Patch by Karthikeyan Singaravelan. From 4a5db78c9a46a9d151635bc09c7c891069c6bf52 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 14 Dec 2019 02:38:35 -0800 Subject: [PATCH 1013/2163] Add PYTHONUTF8 to commandline usage. (GH-17587) Co-Authored-By: Victor Stinner (cherry picked from commit 95826c773a9004fc5b3c89de55f800504685ab21) Co-authored-by: Inada Naoki --- Python/initconfig.c | 1 + 1 file changed, 1 insertion(+) diff --git a/Python/initconfig.c b/Python/initconfig.c index a41a3292d3e2c1..a30fdd1bab9542 100644 --- a/Python/initconfig.c +++ b/Python/initconfig.c @@ -80,6 +80,7 @@ static const char usage_5[] = "PYTHONHOME : alternate directory (or %lc).\n" " The default module search path uses %s.\n" "PYTHONCASEOK : ignore case in 'import' statements (Windows).\n" +"PYTHONUTF8: if set to 1, enable the UTF-8 mode.\n" "PYTHONIOENCODING: Encoding[:errors] used for stdin/stdout/stderr.\n" "PYTHONFAULTHANDLER: dump the Python traceback on fatal errors.\n"; static const char usage_6[] = From ce333cd7d5948b501a988bb56d6a1648e973577c Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 14 Dec 2019 02:43:42 -0800 Subject: [PATCH 1014/2163] Fix elif start column offset when there is an else following (GH-17596) (GH-17600) (cherry picked from commit 5936a4ce914d42af97b9238e5090dedc8d5b0bd2) Co-authored-by: Lysandros Nikolaou --- Lib/test/test_ast.py | 9 +++++++++ Python/ast.c | 4 ++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py index 7df577985de177..caea84f80f6a4b 100644 --- a/Lib/test/test_ast.py +++ b/Lib/test/test_ast.py @@ -70,6 +70,8 @@ def to_tuple(t): "if v:pass", # If-Elif "if a:\n pass\nelif b:\n pass", + # If-Elif-Else + "if a:\n pass\nelif b:\n pass\nelse:\n pass", # With "with x as y: pass", "with x as y, z as q: pass", @@ -807,6 +809,12 @@ def test_elif_stmt_start_position(self): self.assertEqual(elif_stmt.lineno, 3) self.assertEqual(elif_stmt.col_offset, 0) + def test_elif_stmt_start_position_with_else(self): + node = ast.parse('if a:\n pass\nelif b:\n pass\nelse:\n pass\n') + elif_stmt = node.body[0].orelse[0] + self.assertEqual(elif_stmt.lineno, 3) + self.assertEqual(elif_stmt.col_offset, 0) + def test_literal_eval(self): self.assertEqual(ast.literal_eval('[1, 2, 3]'), [1, 2, 3]) self.assertEqual(ast.literal_eval('{"foo": 42}'), {"foo": 42}) @@ -1790,6 +1798,7 @@ def main(): ('Module', [('While', (1, 0), ('Name', (1, 6), 'v', ('Load',)), [('Pass', (1, 8))], [])], []), ('Module', [('If', (1, 0), ('Name', (1, 3), 'v', ('Load',)), [('Pass', (1, 5))], [])], []), ('Module', [('If', (1, 0), ('Name', (1, 3), 'a', ('Load',)), [('Pass', (2, 2))], [('If', (3, 0), ('Name', (3, 5), 'b', ('Load',)), [('Pass', (4, 2))], [])])], []), +('Module', [('If', (1, 0), ('Name', (1, 3), 'a', ('Load',)), [('Pass', (2, 2))], [('If', (3, 0), ('Name', (3, 5), 'b', ('Load',)), [('Pass', (4, 2))], [('Pass', (6, 2))])])], []), ('Module', [('With', (1, 0), [('withitem', ('Name', (1, 5), 'x', ('Load',)), ('Name', (1, 10), 'y', ('Store',)))], [('Pass', (1, 13))], None)], []), ('Module', [('With', (1, 0), [('withitem', ('Name', (1, 5), 'x', ('Load',)), ('Name', (1, 10), 'y', ('Store',))), ('withitem', ('Name', (1, 13), 'z', ('Load',)), ('Name', (1, 18), 'q', ('Store',)))], [('Pass', (1, 21))], None)], []), ('Module', [('Raise', (1, 0), ('Call', (1, 6), ('Name', (1, 6), 'Exception', ('Load',)), [('Constant', (1, 16), 'string', None)], []), None)], []), diff --git a/Python/ast.c b/Python/ast.c index 15d20279b16e30..95caeffc8ff78e 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -4053,8 +4053,8 @@ ast_for_if_stmt(struct compiling *c, const node *n) asdl_seq_SET(orelse, 0, If(expression, suite_seq, suite_seq2, - LINENO(CHILD(n, NCH(n) - 6)), - CHILD(n, NCH(n) - 6)->n_col_offset, + LINENO(CHILD(n, NCH(n) - 7)), + CHILD(n, NCH(n) - 7)->n_col_offset, end_lineno, end_col_offset, c->c_arena)); /* the just-created orelse handled the last elif */ n_elif--; From 5c5d8f63d7d235e557ad20e7d722b22772681759 Mon Sep 17 00:00:00 2001 From: Inada Naoki Date: Sat, 14 Dec 2019 23:01:54 +0900 Subject: [PATCH 1015/2163] bpo-39035: travis: Don't use beta group (GH-17603) --- .travis.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 8a960c53426f59..8954a3a8ca0add 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,6 +1,5 @@ language: c dist: xenial -group: beta # To cache doc-building dependencies and C compiler output. cache: From cd968dea28569960e0b78350d43b89b83dbd49c5 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 15 Dec 2019 12:04:07 -0800 Subject: [PATCH 1016/2163] The comment in ast_for_namedexpr shouldn't include if_stmt (GH-17586) Automerge-Triggered-By: @gvanrossum (cherry picked from commit b08d3f71beab59653edfbbcf7b92a7bc8050d6b8) Co-authored-by: Guido van Rossum --- Python/ast.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/Python/ast.c b/Python/ast.c index 95caeffc8ff78e..7081394e5844b1 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -1935,9 +1935,7 @@ ast_for_decorated(struct compiling *c, const node *n) static expr_ty ast_for_namedexpr(struct compiling *c, const node *n) { - /* if_stmt: 'if' namedexpr_test ':' suite ('elif' namedexpr_test ':' suite)* - ['else' ':' suite] - namedexpr_test: test [':=' test] + /* namedexpr_test: test [':=' test] argument: ( test [comp_for] | test ':=' test | test '=' test | From 8d0f36940e728989822c3789025b0813a8fe249a Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 16 Dec 2019 04:42:20 -0800 Subject: [PATCH 1017/2163] bpo-38811: Check for presence of os.link method in pathlib (GH-17225) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Commit 6b5b013bcc22 ("bpo-26978: Implement pathlib.Path.link_to (Using os.link) (GH-12990)") introduced a new link_to method in pathlib. However, this makes pathlib crash when the 'os' module is missing a 'link' method. Fix this by checking for the presence of the 'link' method on pathlib module import, and if it's not present, turn it into a runtime error like those emitted when there is no lchmod() or symlink(). Signed-off-by: Toke Høiland-Jørgensen (cherry picked from commit 092435e932dee1802784ec28f39454f50fdd879a) Co-authored-by: Toke Høiland-Jørgensen --- Lib/pathlib.py | 7 ++++++- Lib/test/test_pathlib.py | 10 ++++++++++ .../Library/2019-11-15-18-06-04.bpo-38811.AmdQ6M.rst | 1 + 3 files changed, 17 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2019-11-15-18-06-04.bpo-38811.AmdQ6M.rst diff --git a/Lib/pathlib.py b/Lib/pathlib.py index c42bde459b3686..8ed3c883c60067 100644 --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -418,7 +418,12 @@ def lchmod(self, pathobj, mode): unlink = os.unlink - link_to = os.link + if hasattr(os, "link"): + link_to = os.link + else: + @staticmethod + def link_to(self, target): + raise NotImplementedError("os.link() not available on this system") rmdir = os.rmdir diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index 4543e3773dbf89..b127343982ca9a 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -1672,6 +1672,7 @@ def test_rmdir(self): self.assertFileNotFound(p.stat) self.assertFileNotFound(p.unlink) + @unittest.skipUnless(hasattr(os, "link"), "os.link() is not present") def test_link_to(self): P = self.cls(BASE) p = P / 'fileA' @@ -1691,6 +1692,15 @@ def test_link_to(self): self.assertEqual(os.stat(r).st_size, size) self.assertTrue(q.stat) + @unittest.skipIf(hasattr(os, "link"), "os.link() is present") + def test_link_to_not_implemented(self): + P = self.cls(BASE) + p = P / 'fileA' + # linking to another path. + q = P / 'dirA' / 'fileAA' + with self.assertRaises(NotImplementedError): + p.link_to(q) + def test_rename(self): P = self.cls(BASE) p = P / 'fileA' diff --git a/Misc/NEWS.d/next/Library/2019-11-15-18-06-04.bpo-38811.AmdQ6M.rst b/Misc/NEWS.d/next/Library/2019-11-15-18-06-04.bpo-38811.AmdQ6M.rst new file mode 100644 index 00000000000000..0e4a7f5bdf5383 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-11-15-18-06-04.bpo-38811.AmdQ6M.rst @@ -0,0 +1 @@ +Fix an unhandled exception in :mod:`pathlib` when :meth:`os.link` is missing. Patch by Toke Høiland-Jørgensen. From a96e938fc3e266a56d8cebc53687eb2fd56e7589 Mon Sep 17 00:00:00 2001 From: Kyle Stanley Date: Mon, 16 Dec 2019 23:01:19 -0500 Subject: [PATCH 1018/2163] [3.8] Add whatsnew for removal of asyncio.loop.create_datagram_endpoint()'s *reuse_address* parameter (GH-17595). (#17630) (cherry picked from commit f501db2b93a9d3d840b6fb38d6bdda8bcc400d4a) Co-authored-by: Kyle Stanley --- Doc/whatsnew/3.6.rst | 10 ++++++++++ Doc/whatsnew/3.7.rst | 10 ++++++++++ Doc/whatsnew/3.8.rst | 10 ++++++++++ 3 files changed, 30 insertions(+) diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst index 3f5f5200f122f6..04c1f7e71db321 100644 --- a/Doc/whatsnew/3.6.rst +++ b/Doc/whatsnew/3.6.rst @@ -2433,3 +2433,13 @@ In 3.6.7 the :mod:`tokenize` module now implicitly emits a ``NEWLINE`` token when provided with input that does not have a trailing new line. This behavior now matches what the C tokenizer does internally. (Contributed by Ammar Askar in :issue:`33899`.) + +Notable changes in Python 3.6.10 +================================ + +Due to significant security concerns, the *reuse_address* parameter of +:meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is +because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more +details, see the documentation for ``loop.create_datagram_endpoint()``. +(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in +:issue:`37228`.) diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 2197bd9739299e..8a70fe22d52bdc 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -2541,3 +2541,13 @@ This resolves a long standing issue where all virtual environments would have to be upgraded or recreated with each Python update. However, note that this release will still require recreation of virtual environments in order to get the new scripts. + +Notable changes in Python 3.7.6 +=============================== + +Due to significant security concerns, the *reuse_address* parameter of +:meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is +because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more +details, see the documentation for ``loop.create_datagram_endpoint()``. +(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in +:issue:`37228`.) diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index d975fb78154746..546fa2d5f016ec 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -2206,3 +2206,13 @@ Here's a summary of performance improvements since Python 3.3: loop_overhead 0.3 0.5 0.6 0.4 0.3 0.3 (Measured from the macOS 64-bit builds found at python.org) + +Notable changes in Python 3.8.1 +=============================== + +Due to significant security concerns, the *reuse_address* parameter of +:meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is +because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more +details, see the documentation for ``loop.create_datagram_endpoint()``. +(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in +:issue:`37228`.) From d21ad67d5ec7a08e760231c967f4d8b0c148d18f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 16 Dec 2019 20:11:11 -0800 Subject: [PATCH 1019/2163] Fix warnings in test_asyncio.test_base_events (GH-17577) (GH-17579) Co-authored-by: tirkarthi (cherry picked from commit 1988344a6bff253f017e053f69318ecf03587294) Co-authored-by: Kyle Stanley --- Lib/test/test_asyncio/test_base_events.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py index 3a23059911211f..0f9e3d04d78eb8 100644 --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -1808,7 +1808,10 @@ def test_create_datagram_endpoint_reuse_address_warning(self): reuse_address=False) with self.assertWarns(DeprecationWarning): - self.loop.run_until_complete(coro) + transport, protocol = self.loop.run_until_complete(coro) + transport.close() + self.loop.run_until_complete(protocol.done) + self.assertEqual('CLOSED', protocol.state) @patch_socket def test_create_datagram_endpoint_nosoreuseport(self, m_socket): @@ -1818,7 +1821,6 @@ def test_create_datagram_endpoint_nosoreuseport(self, m_socket): coro = self.loop.create_datagram_endpoint( lambda: MyDatagramProto(loop=self.loop), local_addr=('127.0.0.1', 0), - reuse_address=False, reuse_port=True) self.assertRaises(ValueError, self.loop.run_until_complete, coro) @@ -1837,7 +1839,6 @@ def getaddrinfo(*args, **kw): coro = self.loop.create_datagram_endpoint( lambda: MyDatagramProto(loop=self.loop), local_addr=('1.2.3.4', 0), - reuse_address=False, reuse_port=reuseport_supported) t, p = self.loop.run_until_complete(coro) From bf3aa1060a29a05813abbe877193af16e3e7131e Mon Sep 17 00:00:00 2001 From: Ned Deily Date: Tue, 17 Dec 2019 04:05:41 -0500 Subject: [PATCH 1020/2163] bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636) --- Lib/test/test_py_compile.py | 2 +- Misc/NEWS.d/next/macOS/2019-12-17-03-43-04.bpo-38295.hgDvlB.rst | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/macOS/2019-12-17-03-43-04.bpo-38295.hgDvlB.rst diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py index d6677ab45ff509..d4a68c9320d04f 100644 --- a/Lib/test/test_py_compile.py +++ b/Lib/test/test_py_compile.py @@ -51,7 +51,7 @@ def __new__(mcls, name, bases, dct, *, source_date_epoch): class PyCompileTestsBase: def setUp(self): - self.directory = tempfile.mkdtemp() + self.directory = tempfile.mkdtemp(dir=os.getcwd()) self.source_path = os.path.join(self.directory, '_test.py') self.pyc_path = self.source_path + 'c' self.cache_path = importlib.util.cache_from_source(self.source_path) diff --git a/Misc/NEWS.d/next/macOS/2019-12-17-03-43-04.bpo-38295.hgDvlB.rst b/Misc/NEWS.d/next/macOS/2019-12-17-03-43-04.bpo-38295.hgDvlB.rst new file mode 100644 index 00000000000000..cc9ceb4cc50b30 --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2019-12-17-03-43-04.bpo-38295.hgDvlB.rst @@ -0,0 +1 @@ +Prevent failure of test_relative_path in test_py_compile on macOS Catalina. From b1f204471092678dd89117e608fa041a9589d14c Mon Sep 17 00:00:00 2001 From: Pablo Galindo Date: Wed, 18 Dec 2019 01:41:58 +0000 Subject: [PATCH 1021/2163] [3.8] bpo-39080: Starred Expression's column offset fix when inside a CALL (GH-17645) (GH-17649) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit … Co-Authored-By: Pablo Galindo (cherry picked from commit 50d4f12958bf806a4e1a1021d70cfd5d448c5cba) Co-authored-by: Lysandros Nikolaou https://bugs.python.org/issue39080 --- Lib/test/test_ast.py | 9 +++++++++ .../2019-12-17-21-45-36.bpo-39080.OrxEVS.rst | 1 + Python/ast.c | 2 +- 3 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-12-17-21-45-36.bpo-39080.OrxEVS.rst diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py index caea84f80f6a4b..69ed83cd8b68b6 100644 --- a/Lib/test/test_ast.py +++ b/Lib/test/test_ast.py @@ -209,6 +209,8 @@ def to_tuple(t): "1 < 2 < 3", # Call "f(1,2,c=3,*d,**e)", + # Call with multi-character starred + "f(*[0, 1])", # Call with a generator argument "f(a for a in b)", # Num @@ -815,6 +817,12 @@ def test_elif_stmt_start_position_with_else(self): self.assertEqual(elif_stmt.lineno, 3) self.assertEqual(elif_stmt.col_offset, 0) + def test_starred_expr_end_position_within_call(self): + node = ast.parse('f(*[0, 1])') + starred_expr = node.body[0].value.args[0] + self.assertEqual(starred_expr.end_lineno, 1) + self.assertEqual(starred_expr.end_col_offset, 9) + def test_literal_eval(self): self.assertEqual(ast.literal_eval('[1, 2, 3]'), [1, 2, 3]) self.assertEqual(ast.literal_eval('{"foo": 42}'), {"foo": 42}) @@ -1868,6 +1876,7 @@ def main(): ('Expression', ('GeneratorExp', (1, 0), ('Tuple', (1, 1), [('Name', (1, 2), 'a', ('Load',)), ('Name', (1, 4), 'b', ('Load',))], ('Load',)), [('comprehension', ('List', (1, 11), [('Name', (1, 12), 'a', ('Store',)), ('Name', (1, 14), 'b', ('Store',))], ('Store',)), ('Name', (1, 20), 'c', ('Load',)), [], 0)])), ('Expression', ('Compare', (1, 0), ('Constant', (1, 0), 1, None), [('Lt',), ('Lt',)], [('Constant', (1, 4), 2, None), ('Constant', (1, 8), 3, None)])), ('Expression', ('Call', (1, 0), ('Name', (1, 0), 'f', ('Load',)), [('Constant', (1, 2), 1, None), ('Constant', (1, 4), 2, None), ('Starred', (1, 10), ('Name', (1, 11), 'd', ('Load',)), ('Load',))], [('keyword', 'c', ('Constant', (1, 8), 3, None)), ('keyword', None, ('Name', (1, 15), 'e', ('Load',)))])), +('Expression', ('Call', (1, 0), ('Name', (1, 0), 'f', ('Load',)), [('Starred', (1, 2), ('List', (1, 3), [('Constant', (1, 4), 0, None), ('Constant', (1, 7), 1, None)], ('Load',)), ('Load',))], [])), ('Expression', ('Call', (1, 0), ('Name', (1, 0), 'f', ('Load',)), [('GeneratorExp', (1, 1), ('Name', (1, 2), 'a', ('Load',)), [('comprehension', ('Name', (1, 8), 'a', ('Store',)), ('Name', (1, 13), 'b', ('Load',)), [], 0)])], [])), ('Expression', ('Constant', (1, 0), 10, None)), ('Expression', ('Constant', (1, 0), 'string', None)), diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-12-17-21-45-36.bpo-39080.OrxEVS.rst b/Misc/NEWS.d/next/Core and Builtins/2019-12-17-21-45-36.bpo-39080.OrxEVS.rst new file mode 100644 index 00000000000000..b120d496b81e09 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2019-12-17-21-45-36.bpo-39080.OrxEVS.rst @@ -0,0 +1 @@ +Fix the value of *end_col_offset* for Starred Expression AST nodes when they are among the elements in the *args* attribute of Call AST nodes. diff --git a/Python/ast.c b/Python/ast.c index 7081394e5844b1..e70ab519e66e1b 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -3126,7 +3126,7 @@ ast_for_call(struct compiling *c, const node *n, expr_ty func, return NULL; starred = Starred(e, Load, LINENO(chch), chch->n_col_offset, - chch->n_end_lineno, chch->n_end_col_offset, + e->end_lineno, e->end_col_offset, c->c_arena); if (!starred) return NULL; From 35acb3597208e10a101140474adec86859d57f61 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 18 Dec 2019 10:11:05 +0100 Subject: [PATCH 1022/2163] bpo-38546: multiprocessing tests stop the resource tracker (GH-17641) (GH-17647) Multiprocessing and concurrent.futures tests now stop the resource tracker process when tests complete. Add ResourceTracker._stop() method to multiprocessing.resource_tracker. Add _cleanup_tests() helper function to multiprocessing.util: share code between multiprocessing and concurrent.futures tests. (cherry picked from commit 9707e8e22d80ca97bf7a9812816701cecde6d226) --- Lib/multiprocessing/resource_tracker.py | 13 ++++++++++ Lib/multiprocessing/util.py | 25 +++++++++++++++++++ Lib/test/_test_multiprocessing.py | 11 +------- Lib/test/test_concurrent_futures.py | 12 +-------- .../2019-12-17-15-27-07.bpo-38546.82JwN2.rst | 2 ++ 5 files changed, 42 insertions(+), 21 deletions(-) create mode 100644 Misc/NEWS.d/next/Tests/2019-12-17-15-27-07.bpo-38546.82JwN2.rst diff --git a/Lib/multiprocessing/resource_tracker.py b/Lib/multiprocessing/resource_tracker.py index 61a6dd66e72e67..c9bfa9b82b6e6c 100644 --- a/Lib/multiprocessing/resource_tracker.py +++ b/Lib/multiprocessing/resource_tracker.py @@ -50,6 +50,19 @@ def __init__(self): self._fd = None self._pid = None + def _stop(self): + with self._lock: + if self._fd is None: + # not running + return + + # closing the "alive" file descriptor stops main() + os.close(self._fd) + self._fd = None + + os.waitpid(self._pid, 0) + self._pid = None + def getfd(self): self.ensure_running() return self._fd diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py index 32b51b04373f91..745f2b2651b11f 100644 --- a/Lib/multiprocessing/util.py +++ b/Lib/multiprocessing/util.py @@ -439,3 +439,28 @@ def close_fds(*fds): """Close each file descriptor given as an argument""" for fd in fds: os.close(fd) + + +def _cleanup_tests(): + """Cleanup multiprocessing resources when multiprocessing tests + completed.""" + + from test import support + + # cleanup multiprocessing + process._cleanup() + + # Stop the ForkServer process if it's running + from multiprocessing import forkserver + forkserver._forkserver._stop() + + # Stop the ResourceTracker process if it's running + from multiprocessing import resource_tracker + resource_tracker._resource_tracker._stop() + + # bpo-37421: Explicitly call _run_finalizers() to remove immediately + # temporary directories created by multiprocessing.util.get_temp_dir(). + _run_finalizers() + support.gc_collect() + + support.reap_children() diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index b7f3d253c4d131..983770f35761d0 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -5673,16 +5673,7 @@ def tearDownModule(): if need_sleep: time.sleep(0.5) - multiprocessing.process._cleanup() - - # Stop the ForkServer process if it's running - from multiprocessing import forkserver - forkserver._forkserver._stop() - - # bpo-37421: Explicitly call _run_finalizers() to remove immediately - # temporary directories created by multiprocessing.util.get_temp_dir(). - multiprocessing.util._run_finalizers() - test.support.gc_collect() + multiprocessing.util._cleanup_tests() remote_globs['setUpModule'] = setUpModule remote_globs['tearDownModule'] = tearDownModule diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py index 98c9bc9b507a47..e01f2e4b78759e 100644 --- a/Lib/test/test_concurrent_futures.py +++ b/Lib/test/test_concurrent_futures.py @@ -1306,17 +1306,7 @@ def setUpModule(): def tearDownModule(): test.support.threading_cleanup(*_threads_key) - test.support.reap_children() - - # cleanup multiprocessing - multiprocessing.process._cleanup() - # Stop the ForkServer process if it's running - from multiprocessing import forkserver - forkserver._forkserver._stop() - # bpo-37421: Explicitly call _run_finalizers() to remove immediately - # temporary directories created by multiprocessing.util.get_temp_dir(). - multiprocessing.util._run_finalizers() - test.support.gc_collect() + multiprocessing.util._cleanup_tests() if __name__ == "__main__": diff --git a/Misc/NEWS.d/next/Tests/2019-12-17-15-27-07.bpo-38546.82JwN2.rst b/Misc/NEWS.d/next/Tests/2019-12-17-15-27-07.bpo-38546.82JwN2.rst new file mode 100644 index 00000000000000..78d9df34691822 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2019-12-17-15-27-07.bpo-38546.82JwN2.rst @@ -0,0 +1,2 @@ +Multiprocessing and concurrent.futures tests now stop the resource tracker +process when tests complete. From 1b293b60067f6f4a95984d064ce0f6b6d34c1216 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Wed, 18 Dec 2019 18:21:23 +0100 Subject: [PATCH 1023/2163] Python 3.8.1 --- Include/patchlevel.h | 6 +- Lib/pydoc_data/topics.py | 2 +- Misc/NEWS.d/3.8.1.rst | 108 ++++++++++++++++++ .../2019-12-09-10-38-51.bpo-39008.Rrp6f1.rst | 3 - .../2019-12-12-21-05-43.bpo-39031.imlCYZ.rst | 2 - .../2019-12-17-21-45-36.bpo-39080.OrxEVS.rst | 1 - .../2019-11-29-23-44-11.bpo-38943.8pUKKs.rst | 2 - .../2019-11-30-12-10-36.bpo-38944._3xjKG.rst | 1 - .../2019-03-24-12-12-27.bpo-36406.mCEkOl.rst | 1 - .../2019-11-15-18-06-04.bpo-38811.AmdQ6M.rst | 1 - .../2019-12-10-23-34-48.bpo-39022.QDtIxI.rst | 1 - .../2019-12-17-15-27-07.bpo-38546.82JwN2.rst | 2 - .../2019-12-09-10-40-34.bpo-39007.vtarxo.rst | 1 - .../2019-12-17-03-43-04.bpo-38295.hgDvlB.rst | 1 - README.rst | 4 +- 15 files changed, 114 insertions(+), 22 deletions(-) create mode 100644 Misc/NEWS.d/3.8.1.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-12-09-10-38-51.bpo-39008.Rrp6f1.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-12-12-21-05-43.bpo-39031.imlCYZ.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-12-17-21-45-36.bpo-39080.OrxEVS.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2019-11-29-23-44-11.bpo-38943.8pUKKs.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2019-11-30-12-10-36.bpo-38944._3xjKG.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-03-24-12-12-27.bpo-36406.mCEkOl.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-15-18-06-04.bpo-38811.AmdQ6M.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-10-23-34-48.bpo-39022.QDtIxI.rst delete mode 100644 Misc/NEWS.d/next/Tests/2019-12-17-15-27-07.bpo-38546.82JwN2.rst delete mode 100644 Misc/NEWS.d/next/Windows/2019-12-09-10-40-34.bpo-39007.vtarxo.rst delete mode 100644 Misc/NEWS.d/next/macOS/2019-12-17-03-43-04.bpo-38295.hgDvlB.rst diff --git a/Include/patchlevel.h b/Include/patchlevel.h index cbffcd13a858e4..6936a3feee7c0f 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -19,11 +19,11 @@ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 8 #define PY_MICRO_VERSION 1 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA -#define PY_RELEASE_SERIAL 1 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL +#define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.8.1rc1+" +#define PY_VERSION "3.8.1" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index a11be7ab0f836a..9d779d1e69f23b 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Mon Dec 9 18:44:17 2019 +# Autogenerated by Sphinx on Wed Dec 18 18:17:58 2019 topics = {'assert': 'The "assert" statement\n' '**********************\n' '\n' diff --git a/Misc/NEWS.d/3.8.1.rst b/Misc/NEWS.d/3.8.1.rst new file mode 100644 index 00000000000000..8cd4b359f7b4f0 --- /dev/null +++ b/Misc/NEWS.d/3.8.1.rst @@ -0,0 +1,108 @@ +.. bpo: 39080 +.. date: 2019-12-17-21-45-36 +.. nonce: OrxEVS +.. release date: 2019-12-18 +.. section: Core and Builtins + +Fix the value of *end_col_offset* for Starred Expression AST nodes when they +are among the elements in the *args* attribute of Call AST nodes. + +.. + +.. bpo: 39031 +.. date: 2019-12-12-21-05-43 +.. nonce: imlCYZ +.. section: Core and Builtins + +When parsing an "elif" node, lineno and col_offset of the node now point to +the "elif" keyword and not to its condition, making it consistent with the +"if" node. Patch by Lysandros Nikolaou. + +.. + +.. bpo: 39008 +.. date: 2019-12-09-10-38-51 +.. nonce: Rrp6f1 +.. section: Core and Builtins + +:c:func:`PySys_Audit` now requires ``Py_ssize_t`` to be used for size +arguments in the format string, regardless of whethen ``PY_SSIZE_T_CLEAN`` +was defined at include time. + +.. + +.. bpo: 39022 +.. date: 2019-12-10-23-34-48 +.. nonce: QDtIxI +.. section: Library + +Update importliib.metadata to include improvements from importlib_metadata +1.3 including better serialization of EntryPoints and improved documentation +for custom finders. + +.. + +.. bpo: 38811 +.. date: 2019-11-15-18-06-04 +.. nonce: AmdQ6M +.. section: Library + +Fix an unhandled exception in :mod:`pathlib` when :meth:`os.link` is +missing. Patch by Toke Høiland-Jørgensen. + +.. + +.. bpo: 36406 +.. date: 2019-03-24-12-12-27 +.. nonce: mCEkOl +.. section: Library + +Handle namespace packages in :mod:`doctest`. Patch by Karthikeyan +Singaravelan. + +.. + +.. bpo: 38546 +.. date: 2019-12-17-15-27-07 +.. nonce: 82JwN2 +.. section: Tests + +Multiprocessing and concurrent.futures tests now stop the resource tracker +process when tests complete. + +.. + +.. bpo: 39007 +.. date: 2019-12-09-10-40-34 +.. nonce: vtarxo +.. section: Windows + +Add auditing events to functions in :mod:`winreg`. + +.. + +.. bpo: 38295 +.. date: 2019-12-17-03-43-04 +.. nonce: hgDvlB +.. section: macOS + +Prevent failure of test_relative_path in test_py_compile on macOS Catalina. + +.. + +.. bpo: 38944 +.. date: 2019-11-30-12-10-36 +.. nonce: _3xjKG +.. section: IDLE + +Excape key now closes IDLE completion windows. Patch by Johnny Najera. + +.. + +.. bpo: 38943 +.. date: 2019-11-29-23-44-11 +.. nonce: 8pUKKs +.. section: IDLE + +Fix IDLE autocomplete windows not always appearing on some systems. Patch by +Johnny Najera. diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-12-09-10-38-51.bpo-39008.Rrp6f1.rst b/Misc/NEWS.d/next/Core and Builtins/2019-12-09-10-38-51.bpo-39008.Rrp6f1.rst deleted file mode 100644 index 35237ce2714a01..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-12-09-10-38-51.bpo-39008.Rrp6f1.rst +++ /dev/null @@ -1,3 +0,0 @@ -:c:func:`PySys_Audit` now requires ``Py_ssize_t`` to be used for size -arguments in the format string, regardless of whethen ``PY_SSIZE_T_CLEAN`` -was defined at include time. diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-12-12-21-05-43.bpo-39031.imlCYZ.rst b/Misc/NEWS.d/next/Core and Builtins/2019-12-12-21-05-43.bpo-39031.imlCYZ.rst deleted file mode 100644 index 738902ce907ad3..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-12-12-21-05-43.bpo-39031.imlCYZ.rst +++ /dev/null @@ -1,2 +0,0 @@ -When parsing an "elif" node, lineno and col_offset of the node now point to the "elif" keyword and not to its condition, making it consistent with the "if" node. -Patch by Lysandros Nikolaou. diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-12-17-21-45-36.bpo-39080.OrxEVS.rst b/Misc/NEWS.d/next/Core and Builtins/2019-12-17-21-45-36.bpo-39080.OrxEVS.rst deleted file mode 100644 index b120d496b81e09..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-12-17-21-45-36.bpo-39080.OrxEVS.rst +++ /dev/null @@ -1 +0,0 @@ -Fix the value of *end_col_offset* for Starred Expression AST nodes when they are among the elements in the *args* attribute of Call AST nodes. diff --git a/Misc/NEWS.d/next/IDLE/2019-11-29-23-44-11.bpo-38943.8pUKKs.rst b/Misc/NEWS.d/next/IDLE/2019-11-29-23-44-11.bpo-38943.8pUKKs.rst deleted file mode 100644 index 5c9323e2467876..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2019-11-29-23-44-11.bpo-38943.8pUKKs.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix IDLE autocomplete windows not always appearing on some systems. -Patch by Johnny Najera. diff --git a/Misc/NEWS.d/next/IDLE/2019-11-30-12-10-36.bpo-38944._3xjKG.rst b/Misc/NEWS.d/next/IDLE/2019-11-30-12-10-36.bpo-38944._3xjKG.rst deleted file mode 100644 index 38084fafd2f561..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2019-11-30-12-10-36.bpo-38944._3xjKG.rst +++ /dev/null @@ -1 +0,0 @@ -Excape key now closes IDLE completion windows. Patch by Johnny Najera. diff --git a/Misc/NEWS.d/next/Library/2019-03-24-12-12-27.bpo-36406.mCEkOl.rst b/Misc/NEWS.d/next/Library/2019-03-24-12-12-27.bpo-36406.mCEkOl.rst deleted file mode 100644 index 3d81eb50418b04..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-03-24-12-12-27.bpo-36406.mCEkOl.rst +++ /dev/null @@ -1 +0,0 @@ -Handle namespace packages in :mod:`doctest`. Patch by Karthikeyan Singaravelan. diff --git a/Misc/NEWS.d/next/Library/2019-11-15-18-06-04.bpo-38811.AmdQ6M.rst b/Misc/NEWS.d/next/Library/2019-11-15-18-06-04.bpo-38811.AmdQ6M.rst deleted file mode 100644 index 0e4a7f5bdf5383..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-15-18-06-04.bpo-38811.AmdQ6M.rst +++ /dev/null @@ -1 +0,0 @@ -Fix an unhandled exception in :mod:`pathlib` when :meth:`os.link` is missing. Patch by Toke Høiland-Jørgensen. diff --git a/Misc/NEWS.d/next/Library/2019-12-10-23-34-48.bpo-39022.QDtIxI.rst b/Misc/NEWS.d/next/Library/2019-12-10-23-34-48.bpo-39022.QDtIxI.rst deleted file mode 100644 index 4af21be6074269..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-10-23-34-48.bpo-39022.QDtIxI.rst +++ /dev/null @@ -1 +0,0 @@ -Update importliib.metadata to include improvements from importlib_metadata 1.3 including better serialization of EntryPoints and improved documentation for custom finders. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Tests/2019-12-17-15-27-07.bpo-38546.82JwN2.rst b/Misc/NEWS.d/next/Tests/2019-12-17-15-27-07.bpo-38546.82JwN2.rst deleted file mode 100644 index 78d9df34691822..00000000000000 --- a/Misc/NEWS.d/next/Tests/2019-12-17-15-27-07.bpo-38546.82JwN2.rst +++ /dev/null @@ -1,2 +0,0 @@ -Multiprocessing and concurrent.futures tests now stop the resource tracker -process when tests complete. diff --git a/Misc/NEWS.d/next/Windows/2019-12-09-10-40-34.bpo-39007.vtarxo.rst b/Misc/NEWS.d/next/Windows/2019-12-09-10-40-34.bpo-39007.vtarxo.rst deleted file mode 100644 index f2f72f9dad3f6e..00000000000000 --- a/Misc/NEWS.d/next/Windows/2019-12-09-10-40-34.bpo-39007.vtarxo.rst +++ /dev/null @@ -1 +0,0 @@ -Add auditing events to functions in :mod:`winreg`. diff --git a/Misc/NEWS.d/next/macOS/2019-12-17-03-43-04.bpo-38295.hgDvlB.rst b/Misc/NEWS.d/next/macOS/2019-12-17-03-43-04.bpo-38295.hgDvlB.rst deleted file mode 100644 index cc9ceb4cc50b30..00000000000000 --- a/Misc/NEWS.d/next/macOS/2019-12-17-03-43-04.bpo-38295.hgDvlB.rst +++ /dev/null @@ -1 +0,0 @@ -Prevent failure of test_relative_path in test_py_compile on macOS Catalina. diff --git a/README.rst b/README.rst index 4064b511a87a16..d14f2b3ad99a68 100644 --- a/README.rst +++ b/README.rst @@ -1,5 +1,5 @@ -This is Python version 3.8.1rc1 -=============================== +This is Python version 3.8.1 +============================ .. image:: https://travis-ci.org/python/cpython.svg?branch=3.8 :alt: CPython build status on Travis CI From b0eb046cbd0dbb7b17f16aad6de20fac5305f387 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 18 Dec 2019 21:30:43 +0100 Subject: [PATCH 1024/2163] bpo-38546: Fix concurrent.futures test_ressources_gced_in_workers() (GH-17652) (GH-17655) Fix test_ressources_gced_in_workers() of test_concurrent_futures: explicitly stop the manager to prevent leaking a child process running in the background after the test completes. (cherry picked from commit 673c39331f844a80c465efd7cff88ac55c432bfb) --- Lib/test/test_concurrent_futures.py | 14 +++++++++++--- .../Tests/2019-12-18-14-52-08.bpo-38546.2kxNuM.rst | 3 +++ 2 files changed, 14 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Tests/2019-12-18-14-52-08.bpo-38546.2kxNuM.rst diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py index e01f2e4b78759e..ac722981659d33 100644 --- a/Lib/test/test_concurrent_futures.py +++ b/Lib/test/test_concurrent_futures.py @@ -87,8 +87,7 @@ def my_method(self): class EventfulGCObj(): - def __init__(self, ctx): - mgr = get_context(ctx).Manager() + def __init__(self, mgr): self.event = mgr.Event() def __del__(self): @@ -848,12 +847,21 @@ def test_traceback(self): def test_ressources_gced_in_workers(self): # Ensure that argument for a job are correctly gc-ed after the job # is finished - obj = EventfulGCObj(self.ctx) + mgr = get_context(self.ctx).Manager() + obj = EventfulGCObj(mgr) future = self.executor.submit(id, obj) future.result() self.assertTrue(obj.event.wait(timeout=1)) + # explicitly destroy the object to ensure that EventfulGCObj.__del__() + # is called while manager is still running. + obj = None + test.support.gc_collect() + + mgr.shutdown() + mgr.join() + create_executor_tests(ProcessPoolExecutorTest, executor_mixins=(ProcessPoolForkMixin, diff --git a/Misc/NEWS.d/next/Tests/2019-12-18-14-52-08.bpo-38546.2kxNuM.rst b/Misc/NEWS.d/next/Tests/2019-12-18-14-52-08.bpo-38546.2kxNuM.rst new file mode 100644 index 00000000000000..d8ec7cabbbab82 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2019-12-18-14-52-08.bpo-38546.2kxNuM.rst @@ -0,0 +1,3 @@ +Fix test_ressources_gced_in_workers() of test_concurrent_futures: explicitly +stop the manager to prevent leaking a child process running in the background +after the test completes. From f89c86511af4821b117cd61f4a974ea895d476ad Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Thu, 19 Dec 2019 09:01:48 +0100 Subject: [PATCH 1025/2163] Post 3.8.1 --- Include/patchlevel.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h index 6936a3feee7c0f..503d3aa85676f0 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.8.1" +#define PY_VERSION "3.8.1+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. From eba61f33d60cc63e35d5f5fcada837a66c89774a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Batuhan=20Ta=C5=9Fkaya?= <47358913+isidentical@users.noreply.github.com> Date: Thu, 19 Dec 2019 17:44:27 +0300 Subject: [PATCH 1026/2163] [3.8] bpo-38316: Fix co_stacksize documentation (GH-16983) (GH-17661) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit (cherry picked from commit d587272fe3b0fcad2f23a490e76f9f82ca7d64ef) Co-authored-by: Batuhan Taşkaya <47358913+isidentical@users.noreply.github.com> https://bugs.python.org/issue38316 Automerge-Triggered-By: @vstinner --- Doc/reference/datamodel.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 46d50ad600ff52..c530e4117af97c 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -925,8 +925,8 @@ Internal types the first line number of the function; :attr:`co_lnotab` is a string encoding the mapping from bytecode offsets to line numbers (for details see the source code of the interpreter); :attr:`co_stacksize` is the - required stack size (including local variables); :attr:`co_flags` is an - integer encoding a number of flags for the interpreter. + required stack size; :attr:`co_flags` is an integer encoding a number + of flags for the interpreter. .. index:: object: generator From 44683bbc634bd007df572dce0d8bb348a469a6bb Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 20 Dec 2019 11:26:39 -0800 Subject: [PATCH 1027/2163] bpo-38918: Add __module__ entry for function & method type in inspect docs table (GH-17408) Adds` __module__ ` entries for function & method types in inspect docs table. https://bugs.python.org/issue38918 (cherry picked from commit f522a6ddb67a238bab5673608111f74ec4e22205) Co-authored-by: Parth Sharma --- Doc/library/inspect.rst | 6 ++++++ .../Documentation/2019-12-15-22-04-41.bpo-38918.8JnDTS.rst | 3 +++ 2 files changed, 9 insertions(+) create mode 100644 Misc/NEWS.d/next/Documentation/2019-12-15-22-04-41.bpo-38918.8JnDTS.rst diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 10f347dd420f31..bab2c41e4e2242 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -70,6 +70,9 @@ attributes: | | | method is bound, or | | | | ``None`` | +-----------+-------------------+---------------------------+ +| | __module__ | name of module in which | +| | | this method was defined | ++-----------+-------------------+---------------------------+ | function | __doc__ | documentation string | +-----------+-------------------+---------------------------+ | | __name__ | name with which this | @@ -98,6 +101,9 @@ attributes: | | | reserved for return | | | | annotations. | +-----------+-------------------+---------------------------+ +| | __module__ | name of module in which | +| | | this function was defined | ++-----------+-------------------+---------------------------+ | traceback | tb_frame | frame object at this | | | | level | +-----------+-------------------+---------------------------+ diff --git a/Misc/NEWS.d/next/Documentation/2019-12-15-22-04-41.bpo-38918.8JnDTS.rst b/Misc/NEWS.d/next/Documentation/2019-12-15-22-04-41.bpo-38918.8JnDTS.rst new file mode 100644 index 00000000000000..5747936dd64d50 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2019-12-15-22-04-41.bpo-38918.8JnDTS.rst @@ -0,0 +1,3 @@ +Add an entry for ``__module__`` in the "function" & "method" sections of the +`inspect docs types and members table +`_ From 35590c5ee178ce101955e41a213febccb077684d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 22 Dec 2019 09:54:33 -0800 Subject: [PATCH 1028/2163] Add missing markup (GH-17680) "HH", "MM" and "ffffff" are enclosed with double back quotes, but "SS" is left being bare (cherry picked from commit 068768faf6b82478de239d7ab903dfb249ad96a4) Co-authored-by: cocoatomo --- Doc/library/datetime.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index b49eab44aef9e3..2bc4652c263965 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -2514,7 +2514,7 @@ Notes: :meth:`utcoffset` is transformed into a string of the form ``±HHMM[SS[.ffffff]]``, where ``HH`` is a 2-digit string giving the number of UTC offset hours, ``MM`` is a 2-digit string giving the number of UTC - offset minutes, SS is a 2-digit string giving the number of UTC offset + offset minutes, ``SS`` is a 2-digit string giving the number of UTC offset seconds and ``ffffff`` is a 6-digit string giving the number of UTC offset microseconds. The ``ffffff`` part is omitted when the offset is a whole number of seconds and both the ``ffffff`` and the ``SS`` part is From a542aec092404c5cbb89f9e88bf2a9f4548ef179 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 23 Dec 2019 06:44:04 -0800 Subject: [PATCH 1029/2163] [typo] fix dupe in datetime.fromisoformat docs (GH-17295) Fixes a nearly word for word duplication of a sentence that appears earlier in the caution section of datetime.datetime.fromisoformat in Doc/Library/datetime.rst. No issue created as it's a trivial change. Automerge-Triggered-By: @pganssle (cherry picked from commit e7b406f4e20e248d24079f97d7c8b93e5a45249e) Co-authored-by: Michael Morehouse <640167+yawpitch@users.noreply.github.com> --- Doc/library/datetime.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 2bc4652c263965..29c6b5a4736c0d 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -997,8 +997,6 @@ Other constructors, all class methods: as the inverse operation of :meth:`datetime.isoformat`. A more full-featured ISO 8601 parser, ``dateutil.parser.isoparse`` is available in the third-party package `dateutil `__. - This does not support parsing arbitrary ISO 8601 strings - it is only intended - as the inverse operation of :meth:`datetime.isoformat`. Examples:: From 7eb8c6d2c89a8a7ba3af4a99ab20456dff544881 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 23 Dec 2019 07:52:29 -0800 Subject: [PATCH 1030/2163] Doc typo (GH-17667) (#17668) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit (cherry picked from commit b0d4949f1fb04f83691e10a5453d1e10e4598bb9) Co-authored-by: Jesús Cea Co-authored-by: Jesús Cea --- Doc/library/asyncio-eventloop.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index ecd69fd669f1a5..7fed2c23df7d5d 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -467,7 +467,7 @@ Opening network connections UDP. Explicitly passing ``reuse_address=True`` will raise an exception. When multiple processes with differing UIDs assign sockets to an - indentical UDP socket address with ``SO_REUSEADDR``, incoming packets can + identical UDP socket address with ``SO_REUSEADDR``, incoming packets can become randomly distributed among the sockets. For supported platforms, *reuse_port* can be used as a replacement for From 0846e5d4603434c2bbf8a528677cf1ff3fe29b95 Mon Sep 17 00:00:00 2001 From: Bar Harel Date: Mon, 23 Dec 2019 20:31:00 +0200 Subject: [PATCH 1031/2163] [3.8] bpo-38878: Fix os.PathLike __subclasshook__ (GH-17336) (GH-17684) https://bugs.python.org/issue38878 --- Lib/os.py | 6 +++++- Lib/test/test_os.py | 8 ++++++++ .../next/Library/2019-11-22-12-08-52.bpo-38878.EJ0cFf.rst | 2 ++ 3 files changed, 15 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2019-11-22-12-08-52.bpo-38878.EJ0cFf.rst diff --git a/Lib/os.py b/Lib/os.py index 52d3f1d7415854..253cad1a59f35a 100644 --- a/Lib/os.py +++ b/Lib/os.py @@ -26,6 +26,8 @@ import sys import stat as st +from _collections_abc import _check_methods + _names = sys.builtin_module_names # Note: more names are added to __all__ later. @@ -1070,7 +1072,9 @@ def __fspath__(self): @classmethod def __subclasshook__(cls, subclass): - return hasattr(subclass, '__fspath__') + if cls is PathLike: + return _check_methods(subclass, '__fspath__') + return NotImplemented if name == 'nt': diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py index 4a076e3bbf5426..11454b2e88ff65 100644 --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -4017,6 +4017,14 @@ def test_bad_pathlike(self): self.assertRaises(ZeroDivisionError, self.fspath, FakePath(ZeroDivisionError())) + def test_pathlike_subclasshook(self): + # bpo-38878: subclasshook causes subclass checks + # true on abstract implementation. + class A(os.PathLike): + pass + self.assertFalse(issubclass(FakePath, A)) + self.assertTrue(issubclass(FakePath, os.PathLike)) + class TimesTests(unittest.TestCase): def test_times(self): diff --git a/Misc/NEWS.d/next/Library/2019-11-22-12-08-52.bpo-38878.EJ0cFf.rst b/Misc/NEWS.d/next/Library/2019-11-22-12-08-52.bpo-38878.EJ0cFf.rst new file mode 100644 index 00000000000000..9cbdf08dd53e31 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-11-22-12-08-52.bpo-38878.EJ0cFf.rst @@ -0,0 +1,2 @@ +Fixed __subclasshook__ of :class:`os.PathLike` to return a correct result +upon inheritence. Patch by Bar Harel. From 4ed79b78002d1c1bca8ea72aaadea07710aca4e9 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 24 Dec 2019 03:04:12 -0800 Subject: [PATCH 1032/2163] Fix import path for asyncio.TimeoutError (GH-17691) (cherry picked from commit 025eeaa19607b2a80c979668dad405f567444573) Co-authored-by: Andrew Svetlov --- Lib/asyncio/staggered.py | 4 ++-- .../next/Library/2019-12-24-10-43-13.bpo-39129.jVx5rW.rst | 1 + 2 files changed, 3 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-24-10-43-13.bpo-39129.jVx5rW.rst diff --git a/Lib/asyncio/staggered.py b/Lib/asyncio/staggered.py index 27c665a9910ab2..451a53a16f3831 100644 --- a/Lib/asyncio/staggered.py +++ b/Lib/asyncio/staggered.py @@ -6,7 +6,7 @@ import typing from . import events -from . import futures +from . import exceptions as exceptions_mod from . import locks from . import tasks @@ -83,7 +83,7 @@ async def run_one_coro( previous_failed: typing.Optional[locks.Event]) -> None: # Wait for the previous task to finish, or for delay seconds if previous_failed is not None: - with contextlib.suppress(futures.TimeoutError): + with contextlib.suppress(exceptions_mod.TimeoutError): # Use asyncio.wait_for() instead of asyncio.wait() here, so # that if we get cancelled at this point, Event.wait() is also # cancelled, otherwise there will be a "Task destroyed but it is diff --git a/Misc/NEWS.d/next/Library/2019-12-24-10-43-13.bpo-39129.jVx5rW.rst b/Misc/NEWS.d/next/Library/2019-12-24-10-43-13.bpo-39129.jVx5rW.rst new file mode 100644 index 00000000000000..6667697671a280 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-24-10-43-13.bpo-39129.jVx5rW.rst @@ -0,0 +1 @@ +Fix import path for ``asyncio.TimeoutError`` From e9039f32b3e8b827106cf74db80fbf9477035687 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 24 Dec 2019 20:35:01 -0800 Subject: [PATCH 1033/2163] Minor C API documentation improvements. (GH-17697) The added parentheses around the PyIter_Next assignment suppress the following warning which gcc throws without: ``` warning: using the result of an assignment as a condition without parentheses [-Wparentheses] ``` The other change is a typo fix (cherry picked from commit 5c7ed7550ec2da16d7679e538fcd7c1a5631811f) Co-authored-by: William Ayd --- Doc/c-api/iter.rst | 2 +- Doc/includes/custom.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/c-api/iter.rst b/Doc/c-api/iter.rst index 0224d37f1a41a0..a2992b3452f91c 100644 --- a/Doc/c-api/iter.rst +++ b/Doc/c-api/iter.rst @@ -29,7 +29,7 @@ something like this:: /* propagate error */ } - while (item = PyIter_Next(iterator)) { + while ((item = PyIter_Next(iterator))) { /* do something with item */ ... /* release reference when done */ diff --git a/Doc/includes/custom.c b/Doc/includes/custom.c index bda32e2ad81d46..f361baf830dd1b 100644 --- a/Doc/includes/custom.c +++ b/Doc/includes/custom.c @@ -37,7 +37,7 @@ PyInit_custom(void) Py_INCREF(&CustomType); if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) < 0) { Py_DECREF(&CustomType); - PY_DECREF(m); + Py_DECREF(m); return NULL; } From 8c0f0016e2d39f86589187db75012238c4e760e9 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 24 Dec 2019 20:42:24 -0800 Subject: [PATCH 1034/2163] bpo-38753: AsyncMock added in version 3.8 (GH-17102) (cherry picked from commit 279d8df5e5e8bbd4429420649359f7afcb4c8cce) Co-authored-by: John Belmonte --- Doc/library/unittest.mock.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index ebfaf7d643956a..e92f5545d3eb05 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -915,6 +915,7 @@ object:: >>> mock.async_foo + .. versionadded:: 3.8 .. method:: assert_awaited() From ff67612fa3c947c1dcc0a4cb0b55ee472b7edc66 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 25 Dec 2019 07:45:52 -0800 Subject: [PATCH 1035/2163] Fix the miscellaneous typo (GH-17700) A character "i" is omitted. (cherry picked from commit 527f9de6efdcf09983d0764be0b978ddc1fd1653) Co-authored-by: cocoatomo --- Doc/c-api/code.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst index 45a6b4a753a729..6f8c41ccbf6e85 100644 --- a/Doc/c-api/code.rst +++ b/Doc/c-api/code.rst @@ -42,7 +42,7 @@ bound into a function. .. c:function:: PyCodeObject* PyCode_NewWithPosOnlyArgs(int argcount, int posonlyargcount, int kwonlyargcount, int nlocals, int stacksize, int flags, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, int firstlineno, PyObject *lnotab) - Similar to :c:func:`PyCode_New`, but with an extra "posonlyargcount" for positonal-only arguments. + Similar to :c:func:`PyCode_New`, but with an extra "posonlyargcount" for positional-only arguments. .. versionadded:: 3.8 From a5deabd8e43a76bf2d7923342b6c4395c1a5efcd Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 25 Dec 2019 14:51:22 -0800 Subject: [PATCH 1036/2163] bpo-33961: Adjusted dataclasses docs to correct exceptions raised. (GH-7917) (GH-17677) (cherry picked from commit e28aff54d95236ea1b64b648d89a1516e446e621) Co-authored-by: Fabio Sangiovanni <4040184+sanjioh@users.noreply.github.com> --- Doc/library/dataclasses.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/Doc/library/dataclasses.rst b/Doc/library/dataclasses.rst index 9aa4a19d57ac74..37258d4ebc7386 100644 --- a/Doc/library/dataclasses.rst +++ b/Doc/library/dataclasses.rst @@ -60,8 +60,9 @@ Module-level decorators, classes, and functions The :func:`dataclass` decorator will add various "dunder" methods to the class, described below. If any of the added methods already - exist on the class, a :exc:`TypeError` will be raised. The decorator - returns the same class that is called on: no new class is created. + exist on the class, the behavior depends on the parameter, as documented + below. The decorator returns the same class that is called on; no new + class is created. If :func:`dataclass` is used just as a simple decorator with no parameters, it acts as if it has the default values documented in this @@ -115,7 +116,7 @@ Module-level decorators, classes, and functions If the class already defines any of :meth:`__lt__`, :meth:`__le__`, :meth:`__gt__`, or :meth:`__ge__`, then - :exc:`ValueError` is raised. + :exc:`TypeError` is raised. - ``unsafe_hash``: If ``False`` (the default), a :meth:`__hash__` method is generated according to how ``eq`` and ``frozen`` are set. From b9b92d34646d6a1fca120f98476912f5e4ca7f34 Mon Sep 17 00:00:00 2001 From: Benjamin Peterson Date: Thu, 26 Dec 2019 21:07:21 -0600 Subject: [PATCH 1037/2163] [3.8] closes bpo-39135: Remove 'time.clock()' mention in docs. (GH-17713) `time.clock()` was removed in Python 3.8, but it was still mentioned in the documentation for when `time.get_clock_info()` is given the argument `'clock'`. This commit removes that mention. (cherry picked from commit 91874bb07161bb481b6f5ea18ffafe69cb8cac30) Co-authored-by: Michael Wayne Goodman --- Doc/library/time.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/Doc/library/time.rst b/Doc/library/time.rst index e628ac44e80643..6842e9075e1a68 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -218,7 +218,6 @@ Functions Supported clock names and the corresponding functions to read their value are: - * ``'clock'``: :func:`time.clock` * ``'monotonic'``: :func:`time.monotonic` * ``'perf_counter'``: :func:`time.perf_counter` * ``'process_time'``: :func:`time.process_time` From 5dd1909680a5be76397b19bc235a8205b79787c7 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 26 Dec 2019 19:10:52 -0800 Subject: [PATCH 1038/2163] closes bpo-30364: Replace deprecated no_address_safety_analysis attribute. (GH-17702) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit (cherry picked from commit c0052f3fe3d19820b2d4f76e383035439affe32c) Co-authored-by: Batuhan Taşkaya <47358913+isidentical@users.noreply.github.com> --- Objects/obmalloc.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c index bd6480a625e8a8..c483ff3e3b7018 100644 --- a/Objects/obmalloc.c +++ b/Objects/obmalloc.c @@ -31,8 +31,8 @@ static void _PyMem_SetupDebugHooksDomain(PyMemAllocatorDomain domain); #if defined(__has_feature) /* Clang */ # if __has_feature(address_sanitizer) /* is ASAN enabled? */ -# define _Py_NO_ADDRESS_SAFETY_ANALYSIS \ - __attribute__((no_address_safety_analysis)) +# define _Py_NO_SANITIZE_ADDRESS \ + __attribute__((no_sanitize("address"))) # endif # if __has_feature(thread_sanitizer) /* is TSAN enabled? */ # define _Py_NO_SANITIZE_THREAD __attribute__((no_sanitize_thread)) @@ -42,8 +42,8 @@ static void _PyMem_SetupDebugHooksDomain(PyMemAllocatorDomain domain); # endif #elif defined(__GNUC__) # if defined(__SANITIZE_ADDRESS__) /* GCC 4.8+, is ASAN enabled? */ -# define _Py_NO_ADDRESS_SAFETY_ANALYSIS \ - __attribute__((no_address_safety_analysis)) +# define _Py_NO_SANITIZE_ADDRESS \ + __attribute__((no_sanitize_address)) # endif // TSAN is supported since GCC 5.1, but __SANITIZE_THREAD__ macro // is provided only since GCC 7. @@ -52,8 +52,8 @@ static void _PyMem_SetupDebugHooksDomain(PyMemAllocatorDomain domain); # endif #endif -#ifndef _Py_NO_ADDRESS_SAFETY_ANALYSIS -# define _Py_NO_ADDRESS_SAFETY_ANALYSIS +#ifndef _Py_NO_SANITIZE_ADDRESS +# define _Py_NO_SANITIZE_ADDRESS #endif #ifndef _Py_NO_SANITIZE_THREAD # define _Py_NO_SANITIZE_THREAD @@ -1388,7 +1388,7 @@ obmalloc controls. Since this test is needed at every entry point, it's extremely desirable that it be this fast. */ -static bool _Py_NO_ADDRESS_SAFETY_ANALYSIS +static bool _Py_NO_SANITIZE_ADDRESS _Py_NO_SANITIZE_THREAD _Py_NO_SANITIZE_MEMORY address_in_range(void *p, poolp pool) From 2786fdec79c35b4a68afea2bbbedbba3b6eb2269 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 27 Dec 2019 18:50:34 -0800 Subject: [PATCH 1039/2163] bpo-39144 Align ctags and etags behaviours in the makefile and include Python stdlib files (GH-17721) (cherry picked from commit ef7eaafc9d2e370cf79b3674e56f643bbfe239e2) Co-authored-by: Anthony Shaw --- Makefile.pre.in | 3 +++ .../NEWS.d/next/Build/2019-12-27-22-18-26.bpo-39144.dwHMlR.rst | 1 + 2 files changed, 4 insertions(+) create mode 100644 Misc/NEWS.d/next/Build/2019-12-27-22-18-26.bpo-39144.dwHMlR.rst diff --git a/Makefile.pre.in b/Makefile.pre.in index 502317aa0c783a..a914a9c70f6794 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1725,6 +1725,7 @@ tags:: ctags -w $(srcdir)/Include/*.h $(srcdir)/Include/cpython/*.h $(srcdir)/Include/internal/*.h for i in $(SRCDIRS); do ctags -f tags -w -a $(srcdir)/$$i/*.[ch]; done ctags -f tags -w -a $(srcdir)/Modules/_ctypes/*.[ch] + find $(srcdir)/Lib -type f -name "*.py" -not -name "test_*.py" -not -path "*/test/*" -not -path "*/tests/*" -not -path "*/*_test/*" | ctags -f tags -w -a -L - LC_ALL=C sort -o tags tags # Create a tags file for GNU Emacs @@ -1732,6 +1733,8 @@ TAGS:: cd $(srcdir); \ etags Include/*.h Include/cpython/*.h Include/internal/*.h; \ for i in $(SRCDIRS); do etags -a $$i/*.[ch]; done + etags -a $(srcdir)/Modules/_ctypes/*.[ch] + find $(srcdir)/Lib -type f -name "*.py" -not -name "test_*.py" -not -path "*/test/*" -not -path "*/tests/*" -not -path "*/*_test/*" | etags - -a # Sanitation targets -- clean leaves libraries, executables and tags # files, which clobber removes as well diff --git a/Misc/NEWS.d/next/Build/2019-12-27-22-18-26.bpo-39144.dwHMlR.rst b/Misc/NEWS.d/next/Build/2019-12-27-22-18-26.bpo-39144.dwHMlR.rst new file mode 100644 index 00000000000000..8b90da19622e61 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2019-12-27-22-18-26.bpo-39144.dwHMlR.rst @@ -0,0 +1 @@ +The ctags and etags build targets both include Modules/_ctypes and Python standard library source files. \ No newline at end of file From 04c1efe5acc89b6f3f2c208604a1c35641e8a380 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 27 Dec 2019 18:58:47 -0800 Subject: [PATCH 1040/2163] bpo-38731: Fix function signature of quiet in docs (GH-17719) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit (cherry picked from commit 98f0f04b5016e63561d313a3446b7b58f2c12611) Co-authored-by: Batuhan Taşkaya <47358913+isidentical@users.noreply.github.com> --- Doc/library/py_compile.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/py_compile.rst b/Doc/library/py_compile.rst index 3824353abda1f8..a12a5bb0b1aa2c 100644 --- a/Doc/library/py_compile.rst +++ b/Doc/library/py_compile.rst @@ -27,7 +27,7 @@ byte-code cache files in the directory containing the source code. Exception raised when an error occurs while attempting to compile the file. -.. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1, invalidation_mode=PycInvalidationMode.TIMESTAMP) +.. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1, invalidation_mode=PycInvalidationMode.TIMESTAMP, quiet=0) Compile a source file to byte-code and write out the byte-code cache file. The source code is loaded from the file named *file*. The byte-code is From 9ee1b156146cb8114ba6d24811bdec456bbf7b8f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=89ric=20Araujo?= Date: Sat, 28 Dec 2019 15:41:21 -0500 Subject: [PATCH 1041/2163] Reorder entries in Misc/ACKS (GH-17663) (GH-17673) --- Misc/ACKS | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS index a17db456576318..21ee85b4558ed1 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -12,6 +12,7 @@ PS: In the standard Python distribution, this file is encoded in UTF-8 and the list is in rough alphabetical order by last names. Aahz +Edison Abahurire Michael Abbott Rajiv Abraham David Abrahams @@ -82,6 +83,7 @@ Marcin Bachry Alfonso Baciero Dwayne Bailey Stig Bakken +Aleksandr Balezin Greg Ball Luigi Ballabio Thomas Ballinger @@ -551,6 +553,7 @@ Fred Gansevles Lars Marius Garshol Jake Garver Dan Gass +Tim Gates Andrew Gaul Matthieu Gautier Stephen M. Gava @@ -856,6 +859,7 @@ Kamil Kisiel Akira Kitada Ron Klatchko Reid Kleckner +Carsten Klein Bastian Kleineidam Bob Kline Matthias Klose @@ -948,6 +952,7 @@ Inyeol Lee James Lee John J. Lee Thomas Lee +Robert Leenders Cooper Ry Lees Yaron de Leeuw Tennessee Leeuwenburg @@ -979,6 +984,7 @@ Akira Li Robert Li Xuanji Li Zekun Li +Zheao Li Robert van Liere Ross Light Shawn Ligocki @@ -1397,6 +1403,7 @@ Giampaolo Rodola Mauro S. M. Rodrigues Elson Rodriguez Adi Roiban +Diego Rojas Luis Rojas Mike Romberg Armin Ronacher @@ -1506,6 +1513,7 @@ Dmitry Shachnev Anish Shah Daniel Shahaf Hui Shang +Geoff Shannon Mark Shannon Ha Shao Richard Shapiro @@ -1537,8 +1545,8 @@ Kirill Simonov Nathan Paul Simons Guilherme Simões Adam Simpkins -Ravi Sinha Mandeep Singh +Ravi Sinha Janne Sinkkonen Ng Pheng Siong Yann Sionneau @@ -1641,6 +1649,7 @@ Musashi Tamura William Tanksley Christian Tanzer Steven Taschuk +Batuhan Taskaya Amy Taylor Julian Taylor Monty Taylor @@ -1876,13 +1885,5 @@ Jelle Zijlstra Gennadiy Zlobin Doug Zongker Peter Åstrand -Zheao Li -Carsten Klein -Diego Rojas -Edison Abahurire -Geoff Shannon -Batuhan Taskaya -Aleksandr Balezin -Robert Leenders -Ngalim Siregar -Tim Gates + +(Entries should be added in rough alphabetical order by last names) From df647f3340d7a40628ba731f80425cb469e72653 Mon Sep 17 00:00:00 2001 From: Terry Jan Reedy Date: Sat, 28 Dec 2019 19:05:15 -0500 Subject: [PATCH 1042/2163] [3.8] bpo-39136: Fixed typos (GH-17720) funtion -> function; configuraton -> configuration; defintitions -> definitions; focusses -> focuses; necesarily -> necessarily; follwing -> following; Excape -> Escape, (cherry picked from commit 6c7bb38ff2799ac218e6df598b2b262f89e2bc1e) --- Doc/c-api/init.rst | 2 +- Doc/c-api/init_config.rst | 2 +- Doc/faq/programming.rst | 2 +- Lib/idlelib/NEWS.txt | 2 +- Lib/tkinter/tix.py | 4 ++-- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 155edba82f96df..81cb4f825fee7a 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1181,7 +1181,7 @@ It is usually the only Python interpreter in a process. Unlike sub-interpreters the main interpreter has unique process-global responsibilities like signal handling. It is also responsible for execution during runtime initialization and is usually the active interpreter during runtime finalization. The -:c:func:`PyInterpreterState_Main` funtion returns a pointer to its state. +:c:func:`PyInterpreterState_Main` function returns a pointer to its state. You can switch between sub-interpreters using the :c:func:`PyThreadState_Swap` function. You can create and destroy them using the following functions: diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst index 6b16b5bb72859a..79a8815ed4199d 100644 --- a/Doc/c-api/init_config.rst +++ b/Doc/c-api/init_config.rst @@ -757,7 +757,7 @@ configuration, and then override some parameters:: PyConfig config; PyConfig_InitPythonConfig(&config); - /* Set the program name before reading the configuraton + /* Set the program name before reading the configuration (decode byte string from the locale encoding). Implicitly preinitialize Python. */ diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 9d45765abaacf2..70b11d6e93056f 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -1019,7 +1019,7 @@ That's a tough one, in general. First, here are a list of things to remember before diving further: * Performance characteristics vary across Python implementations. This FAQ - focusses on :term:`CPython`. + focuses on :term:`CPython`. * Behaviour can vary across operating systems, especially when talking about I/O or multi-threading. * You should always find the hot spots in your program *before* attempting to diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index e829bc910ea877..0baec813b044d1 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -6,7 +6,7 @@ Released on 2019-12-16? bpo-38943: Fix autocomplete windows not always appearing on some systems. Patch by Johnny Najera. -bpo-38944: Excape key now closes IDLE completion windows. Patch by +bpo-38944: Escape key now closes IDLE completion windows. Patch by Johnny Najera. bpo-38862: 'Strip Trailing Whitespace' on the Format menu removes extra diff --git a/Lib/tkinter/tix.py b/Lib/tkinter/tix.py index d9c097a77cdd18..ac545502e45c30 100644 --- a/Lib/tkinter/tix.py +++ b/Lib/tkinter/tix.py @@ -1890,7 +1890,7 @@ def size_column(self, index, **kw): containing the current size setting of the given column. When option-value pairs are given, the corresponding options of the size setting of the given column are changed. Options may be one - of the follwing: + of the following: pad0 pixels Specifies the paddings to the left of a column. pad1 pixels @@ -1915,7 +1915,7 @@ def size_row(self, index, **kw): When no option-value pair is given, this command returns a list con- taining the current size setting of the given row . When option-value pairs are given, the corresponding options of the size setting of the - given row are changed. Options may be one of the follwing: + given row are changed. Options may be one of the following: pad0 pixels Specifies the paddings to the top of a row. pad1 pixels From e06be0820f3304b1ea940cd643214d332ed02ea8 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 29 Dec 2019 14:20:52 -0800 Subject: [PATCH 1043/2163] Fix typos and remove deprecated deprecation warning. (GH-17741) (cherry picked from commit 32a12aed6da41f49a5ca05e6de34f5f93ea1dc33) Co-authored-by: Antoine <43954001+awecx@users.noreply.github.com> --- Doc/library/ctypes.rst | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index e0bc28f5e5058c..2d6c6d0a1c3c57 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -161,13 +161,6 @@ as the ``NULL`` pointer):: 0x1d000000 >>> -.. note:: - - :mod:`ctypes` may raise a :exc:`ValueError` after calling the function, if - it detects that an invalid number of arguments were passed. This behavior - should not be relied upon. It is deprecated in 3.6.2, and will be removed - in 3.7. - :exc:`ValueError` is raised when you call an ``stdcall`` function with the ``cdecl`` calling convention, or vice versa:: @@ -624,7 +617,7 @@ Structure/union alignment and byte order ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ By default, Structure and Union fields are aligned in the same way the C -compiler does it. It is possible to override this behavior be specifying a +compiler does it. It is possible to override this behavior by specifying a :attr:`_pack_` class attribute in the subclass definition. This must be set to a positive integer and specifies the maximum alignment for the fields. This is what ``#pragma pack(n)`` also does in MSVC. @@ -922,7 +915,7 @@ attribute later, after the class statement:: ... ("next", POINTER(cell))] >>> -Lets try it. We create two instances of ``cell``, and let them point to each +Let's try it. We create two instances of ``cell``, and let them point to each other, and finally follow the pointer chain a few times:: >>> c1 = cell() @@ -1125,8 +1118,8 @@ hit the ``NULL`` entry:: >>> The fact that standard Python has a frozen module and a frozen package -(indicated by the negative size member) is not well known, it is only used for -testing. Try it out with ``import __hello__`` for example. +(indicated by the negative ``size`` member) is not well known, it is only used +for testing. Try it out with ``import __hello__`` for example. .. _ctypes-surprises: From cbfafa3e3625dae96ce392c88c793f8af55167bf Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 29 Dec 2019 21:31:18 -0800 Subject: [PATCH 1044/2163] bpo-39037: Fix lookup order of magic methods in with statement documentation (GH-17608) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * __enter__ is now looked up before __exit__ to give a more intuitive error message * add pseudo-code equivalent for the with statement * fix pseudo-code for the async with statement to use a finally clause * use SUITE rather than BLOCK for consistency with the language grammar Patch by Géry Ogam. (cherry picked from commit 226e6e7d4326cf91ef37e13528eb1f62de1bb832) Co-authored-by: Géry Ogam --- Doc/reference/compound_stmts.rst | 67 +++++++++++++++++++++++--------- 1 file changed, 49 insertions(+), 18 deletions(-) diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 988eec6d254e10..564d6cc42136da 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -399,6 +399,8 @@ The execution of the :keyword:`with` statement with one "item" proceeds as follo #. The context expression (the expression given in the :token:`with_item`) is evaluated to obtain a context manager. +#. The context manager's :meth:`__enter__` is loaded for later use. + #. The context manager's :meth:`__exit__` is loaded for later use. #. The context manager's :meth:`__enter__` method is invoked. @@ -430,17 +432,41 @@ The execution of the :keyword:`with` statement with one "item" proceeds as follo value from :meth:`__exit__` is ignored, and execution proceeds at the normal location for the kind of exit that was taken. +The following code:: + + with EXPRESSION as TARGET: + SUITE + +is semantically equivalent to:: + + manager = (EXPRESSION) + enter = type(manager).__enter__ + exit = type(manager).__exit__ + value = enter(manager) + hit_except = False + + try: + TARGET = value + SUITE + except: + hit_except = True + if not exit(manager, *sys.exc_info()): + raise + finally: + if not hit_except: + exit(manager, None, None, None) + With more than one item, the context managers are processed as if multiple :keyword:`with` statements were nested:: with A() as a, B() as b: - suite + SUITE -is equivalent to :: +is semantically equivalent to:: with A() as a: with B() as b: - suite + SUITE .. versionchanged:: 3.1 Support for multiple context expressions. @@ -772,24 +798,25 @@ iterators. The following code:: async for TARGET in ITER: - BLOCK + SUITE else: - BLOCK2 + SUITE2 Is semantically equivalent to:: iter = (ITER) iter = type(iter).__aiter__(iter) running = True + while running: try: TARGET = await type(iter).__anext__(iter) except StopAsyncIteration: running = False else: - BLOCK + SUITE else: - BLOCK2 + SUITE2 See also :meth:`__aiter__` and :meth:`__anext__` for details. @@ -811,23 +838,27 @@ able to suspend execution in its *enter* and *exit* methods. The following code:: - async with EXPR as VAR: - BLOCK + async with EXPRESSION as TARGET: + SUITE -Is semantically equivalent to:: +is semantically equivalent to:: - mgr = (EXPR) - aexit = type(mgr).__aexit__ - aenter = type(mgr).__aenter__(mgr) + manager = (EXPRESSION) + aexit = type(manager).__aexit__ + aenter = type(manager).__aenter__ + value = await aenter(manager) + hit_except = False - VAR = await aenter try: - BLOCK + TARGET = value + SUITE except: - if not await aexit(mgr, *sys.exc_info()): + hit_except = True + if not await aexit(manager, *sys.exc_info()): raise - else: - await aexit(mgr, None, None, None) + finally: + if not hit_except: + await aexit(manager, None, None, None) See also :meth:`__aenter__` and :meth:`__aexit__` for details. From fcaf14cd9179bb48850f8f81ce8d5cee28129745 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 30 Dec 2019 11:51:06 -0800 Subject: [PATCH 1045/2163] bpo-38610: Fix possible crashes in several list methods (GH-17022) Hold strong references to list elements while calling PyObject_RichCompareBool(). (cherry picked from commit d9e561d23d994e3ed15f4fcbd7ee5c8fe50f190b) Co-authored-by: Zackery Spytz --- Lib/test/test_list.py | 26 +++++++++++++++++++ .../2019-10-31-14-30-39.bpo-38610.fHdVMS.rst | 2 ++ Objects/listobject.c | 15 ++++++++--- 3 files changed, 40 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-10-31-14-30-39.bpo-38610.fHdVMS.rst diff --git a/Lib/test/test_list.py b/Lib/test/test_list.py index c5002b12732c98..55c6eede527c2a 100644 --- a/Lib/test/test_list.py +++ b/Lib/test/test_list.py @@ -166,5 +166,31 @@ def test_preallocation(self): self.assertEqual(iter_size, sys.getsizeof(list([0] * 10))) self.assertEqual(iter_size, sys.getsizeof(list(range(10)))) + def test_count_index_remove_crashes(self): + # bpo-38610: The count(), index(), and remove() methods were not + # holding strong references to list elements while calling + # PyObject_RichCompareBool(). + class X: + def __eq__(self, other): + lst.clear() + return NotImplemented + + lst = [X()] + with self.assertRaises(ValueError): + lst.index(lst) + + class L(list): + def __eq__(self, other): + str(other) + return NotImplemented + + lst = L([X()]) + lst.count(lst) + + lst = L([X()]) + with self.assertRaises(ValueError): + lst.remove(lst) + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-31-14-30-39.bpo-38610.fHdVMS.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-31-14-30-39.bpo-38610.fHdVMS.rst new file mode 100644 index 00000000000000..0ee63bbb40dc66 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-31-14-30-39.bpo-38610.fHdVMS.rst @@ -0,0 +1,2 @@ +Fix possible crashes in several list methods by holding strong references to +list elements when calling :c:func:`PyObject_RichCompareBool`. diff --git a/Objects/listobject.c b/Objects/listobject.c index f8bf45e5f8cda2..8b52fa5ef267ab 100644 --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -2555,7 +2555,10 @@ list_index_impl(PyListObject *self, PyObject *value, Py_ssize_t start, stop = 0; } for (i = start; i < stop && i < Py_SIZE(self); i++) { - int cmp = PyObject_RichCompareBool(self->ob_item[i], value, Py_EQ); + PyObject *obj = self->ob_item[i]; + Py_INCREF(obj); + int cmp = PyObject_RichCompareBool(obj, value, Py_EQ); + Py_DECREF(obj); if (cmp > 0) return PyLong_FromSsize_t(i); else if (cmp < 0) @@ -2582,7 +2585,10 @@ list_count(PyListObject *self, PyObject *value) Py_ssize_t i; for (i = 0; i < Py_SIZE(self); i++) { - int cmp = PyObject_RichCompareBool(self->ob_item[i], value, Py_EQ); + PyObject *obj = self->ob_item[i]; + Py_INCREF(obj); + int cmp = PyObject_RichCompareBool(obj, value, Py_EQ); + Py_DECREF(obj); if (cmp > 0) count++; else if (cmp < 0) @@ -2609,7 +2615,10 @@ list_remove(PyListObject *self, PyObject *value) Py_ssize_t i; for (i = 0; i < Py_SIZE(self); i++) { - int cmp = PyObject_RichCompareBool(self->ob_item[i], value, Py_EQ); + PyObject *obj = self->ob_item[i]; + Py_INCREF(obj); + int cmp = PyObject_RichCompareBool(obj, value, Py_EQ); + Py_DECREF(obj); if (cmp > 0) { if (list_ass_slice(self, i, i+1, (PyObject *)NULL) == 0) From ec941568bdb25e164a87a23cf1b8870ac047b4e3 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 30 Dec 2019 14:24:27 -0800 Subject: [PATCH 1046/2163] bpo-34118: memoryview, range, and tuple are classes (GH-17761) Tag memoryview, range, and tuple as classes, the same as list, etcetera, in the library manual built-in functions list. (cherry picked from commit ee9ff05ec22ecd47dbffdd361967ccd55963dad2) Co-authored-by: Terry Jan Reedy --- Doc/library/functions.rst | 6 +++--- .../next/IDLE/2019-12-30-16-44-07.bpo-34118.FaNW0a.rst | 2 ++ 2 files changed, 5 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/IDLE/2019-12-30-16-44-07.bpo-34118.FaNW0a.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 7968eef4fb96bd..6feb26c4f25a65 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -950,7 +950,7 @@ are always available. They are listed here in alphabetical order. .. _func-memoryview: -.. function:: memoryview(obj) +.. class:: memoryview(obj) :noindex: Return a "memory view" object created from the given argument. See @@ -1412,7 +1412,7 @@ are always available. They are listed here in alphabetical order. .. _func-range: -.. function:: range(stop) +.. class:: range(stop) range(start, stop[, step]) :noindex: @@ -1659,7 +1659,7 @@ are always available. They are listed here in alphabetical order. .. _func-tuple: -.. function:: tuple([iterable]) +.. class:: tuple([iterable]) :noindex: Rather than being a function, :class:`tuple` is actually an immutable diff --git a/Misc/NEWS.d/next/IDLE/2019-12-30-16-44-07.bpo-34118.FaNW0a.rst b/Misc/NEWS.d/next/IDLE/2019-12-30-16-44-07.bpo-34118.FaNW0a.rst new file mode 100644 index 00000000000000..ce95eb5482f2bf --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2019-12-30-16-44-07.bpo-34118.FaNW0a.rst @@ -0,0 +1,2 @@ +Tag memoryview, range, and tuple as classes, the same as list, etcetera, in +the library manual built-in functions list. From a278ae19b4daa1deb11e2a8eed38838027e90ece Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 30 Dec 2019 18:51:18 -0800 Subject: [PATCH 1047/2163] closes bpo-37446: resolve undefined behavior in Python/hamt.c (GH-17727) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit (cherry picked from commit d0c92e81aa2171228a23cb2bed36f7dab975257d) Co-authored-by: Batuhan Taşkaya <47358913+isidentical@users.noreply.github.com> --- Python/hamt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Python/hamt.c b/Python/hamt.c index 38412596a37b02..5efc8d7fabe8e3 100644 --- a/Python/hamt.c +++ b/Python/hamt.c @@ -1864,7 +1864,7 @@ hamt_node_array_without(PyHamtNode_Array *self, continue; } - bitmap |= 1 << i; + bitmap |= 1U << i; if (IS_BITMAP_NODE(node)) { PyHamtNode_Bitmap *child = (PyHamtNode_Bitmap *)node; From 2ee87913dde038436a25f1db13ee3fddd2bcc983 Mon Sep 17 00:00:00 2001 From: Dong-hee Na Date: Tue, 31 Dec 2019 13:15:36 +0900 Subject: [PATCH 1048/2163] =?UTF-8?q?[3.8]=20bpo-38588:=20Fix=20possible?= =?UTF-8?q?=20crashes=20in=20dict=20and=20list=20when=20calling=20P?= =?UTF-8?q?=E2=80=A6=20(GH-17764)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * [3.8] bpo-38588: Fix possible crashes in dict and list when calling PyObject_RichCompareBool (GH-17734) Take strong references before calling PyObject_RichCompareBool to protect against the case where the object dies during the call. (cherry picked from commit 2d5bf568eaa5059402ccce9ba5a366986ba27c8a) Co-authored-by: Dong-hee Na * Update Objects/listobject.c @methane's suggestion Co-Authored-By: Inada Naoki Co-authored-by: Inada Naoki --- Lib/test/test_dict.py | 12 ++++++++- Lib/test/test_list.py | 25 +++++++++++++++++++ .../2019-12-29-19-13-54.bpo-38588.pgXnNS.rst | 2 ++ Objects/dictobject.c | 2 ++ Objects/listobject.c | 10 ++++++++ 5 files changed, 50 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-12-29-19-13-54.bpo-38588.pgXnNS.rst diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py index 5b513765f7b08a..de483ab552155a 100644 --- a/Lib/test/test_dict.py +++ b/Lib/test/test_dict.py @@ -1221,7 +1221,7 @@ def test_free_after_iterating(self): support.check_free_after_iterating(self, lambda d: iter(d.items()), dict) def test_equal_operator_modifying_operand(self): - # test fix for seg fault reported in issue 27945 part 3. + # test fix for seg fault reported in bpo-27945 part 3. class X(): def __del__(self): dict_b.clear() @@ -1237,6 +1237,16 @@ def __hash__(self): dict_b = {X(): X()} self.assertTrue(dict_a == dict_b) + # test fix for seg fault reported in bpo-38588 part 1. + class Y: + def __eq__(self, other): + dict_d.clear() + return True + + dict_c = {0: Y()} + dict_d = {0: set()} + self.assertTrue(dict_c == dict_d) + def test_fromkeys_operator_modifying_dict_operand(self): # test fix for seg fault reported in issue 27945 part 4a. class X(int): diff --git a/Lib/test/test_list.py b/Lib/test/test_list.py index 55c6eede527c2a..f4dcced9c167f8 100644 --- a/Lib/test/test_list.py +++ b/Lib/test/test_list.py @@ -158,6 +158,31 @@ class L(list): pass with self.assertRaises(TypeError): (3,) + L([1,2]) + def test_equal_operator_modifying_operand(self): + # test fix for seg fault reported in bpo-38588 part 2. + class X: + def __eq__(self,other) : + list2.clear() + return NotImplemented + + class Y: + def __eq__(self, other): + list1.clear() + return NotImplemented + + class Z: + def __eq__(self, other): + list3.clear() + return NotImplemented + + list1 = [X()] + list2 = [Y()] + self.assertTrue(list1 == list2) + + list3 = [Z()] + list4 = [1] + self.assertFalse(list3 == list4) + @cpython_only def test_preallocation(self): iterable = [0] * 10 diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-12-29-19-13-54.bpo-38588.pgXnNS.rst b/Misc/NEWS.d/next/Core and Builtins/2019-12-29-19-13-54.bpo-38588.pgXnNS.rst new file mode 100644 index 00000000000000..0b81085a89d254 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2019-12-29-19-13-54.bpo-38588.pgXnNS.rst @@ -0,0 +1,2 @@ +Fix possible crashes in dict and list when calling +:c:func:`PyObject_RichCompareBool`. diff --git a/Objects/dictobject.c b/Objects/dictobject.c index 76f4fefe0979ff..3c56f4a515e8a4 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -2773,9 +2773,11 @@ dict_equal(PyDictObject *a, PyDictObject *b) return -1; return 0; } + Py_INCREF(bval); cmp = PyObject_RichCompareBool(aval, bval, Py_EQ); Py_DECREF(key); Py_DECREF(aval); + Py_DECREF(bval); if (cmp <= 0) /* error or not equal */ return cmp; } diff --git a/Objects/listobject.c b/Objects/listobject.c index 8b52fa5ef267ab..d506c0817336b2 100644 --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -2664,8 +2664,18 @@ list_richcompare(PyObject *v, PyObject *w, int op) /* Search for the first index where items are different */ for (i = 0; i < Py_SIZE(vl) && i < Py_SIZE(wl); i++) { + PyObject *vitem = vl->ob_item[i]; + PyObject *witem = wl->ob_item[i]; + if (vitem == witem) { + continue; + } + + Py_INCREF(vitem); + Py_INCREF(witem); int k = PyObject_RichCompareBool(vl->ob_item[i], wl->ob_item[i], Py_EQ); + Py_DECREF(vitem); + Py_DECREF(witem); if (k < 0) return NULL; if (!k) From 302b35f82bdaac817966638e2630b452f4123958 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 31 Dec 2019 10:52:38 -0800 Subject: [PATCH 1049/2163] Fix idlelib README typo. (GH-17770) (cherry picked from commit ba82ee894cf0f6ec9e9f6a313c870ffd2db377e6) Co-authored-by: Terry Jan Reedy --- Lib/idlelib/README.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Lib/idlelib/README.txt b/Lib/idlelib/README.txt index 48a1f4a425c9e3..bc3d978f43f1ad 100644 --- a/Lib/idlelib/README.txt +++ b/Lib/idlelib/README.txt @@ -90,7 +90,7 @@ Configuration config-extensions.def # Defaults for extensions config-highlight.def # Defaults for colorizing config-keys.def # Defaults for key bindings -config-main.def # Defai;ts fpr font and geneal +config-main.def # Defaults for font and general tabs Text ---- From 6c004955aceb8a0cd8e14afbc608ebfdf7c8aa4a Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 31 Dec 2019 19:28:08 -0800 Subject: [PATCH 1050/2163] bpo-39176: Improve error message for 'named assignment' (GH-17777) (GH-17778) (cherry picked from commit 37143a8e3b2e9245d52f4ddebbdd1c6121c96884) Co-authored-by: Ned Batchelder Co-authored-by: Ned Batchelder --- Lib/test/test_named_expressions.py | 4 ++-- Lib/test/test_syntax.py | 2 +- Python/ast.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_named_expressions.py b/Lib/test/test_named_expressions.py index 01e26c8dfaf259..3ae557f78d273a 100644 --- a/Lib/test/test_named_expressions.py +++ b/Lib/test/test_named_expressions.py @@ -32,7 +32,7 @@ def test_named_expression_invalid_04(self): def test_named_expression_invalid_06(self): code = """((a, b) := (1, 2))""" - with self.assertRaisesRegex(SyntaxError, "cannot use named assignment with tuple"): + with self.assertRaisesRegex(SyntaxError, "cannot use assignment expressions with tuple"): exec(code, {}, {}) def test_named_expression_invalid_07(self): @@ -90,7 +90,7 @@ def test_named_expression_invalid_15(self): code = """(lambda: x := 1)""" with self.assertRaisesRegex(SyntaxError, - "cannot use named assignment with lambda"): + "cannot use assignment expressions with lambda"): exec(code, {}, {}) def test_named_expression_invalid_16(self): diff --git a/Lib/test/test_syntax.py b/Lib/test/test_syntax.py index 3829746f1799a2..128c4da1438414 100644 --- a/Lib/test/test_syntax.py +++ b/Lib/test/test_syntax.py @@ -45,7 +45,7 @@ >>> (True := 1) Traceback (most recent call last): -SyntaxError: cannot use named assignment with True +SyntaxError: cannot use assignment expressions with True >>> obj.__debug__ = 1 Traceback (most recent call last): diff --git a/Python/ast.c b/Python/ast.c index e70ab519e66e1b..6cf71ce7bb8104 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -1955,7 +1955,7 @@ ast_for_namedexpr(struct compiling *c, const node *n) if (target->kind != Name_kind) { const char *expr_name = get_expr_name(target); if (expr_name != NULL) { - ast_error(c, n, "cannot use named assignment with %s", expr_name); + ast_error(c, n, "cannot use assignment expressions with %s", expr_name); } return NULL; } From dad526eb36530186f625a2724c8835fe3df3c8c1 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 31 Dec 2019 22:27:56 -0800 Subject: [PATCH 1051/2163] Document CodeType.replace (GH-17776) (cherry picked from commit 22424c02e51fab3b62cbe255d0b87d1b55b9a6c3) Co-authored-by: Anthony Sottile --- Doc/library/types.rst | 8 +++++++- Objects/clinic/codeobject.c.h | 4 ++-- Objects/codeobject.c | 4 ++-- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 9393f9e6db9909..3529c2b0edb896 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -132,7 +132,7 @@ Standard names are defined for the following types: .. versionadded:: 3.6 -.. data:: CodeType +.. class:: CodeType(**kwargs) .. index:: builtin: compile @@ -143,6 +143,12 @@ Standard names are defined for the following types: Note that the audited arguments may not match the names or positions required by the initializer. + .. method:: CodeType.replace(**kwargs) + + Return a copy of the code object with new values for the specified fields. + + .. versionadded:: 3.8 + .. data:: CellType The type for cell objects: such objects are used as containers for diff --git a/Objects/clinic/codeobject.c.h b/Objects/clinic/codeobject.c.h index 6596de051cacb3..1dd82278cf3d52 100644 --- a/Objects/clinic/codeobject.c.h +++ b/Objects/clinic/codeobject.c.h @@ -11,7 +11,7 @@ PyDoc_STRVAR(code_replace__doc__, " co_lnotab=None)\n" "--\n" "\n" -"Return a new code object with new specified fields."); +"Return a copy of the code object with new values for the specified fields."); #define CODE_REPLACE_METHODDEF \ {"replace", (PyCFunction)(void(*)(void))code_replace, METH_FASTCALL|METH_KEYWORDS, code_replace__doc__}, @@ -253,4 +253,4 @@ code_replace(PyCodeObject *self, PyObject *const *args, Py_ssize_t nargs, PyObje exit: return return_value; } -/*[clinic end generated code: output=fade581d6313a0c2 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=27fe34e82106b220 input=a9049054013a1b77]*/ diff --git a/Objects/codeobject.c b/Objects/codeobject.c index f0b62ec94148d6..522e1a9f2a4195 100644 --- a/Objects/codeobject.c +++ b/Objects/codeobject.c @@ -610,7 +610,7 @@ code.replace co_name: unicode(c_default="self->co_name") = None co_lnotab: PyBytesObject(c_default="(PyBytesObject *)self->co_lnotab") = None -Return a new code object with new specified fields. +Return a copy of the code object with new values for the specified fields. [clinic start generated code]*/ static PyObject * @@ -622,7 +622,7 @@ code_replace_impl(PyCodeObject *self, int co_argcount, PyObject *co_varnames, PyObject *co_freevars, PyObject *co_cellvars, PyObject *co_filename, PyObject *co_name, PyBytesObject *co_lnotab) -/*[clinic end generated code: output=25c8e303913bcace input=77189e46579ec426]*/ +/*[clinic end generated code: output=25c8e303913bcace input=d9051bc8f24e6b28]*/ { #define CHECK_INT_ARG(ARG) \ if (ARG < 0) { \ From 1d5a7e5694bd9104f56f4f28357c2d13afd58a29 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 1 Jan 2020 12:06:52 -0800 Subject: [PATCH 1052/2163] bpo-39142: Avoid converting namedtuple instances to ConvertingTuple. (GH-17773) (GH-17785) (cherry picked from commit 46abfc1416ff8e450999611ef8f231ff871ab133) --- Lib/logging/config.py | 2 +- Lib/test/test_logging.py | 31 +++++++++++++++++++ .../2019-12-31-19-27-23.bpo-39142.oqU5iD.rst | 5 +++ 3 files changed, 37 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-31-19-27-23.bpo-39142.oqU5iD.rst diff --git a/Lib/logging/config.py b/Lib/logging/config.py index 9dd35e11aab395..3cd5fea8524494 100644 --- a/Lib/logging/config.py +++ b/Lib/logging/config.py @@ -447,7 +447,7 @@ def convert(self, value): value = ConvertingList(value) value.configurator = self elif not isinstance(value, ConvertingTuple) and\ - isinstance(value, tuple): + isinstance(value, tuple) and not hasattr(value, '_fields'): value = ConvertingTuple(value) value.configurator = self elif isinstance(value, str): # str for py3k diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index 37655a5ccc40c6..90bf2a4d3ac06f 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -3361,6 +3361,37 @@ def test_baseconfig(self): self.assertRaises(ValueError, bc.convert, 'cfg://!') self.assertRaises(KeyError, bc.convert, 'cfg://adict[2]') + def test_namedtuple(self): + # see bpo-39142 + from collections import namedtuple + + class MyHandler(logging.StreamHandler): + def __init__(self, resource, *args, **kwargs): + super().__init__(*args, **kwargs) + self.resource: namedtuple = resource + + def emit(self, record): + record.msg += f' {self.resource.type}' + return super().emit(record) + + Resource = namedtuple('Resource', ['type', 'labels']) + resource = Resource(type='my_type', labels=['a']) + + config = { + 'version': 1, + 'handlers': { + 'myhandler': { + '()': MyHandler, + 'resource': resource + } + }, + 'root': {'level': 'INFO', 'handlers': ['myhandler']}, + } + with support.captured_stderr() as stderr: + self.apply_config(config) + logging.info('some log') + self.assertEqual(stderr.getvalue(), 'some log my_type\n') + class ManagerTest(BaseTest): def test_manager_loggerclass(self): logged = [] diff --git a/Misc/NEWS.d/next/Library/2019-12-31-19-27-23.bpo-39142.oqU5iD.rst b/Misc/NEWS.d/next/Library/2019-12-31-19-27-23.bpo-39142.oqU5iD.rst new file mode 100644 index 00000000000000..508d1338d7c317 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-31-19-27-23.bpo-39142.oqU5iD.rst @@ -0,0 +1,5 @@ +A change was made to logging.config.dictConfig to avoid converting instances +of named tuples to ConvertingTuple. It's assumed that named tuples are too +specialised to be treated like ordinary tuples; if a user of named tuples +requires ConvertingTuple functionality, they will have to implement that +themselves in their named tuple class. From 6bf382ac9a07f42c6f91f35a7f0df4c63e35a8a7 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 1 Jan 2020 14:32:23 -0800 Subject: [PATCH 1053/2163] bpo-39183: Fix formatting in library/ensurepip (GH-17787) Remove extra space to fix formatting and avoid from splitting text in to strings. https://bugs.python.org/issue39183 (cherry picked from commit 149175c6dfc8455023e4335575f3fe3d606729f9) Co-authored-by: Rafael Fontenelle --- Doc/library/ensurepip.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/ensurepip.rst b/Doc/library/ensurepip.rst index a2bb045e57e3c0..a5221250c40486 100644 --- a/Doc/library/ensurepip.rst +++ b/Doc/library/ensurepip.rst @@ -74,7 +74,7 @@ options: script will *not* be installed. * ``--default-pip``: if a "default pip" installation is requested, the - ``pip`` script will be installed in addition to the two regular scripts. + ``pip`` script will be installed in addition to the two regular scripts. Providing both of the script selection options will trigger an exception. From 10dc738bdac7ef4e2ba0785f65c9cb4696aa241d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 2 Jan 2020 19:15:37 -0800 Subject: [PATCH 1054/2163] Bring Python into the next decade. (GH-17801) (cherry picked from commit 946b29ea0b3b386ed05e87e60b8617c9dc19cd53) Co-authored-by: Benjamin Peterson --- Doc/copyright.rst | 2 +- Doc/license.rst | 2 +- LICENSE | 2 +- Mac/IDLE/IDLE.app/Contents/Info.plist | 2 +- Mac/PythonLauncher/Info.plist.in | 2 +- Mac/Resources/app/Info.plist.in | 2 +- Python/getcopyright.c | 2 +- README.rst | 4 ++-- 8 files changed, 9 insertions(+), 9 deletions(-) diff --git a/Doc/copyright.rst b/Doc/copyright.rst index 393a1f03751f82..1b90d9f172c992 100644 --- a/Doc/copyright.rst +++ b/Doc/copyright.rst @@ -4,7 +4,7 @@ Copyright Python and this documentation is: -Copyright © 2001-2019 Python Software Foundation. All rights reserved. +Copyright © 2001-2020 Python Software Foundation. All rights reserved. Copyright © 2000 BeOpen.com. All rights reserved. diff --git a/Doc/license.rst b/Doc/license.rst index 810d2e63fd48c4..472a5cf3d88b34 100644 --- a/Doc/license.rst +++ b/Doc/license.rst @@ -87,7 +87,7 @@ PSF LICENSE AGREEMENT FOR PYTHON |release| analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python |release| alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of - copyright, i.e., "Copyright © 2001-2019 Python Software Foundation; All Rights + copyright, i.e., "Copyright © 2001-2020 Python Software Foundation; All Rights Reserved" are retained in Python |release| alone or in any derivative version prepared by Licensee. diff --git a/LICENSE b/LICENSE index 9dc010d80348fc..66a3ac80d729a3 100644 --- a/LICENSE +++ b/LICENSE @@ -73,7 +73,7 @@ analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, -2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Python Software Foundation; +2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee. diff --git a/Mac/IDLE/IDLE.app/Contents/Info.plist b/Mac/IDLE/IDLE.app/Contents/Info.plist index 04a0a08c836395..dcc48abdd2a395 100644 --- a/Mac/IDLE/IDLE.app/Contents/Info.plist +++ b/Mac/IDLE/IDLE.app/Contents/Info.plist @@ -36,7 +36,7 @@ CFBundleExecutable IDLE CFBundleGetInfoString - %version%, © 2001-2019 Python Software Foundation + %version%, © 2001-2020 Python Software Foundation CFBundleIconFile IDLE.icns CFBundleIdentifier diff --git a/Mac/PythonLauncher/Info.plist.in b/Mac/PythonLauncher/Info.plist.in index 9fb4e0affd9c43..21a051535fb925 100644 --- a/Mac/PythonLauncher/Info.plist.in +++ b/Mac/PythonLauncher/Info.plist.in @@ -40,7 +40,7 @@ CFBundleExecutable Python Launcher CFBundleGetInfoString - %VERSION%, © 2001-2019 Python Software Foundation + %VERSION%, © 2001-2020 Python Software Foundation CFBundleIconFile PythonLauncher.icns CFBundleIdentifier diff --git a/Mac/Resources/app/Info.plist.in b/Mac/Resources/app/Info.plist.in index b7581984dd6764..66b5e764c54b0b 100644 --- a/Mac/Resources/app/Info.plist.in +++ b/Mac/Resources/app/Info.plist.in @@ -37,7 +37,7 @@ CFBundleInfoDictionaryVersion 6.0 CFBundleLongVersionString - %version%, (c) 2001-2019 Python Software Foundation. + %version%, (c) 2001-2020 Python Software Foundation. CFBundleName Python CFBundlePackageType diff --git a/Python/getcopyright.c b/Python/getcopyright.c index 27a1731f46ded4..299ccc08c44f83 100644 --- a/Python/getcopyright.c +++ b/Python/getcopyright.c @@ -4,7 +4,7 @@ static const char cprt[] = "\ -Copyright (c) 2001-2019 Python Software Foundation.\n\ +Copyright (c) 2001-2020 Python Software Foundation.\n\ All Rights Reserved.\n\ \n\ Copyright (c) 2000 BeOpen.com.\n\ diff --git a/README.rst b/README.rst index d14f2b3ad99a68..be371c64980794 100644 --- a/README.rst +++ b/README.rst @@ -18,7 +18,7 @@ This is Python version 3.8.1 :target: https://python.zulipchat.com -Copyright (c) 2001-2019 Python Software Foundation. All rights reserved. +Copyright (c) 2001-2020 Python Software Foundation. All rights reserved. See the end of this file for further copyright and license information. @@ -246,7 +246,7 @@ See :pep:`569` for Python 3.8 release details. Copyright and License Information --------------------------------- -Copyright (c) 2001-2019 Python Software Foundation. All rights reserved. +Copyright (c) 2001-2020 Python Software Foundation. All rights reserved. Copyright (c) 2000 BeOpen.com. All rights reserved. From 526f0b3d93b50dda26cff03e92ed0d361dbea828 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 2 Jan 2020 19:49:59 -0800 Subject: [PATCH 1055/2163] Update copyright year in macOS installer license copy (GH-17806) (cherry picked from commit 32f1443aa98db769d87db497b45bd0dcb732445b) Co-authored-by: Ned Deily --- Mac/BuildScript/resources/License.rtf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Mac/BuildScript/resources/License.rtf b/Mac/BuildScript/resources/License.rtf index 7566cf3b44618a..25d53386da01a5 100644 --- a/Mac/BuildScript/resources/License.rtf +++ b/Mac/BuildScript/resources/License.rtf @@ -1,5 +1,5 @@ -{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600 -{\fonttbl\f0\fswiss\fcharset0 Helvetica-Bold;\f1\fswiss\fcharset0 Helvetica;\f2\fmodern\fcharset0 CourierNewPS-BoldMT; +{\rtf1\ansi\ansicpg1252\cocoartf2511 +\cocoatextscaling0\cocoaplatform0{\fonttbl\f0\fswiss\fcharset0 Helvetica-Bold;\f1\fswiss\fcharset0 Helvetica;\f2\fmodern\fcharset0 CourierNewPS-BoldMT; \f3\fmodern\fcharset0 CourierNewPSMT;} {\colortbl;\red255\green255\blue255;} {\*\expandedcolortbl;;} @@ -55,7 +55,7 @@ Thanks to the many outside volunteers who have worked under Guido's direction to \f1\b0 \ 1. This LICENSE AGREEMENT is between the Python Software Foundation ("PSF"), and the Individual or Organization ("Licensee") accessing and otherwise using this software ("Python") in source or binary form and its associated documentation.\ \ -2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\ +2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright \'a9 2001-2020 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\ \ 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make the derivative work available to others as provided herein, then Licensee hereby agrees to include in any such work a brief summary of the changes made to Python.\ \ From aa3efea9c5f4d25afc3fa4cfd5e6d789943893c9 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 3 Jan 2020 04:50:11 -0800 Subject: [PATCH 1056/2163] bpo-38532: Add missing decrefs in PyCFuncPtr_FromDll() (GH-17811) (cherry picked from commit e02ab59fdffa0bb841182c30ef1355c89578d945) Co-authored-by: Zackery Spytz --- Modules/_ctypes/_ctypes.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 347a3656b62578..c6da0d804453e7 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -3554,10 +3554,12 @@ PyCFuncPtr_FromDll(PyTypeObject *type, PyObject *args, PyObject *kwds) if (PySys_Audit("ctypes.dlsym", ((uintptr_t)name & ~0xFFFF) ? "Os" : "On", dll, name) < 0) { + Py_DECREF(ftuple); return NULL; } #else if (PySys_Audit("ctypes.dlsym", "Os", dll, name) < 0) { + Py_DECREF(ftuple); return NULL; } #endif From 867d8333ce6a7f74191ad464acc609d4a04e4acb Mon Sep 17 00:00:00 2001 From: Andrew Svetlov Date: Sat, 4 Jan 2020 11:49:11 +0200 Subject: [PATCH 1057/2163] [3.8] bpo-39191: Don't spawn a task before failing (GH-17796) (GH-17820) (cherry picked from commit 3a5de511596f17575de082dcb8d43d63b2bd2da9) Co-authored-by: Andrew Svetlov --- Lib/asyncio/base_events.py | 10 +++++++--- .../Library/2020-01-02-17-28-03.bpo-39191.ur_scy.rst | 3 +++ 2 files changed, 10 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-01-02-17-28-03.bpo-39191.ur_scy.rst diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py index bfd40115bed38a..aedf0c5e6d3d23 100644 --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -547,14 +547,17 @@ async def shutdown_asyncgens(self): 'asyncgen': agen }) - def run_forever(self): - """Run until stop() is called.""" - self._check_closed() + def _check_runnung(self): if self.is_running(): raise RuntimeError('This event loop is already running') if events._get_running_loop() is not None: raise RuntimeError( 'Cannot run the event loop while another loop is running') + + def run_forever(self): + """Run until stop() is called.""" + self._check_closed() + self._check_runnung() self._set_coroutine_origin_tracking(self._debug) self._thread_id = threading.get_ident() @@ -586,6 +589,7 @@ def run_until_complete(self, future): Return the Future's result, or raise its exception. """ self._check_closed() + self._check_runnung() new_task = not futures.isfuture(future) future = tasks.ensure_future(future, loop=self) diff --git a/Misc/NEWS.d/next/Library/2020-01-02-17-28-03.bpo-39191.ur_scy.rst b/Misc/NEWS.d/next/Library/2020-01-02-17-28-03.bpo-39191.ur_scy.rst new file mode 100644 index 00000000000000..138c93c2e4877a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-02-17-28-03.bpo-39191.ur_scy.rst @@ -0,0 +1,3 @@ +Perform a check for running loop before starting a new task in +``loop.run_until_complete()`` to fail fast; it prevents the side effect of +new task spawning before exception raising. From 859525590c7aad210ae5f2557140a52033c498cd Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 4 Jan 2020 18:14:58 -0800 Subject: [PATCH 1058/2163] Fix SystemError when nested function has annotation on positional-only argument (GH-17826) (cherry picked from commit ec007cb43faf5f33d06efbc28152c7fdcb2edb9c) Co-authored-by: Anthony Sottile --- Lib/test/test_positional_only_arg.py | 7 +++++++ .../2020-01-04-17-25-34.bpo-39215.xiqiIz.rst | 2 ++ Python/symtable.c | 2 ++ 3 files changed, 11 insertions(+) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-04-17-25-34.bpo-39215.xiqiIz.rst diff --git a/Lib/test/test_positional_only_arg.py b/Lib/test/test_positional_only_arg.py index 59b0b8fb55621c..63dee7ca434b21 100644 --- a/Lib/test/test_positional_only_arg.py +++ b/Lib/test/test_positional_only_arg.py @@ -15,6 +15,10 @@ def global_pos_only_and_normal(a, /, b): def global_pos_only_defaults(a=1, /, b=2): return a, b +def global_inner_has_pos_only(): + def f(x: int, /): ... + return f + class PositionalOnlyTestCase(unittest.TestCase): @@ -412,6 +416,9 @@ def method(self, /): self.assertEqual(C().method(), sentinel) + def test_annotations(self): + assert global_inner_has_pos_only().__annotations__ == {'x': int} + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-04-17-25-34.bpo-39215.xiqiIz.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-04-17-25-34.bpo-39215.xiqiIz.rst new file mode 100644 index 00000000000000..9a3178f9c62189 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-01-04-17-25-34.bpo-39215.xiqiIz.rst @@ -0,0 +1,2 @@ +Fix ``SystemError`` when nested function has annotation on positional-only +argument - by Anthony Sottile. diff --git a/Python/symtable.c b/Python/symtable.c index b8713588b9a914..30482d99b3ca92 100644 --- a/Python/symtable.c +++ b/Python/symtable.c @@ -1717,6 +1717,8 @@ static int symtable_visit_annotations(struct symtable *st, stmt_ty s, arguments_ty a, expr_ty returns) { + if (a->posonlyargs && !symtable_visit_argannotations(st, a->posonlyargs)) + return 0; if (a->args && !symtable_visit_argannotations(st, a->args)) return 0; if (a->vararg && a->vararg->annotation) From fc84d501b9d77701cbdd485de45e200bf027c0e9 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 5 Jan 2020 04:32:00 -0800 Subject: [PATCH 1059/2163] bpo-39057: Fix urllib.request.proxy_bypass_environment(). (GH-17619) Ignore leading dots and no longer ignore a trailing newline. (cherry picked from commit 6a265f0d0c0a4b3b8fecf4275d49187a384167f4) Co-authored-by: Serhiy Storchaka --- Lib/test/test_urllib.py | 22 +++++++++++++++++ Lib/urllib/parse.py | 4 ++-- Lib/urllib/request.py | 24 ++++++++++--------- .../2019-12-15-21-47-54.bpo-39057.FOxn-w.rst | 2 ++ 4 files changed, 39 insertions(+), 13 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-15-21-47-54.bpo-39057.FOxn-w.rst diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py index 801f0fd647f4ab..e9c656c583bbdb 100644 --- a/Lib/test/test_urllib.py +++ b/Lib/test/test_urllib.py @@ -261,14 +261,36 @@ def test_proxy_bypass_environment_host_match(self): self.assertTrue(bypass('localhost')) self.assertTrue(bypass('LocalHost')) # MixedCase self.assertTrue(bypass('LOCALHOST')) # UPPERCASE + self.assertTrue(bypass('.localhost')) self.assertTrue(bypass('newdomain.com:1234')) + self.assertTrue(bypass('.newdomain.com:1234')) self.assertTrue(bypass('foo.d.o.t')) # issue 29142 + self.assertTrue(bypass('d.o.t')) self.assertTrue(bypass('anotherdomain.com:8888')) + self.assertTrue(bypass('.anotherdomain.com:8888')) self.assertTrue(bypass('www.newdomain.com:1234')) self.assertFalse(bypass('prelocalhost')) self.assertFalse(bypass('newdomain.com')) # no port self.assertFalse(bypass('newdomain.com:1235')) # wrong port + def test_proxy_bypass_environment_always_match(self): + bypass = urllib.request.proxy_bypass_environment + self.env.set('NO_PROXY', '*') + self.assertTrue(bypass('newdomain.com')) + self.assertTrue(bypass('newdomain.com:1234')) + self.env.set('NO_PROXY', '*, anotherdomain.com') + self.assertTrue(bypass('anotherdomain.com')) + self.assertFalse(bypass('newdomain.com')) + self.assertFalse(bypass('newdomain.com:1234')) + + def test_proxy_bypass_environment_newline(self): + bypass = urllib.request.proxy_bypass_environment + self.env.set('NO_PROXY', + 'localhost, anotherdomain.com, newdomain.com:1234') + self.assertFalse(bypass('localhost\n')) + self.assertFalse(bypass('anotherdomain.com:8888\n')) + self.assertFalse(bypass('newdomain.com:1234\n')) + class ProxyTests_withOrderedEnv(unittest.TestCase): diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py index d497925b943f44..0b39b6eaf7ded1 100644 --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -1054,9 +1054,9 @@ def _splitport(host): """splitport('host:port') --> 'host', 'port'.""" global _portprog if _portprog is None: - _portprog = re.compile('(.*):([0-9]*)$', re.DOTALL) + _portprog = re.compile('(.*):([0-9]*)', re.DOTALL) - match = _portprog.match(host) + match = _portprog.fullmatch(host) if match: host, port = match.groups() if port: diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py index 51c4759629fd1f..6f6577bf1d9029 100644 --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -2500,24 +2500,26 @@ def proxy_bypass_environment(host, proxies=None): try: no_proxy = proxies['no'] except KeyError: - return 0 + return False # '*' is special case for always bypass if no_proxy == '*': - return 1 + return True + host = host.lower() # strip port off host hostonly, port = _splitport(host) # check if the host ends with any of the DNS suffixes - no_proxy_list = [proxy.strip() for proxy in no_proxy.split(',')] - for name in no_proxy_list: + for name in no_proxy.split(','): + name = name.strip() if name: name = name.lstrip('.') # ignore leading dots - name = re.escape(name) - pattern = r'(.+\.)?%s$' % name - if (re.match(pattern, hostonly, re.I) - or re.match(pattern, host, re.I)): - return 1 + name = name.lower() + if hostonly == name or host == name: + return True + name = '.' + name + if hostonly.endswith(name) or host.endswith(name): + return True # otherwise, don't bypass - return 0 + return False # This code tests an OSX specific data structure but is testable on all @@ -2643,7 +2645,7 @@ def getproxies_registry(): for p in proxyServer.split(';'): protocol, address = p.split('=', 1) # See if address has a type:// prefix - if not re.match('^([^/:]+)://', address): + if not re.match('(?:[^/:]+)://', address): address = '%s://%s' % (protocol, address) proxies[protocol] = address else: diff --git a/Misc/NEWS.d/next/Library/2019-12-15-21-47-54.bpo-39057.FOxn-w.rst b/Misc/NEWS.d/next/Library/2019-12-15-21-47-54.bpo-39057.FOxn-w.rst new file mode 100644 index 00000000000000..24a17444b97daa --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-15-21-47-54.bpo-39057.FOxn-w.rst @@ -0,0 +1,2 @@ +:func:`urllib.request.proxy_bypass_environment` now ignores leading dots and +no longer ignores a trailing newline. From e1caa49f68dd63b534774aebad0c240143e6fb5d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 5 Jan 2020 04:33:08 -0800 Subject: [PATCH 1060/2163] bpo-39056: Fix handling invalid warning category in the -W option. (GH-17618) No longer import the re module if it is not needed. (cherry picked from commit 41ec17e45d54473d32f543396293256f1581e44d) Co-authored-by: Serhiy Storchaka --- Lib/test/test_warnings/__init__.py | 23 ++++++++++++++ Lib/warnings.py | 30 ++++++++----------- .../2019-12-15-21-05-16.bpo-39056.nEfUM9.rst | 2 ++ 3 files changed, 38 insertions(+), 17 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-15-21-05-16.bpo-39056.nEfUM9.rst diff --git a/Lib/test/test_warnings/__init__.py b/Lib/test/test_warnings/__init__.py index fc3f8f6fe7f0c2..985adc1cda781c 100644 --- a/Lib/test/test_warnings/__init__.py +++ b/Lib/test/test_warnings/__init__.py @@ -43,6 +43,10 @@ def warnings_state(module): module.filters = original_filters +class TestWarning(Warning): + pass + + class BaseTest: """Basic bookkeeping required for testing.""" @@ -566,9 +570,28 @@ def test_improper_input(self): self.module._setoption, 'bogus::Warning') self.assertRaises(self.module._OptionError, self.module._setoption, 'ignore:2::4:-5') + with self.assertRaises(self.module._OptionError): + self.module._setoption('ignore::123') + with self.assertRaises(self.module._OptionError): + self.module._setoption('ignore::123abc') + with self.assertRaises(self.module._OptionError): + self.module._setoption('ignore::===') + with self.assertRaisesRegex(self.module._OptionError, 'Wärning'): + self.module._setoption('ignore::Wärning') self.module._setoption('error::Warning::0') self.assertRaises(UserWarning, self.module.warn, 'convert to error') + def test_import_from_module(self): + with original_warnings.catch_warnings(module=self.module): + self.module._setoption('ignore::Warning') + with self.assertRaises(self.module._OptionError): + self.module._setoption('ignore::TestWarning') + with self.assertRaises(self.module._OptionError): + self.module._setoption('ignore::test.test_warnings.bogus') + self.module._setoption('error::test.test_warnings.TestWarning') + with self.assertRaises(TestWarning): + self.module.warn('test warning', TestWarning) + class CWCmdLineTests(WCmdLineTests, unittest.TestCase): module = c_warnings diff --git a/Lib/warnings.py b/Lib/warnings.py index 00f740ca3a95ba..691ccddfa450ad 100644 --- a/Lib/warnings.py +++ b/Lib/warnings.py @@ -211,7 +211,6 @@ def _processoptions(args): # Helper for _processoptions() def _setoption(arg): - import re parts = arg.split(':') if len(parts) > 5: raise _OptionError("too many fields (max 5): %r" % (arg,)) @@ -220,11 +219,13 @@ def _setoption(arg): action, message, category, module, lineno = [s.strip() for s in parts] action = _getaction(action) - message = re.escape(message) category = _getcategory(category) - module = re.escape(module) + if message or module: + import re + if message: + message = re.escape(message) if module: - module = module + '$' + module = re.escape(module) + r'\Z' if lineno: try: lineno = int(lineno) @@ -248,26 +249,21 @@ def _getaction(action): # Helper for _setoption() def _getcategory(category): - import re if not category: return Warning - if re.match("^[a-zA-Z0-9_]+$", category): - try: - cat = eval(category) - except NameError: - raise _OptionError("unknown warning category: %r" % (category,)) from None + if '.' not in category: + import builtins as m + klass = category else: - i = category.rfind(".") - module = category[:i] - klass = category[i+1:] + module, _, klass = category.rpartition('.') try: m = __import__(module, None, None, [klass]) except ImportError: raise _OptionError("invalid module name: %r" % (module,)) from None - try: - cat = getattr(m, klass) - except AttributeError: - raise _OptionError("unknown warning category: %r" % (category,)) from None + try: + cat = getattr(m, klass) + except AttributeError: + raise _OptionError("unknown warning category: %r" % (category,)) from None if not issubclass(cat, Warning): raise _OptionError("invalid warning category: %r" % (category,)) return cat diff --git a/Misc/NEWS.d/next/Library/2019-12-15-21-05-16.bpo-39056.nEfUM9.rst b/Misc/NEWS.d/next/Library/2019-12-15-21-05-16.bpo-39056.nEfUM9.rst new file mode 100644 index 00000000000000..d5d2b98e9b0b38 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-15-21-05-16.bpo-39056.nEfUM9.rst @@ -0,0 +1,2 @@ +Fixed handling invalid warning category in the -W option. No longer import +the re module if it is not needed. From 34aa3e71dc52c1a31336302905b9ac011a310412 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 5 Jan 2020 04:36:48 -0800 Subject: [PATCH 1061/2163] bpo-39055: Reject a trailing \n in base64.b64decode() with validate=True. (GH-17616) (cherry picked from commit b19c0d77e6f25ea831ab608c71f15d0d9266c8c4) Co-authored-by: Serhiy Storchaka --- Lib/base64.py | 2 +- Lib/test/test_base64.py | 1 + .../next/Library/2019-12-15-19-23-23.bpo-39055.FmN3un.rst | 2 ++ 3 files changed, 4 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-15-19-23-23.bpo-39055.FmN3un.rst diff --git a/Lib/base64.py b/Lib/base64.py index 2be9c395a96674..2e70223dfe7824 100755 --- a/Lib/base64.py +++ b/Lib/base64.py @@ -82,7 +82,7 @@ def b64decode(s, altchars=None, validate=False): altchars = _bytes_from_decode_data(altchars) assert len(altchars) == 2, repr(altchars) s = s.translate(bytes.maketrans(altchars, b'+/')) - if validate and not re.match(b'^[A-Za-z0-9+/]*={0,2}$', s): + if validate and not re.fullmatch(b'[A-Za-z0-9+/]*={0,2}', s): raise binascii.Error('Non-base64 digit found') return binascii.a2b_base64(s) diff --git a/Lib/test/test_base64.py b/Lib/test/test_base64.py index 2a4cc2acad24b1..7dba6635d4eae7 100644 --- a/Lib/test/test_base64.py +++ b/Lib/test/test_base64.py @@ -250,6 +250,7 @@ def test_b64decode_invalid_chars(self): (b'3d}==', b'\xdd'), (b'@@', b''), (b'!', b''), + (b"YWJj\n", b"abc"), (b'YWJj\nYWI=', b'abcab')) funcs = ( base64.b64decode, diff --git a/Misc/NEWS.d/next/Library/2019-12-15-19-23-23.bpo-39055.FmN3un.rst b/Misc/NEWS.d/next/Library/2019-12-15-19-23-23.bpo-39055.FmN3un.rst new file mode 100644 index 00000000000000..83b1431e92fcb2 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-15-19-23-23.bpo-39055.FmN3un.rst @@ -0,0 +1,2 @@ +:func:`base64.b64decode` with ``validate=True`` raises now a binascii.Error +if the input ends with a single ``\n``. From 636a850ed81cf9b8feed523f277b1538bfc5230b Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 5 Jan 2020 09:07:30 -0800 Subject: [PATCH 1062/2163] bpo-39152: add missing ttk.Scale.configure return value (GH-17815) tkinter.ttk.Scale().configure([name]) now returns a configuration tuple for name or a list thereof for all options. Based on patch Giovanni Lombardo. (cherry picked from commit 5ea7bb25e3b192d6c49a49c9e3b316f8559602aa) Co-authored-by: Terry Jan Reedy --- Lib/tkinter/test/widget_tests.py | 13 ++++--------- Lib/tkinter/ttk.py | 5 +++-- .../2020-01-03-18-02-50.bpo-39152.JgPjCC.rst | 2 ++ 3 files changed, 9 insertions(+), 11 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-01-03-18-02-50.bpo-39152.JgPjCC.rst diff --git a/Lib/tkinter/test/widget_tests.py b/Lib/tkinter/test/widget_tests.py index 75a068fbbf26b6..b42ff52178f29e 100644 --- a/Lib/tkinter/test/widget_tests.py +++ b/Lib/tkinter/test/widget_tests.py @@ -3,7 +3,6 @@ import unittest import sys import tkinter -from tkinter.ttk import Scale from tkinter.test.support import (AbstractTkTest, tcl_version, requires_tcl, get_tk_patchlevel, pixels_conv, tcl_obj_eq) import test.support @@ -63,11 +62,9 @@ def checkParam(self, widget, name, value, *, expected=_sentinel, eq = tcl_obj_eq self.assertEqual2(widget[name], expected, eq=eq) self.assertEqual2(widget.cget(name), expected, eq=eq) - # XXX - if not isinstance(widget, Scale): - t = widget.configure(name) - self.assertEqual(len(t), 5) - self.assertEqual2(t[4], expected, eq=eq) + t = widget.configure(name) + self.assertEqual(len(t), 5) + self.assertEqual2(t[4], expected, eq=eq) def checkInvalidParam(self, widget, name, value, errmsg=None, *, keep_orig=True): @@ -209,9 +206,7 @@ def assertIsBoundingBox(self, bbox): def test_keys(self): widget = self.create() keys = widget.keys() - # XXX - if not isinstance(widget, Scale): - self.assertEqual(sorted(keys), sorted(widget.configure())) + self.assertEqual(sorted(keys), sorted(widget.configure())) for k in keys: widget[k] # Test if OPTIONS contains all keys diff --git a/Lib/tkinter/ttk.py b/Lib/tkinter/ttk.py index 573544dd84a390..c7c71cd5a559cf 100644 --- a/Lib/tkinter/ttk.py +++ b/Lib/tkinter/ttk.py @@ -1084,11 +1084,12 @@ def configure(self, cnf=None, **kw): Setting a value for any of the "from", "from_" or "to" options generates a <> event.""" - if cnf: + retval = Widget.configure(self, cnf, **kw) + if not isinstance(cnf, (type(None), str)): kw.update(cnf) - Widget.configure(self, **kw) if any(['from' in kw, 'from_' in kw, 'to' in kw]): self.event_generate('<>') + return retval def get(self, x=None, y=None): diff --git a/Misc/NEWS.d/next/Library/2020-01-03-18-02-50.bpo-39152.JgPjCC.rst b/Misc/NEWS.d/next/Library/2020-01-03-18-02-50.bpo-39152.JgPjCC.rst new file mode 100644 index 00000000000000..abb3df0da0fe46 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-03-18-02-50.bpo-39152.JgPjCC.rst @@ -0,0 +1,2 @@ +Fix ttk.Scale.configure([name]) to return configuration tuple for name +or all options. Giovanni Lombardo contributed part of the patch. From 49c108c8324c14e41ddf2a909dcab687f3d34b14 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 5 Jan 2020 09:21:47 -0800 Subject: [PATCH 1063/2163] Fix constant folding optimization for positional only arguments (GH-17837) (cherry picked from commit b121a4a45ff4bab8812a9b26ceffe5ad642f5d5a) Co-authored-by: Anthony Sottile --- Lib/test/test_positional_only_arg.py | 12 ++++++++++++ .../2020-01-05-06-55-52.bpo-39216.74jLh9.rst | 2 ++ Python/ast_opt.c | 1 + 3 files changed, 15 insertions(+) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-05-06-55-52.bpo-39216.74jLh9.rst diff --git a/Lib/test/test_positional_only_arg.py b/Lib/test/test_positional_only_arg.py index 63dee7ca434b21..2ef40e3a5a1138 100644 --- a/Lib/test/test_positional_only_arg.py +++ b/Lib/test/test_positional_only_arg.py @@ -1,5 +1,6 @@ """Unit tests for the positional only argument syntax specified in PEP 570.""" +import dis import pickle import unittest @@ -419,6 +420,17 @@ def method(self, /): def test_annotations(self): assert global_inner_has_pos_only().__annotations__ == {'x': int} + def test_annotations_constant_fold(self): + def g(): + def f(x: not (int is int), /): ... + + # without constant folding we end up with + # COMPARE_OP(is), UNARY_NOT + # with constant folding we should expect a COMPARE_OP(is not) + codes = [(i.opname, i.argval) for i in dis.get_instructions(g)] + self.assertNotIn(('UNARY_NOT', None), codes) + self.assertIn(('COMPARE_OP', 'is not'), codes) + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-05-06-55-52.bpo-39216.74jLh9.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-05-06-55-52.bpo-39216.74jLh9.rst new file mode 100644 index 00000000000000..971b06552973ea --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-01-05-06-55-52.bpo-39216.74jLh9.rst @@ -0,0 +1,2 @@ +Fix constant folding optimization for positional only arguments - by Anthony +Sottile. diff --git a/Python/ast_opt.c b/Python/ast_opt.c index 96c766fc0957d4..f2a2c259149932 100644 --- a/Python/ast_opt.c +++ b/Python/ast_opt.c @@ -617,6 +617,7 @@ astfold_comprehension(comprehension_ty node_, PyArena *ctx_, int optimize_) static int astfold_arguments(arguments_ty node_, PyArena *ctx_, int optimize_) { + CALL_SEQ(astfold_arg, arg_ty, node_->posonlyargs); CALL_SEQ(astfold_arg, arg_ty, node_->args); CALL_OPT(astfold_arg, arg_ty, node_->vararg); CALL_SEQ(astfold_arg, arg_ty, node_->kwonlyargs); From 183fbd1982a24465487b391debc0940b9bad20b3 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 5 Jan 2020 14:13:57 -0800 Subject: [PATCH 1064/2163] Replace links in howto/pyporting.rst with sphinx references (GH-17781) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Oleg Höfling (cherry picked from commit e6ae90dede07e8599cc6906417ca4aa99d8aa6e4) Co-authored-by: Oleg Höfling --- Doc/howto/pyporting.rst | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst index 3be6bb380d663b..8608162bace357 100644 --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -119,7 +119,7 @@ Once you have your code well-tested you are ready to begin porting your code to Python 3! But to fully understand how your code is going to change and what you want to look out for while you code, you will want to learn what changes Python 3 makes in terms of Python 2. Typically the two best ways of doing that -is reading the `"What's New"`_ doc for each release of Python 3 and the +is reading the :ref:`"What's New" ` doc for each release of Python 3 and the `Porting to Python 3`_ book (which is free online). There is also a handy `cheat sheet`_ from the Python-Future project. @@ -302,10 +302,10 @@ If for some reason that doesn't work then you should make the version check be against Python 2 and not Python 3. To help explain this, let's look at an example. -Let's pretend that you need access to a feature of importlib_ that +Let's pretend that you need access to a feature of :mod:`importlib` that is available in Python's standard library since Python 3.3 and available for Python 2 through importlib2_ on PyPI. You might be tempted to write code to -access e.g. the ``importlib.abc`` module by doing the following:: +access e.g. the :mod:`importlib.abc` module by doing the following:: import sys @@ -426,12 +426,10 @@ can also explicitly state whether your APIs use textual or binary data, helping to make sure everything functions as expected in both versions of Python. -.. _2to3: https://docs.python.org/3/library/2to3.html .. _caniusepython3: https://pypi.org/project/caniusepython3 .. _cheat sheet: http://python-future.org/compatible_idioms.html .. _coverage.py: https://pypi.org/project/coverage .. _Futurize: http://python-future.org/automatic_conversion.html -.. _importlib: https://docs.python.org/3/library/importlib.html#module-importlib .. _importlib2: https://pypi.org/project/importlib2 .. _Modernize: https://python-modernize.readthedocs.io/ .. _mypy: http://mypy-lang.org/ @@ -447,6 +445,4 @@ to make sure everything functions as expected in both versions of Python. .. _tox: https://pypi.org/project/tox .. _trove classifier: https://pypi.org/classifiers -.. _"What's New": https://docs.python.org/3/whatsnew/index.html - .. _Why Python 3 exists: https://snarky.ca/why-python-3-exists From cfeacf228924bed5e79d420eac682708b7fad727 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 5 Jan 2020 14:37:44 -0800 Subject: [PATCH 1065/2163] Fix the parameter list of object. _rpow_ (GH-GH-16477) (cherry picked from commit abc0c4fa9970931849b3da598c5980a5b170661e) Co-authored-by: HongWeipeng <961365124@qq.com> --- Doc/reference/datamodel.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index c530e4117af97c..e801ee392f195e 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -2331,7 +2331,7 @@ left undefined. object.__rfloordiv__(self, other) object.__rmod__(self, other) object.__rdivmod__(self, other) - object.__rpow__(self, other) + object.__rpow__(self, other[, modulo]) object.__rlshift__(self, other) object.__rrshift__(self, other) object.__rand__(self, other) From 89947881ac36324148fab75190af453dc09fd862 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 5 Jan 2020 14:45:35 -0800 Subject: [PATCH 1066/2163] bpo-39130: Dict reversed was added in v3.8 so should say in the doc as well (GH-17694) To be consistent with document layout, it should say when the feature was added. Although it's mentioned few other places in the doc but it's not explicitly say that at that place. https://bugs.python.org/issue39130 (cherry picked from commit 94d9cfc4ed9dd3c4a3a359bc194b4dc3f6ba63eb) Co-authored-by: Khalid Mammadov --- Doc/library/stdtypes.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index be72695462c7bc..d6db9b5411d8a6 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -4326,6 +4326,8 @@ pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098: Return a reverse iterator over the keys of the dictionary. This is a shortcut for ``reversed(d.keys())``. + .. versionadded:: 3.8 + .. method:: setdefault(key[, default]) If *key* is in the dictionary, return its value. If not, insert *key* From 5a065ac1811f07fcea955be402b1a264fbcc43d5 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 5 Jan 2020 16:09:56 -0800 Subject: [PATCH 1067/2163] Minor formatting improvements and fixes to idle.rst (GH-17165) (cherry picked from commit d6c08db8538d046d783db44fe4e70a60af0fb02e) Co-authored-by: Tal Einat --- Doc/library/idle.rst | 15 ++++++++------- Lib/idlelib/help.html | 15 ++++++++------- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst index 273b5830e4293e..f15f46b788b36a 100644 --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -370,7 +370,8 @@ Paste Editor windows also have breakpoint functions. Lines with a breakpoint set are specially marked. Breakpoints only have an effect when running under the -debugger. Breakpoints for a file are saved in the user's .idlerc directory. +debugger. Breakpoints for a file are saved in the user's ``.idlerc`` +directory. Set Breakpoint Set a breakpoint on the current line. @@ -685,14 +686,14 @@ crash or Keyboard Interrupt (control-C) may fail to connect. Dismissing the error box or Restart Shell on the Shell menu may fix a temporary problem. When IDLE first starts, it attempts to read user configuration files in -~/.idlerc/ (~ is one's home directory). If there is a problem, an error +``~/.idlerc/`` (~ is one's home directory). If there is a problem, an error message should be displayed. Leaving aside random disk glitches, this can be prevented by never editing the files by hand, using the configuration dialog, under Options, instead Options. Once it happens, the solution may be to delete one or more of the configuration files. If IDLE quits with no message, and it was not started from a console, try -starting from a console (``python -m idlelib)`` and see if a message appears. +starting from a console (``python -m idlelib``) and see if a message appears. Running user code ^^^^^^^^^^^^^^^^^ @@ -863,13 +864,13 @@ Or click the TOC (Table of Contents) button and select a section header in the opened box. Help menu entry "Python Docs" opens the extensive sources of help, -including tutorials, available at docs.python.org/x.y, where 'x.y' +including tutorials, available at ``docs.python.org/x.y``, where 'x.y' is the currently running Python version. If your system has an off-line copy of the docs (this may be an installation option), that will be opened instead. Selected URLs can be added or removed from the help menu at any time using the -General tab of the Configure IDLE dialog . +General tab of the Configure IDLE dialog. .. _preferences: @@ -878,9 +879,9 @@ Setting preferences The font preferences, highlighting, keys, and general preferences can be changed via Configure IDLE on the Option menu. -Non-default user settings are saved in a .idlerc directory in the user's +Non-default user settings are saved in a ``.idlerc`` directory in the user's home directory. Problems caused by bad user configuration files are solved -by editing or deleting one or more of the files in .idlerc. +by editing or deleting one or more of the files in ``.idlerc``. On the Font tab, see the text sample for the effect of font face and size on multiple characters in multiple languages. Edit the sample to add diff --git a/Lib/idlelib/help.html b/Lib/idlelib/help.html index 09dc4c57bcdc0f..0b2bdd2e174ccf 100644 --- a/Lib/idlelib/help.html +++ b/Lib/idlelib/help.html @@ -382,7 +382,8 @@

    Help menu (Shell and Editor).idlerc +directory.

    Set Breakpoint

    Set a breakpoint on the current line.

    @@ -638,13 +639,13 @@

    Startup failure~/.idlerc/ (~ is one’s home directory). If there is a problem, an error message should be displayed. Leaving aside random disk glitches, this can be prevented by never editing the files by hand, using the configuration dialog, under Options, instead Options. Once it happens, the solution may be to delete one or more of the configuration files.

    If IDLE quits with no message, and it was not started from a console, try -starting from a console (python -m idlelib) and see if a message appears.

    +starting from a console (python -m idlelib) and see if a message appears.

    Setting preferences¶

    The font preferences, highlighting, keys, and general preferences can be changed via Configure IDLE on the Option menu. -Non-default user settings are saved in a .idlerc directory in the user’s +Non-default user settings are saved in a .idlerc directory in the user’s home directory. Problems caused by bad user configuration files are solved -by editing or deleting one or more of the files in .idlerc.

    +by editing or deleting one or more of the files in .idlerc.

    On the Font tab, see the text sample for the effect of font face and size on multiple characters in multiple languages. Edit the sample to add other characters of personal interest. Use the sample to select From 5ed9d60bc53e2eb0a88f07d5afe5299acdc0b216 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 6 Jan 2020 04:34:10 -0800 Subject: [PATCH 1068/2163] bpo-38907: In http.server script, restore binding to IPv4 on Windows. (GH-17851) (#17854) (cherry picked from commit ee94bdb0598f9bc47d6a49e58fffc97aa617be96) Co-authored-by: Jason R. Coombs Co-authored-by: Jason R. Coombs --- Lib/http/server.py | 14 +++++++++++++- .../2020-01-06-02-14-38.bpo-38907.F1RkCR.rst | 1 + 2 files changed, 14 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2020-01-06-02-14-38.bpo-38907.F1RkCR.rst diff --git a/Lib/http/server.py b/Lib/http/server.py index b247675ec45e81..0fe44bdaa73feb 100644 --- a/Lib/http/server.py +++ b/Lib/http/server.py @@ -1280,4 +1280,16 @@ def test(HandlerClass=BaseHTTPRequestHandler, else: handler_class = partial(SimpleHTTPRequestHandler, directory=args.directory) - test(HandlerClass=handler_class, port=args.port, bind=args.bind) + + # ensure dual-stack is not disabled; ref #38907 + class DualStackServer(ThreadingHTTPServer): + def server_bind(self): + self.socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0) + return super().server_bind() + + test( + HandlerClass=handler_class, + ServerClass=DualStackServer, + port=args.port, + bind=args.bind, + ) diff --git a/Misc/NEWS.d/next/Library/2020-01-06-02-14-38.bpo-38907.F1RkCR.rst b/Misc/NEWS.d/next/Library/2020-01-06-02-14-38.bpo-38907.F1RkCR.rst new file mode 100644 index 00000000000000..a6e79f7809521d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-06-02-14-38.bpo-38907.F1RkCR.rst @@ -0,0 +1 @@ +In http.server script, restore binding to IPv4 on Windows. \ No newline at end of file From 33cb4a62bf6848093b7a05c9794582d204798b1b Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 6 Jan 2020 05:28:27 -0800 Subject: [PATCH 1069/2163] bpo-38907: Suppress any exception when attempting to set V6ONLY. (GH-17864) (GH-17865) Fixes error attempting to bind to IPv4 address. (cherry picked from commit 7cdc31a14c824000cbe8b487900c9826a33f6940) Co-authored-by: Jason R. Coombs Co-authored-by: Jason R. Coombs --- Lib/http/server.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/Lib/http/server.py b/Lib/http/server.py index 0fe44bdaa73feb..38f7accad7a346 100644 --- a/Lib/http/server.py +++ b/Lib/http/server.py @@ -103,6 +103,7 @@ import sys import time import urllib.parse +import contextlib from functools import partial from http import HTTPStatus @@ -1284,7 +1285,10 @@ def test(HandlerClass=BaseHTTPRequestHandler, # ensure dual-stack is not disabled; ref #38907 class DualStackServer(ThreadingHTTPServer): def server_bind(self): - self.socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0) + # suppress exception when protocol is IPv4 + with contextlib.suppress(Exception): + self.socket.setsockopt( + socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0) return super().server_bind() test( From b2e281aaa2e4a1f24671d293dfd06b23bb052e47 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 6 Jan 2020 08:26:13 -0800 Subject: [PATCH 1070/2163] bpo-39209: Manage correctly multi-line tokens in interactive mode (GH-17860) (cherry picked from commit 5ec91f78d59d9c39b984f284e00cd04b96ddb5db) Co-authored-by: Pablo Galindo --- Lib/test/test_repl.py | 36 +++++++++++++++++++ .../2020-01-06-10-29-16.bpo-39209.QHAONe.rst | 2 ++ Parser/tokenizer.c | 2 ++ 3 files changed, 40 insertions(+) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-06-10-29-16.bpo-39209.QHAONe.rst diff --git a/Lib/test/test_repl.py b/Lib/test/test_repl.py index 9efd459a6f0763..71f192f90d9a1d 100644 --- a/Lib/test/test_repl.py +++ b/Lib/test/test_repl.py @@ -58,5 +58,41 @@ def test_no_memory(self): # Exit code 120: Py_FinalizeEx() failed to flush stdout and stderr. self.assertIn(p.returncode, (1, 120)) + @cpython_only + def test_multiline_string_parsing(self): + # bpo-39209: Multiline string tokens need to be handled in the tokenizer + # in two places: the interactive path and the non-interactive path. + user_input = '''\ + x = """ + + + + + 0KiB + 0 + 1.3 + 0 + + + 16738211KiB + 237.15 + 1.3 + 0 + + never + none + + + """ + ''' + user_input = dedent(user_input) + user_input = user_input.encode() + p = spawn_repl() + with SuppressCrashReport(): + p.stdin.write(user_input) + output = kill_python(p) + self.assertEqual(p.returncode, 0) + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-06-10-29-16.bpo-39209.QHAONe.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-06-10-29-16.bpo-39209.QHAONe.rst new file mode 100644 index 00000000000000..c05b3f8dfa4d41 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-01-06-10-29-16.bpo-39209.QHAONe.rst @@ -0,0 +1,2 @@ +Correctly handle multi-line tokens in interactive mode. Patch by Pablo +Galindo. diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index f84093dae5b62e..f73c32684c7b73 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -886,6 +886,7 @@ tok_nextc(struct tok_state *tok) size_t start = tok->start - tok->buf; size_t oldlen = tok->cur - tok->buf; size_t newlen = oldlen + strlen(newtok); + Py_ssize_t cur_multi_line_start = tok->multi_line_start - tok->buf; char *buf = tok->buf; buf = (char *)PyMem_REALLOC(buf, newlen+1); tok->lineno++; @@ -898,6 +899,7 @@ tok_nextc(struct tok_state *tok) } tok->buf = buf; tok->cur = tok->buf + oldlen; + tok->multi_line_start = tok->buf + cur_multi_line_start; tok->line_start = tok->cur; strcpy(tok->buf + oldlen, newtok); PyMem_FREE(newtok); From 0048833e1308d39dc9c6489da7872ade0f14486f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 6 Jan 2020 08:46:55 -0800 Subject: [PATCH 1071/2163] bpo-39041: Add GitHub Actions support (GH-17594) (cherry picked from commit a76ba362c4d86adf5e7f8254398135d12d7afd25) Co-authored-by: Steve Dower --- .github/workflows/build.yml | 85 +++++++++++++++++++++++++++ .github/workflows/coverage.yml | 89 +++++++++++++++++++++++++++++ .github/workflows/doc.yml | 40 +++++++++++++ .github/workflows/posix-deps-apt.sh | 21 +++++++ Doc/make.bat | 6 +- 5 files changed, 238 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/build.yml create mode 100644 .github/workflows/coverage.yml create mode 100644 .github/workflows/doc.yml create mode 100755 .github/workflows/posix-deps-apt.sh diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml new file mode 100644 index 00000000000000..16d6f0db8c908f --- /dev/null +++ b/.github/workflows/build.yml @@ -0,0 +1,85 @@ +name: Tests + +on: + #push: + # branches: + # - master + # - 3.8 + # - 3.7 + # paths-ignore: + # - 'Doc/**' + # - 'Misc/**' + pull_request: + branches: + - master + - 3.8 + - 3.7 + paths-ignore: + - 'Doc/**' + - 'Misc/**' + +jobs: + build_win32: + name: 'Windows (x86)' + runs-on: windows-latest + steps: + - uses: actions/checkout@v1 + - name: Build CPython + run: .\PCbuild\build.bat -e -p Win32 + - name: Display build info + run: .\python.bat -m test.pythoninfo + - name: Tests + run: .\PCbuild\rt.bat -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0 + + build_win_amd64: + name: 'Windows (x64)' + runs-on: windows-latest + steps: + - uses: actions/checkout@v1 + - name: Build CPython + run: .\PCbuild\build.bat -e -p x64 + - name: Display build info + run: .\python.bat -m test.pythoninfo + - name: Tests + run: .\PCbuild\rt.bat -x64 -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0 + + build_macos: + name: 'macOS' + runs-on: macos-latest + steps: + - uses: actions/checkout@v1 + - name: Configure CPython + run: ./configure --with-pydebug --with-openssl=/usr/local/opt/openssl --prefix=/opt/python-dev + - name: Build CPython + run: make -s -j4 + - name: Display build info + run: make pythoninfo + - name: Tests + run: make buildbottest TESTOPTS="-j4 -uall,-cpu" + + build_ubuntu: + name: 'Ubuntu' + runs-on: ubuntu-latest + env: + OPENSSL_VER: 1.1.1d + steps: + - uses: actions/checkout@v1 + - name: Install Dependencies + run: sudo ./.github/workflows/posix-deps-apt.sh + - name: 'Restore OpenSSL build' + id: cache-openssl + uses: actions/cache@v1 + with: + path: ./multissl/openssl/${{ env.OPENSSL_VER }} + key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }} + - name: Install OpenSSL + if: steps.cache-openssl.outputs.cache-hit != 'true' + run: python3 Tools/ssl/multissltests.py --steps=library --base-directory $PWD/multissl --openssl $OPENSSL_VER --system Linux + - name: Configure CPython + run: ./configure --with-pydebug --with-openssl=$PWD/multissl/openssl/$OPENSSL_VER + - name: Build CPython + run: make -s -j4 + - name: Display build info + run: make pythoninfo + - name: Tests + run: xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu" diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml new file mode 100644 index 00000000000000..cb05e8e2f71f05 --- /dev/null +++ b/.github/workflows/coverage.yml @@ -0,0 +1,89 @@ +name: Coverage + +on: + push: + branches: + - master + - 3.8 + - 3.7 + paths-ignore: + - 'Doc/**' + - 'Misc/**' + #pull_request: + # branches: + # - master + # - 3.8 + # - 3.7 + # paths-ignore: + # - 'Doc/**' + # - 'Misc/**' + +jobs: + coverage_ubuntu: + name: 'Ubuntu (Coverage)' + runs-on: ubuntu-latest + env: + OPENSSL_VER: 1.1.1d + steps: + - uses: actions/checkout@v1 + - name: Install Dependencies + run: sudo ./.github/workflows/posix-deps-apt.sh + - name: 'Restore OpenSSL build' + id: cache-openssl + uses: actions/cache@v1 + with: + path: ./multissl/openssl/${{ env.OPENSSL_VER }} + key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }} + - name: Install OpenSSL + if: steps.cache-openssl.outputs.cache-hit != 'true' + run: python3 Tools/ssl/multissltests.py --steps=library --base-directory $PWD/multissl --openssl $OPENSSL_VER --system Linux + - name: Configure CPython + run: ./configure --with-openssl=$PWD/multissl/openssl/$OPENSSL_VER + - name: Build CPython + run: make -s -j4 + - name: Display build info + run: make pythoninfo + - name: 'Coverage Preparation' + run: | + ./python -m venv .venv + source ./.venv/bin/activate + python -m pip install -U coverage + python -m test.pythoninfo + - name: 'Tests with coverage' + run: > + source ./.venv/bin/activate && + xvfb-run python -m coverage + run --branch --pylib + -m test + --fail-env-changed + -uall,-cpu + -x test_multiprocessing_fork + -x test_multiprocessing_forkserver + -x test_multiprocessing_spawn + -x test_concurrent_futures + || true + - name: 'Publish code coverage results' + run: | + ./.venv/bin/python -m coverage xml + bash <(curl -s https://codecov.io/bash) + env: + CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} + + c_coverage_ubuntu: + name: 'Ubuntu (C Coverage)' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v1 + - name: Install Dependencies + run: sudo ./.github/workflows/posix-deps-apt.sh + - name: Configure CPython + run: ./configure + - name: 'Build CPython and measure coverage' + run: xvfb-run make -j4 coverage-report + - name: 'Publish code coverage results' + if: always() + run: | + make pythoninfo + bash <(curl -s https://codecov.io/bash) + env: + CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml new file mode 100644 index 00000000000000..405b12e3d29c95 --- /dev/null +++ b/.github/workflows/doc.yml @@ -0,0 +1,40 @@ +name: Docs + +on: + #push: + # branches: + # - master + # - 3.8 + # - 3.7 + # paths: + # - 'Doc/**' + pull_request: + branches: + - master + - 3.8 + - 3.7 + paths: + - 'Doc/**' + - 'Misc/**' + +jobs: + build_doc: + name: 'Docs' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v1 + - uses: actions/setup-python@v1 + with: + python-version: '3.7' + architecture: 'x64' + - name: 'Install build dependencies' + run: python -m pip install sphinx==2.2.0 blurb python-docs-theme + - name: 'Build documentation' + run: | + cd Doc + make check suspicious html PYTHON=python + - name: Upload + uses: actions/upload-artifact@v1 + with: + name: doc-html + path: Doc/build/html diff --git a/.github/workflows/posix-deps-apt.sh b/.github/workflows/posix-deps-apt.sh new file mode 100755 index 00000000000000..2b879d32f8150d --- /dev/null +++ b/.github/workflows/posix-deps-apt.sh @@ -0,0 +1,21 @@ +#!/bin/sh +apt-get update + +apt-get -yq install \ + build-essential \ + zlib1g-dev \ + libbz2-dev \ + liblzma-dev \ + libncurses5-dev \ + libreadline6-dev \ + libsqlite3-dev \ + libssl-dev \ + libgdbm-dev \ + tk-dev \ + lzma \ + lzma-dev \ + liblzma-dev \ + libffi-dev \ + uuid-dev \ + xvfb \ + lcov diff --git a/Doc/make.bat b/Doc/make.bat index dfc622f66615df..6f8f172e95eb8e 100644 --- a/Doc/make.bat +++ b/Doc/make.bat @@ -54,9 +54,9 @@ if not exist "%HTMLHELP%" ( ) :skiphhcsearch -if "%DISTVERSION%" EQU "" for /f "usebackq" %%v in (`%PYTHON% tools/extensions/patchlevel.py`) do set DISTVERSION=%%v +if not defined DISTVERSION for /f "usebackq" %%v in (`%PYTHON% tools/extensions/patchlevel.py`) do set DISTVERSION=%%v -if "%BUILDDIR%" EQU "" set BUILDDIR=build +if not defined BUILDDIR set BUILDDIR=build rem Targets that don't require sphinx-build if "%1" EQU "" goto help @@ -131,7 +131,7 @@ if exist ..\Misc\NEWS ( ) ) -if NOT "%PAPER%" == "" ( +if defined PAPER ( set SPHINXOPTS=-D latex_elements.papersize=%PAPER% %SPHINXOPTS% ) if "%1" EQU "htmlhelp" ( From a9a43c221bf3896ed1d1c2eee2531b7121cf78e4 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 6 Jan 2020 09:17:36 -0800 Subject: [PATCH 1072/2163] bpo-29778: Fix incorrect NULL check in _PyPathConfig_InitDLLPath() (GH-17818) (cherry picked from commit 7b79dc9200a19ecbac667111dffd58e314be02a8) Co-authored-by: Anthony Wee --- Python/pathconfig.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Python/pathconfig.c b/Python/pathconfig.c index 7f3fdcc103f7b0..258ff613a066c1 100644 --- a/Python/pathconfig.c +++ b/Python/pathconfig.c @@ -150,7 +150,7 @@ _PyWideStringList_Join(const PyWideStringList *list, wchar_t sep) static PyStatus _PyPathConfig_InitDLLPath(void) { - if (_Py_dll_path == NULL) { + if (_Py_dll_path != NULL) { /* Already set: nothing to do */ return _PyStatus_OK(); } From fb59f5ffe80a1f2dcf7c6cbd2406e15bea49da21 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 6 Jan 2020 13:41:33 -0800 Subject: [PATCH 1073/2163] bpo-39041: Fix coverage upload command for GitHub Actions (GH-17873) https://bugs.python.org/issue39041 Automerge-Triggered-By: @zooba (cherry picked from commit b1ce22d086660d2505010694c8813cc67adf8f9e) Co-authored-by: Steve Dower --- .github/workflows/coverage.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index cb05e8e2f71f05..e8b47b390e5a79 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -64,7 +64,7 @@ jobs: || true - name: 'Publish code coverage results' run: | - ./.venv/bin/python -m coverage xml + source ./.venv/bin/activate bash <(curl -s https://codecov.io/bash) env: CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} From 6a8284d8f1583c952e3219c7e3671ba4979e340f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 6 Jan 2020 22:59:28 -0800 Subject: [PATCH 1074/2163] Doc: Change Python 2 status to EOL. (GH-17885) (cherry picked from commit f4800b8ed3dbe15a0078869a836d968ab3362b8c) Co-authored-by: Inada Naoki --- Doc/tools/templates/indexsidebar.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html index c51dcc7582e54f..4730a5fe5db7bc 100644 --- a/Doc/tools/templates/indexsidebar.html +++ b/Doc/tools/templates/indexsidebar.html @@ -7,7 +7,7 @@

    {% trans %}Docs by version{% endtrans %}

  • {% trans %}Python 3.7 (stable){% endtrans %}
  • {% trans %}Python 3.6 (security-fixes){% endtrans %}
  • {% trans %}Python 3.5 (security-fixes){% endtrans %}
  • -
  • {% trans %}Python 2.7 (stable){% endtrans %}
  • +
  • {% trans %}Python 2.7 (EOL){% endtrans %}
  • {% trans %}All versions{% endtrans %}
  • From a6b37589a05c63abec122d3a00785641a3bcd85a Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 7 Jan 2020 00:04:43 -0800 Subject: [PATCH 1075/2163] bpo-38623: Doc: Add section for site module CLI. (GH-17858) (cherry picked from commit ca94677a6216e2d41b04574986ce49d31a0b329c) Co-authored-by: Inada Naoki --- Doc/library/site.rst | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/Doc/library/site.rst b/Doc/library/site.rst index e1ca160c107b8b..b424e1ba348d87 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -236,6 +236,13 @@ Module contents .. versionadded:: 3.2 +.. _site-commandline: + +Command Line Interface +---------------------- + +.. program:: site + The :mod:`site` module also provides a way to get the user directories from the command line: @@ -244,8 +251,6 @@ command line: $ python3 -m site --user-site /home/user/.local/lib/python3.3/site-packages -.. program:: site - If it is called without arguments, it will print the contents of :data:`sys.path` on the standard output, followed by the value of :data:`USER_BASE` and whether the directory exists, then the same thing for From 4112a3da2e01ecc19e9f54b8ac7b383b6f5e85c8 Mon Sep 17 00:00:00 2001 From: Andrew Svetlov Date: Tue, 7 Jan 2020 16:55:19 +0200 Subject: [PATCH 1076/2163] [3.8] bpo-39191: Fix RuntimeWarning in asyncio test (GH-17863) (#17894) https://bugs.python.org/issue39191. (cherry picked from commit 10ac0cded26d91c3468e5e5a87cecad7fc0bcebd) Co-authored-by: Andrew Svetlov --- Lib/asyncio/base_events.py | 6 +++--- Lib/test/test_asyncio/test_events.py | 8 ++++++-- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py index aedf0c5e6d3d23..799013d5ccccb5 100644 --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -547,7 +547,7 @@ async def shutdown_asyncgens(self): 'asyncgen': agen }) - def _check_runnung(self): + def _check_running(self): if self.is_running(): raise RuntimeError('This event loop is already running') if events._get_running_loop() is not None: @@ -557,7 +557,7 @@ def _check_runnung(self): def run_forever(self): """Run until stop() is called.""" self._check_closed() - self._check_runnung() + self._check_running() self._set_coroutine_origin_tracking(self._debug) self._thread_id = threading.get_ident() @@ -589,7 +589,7 @@ def run_until_complete(self, future): Return the Future's result, or raise its exception. """ self._check_closed() - self._check_runnung() + self._check_running() new_task = not futures.isfuture(future) future = tasks.ensure_future(future, loop=self) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py index b0ade1ed3ba1bb..aec56da8e22504 100644 --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -258,8 +258,12 @@ async def coro2(): self.assertTrue(self.loop.is_running()) self.loop.run_until_complete(coro1()) - self.assertRaises( - RuntimeError, self.loop.run_until_complete, coro2()) + with self.assertWarnsRegex( + RuntimeWarning, + r"coroutine \S+ was never awaited" + ): + self.assertRaises( + RuntimeError, self.loop.run_until_complete, coro2()) # Note: because of the default Windows timing granularity of # 15.6 msec, we use fairly long sleep times here (~100 msec). From bff48c6734f936257b0cfae58dbea67d43e3b245 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 7 Jan 2020 09:03:23 -0800 Subject: [PATCH 1077/2163] bpo-39198: Ensure logging global lock is released on exception in isEnabledFor (GH-17689) (GH-17897) (cherry picked from commit 950c6795aa0ffa85e103a13e7a04e08cb34c66ad) --- Lib/logging/__init__.py | 15 +++++++++------ .../2020-01-02-20-21-03.bpo-39198.nzwGyG.rst | 1 + 2 files changed, 10 insertions(+), 6 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-01-02-20-21-03.bpo-39198.nzwGyG.rst diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py index 16812ec8d55678..0cfaec84bac139 100644 --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -1685,12 +1685,15 @@ def isEnabledFor(self, level): return self._cache[level] except KeyError: _acquireLock() - if self.manager.disable >= level: - is_enabled = self._cache[level] = False - else: - is_enabled = self._cache[level] = level >= self.getEffectiveLevel() - _releaseLock() - + try: + if self.manager.disable >= level: + is_enabled = self._cache[level] = False + else: + is_enabled = self._cache[level] = ( + level >= self.getEffectiveLevel() + ) + finally: + _releaseLock() return is_enabled def getChild(self, suffix): diff --git a/Misc/NEWS.d/next/Library/2020-01-02-20-21-03.bpo-39198.nzwGyG.rst b/Misc/NEWS.d/next/Library/2020-01-02-20-21-03.bpo-39198.nzwGyG.rst new file mode 100644 index 00000000000000..ec4e81e2bbe4a0 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-02-20-21-03.bpo-39198.nzwGyG.rst @@ -0,0 +1 @@ +If an exception were to be thrown in `Logger.isEnabledFor` (say, by asyncio timeouts or stopit) , the `logging` global lock may not be released appropriately, resulting in deadlock. This change wraps that block of code with `try...finally` to ensure the lock is released. \ No newline at end of file From 39a5c889d30d03a88102e56f03ee0c95db198fb3 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 7 Jan 2020 09:52:06 -0800 Subject: [PATCH 1078/2163] bpo-38871: Fix lib2to3 for filter-based statements that contain lambda (GH-17780) Correctly parenthesize filter-based statements that contain lambda expressions in lib2to3. (cherry picked from commit b821173b5458d137c8d5edb6e9b4997aac800a38) Co-authored-by: Dong-hee Na --- Lib/lib2to3/fixes/fix_filter.py | 10 +++++++--- Lib/lib2to3/tests/test_fixers.py | 5 +++++ .../Library/2020-01-01-18-44-52.bpo-38871.3EEOLg.rst | 2 ++ 3 files changed, 14 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-01-01-18-44-52.bpo-38871.3EEOLg.rst diff --git a/Lib/lib2to3/fixes/fix_filter.py b/Lib/lib2to3/fixes/fix_filter.py index a7a5a154f6fb11..38e9078f11ac88 100644 --- a/Lib/lib2to3/fixes/fix_filter.py +++ b/Lib/lib2to3/fixes/fix_filter.py @@ -17,7 +17,7 @@ from .. import fixer_base from ..pytree import Node from ..pygram import python_symbols as syms -from ..fixer_util import Name, ArgList, ListComp, in_special_context +from ..fixer_util import Name, ArgList, ListComp, in_special_context, parenthesize class FixFilter(fixer_base.ConditionalFix): @@ -65,10 +65,14 @@ def transform(self, node, results): trailers.append(t.clone()) if "filter_lambda" in results: + xp = results.get("xp").clone() + if xp.type == syms.test: + xp.prefix = "" + xp = parenthesize(xp) + new = ListComp(results.get("fp").clone(), results.get("fp").clone(), - results.get("it").clone(), - results.get("xp").clone()) + results.get("it").clone(), xp) new = Node(syms.power, [new] + trailers, prefix="") elif "none" in results: diff --git a/Lib/lib2to3/tests/test_fixers.py b/Lib/lib2to3/tests/test_fixers.py index 3da5dd845c93c6..a2852419818133 100644 --- a/Lib/lib2to3/tests/test_fixers.py +++ b/Lib/lib2to3/tests/test_fixers.py @@ -2954,6 +2954,11 @@ def test_filter_basic(self): a = """x = [x for x in range(10) if x%2 == 0]""" self.check(b, a) + # bpo-38871 + b = """filter(lambda x: True if x > 2 else False, [1, 2, 3])""" + a = """[x for x in [1, 2, 3] if (True if x > 2 else False)]""" + self.check(b, a) + def test_filter_trailers(self): b = """x = filter(None, 'abc')[0]""" a = """x = [_f for _f in 'abc' if _f][0]""" diff --git a/Misc/NEWS.d/next/Library/2020-01-01-18-44-52.bpo-38871.3EEOLg.rst b/Misc/NEWS.d/next/Library/2020-01-01-18-44-52.bpo-38871.3EEOLg.rst new file mode 100644 index 00000000000000..fe970fd9e3fa1a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-01-18-44-52.bpo-38871.3EEOLg.rst @@ -0,0 +1,2 @@ +Correctly parenthesize filter-based statements that contain lambda +expressions in mod:`lib2to3`. Patch by Dong-hee Na. From b24e4fac03409c4f845758d7ed884c5a99368493 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 8 Jan 2020 07:48:33 -0800 Subject: [PATCH 1079/2163] bpo-39242: Updated the Gmane domain into news.gmane.io (GH-17903) (cherry picked from commit 2e6a8efa837410327b593dc83c57492253b1201e) Co-authored-by: Dong-hee Na --- Doc/library/nntplib.rst | 8 ++++---- Lib/nntplib.py | 2 +- Lib/test/test_nntplib.py | 4 ++-- .../next/Library/2020-01-08-23-25-27.bpo-39242.bnL65N.rst | 3 +++ 4 files changed, 10 insertions(+), 7 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-01-08-23-25-27.bpo-39242.bnL65N.rst diff --git a/Doc/library/nntplib.rst b/Doc/library/nntplib.rst index 46f1c0783551c2..e8480b548073ab 100644 --- a/Doc/library/nntplib.rst +++ b/Doc/library/nntplib.rst @@ -20,7 +20,7 @@ as well as the older :rfc:`977` and :rfc:`2980`. Here are two small examples of how it can be used. To list some statistics about a newsgroup and print the subjects of the last 10 articles:: - >>> s = nntplib.NNTP('news.gmane.org') + >>> s = nntplib.NNTP('news.gmane.io') >>> resp, count, first, last, name = s.group('gmane.comp.python.committers') >>> print('Group', name, 'has', count, 'articles, range', first, 'to', last) Group gmane.comp.python.committers has 1096 articles, range 1 to 1096 @@ -44,7 +44,7 @@ about a newsgroup and print the subjects of the last 10 articles:: To post an article from a binary file (this assumes that the article has valid headers, and that you have right to post on the particular newsgroup):: - >>> s = nntplib.NNTP('news.gmane.org') + >>> s = nntplib.NNTP('news.gmane.io') >>> f = open('article.txt', 'rb') >>> s.post(f) '240 Article posted successfully.' @@ -73,7 +73,7 @@ The module itself defines the following classes: connection when done, e.g.: >>> from nntplib import NNTP - >>> with NNTP('news.gmane.org') as n: + >>> with NNTP('news.gmane.io') as n: ... n.group('gmane.comp.python.committers') ... # doctest: +SKIP ('211 1755 1 1755 gmane.comp.python.committers', 1755, 1, 1755, 'gmane.comp.python.committers') @@ -225,7 +225,7 @@ tuples or objects that the method normally returns will be empty. of values. On legacy servers which don't understand the ``CAPABILITIES`` command, an empty dictionary is returned instead. - >>> s = NNTP('news.gmane.org') + >>> s = NNTP('news.gmane.io') >>> 'POST' in s.getcapabilities() True diff --git a/Lib/nntplib.py b/Lib/nntplib.py index 1b7e83af01ad00..9036f361b5fb6b 100644 --- a/Lib/nntplib.py +++ b/Lib/nntplib.py @@ -1107,7 +1107,7 @@ def _close(self): nntplib built-in demo - display the latest articles in a newsgroup""") parser.add_argument('-g', '--group', default='gmane.comp.python.general', help='group to fetch messages from (default: %(default)s)') - parser.add_argument('-s', '--server', default='news.gmane.org', + parser.add_argument('-s', '--server', default='news.gmane.io', help='NNTP server hostname (default: %(default)s)') parser.add_argument('-p', '--port', default=-1, type=int, help='NNTP port number (default: %s / %s)' % (NNTP_PORT, NNTP_SSL_PORT)) diff --git a/Lib/test/test_nntplib.py b/Lib/test/test_nntplib.py index 618b403bfb5bd3..fbd7db03defb1d 100644 --- a/Lib/test/test_nntplib.py +++ b/Lib/test/test_nntplib.py @@ -633,7 +633,7 @@ def handle_XOVER(self, message_spec): "\tSat, 19 Jun 2010 18:04:08 -0400" "\t<4FD05F05-F98B-44DC-8111-C6009C925F0C@gmail.com>" "\t\t7103\t16" - "\tXref: news.gmane.org gmane.comp.python.authors:57" + "\tXref: news.gmane.io gmane.comp.python.authors:57" "\n" "58\tLooking for a few good bloggers" "\tDoug Hellmann " @@ -1119,7 +1119,7 @@ def check_over_xover_resp(self, resp, overviews): "references": "", ":bytes": "7103", ":lines": "16", - "xref": "news.gmane.org gmane.comp.python.authors:57" + "xref": "news.gmane.io gmane.comp.python.authors:57" }) art_num, over = overviews[1] self.assertEqual(over["xref"], None) diff --git a/Misc/NEWS.d/next/Library/2020-01-08-23-25-27.bpo-39242.bnL65N.rst b/Misc/NEWS.d/next/Library/2020-01-08-23-25-27.bpo-39242.bnL65N.rst new file mode 100644 index 00000000000000..a87dddf81dcd5c --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-08-23-25-27.bpo-39242.bnL65N.rst @@ -0,0 +1,3 @@ +Updated the Gmane domain from news.gmane.org to news.gmane.io +which is used for examples of :class:`~nntplib.NNTP` news reader server and +nntplib tests. From 45e5750a013291c5729e0ebad2b9e340fdffbd36 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 8 Jan 2020 19:07:38 -0800 Subject: [PATCH 1080/2163] closes bpo-39262: Use specific out-of-memory message in _sharedexception_bind. (GH-17908) (cherry picked from commit 5cae042f686cc174e00093944dc118914c874b7c) Co-authored-by: Alex Henrie --- Modules/_xxsubinterpretersmodule.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/Modules/_xxsubinterpretersmodule.c b/Modules/_xxsubinterpretersmodule.c index 7842947e54ac56..db1116ac6e352d 100644 --- a/Modules/_xxsubinterpretersmodule.c +++ b/Modules/_xxsubinterpretersmodule.c @@ -221,8 +221,9 @@ _sharedexception_bind(PyObject *exctype, PyObject *exc, PyObject *tb) if (err->name == NULL) { if (PyErr_ExceptionMatches(PyExc_MemoryError)) { failure = "out of memory copying exception type name"; + } else { + failure = "unable to encode and copy exception type name"; } - failure = "unable to encode and copy exception type name"; goto finally; } @@ -237,8 +238,9 @@ _sharedexception_bind(PyObject *exctype, PyObject *exc, PyObject *tb) if (err->msg == NULL) { if (PyErr_ExceptionMatches(PyExc_MemoryError)) { failure = "out of memory copying exception message"; + } else { + failure = "unable to encode and copy exception message"; } - failure = "unable to encode and copy exception message"; goto finally; } } From 20c990229e98ad69c03e44fe61f8dce99b96cf9d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 9 Jan 2020 04:27:35 -0800 Subject: [PATCH 1081/2163] bpo-39161: Document multi-phase init modules under Py_NewInterpreter() (GH-17896) \+ this also adds a stronger warning against sharing objects between (sub-)interpreters. https://bugs.python.org/issue39161 (cherry picked from commit 6c5d661342d12f6836580b0e75e3569c764527ae) Co-authored-by: Petr Viktorin --- Doc/c-api/init.rst | 52 +++++++++++++++++++++++++++++++--------------- 1 file changed, 35 insertions(+), 17 deletions(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 81cb4f825fee7a..68d892dcae4046 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1222,15 +1222,31 @@ function. You can create and destroy them using the following functions: single: Py_FinalizeEx() single: Py_Initialize() - Extension modules are shared between (sub-)interpreters as follows: the first - time a particular extension is imported, it is initialized normally, and a - (shallow) copy of its module's dictionary is squirreled away. When the same - extension is imported by another (sub-)interpreter, a new module is initialized - and filled with the contents of this copy; the extension's ``init`` function is - not called. Note that this is different from what happens when an extension is - imported after the interpreter has been completely re-initialized by calling - :c:func:`Py_FinalizeEx` and :c:func:`Py_Initialize`; in that case, the extension's - ``initmodule`` function *is* called again. + Extension modules are shared between (sub-)interpreters as follows: + + * For modules using multi-phase initialization, + e.g. :c:func:`PyModule_FromDefAndSpec`, a separate module object is + created and initialized for each interpreter. + Only C-level static and global variables are shared between these + module objects. + + * For modules using single-phase initialization, + e.g. :c:func:`PyModule_Create`, the first time a particular extension + is imported, it is initialized normally, and a (shallow) copy of its + module's dictionary is squirreled away. + When the same extension is imported by another (sub-)interpreter, a new + module is initialized and filled with the contents of this copy; the + extension's ``init`` function is not called. + Objects in the module's dictionary thus end up shared across + (sub-)interpreters, which might cause unwanted behavior (see + `Bugs and caveats`_ below). + + Note that this is different from what happens when an extension is + imported after the interpreter has been completely re-initialized by + calling :c:func:`Py_FinalizeEx` and :c:func:`Py_Initialize`; in that + case, the extension's ``initmodule`` function *is* called again. + As with multi-phase initialization, this means that only C-level static + and global variables are shared between these modules. .. index:: single: close() (in module os) @@ -1256,14 +1272,16 @@ process, the insulation between them isn't perfect --- for example, using low-level file operations like :func:`os.close` they can (accidentally or maliciously) affect each other's open files. Because of the way extensions are shared between (sub-)interpreters, some extensions may not -work properly; this is especially likely when the extension makes use of -(static) global variables, or when the extension manipulates its module's -dictionary after its initialization. It is possible to insert objects created -in one sub-interpreter into a namespace of another sub-interpreter; this should -be done with great care to avoid sharing user-defined functions, methods, -instances or classes between sub-interpreters, since import operations executed -by such objects may affect the wrong (sub-)interpreter's dictionary of loaded -modules. +work properly; this is especially likely when using single-phase initialization +or (static) global variables. +It is possible to insert objects created in one sub-interpreter into +a namespace of another (sub-)interpreter; this should be avoided if possible. + +Special care should be taken to avoid sharing user-defined functions, +methods, instances or classes between sub-interpreters, since import +operations executed by such objects may affect the wrong (sub-)interpreter's +dictionary of loaded modules. It is equally important to avoid sharing +objects from which the above are reachable. Also note that combining this functionality with :c:func:`PyGILState_\*` APIs is delicate, because these APIs assume a bijection between Python thread states From 8c08518c255747a06d00479f21087f0c934d0ad6 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 9 Jan 2020 09:19:59 -0800 Subject: [PATCH 1082/2163] bpo-25172: Reduce scope of crypt import tests (GH-17881) (cherry picked from commit ed367815eeb9329c48a86a8a7fa3186e27a10f2c) Co-authored-by: Steve Dower --- Lib/test/test_crypt.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_crypt.py b/Lib/test/test_crypt.py index d29e005fdad506..5dc83b4ecbfa00 100644 --- a/Lib/test/test_crypt.py +++ b/Lib/test/test_crypt.py @@ -6,20 +6,21 @@ import crypt IMPORT_ERROR = None except ImportError as ex: + if sys.platform != 'win32': + raise unittest.SkipTest(str(ex)) crypt = None IMPORT_ERROR = str(ex) -@unittest.skipIf(crypt, 'This should only run on windows') +@unittest.skipUnless(sys.platform == 'win32', 'This should only run on windows') +@unittest.skipIf(crypt, 'import succeeded') class TestWhyCryptDidNotImport(unittest.TestCase): - def test_failure_only_for_windows(self): - self.assertEqual(sys.platform, 'win32') def test_import_failure_message(self): self.assertIn('not supported', IMPORT_ERROR) -@unittest.skipUnless(crypt, 'Not supported on Windows') +@unittest.skipUnless(crypt, 'crypt module is required') class CryptTestCase(unittest.TestCase): def test_crypt(self): From e47a7e3a7ab2cfb321e5f276028be286f30bd057 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 9 Jan 2020 09:33:05 -0800 Subject: [PATCH 1083/2163] Fix typo in test's docstring (GH-17856) (GH-17923) * Fix typo in test's docstring. contination -> continuation. (cherry picked from commit 2f65aa465865930f8364645b1466d2751c4086d3) Co-authored-by: Daniel Hahler Co-authored-by: Daniel Hahler --- Lib/test/test_eof.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Lib/test/test_eof.py b/Lib/test/test_eof.py index a091ceaa25bc4f..9ef8eb1187486f 100644 --- a/Lib/test/test_eof.py +++ b/Lib/test/test_eof.py @@ -27,7 +27,7 @@ def test_EOFS(self): raise support.TestFailed def test_line_continuation_EOF(self): - """A contination at the end of input must be an error; bpo2180.""" + """A continuation at the end of input must be an error; bpo2180.""" expect = 'unexpected EOF while parsing (, line 1)' with self.assertRaises(SyntaxError) as excinfo: exec('x = 5\\') From 33e033da3c1472b0aa2ae3cff06649a1ae4aa37f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 9 Jan 2020 11:39:00 -0800 Subject: [PATCH 1084/2163] bpo-39235: Fix end location for genexp in call args (GH-17925) The fix changes copy_location() to require an extra node from which to extract the end location, and fixing all 5 call sites. https://bugs.python.org/issue39235 (cherry picked from commit a796d8ef9dd1af65f7e4d7a857b56f35b7cb6e78) Co-authored-by: Guido van Rossum --- .../2020-01-09-10-01-18.bpo-39235.RYwjoc.rst | 2 ++ Python/ast.c | 16 ++++++++-------- 2 files changed, 10 insertions(+), 8 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-09-10-01-18.bpo-39235.RYwjoc.rst diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-09-10-01-18.bpo-39235.RYwjoc.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-09-10-01-18.bpo-39235.RYwjoc.rst new file mode 100644 index 00000000000000..5fb0d45356badf --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-01-09-10-01-18.bpo-39235.RYwjoc.rst @@ -0,0 +1,2 @@ +Fix AST end location for lone generator expression in function call, e.g. +f(i for i in a). diff --git a/Python/ast.c b/Python/ast.c index 6cf71ce7bb8104..f3263c1e3fcdce 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -1028,13 +1028,13 @@ forbidden_name(struct compiling *c, identifier name, const node *n, } static expr_ty -copy_location(expr_ty e, const node *n) +copy_location(expr_ty e, const node *n, const node *end) { if (e) { e->lineno = LINENO(n); e->col_offset = n->n_col_offset; - e->end_lineno = n->n_end_lineno; - e->end_col_offset = n->n_end_col_offset; + e->end_lineno = end->n_end_lineno; + e->end_col_offset = end->n_end_col_offset; } return e; } @@ -2464,10 +2464,10 @@ ast_for_atom(struct compiling *c, const node *n) } if (TYPE(CHILD(ch, 1)) == comp_for) { - return copy_location(ast_for_genexp(c, ch), n); + return copy_location(ast_for_genexp(c, ch), n, n); } else { - return copy_location(ast_for_testlist(c, ch), n); + return copy_location(ast_for_testlist(c, ch), n, n); } case LSQB: /* list (or list comprehension) */ ch = CHILD(n, 1); @@ -2486,7 +2486,7 @@ ast_for_atom(struct compiling *c, const node *n) n->n_end_lineno, n->n_end_col_offset, c->c_arena); } else { - return copy_location(ast_for_listcomp(c, ch), n); + return copy_location(ast_for_listcomp(c, ch), n, n); } case LBRACE: { /* dictorsetmaker: ( ((test ':' test | '**' test) @@ -2527,7 +2527,7 @@ ast_for_atom(struct compiling *c, const node *n) /* It's a dictionary display. */ res = ast_for_dictdisplay(c, ch); } - return copy_location(res, n); + return copy_location(res, n, n); } } default: @@ -3146,7 +3146,7 @@ ast_for_call(struct compiling *c, const node *n, expr_ty func, } else if (TYPE(CHILD(ch, 1)) == comp_for) { /* the lone generator expression */ - e = copy_location(ast_for_genexp(c, ch), maybegenbeg); + e = copy_location(ast_for_genexp(c, ch), maybegenbeg, closepar); if (!e) return NULL; asdl_seq_SET(args, nargs++, e); From df2fb60cf79bf95497cc15a74d657a3a54a53057 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 10 Jan 2020 04:12:08 -0800 Subject: [PATCH 1085/2163] [3.8] Add test cases for dataclasses. (GH-17909) (GH-17919) * Add test cases for dataclasses. * Add test for repr output of field. * Add test for ValueError to be raised when both default and default_factory are passed. (cherry picked from commit eef1b027ab70704bcaa60a089e4ae1592c504b86) Co-authored-by: Karthikeyan Singaravelan Automerge-Triggered-By: @ericvsmith --- Lib/test/test_dataclasses.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py index 2f3c531fc208ad..21a7d7ec3fe391 100755 --- a/Lib/test/test_dataclasses.py +++ b/Lib/test/test_dataclasses.py @@ -45,6 +45,25 @@ class C: o = C(42) self.assertEqual(o.x, 42) + def test_field_default_default_factory_error(self): + msg = "cannot specify both default and default_factory" + with self.assertRaisesRegex(ValueError, msg): + @dataclass + class C: + x: int = field(default=1, default_factory=int) + + def test_field_repr(self): + int_field = field(default=1, init=True, repr=False) + int_field.name = "id" + repr_output = repr(int_field) + expected_output = "Field(name='id',type=None," \ + f"default=1,default_factory={MISSING!r}," \ + "init=True,repr=False,hash=None," \ + "compare=True,metadata=mappingproxy({})," \ + "_field_type=None)" + + self.assertEqual(repr_output, expected_output) + def test_named_init_params(self): @dataclass class C: From 0f40482fde59ff307569fa5676183dd8432809a8 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 10 Jan 2020 21:39:01 -0800 Subject: [PATCH 1086/2163] Fix host in address of socket.create_server example. (GH-17706) Host as None in address raises TypeError since it should be string, bytes or bytearray. (cherry picked from commit 43682f1e39a3c61f0e8a638b887bcdcbfef766c5) Co-authored-by: Karthikeyan Singaravelan --- Lib/socket.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Lib/socket.py b/Lib/socket.py index 5b17906ef479a1..f83f36d0ada55c 100644 --- a/Lib/socket.py +++ b/Lib/socket.py @@ -843,7 +843,7 @@ def create_server(address, *, family=AF_INET, backlog=None, reuse_port=False, connections. When false it will explicitly disable this option on platforms that enable it by default (e.g. Linux). - >>> with create_server((None, 8000)) as server: + >>> with create_server(('', 8000)) as server: ... while True: ... conn, addr = server.accept() ... # handle new connection From 98b1c0c7ac7c80aac8bce8648fe14b55abeb382a Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 11 Jan 2020 07:56:57 -0800 Subject: [PATCH 1087/2163] bpo-39297: Update for importlib_metadata 1.4. (GH-17947) (GH-17952) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * bpo-39297: Update for importlib_metadata 1.4. Includes performance updates. * 📜🤖 Added by blurb_it. * Update blurb Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> (cherry picked from commit 136735c1a2efb320e4cbb64b40f1191228745b39) Co-authored-by: Jason R. Coombs Co-authored-by: Jason R. Coombs --- Lib/importlib/metadata.py | 108 ++++++++++++------ .../2020-01-11-01-15-37.bpo-39297.y98Z6Q.rst | 1 + 2 files changed, 73 insertions(+), 36 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-01-11-01-15-37.bpo-39297.y98Z6Q.rst diff --git a/Lib/importlib/metadata.py b/Lib/importlib/metadata.py index 53f9fb59346684..ae8ecf9b8500cc 100644 --- a/Lib/importlib/metadata.py +++ b/Lib/importlib/metadata.py @@ -10,6 +10,7 @@ import operator import functools import itertools +import posixpath import collections from configparser import ConfigParser @@ -371,10 +372,6 @@ def path(self): """ return vars(self).get('path', sys.path) - @property - def pattern(self): - return '.*' if self.name is None else re.escape(self.name) - @abc.abstractmethod def find_distributions(self, context=Context()): """ @@ -386,6 +383,73 @@ def find_distributions(self, context=Context()): """ +class FastPath: + """ + Micro-optimized class for searching a path for + children. + """ + + def __init__(self, root): + self.root = root + + def joinpath(self, child): + return pathlib.Path(self.root, child) + + def children(self): + with suppress(Exception): + return os.listdir(self.root or '') + with suppress(Exception): + return self.zip_children() + return [] + + def zip_children(self): + zip_path = zipfile.Path(self.root) + names = zip_path.root.namelist() + self.joinpath = zip_path.joinpath + + return ( + posixpath.split(child)[0] + for child in names + ) + + def is_egg(self, search): + root_n_low = os.path.split(self.root)[1].lower() + + return ( + root_n_low == search.normalized + '.egg' + or root_n_low.startswith(search.prefix) + and root_n_low.endswith('.egg')) + + def search(self, name): + for child in self.children(): + n_low = child.lower() + if (n_low in name.exact_matches + or n_low.startswith(name.prefix) + and n_low.endswith(name.suffixes) + # legacy case: + or self.is_egg(name) and n_low == 'egg-info'): + yield self.joinpath(child) + + +class Prepared: + """ + A prepared search for metadata on a possibly-named package. + """ + normalized = '' + prefix = '' + suffixes = '.dist-info', '.egg-info' + exact_matches = [''][:0] + + def __init__(self, name): + self.name = name + if name is None: + return + self.normalized = name.lower().replace('-', '_') + self.prefix = self.normalized + '-' + self.exact_matches = [ + self.normalized + suffix for suffix in self.suffixes] + + class MetadataPathFinder(DistributionFinder): @classmethod def find_distributions(cls, context=DistributionFinder.Context()): @@ -397,45 +461,17 @@ def find_distributions(cls, context=DistributionFinder.Context()): (or all names if ``None`` indicated) along the paths in the list of directories ``context.path``. """ - found = cls._search_paths(context.pattern, context.path) + found = cls._search_paths(context.name, context.path) return map(PathDistribution, found) @classmethod - def _search_paths(cls, pattern, paths): + def _search_paths(cls, name, paths): """Find metadata directories in paths heuristically.""" return itertools.chain.from_iterable( - cls._search_path(path, pattern) - for path in map(cls._switch_path, paths) + path.search(Prepared(name)) + for path in map(FastPath, paths) ) - @staticmethod - def _switch_path(path): - PYPY_OPEN_BUG = False - if not PYPY_OPEN_BUG or os.path.isfile(path): # pragma: no branch - with suppress(Exception): - return zipfile.Path(path) - return pathlib.Path(path) - - @classmethod - def _matches_info(cls, normalized, item): - template = r'{pattern}(-.*)?\.(dist|egg)-info' - manifest = template.format(pattern=normalized) - return re.match(manifest, item.name, flags=re.IGNORECASE) - - @classmethod - def _matches_legacy(cls, normalized, item): - template = r'{pattern}-.*\.egg[\\/]EGG-INFO' - manifest = template.format(pattern=normalized) - return re.search(manifest, str(item), flags=re.IGNORECASE) - - @classmethod - def _search_path(cls, root, pattern): - if not root.is_dir(): - return () - normalized = pattern.replace('-', '_') - return (item for item in root.iterdir() - if cls._matches_info(normalized, item) - or cls._matches_legacy(normalized, item)) class PathDistribution(Distribution): diff --git a/Misc/NEWS.d/next/Library/2020-01-11-01-15-37.bpo-39297.y98Z6Q.rst b/Misc/NEWS.d/next/Library/2020-01-11-01-15-37.bpo-39297.y98Z6Q.rst new file mode 100644 index 00000000000000..618f6f9f2b7ff2 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-11-01-15-37.bpo-39297.y98Z6Q.rst @@ -0,0 +1 @@ +Improved performance of importlib.metadata distribution discovery and resilients to inaccessible sys.path entries (importlib_metadata v1.4.0). From a240f0545653d961c67c1eb1597cb70b67f4e5f1 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 12 Jan 2020 03:03:15 -0800 Subject: [PATCH 1088/2163] bpo-39233: Update positional-only section in the glossary (GH-17874) https://bugs.python.org/issue39233 (cherry picked from commit 9a669d58e8cb586fba38c84d5b631cd8a95d0c0c) Co-authored-by: Pablo Galindo --- Doc/glossary.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 9ce0357f1cb424..6189cb045049c2 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -824,9 +824,11 @@ Glossary .. _positional-only_parameter: * :dfn:`positional-only`: specifies an argument that can be supplied only - by position. Python has no syntax for defining positional-only - parameters. However, some built-in functions have positional-only - parameters (e.g. :func:`abs`). + by position. Positional-only parameters can be defined by including a + ``/`` character in the parameter list of the function definition after + them, for example *posonly1* and *posonly2* in the following:: + + def func(posonly1, posonly2, /, positional_or_keyword): ... .. _keyword-only_parameter: From 33dd75a28fe2ec6e85c5d3b315b5a9d4cf0652db Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 12 Jan 2020 03:21:00 -0800 Subject: [PATCH 1089/2163] bpo-38356: Fix ThreadedChildWatcher thread leak in test_asyncio (GH-16552) Motivation for this PR (comment from @vstinner in bpo issue): ``` Warning seen o AMD64 Ubuntu Shared 3.x buildbot: https://buildbot.python.org/all/GH-/builders/141/builds/2593 test_devnull_output (test.test_a=syncio.test_subprocess.SubprocessThreadedWatcherTests) ... Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 2) ``` The following implementation details for the new method are TBD: 1) Public vs private 2) Inclusion in `close()` 3) Name 4) Coroutine vs subroutine method 5) *timeout* parameter If it's a private method, 3, 4, and 5 are significantly less important. I started with the most minimal implementation that fixes the dangling threads without modifying the regression tests, which I think is particularly important. I typically try to avoid directly modifying existing tests as much as possible unless it's necessary to do so. However, I am open to changing any part of this. https://bugs.python.org/issue38356 (cherry picked from commit 0ca7cc7fc0518c24dc9b78c38418e6064e64f148) Co-authored-by: Kyle Stanley --- Lib/asyncio/unix_events.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py index d8f653045aee4c..8c0a57482b7a49 100644 --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -1266,7 +1266,14 @@ def is_active(self): return True def close(self): - pass + self._join_threads() + + def _join_threads(self): + """Internal: Join all non-daemon threads""" + threads = [thread for thread in list(self._threads.values()) + if thread.is_alive() and not thread.daemon] + for thread in threads: + thread.join() def __enter__(self): return self From eb9ba2f66df2195a9c6295e73ab3d545a1445f05 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 12 Jan 2020 03:41:07 -0800 Subject: [PATCH 1090/2163] bpo-16575: Disabled checks for union types being passed by value. (GH-17960) (GH-17964) Although the underlying libffi issue remains open, adding these checks have caused problems in third-party projects which are in widespread use. See the issue for examples. The corresponding tests have also been skipped. (cherry picked from commit c12440c371025bea9c3bfb94945f006c486c2c01) --- Lib/ctypes/test/test_structures.py | 3 ++- Modules/_ctypes/_ctypes.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py index 19c4430bea344d..cdbaa7fbd65baf 100644 --- a/Lib/ctypes/test/test_structures.py +++ b/Lib/ctypes/test/test_structures.py @@ -571,6 +571,7 @@ class U(Union): self.assertEqual(f2, [0x4567, 0x0123, 0xcdef, 0x89ab, 0x3210, 0x7654, 0xba98, 0xfedc]) + @unittest.skipIf(True, 'Test disabled for now - see bpo-16575/bpo-16576') def test_union_by_value(self): # See bpo-16575 @@ -651,7 +652,7 @@ class Test5(Structure): self.assertEqual(test5.nested.an_int, 0) self.assertEqual(test5.another_int, 0) - #@unittest.skipIf('s390' in MACHINE, 'Test causes segfault on S390') + @unittest.skipIf(True, 'Test disabled for now - see bpo-16575/bpo-16576') def test_bitfield_by_value(self): # See bpo-16576 diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index c6da0d804453e7..b10b86725ff171 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -2401,6 +2401,23 @@ converters_from_argtypes(PyObject *ob) for (i = 0; i < nArgs; ++i) { PyObject *cnv; PyObject *tp = PyTuple_GET_ITEM(ob, i); +/* + * The following checks, relating to bpo-16575 and bpo-16576, have been + * disabled. The reason is that, although there is a definite problem with + * how libffi handles unions (https://github.com/libffi/libffi/issues/33), + * there are numerous libraries which pass structures containing unions + * by values - especially on Windows but examples also exist on Linux + * (https://bugs.python.org/msg359834). + * + * It may not be possible to get proper support for unions and bitfields + * until support is forthcoming in libffi, but for now, adding the checks + * has caused problems in otherwise-working software, which suggests it + * is better to disable the checks. + * + * Although specific examples reported relate specifically to unions and + * not bitfields, the bitfields check is also being disabled as a + * precaution. + StgDictObject *stgdict = PyType_stgdict(tp); if (stgdict != NULL) { @@ -2428,6 +2445,7 @@ converters_from_argtypes(PyObject *ob) return NULL; } } + */ if (_PyObject_LookupAttrId(tp, &PyId_from_param, &cnv) <= 0) { Py_DECREF(converters); From 3043ec7d6aed402218404c25179e734166c7fbe0 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 12 Jan 2020 10:04:18 -0800 Subject: [PATCH 1091/2163] bpo-38293: Allow shallow and deep copying of property objects (GH-16438) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copying property objects results in a TypeError. Steps to reproduce: ``` >>> import copy >>> obj = property() >>> copy.copy(obj) ```` This affects both shallow and deep copying. My idea for a fix is to add property objects to the list of "atomic" objects in the copy module. These already include types like functions and type objects. I also added property objects to the unit tests test_copy_atomic and test_deepcopy_atomic. This is my first PR, and it's highly likely I've made some mistake, so please be kind :) https://bugs.python.org/issue38293 (cherry picked from commit 9f3fc6c5b4993f2b362263b494f84793a21aa073) Co-authored-by: Guðni Natan Gunnarsson <1493259+GudniNatan@users.noreply.github.com> --- Lib/copy.py | 3 ++- Lib/test/test_copy.py | 4 ++-- .../next/Library/2019-09-29-08-17-03.bpo-38293.wls5s3.rst | 1 + 3 files changed, 5 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-09-29-08-17-03.bpo-38293.wls5s3.rst diff --git a/Lib/copy.py b/Lib/copy.py index f53cd8c5874954..41873f2c046cac 100644 --- a/Lib/copy.py +++ b/Lib/copy.py @@ -107,7 +107,7 @@ def copy(x): def _copy_immutable(x): return x for t in (type(None), int, float, bool, complex, str, tuple, - bytes, frozenset, type, range, slice, + bytes, frozenset, type, range, slice, property, types.BuiltinFunctionType, type(Ellipsis), type(NotImplemented), types.FunctionType, weakref.ref): d[t] = _copy_immutable @@ -195,6 +195,7 @@ def _deepcopy_atomic(x, memo): d[types.BuiltinFunctionType] = _deepcopy_atomic d[types.FunctionType] = _deepcopy_atomic d[weakref.ref] = _deepcopy_atomic +d[property] = _deepcopy_atomic def _deepcopy_list(x, memo, deepcopy=deepcopy): y = [] diff --git a/Lib/test/test_copy.py b/Lib/test/test_copy.py index 45a692022f29bc..35f72fb216b64a 100644 --- a/Lib/test/test_copy.py +++ b/Lib/test/test_copy.py @@ -99,7 +99,7 @@ class WithMetaclass(metaclass=abc.ABCMeta): 42, 2**100, 3.14, True, False, 1j, "hello", "hello\u1234", f.__code__, b"world", bytes(range(256)), range(10), slice(1, 10, 2), - NewStyle, Classic, max, WithMetaclass] + NewStyle, Classic, max, WithMetaclass, property()] for x in tests: self.assertIs(copy.copy(x), x) @@ -357,7 +357,7 @@ def f(): pass tests = [None, 42, 2**100, 3.14, True, False, 1j, "hello", "hello\u1234", f.__code__, - NewStyle, Classic, max] + NewStyle, Classic, max, property()] for x in tests: self.assertIs(copy.deepcopy(x), x) diff --git a/Misc/NEWS.d/next/Library/2019-09-29-08-17-03.bpo-38293.wls5s3.rst b/Misc/NEWS.d/next/Library/2019-09-29-08-17-03.bpo-38293.wls5s3.rst new file mode 100644 index 00000000000000..0b19551970eb09 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-09-29-08-17-03.bpo-38293.wls5s3.rst @@ -0,0 +1 @@ +Add :func:`copy.copy` and :func:`copy.deepcopy` support to :func:`property` objects. \ No newline at end of file From ef0af30e507a29dae03aae40459b9c44c96f260d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 12 Jan 2020 12:44:33 -0800 Subject: [PATCH 1092/2163] bpo-3530: Add advice on when to correctly use fix_missing_locations in the AST docs (GH-17172) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Pablo Galindo (cherry picked from commit 6680f4a9f5d15ab82b2ab6266c6f917cb78c919a) Co-authored-by: Batuhan TaĹźkaya <47358913+isidentical@users.noreply.github.com> --- Doc/library/ast.rst | 10 +++++++++- .../2019-11-17-11-53-10.bpo-3530.8zFUMc.rst | 2 ++ 2 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Documentation/2019-11-17-11-53-10.bpo-3530.8zFUMc.rst diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index 92bf8912eb53dd..a5dd0e1cc70dd6 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -300,7 +300,7 @@ and classes for traversing abstract syntax trees: class RewriteName(NodeTransformer): def visit_Name(self, node): - return copy_location(Subscript( + return Subscript( value=Name(id='data', ctx=Load()), slice=Index(value=Constant(value=node.id)), ctx=node.ctx @@ -314,6 +314,14 @@ and classes for traversing abstract syntax trees: statement nodes), the visitor may also return a list of nodes rather than just a single node. + If :class:`NodeTransformer` introduces new nodes (that weren't part of + original tree) without giving them location information (such as + :attr:`lineno`), :func:`fix_missing_locations` should be called with + the new sub-tree to recalculate the location information:: + + tree = ast.parse('foo', mode='eval') + new_tree = fix_missing_locations(RewriteName().visit(tree)) + Usually you use the transformer like this:: node = YourTransformer().visit(node) diff --git a/Misc/NEWS.d/next/Documentation/2019-11-17-11-53-10.bpo-3530.8zFUMc.rst b/Misc/NEWS.d/next/Documentation/2019-11-17-11-53-10.bpo-3530.8zFUMc.rst new file mode 100644 index 00000000000000..65f1a6d156a120 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2019-11-17-11-53-10.bpo-3530.8zFUMc.rst @@ -0,0 +1,2 @@ +In the :mod:`ast` module documentation, fix a misleading ``NodeTransformer`` example and add +advice on when to use the ``fix_missing_locations`` function. From f1f0c58d3835b63692600436c70478f638c3352f Mon Sep 17 00:00:00 2001 From: toonarmycaptain Date: Sun, 12 Jan 2020 16:56:26 -0600 Subject: [PATCH 1093/2163] [3.8] Fix typos in Misc/NEWS.d (GH-17930) --- Misc/NEWS.d/3.8.0.rst | 4 ++-- Misc/NEWS.d/3.8.1.rst | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Misc/NEWS.d/3.8.0.rst b/Misc/NEWS.d/3.8.0.rst index b7e681f8333bd5..e72025cfca572b 100644 --- a/Misc/NEWS.d/3.8.0.rst +++ b/Misc/NEWS.d/3.8.0.rst @@ -101,7 +101,7 @@ now never fails (except MemoryError). .. nonce: go_jFf .. section: Documentation -Add list of no-longer-escaped chars to re.escape documentation +Add list of no-longer-escaped chars to re.escape documentation. .. @@ -123,7 +123,7 @@ Python slowest buildbots. .. nonce: scr2LO .. section: Windows -Fix error message in activate.bat +Fix error message in activate.bat. .. diff --git a/Misc/NEWS.d/3.8.1.rst b/Misc/NEWS.d/3.8.1.rst index 8cd4b359f7b4f0..55d23dd7fa53e0 100644 --- a/Misc/NEWS.d/3.8.1.rst +++ b/Misc/NEWS.d/3.8.1.rst @@ -26,7 +26,7 @@ the "elif" keyword and not to its condition, making it consistent with the .. section: Core and Builtins :c:func:`PySys_Audit` now requires ``Py_ssize_t`` to be used for size -arguments in the format string, regardless of whethen ``PY_SSIZE_T_CLEAN`` +arguments in the format string, regardless of whether ``PY_SSIZE_T_CLEAN`` was defined at include time. .. @@ -36,7 +36,7 @@ was defined at include time. .. nonce: QDtIxI .. section: Library -Update importliib.metadata to include improvements from importlib_metadata +Update importlib.metadata to include improvements from importlib_metadata 1.3 including better serialization of EntryPoints and improved documentation for custom finders. @@ -95,7 +95,7 @@ Prevent failure of test_relative_path in test_py_compile on macOS Catalina. .. nonce: _3xjKG .. section: IDLE -Excape key now closes IDLE completion windows. Patch by Johnny Najera. +Escape key now closes IDLE completion windows. Patch by Johnny Najera. .. From 9671b6b3b3e480fbc4b14aab7008b90b38767f55 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 13 Jan 2020 02:54:24 -0800 Subject: [PATCH 1094/2163] bpo-39307: Fix memory leak on error path in parsetok (GH-17953) (cherry picked from commit 7ba6f18de2582755ae31888ba6a4237d96dddc48) Co-authored-by: Alex Henrie --- Parser/parsetok.c | 1 + 1 file changed, 1 insertion(+) diff --git a/Parser/parsetok.c b/Parser/parsetok.c index a5d78974b871b3..2bb733d0dcd66b 100644 --- a/Parser/parsetok.c +++ b/Parser/parsetok.c @@ -246,6 +246,7 @@ parsetok(struct tok_state *tok, grammar *g, int start, perrdetail *err_ret, if ((ps = PyParser_New(g, start)) == NULL) { err_ret->error = E_NOMEM; + growable_comment_array_deallocate(&type_ignores); PyTokenizer_Free(tok); return NULL; } From 9362f8526e42157baf27df982b16f23f212c3c3a Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 13 Jan 2020 20:24:12 +0100 Subject: [PATCH 1095/2163] bpo-39310: Update sys.float_info documentation (GH-17982) Specify that sys.float_info.min is only the minimum normalized float. --- Doc/library/sys.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 1cf19b8ad6c025..e1d93f85bd4071 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -479,8 +479,8 @@ always available. +---------------------+----------------+--------------------------------------------------+ | attribute | float.h macro | explanation | +=====================+================+==================================================+ - | :const:`epsilon` | DBL_EPSILON | difference between 1 and the least value greater | - | | | than 1 that is representable as a float | + | :const:`epsilon` | DBL_EPSILON | difference between 1.0 and the least value | + | | | greater than 1.0 that is representable as a float| +---------------------+----------------+--------------------------------------------------+ | :const:`dig` | DBL_DIG | maximum number of decimal digits that can be | | | | faithfully represented in a float; see below | @@ -488,20 +488,20 @@ always available. | :const:`mant_dig` | DBL_MANT_DIG | float precision: the number of base-``radix`` | | | | digits in the significand of a float | +---------------------+----------------+--------------------------------------------------+ - | :const:`max` | DBL_MAX | maximum representable finite float | + | :const:`max` | DBL_MAX | maximum representable positive finite float | +---------------------+----------------+--------------------------------------------------+ - | :const:`max_exp` | DBL_MAX_EXP | maximum integer e such that ``radix**(e-1)`` is | + | :const:`max_exp` | DBL_MAX_EXP | maximum integer *e* such that ``radix**(e-1)`` is| | | | a representable finite float | +---------------------+----------------+--------------------------------------------------+ - | :const:`max_10_exp` | DBL_MAX_10_EXP | maximum integer e such that ``10**e`` is in the | + | :const:`max_10_exp` | DBL_MAX_10_EXP | maximum integer *e* such that ``10**e`` is in the| | | | range of representable finite floats | +---------------------+----------------+--------------------------------------------------+ - | :const:`min` | DBL_MIN | minimum positive normalized float | + | :const:`min` | DBL_MIN | minimum representable positive *normalized* float| +---------------------+----------------+--------------------------------------------------+ - | :const:`min_exp` | DBL_MIN_EXP | minimum integer e such that ``radix**(e-1)`` is | + | :const:`min_exp` | DBL_MIN_EXP | minimum integer *e* such that ``radix**(e-1)`` is| | | | a normalized float | +---------------------+----------------+--------------------------------------------------+ - | :const:`min_10_exp` | DBL_MIN_10_EXP | minimum integer e such that ``10**e`` is a | + | :const:`min_10_exp` | DBL_MIN_10_EXP | minimum integer *e* such that ``10**e`` is a | | | | normalized float | +---------------------+----------------+--------------------------------------------------+ | :const:`radix` | FLT_RADIX | radix of exponent representation | From 9955f33cdbf27de270038dfbad37d15b160ecca2 Mon Sep 17 00:00:00 2001 From: Karthikeyan Singaravelan Date: Tue, 14 Jan 2020 17:09:19 +0530 Subject: [PATCH 1096/2163] [3.8] bpo-39033: Fix NameError in zipimport during hash validation (GH-17588) (GH-17642) Fix `NameError` in `zipimport` during hash validation and add a regression test. (cherry picked from commit 79f02fee1a542c440fd906fd54154c73fc0f8235) https://bugs.python.org/issue39033 --- Lib/test/test_zipimport.py | 16 + Lib/zipimport.py | 2 +- .../2019-12-13-18-54-49.bpo-39033.cepuyD.rst | 1 + Python/importlib_zipimport.h | 540 +++++++++--------- 4 files changed, 287 insertions(+), 272 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-13-18-54-49.bpo-39033.cepuyD.rst diff --git a/Lib/test/test_zipimport.py b/Lib/test/test_zipimport.py index d4f619e9acdb9b..2af8689c1d2cb2 100644 --- a/Lib/test/test_zipimport.py +++ b/Lib/test/test_zipimport.py @@ -6,6 +6,7 @@ import struct import time import unittest +import unittest.mock from test import support @@ -204,6 +205,21 @@ def check(mod): self.assertEqual(mod.state, 'old') self.doTest(None, files, TESTMOD, call=check) + @unittest.mock.patch('_imp.check_hash_based_pycs', 'always') + def test_checked_hash_based_change_pyc(self): + source = b"state = 'old'" + source_hash = importlib.util.source_hash(source) + bytecode = importlib._bootstrap_external._code_to_hash_pyc( + compile(source, "???", "exec"), + source_hash, + False, + ) + files = {TESTMOD + ".py": (NOW, "state = 'new'"), + TESTMOD + ".pyc": (NOW - 20, bytecode)} + def check(mod): + self.assertEqual(mod.state, 'new') + self.doTest(None, files, TESTMOD, call=check) + def testEmptyPy(self): files = {TESTMOD + ".py": (NOW, "")} self.doTest(None, files, TESTMOD) diff --git a/Lib/zipimport.py b/Lib/zipimport.py index fd917c16b01535..5ef0a17c2a5ed2 100644 --- a/Lib/zipimport.py +++ b/Lib/zipimport.py @@ -608,7 +608,7 @@ def _unmarshal_code(self, pathname, fullpath, fullname, data): ) try: - _boostrap_external._validate_hash_pyc( + _bootstrap_external._validate_hash_pyc( data, source_hash, fullname, exc_details) except ImportError: return None diff --git a/Misc/NEWS.d/next/Library/2019-12-13-18-54-49.bpo-39033.cepuyD.rst b/Misc/NEWS.d/next/Library/2019-12-13-18-54-49.bpo-39033.cepuyD.rst new file mode 100644 index 00000000000000..3dee3c08cc5bea --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-13-18-54-49.bpo-39033.cepuyD.rst @@ -0,0 +1 @@ +Fix :exc:`NameError` in :mod:`zipimport`. Patch by Karthikeyan Singaravelan. diff --git a/Python/importlib_zipimport.h b/Python/importlib_zipimport.h index 6c9c4e8465523d..f013b41457fd43 100644 --- a/Python/importlib_zipimport.h +++ b/Python/importlib_zipimport.h @@ -792,17 +792,17 @@ const unsigned char _Py_M__zipimport[] = { 100,5,107,3,114,180,124,8,115,104,116,3,106,4,100,6, 107,2,114,180,116,5,124,0,124,2,131,2,125,9,124,9, 100,0,107,9,114,180,116,3,160,6,116,0,106,7,124,9, - 161,2,125,10,122,20,116,8,160,9,124,4,124,10,124,3, + 161,2,125,10,122,20,116,0,160,8,124,4,124,10,124,3, 124,5,161,4,1,0,87,0,110,22,4,0,116,2,107,10, 114,178,1,0,1,0,1,0,89,0,100,0,83,0,88,0, - 110,84,116,10,124,0,124,2,131,2,92,2,125,11,125,12, - 124,11,144,1,114,10,116,11,116,12,124,4,100,7,100,8, - 133,2,25,0,131,1,124,11,131,2,114,246,116,12,124,4, + 110,84,116,9,124,0,124,2,131,2,92,2,125,11,125,12, + 124,11,144,1,114,10,116,10,116,11,124,4,100,7,100,8, + 133,2,25,0,131,1,124,11,131,2,114,246,116,11,124,4, 100,8,100,9,133,2,25,0,131,1,124,12,107,3,144,1, - 114,10,116,13,160,14,100,10,124,3,155,2,157,2,161,1, - 1,0,100,0,83,0,116,15,160,16,124,4,100,9,100,0, - 133,2,25,0,161,1,125,13,116,17,124,13,116,18,131,2, - 144,1,115,56,116,19,100,11,124,1,155,2,100,12,157,3, + 114,10,116,12,160,13,100,10,124,3,155,2,157,2,161,1, + 1,0,100,0,83,0,116,14,160,15,124,4,100,9,100,0, + 133,2,25,0,161,1,125,13,116,16,124,13,116,17,131,2, + 144,1,115,56,116,18,100,11,124,1,155,2,100,12,157,3, 131,1,130,1,124,13,83,0,41,13,78,41,2,114,59,0, 0,0,114,13,0,0,0,114,5,0,0,0,114,0,0,0, 0,114,86,0,0,0,90,5,110,101,118,101,114,90,6,97, @@ -811,273 +811,271 @@ const unsigned char _Py_M__zipimport[] = { 115,32,115,116,97,108,101,32,102,111,114,32,122,16,99,111, 109,112,105,108,101,100,32,109,111,100,117,108,101,32,122,21, 32,105,115,32,110,111,116,32,97,32,99,111,100,101,32,111, - 98,106,101,99,116,41,20,114,21,0,0,0,90,13,95,99, + 98,106,101,99,116,41,19,114,21,0,0,0,90,13,95,99, 108,97,115,115,105,102,121,95,112,121,99,114,75,0,0,0, 218,4,95,105,109,112,90,21,99,104,101,99,107,95,104,97, 115,104,95,98,97,115,101,100,95,112,121,99,115,218,15,95, 103,101,116,95,112,121,99,95,115,111,117,114,99,101,218,11, 115,111,117,114,99,101,95,104,97,115,104,90,17,95,82,65, 87,95,77,65,71,73,67,95,78,85,77,66,69,82,90,18, - 95,98,111,111,115,116,114,97,112,95,101,120,116,101,114,110, - 97,108,90,18,95,118,97,108,105,100,97,116,101,95,104,97, - 115,104,95,112,121,99,218,29,95,103,101,116,95,109,116,105, - 109,101,95,97,110,100,95,115,105,122,101,95,111,102,95,115, - 111,117,114,99,101,114,147,0,0,0,114,2,0,0,0,114, - 76,0,0,0,114,77,0,0,0,218,7,109,97,114,115,104, - 97,108,90,5,108,111,97,100,115,114,15,0,0,0,218,10, - 95,99,111,100,101,95,116,121,112,101,218,9,84,121,112,101, - 69,114,114,111,114,41,14,114,32,0,0,0,114,53,0,0, - 0,114,63,0,0,0,114,38,0,0,0,114,126,0,0,0, - 90,11,101,120,99,95,100,101,116,97,105,108,115,114,129,0, - 0,0,90,10,104,97,115,104,95,98,97,115,101,100,90,12, - 99,104,101,99,107,95,115,111,117,114,99,101,90,12,115,111, - 117,114,99,101,95,98,121,116,101,115,114,150,0,0,0,90, - 12,115,111,117,114,99,101,95,109,116,105,109,101,90,11,115, - 111,117,114,99,101,95,115,105,122,101,114,46,0,0,0,114, - 9,0,0,0,114,9,0,0,0,114,10,0,0,0,218,15, - 95,117,110,109,97,114,115,104,97,108,95,99,111,100,101,75, - 2,0,0,115,88,0,0,0,0,2,2,1,2,254,6,5, - 2,1,18,1,14,1,8,2,12,1,4,1,12,1,10,1, - 2,255,2,1,8,255,2,2,10,1,8,1,4,1,4,1, - 2,254,4,5,2,1,4,1,2,0,2,0,2,0,2,255, - 8,2,14,1,10,3,8,255,6,3,6,3,22,1,18,255, - 4,2,4,1,8,255,4,2,4,2,18,1,12,1,16,1, - 114,155,0,0,0,99,1,0,0,0,0,0,0,0,0,0, - 0,0,1,0,0,0,4,0,0,0,67,0,0,0,115,28, - 0,0,0,124,0,160,0,100,1,100,2,161,2,125,0,124, - 0,160,0,100,3,100,2,161,2,125,0,124,0,83,0,41, - 4,78,115,2,0,0,0,13,10,243,1,0,0,0,10,243, - 1,0,0,0,13,41,1,114,19,0,0,0,41,1,218,6, - 115,111,117,114,99,101,114,9,0,0,0,114,9,0,0,0, - 114,10,0,0,0,218,23,95,110,111,114,109,97,108,105,122, - 101,95,108,105,110,101,95,101,110,100,105,110,103,115,126,2, - 0,0,115,6,0,0,0,0,1,12,1,12,1,114,159,0, - 0,0,99,2,0,0,0,0,0,0,0,0,0,0,0,2, - 0,0,0,6,0,0,0,67,0,0,0,115,24,0,0,0, - 116,0,124,1,131,1,125,1,116,1,124,1,124,0,100,1, - 100,2,100,3,141,4,83,0,41,4,78,114,74,0,0,0, - 84,41,1,90,12,100,111,110,116,95,105,110,104,101,114,105, - 116,41,2,114,159,0,0,0,218,7,99,111,109,112,105,108, - 101,41,2,114,53,0,0,0,114,158,0,0,0,114,9,0, - 0,0,114,9,0,0,0,114,10,0,0,0,218,15,95,99, - 111,109,112,105,108,101,95,115,111,117,114,99,101,133,2,0, - 0,115,4,0,0,0,0,1,8,1,114,161,0,0,0,99, - 2,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0, - 11,0,0,0,67,0,0,0,115,68,0,0,0,116,0,160, - 1,124,0,100,1,63,0,100,2,23,0,124,0,100,3,63, - 0,100,4,64,0,124,0,100,5,64,0,124,1,100,6,63, - 0,124,1,100,3,63,0,100,7,64,0,124,1,100,5,64, - 0,100,8,20,0,100,9,100,9,100,9,102,9,161,1,83, - 0,41,10,78,233,9,0,0,0,105,188,7,0,0,233,5, - 0,0,0,233,15,0,0,0,233,31,0,0,0,233,11,0, - 0,0,233,63,0,0,0,114,86,0,0,0,114,14,0,0, - 0,41,2,114,131,0,0,0,90,6,109,107,116,105,109,101, - 41,2,218,1,100,114,138,0,0,0,114,9,0,0,0,114, - 9,0,0,0,114,10,0,0,0,218,14,95,112,97,114,115, - 101,95,100,111,115,116,105,109,101,139,2,0,0,115,22,0, - 0,0,0,1,4,1,10,1,10,1,6,1,6,1,10,1, - 10,1,2,0,2,0,2,249,114,169,0,0,0,99,2,0, - 0,0,0,0,0,0,0,0,0,0,6,0,0,0,10,0, - 0,0,67,0,0,0,115,116,0,0,0,122,82,124,1,100, - 1,100,0,133,2,25,0,100,2,107,6,115,22,116,0,130, - 1,124,1,100,0,100,1,133,2,25,0,125,1,124,0,106, - 1,124,1,25,0,125,2,124,2,100,3,25,0,125,3,124, - 2,100,4,25,0,125,4,124,2,100,5,25,0,125,5,116, - 2,124,4,124,3,131,2,124,5,102,2,87,0,83,0,4, - 0,116,3,116,4,116,5,102,3,107,10,114,110,1,0,1, - 0,1,0,89,0,100,6,83,0,88,0,100,0,83,0,41, - 7,78,114,14,0,0,0,169,2,218,1,99,218,1,111,114, - 163,0,0,0,233,6,0,0,0,233,3,0,0,0,41,2, - 114,0,0,0,0,114,0,0,0,0,41,6,218,14,65,115, - 115,101,114,116,105,111,110,69,114,114,111,114,114,28,0,0, - 0,114,169,0,0,0,114,26,0,0,0,218,10,73,110,100, - 101,120,69,114,114,111,114,114,154,0,0,0,41,6,114,32, - 0,0,0,114,13,0,0,0,114,54,0,0,0,114,131,0, - 0,0,114,132,0,0,0,90,17,117,110,99,111,109,112,114, - 101,115,115,101,100,95,115,105,122,101,114,9,0,0,0,114, - 9,0,0,0,114,10,0,0,0,114,151,0,0,0,152,2, - 0,0,115,20,0,0,0,0,1,2,2,20,1,12,1,10, - 3,8,1,8,1,8,1,16,1,20,1,114,151,0,0,0, - 99,2,0,0,0,0,0,0,0,0,0,0,0,3,0,0, - 0,8,0,0,0,67,0,0,0,115,86,0,0,0,124,1, - 100,1,100,0,133,2,25,0,100,2,107,6,115,20,116,0, - 130,1,124,1,100,0,100,1,133,2,25,0,125,1,122,14, - 124,0,106,1,124,1,25,0,125,2,87,0,110,22,4,0, - 116,2,107,10,114,68,1,0,1,0,1,0,89,0,100,0, - 83,0,88,0,116,3,124,0,106,4,124,2,131,2,83,0, - 100,0,83,0,41,3,78,114,14,0,0,0,114,170,0,0, - 0,41,5,114,175,0,0,0,114,28,0,0,0,114,26,0, - 0,0,114,52,0,0,0,114,29,0,0,0,41,3,114,32, - 0,0,0,114,13,0,0,0,114,54,0,0,0,114,9,0, - 0,0,114,9,0,0,0,114,10,0,0,0,114,149,0,0, - 0,171,2,0,0,115,14,0,0,0,0,2,20,1,12,2, - 2,1,14,1,14,1,8,2,114,149,0,0,0,99,2,0, - 0,0,0,0,0,0,0,0,0,0,11,0,0,0,9,0, - 0,0,67,0,0,0,115,198,0,0,0,116,0,124,0,124, - 1,131,2,125,2,116,1,68,0,93,160,92,3,125,3,125, - 4,125,5,124,2,124,3,23,0,125,6,116,2,106,3,100, - 1,124,0,106,4,116,5,124,6,100,2,100,3,141,5,1, - 0,122,14,124,0,106,6,124,6,25,0,125,7,87,0,110, - 20,4,0,116,7,107,10,114,88,1,0,1,0,1,0,89, - 0,113,14,88,0,124,7,100,4,25,0,125,8,116,8,124, - 0,106,4,124,7,131,2,125,9,124,4,114,132,116,9,124, - 0,124,8,124,6,124,1,124,9,131,5,125,10,110,10,116, - 10,124,8,124,9,131,2,125,10,124,10,100,0,107,8,114, - 152,113,14,124,7,100,4,25,0,125,8,124,10,124,5,124, - 8,102,3,2,0,1,0,83,0,113,14,116,11,100,5,124, - 1,155,2,157,2,124,1,100,6,141,2,130,1,100,0,83, - 0,41,7,78,122,13,116,114,121,105,110,103,32,123,125,123, - 125,123,125,114,86,0,0,0,41,1,90,9,118,101,114,98, - 111,115,105,116,121,114,0,0,0,0,114,57,0,0,0,114, - 58,0,0,0,41,12,114,36,0,0,0,114,89,0,0,0, - 114,76,0,0,0,114,77,0,0,0,114,29,0,0,0,114, - 20,0,0,0,114,28,0,0,0,114,26,0,0,0,114,52, - 0,0,0,114,155,0,0,0,114,161,0,0,0,114,3,0, - 0,0,41,11,114,32,0,0,0,114,38,0,0,0,114,13, - 0,0,0,114,90,0,0,0,114,91,0,0,0,114,47,0, - 0,0,114,63,0,0,0,114,54,0,0,0,114,40,0,0, - 0,114,126,0,0,0,114,46,0,0,0,114,9,0,0,0, - 114,9,0,0,0,114,10,0,0,0,114,44,0,0,0,186, - 2,0,0,115,36,0,0,0,0,1,10,1,14,1,8,1, - 22,1,2,1,14,1,14,1,6,2,8,1,12,1,4,1, - 18,2,10,1,8,3,2,1,8,1,16,2,114,44,0,0, - 0,99,0,0,0,0,0,0,0,0,0,0,0,0,0,0, - 0,0,2,0,0,0,64,0,0,0,115,60,0,0,0,101, - 0,90,1,100,0,90,2,100,1,90,3,100,2,90,4,100, - 3,100,4,132,0,90,5,100,5,100,6,132,0,90,6,100, - 7,100,8,132,0,90,7,100,9,100,10,132,0,90,8,100, - 11,100,12,132,0,90,9,100,13,83,0,41,14,114,80,0, - 0,0,122,165,80,114,105,118,97,116,101,32,99,108,97,115, - 115,32,117,115,101,100,32,116,111,32,115,117,112,112,111,114, - 116,32,90,105,112,73,109,112,111,114,116,46,103,101,116,95, - 114,101,115,111,117,114,99,101,95,114,101,97,100,101,114,40, - 41,46,10,10,32,32,32,32,84,104,105,115,32,99,108,97, - 115,115,32,105,115,32,97,108,108,111,119,101,100,32,116,111, - 32,114,101,102,101,114,101,110,99,101,32,97,108,108,32,116, - 104,101,32,105,110,110,97,114,100,115,32,97,110,100,32,112, - 114,105,118,97,116,101,32,112,97,114,116,115,32,111,102,10, - 32,32,32,32,116,104,101,32,122,105,112,105,109,112,111,114, - 116,101,114,46,10,32,32,32,32,70,99,3,0,0,0,0, - 0,0,0,0,0,0,0,3,0,0,0,2,0,0,0,67, - 0,0,0,115,16,0,0,0,124,1,124,0,95,0,124,2, - 124,0,95,1,100,0,83,0,114,88,0,0,0,41,2,114, - 4,0,0,0,114,38,0,0,0,41,3,114,32,0,0,0, - 114,4,0,0,0,114,38,0,0,0,114,9,0,0,0,114, - 9,0,0,0,114,10,0,0,0,114,34,0,0,0,220,2, - 0,0,115,4,0,0,0,0,1,6,1,122,33,95,90,105, - 112,73,109,112,111,114,116,82,101,115,111,117,114,99,101,82, - 101,97,100,101,114,46,95,95,105,110,105,116,95,95,99,2, - 0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,8, - 0,0,0,67,0,0,0,115,92,0,0,0,124,0,106,0, - 160,1,100,1,100,2,161,2,125,2,124,2,155,0,100,2, - 124,1,155,0,157,3,125,3,100,3,100,4,108,2,109,3, - 125,4,1,0,122,18,124,4,124,0,106,4,160,5,124,3, - 161,1,131,1,87,0,83,0,4,0,116,6,107,10,114,86, - 1,0,1,0,1,0,116,7,124,3,131,1,130,1,89,0, - 110,2,88,0,100,0,83,0,41,5,78,114,85,0,0,0, - 114,109,0,0,0,114,0,0,0,0,41,1,218,7,66,121, - 116,101,115,73,79,41,8,114,38,0,0,0,114,19,0,0, - 0,90,2,105,111,114,177,0,0,0,114,4,0,0,0,114, - 55,0,0,0,114,22,0,0,0,218,17,70,105,108,101,78, - 111,116,70,111,117,110,100,69,114,114,111,114,41,5,114,32, - 0,0,0,218,8,114,101,115,111,117,114,99,101,218,16,102, - 117,108,108,110,97,109,101,95,97,115,95,112,97,116,104,114, - 13,0,0,0,114,177,0,0,0,114,9,0,0,0,114,9, - 0,0,0,114,10,0,0,0,218,13,111,112,101,110,95,114, - 101,115,111,117,114,99,101,224,2,0,0,115,14,0,0,0, - 0,1,14,1,14,1,12,1,2,1,18,1,14,1,122,38, - 95,90,105,112,73,109,112,111,114,116,82,101,115,111,117,114, - 99,101,82,101,97,100,101,114,46,111,112,101,110,95,114,101, - 115,111,117,114,99,101,99,2,0,0,0,0,0,0,0,0, - 0,0,0,2,0,0,0,1,0,0,0,67,0,0,0,115, - 8,0,0,0,116,0,130,1,100,0,83,0,114,88,0,0, - 0,41,1,114,178,0,0,0,41,2,114,32,0,0,0,114, - 179,0,0,0,114,9,0,0,0,114,9,0,0,0,114,10, - 0,0,0,218,13,114,101,115,111,117,114,99,101,95,112,97, - 116,104,233,2,0,0,115,2,0,0,0,0,4,122,38,95, - 90,105,112,73,109,112,111,114,116,82,101,115,111,117,114,99, - 101,82,101,97,100,101,114,46,114,101,115,111,117,114,99,101, - 95,112,97,116,104,99,2,0,0,0,0,0,0,0,0,0, - 0,0,4,0,0,0,8,0,0,0,67,0,0,0,115,72, - 0,0,0,124,0,106,0,160,1,100,1,100,2,161,2,125, - 2,124,2,155,0,100,2,124,1,155,0,157,3,125,3,122, - 16,124,0,106,2,160,3,124,3,161,1,1,0,87,0,110, - 22,4,0,116,4,107,10,114,66,1,0,1,0,1,0,89, - 0,100,3,83,0,88,0,100,4,83,0,41,5,78,114,85, - 0,0,0,114,109,0,0,0,70,84,41,5,114,38,0,0, - 0,114,19,0,0,0,114,4,0,0,0,114,55,0,0,0, - 114,22,0,0,0,41,4,114,32,0,0,0,114,59,0,0, - 0,114,180,0,0,0,114,13,0,0,0,114,9,0,0,0, - 114,9,0,0,0,114,10,0,0,0,218,11,105,115,95,114, - 101,115,111,117,114,99,101,239,2,0,0,115,14,0,0,0, - 0,3,14,1,14,1,2,1,16,1,14,1,8,1,122,36, - 95,90,105,112,73,109,112,111,114,116,82,101,115,111,117,114, - 99,101,82,101,97,100,101,114,46,105,115,95,114,101,115,111, - 117,114,99,101,99,1,0,0,0,0,0,0,0,0,0,0, - 0,9,0,0,0,9,0,0,0,99,0,0,0,115,186,0, - 0,0,100,1,100,2,108,0,109,1,125,1,1,0,124,1, - 124,0,106,2,160,3,124,0,106,4,161,1,131,1,125,2, - 124,2,160,5,124,0,106,2,106,6,161,1,125,3,124,3, - 106,7,100,3,107,2,115,58,116,8,130,1,124,3,106,9, - 125,4,116,10,131,0,125,5,124,0,106,2,106,11,68,0, - 93,102,125,6,122,18,124,1,124,6,131,1,160,5,124,4, - 161,1,125,7,87,0,110,24,4,0,116,12,107,10,114,124, - 1,0,1,0,1,0,89,0,113,78,89,0,110,2,88,0, - 124,7,106,9,106,7,125,8,116,13,124,8,131,1,100,1, - 107,2,114,156,124,7,106,7,86,0,1,0,113,78,124,8, - 124,5,107,7,114,78,124,5,160,14,124,8,161,1,1,0, - 124,8,86,0,1,0,113,78,100,0,83,0,41,4,78,114, - 0,0,0,0,41,1,218,4,80,97,116,104,114,60,0,0, - 0,41,15,90,7,112,97,116,104,108,105,98,114,184,0,0, - 0,114,4,0,0,0,114,56,0,0,0,114,38,0,0,0, - 90,11,114,101,108,97,116,105,118,101,95,116,111,114,29,0, - 0,0,114,59,0,0,0,114,175,0,0,0,90,6,112,97, - 114,101,110,116,218,3,115,101,116,114,28,0,0,0,114,23, - 0,0,0,114,51,0,0,0,218,3,97,100,100,41,9,114, - 32,0,0,0,114,184,0,0,0,90,13,102,117,108,108,110, - 97,109,101,95,112,97,116,104,90,13,114,101,108,97,116,105, - 118,101,95,112,97,116,104,90,12,112,97,99,107,97,103,101, - 95,112,97,116,104,90,12,115,117,98,100,105,114,115,95,115, - 101,101,110,218,8,102,105,108,101,110,97,109,101,90,8,114, - 101,108,97,116,105,118,101,90,11,112,97,114,101,110,116,95, - 110,97,109,101,114,9,0,0,0,114,9,0,0,0,114,10, - 0,0,0,218,8,99,111,110,116,101,110,116,115,250,2,0, - 0,115,34,0,0,0,0,8,12,1,18,1,14,3,14,1, - 6,1,6,1,12,1,2,1,18,1,14,1,10,5,8,1, - 12,1,10,1,8,1,10,1,122,33,95,90,105,112,73,109, - 112,111,114,116,82,101,115,111,117,114,99,101,82,101,97,100, - 101,114,46,99,111,110,116,101,110,116,115,78,41,10,114,6, - 0,0,0,114,7,0,0,0,114,8,0,0,0,114,84,0, - 0,0,114,81,0,0,0,114,34,0,0,0,114,181,0,0, - 0,114,182,0,0,0,114,183,0,0,0,114,188,0,0,0, - 114,9,0,0,0,114,9,0,0,0,114,9,0,0,0,114, - 10,0,0,0,114,80,0,0,0,212,2,0,0,115,14,0, - 0,0,8,1,4,5,4,2,8,4,8,9,8,6,8,11, - 114,80,0,0,0,41,45,114,84,0,0,0,90,26,95,102, - 114,111,122,101,110,95,105,109,112,111,114,116,108,105,98,95, - 101,120,116,101,114,110,97,108,114,21,0,0,0,114,1,0, - 0,0,114,2,0,0,0,90,17,95,102,114,111,122,101,110, - 95,105,109,112,111,114,116,108,105,98,114,76,0,0,0,114, - 148,0,0,0,114,110,0,0,0,114,152,0,0,0,114,67, - 0,0,0,114,131,0,0,0,90,7,95,95,97,108,108,95, - 95,114,20,0,0,0,90,15,112,97,116,104,95,115,101,112, - 97,114,97,116,111,114,115,114,18,0,0,0,114,75,0,0, - 0,114,3,0,0,0,114,25,0,0,0,218,4,116,121,112, - 101,114,70,0,0,0,114,113,0,0,0,114,115,0,0,0, - 114,117,0,0,0,114,4,0,0,0,114,89,0,0,0,114, - 36,0,0,0,114,37,0,0,0,114,35,0,0,0,114,27, - 0,0,0,114,122,0,0,0,114,142,0,0,0,114,144,0, - 0,0,114,52,0,0,0,114,147,0,0,0,114,155,0,0, - 0,218,8,95,95,99,111,100,101,95,95,114,153,0,0,0, - 114,159,0,0,0,114,161,0,0,0,114,169,0,0,0,114, - 151,0,0,0,114,149,0,0,0,114,44,0,0,0,114,80, - 0,0,0,114,9,0,0,0,114,9,0,0,0,114,9,0, - 0,0,114,10,0,0,0,218,8,60,109,111,100,117,108,101, - 62,1,0,0,0,115,88,0,0,0,4,16,8,1,16,1, - 8,1,8,1,8,1,8,1,8,1,8,2,8,3,6,1, - 14,3,16,4,4,2,8,2,4,1,4,1,4,2,14,127, - 0,127,0,1,12,1,12,1,2,1,2,252,4,9,8,4, - 8,9,8,31,8,126,2,254,2,29,4,5,8,21,8,46, - 8,10,8,46,10,5,8,7,8,6,8,13,8,19,8,15, - 8,26, + 95,118,97,108,105,100,97,116,101,95,104,97,115,104,95,112, + 121,99,218,29,95,103,101,116,95,109,116,105,109,101,95,97, + 110,100,95,115,105,122,101,95,111,102,95,115,111,117,114,99, + 101,114,147,0,0,0,114,2,0,0,0,114,76,0,0,0, + 114,77,0,0,0,218,7,109,97,114,115,104,97,108,90,5, + 108,111,97,100,115,114,15,0,0,0,218,10,95,99,111,100, + 101,95,116,121,112,101,218,9,84,121,112,101,69,114,114,111, + 114,41,14,114,32,0,0,0,114,53,0,0,0,114,63,0, + 0,0,114,38,0,0,0,114,126,0,0,0,90,11,101,120, + 99,95,100,101,116,97,105,108,115,114,129,0,0,0,90,10, + 104,97,115,104,95,98,97,115,101,100,90,12,99,104,101,99, + 107,95,115,111,117,114,99,101,90,12,115,111,117,114,99,101, + 95,98,121,116,101,115,114,150,0,0,0,90,12,115,111,117, + 114,99,101,95,109,116,105,109,101,90,11,115,111,117,114,99, + 101,95,115,105,122,101,114,46,0,0,0,114,9,0,0,0, + 114,9,0,0,0,114,10,0,0,0,218,15,95,117,110,109, + 97,114,115,104,97,108,95,99,111,100,101,75,2,0,0,115, + 88,0,0,0,0,2,2,1,2,254,6,5,2,1,18,1, + 14,1,8,2,12,1,4,1,12,1,10,1,2,255,2,1, + 8,255,2,2,10,1,8,1,4,1,4,1,2,254,4,5, + 2,1,4,1,2,0,2,0,2,0,2,255,8,2,14,1, + 10,3,8,255,6,3,6,3,22,1,18,255,4,2,4,1, + 8,255,4,2,4,2,18,1,12,1,16,1,114,155,0,0, + 0,99,1,0,0,0,0,0,0,0,0,0,0,0,1,0, + 0,0,4,0,0,0,67,0,0,0,115,28,0,0,0,124, + 0,160,0,100,1,100,2,161,2,125,0,124,0,160,0,100, + 3,100,2,161,2,125,0,124,0,83,0,41,4,78,115,2, + 0,0,0,13,10,243,1,0,0,0,10,243,1,0,0,0, + 13,41,1,114,19,0,0,0,41,1,218,6,115,111,117,114, + 99,101,114,9,0,0,0,114,9,0,0,0,114,10,0,0, + 0,218,23,95,110,111,114,109,97,108,105,122,101,95,108,105, + 110,101,95,101,110,100,105,110,103,115,126,2,0,0,115,6, + 0,0,0,0,1,12,1,12,1,114,159,0,0,0,99,2, + 0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,6, + 0,0,0,67,0,0,0,115,24,0,0,0,116,0,124,1, + 131,1,125,1,116,1,124,1,124,0,100,1,100,2,100,3, + 141,4,83,0,41,4,78,114,74,0,0,0,84,41,1,90, + 12,100,111,110,116,95,105,110,104,101,114,105,116,41,2,114, + 159,0,0,0,218,7,99,111,109,112,105,108,101,41,2,114, + 53,0,0,0,114,158,0,0,0,114,9,0,0,0,114,9, + 0,0,0,114,10,0,0,0,218,15,95,99,111,109,112,105, + 108,101,95,115,111,117,114,99,101,133,2,0,0,115,4,0, + 0,0,0,1,8,1,114,161,0,0,0,99,2,0,0,0, + 0,0,0,0,0,0,0,0,2,0,0,0,11,0,0,0, + 67,0,0,0,115,68,0,0,0,116,0,160,1,124,0,100, + 1,63,0,100,2,23,0,124,0,100,3,63,0,100,4,64, + 0,124,0,100,5,64,0,124,1,100,6,63,0,124,1,100, + 3,63,0,100,7,64,0,124,1,100,5,64,0,100,8,20, + 0,100,9,100,9,100,9,102,9,161,1,83,0,41,10,78, + 233,9,0,0,0,105,188,7,0,0,233,5,0,0,0,233, + 15,0,0,0,233,31,0,0,0,233,11,0,0,0,233,63, + 0,0,0,114,86,0,0,0,114,14,0,0,0,41,2,114, + 131,0,0,0,90,6,109,107,116,105,109,101,41,2,218,1, + 100,114,138,0,0,0,114,9,0,0,0,114,9,0,0,0, + 114,10,0,0,0,218,14,95,112,97,114,115,101,95,100,111, + 115,116,105,109,101,139,2,0,0,115,22,0,0,0,0,1, + 4,1,10,1,10,1,6,1,6,1,10,1,10,1,2,0, + 2,0,2,249,114,169,0,0,0,99,2,0,0,0,0,0, + 0,0,0,0,0,0,6,0,0,0,10,0,0,0,67,0, + 0,0,115,116,0,0,0,122,82,124,1,100,1,100,0,133, + 2,25,0,100,2,107,6,115,22,116,0,130,1,124,1,100, + 0,100,1,133,2,25,0,125,1,124,0,106,1,124,1,25, + 0,125,2,124,2,100,3,25,0,125,3,124,2,100,4,25, + 0,125,4,124,2,100,5,25,0,125,5,116,2,124,4,124, + 3,131,2,124,5,102,2,87,0,83,0,4,0,116,3,116, + 4,116,5,102,3,107,10,114,110,1,0,1,0,1,0,89, + 0,100,6,83,0,88,0,100,0,83,0,41,7,78,114,14, + 0,0,0,169,2,218,1,99,218,1,111,114,163,0,0,0, + 233,6,0,0,0,233,3,0,0,0,41,2,114,0,0,0, + 0,114,0,0,0,0,41,6,218,14,65,115,115,101,114,116, + 105,111,110,69,114,114,111,114,114,28,0,0,0,114,169,0, + 0,0,114,26,0,0,0,218,10,73,110,100,101,120,69,114, + 114,111,114,114,154,0,0,0,41,6,114,32,0,0,0,114, + 13,0,0,0,114,54,0,0,0,114,131,0,0,0,114,132, + 0,0,0,90,17,117,110,99,111,109,112,114,101,115,115,101, + 100,95,115,105,122,101,114,9,0,0,0,114,9,0,0,0, + 114,10,0,0,0,114,151,0,0,0,152,2,0,0,115,20, + 0,0,0,0,1,2,2,20,1,12,1,10,3,8,1,8, + 1,8,1,16,1,20,1,114,151,0,0,0,99,2,0,0, + 0,0,0,0,0,0,0,0,0,3,0,0,0,8,0,0, + 0,67,0,0,0,115,86,0,0,0,124,1,100,1,100,0, + 133,2,25,0,100,2,107,6,115,20,116,0,130,1,124,1, + 100,0,100,1,133,2,25,0,125,1,122,14,124,0,106,1, + 124,1,25,0,125,2,87,0,110,22,4,0,116,2,107,10, + 114,68,1,0,1,0,1,0,89,0,100,0,83,0,88,0, + 116,3,124,0,106,4,124,2,131,2,83,0,100,0,83,0, + 41,3,78,114,14,0,0,0,114,170,0,0,0,41,5,114, + 175,0,0,0,114,28,0,0,0,114,26,0,0,0,114,52, + 0,0,0,114,29,0,0,0,41,3,114,32,0,0,0,114, + 13,0,0,0,114,54,0,0,0,114,9,0,0,0,114,9, + 0,0,0,114,10,0,0,0,114,149,0,0,0,171,2,0, + 0,115,14,0,0,0,0,2,20,1,12,2,2,1,14,1, + 14,1,8,2,114,149,0,0,0,99,2,0,0,0,0,0, + 0,0,0,0,0,0,11,0,0,0,9,0,0,0,67,0, + 0,0,115,198,0,0,0,116,0,124,0,124,1,131,2,125, + 2,116,1,68,0,93,160,92,3,125,3,125,4,125,5,124, + 2,124,3,23,0,125,6,116,2,106,3,100,1,124,0,106, + 4,116,5,124,6,100,2,100,3,141,5,1,0,122,14,124, + 0,106,6,124,6,25,0,125,7,87,0,110,20,4,0,116, + 7,107,10,114,88,1,0,1,0,1,0,89,0,113,14,88, + 0,124,7,100,4,25,0,125,8,116,8,124,0,106,4,124, + 7,131,2,125,9,124,4,114,132,116,9,124,0,124,8,124, + 6,124,1,124,9,131,5,125,10,110,10,116,10,124,8,124, + 9,131,2,125,10,124,10,100,0,107,8,114,152,113,14,124, + 7,100,4,25,0,125,8,124,10,124,5,124,8,102,3,2, + 0,1,0,83,0,113,14,116,11,100,5,124,1,155,2,157, + 2,124,1,100,6,141,2,130,1,100,0,83,0,41,7,78, + 122,13,116,114,121,105,110,103,32,123,125,123,125,123,125,114, + 86,0,0,0,41,1,90,9,118,101,114,98,111,115,105,116, + 121,114,0,0,0,0,114,57,0,0,0,114,58,0,0,0, + 41,12,114,36,0,0,0,114,89,0,0,0,114,76,0,0, + 0,114,77,0,0,0,114,29,0,0,0,114,20,0,0,0, + 114,28,0,0,0,114,26,0,0,0,114,52,0,0,0,114, + 155,0,0,0,114,161,0,0,0,114,3,0,0,0,41,11, + 114,32,0,0,0,114,38,0,0,0,114,13,0,0,0,114, + 90,0,0,0,114,91,0,0,0,114,47,0,0,0,114,63, + 0,0,0,114,54,0,0,0,114,40,0,0,0,114,126,0, + 0,0,114,46,0,0,0,114,9,0,0,0,114,9,0,0, + 0,114,10,0,0,0,114,44,0,0,0,186,2,0,0,115, + 36,0,0,0,0,1,10,1,14,1,8,1,22,1,2,1, + 14,1,14,1,6,2,8,1,12,1,4,1,18,2,10,1, + 8,3,2,1,8,1,16,2,114,44,0,0,0,99,0,0, + 0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0, + 0,0,64,0,0,0,115,60,0,0,0,101,0,90,1,100, + 0,90,2,100,1,90,3,100,2,90,4,100,3,100,4,132, + 0,90,5,100,5,100,6,132,0,90,6,100,7,100,8,132, + 0,90,7,100,9,100,10,132,0,90,8,100,11,100,12,132, + 0,90,9,100,13,83,0,41,14,114,80,0,0,0,122,165, + 80,114,105,118,97,116,101,32,99,108,97,115,115,32,117,115, + 101,100,32,116,111,32,115,117,112,112,111,114,116,32,90,105, + 112,73,109,112,111,114,116,46,103,101,116,95,114,101,115,111, + 117,114,99,101,95,114,101,97,100,101,114,40,41,46,10,10, + 32,32,32,32,84,104,105,115,32,99,108,97,115,115,32,105, + 115,32,97,108,108,111,119,101,100,32,116,111,32,114,101,102, + 101,114,101,110,99,101,32,97,108,108,32,116,104,101,32,105, + 110,110,97,114,100,115,32,97,110,100,32,112,114,105,118,97, + 116,101,32,112,97,114,116,115,32,111,102,10,32,32,32,32, + 116,104,101,32,122,105,112,105,109,112,111,114,116,101,114,46, + 10,32,32,32,32,70,99,3,0,0,0,0,0,0,0,0, + 0,0,0,3,0,0,0,2,0,0,0,67,0,0,0,115, + 16,0,0,0,124,1,124,0,95,0,124,2,124,0,95,1, + 100,0,83,0,114,88,0,0,0,41,2,114,4,0,0,0, + 114,38,0,0,0,41,3,114,32,0,0,0,114,4,0,0, + 0,114,38,0,0,0,114,9,0,0,0,114,9,0,0,0, + 114,10,0,0,0,114,34,0,0,0,220,2,0,0,115,4, + 0,0,0,0,1,6,1,122,33,95,90,105,112,73,109,112, + 111,114,116,82,101,115,111,117,114,99,101,82,101,97,100,101, + 114,46,95,95,105,110,105,116,95,95,99,2,0,0,0,0, + 0,0,0,0,0,0,0,5,0,0,0,8,0,0,0,67, + 0,0,0,115,92,0,0,0,124,0,106,0,160,1,100,1, + 100,2,161,2,125,2,124,2,155,0,100,2,124,1,155,0, + 157,3,125,3,100,3,100,4,108,2,109,3,125,4,1,0, + 122,18,124,4,124,0,106,4,160,5,124,3,161,1,131,1, + 87,0,83,0,4,0,116,6,107,10,114,86,1,0,1,0, + 1,0,116,7,124,3,131,1,130,1,89,0,110,2,88,0, + 100,0,83,0,41,5,78,114,85,0,0,0,114,109,0,0, + 0,114,0,0,0,0,41,1,218,7,66,121,116,101,115,73, + 79,41,8,114,38,0,0,0,114,19,0,0,0,90,2,105, + 111,114,177,0,0,0,114,4,0,0,0,114,55,0,0,0, + 114,22,0,0,0,218,17,70,105,108,101,78,111,116,70,111, + 117,110,100,69,114,114,111,114,41,5,114,32,0,0,0,218, + 8,114,101,115,111,117,114,99,101,218,16,102,117,108,108,110, + 97,109,101,95,97,115,95,112,97,116,104,114,13,0,0,0, + 114,177,0,0,0,114,9,0,0,0,114,9,0,0,0,114, + 10,0,0,0,218,13,111,112,101,110,95,114,101,115,111,117, + 114,99,101,224,2,0,0,115,14,0,0,0,0,1,14,1, + 14,1,12,1,2,1,18,1,14,1,122,38,95,90,105,112, + 73,109,112,111,114,116,82,101,115,111,117,114,99,101,82,101, + 97,100,101,114,46,111,112,101,110,95,114,101,115,111,117,114, + 99,101,99,2,0,0,0,0,0,0,0,0,0,0,0,2, + 0,0,0,1,0,0,0,67,0,0,0,115,8,0,0,0, + 116,0,130,1,100,0,83,0,114,88,0,0,0,41,1,114, + 178,0,0,0,41,2,114,32,0,0,0,114,179,0,0,0, + 114,9,0,0,0,114,9,0,0,0,114,10,0,0,0,218, + 13,114,101,115,111,117,114,99,101,95,112,97,116,104,233,2, + 0,0,115,2,0,0,0,0,4,122,38,95,90,105,112,73, + 109,112,111,114,116,82,101,115,111,117,114,99,101,82,101,97, + 100,101,114,46,114,101,115,111,117,114,99,101,95,112,97,116, + 104,99,2,0,0,0,0,0,0,0,0,0,0,0,4,0, + 0,0,8,0,0,0,67,0,0,0,115,72,0,0,0,124, + 0,106,0,160,1,100,1,100,2,161,2,125,2,124,2,155, + 0,100,2,124,1,155,0,157,3,125,3,122,16,124,0,106, + 2,160,3,124,3,161,1,1,0,87,0,110,22,4,0,116, + 4,107,10,114,66,1,0,1,0,1,0,89,0,100,3,83, + 0,88,0,100,4,83,0,41,5,78,114,85,0,0,0,114, + 109,0,0,0,70,84,41,5,114,38,0,0,0,114,19,0, + 0,0,114,4,0,0,0,114,55,0,0,0,114,22,0,0, + 0,41,4,114,32,0,0,0,114,59,0,0,0,114,180,0, + 0,0,114,13,0,0,0,114,9,0,0,0,114,9,0,0, + 0,114,10,0,0,0,218,11,105,115,95,114,101,115,111,117, + 114,99,101,239,2,0,0,115,14,0,0,0,0,3,14,1, + 14,1,2,1,16,1,14,1,8,1,122,36,95,90,105,112, + 73,109,112,111,114,116,82,101,115,111,117,114,99,101,82,101, + 97,100,101,114,46,105,115,95,114,101,115,111,117,114,99,101, + 99,1,0,0,0,0,0,0,0,0,0,0,0,9,0,0, + 0,9,0,0,0,99,0,0,0,115,186,0,0,0,100,1, + 100,2,108,0,109,1,125,1,1,0,124,1,124,0,106,2, + 160,3,124,0,106,4,161,1,131,1,125,2,124,2,160,5, + 124,0,106,2,106,6,161,1,125,3,124,3,106,7,100,3, + 107,2,115,58,116,8,130,1,124,3,106,9,125,4,116,10, + 131,0,125,5,124,0,106,2,106,11,68,0,93,102,125,6, + 122,18,124,1,124,6,131,1,160,5,124,4,161,1,125,7, + 87,0,110,24,4,0,116,12,107,10,114,124,1,0,1,0, + 1,0,89,0,113,78,89,0,110,2,88,0,124,7,106,9, + 106,7,125,8,116,13,124,8,131,1,100,1,107,2,114,156, + 124,7,106,7,86,0,1,0,113,78,124,8,124,5,107,7, + 114,78,124,5,160,14,124,8,161,1,1,0,124,8,86,0, + 1,0,113,78,100,0,83,0,41,4,78,114,0,0,0,0, + 41,1,218,4,80,97,116,104,114,60,0,0,0,41,15,90, + 7,112,97,116,104,108,105,98,114,184,0,0,0,114,4,0, + 0,0,114,56,0,0,0,114,38,0,0,0,90,11,114,101, + 108,97,116,105,118,101,95,116,111,114,29,0,0,0,114,59, + 0,0,0,114,175,0,0,0,90,6,112,97,114,101,110,116, + 218,3,115,101,116,114,28,0,0,0,114,23,0,0,0,114, + 51,0,0,0,218,3,97,100,100,41,9,114,32,0,0,0, + 114,184,0,0,0,90,13,102,117,108,108,110,97,109,101,95, + 112,97,116,104,90,13,114,101,108,97,116,105,118,101,95,112, + 97,116,104,90,12,112,97,99,107,97,103,101,95,112,97,116, + 104,90,12,115,117,98,100,105,114,115,95,115,101,101,110,218, + 8,102,105,108,101,110,97,109,101,90,8,114,101,108,97,116, + 105,118,101,90,11,112,97,114,101,110,116,95,110,97,109,101, + 114,9,0,0,0,114,9,0,0,0,114,10,0,0,0,218, + 8,99,111,110,116,101,110,116,115,250,2,0,0,115,34,0, + 0,0,0,8,12,1,18,1,14,3,14,1,6,1,6,1, + 12,1,2,1,18,1,14,1,10,5,8,1,12,1,10,1, + 8,1,10,1,122,33,95,90,105,112,73,109,112,111,114,116, + 82,101,115,111,117,114,99,101,82,101,97,100,101,114,46,99, + 111,110,116,101,110,116,115,78,41,10,114,6,0,0,0,114, + 7,0,0,0,114,8,0,0,0,114,84,0,0,0,114,81, + 0,0,0,114,34,0,0,0,114,181,0,0,0,114,182,0, + 0,0,114,183,0,0,0,114,188,0,0,0,114,9,0,0, + 0,114,9,0,0,0,114,9,0,0,0,114,10,0,0,0, + 114,80,0,0,0,212,2,0,0,115,14,0,0,0,8,1, + 4,5,4,2,8,4,8,9,8,6,8,11,114,80,0,0, + 0,41,45,114,84,0,0,0,90,26,95,102,114,111,122,101, + 110,95,105,109,112,111,114,116,108,105,98,95,101,120,116,101, + 114,110,97,108,114,21,0,0,0,114,1,0,0,0,114,2, + 0,0,0,90,17,95,102,114,111,122,101,110,95,105,109,112, + 111,114,116,108,105,98,114,76,0,0,0,114,148,0,0,0, + 114,110,0,0,0,114,152,0,0,0,114,67,0,0,0,114, + 131,0,0,0,90,7,95,95,97,108,108,95,95,114,20,0, + 0,0,90,15,112,97,116,104,95,115,101,112,97,114,97,116, + 111,114,115,114,18,0,0,0,114,75,0,0,0,114,3,0, + 0,0,114,25,0,0,0,218,4,116,121,112,101,114,70,0, + 0,0,114,113,0,0,0,114,115,0,0,0,114,117,0,0, + 0,114,4,0,0,0,114,89,0,0,0,114,36,0,0,0, + 114,37,0,0,0,114,35,0,0,0,114,27,0,0,0,114, + 122,0,0,0,114,142,0,0,0,114,144,0,0,0,114,52, + 0,0,0,114,147,0,0,0,114,155,0,0,0,218,8,95, + 95,99,111,100,101,95,95,114,153,0,0,0,114,159,0,0, + 0,114,161,0,0,0,114,169,0,0,0,114,151,0,0,0, + 114,149,0,0,0,114,44,0,0,0,114,80,0,0,0,114, + 9,0,0,0,114,9,0,0,0,114,9,0,0,0,114,10, + 0,0,0,218,8,60,109,111,100,117,108,101,62,1,0,0, + 0,115,88,0,0,0,4,16,8,1,16,1,8,1,8,1, + 8,1,8,1,8,1,8,2,8,3,6,1,14,3,16,4, + 4,2,8,2,4,1,4,1,4,2,14,127,0,127,0,1, + 12,1,12,1,2,1,2,252,4,9,8,4,8,9,8,31, + 8,126,2,254,2,29,4,5,8,21,8,46,8,10,8,46, + 10,5,8,7,8,6,8,13,8,19,8,15,8,26, }; From 9f220e4968cf73fa60440120ee46881e7974e47d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 14 Jan 2020 12:13:59 -0800 Subject: [PATCH 1097/2163] Fix documentation in code.py (GH-17988) (cherry picked from commit b4cdb3f60e71888d7f3d4e0d40cb31e968ea160c) Co-authored-by: Kyle Pollina --- Doc/library/code.rst | 2 +- Lib/code.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/code.rst b/Doc/library/code.rst index e2c47bab5a0b91..6708079f778c1a 100644 --- a/Doc/library/code.rst +++ b/Doc/library/code.rst @@ -76,7 +76,7 @@ Interactive Interpreter Objects Compile and run some source in the interpreter. Arguments are the same as for :func:`compile_command`; the default for *filename* is ``''``, and for - *symbol* is ``'single'``. One several things can happen: + *symbol* is ``'single'``. One of several things can happen: * The input is incorrect; :func:`compile_command` raised an exception (:exc:`SyntaxError` or :exc:`OverflowError`). A syntax traceback will be diff --git a/Lib/code.py b/Lib/code.py index d8106ae612c4b4..76000f8c8b2c1e 100644 --- a/Lib/code.py +++ b/Lib/code.py @@ -40,7 +40,7 @@ def runsource(self, source, filename="", symbol="single"): Arguments are as for compile_command(). - One several things can happen: + One of several things can happen: 1) The input is incorrect; compile_command() raised an exception (SyntaxError or OverflowError). A syntax traceback From 92135775ce12fd71f0878c0a8a0652c197fcc60b Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 15 Jan 2020 09:07:09 -0800 Subject: [PATCH 1098/2163] Fix compiler warning on Windows (GH-18012) Python-ast.h contains a macro named Yield that conflicts with the Yield macro in Windows system headers. While Python-ast.h has an "undef Yield" directive to prevent this, it means that Python-ast.h must be included before Windows header files or we run into a re-declaration warning. In commit c96be811fa7d an include for pycore_pystate.h was added which indirectly includes Windows header files. In this commit we re-order the includes to fix this warning. (cherry picked from commit e92d39303feb1d3b4194c6a8275b1fc63b2153b2) Co-authored-by: Ammar Askar --- Python/compile.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Python/compile.c b/Python/compile.c index f09e99fd4a43d3..11974c3509a32c 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -23,8 +23,8 @@ #include "Python.h" -#include "Python-ast.h" #include "pycore_pystate.h" /* _PyInterpreterState_GET_UNSAFE() */ +#include "Python-ast.h" #include "ast.h" #include "code.h" #include "symtable.h" From 54abd28b2c4fab37f38e9d6c5e4120c9428a759e Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 15 Jan 2020 12:19:21 -0800 Subject: [PATCH 1099/2163] [3.8] Fix typo in multiprocessing.pool.AsyncResult.successful doc. (GH-17932) (GH-18015) Since 3.7 `successful` raises a `ValueError` as explained in the next text block from the documentation: _Changed in version 3.7: If the result is not ready, ValueError is raised instead of AssertionError._ No issue associated with this PR. Should be backported in 3.7 and 3.8. (cherry picked from commit dc0284ee8f7a270b6005467f26d8e5773d76e959) Co-authored-by: Antoine <43954001+awecx@users.noreply.github.com> Automerge-Triggered-By: @pitrou --- Doc/library/multiprocessing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 3c7b5cc1262097..492f94c30017f0 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -2279,7 +2279,7 @@ with the :class:`Pool` class. .. method:: successful() Return whether the call completed without raising an exception. Will - raise :exc:`AssertionError` if the result is not ready. + raise :exc:`ValueError` if the result is not ready. .. versionchanged:: 3.7 If the result is not ready, :exc:`ValueError` is raised instead of From 19bd3a18626b504dc92abb44019cad600c7ef221 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 15 Jan 2020 16:01:21 -0800 Subject: [PATCH 1100/2163] bpo-39348: Fix code highlight for the SOCK_NONBLOCK example (GH-18018) The previous double colon was wrongly place directly after Therefore. Which produced a block without syntax highlighting. This fixes it by separating the double colon from the text. As a result, sphinx now properly highlights the python code. https://bugs.python.org/issue39348 (cherry picked from commit fad8b5674c66d9e00bb788e30adddb0c256c787b) Co-authored-by: Oz N Tiram --- Doc/library/socket.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index ae82e43e00ff08..c72f22b5f6699e 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -548,7 +548,9 @@ The following functions all create :ref:`socket objects `. When :const:`SOCK_NONBLOCK` or :const:`SOCK_CLOEXEC` bit flags are applied to *type* they are cleared, and :attr:`socket.type` will not reflect them. They are still passed - to the underlying system `socket()` call. Therefore:: + to the underlying system `socket()` call. Therefore, + + :: sock = socket.socket( socket.AF_INET, From 34ece35a0d52f0c332b2c0d55b567daa7bcad9eb Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 17 Jan 2020 15:02:06 -0800 Subject: [PATCH 1101/2163] Fix Lock.locked() to remove extra bold highlighting (GH-18042) (#18043) (cherry picked from commit ef8844f1bcbea994a2a69b5a70309369d08b555c) Co-authored-by: Grant Jenks Co-authored-by: Grant Jenks --- Doc/library/threading.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index 93ea4bda7cd7aa..f4b58d3d3aebb1 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -489,6 +489,7 @@ All methods are executed atomically. There is no return value. .. method:: locked() + Return true if the lock is acquired. From c7b16f858844546f02f7a3521c982038b44b3728 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 19 Jan 2020 02:48:04 -0800 Subject: [PATCH 1102/2163] Fix typo from base to based (GH-18055) (cherry picked from commit 558f07891170fe5173f277d3749e92d844de0a27) Co-authored-by: Michael Haas --- Lib/webbrowser.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Lib/webbrowser.py b/Lib/webbrowser.py index 0af36c4301d798..1ef179a91a6f19 100755 --- a/Lib/webbrowser.py +++ b/Lib/webbrowser.py @@ -87,7 +87,7 @@ def open_new_tab(url): def _synthesize(browser, *, preferred=False): - """Attempt to synthesize a controller base on existing controllers. + """Attempt to synthesize a controller based on existing controllers. This is useful to create a controller when a user specifies a path to an entry in the BROWSER environment variable -- we can copy a general From 23793edf0d61aa98478541d0ff8f2b900ff1813d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 19 Jan 2020 14:44:35 -0800 Subject: [PATCH 1103/2163] bpo-35561: Supress valgrind false alarm on epoll_ctl(event) (GH-18060) Update Misc/valgrind-python.supp to suppress the false alarm. (cherry picked from commit d8ef64422a75f40cecdb1a7ee43492607d3daaf6) Co-authored-by: Zackery Spytz --- Misc/valgrind-python.supp | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Misc/valgrind-python.supp b/Misc/valgrind-python.supp index 38a5ea3cd2b828..c9c45ba7ed6de4 100644 --- a/Misc/valgrind-python.supp +++ b/Misc/valgrind-python.supp @@ -263,6 +263,14 @@ } +{ + Uninitialised byte(s) false alarm, see bpo-35561 + Memcheck:Param + epoll_ctl(event) + fun:epoll_ctl + fun:pyepoll_internal_ctl +} + { ZLIB problems, see test_gzip Memcheck:Cond From 4cdb75890abd4ee7694744d5c24248f6735b0534 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 19 Jan 2020 15:43:37 -0800 Subject: [PATCH 1104/2163] bpo-38400 Don't check for NULL linked list pointers in _PyObject_IsFreed (GH-16630) Some objects like Py_None are not initialized with conventional means that prepare the circular linked list pointers, leaving them unlinked from the rest of the objects. For those objects, NULL pointers does not mean that they are freed, so we need to skip the check in those cases. (cherry picked from commit 36e33c360ed7716a2b5ab2b53210da81f8ce1295) Co-authored-by: Pablo Galindo --- Objects/object.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Objects/object.c b/Objects/object.c index 566593a9c67cc3..74b1b15d30bbf5 100644 --- a/Objects/object.c +++ b/Objects/object.c @@ -454,9 +454,12 @@ _PyObject_IsFreed(PyObject *op) /* ignore op->ob_ref: its value can have be modified by Py_INCREF() and Py_DECREF(). */ #ifdef Py_TRACE_REFS - if (_PyMem_IsPtrFreed(op->_ob_next) || _PyMem_IsPtrFreed(op->_ob_prev)) { + if (op->_ob_next != NULL && _PyMem_IsPtrFreed(op->_ob_next)) { return 1; } + if (op->_ob_prev != NULL && _PyMem_IsPtrFreed(op->_ob_prev)) { + return 1; + } #endif return 0; } From 6aeed01901d020363e383821b38614816d0b4032 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 20 Jan 2020 14:52:35 -0800 Subject: [PATCH 1105/2163] Fix asyncio.get_event_loop() documentation (GH-18051) Mention that the function implicitly creates new event loop only if called from the main thread. (cherry picked from commit 2c49becc69c05934996a00b902e4a4f089b91954) Co-authored-by: Andrew Svetlov --- Doc/library/asyncio-eventloop.rst | 6 ++++-- .../Documentation/2020-01-18-15-37-56.bpo-39381.wTWe8d.rst | 2 ++ 2 files changed, 6 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Documentation/2020-01-18-15-37-56.bpo-39381.wTWe8d.rst diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 7fed2c23df7d5d..24f621e2d9fffa 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -38,8 +38,10 @@ an event loop: .. function:: get_event_loop() - Get the current event loop. If there is no current event loop set - in the current OS thread and :func:`set_event_loop` has not yet + Get the current event loop. + + If there is no current event loop set in the current OS thread, + the OS thread is main, and :func:`set_event_loop` has not yet been called, asyncio will create a new event loop and set it as the current one. diff --git a/Misc/NEWS.d/next/Documentation/2020-01-18-15-37-56.bpo-39381.wTWe8d.rst b/Misc/NEWS.d/next/Documentation/2020-01-18-15-37-56.bpo-39381.wTWe8d.rst new file mode 100644 index 00000000000000..37b66ad9dfd178 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-01-18-15-37-56.bpo-39381.wTWe8d.rst @@ -0,0 +1,2 @@ +Mention in docs that :func:`asyncio.get_event_loop` implicitly creates new +event loop only if called from the main thread. From 5cadd3fe3aead1b5bee1438dc03383d6739d4209 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 20 Jan 2020 15:06:40 -0800 Subject: [PATCH 1106/2163] bpo-39386: Prevent double awaiting of async iterator (GH-18081) (cherry picked from commit a96e06db77dcbd3433d39761ddb4615d7d96284a) Co-authored-by: Andrew Svetlov --- Lib/test/test_asyncgen.py | 36 +++++++++++++++++++ .../2020-01-20-21-40-57.bpo-39386.ULqD8t.rst | 1 + Objects/genobject.c | 16 ++++++--- 3 files changed, 49 insertions(+), 4 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-20-21-40-57.bpo-39386.ULqD8t.rst diff --git a/Lib/test/test_asyncgen.py b/Lib/test/test_asyncgen.py index 58d8aee19adac6..24b20bec2b2d1e 100644 --- a/Lib/test/test_asyncgen.py +++ b/Lib/test/test_asyncgen.py @@ -1128,6 +1128,42 @@ async def main(): self.assertEqual([], messages) + def test_async_gen_await_anext_twice(self): + async def async_iterate(): + yield 1 + yield 2 + + async def run(): + it = async_iterate() + nxt = it.__anext__() + await nxt + with self.assertRaisesRegex( + RuntimeError, + r"cannot reuse already awaited __anext__\(\)/asend\(\)" + ): + await nxt + + await it.aclose() # prevent unfinished iterator warning + + self.loop.run_until_complete(run()) + + def test_async_gen_await_aclose_twice(self): + async def async_iterate(): + yield 1 + yield 2 + + async def run(): + it = async_iterate() + nxt = it.aclose() + await nxt + with self.assertRaisesRegex( + RuntimeError, + r"cannot reuse already awaited aclose\(\)/athrow\(\)" + ): + await nxt + + self.loop.run_until_complete(run()) + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-20-21-40-57.bpo-39386.ULqD8t.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-20-21-40-57.bpo-39386.ULqD8t.rst new file mode 100644 index 00000000000000..f24e1f4e8a1831 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-01-20-21-40-57.bpo-39386.ULqD8t.rst @@ -0,0 +1 @@ +Prevent double awaiting of async iterator. diff --git a/Objects/genobject.c b/Objects/genobject.c index 5643553b708d15..72aa872c6b59f5 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -1533,7 +1533,9 @@ async_gen_asend_send(PyAsyncGenASend *o, PyObject *arg) PyObject *result; if (o->ags_state == AWAITABLE_STATE_CLOSED) { - PyErr_SetNone(PyExc_StopIteration); + PyErr_SetString( + PyExc_RuntimeError, + "cannot reuse already awaited __anext__()/asend()"); return NULL; } @@ -1576,7 +1578,9 @@ async_gen_asend_throw(PyAsyncGenASend *o, PyObject *args) PyObject *result; if (o->ags_state == AWAITABLE_STATE_CLOSED) { - PyErr_SetNone(PyExc_StopIteration); + PyErr_SetString( + PyExc_RuntimeError, + "cannot reuse already awaited __anext__()/asend()"); return NULL; } @@ -1810,7 +1814,9 @@ async_gen_athrow_send(PyAsyncGenAThrow *o, PyObject *arg) if (f == NULL || f->f_stacktop == NULL || o->agt_state == AWAITABLE_STATE_CLOSED) { - PyErr_SetNone(PyExc_StopIteration); + PyErr_SetString( + PyExc_RuntimeError, + "cannot reuse already awaited aclose()/athrow()"); return NULL; } @@ -1932,7 +1938,9 @@ async_gen_athrow_throw(PyAsyncGenAThrow *o, PyObject *args) PyObject *retval; if (o->agt_state == AWAITABLE_STATE_CLOSED) { - PyErr_SetNone(PyExc_StopIteration); + PyErr_SetString( + PyExc_RuntimeError, + "cannot reuse already awaited aclose()/athrow()"); return NULL; } From c1964e09421e7ab61179eb4e2691e858a50d70e5 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 20 Jan 2020 15:28:43 -0800 Subject: [PATCH 1107/2163] bpo-39383: Mention Darwin as a potential value for platform.system() (GH-18054) (cherry picked from commit 8d57a4182f0aa68e16d66dea31ba59e732612b4f) Co-authored-by: Peter Bittner --- Doc/library/platform.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index 1d33afc75870a5..8e8e3775aaff4a 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -145,8 +145,8 @@ Cross Platform .. function:: system() - Returns the system/OS name, e.g. ``'Linux'``, ``'Windows'``, or ``'Java'``. An - empty string is returned if the value cannot be determined. + Returns the system/OS name, such as ``'Linux'``, ``'Darwin'``, ``'Java'``, + ``'Windows'``. An empty string is returned if the value cannot be determined. .. function:: system_alias(system, release, version) @@ -260,4 +260,3 @@ Unix Platforms using :program:`gcc`. The file is read and scanned in chunks of *chunksize* bytes. - From 070e68a59d4a3e36085957eea83b79181bd1e0ba Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 20 Jan 2020 17:12:07 -0800 Subject: [PATCH 1108/2163] improve the documentation of the LOAD_METHOD and CALL_METHOD (GH-18079) (cherry picked from commit 8698b34b68065b80bd9bd18b8decb425208fa386) Co-authored-by: Carl Friedrich Bolz-Tereick --- Doc/library/dis.rst | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 39a3e130afd3e0..b1404af9c00eba 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -1176,22 +1176,24 @@ All of the following opcodes use their arguments. .. opcode:: LOAD_METHOD (namei) - Loads a method named ``co_names[namei]`` from TOS object. TOS is popped and - method and TOS are pushed when interpreter can call unbound method directly. - TOS will be used as the first argument (``self``) by :opcode:`CALL_METHOD`. - Otherwise, ``NULL`` and method is pushed (method is bound method or - something else). + Loads a method named ``co_names[namei]`` from the TOS object. TOS is popped. + This bytecode distinguishes two cases: if TOS has a method with the correct + name, the bytecode pushes the unbound method and TOS. TOS will be used as + the first argument (``self``) by :opcode:`CALL_METHOD` when calling the + unbound method. Otherwise, ``NULL`` and the object return by the attribute + lookup are pushed. .. versionadded:: 3.7 .. opcode:: CALL_METHOD (argc) - Calls a method. *argc* is number of positional arguments. + Calls a method. *argc* is the number of positional arguments. Keyword arguments are not supported. This opcode is designed to be used with :opcode:`LOAD_METHOD`. Positional arguments are on top of the stack. - Below them, two items described in :opcode:`LOAD_METHOD` on the stack. - All of them are popped and return value is pushed. + Below them, the two items described in :opcode:`LOAD_METHOD` are on the + stack (either ``self`` and an unbound method object or ``NULL`` and an + arbitrary callable). All of them are popped and the return value is pushed. .. versionadded:: 3.7 From 060ad2fc1535adc76f96be8269b4af0f14429161 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 21 Jan 2020 02:29:39 -0800 Subject: [PATCH 1109/2163] bpo-32989: IDLE - fix bad editor call of pyparse method (GH-5968) Fix comments and add tests for editor newline_and_indent_event method. Remove unused None default for function parameter of pyparse find_good_parse_start method and code triggered by that default. Co-authored-by: Terry Jan Reedy (cherry picked from commit ec64640a2c5236d7a5d5470d759172a3d93eab0b) Co-authored-by: Cheryl Sabella --- Lib/idlelib/NEWS.txt | 3 + Lib/idlelib/editor.py | 55 +++++++---- Lib/idlelib/idle_test/test_editor.py | 99 +++++++++++++++++++ Lib/idlelib/idle_test/test_pyparse.py | 27 ++--- Lib/idlelib/pyparse.py | 7 +- .../2018-03-03-12-56-26.bpo-32989.FVhmhH.rst | 2 + 6 files changed, 154 insertions(+), 39 deletions(-) create mode 100644 Misc/NEWS.d/next/IDLE/2018-03-03-12-56-26.bpo-32989.FVhmhH.rst diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 0baec813b044d1..6e30ef35081f21 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,9 @@ Released on 2019-12-16? ====================================== +bpo-32989: Add tests for editor newline_and_indent_event method. +Remove dead code from pyparse find_good_parse_start method. + bpo-38943: Fix autocomplete windows not always appearing on some systems. Patch by Johnny Najera. diff --git a/Lib/idlelib/editor.py b/Lib/idlelib/editor.py index 92dcf57c4ff260..c9f1a1625ca5ea 100644 --- a/Lib/idlelib/editor.py +++ b/Lib/idlelib/editor.py @@ -1342,38 +1342,51 @@ def smart_indent_event(self, event): text.undo_block_stop() def newline_and_indent_event(self, event): + """Insert a newline and indentation after Enter keypress event. + + Properly position the cursor on the new line based on information + from the current line. This takes into account if the current line + is a shell prompt, is empty, has selected text, contains a block + opener, contains a block closer, is a continuation line, or + is inside a string. + """ text = self.text first, last = self.get_selection_indices() text.undo_block_start() - try: + try: # Close undo block and expose new line in finally clause. if first and last: text.delete(first, last) text.mark_set("insert", first) line = text.get("insert linestart", "insert") + + # Count leading whitespace for indent size. i, n = 0, len(line) while i < n and line[i] in " \t": - i = i+1 + i += 1 if i == n: - # the cursor is in or at leading indentation in a continuation - # line; just inject an empty line at the start + # The cursor is in or at leading indentation in a continuation + # line; just inject an empty line at the start. text.insert("insert linestart", '\n') return "break" indent = line[:i] - # strip whitespace before insert point unless it's in the prompt + + # Strip whitespace before insert point unless it's in the prompt. i = 0 while line and line[-1] in " \t" and line != self.prompt_last_line: line = line[:-1] - i = i+1 + i += 1 if i: text.delete("insert - %d chars" % i, "insert") - # strip whitespace after insert point + + # Strip whitespace after insert point. while text.get("insert") in " \t": text.delete("insert") - # start new line + + # Insert new line. text.insert("insert", '\n') - # adjust indentation for continuations and block - # open/close first need to find the last stmt + # Adjust indentation for continuations and block open/close. + # First need to find the last statement. lno = index2line(text.index('insert')) y = pyparse.Parser(self.indentwidth, self.tabwidth) if not self.prompt_last_line: @@ -1383,7 +1396,7 @@ def newline_and_indent_event(self, event): rawtext = text.get(startatindex, "insert") y.set_code(rawtext) bod = y.find_good_parse_start( - self._build_char_in_string_func(startatindex)) + self._build_char_in_string_func(startatindex)) if bod is not None or startat == 1: break y.set_lo(bod or 0) @@ -1399,26 +1412,26 @@ def newline_and_indent_event(self, event): c = y.get_continuation_type() if c != pyparse.C_NONE: - # The current stmt hasn't ended yet. + # The current statement hasn't ended yet. if c == pyparse.C_STRING_FIRST_LINE: - # after the first line of a string; do not indent at all + # After the first line of a string do not indent at all. pass elif c == pyparse.C_STRING_NEXT_LINES: - # inside a string which started before this line; - # just mimic the current indent + # Inside a string which started before this line; + # just mimic the current indent. text.insert("insert", indent) elif c == pyparse.C_BRACKET: - # line up with the first (if any) element of the + # Line up with the first (if any) element of the # last open bracket structure; else indent one # level beyond the indent of the line with the - # last open bracket + # last open bracket. self.reindent_to(y.compute_bracket_indent()) elif c == pyparse.C_BACKSLASH: - # if more than one line in this stmt already, just + # If more than one line in this statement already, just # mimic the current indent; else if initial line # has a start on an assignment stmt, indent to # beyond leftmost =; else to beyond first chunk of - # non-whitespace on initial line + # non-whitespace on initial line. if y.get_num_lines_in_stmt() > 1: text.insert("insert", indent) else: @@ -1427,9 +1440,9 @@ def newline_and_indent_event(self, event): assert 0, "bogus continuation type %r" % (c,) return "break" - # This line starts a brand new stmt; indent relative to + # This line starts a brand new statement; indent relative to # indentation of initial line of closest preceding - # interesting stmt. + # interesting statement. indent = y.get_base_indent_string() text.insert("insert", indent) if y.is_block_opener(): diff --git a/Lib/idlelib/idle_test/test_editor.py b/Lib/idlelib/idle_test/test_editor.py index 240db71747a284..91e8ef89d1d72a 100644 --- a/Lib/idlelib/idle_test/test_editor.py +++ b/Lib/idlelib/idle_test/test_editor.py @@ -2,6 +2,7 @@ from idlelib import editor import unittest +from collections import namedtuple from test.support import requires from tkinter import Tk @@ -91,5 +92,103 @@ def test_tabwidth_8(self): ) +class IndentAndNewlineTest(unittest.TestCase): + + @classmethod + def setUpClass(cls): + requires('gui') + cls.root = Tk() + cls.root.withdraw() + cls.window = Editor(root=cls.root) + cls.window.indentwidth = 2 + cls.window.tabwidth = 2 + + @classmethod + def tearDownClass(cls): + cls.window._close() + del cls.window + cls.root.update_idletasks() + for id in cls.root.tk.call('after', 'info'): + cls.root.after_cancel(id) + cls.root.destroy() + del cls.root + + def insert(self, text): + t = self.window.text + t.delete('1.0', 'end') + t.insert('end', text) + # Force update for colorizer to finish. + t.update() + + def test_indent_and_newline_event(self): + eq = self.assertEqual + w = self.window + text = w.text + get = text.get + nl = w.newline_and_indent_event + + TestInfo = namedtuple('Tests', ['label', 'text', 'expected', 'mark']) + + tests = (TestInfo('Empty line inserts with no indent.', + ' \n def __init__(self):', + '\n \n def __init__(self):\n', + '1.end'), + TestInfo('Inside bracket before space, deletes space.', + ' def f1(self, a, b):', + ' def f1(self,\n a, b):\n', + '1.14'), + TestInfo('Inside bracket after space, deletes space.', + ' def f1(self, a, b):', + ' def f1(self,\n a, b):\n', + '1.15'), + TestInfo('Inside string with one line - no indent.', + ' """Docstring."""', + ' """Docstring.\n"""\n', + '1.15'), + TestInfo('Inside string with more than one line.', + ' """Docstring.\n Docstring Line 2"""', + ' """Docstring.\n Docstring Line 2\n """\n', + '2.18'), + TestInfo('Backslash with one line.', + 'a =\\', + 'a =\\\n \n', + '1.end'), + TestInfo('Backslash with more than one line.', + 'a =\\\n multiline\\', + 'a =\\\n multiline\\\n \n', + '2.end'), + TestInfo('Block opener - indents +1 level.', + ' def f1(self):\n pass', + ' def f1(self):\n \n pass\n', + '1.end'), + TestInfo('Block closer - dedents -1 level.', + ' def f1(self):\n pass', + ' def f1(self):\n pass\n \n', + '2.end'), + ) + + w.prompt_last_line = '' + for test in tests: + with self.subTest(label=test.label): + self.insert(test.text) + text.mark_set('insert', test.mark) + nl(event=None) + eq(get('1.0', 'end'), test.expected) + + # Selected text. + self.insert(' def f1(self, a, b):\n return a + b') + text.tag_add('sel', '1.17', '1.end') + nl(None) + # Deletes selected text before adding new line. + eq(get('1.0', 'end'), ' def f1(self, a,\n \n return a + b\n') + + # Preserves the whitespace in shell prompt. + w.prompt_last_line = '>>> ' + self.insert('>>> \t\ta =') + text.mark_set('insert', '1.5') + nl(None) + eq(get('1.0', 'end'), '>>> \na =\n') + + if __name__ == '__main__': unittest.main(verbosity=2) diff --git a/Lib/idlelib/idle_test/test_pyparse.py b/Lib/idlelib/idle_test/test_pyparse.py index f7154e6ded9574..a2b13c38d80d5f 100644 --- a/Lib/idlelib/idle_test/test_pyparse.py +++ b/Lib/idlelib/idle_test/test_pyparse.py @@ -18,7 +18,7 @@ def test_trans(self): # trans is the production instance of ParseMap, used in _study1 parser = pyparse.Parser(4, 4) self.assertEqual('\t a([{b}])b"c\'d\n'.translate(pyparse.trans), - 'xxx(((x)))x"x\'x\n') + 'xxx(((x)))x"x\'x\n') class PyParseTest(unittest.TestCase): @@ -61,14 +61,17 @@ def test_find_good_parse_start(self): # Split def across lines. setcode('"""This is a module docstring"""\n' - 'class C():\n' - ' def __init__(self, a,\n' - ' b=True):\n' - ' pass\n' - ) + 'class C():\n' + ' def __init__(self, a,\n' + ' b=True):\n' + ' pass\n' + ) - # No value sent for is_char_in_string(). - self.assertIsNone(start()) + # Passing no value or non-callable should fail (issue 32989). + with self.assertRaises(TypeError): + start() + with self.assertRaises(TypeError): + start(False) # Make text look like a string. This returns pos as the start # position, but it's set to None. @@ -91,10 +94,10 @@ def test_find_good_parse_start(self): # Code without extra line break in def line - mostly returns the same # values. setcode('"""This is a module docstring"""\n' - 'class C():\n' - ' def __init__(self, a, b=True):\n' - ' pass\n' - ) + 'class C():\n' + ' def __init__(self, a, b=True):\n' + ' pass\n' + ) eq(start(is_char_in_string=lambda index: False), 44) eq(start(is_char_in_string=lambda index: index > 44), 44) eq(start(is_char_in_string=lambda index: index >= 44), 33) diff --git a/Lib/idlelib/pyparse.py b/Lib/idlelib/pyparse.py index feb57cbb740564..9fa20108960711 100644 --- a/Lib/idlelib/pyparse.py +++ b/Lib/idlelib/pyparse.py @@ -133,8 +133,7 @@ def set_code(self, s): self.code = s self.study_level = 0 - def find_good_parse_start(self, is_char_in_string=None, - _synchre=_synchre): + def find_good_parse_start(self, is_char_in_string, _synchre=_synchre): """ Return index of a good place to begin parsing, as close to the end of the string as possible. This will be the start of some @@ -149,10 +148,6 @@ def find_good_parse_start(self, is_char_in_string=None, """ code, pos = self.code, None - if not is_char_in_string: - # no clue -- make the caller pass everything - return None - # Peek back from the end for a good place to start, # but don't try too often; pos will be left None, or # bumped to a legitimate synch point. diff --git a/Misc/NEWS.d/next/IDLE/2018-03-03-12-56-26.bpo-32989.FVhmhH.rst b/Misc/NEWS.d/next/IDLE/2018-03-03-12-56-26.bpo-32989.FVhmhH.rst new file mode 100644 index 00000000000000..38f0fb6e104527 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2018-03-03-12-56-26.bpo-32989.FVhmhH.rst @@ -0,0 +1,2 @@ +Add tests for editor newline_and_indent_event method. +Remove dead code from pyparse find_good_parse_start method. From ab0d8e356ecd351d55f89519a6a97a1e69c0dfab Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 21 Jan 2020 03:42:49 -0800 Subject: [PATCH 1110/2163] bpo-39389: gzip: fix compression level metadata (GH-18077) As described in RFC 1952, section 2.3.1, the XFL (eXtra FLags) byte of a gzip member header should indicate whether the DEFLATE algorithm was tuned for speed or compression ratio. Prior to this patch, archives emitted by the `gzip` module always indicated maximum compression. (cherry picked from commit eab3b3f1c60afecfb4db3c3619109684cb04bd60) Co-authored-by: William Chargin --- Lib/gzip.py | 12 ++++++++--- Lib/test/test_gzip.py | 20 +++++++++++++++++++ .../2020-01-20-00-56-01.bpo-39389.fEirIS.rst | 2 ++ 3 files changed, 31 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-01-20-00-56-01.bpo-39389.fEirIS.rst diff --git a/Lib/gzip.py b/Lib/gzip.py index 2968f475efad31..87b553df66cd85 100644 --- a/Lib/gzip.py +++ b/Lib/gzip.py @@ -201,7 +201,7 @@ def __init__(self, filename=None, mode=None, self.fileobj = fileobj if self.mode == WRITE: - self._write_gzip_header() + self._write_gzip_header(compresslevel) @property def filename(self): @@ -228,7 +228,7 @@ def _init_write(self, filename): self.bufsize = 0 self.offset = 0 # Current file offset for seek(), tell(), etc - def _write_gzip_header(self): + def _write_gzip_header(self, compresslevel): self.fileobj.write(b'\037\213') # magic header self.fileobj.write(b'\010') # compression method try: @@ -249,7 +249,13 @@ def _write_gzip_header(self): if mtime is None: mtime = time.time() write32u(self.fileobj, int(mtime)) - self.fileobj.write(b'\002') + if compresslevel == _COMPRESS_LEVEL_BEST: + xfl = b'\002' + elif compresslevel == _COMPRESS_LEVEL_FAST: + xfl = b'\004' + else: + xfl = b'\000' + self.fileobj.write(xfl) self.fileobj.write(b'\377') if fname: self.fileobj.write(fname + b'\000') diff --git a/Lib/test/test_gzip.py b/Lib/test/test_gzip.py index 48a36a30913f6a..646828621a53c8 100644 --- a/Lib/test/test_gzip.py +++ b/Lib/test/test_gzip.py @@ -358,6 +358,26 @@ def test_metadata(self): isizeBytes = fRead.read(4) self.assertEqual(isizeBytes, struct.pack(' Date: Tue, 21 Jan 2020 15:04:33 +0100 Subject: [PATCH 1111/2163] Move test_math tests (GH-18098) (GH-18102) testPerm() and testComb() belong to MathTests, not to IsCloseTests(). (cherry picked from commit 59e2d26b258c12f18d8d2e789ef741703d6c52d5) --- Lib/test/test_math.py | 258 +++++++++++++++++++++--------------------- 1 file changed, 129 insertions(+), 129 deletions(-) diff --git a/Lib/test/test_math.py b/Lib/test/test_math.py index ac978247fc917f..6abaeead5c20cd 100644 --- a/Lib/test/test_math.py +++ b/Lib/test/test_math.py @@ -1744,135 +1744,6 @@ def _naive_prod(iterable, start=1): self.assertEqual(type(prod([1, decimal.Decimal(2.0), 3, 4, 5, 6])), decimal.Decimal) - # Custom assertions. - - def assertIsNaN(self, value): - if not math.isnan(value): - self.fail("Expected a NaN, got {!r}.".format(value)) - - -class IsCloseTests(unittest.TestCase): - isclose = math.isclose # subclasses should override this - - def assertIsClose(self, a, b, *args, **kwargs): - self.assertTrue(self.isclose(a, b, *args, **kwargs), - msg="%s and %s should be close!" % (a, b)) - - def assertIsNotClose(self, a, b, *args, **kwargs): - self.assertFalse(self.isclose(a, b, *args, **kwargs), - msg="%s and %s should not be close!" % (a, b)) - - def assertAllClose(self, examples, *args, **kwargs): - for a, b in examples: - self.assertIsClose(a, b, *args, **kwargs) - - def assertAllNotClose(self, examples, *args, **kwargs): - for a, b in examples: - self.assertIsNotClose(a, b, *args, **kwargs) - - def test_negative_tolerances(self): - # ValueError should be raised if either tolerance is less than zero - with self.assertRaises(ValueError): - self.assertIsClose(1, 1, rel_tol=-1e-100) - with self.assertRaises(ValueError): - self.assertIsClose(1, 1, rel_tol=1e-100, abs_tol=-1e10) - - def test_identical(self): - # identical values must test as close - identical_examples = [(2.0, 2.0), - (0.1e200, 0.1e200), - (1.123e-300, 1.123e-300), - (12345, 12345.0), - (0.0, -0.0), - (345678, 345678)] - self.assertAllClose(identical_examples, rel_tol=0.0, abs_tol=0.0) - - def test_eight_decimal_places(self): - # examples that are close to 1e-8, but not 1e-9 - eight_decimal_places_examples = [(1e8, 1e8 + 1), - (-1e-8, -1.000000009e-8), - (1.12345678, 1.12345679)] - self.assertAllClose(eight_decimal_places_examples, rel_tol=1e-8) - self.assertAllNotClose(eight_decimal_places_examples, rel_tol=1e-9) - - def test_near_zero(self): - # values close to zero - near_zero_examples = [(1e-9, 0.0), - (-1e-9, 0.0), - (-1e-150, 0.0)] - # these should not be close to any rel_tol - self.assertAllNotClose(near_zero_examples, rel_tol=0.9) - # these should be close to abs_tol=1e-8 - self.assertAllClose(near_zero_examples, abs_tol=1e-8) - - def test_identical_infinite(self): - # these are close regardless of tolerance -- i.e. they are equal - self.assertIsClose(INF, INF) - self.assertIsClose(INF, INF, abs_tol=0.0) - self.assertIsClose(NINF, NINF) - self.assertIsClose(NINF, NINF, abs_tol=0.0) - - def test_inf_ninf_nan(self): - # these should never be close (following IEEE 754 rules for equality) - not_close_examples = [(NAN, NAN), - (NAN, 1e-100), - (1e-100, NAN), - (INF, NAN), - (NAN, INF), - (INF, NINF), - (INF, 1.0), - (1.0, INF), - (INF, 1e308), - (1e308, INF)] - # use largest reasonable tolerance - self.assertAllNotClose(not_close_examples, abs_tol=0.999999999999999) - - def test_zero_tolerance(self): - # test with zero tolerance - zero_tolerance_close_examples = [(1.0, 1.0), - (-3.4, -3.4), - (-1e-300, -1e-300)] - self.assertAllClose(zero_tolerance_close_examples, rel_tol=0.0) - - zero_tolerance_not_close_examples = [(1.0, 1.000000000000001), - (0.99999999999999, 1.0), - (1.0e200, .999999999999999e200)] - self.assertAllNotClose(zero_tolerance_not_close_examples, rel_tol=0.0) - - def test_asymmetry(self): - # test the asymmetry example from PEP 485 - self.assertAllClose([(9, 10), (10, 9)], rel_tol=0.1) - - def test_integers(self): - # test with integer values - integer_examples = [(100000001, 100000000), - (123456789, 123456788)] - - self.assertAllClose(integer_examples, rel_tol=1e-8) - self.assertAllNotClose(integer_examples, rel_tol=1e-9) - - def test_decimals(self): - # test with Decimal values - from decimal import Decimal - - decimal_examples = [(Decimal('1.00000001'), Decimal('1.0')), - (Decimal('1.00000001e-20'), Decimal('1.0e-20')), - (Decimal('1.00000001e-100'), Decimal('1.0e-100')), - (Decimal('1.00000001e20'), Decimal('1.0e20'))] - self.assertAllClose(decimal_examples, rel_tol=1e-8) - self.assertAllNotClose(decimal_examples, rel_tol=1e-9) - - def test_fractions(self): - # test with Fraction values - from fractions import Fraction - - fraction_examples = [ - (Fraction(1, 100000000) + 1, Fraction(1)), - (Fraction(100000001), Fraction(100000000)), - (Fraction(10**8 + 1, 10**28), Fraction(1, 10**20))] - self.assertAllClose(fraction_examples, rel_tol=1e-8) - self.assertAllNotClose(fraction_examples, rel_tol=1e-9) - def testPerm(self): perm = math.perm factorial = math.factorial @@ -2007,6 +1878,135 @@ def testComb(self): self.assertIs(type(comb(IntSubclass(5), IntSubclass(k))), int) self.assertIs(type(comb(MyIndexable(5), MyIndexable(k))), int) + # Custom assertions. + + def assertIsNaN(self, value): + if not math.isnan(value): + self.fail("Expected a NaN, got {!r}.".format(value)) + + +class IsCloseTests(unittest.TestCase): + isclose = math.isclose # subclasses should override this + + def assertIsClose(self, a, b, *args, **kwargs): + self.assertTrue(self.isclose(a, b, *args, **kwargs), + msg="%s and %s should be close!" % (a, b)) + + def assertIsNotClose(self, a, b, *args, **kwargs): + self.assertFalse(self.isclose(a, b, *args, **kwargs), + msg="%s and %s should not be close!" % (a, b)) + + def assertAllClose(self, examples, *args, **kwargs): + for a, b in examples: + self.assertIsClose(a, b, *args, **kwargs) + + def assertAllNotClose(self, examples, *args, **kwargs): + for a, b in examples: + self.assertIsNotClose(a, b, *args, **kwargs) + + def test_negative_tolerances(self): + # ValueError should be raised if either tolerance is less than zero + with self.assertRaises(ValueError): + self.assertIsClose(1, 1, rel_tol=-1e-100) + with self.assertRaises(ValueError): + self.assertIsClose(1, 1, rel_tol=1e-100, abs_tol=-1e10) + + def test_identical(self): + # identical values must test as close + identical_examples = [(2.0, 2.0), + (0.1e200, 0.1e200), + (1.123e-300, 1.123e-300), + (12345, 12345.0), + (0.0, -0.0), + (345678, 345678)] + self.assertAllClose(identical_examples, rel_tol=0.0, abs_tol=0.0) + + def test_eight_decimal_places(self): + # examples that are close to 1e-8, but not 1e-9 + eight_decimal_places_examples = [(1e8, 1e8 + 1), + (-1e-8, -1.000000009e-8), + (1.12345678, 1.12345679)] + self.assertAllClose(eight_decimal_places_examples, rel_tol=1e-8) + self.assertAllNotClose(eight_decimal_places_examples, rel_tol=1e-9) + + def test_near_zero(self): + # values close to zero + near_zero_examples = [(1e-9, 0.0), + (-1e-9, 0.0), + (-1e-150, 0.0)] + # these should not be close to any rel_tol + self.assertAllNotClose(near_zero_examples, rel_tol=0.9) + # these should be close to abs_tol=1e-8 + self.assertAllClose(near_zero_examples, abs_tol=1e-8) + + def test_identical_infinite(self): + # these are close regardless of tolerance -- i.e. they are equal + self.assertIsClose(INF, INF) + self.assertIsClose(INF, INF, abs_tol=0.0) + self.assertIsClose(NINF, NINF) + self.assertIsClose(NINF, NINF, abs_tol=0.0) + + def test_inf_ninf_nan(self): + # these should never be close (following IEEE 754 rules for equality) + not_close_examples = [(NAN, NAN), + (NAN, 1e-100), + (1e-100, NAN), + (INF, NAN), + (NAN, INF), + (INF, NINF), + (INF, 1.0), + (1.0, INF), + (INF, 1e308), + (1e308, INF)] + # use largest reasonable tolerance + self.assertAllNotClose(not_close_examples, abs_tol=0.999999999999999) + + def test_zero_tolerance(self): + # test with zero tolerance + zero_tolerance_close_examples = [(1.0, 1.0), + (-3.4, -3.4), + (-1e-300, -1e-300)] + self.assertAllClose(zero_tolerance_close_examples, rel_tol=0.0) + + zero_tolerance_not_close_examples = [(1.0, 1.000000000000001), + (0.99999999999999, 1.0), + (1.0e200, .999999999999999e200)] + self.assertAllNotClose(zero_tolerance_not_close_examples, rel_tol=0.0) + + def test_asymmetry(self): + # test the asymmetry example from PEP 485 + self.assertAllClose([(9, 10), (10, 9)], rel_tol=0.1) + + def test_integers(self): + # test with integer values + integer_examples = [(100000001, 100000000), + (123456789, 123456788)] + + self.assertAllClose(integer_examples, rel_tol=1e-8) + self.assertAllNotClose(integer_examples, rel_tol=1e-9) + + def test_decimals(self): + # test with Decimal values + from decimal import Decimal + + decimal_examples = [(Decimal('1.00000001'), Decimal('1.0')), + (Decimal('1.00000001e-20'), Decimal('1.0e-20')), + (Decimal('1.00000001e-100'), Decimal('1.0e-100')), + (Decimal('1.00000001e20'), Decimal('1.0e20'))] + self.assertAllClose(decimal_examples, rel_tol=1e-8) + self.assertAllNotClose(decimal_examples, rel_tol=1e-9) + + def test_fractions(self): + # test with Fraction values + from fractions import Fraction + + fraction_examples = [ + (Fraction(1, 100000000) + 1, Fraction(1)), + (Fraction(100000001), Fraction(100000000)), + (Fraction(10**8 + 1, 10**28), Fraction(1, 10**20))] + self.assertAllClose(fraction_examples, rel_tol=1e-8) + self.assertAllNotClose(fraction_examples, rel_tol=1e-9) + def test_main(): from doctest import DocFileSuite From 41b3e879303d147bb46f0d230d5a101bca40bb9f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 21 Jan 2020 11:22:21 -0800 Subject: [PATCH 1112/2163] PyLong_AsLongLong() docs should say 'long long' (GH-18082) (GH-18110) (cherry picked from commit 47be7d0108b4021ede111dbd15a095c725be46b7) Co-authored-by: Keith Erskine Co-authored-by: Keith Erskine --- Doc/c-api/long.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/c-api/long.rst b/Doc/c-api/long.rst index 33f6b3be58307a..f41d419bd37758 100644 --- a/Doc/c-api/long.rst +++ b/Doc/c-api/long.rst @@ -179,7 +179,7 @@ distinguished from a number. Use :c:func:`PyErr_Occurred` to disambiguate. :c:type:`PyLongObject`. Raise :exc:`OverflowError` if the value of *obj* is out of range for a - :c:type:`long`. + :c:type:`long long`. Returns ``-1`` on error. Use :c:func:`PyErr_Occurred` to disambiguate. From a755410e054e1e2390de5830befc08fe80706c66 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 21 Jan 2020 19:26:34 -0800 Subject: [PATCH 1113/2163] closes bpo-39415: Remove unused codes from longobject.c complexobject.c floatobject.c. (GH-18105) (cherry picked from commit 0d5eac8c327251f8edde5261cee43975d81311f6) Co-authored-by: Dong-hee Na --- Objects/complexobject.c | 20 -------------------- Objects/floatobject.c | 37 ------------------------------------- Objects/longobject.c | 11 ----------- 3 files changed, 68 deletions(-) diff --git a/Objects/complexobject.c b/Objects/complexobject.c index a49da4018411e7..e01409b971b0e1 100644 --- a/Objects/complexobject.c +++ b/Objects/complexobject.c @@ -743,29 +743,9 @@ complex__format__(PyObject* self, PyObject* args) return _PyUnicodeWriter_Finish(&writer); } -#if 0 -static PyObject * -complex_is_finite(PyObject *self) -{ - Py_complex c; - c = ((PyComplexObject *)self)->cval; - return PyBool_FromLong((long)(Py_IS_FINITE(c.real) && - Py_IS_FINITE(c.imag))); -} - -PyDoc_STRVAR(complex_is_finite_doc, -"complex.is_finite() -> bool\n" -"\n" -"Returns True if the real and the imaginary part is finite."); -#endif - static PyMethodDef complex_methods[] = { {"conjugate", (PyCFunction)complex_conjugate, METH_NOARGS, complex_conjugate_doc}, -#if 0 - {"is_finite", (PyCFunction)complex_is_finite, METH_NOARGS, - complex_is_finite_doc}, -#endif {"__getnewargs__", (PyCFunction)complex_getnewargs, METH_NOARGS}, {"__format__", (PyCFunction)complex__format__, METH_VARARGS, complex__format__doc}, diff --git a/Objects/floatobject.c b/Objects/floatobject.c index 1cb8ff795fb87f..609f66f8b32ca9 100644 --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -861,35 +861,6 @@ float_is_integer_impl(PyObject *self) return o; } -#if 0 -static PyObject * -float_is_inf(PyObject *v) -{ - double x = PyFloat_AsDouble(v); - if (x == -1.0 && PyErr_Occurred()) - return NULL; - return PyBool_FromLong((long)Py_IS_INFINITY(x)); -} - -static PyObject * -float_is_nan(PyObject *v) -{ - double x = PyFloat_AsDouble(v); - if (x == -1.0 && PyErr_Occurred()) - return NULL; - return PyBool_FromLong((long)Py_IS_NAN(x)); -} - -static PyObject * -float_is_finite(PyObject *v) -{ - double x = PyFloat_AsDouble(v); - if (x == -1.0 && PyErr_Occurred()) - return NULL; - return PyBool_FromLong((long)Py_IS_FINITE(x)); -} -#endif - /*[clinic input] float.__trunc__ @@ -1852,14 +1823,6 @@ static PyMethodDef float_methods[] = { FLOAT_FROMHEX_METHODDEF FLOAT_HEX_METHODDEF FLOAT_IS_INTEGER_METHODDEF -#if 0 - {"is_inf", (PyCFunction)float_is_inf, METH_NOARGS, - "Return True if the float is positive or negative infinite."}, - {"is_finite", (PyCFunction)float_is_finite, METH_NOARGS, - "Return True if the float is finite, neither infinite nor NaN."}, - {"is_nan", (PyCFunction)float_is_nan, METH_NOARGS, - "Return True if the float is not a number (NaN)."}, -#endif FLOAT___GETNEWARGS___METHODDEF FLOAT___GETFORMAT___METHODDEF FLOAT___SET_FORMAT___METHODDEF diff --git a/Objects/longobject.c b/Objects/longobject.c index 708934c51fe250..67dce97471522c 100644 --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -5464,13 +5464,6 @@ int_bit_length_impl(PyObject *self) return NULL; } -#if 0 -static PyObject * -long_is_finite(PyObject *v) -{ - Py_RETURN_TRUE; -} -#endif /*[clinic input] int.as_integer_ratio @@ -5628,10 +5621,6 @@ static PyMethodDef long_methods[] = { {"conjugate", long_long_meth, METH_NOARGS, "Returns self, the complex conjugate of any int."}, INT_BIT_LENGTH_METHODDEF -#if 0 - {"is_finite", (PyCFunction)long_is_finite, METH_NOARGS, - "Returns always True."}, -#endif INT_TO_BYTES_METHODDEF INT_FROM_BYTES_METHODDEF INT_AS_INTEGER_RATIO_METHODDEF From f84f65be5602e561fef04b66bb487fbc4e560db5 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 22 Jan 2020 13:50:40 +0100 Subject: [PATCH 1114/2163] bpo-39413: os.unsetenv() is not available on Windows (GH-18108) Update os.unsetenv() documentation: it is not available on Windows. --- Doc/library/os.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 9d2e8b2eea2392..7aadbcf773b6a7 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -643,7 +643,7 @@ process and user. calls to :func:`unsetenv` don't update ``os.environ``, so it is actually preferable to delete items of ``os.environ``. - .. availability:: most flavors of Unix, Windows. + .. availability:: most flavors of Unix. .. _os-newstreams: From fdb21609d944941f0732df72dc3d07a7a9a7efea Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 22 Jan 2020 10:11:22 -0800 Subject: [PATCH 1115/2163] bpo-39425: Fix list.count performance regression (GH-18119) (GH-18120) https://bugs.python.org/issue39425 Automerge-Triggered-By: @pablogsal (cherry picked from commit 14d80d0b605d8b148e14458e4c1853a940071462) Co-authored-by: Dong-hee Na Co-authored-by: Dong-hee Na --- Objects/listobject.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Objects/listobject.c b/Objects/listobject.c index d506c0817336b2..73afc44c39e36e 100644 --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -2586,6 +2586,10 @@ list_count(PyListObject *self, PyObject *value) for (i = 0; i < Py_SIZE(self); i++) { PyObject *obj = self->ob_item[i]; + if (obj == value) { + count++; + continue; + } Py_INCREF(obj); int cmp = PyObject_RichCompareBool(obj, value, Py_EQ); Py_DECREF(obj); From 3b7618809df647574e8aad4c2d869a1d55df147c Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 22 Jan 2020 20:13:29 -0800 Subject: [PATCH 1116/2163] bpo-39050: The Help button in IDLE's config menu works again (GH-17611) Co-authored-by: Terry Jan Reedy (cherry picked from commit 2e43b64c94e49f7133b9c26e84c9519935c49063) Co-authored-by: Zackery Spytz --- Lib/idlelib/NEWS.txt | 2 ++ Lib/idlelib/configdialog.py | 2 +- Lib/idlelib/idle_test/test_configdialog.py | 11 +++++++++++ .../IDLE/2020-01-22-22-28-06.bpo-39050.zkn0FO.rst | 1 + 4 files changed, 15 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/IDLE/2020-01-22-22-28-06.bpo-39050.zkn0FO.rst diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 6e30ef35081f21..b0bcacdf586042 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,8 @@ Released on 2019-12-16? ====================================== +bpo-39050: Make Settings dialog Help button work again. + bpo-32989: Add tests for editor newline_and_indent_event method. Remove dead code from pyparse find_good_parse_start method. diff --git a/Lib/idlelib/configdialog.py b/Lib/idlelib/configdialog.py index aaf319bbe1befd..0e007b516ea5e1 100644 --- a/Lib/idlelib/configdialog.py +++ b/Lib/idlelib/configdialog.py @@ -210,7 +210,7 @@ def help(self): """ page = self.note.tab(self.note.select(), option='text').strip() view_text(self, title='Help for IDLE preferences', - text=help_common+help_pages.get(page, '')) + contents=help_common+help_pages.get(page, '')) def deactivate_current_config(self): """Remove current key bindings. diff --git a/Lib/idlelib/idle_test/test_configdialog.py b/Lib/idlelib/idle_test/test_configdialog.py index 1f14ed1f264730..7c575d0e5992c2 100644 --- a/Lib/idlelib/idle_test/test_configdialog.py +++ b/Lib/idlelib/idle_test/test_configdialog.py @@ -47,6 +47,17 @@ def tearDownModule(): root.destroy() root = dialog = None +class ConfigDialogTest(unittest.TestCase): + + def test_help(self): + dialog.note.select(dialog.keyspage) + saved = configdialog.view_text + view = configdialog.view_text = Func() + dialog.help() + s = view.kwds['contents'] + self.assertTrue(s.startswith('When you click')) + self.assertTrue(s.endswith('a different name.\n')) + configdialog.view_text = saved class FontPageTest(unittest.TestCase): """Test that font widgets enable users to make font changes. diff --git a/Misc/NEWS.d/next/IDLE/2020-01-22-22-28-06.bpo-39050.zkn0FO.rst b/Misc/NEWS.d/next/IDLE/2020-01-22-22-28-06.bpo-39050.zkn0FO.rst new file mode 100644 index 00000000000000..e71265cdf109b6 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2020-01-22-22-28-06.bpo-39050.zkn0FO.rst @@ -0,0 +1 @@ +Make IDLE Settings dialog Help button work again. From 545fc51d950558ecec9ff64cb2f9c11469051524 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 22 Jan 2020 21:13:41 -0800 Subject: [PATCH 1117/2163] bpo-32989: IDLE - remove unneeded parameter (GH-18138) IDLE does not pass a non-default _synchre in any of its calls to pyparse.find_good_parse_start. (cherry picked from commit f9e07e116c32b6dc4561d0bdeb452ccde13b0e7c) Co-authored-by: Terry Jan Reedy --- Lib/idlelib/NEWS.txt | 3 ++- Lib/idlelib/pyparse.py | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index b0bcacdf586042..729d405116a3d9 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -6,7 +6,8 @@ Released on 2019-12-16? bpo-39050: Make Settings dialog Help button work again. bpo-32989: Add tests for editor newline_and_indent_event method. -Remove dead code from pyparse find_good_parse_start method. +Remove unneeded arguments and dead code from pyparse +find_good_parse_start method. bpo-38943: Fix autocomplete windows not always appearing on some systems. Patch by Johnny Najera. diff --git a/Lib/idlelib/pyparse.py b/Lib/idlelib/pyparse.py index 9fa20108960711..d34872b4396e1e 100644 --- a/Lib/idlelib/pyparse.py +++ b/Lib/idlelib/pyparse.py @@ -133,7 +133,7 @@ def set_code(self, s): self.code = s self.study_level = 0 - def find_good_parse_start(self, is_char_in_string, _synchre=_synchre): + def find_good_parse_start(self, is_char_in_string): """ Return index of a good place to begin parsing, as close to the end of the string as possible. This will be the start of some From 993811ffe75c2573f97fb3fd1414b34609b8c8db Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 23 Jan 2020 07:22:25 -0800 Subject: [PATCH 1118/2163] bpo-39421: Fix posible crash in heapq with custom comparison operators (GH-18118) * bpo-39421: Fix posible crash in heapq with custom comparison operators * fixup! bpo-39421: Fix posible crash in heapq with custom comparison operators * fixup! fixup! bpo-39421: Fix posible crash in heapq with custom comparison operators (cherry picked from commit 79f89e6e5a659846d1068e8b1bd8e491ccdef861) Co-authored-by: Pablo Galindo --- Lib/test/test_heapq.py | 31 ++++++++++++++++ .../2020-01-22-15-53-37.bpo-39421.O3nG7u.rst | 2 ++ Modules/_heapqmodule.c | 35 ++++++++++++++----- 3 files changed, 59 insertions(+), 9 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-22-15-53-37.bpo-39421.O3nG7u.rst diff --git a/Lib/test/test_heapq.py b/Lib/test/test_heapq.py index 861ba7540df2bc..6902573e8fa85a 100644 --- a/Lib/test/test_heapq.py +++ b/Lib/test/test_heapq.py @@ -432,6 +432,37 @@ def test_heappop_mutating_heap(self): with self.assertRaises((IndexError, RuntimeError)): self.module.heappop(heap) + def test_comparison_operator_modifiying_heap(self): + # See bpo-39421: Strong references need to be taken + # when comparing objects as they can alter the heap + class EvilClass(int): + def __lt__(self, o): + heap.clear() + return NotImplemented + + heap = [] + self.module.heappush(heap, EvilClass(0)) + self.assertRaises(IndexError, self.module.heappushpop, heap, 1) + + def test_comparison_operator_modifiying_heap_two_heaps(self): + + class h(int): + def __lt__(self, o): + list2.clear() + return NotImplemented + + class g(int): + def __lt__(self, o): + list1.clear() + return NotImplemented + + list1, list2 = [], [] + + self.module.heappush(list1, h(0)) + self.module.heappush(list2, g(0)) + + self.assertRaises((IndexError, RuntimeError), self.module.heappush, list1, g(1)) + self.assertRaises((IndexError, RuntimeError), self.module.heappush, list2, h(1)) class TestErrorHandlingPython(TestErrorHandling, TestCase): module = py_heapq diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-22-15-53-37.bpo-39421.O3nG7u.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-22-15-53-37.bpo-39421.O3nG7u.rst new file mode 100644 index 00000000000000..bae008150ee127 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-01-22-15-53-37.bpo-39421.O3nG7u.rst @@ -0,0 +1,2 @@ +Fix possible crashes when operating with the functions in the :mod:`heapq` +module and custom comparison operators. diff --git a/Modules/_heapqmodule.c b/Modules/_heapqmodule.c index a84cade3aaa16a..6bc18b5f82fb82 100644 --- a/Modules/_heapqmodule.c +++ b/Modules/_heapqmodule.c @@ -36,7 +36,11 @@ siftdown(PyListObject *heap, Py_ssize_t startpos, Py_ssize_t pos) while (pos > startpos) { parentpos = (pos - 1) >> 1; parent = arr[parentpos]; + Py_INCREF(newitem); + Py_INCREF(parent); cmp = PyObject_RichCompareBool(newitem, parent, Py_LT); + Py_DECREF(parent); + Py_DECREF(newitem); if (cmp < 0) return -1; if (size != PyList_GET_SIZE(heap)) { @@ -78,10 +82,13 @@ siftup(PyListObject *heap, Py_ssize_t pos) /* Set childpos to index of smaller child. */ childpos = 2*pos + 1; /* leftmost child position */ if (childpos + 1 < endpos) { - cmp = PyObject_RichCompareBool( - arr[childpos], - arr[childpos + 1], - Py_LT); + PyObject* a = arr[childpos]; + PyObject* b = arr[childpos + 1]; + Py_INCREF(a); + Py_INCREF(b); + cmp = PyObject_RichCompareBool(a, b, Py_LT); + Py_DECREF(a); + Py_DECREF(b); if (cmp < 0) return -1; childpos += ((unsigned)cmp ^ 1); /* increment when cmp==0 */ @@ -264,7 +271,10 @@ _heapq_heappushpop_impl(PyObject *module, PyObject *heap, PyObject *item) return item; } - cmp = PyObject_RichCompareBool(PyList_GET_ITEM(heap, 0), item, Py_LT); + PyObject* top = PyList_GET_ITEM(heap, 0); + Py_INCREF(top); + cmp = PyObject_RichCompareBool(top, item, Py_LT); + Py_DECREF(top); if (cmp < 0) return NULL; if (cmp == 0) { @@ -420,7 +430,11 @@ siftdown_max(PyListObject *heap, Py_ssize_t startpos, Py_ssize_t pos) while (pos > startpos) { parentpos = (pos - 1) >> 1; parent = arr[parentpos]; + Py_INCREF(parent); + Py_INCREF(newitem); cmp = PyObject_RichCompareBool(parent, newitem, Py_LT); + Py_DECREF(parent); + Py_DECREF(newitem); if (cmp < 0) return -1; if (size != PyList_GET_SIZE(heap)) { @@ -462,10 +476,13 @@ siftup_max(PyListObject *heap, Py_ssize_t pos) /* Set childpos to index of smaller child. */ childpos = 2*pos + 1; /* leftmost child position */ if (childpos + 1 < endpos) { - cmp = PyObject_RichCompareBool( - arr[childpos + 1], - arr[childpos], - Py_LT); + PyObject* a = arr[childpos + 1]; + PyObject* b = arr[childpos]; + Py_INCREF(a); + Py_INCREF(b); + cmp = PyObject_RichCompareBool(a, b, Py_LT); + Py_DECREF(a); + Py_DECREF(b); if (cmp < 0) return -1; childpos += ((unsigned)cmp ^ 1); /* increment when cmp==0 */ From 5a2356be1a3f4a9b2e509a16062016b7d9dee7e9 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 23 Jan 2020 10:22:36 -0800 Subject: [PATCH 1119/2163] bpo-39431: Also mention nonlocal in assignment quirk (GH-17375) (cherry picked from commit 7142df5ea23b4ce0efb72746b4b3b65414e8dcb1) Co-authored-by: Shanavas M --- Doc/tutorial/classes.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/tutorial/classes.rst b/Doc/tutorial/classes.rst index 0c0dca99f21f28..f781fecf832f08 100644 --- a/Doc/tutorial/classes.rst +++ b/Doc/tutorial/classes.rst @@ -143,10 +143,10 @@ language definition is evolving towards static name resolution, at "compile" time, so don't rely on dynamic name resolution! (In fact, local variables are already determined statically.) -A special quirk of Python is that -- if no :keyword:`global` statement is in -effect -- assignments to names always go into the innermost scope. Assignments -do not copy data --- they just bind names to objects. The same is true for -deletions: the statement ``del x`` removes the binding of ``x`` from the +A special quirk of Python is that -- if no :keyword:`global` or :keyword:`nonlocal` +statement is in effect -- assignments to names always go into the innermost scope. +Assignments do not copy data --- they just bind names to objects. The same is true +for deletions: the statement ``del x`` removes the binding of ``x`` from the namespace referenced by the local scope. In fact, all operations that introduce new names use the local scope: in particular, :keyword:`import` statements and function definitions bind the module or function name in the local scope. From 5654f83b9706af88040f515791f1cdc5d81cd9d6 Mon Sep 17 00:00:00 2001 From: Alex Rebert Date: Thu, 23 Jan 2020 18:31:29 -0500 Subject: [PATCH 1120/2163] [3.8] bpo-35182: fix communicate() crash after child closes its pipes (GH-18117) (GH-18148) When communicate() is called in a loop, it crashes when the child process has already closed any piped standard stream, but still continues to be running Co-authored-by: Andriy Maletsky . (cherry picked from commit d3ae95e1e945ed20297e1c38ba43a18b7a868ab6) Co-authored-by: Alex Rebert https://bugs.python.org/issue35182 --- Lib/subprocess.py | 4 ++-- Lib/test/test_subprocess.py | 11 +++++++++++ .../Library/2019-10-31-19-23-25.bpo-35182.hzeNl9.rst | 3 +++ 3 files changed, 16 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-10-31-19-23-25.bpo-35182.hzeNl9.rst diff --git a/Lib/subprocess.py b/Lib/subprocess.py index 85b9ea07854669..5c2c2f05093f7e 100644 --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1848,9 +1848,9 @@ def _communicate(self, input, endtime, orig_timeout): with _PopenSelector() as selector: if self.stdin and input: selector.register(self.stdin, selectors.EVENT_WRITE) - if self.stdout: + if self.stdout and not self.stdout.closed: selector.register(self.stdout, selectors.EVENT_READ) - if self.stderr: + if self.stderr and not self.stderr.closed: selector.register(self.stderr, selectors.EVENT_READ) while selector.get_map(): diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index b2afd127dfed87..80acb06e4f9204 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -2910,6 +2910,17 @@ def test_stopped(self): self.assertEqual(returncode, -3) + def test_communicate_repeated_call_after_stdout_close(self): + proc = subprocess.Popen([sys.executable, '-c', + 'import os, time; os.close(1), time.sleep(2)'], + stdout=subprocess.PIPE) + while True: + try: + proc.communicate(timeout=0.1) + return + except subprocess.TimeoutExpired: + pass + @unittest.skipUnless(mswindows, "Windows specific tests") class Win32ProcessTestCase(BaseTestCase): diff --git a/Misc/NEWS.d/next/Library/2019-10-31-19-23-25.bpo-35182.hzeNl9.rst b/Misc/NEWS.d/next/Library/2019-10-31-19-23-25.bpo-35182.hzeNl9.rst new file mode 100644 index 00000000000000..9438cd8f9fd0ba --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-10-31-19-23-25.bpo-35182.hzeNl9.rst @@ -0,0 +1,3 @@ +Fixed :func:`Popen.communicate` subsequent call crash when the child process +has already closed any piped standard stream, but still continues to be +running. Patch by Andriy Maletsky. From a46728a570e30e88df253eab17ad6c4372a422da Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 24 Jan 2020 02:20:35 -0800 Subject: [PATCH 1121/2163] bpo-39426: Fix outdated default and highest protocols in docs (GH-18154) Some portions of the pickle documentation hadn't been updated for the pickle protocol changes in Python 3.8 (new protocol 5, default protocol 4). This PR fixes those docs. https://bugs.python.org/issue39426 (cherry picked from commit e9652e8d58392f5022759ba06b444ce970eb12db) Co-authored-by: Mark Dickinson --- Lib/pickle.py | 6 +++--- Modules/_pickle.c | 19 ++++++++++--------- Modules/clinic/_pickle.c.h | 15 ++++++++------- 3 files changed, 21 insertions(+), 19 deletions(-) diff --git a/Lib/pickle.py b/Lib/pickle.py index 71aa57d500ecce..515cb8a0bb3f1a 100644 --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -409,9 +409,9 @@ def __init__(self, file, protocol=None, *, fix_imports=True, """This takes a binary file for writing a pickle data stream. The optional *protocol* argument tells the pickler to use the - given protocol; supported protocols are 0, 1, 2, 3 and 4. The - default protocol is 4. It was introduced in Python 3.4, it is - incompatible with previous versions. + given protocol; supported protocols are 0, 1, 2, 3, 4 and 5. + The default protocol is 4. It was introduced in Python 3.4, and + is incompatible with previous versions. Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the diff --git a/Modules/_pickle.c b/Modules/_pickle.c index 8ee7d8ee2a7931..8150bf3b33dc38 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -4647,8 +4647,9 @@ _pickle.Pickler.__init__ This takes a binary file for writing a pickle data stream. The optional *protocol* argument tells the pickler to use the given -protocol; supported protocols are 0, 1, 2, 3 and 4. The default -protocol is 3; a backward-incompatible protocol designed for Python 3. +protocol; supported protocols are 0, 1, 2, 3, 4 and 5. The default +protocol is 4. It was introduced in Python 3.4, and is incompatible +with previous versions. Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the more recent the @@ -4680,7 +4681,7 @@ static int _pickle_Pickler___init___impl(PicklerObject *self, PyObject *file, PyObject *protocol, int fix_imports, PyObject *buffer_callback) -/*[clinic end generated code: output=0abedc50590d259b input=bb886e00443a7811]*/ +/*[clinic end generated code: output=0abedc50590d259b input=a7c969699bf5dad3]*/ { _Py_IDENTIFIER(persistent_id); _Py_IDENTIFIER(dispatch_table); @@ -7633,8 +7634,8 @@ This is equivalent to ``Pickler(file, protocol).dump(obj)``, but may be more efficient. The optional *protocol* argument tells the pickler to use the given -protocol; supported protocols are 0, 1, 2, 3 and 4. The default -protocol is 4. It was introduced in Python 3.4, it is incompatible +protocol; supported protocols are 0, 1, 2, 3, 4 and 5. The default +protocol is 4. It was introduced in Python 3.4, and is incompatible with previous versions. Specifying a negative protocol version selects the highest protocol @@ -7660,7 +7661,7 @@ static PyObject * _pickle_dump_impl(PyObject *module, PyObject *obj, PyObject *file, PyObject *protocol, int fix_imports, PyObject *buffer_callback) -/*[clinic end generated code: output=706186dba996490c input=cfdcaf573ed6e46c]*/ +/*[clinic end generated code: output=706186dba996490c input=5ed6653da99cd97c]*/ { PicklerObject *pickler = _Pickler_New(); @@ -7703,8 +7704,8 @@ _pickle.dumps Return the pickled representation of the object as a bytes object. The optional *protocol* argument tells the pickler to use the given -protocol; supported protocols are 0, 1, 2, 3 and 4. The default -protocol is 4. It was introduced in Python 3.4, it is incompatible +protocol; supported protocols are 0, 1, 2, 3, 4 and 5. The default +protocol is 4. It was introduced in Python 3.4, and is incompatible with previous versions. Specifying a negative protocol version selects the highest protocol @@ -7724,7 +7725,7 @@ into *file* as part of the pickle stream. It is an error if static PyObject * _pickle_dumps_impl(PyObject *module, PyObject *obj, PyObject *protocol, int fix_imports, PyObject *buffer_callback) -/*[clinic end generated code: output=fbab0093a5580fdf input=9f334d535ff7194f]*/ +/*[clinic end generated code: output=fbab0093a5580fdf input=e543272436c6f987]*/ { PyObject *result; PicklerObject *pickler = _Pickler_New(); diff --git a/Modules/clinic/_pickle.c.h b/Modules/clinic/_pickle.c.h index 9da3f1195a3bf2..0457a433e79fb0 100644 --- a/Modules/clinic/_pickle.c.h +++ b/Modules/clinic/_pickle.c.h @@ -69,8 +69,9 @@ PyDoc_STRVAR(_pickle_Pickler___init____doc__, "This takes a binary file for writing a pickle data stream.\n" "\n" "The optional *protocol* argument tells the pickler to use the given\n" -"protocol; supported protocols are 0, 1, 2, 3 and 4. The default\n" -"protocol is 3; a backward-incompatible protocol designed for Python 3.\n" +"protocol; supported protocols are 0, 1, 2, 3, 4 and 5. The default\n" +"protocol is 4. It was introduced in Python 3.4, and is incompatible\n" +"with previous versions.\n" "\n" "Specifying a negative protocol version selects the highest protocol\n" "version supported. The higher the protocol used, the more recent the\n" @@ -463,8 +464,8 @@ PyDoc_STRVAR(_pickle_dump__doc__, "be more efficient.\n" "\n" "The optional *protocol* argument tells the pickler to use the given\n" -"protocol; supported protocols are 0, 1, 2, 3 and 4. The default\n" -"protocol is 4. It was introduced in Python 3.4, it is incompatible\n" +"protocol; supported protocols are 0, 1, 2, 3, 4 and 5. The default\n" +"protocol is 4. It was introduced in Python 3.4, and is incompatible\n" "with previous versions.\n" "\n" "Specifying a negative protocol version selects the highest protocol\n" @@ -550,8 +551,8 @@ PyDoc_STRVAR(_pickle_dumps__doc__, "Return the pickled representation of the object as a bytes object.\n" "\n" "The optional *protocol* argument tells the pickler to use the given\n" -"protocol; supported protocols are 0, 1, 2, 3 and 4. The default\n" -"protocol is 4. It was introduced in Python 3.4, it is incompatible\n" +"protocol; supported protocols are 0, 1, 2, 3, 4 and 5. The default\n" +"protocol is 4. It was introduced in Python 3.4, and is incompatible\n" "with previous versions.\n" "\n" "Specifying a negative protocol version selects the highest protocol\n" @@ -835,4 +836,4 @@ _pickle_loads(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObjec exit: return return_value; } -/*[clinic end generated code: output=de075ec48d4ee0e1 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=e2506823be1960c5 input=a9049054013a1b77]*/ From 0ea7309927d1b3efde9be986528b8855f0f4c955 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 24 Jan 2020 05:27:04 -0800 Subject: [PATCH 1122/2163] Improve test coverage for AsyncMock. (GH-17906) * Add test for nested async decorator patch. * Add test for side_effect and wraps with a function. * Add test for side_effect with an exception in the iterable. (cherry picked from commit 54f743eb315f00b0ff45e115dde7a5d506034153) Co-authored-by: Karthikeyan Singaravelan --- Lib/unittest/test/testmock/testasync.py | 53 +++++++++++++++++++++++-- 1 file changed, 49 insertions(+), 4 deletions(-) diff --git a/Lib/unittest/test/testmock/testasync.py b/Lib/unittest/test/testmock/testasync.py index be8e3117c466b3..0bd7ae4929c3e7 100644 --- a/Lib/unittest/test/testmock/testasync.py +++ b/Lib/unittest/test/testmock/testasync.py @@ -72,9 +72,17 @@ def test_async(mock_method): test_async() def test_async_def_patch(self): - @patch(f"{__name__}.async_func", AsyncMock()) - async def test_async(): + @patch(f"{__name__}.async_func", return_value=1) + @patch(f"{__name__}.async_func_args", return_value=2) + async def test_async(func_args_mock, func_mock): + self.assertEqual(func_args_mock._mock_name, "async_func_args") + self.assertEqual(func_mock._mock_name, "async_func") + self.assertIsInstance(async_func, AsyncMock) + self.assertIsInstance(async_func_args, AsyncMock) + + self.assertEqual(await async_func(), 1) + self.assertEqual(await async_func_args(1, 2, c=3), 2) asyncio.run(test_async()) self.assertTrue(inspect.iscoroutinefunction(async_func)) @@ -370,22 +378,40 @@ async def addition(var): with self.assertRaises(Exception): await mock(5) - async def test_add_side_effect_function(self): + async def test_add_side_effect_coroutine(self): async def addition(var): return var + 1 mock = AsyncMock(side_effect=addition) result = await mock(5) self.assertEqual(result, 6) + async def test_add_side_effect_normal_function(self): + def addition(var): + return var + 1 + mock = AsyncMock(side_effect=addition) + result = await mock(5) + self.assertEqual(result, 6) + async def test_add_side_effect_iterable(self): vals = [1, 2, 3] mock = AsyncMock(side_effect=vals) for item in vals: - self.assertEqual(item, await mock()) + self.assertEqual(await mock(), item) with self.assertRaises(StopAsyncIteration) as e: await mock() + async def test_add_side_effect_exception_iterable(self): + class SampleException(Exception): + pass + + vals = [1, SampleException("foo")] + mock = AsyncMock(side_effect=vals) + self.assertEqual(await mock(), 1) + + with self.assertRaises(SampleException) as e: + await mock() + async def test_return_value_AsyncMock(self): value = AsyncMock(return_value=10) mock = AsyncMock(return_value=value) @@ -432,6 +458,21 @@ async def inner(): mock.assert_awaited() self.assertTrue(ran) + async def test_wraps_normal_function(self): + value = 1 + + ran = False + def inner(): + nonlocal ran + ran = True + return value + + mock = AsyncMock(wraps=inner) + result = await mock() + self.assertEqual(result, value) + mock.assert_awaited() + self.assertTrue(ran) + class AsyncMagicMethods(unittest.TestCase): def test_async_magic_methods_return_async_mocks(self): m_mock = MagicMock() @@ -854,6 +895,10 @@ def test_assert_awaited_once(self): self.mock.assert_awaited_once() def test_assert_awaited_with(self): + msg = 'Not awaited' + with self.assertRaisesRegex(AssertionError, msg): + self.mock.assert_awaited_with('foo') + asyncio.run(self._runnable_test()) msg = 'expected await not found' with self.assertRaisesRegex(AssertionError, msg): From ea4a61fec842c94107eef46e5030b89a086f94bb Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 24 Jan 2020 12:10:42 -0800 Subject: [PATCH 1123/2163] bpo-39430: Fix race condition in lazy imports in tarfile. (GH-18161) Use `from ... import ...` to ensure module is fully loaded before accessing its attributes. (cherry picked from commit 9017e0bd5e124ae6d2ed94b9e9cacb2e86270980) Co-authored-by: Serhiy Storchaka --- Lib/tarfile.py | 18 ++++++++---------- .../2020-01-24-11-05-21.bpo-39430.I0UQzM.rst | 1 + 2 files changed, 9 insertions(+), 10 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-01-24-11-05-21.bpo-39430.I0UQzM.rst diff --git a/Lib/tarfile.py b/Lib/tarfile.py index 2c06f9160c658a..d31b9cbb51d654 100755 --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -1655,13 +1655,12 @@ def gzopen(cls, name, mode="r", fileobj=None, compresslevel=9, **kwargs): raise ValueError("mode must be 'r', 'w' or 'x'") try: - import gzip - gzip.GzipFile - except (ImportError, AttributeError): + from gzip import GzipFile + except ImportError: raise CompressionError("gzip module is not available") try: - fileobj = gzip.GzipFile(name, mode + "b", compresslevel, fileobj) + fileobj = GzipFile(name, mode + "b", compresslevel, fileobj) except OSError: if fileobj is not None and mode == 'r': raise ReadError("not a gzip file") @@ -1689,12 +1688,11 @@ def bz2open(cls, name, mode="r", fileobj=None, compresslevel=9, **kwargs): raise ValueError("mode must be 'r', 'w' or 'x'") try: - import bz2 + from bz2 import BZ2File except ImportError: raise CompressionError("bz2 module is not available") - fileobj = bz2.BZ2File(fileobj or name, mode, - compresslevel=compresslevel) + fileobj = BZ2File(fileobj or name, mode, compresslevel=compresslevel) try: t = cls.taropen(name, mode, fileobj, **kwargs) @@ -1718,15 +1716,15 @@ def xzopen(cls, name, mode="r", fileobj=None, preset=None, **kwargs): raise ValueError("mode must be 'r', 'w' or 'x'") try: - import lzma + from lzma import LZMAFile, LZMAError except ImportError: raise CompressionError("lzma module is not available") - fileobj = lzma.LZMAFile(fileobj or name, mode, preset=preset) + fileobj = LZMAFile(fileobj or name, mode, preset=preset) try: t = cls.taropen(name, mode, fileobj, **kwargs) - except (lzma.LZMAError, EOFError): + except (LZMAError, EOFError): fileobj.close() if mode == 'r': raise ReadError("not an lzma file") diff --git a/Misc/NEWS.d/next/Library/2020-01-24-11-05-21.bpo-39430.I0UQzM.rst b/Misc/NEWS.d/next/Library/2020-01-24-11-05-21.bpo-39430.I0UQzM.rst new file mode 100644 index 00000000000000..712fc5d34bbe08 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-24-11-05-21.bpo-39430.I0UQzM.rst @@ -0,0 +1 @@ +Fixed race condition in lazy imports in :mod:`tarfile`. From a23449285430081ea317cc1f203c80c410bbd8b9 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 25 Jan 2020 01:17:54 -0800 Subject: [PATCH 1124/2163] bpo-39388: IDLE: Fix bug when cancelling out of configdialog (GH-18068) Co-authored-by: Terry Jan Reedy (cherry picked from commit d0d9fa8c5e30aff71b6d5e8b2673396622f33270) Co-authored-by: Cheryl Sabella --- Lib/idlelib/NEWS.txt | 4 +++- Lib/idlelib/configdialog.py | 1 + Lib/idlelib/idle_test/test_configdialog.py | 21 ++++++++++++------- .../2020-01-25-02-26-45.bpo-39388.x4TQNh.rst | 1 + 4 files changed, 19 insertions(+), 8 deletions(-) create mode 100644 Misc/NEWS.d/next/IDLE/2020-01-25-02-26-45.bpo-39388.x4TQNh.rst diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 729d405116a3d9..b7217335fe2814 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,7 +3,9 @@ Released on 2019-12-16? ====================================== -bpo-39050: Make Settings dialog Help button work again. +bpo-39388: Settings dialog Cancel button cancels pending changes. + +bpo-39050: Settings dialog Help button again displays help text. bpo-32989: Add tests for editor newline_and_indent_event method. Remove unneeded arguments and dead code from pyparse diff --git a/Lib/idlelib/configdialog.py b/Lib/idlelib/configdialog.py index 0e007b516ea5e1..2f95c9ccaa0c57 100644 --- a/Lib/idlelib/configdialog.py +++ b/Lib/idlelib/configdialog.py @@ -191,6 +191,7 @@ def cancel(self): Methods: destroy: inherited """ + changes.clear() self.destroy() def destroy(self): diff --git a/Lib/idlelib/idle_test/test_configdialog.py b/Lib/idlelib/idle_test/test_configdialog.py index 7c575d0e5992c2..817a35217bf3cd 100644 --- a/Lib/idlelib/idle_test/test_configdialog.py +++ b/Lib/idlelib/idle_test/test_configdialog.py @@ -47,17 +47,24 @@ def tearDownModule(): root.destroy() root = dialog = None -class ConfigDialogTest(unittest.TestCase): - def test_help(self): +class DialogTest(unittest.TestCase): + + @mock.patch(__name__+'.dialog.destroy', new_callable=Func) + def test_cancel(self, destroy): + changes['main']['something'] = 1 + dialog.cancel() + self.assertEqual(changes['main'], {}) + self.assertEqual(destroy.called, 1) + + @mock.patch('idlelib.configdialog.view_text', new_callable=Func) + def test_help(self, view): dialog.note.select(dialog.keyspage) - saved = configdialog.view_text - view = configdialog.view_text = Func() dialog.help() s = view.kwds['contents'] - self.assertTrue(s.startswith('When you click')) - self.assertTrue(s.endswith('a different name.\n')) - configdialog.view_text = saved + self.assertTrue(s.startswith('When you click') and + s.endswith('a different name.\n')) + class FontPageTest(unittest.TestCase): """Test that font widgets enable users to make font changes. diff --git a/Misc/NEWS.d/next/IDLE/2020-01-25-02-26-45.bpo-39388.x4TQNh.rst b/Misc/NEWS.d/next/IDLE/2020-01-25-02-26-45.bpo-39388.x4TQNh.rst new file mode 100644 index 00000000000000..42bbfb168c19de --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2020-01-25-02-26-45.bpo-39388.x4TQNh.rst @@ -0,0 +1 @@ +IDLE Settings Cancel button now cancels pending changes From a5906b2bfce9560568dee1dcc3550e74e742dd34 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 25 Jan 2020 06:53:08 -0800 Subject: [PATCH 1125/2163] bpo-38473: Handle autospecced functions and methods used with attach_mock (GH-16784) (GH-18167) If an autospecced object is attached using attach_mock the child would be a function with mock object as attribute from which signature has to be derived. (cherry picked from commit 66b00a9d3aacf6ed49412f48743e4913104a2bb3) Co-authored-by: Karthikeyan Singaravelan --- Lib/unittest/mock.py | 4 +++ Lib/unittest/test/testmock/testmock.py | 29 +++++++++++++++++++ .../2019-10-14-21-14-55.bpo-38473.uXpVld.rst | 2 ++ 3 files changed, 35 insertions(+) create mode 100644 Misc/NEWS.d/next/Library/2019-10-14-21-14-55.bpo-38473.uXpVld.rst diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py index e92ccf168dbbde..34f2dd77ec6cd3 100644 --- a/Lib/unittest/mock.py +++ b/Lib/unittest/mock.py @@ -825,6 +825,10 @@ def _get_call_signature_from_name(self, name): if child is None or isinstance(child, _SpecState): break else: + # If an autospecced object is attached using attach_mock the + # child would be a function with mock object as attribute from + # which signature has to be derived. + child = _extract_mock(child) children = child._mock_children sig = child._spec_signature diff --git a/Lib/unittest/test/testmock/testmock.py b/Lib/unittest/test/testmock/testmock.py index 6dc2725427ec56..1cde45e9aea555 100644 --- a/Lib/unittest/test/testmock/testmock.py +++ b/Lib/unittest/test/testmock/testmock.py @@ -1914,6 +1914,35 @@ def test_attach_mock_patch_autospec(self): self.assertEqual(mock_func.mock._extract_mock_name(), 'mock.child') + def test_attach_mock_patch_autospec_signature(self): + with mock.patch(f'{__name__}.Something.meth', autospec=True) as mocked: + manager = Mock() + manager.attach_mock(mocked, 'attach_meth') + obj = Something() + obj.meth(1, 2, 3, d=4) + manager.assert_has_calls([call.attach_meth(mock.ANY, 1, 2, 3, d=4)]) + obj.meth.assert_has_calls([call(mock.ANY, 1, 2, 3, d=4)]) + mocked.assert_has_calls([call(mock.ANY, 1, 2, 3, d=4)]) + + with mock.patch(f'{__name__}.something', autospec=True) as mocked: + manager = Mock() + manager.attach_mock(mocked, 'attach_func') + something(1) + manager.assert_has_calls([call.attach_func(1)]) + something.assert_has_calls([call(1)]) + mocked.assert_has_calls([call(1)]) + + with mock.patch(f'{__name__}.Something', autospec=True) as mocked: + manager = Mock() + manager.attach_mock(mocked, 'attach_obj') + obj = Something() + obj.meth(1, 2, 3, d=4) + manager.assert_has_calls([call.attach_obj(), + call.attach_obj().meth(1, 2, 3, d=4)]) + obj.meth.assert_has_calls([call(1, 2, 3, d=4)]) + mocked.assert_has_calls([call(), call().meth(1, 2, 3, d=4)]) + + def test_attribute_deletion(self): for mock in (Mock(), MagicMock(), NonCallableMagicMock(), NonCallableMock()): diff --git a/Misc/NEWS.d/next/Library/2019-10-14-21-14-55.bpo-38473.uXpVld.rst b/Misc/NEWS.d/next/Library/2019-10-14-21-14-55.bpo-38473.uXpVld.rst new file mode 100644 index 00000000000000..de80e89e00e2d3 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-10-14-21-14-55.bpo-38473.uXpVld.rst @@ -0,0 +1,2 @@ +Use signature from inner mock for autospecced methods attached with +:func:`unittest.mock.attach_mock`. Patch by Karthikeyan Singaravelan. From 079dfe082a4c7eb4038afbb1097fca9d90b83e1a Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 25 Jan 2020 09:33:36 -0800 Subject: [PATCH 1126/2163] Update 3.8.rst (GH-18173) (#18182) Fixed the name of the contributor (@selik). (cherry picked from commit 9bfb4a7061a3bc4fc5632bccfdf9ed61f62679f7) Co-authored-by: fireattack --- Doc/whatsnew/3.8.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index 546fa2d5f016ec..7bc91a9fc19bf5 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -727,7 +727,7 @@ csv The :class:`csv.DictReader` now returns instances of :class:`dict` instead of a :class:`collections.OrderedDict`. The tool is now faster and uses less memory while still preserving the field order. -(Contributed by Michael Seek in :issue:`34003`.) +(Contributed by Michael Selik in :issue:`34003`.) curses From 321491a536c378227f9d574703f7c06f89c67dcf Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 25 Jan 2020 11:04:16 -0800 Subject: [PATCH 1127/2163] bpo-15243: Document __prepare__ as classmethod (GH-17124) (cherry picked from commit 7de617455ed788e6730c40cf854c4b72b0432194) Co-authored-by: alclarks <57201106+alclarks@users.noreply.github.com> --- Doc/reference/datamodel.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index e801ee392f195e..7b3bf0d2f5f366 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1945,7 +1945,8 @@ Preparing the class namespace Once the appropriate metaclass has been identified, then the class namespace is prepared. If the metaclass has a ``__prepare__`` attribute, it is called as ``namespace = metaclass.__prepare__(name, bases, **kwds)`` (where the -additional keyword arguments, if any, come from the class definition). +additional keyword arguments, if any, come from the class definition). The +``__prepare__`` method should be implemented as a :func:`classmethod`. If the metaclass has no ``__prepare__`` attribute, then the class namespace is initialised as an empty ordered mapping. From 1cf0df4f1bcc38dfd70a152af20cf584de531ea7 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 25 Jan 2020 11:34:36 -0800 Subject: [PATCH 1128/2163] bpo-36654: Add examples for using tokenize module programmatically (GH-18187) (cherry picked from commit 4b09dc79f4d08d85f2cc945563e9c8ef1e531d7b) Co-authored-by: Windson yang --- Doc/library/tokenize.rst | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index b208ba46d17d99..96778f23f8f062 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -278,3 +278,22 @@ The exact token type names can be displayed using the :option:`-e` option: 4,10-4,11: RPAR ')' 4,11-4,12: NEWLINE '\n' 5,0-5,0: ENDMARKER '' + +Example of tokenizing a file programmatically, reading unicode +strings instead of bytes with :func:`generate_tokens`:: + + import tokenize + + with tokenize.open('hello.py') as f: + tokens = tokenize.generate_tokens(f.readline) + for token in tokens: + print(token) + +Or reading bytes directly with :func:`.tokenize`:: + + import tokenize + + with open('hello.py', 'rb') as f: + tokens = tokenize.tokenize(f.readline) + for token in tokens: + print(token) From eec7636bfd07412b5872c0683636e9e98bf79a8c Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 25 Jan 2020 11:40:27 -0800 Subject: [PATCH 1129/2163] bpo-39234: Doc: `enum.auto()` incrementation value not specified. (GH-17872) (GH-17875) (cherry picked from commit 2e9012a3e1e316c54e27f51ba5849ba06eab7da2) Co-authored-by: YoSTEALTH <35307184+YoSTEALTH@users.noreply.github.com> --- Doc/library/enum.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index 19277d76995fed..7538084767d760 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -55,7 +55,7 @@ helper, :class:`auto`. .. class:: auto - Instances are replaced with an appropriate value for Enum members. + Instances are replaced with an appropriate value for Enum members. Initial value starts at 1. .. versionadded:: 3.6 ``Flag``, ``IntFlag``, ``auto`` From eebcff8c071b38b53bd429892524ba8518cbeb98 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 25 Jan 2020 21:24:13 -0800 Subject: [PATCH 1130/2163] bpo-36018: Add another example for NormalDist() (GH-18191) (GH-18192) --- Doc/library/statistics.rst | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst index 4c7239c1895fbf..09b02cabf21f8e 100644 --- a/Doc/library/statistics.rst +++ b/Doc/library/statistics.rst @@ -772,6 +772,42 @@ Carlo simulation `_: >>> quantiles(map(model, X, Y, Z)) # doctest: +SKIP [1.4591308524824727, 1.8035946855390597, 2.175091447274739] +Normal distributions can be used to approximate `Binomial +distributions `_ +when the sample size is large and when the probability of a successful +trial is near 50%. + +For example, an open source conference has 750 attendees and two rooms with a +500 person capacity. There is a talk about Python and another about Ruby. +In previous conferences, 65% of the attendees preferred to listen to Python +talks. Assuming the population preferences haven't changed, what is the +probability that the rooms will stay within their capacity limits? + +.. doctest:: + + >>> n = 750 # Sample size + >>> p = 0.65 # Preference for Python + >>> q = 1.0 - p # Preference for Ruby + >>> k = 500 # Room capacity + + >>> # Approximation using the cumulative normal distribution + >>> from math import sqrt + >>> round(NormalDist(mu=n*p, sigma=sqrt(n*p*q)).cdf(k + 0.5), 4) + 0.8402 + + >>> # Solution using the cumulative binomial distribution + >>> from math import comb, fsum + >>> round(fsum(comb(n, r) * p**r * q**(n-r) for r in range(k+1)), 4) + 0.8402 + + >>> # Approximation using a simulation + >>> from random import seed, choices + >>> seed(8675309) + >>> def trial(): + ... return choices(('Python', 'Ruby'), (p, q), k=n).count('Python') + >>> mean(trial() <= k for i in range(10_000)) + 0.8398 + Normal distributions commonly arise in machine learning problems. Wikipedia has a `nice example of a Naive Bayesian Classifier From 19be85c76503535c101b38194d282187de0ff631 Mon Sep 17 00:00:00 2001 From: Matthew Kokotovich Date: Sun, 26 Jan 2020 09:30:27 -0600 Subject: [PATCH 1131/2163] [3.8] bpo-39082: Allow AsyncMock to correctly patch static/class methods (GH-18190) (cherry picked from commit 62865f4532094017a9b780b704686ca9734bc329) Co-authored-by: Matthew Kokotovich --- Lib/unittest/mock.py | 2 ++ Lib/unittest/test/testmock/testasync.py | 23 +++++++++++++++++++ .../2020-01-24-13-24-35.bpo-39082.qKgrq_.rst | 1 + 3 files changed, 26 insertions(+) create mode 100644 Misc/NEWS.d/next/Library/2020-01-24-13-24-35.bpo-39082.qKgrq_.rst diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py index 34f2dd77ec6cd3..66ace80cb5f9c3 100644 --- a/Lib/unittest/mock.py +++ b/Lib/unittest/mock.py @@ -48,6 +48,8 @@ def _is_async_obj(obj): if _is_instance_mock(obj) and not isinstance(obj, AsyncMock): return False + if hasattr(obj, '__func__'): + obj = getattr(obj, '__func__') return asyncio.iscoroutinefunction(obj) or inspect.isawaitable(obj) diff --git a/Lib/unittest/test/testmock/testasync.py b/Lib/unittest/test/testmock/testasync.py index 0bd7ae4929c3e7..e68022afdcceaa 100644 --- a/Lib/unittest/test/testmock/testasync.py +++ b/Lib/unittest/test/testmock/testasync.py @@ -19,6 +19,15 @@ async def async_method(self): def normal_method(self): pass + @classmethod + async def async_class_method(cls): + pass + + @staticmethod + async def async_static_method(): + pass + + class AwaitableClass: def __await__(self): yield @@ -71,6 +80,20 @@ def test_async(mock_method): test_async() + def test_is_AsyncMock_patch_staticmethod(self): + @patch.object(AsyncClass, 'async_static_method') + def test_async(mock_method): + self.assertIsInstance(mock_method, AsyncMock) + + test_async() + + def test_is_AsyncMock_patch_classmethod(self): + @patch.object(AsyncClass, 'async_class_method') + def test_async(mock_method): + self.assertIsInstance(mock_method, AsyncMock) + + test_async() + def test_async_def_patch(self): @patch(f"{__name__}.async_func", return_value=1) @patch(f"{__name__}.async_func_args", return_value=2) diff --git a/Misc/NEWS.d/next/Library/2020-01-24-13-24-35.bpo-39082.qKgrq_.rst b/Misc/NEWS.d/next/Library/2020-01-24-13-24-35.bpo-39082.qKgrq_.rst new file mode 100644 index 00000000000000..52c4ee1b33bdae --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-24-13-24-35.bpo-39082.qKgrq_.rst @@ -0,0 +1 @@ +Allow AsyncMock to correctly patch static/class methods From 57e7e5775e52926c983e71e517d900b5560ea2ff Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 27 Jan 2020 08:27:59 -0800 Subject: [PATCH 1132/2163] Ignore NEWS snippets in code coverage stats (GH-18194) (cherry picked from commit 7023288dc500008609e7a4d12ae710c2093c3fc6) Co-authored-by: Nick Coghlan --- .github/codecov.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/codecov.yml b/.github/codecov.yml index 9d97dfbc43f8d0..ea504f48672eac 100644 --- a/.github/codecov.yml +++ b/.github/codecov.yml @@ -5,7 +5,7 @@ codecov: comment: off ignore: - "Doc/**/*" - - "Misc/*" + - "Misc/**/*" - "Mac/**/*" - "PC/**/*" - "PCbuild/**/*" From 7b57b15bd83879ee35f8758a84a7857a9968c145 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 27 Jan 2020 14:35:08 -0800 Subject: [PATCH 1133/2163] bpo-30780: Add IDLE configdialog tests (GH-3592) Expose dialog buttons to test code and complete their test coverage. Complete test coverage for highlights and keys tabs. Co-authored-by: Terry Jan Reedy (cherry picked from commit dd023ad1619b6f1ab313986e8953eea32c18f50c) Co-authored-by: Cheryl Sabella --- Lib/idlelib/NEWS.txt | 3 + Lib/idlelib/configdialog.py | 18 +- Lib/idlelib/idle_test/test_configdialog.py | 158 +++++++++++++++--- .../2020-01-27-16-44-29.bpo-30780.nR80qu.rst | 1 + 4 files changed, 149 insertions(+), 31 deletions(-) create mode 100644 Misc/NEWS.d/next/IDLE/2020-01-27-16-44-29.bpo-30780.nR80qu.rst diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index b7217335fe2814..4d2dd6d7ff2b95 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,9 @@ Released on 2019-12-16? ====================================== +bpo-30780: Add remaining configdialog tests for buttons and +highlights and keys tabs. + bpo-39388: Settings dialog Cancel button cancels pending changes. bpo-39050: Settings dialog Help button again displays help text. diff --git a/Lib/idlelib/configdialog.py b/Lib/idlelib/configdialog.py index 2f95c9ccaa0c57..22359735874d19 100644 --- a/Lib/idlelib/configdialog.py +++ b/Lib/idlelib/configdialog.py @@ -149,17 +149,19 @@ def create_action_buttons(self): else: padding_args = {'padding': (6, 3)} outer = Frame(self, padding=2) - buttons = Frame(outer, padding=2) + buttons_frame = Frame(outer, padding=2) + self.buttons = {} for txt, cmd in ( ('Ok', self.ok), ('Apply', self.apply), ('Cancel', self.cancel), ('Help', self.help)): - Button(buttons, text=txt, command=cmd, takefocus=FALSE, - **padding_args).pack(side=LEFT, padx=5) + self.buttons[txt] = Button(buttons_frame, text=txt, command=cmd, + takefocus=FALSE, **padding_args) + self.buttons[txt].pack(side=LEFT, padx=5) # Add space above buttons. Frame(outer, height=2, borderwidth=0).pack(side=TOP) - buttons.pack(side=BOTTOM) + buttons_frame.pack(side=BOTTOM) return outer def ok(self): @@ -205,7 +207,6 @@ def help(self): Attributes accessed: note - Methods: view_text: Method from textview module. """ @@ -852,6 +853,7 @@ def create_page_highlight(self): text.configure( font=('courier', 12, ''), cursor='hand2', width=1, height=1, takefocus=FALSE, highlightthickness=0, wrap=NONE) + # Prevent perhaps invisible selection of word or slice. text.bind('', lambda e: 'break') text.bind('', lambda e: 'break') string_tags=( @@ -1284,8 +1286,7 @@ def save_new(self, theme_name, theme): theme_name - string, the name of the new theme theme - dictionary containing the new theme """ - if not idleConf.userCfg['highlight'].has_section(theme_name): - idleConf.userCfg['highlight'].add_section(theme_name) + idleConf.userCfg['highlight'].AddSection(theme_name) for element in theme: value = theme[element] idleConf.userCfg['highlight'].SetOption(theme_name, element, value) @@ -1730,8 +1731,7 @@ def save_new_key_set(keyset_name, keyset): keyset_name - string, the name of the new key set keyset - dictionary containing the new keybindings """ - if not idleConf.userCfg['keys'].has_section(keyset_name): - idleConf.userCfg['keys'].add_section(keyset_name) + idleConf.userCfg['keys'].AddSection(keyset_name) for event in keyset: value = keyset[event] idleConf.userCfg['keys'].SetOption(keyset_name, event, value) diff --git a/Lib/idlelib/idle_test/test_configdialog.py b/Lib/idlelib/idle_test/test_configdialog.py index 817a35217bf3cd..1fea6d41df811c 100644 --- a/Lib/idlelib/idle_test/test_configdialog.py +++ b/Lib/idlelib/idle_test/test_configdialog.py @@ -8,7 +8,7 @@ import unittest from unittest import mock from idlelib.idle_test.mock_idle import Func -from tkinter import Tk, StringVar, IntVar, BooleanVar, DISABLED, NORMAL +from tkinter import (Tk, StringVar, IntVar, BooleanVar, DISABLED, NORMAL) from idlelib import config from idlelib.configdialog import idleConf, changes, tracers @@ -30,6 +30,7 @@ keyspage = changes['keys'] extpage = changes['extensions'] + def setUpModule(): global root, dialog idleConf.userCfg = testcfg @@ -37,6 +38,7 @@ def setUpModule(): # root.withdraw() # Comment out, see issue 30870 dialog = configdialog.ConfigDialog(root, 'Test', _utest=True) + def tearDownModule(): global root, dialog idleConf.userCfg = usercfg @@ -48,22 +50,56 @@ def tearDownModule(): root = dialog = None -class DialogTest(unittest.TestCase): +class ConfigDialogTest(unittest.TestCase): + + def test_deactivate_current_config(self): + pass + + def activate_config_changes(self): + pass + - @mock.patch(__name__+'.dialog.destroy', new_callable=Func) - def test_cancel(self, destroy): +class ButtonTest(unittest.TestCase): + + def test_click_ok(self): + d = dialog + apply = d.apply = mock.Mock() + destroy = d.destroy = mock.Mock() + d.buttons['Ok'].invoke() + apply.assert_called_once() + destroy.assert_called_once() + del d.destroy, d.apply + + def test_click_apply(self): + d = dialog + deactivate = d.deactivate_current_config = mock.Mock() + save_ext = d.save_all_changed_extensions = mock.Mock() + activate = d.activate_config_changes = mock.Mock() + d.buttons['Apply'].invoke() + deactivate.assert_called_once() + save_ext.assert_called_once() + activate.assert_called_once() + del d.save_all_changed_extensions + del d.activate_config_changes, d.deactivate_current_config + + def test_click_cancel(self): + d = dialog + d.destroy = Func() changes['main']['something'] = 1 - dialog.cancel() + d.buttons['Cancel'].invoke() self.assertEqual(changes['main'], {}) - self.assertEqual(destroy.called, 1) + self.assertEqual(d.destroy.called, 1) + del d.destroy - @mock.patch('idlelib.configdialog.view_text', new_callable=Func) - def test_help(self, view): + def test_click_help(self): dialog.note.select(dialog.keyspage) - dialog.help() - s = view.kwds['contents'] - self.assertTrue(s.startswith('When you click') and - s.endswith('a different name.\n')) + with mock.patch.object(configdialog, 'view_text', + new_callable=Func) as view: + dialog.buttons['Help'].invoke() + title, contents = view.kwds['title'], view.kwds['contents'] + self.assertEqual(title, 'Help for IDLE preferences') + self.assertTrue(contents.startswith('When you click') and + contents.endswith('a different name.\n')) class FontPageTest(unittest.TestCase): @@ -438,6 +474,48 @@ def click_it(start): eq(d.highlight_target.get(), elem[tag]) eq(d.set_highlight_target.called, count) + def test_highlight_sample_double_click(self): + # Test double click on highlight_sample. + eq = self.assertEqual + d = self.page + + hs = d.highlight_sample + hs.focus_force() + hs.see(1.0) + hs.update_idletasks() + + # Test binding from configdialog. + hs.event_generate('', x=0, y=0) + hs.event_generate('', x=0, y=0) + # Double click is a sequence of two clicks in a row. + for _ in range(2): + hs.event_generate('', x=0, y=0) + hs.event_generate('', x=0, y=0) + + eq(hs.tag_ranges('sel'), ()) + + def test_highlight_sample_b1_motion(self): + # Test button motion on highlight_sample. + eq = self.assertEqual + d = self.page + + hs = d.highlight_sample + hs.focus_force() + hs.see(1.0) + hs.update_idletasks() + + x, y, dx, dy, offset = hs.dlineinfo('1.0') + + # Test binding from configdialog. + hs.event_generate('') + hs.event_generate('') + hs.event_generate('', x=x, y=y) + hs.event_generate('', x=x, y=y) + hs.event_generate('', x=dx, y=dy) + hs.event_generate('', x=dx, y=dy) + + eq(hs.tag_ranges('sel'), ()) + def test_set_theme_type(self): eq = self.assertEqual d = self.page @@ -666,8 +744,13 @@ def test_delete_custom(self): idleConf.userCfg['highlight'].SetOption(theme_name, 'name', 'value') highpage[theme_name] = {'option': 'True'} + theme_name2 = 'other theme' + idleConf.userCfg['highlight'].SetOption(theme_name2, 'name', 'value') + highpage[theme_name2] = {'option': 'False'} + # Force custom theme. - d.theme_source.set(False) + d.custom_theme_on.state(('!disabled',)) + d.custom_theme_on.invoke() d.custom_name.set(theme_name) # Cancel deletion. @@ -675,7 +758,7 @@ def test_delete_custom(self): d.button_delete_custom.invoke() eq(yesno.called, 1) eq(highpage[theme_name], {'option': 'True'}) - eq(idleConf.GetSectionList('user', 'highlight'), ['spam theme']) + eq(idleConf.GetSectionList('user', 'highlight'), [theme_name, theme_name2]) eq(dialog.deactivate_current_config.called, 0) eq(dialog.activate_config_changes.called, 0) eq(d.set_theme_type.called, 0) @@ -685,13 +768,26 @@ def test_delete_custom(self): d.button_delete_custom.invoke() eq(yesno.called, 2) self.assertNotIn(theme_name, highpage) - eq(idleConf.GetSectionList('user', 'highlight'), []) - eq(d.custom_theme_on.state(), ('disabled',)) - eq(d.custom_name.get(), '- no custom themes -') + eq(idleConf.GetSectionList('user', 'highlight'), [theme_name2]) + eq(d.custom_theme_on.state(), ()) + eq(d.custom_name.get(), theme_name2) eq(dialog.deactivate_current_config.called, 1) eq(dialog.activate_config_changes.called, 1) eq(d.set_theme_type.called, 1) + # Confirm deletion of second theme - empties list. + d.custom_name.set(theme_name2) + yesno.result = True + d.button_delete_custom.invoke() + eq(yesno.called, 3) + self.assertNotIn(theme_name, highpage) + eq(idleConf.GetSectionList('user', 'highlight'), []) + eq(d.custom_theme_on.state(), ('disabled',)) + eq(d.custom_name.get(), '- no custom themes -') + eq(dialog.deactivate_current_config.called, 2) + eq(dialog.activate_config_changes.called, 2) + eq(d.set_theme_type.called, 2) + del dialog.activate_config_changes, dialog.deactivate_current_config del d.askyesno @@ -1059,8 +1155,13 @@ def test_delete_custom_keys(self): idleConf.userCfg['keys'].SetOption(keyset_name, 'name', 'value') keyspage[keyset_name] = {'option': 'True'} + keyset_name2 = 'other key set' + idleConf.userCfg['keys'].SetOption(keyset_name2, 'name', 'value') + keyspage[keyset_name2] = {'option': 'False'} + # Force custom keyset. - d.keyset_source.set(False) + d.custom_keyset_on.state(('!disabled',)) + d.custom_keyset_on.invoke() d.custom_name.set(keyset_name) # Cancel deletion. @@ -1068,7 +1169,7 @@ def test_delete_custom_keys(self): d.button_delete_custom_keys.invoke() eq(yesno.called, 1) eq(keyspage[keyset_name], {'option': 'True'}) - eq(idleConf.GetSectionList('user', 'keys'), ['spam key set']) + eq(idleConf.GetSectionList('user', 'keys'), [keyset_name, keyset_name2]) eq(dialog.deactivate_current_config.called, 0) eq(dialog.activate_config_changes.called, 0) eq(d.set_keys_type.called, 0) @@ -1078,13 +1179,26 @@ def test_delete_custom_keys(self): d.button_delete_custom_keys.invoke() eq(yesno.called, 2) self.assertNotIn(keyset_name, keyspage) - eq(idleConf.GetSectionList('user', 'keys'), []) - eq(d.custom_keyset_on.state(), ('disabled',)) - eq(d.custom_name.get(), '- no custom keys -') + eq(idleConf.GetSectionList('user', 'keys'), [keyset_name2]) + eq(d.custom_keyset_on.state(), ()) + eq(d.custom_name.get(), keyset_name2) eq(dialog.deactivate_current_config.called, 1) eq(dialog.activate_config_changes.called, 1) eq(d.set_keys_type.called, 1) + # Confirm deletion of second keyset - empties list. + d.custom_name.set(keyset_name2) + yesno.result = True + d.button_delete_custom_keys.invoke() + eq(yesno.called, 3) + self.assertNotIn(keyset_name, keyspage) + eq(idleConf.GetSectionList('user', 'keys'), []) + eq(d.custom_keyset_on.state(), ('disabled',)) + eq(d.custom_name.get(), '- no custom keys -') + eq(dialog.deactivate_current_config.called, 2) + eq(dialog.activate_config_changes.called, 2) + eq(d.set_keys_type.called, 2) + del dialog.activate_config_changes, dialog.deactivate_current_config del d.askyesno diff --git a/Misc/NEWS.d/next/IDLE/2020-01-27-16-44-29.bpo-30780.nR80qu.rst b/Misc/NEWS.d/next/IDLE/2020-01-27-16-44-29.bpo-30780.nR80qu.rst new file mode 100644 index 00000000000000..2f65a00a5af3b4 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2020-01-27-16-44-29.bpo-30780.nR80qu.rst @@ -0,0 +1 @@ +Add remaining configdialog tests for buttons and highlights and keys tabs. From cf9d00554715febf21cf94950da4f42723b80498 Mon Sep 17 00:00:00 2001 From: mbarkhau Date: Mon, 27 Jan 2020 23:46:29 +0000 Subject: [PATCH 1134/2163] [3.8] bpo-39390 shutil: fix argument types for ignore callback (GH-18122) --- Doc/whatsnew/3.8.rst | 7 ++++ Lib/shutil.py | 2 +- Lib/test/test_shutil.py | 42 +++++++++++++++++++ .../2020-01-23-21-34-29.bpo-39390.D2tSXk.rst | 2 + 4 files changed, 52 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2020-01-23-21-34-29.bpo-39390.D2tSXk.rst diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index 7bc91a9fc19bf5..007e3228f9ae6d 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -2216,3 +2216,10 @@ because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more details, see the documentation for ``loop.create_datagram_endpoint()``. (Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in :issue:`37228`.) + +Notable changes in Python 3.8.2 +=============================== + +Fixed a regression with the ``ignore`` callback of :func:`shutil.copytree`. +The argument types are now str and List[str] again. +(Contributed by Manuel Barkhau and Giampaolo Rodola in :issue:`39390`.) diff --git a/Lib/shutil.py b/Lib/shutil.py index 755ce392e6d9ae..cde7b860050a26 100644 --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -442,7 +442,7 @@ def _ignore_patterns(path, names): def _copytree(entries, src, dst, symlinks, ignore, copy_function, ignore_dangling_symlinks, dirs_exist_ok=False): if ignore is not None: - ignored_names = ignore(src, {x.name for x in entries}) + ignored_names = ignore(os.fspath(src), [x.name for x in entries]) else: ignored_names = set() diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py index 286e333a8aa9cd..bcb7e498e198e2 100644 --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -880,6 +880,48 @@ def _filter(src, names): shutil.rmtree(src_dir) shutil.rmtree(os.path.dirname(dst_dir)) + def test_copytree_arg_types_of_ignore(self): + join = os.path.join + exists = os.path.exists + + tmp_dir = self.mkdtemp() + src_dir = join(tmp_dir, "source") + + os.mkdir(join(src_dir)) + os.mkdir(join(src_dir, 'test_dir')) + os.mkdir(os.path.join(src_dir, 'test_dir', 'subdir')) + write_file((src_dir, 'test_dir', 'subdir', 'test.txt'), '456') + + invokations = [] + + def _ignore(src, names): + invokations.append(src) + self.assertIsInstance(src, str) + self.assertIsInstance(names, list) + self.assertEqual(len(names), len(set(names))) + for name in names: + self.assertIsInstance(name, str) + return [] + + dst_dir = join(self.mkdtemp(), 'destination') + shutil.copytree(src_dir, dst_dir, ignore=_ignore) + self.assertTrue(exists(join(dst_dir, 'test_dir', 'subdir', + 'test.txt'))) + + dst_dir = join(self.mkdtemp(), 'destination') + shutil.copytree(pathlib.Path(src_dir), dst_dir, ignore=_ignore) + self.assertTrue(exists(join(dst_dir, 'test_dir', 'subdir', + 'test.txt'))) + + dst_dir = join(self.mkdtemp(), 'destination') + src_dir_entry = list(os.scandir(tmp_dir))[0] + self.assertIsInstance(src_dir_entry, os.DirEntry) + shutil.copytree(src_dir_entry, dst_dir, ignore=_ignore) + self.assertTrue(exists(join(dst_dir, 'test_dir', 'subdir', + 'test.txt'))) + + self.assertEqual(len(invokations), 9) + def test_copytree_retains_permissions(self): tmp_dir = tempfile.mkdtemp() src_dir = os.path.join(tmp_dir, 'source') diff --git a/Misc/NEWS.d/next/Library/2020-01-23-21-34-29.bpo-39390.D2tSXk.rst b/Misc/NEWS.d/next/Library/2020-01-23-21-34-29.bpo-39390.D2tSXk.rst new file mode 100644 index 00000000000000..ffa961ea4cd22c --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-23-21-34-29.bpo-39390.D2tSXk.rst @@ -0,0 +1,2 @@ +Fixed a regression with the `ignore` callback of :func:`shutil.copytree`. +The argument types are now str and List[str] again. From b487a8ed5bd267d62a05c3cab7def6b1f36999ea Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 27 Jan 2020 15:47:08 -0800 Subject: [PATCH 1135/2163] bpo-39392: Turtle overlap fill depends on OS (GH-18223) Whether or not overlap regions for self-intersecting polygons or multiple shapes are filled depends on the operating system graphics, typeof overlap, and number of overlaps. (cherry picked from commit 2824c45a0a020f12f27da7e7162e8636c21bf869) Co-authored-by: Terry Jan Reedy --- Doc/library/turtle.rst | 5 +++++ .../Documentation/2020-01-27-18-18-42.bpo-39392.oiqcLO.rst | 1 + 2 files changed, 6 insertions(+) create mode 100644 Misc/NEWS.d/next/Documentation/2020-01-27-18-18-42.bpo-39392.oiqcLO.rst diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index 7f9f0c34386799..fed85045435b1b 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -1051,6 +1051,11 @@ Filling Fill the shape drawn after the last call to :func:`begin_fill`. + Whether or not overlap regions for self-intersecting polygons + or multiple shapes are filled depends on the operating system graphics, + type of overlap, and number of overlaps. For example, the Turtle star + above may be either all yellow or have some white regions. + .. doctest:: :skipif: _tkinter is None diff --git a/Misc/NEWS.d/next/Documentation/2020-01-27-18-18-42.bpo-39392.oiqcLO.rst b/Misc/NEWS.d/next/Documentation/2020-01-27-18-18-42.bpo-39392.oiqcLO.rst new file mode 100644 index 00000000000000..715874981f7356 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-01-27-18-18-42.bpo-39392.oiqcLO.rst @@ -0,0 +1 @@ +Explain that when filling with turtle, overlap regions may be left unfilled. From 41f4dc3bcf30cb8362a062a26818311c704ea89f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 27 Jan 2020 19:40:14 -0800 Subject: [PATCH 1136/2163] bpo-36018: Minor fixes to the NormalDist() examples and recipes. (GH-18226) (GH-18227) * Change the source for the SAT data to a primary source. * Fix typo in the standard deviation * Clarify that the binomial probabalities are just for the Python room. (cherry picked from commit 01bf2196d842fc20667c5336e0a7a77eb4fdc25c) Co-authored-by: Raymond Hettinger Co-authored-by: Raymond Hettinger --- Doc/library/statistics.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst index 09b02cabf21f8e..026f4aa462d3d7 100644 --- a/Doc/library/statistics.rst +++ b/Doc/library/statistics.rst @@ -734,10 +734,10 @@ of applications in statistics. :class:`NormalDist` readily solves classic probability problems. For example, given `historical data for SAT exams -`_ showing that scores -are normally distributed with a mean of 1060 and a standard deviation of 192, -determine the percentage of students with test scores between 1100 and -1200, after rounding to the nearest whole number: +`_ showing +that scores are normally distributed with a mean of 1060 and a standard +deviation of 195, determine the percentage of students with test scores +between 1100 and 1200, after rounding to the nearest whole number: .. doctest:: @@ -781,7 +781,7 @@ For example, an open source conference has 750 attendees and two rooms with a 500 person capacity. There is a talk about Python and another about Ruby. In previous conferences, 65% of the attendees preferred to listen to Python talks. Assuming the population preferences haven't changed, what is the -probability that the rooms will stay within their capacity limits? +probability that the Python room will stay within its capacity limits? .. doctest:: From 595b516965ee88b817dc1b3d7713e1f3f263634d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 28 Jan 2020 01:59:43 -0800 Subject: [PATCH 1137/2163] bpo-38883: Don't use POSIX `$HOME` in `pathlib.Path.home/expanduser` on Windows (GH-17961) In bpo-36264 os.path.expanduser was changed to ignore HOME on Windows. Path.expanduser/home still honored HOME despite being documented as behaving the same as os.path.expanduser. This makes them also ignore HOME so that both implementations behave the same way again. (cherry picked from commit c45a2aa9e255b5c7c211faa79f6b23895b64ab27) Co-authored-by: Christoph Reiter --- Lib/pathlib.py | 4 +--- Lib/test/test_pathlib.py | 22 ++++++++++++------- .../2020-01-11-22-53-55.bpo-38883.X7FRaN.rst | 5 +++++ 3 files changed, 20 insertions(+), 11 deletions(-) create mode 100644 Misc/NEWS.d/next/Windows/2020-01-11-22-53-55.bpo-38883.X7FRaN.rst diff --git a/Lib/pathlib.py b/Lib/pathlib.py index 8ed3c883c60067..015370a860e656 100644 --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -253,9 +253,7 @@ def make_uri(self, path): return 'file:' + urlquote_from_bytes(path.as_posix().encode('utf-8')) def gethomedir(self, username): - if 'HOME' in os.environ: - userhome = os.environ['HOME'] - elif 'USERPROFILE' in os.environ: + if 'USERPROFILE' in os.environ: userhome = os.environ['USERPROFILE'] elif 'HOMEPATH' in os.environ: try: diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index b127343982ca9a..97fc5d8ad78382 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -1296,8 +1296,16 @@ def _test_home(self, p): self.assertTrue(p.is_absolute()) def test_home(self): - p = self.cls.home() - self._test_home(p) + with support.EnvironmentVarGuard() as env: + self._test_home(self.cls.home()) + + env.clear() + env['USERPROFILE'] = os.path.join(BASE, 'userprofile') + self._test_home(self.cls.home()) + + # bpo-38883: ignore `HOME` when set on windows + env['HOME'] = os.path.join(BASE, 'home') + self._test_home(self.cls.home()) def test_samefile(self): fileA_path = os.path.join(BASE, 'fileA') @@ -2348,12 +2356,6 @@ def check(): self.assertEqual(p5.expanduser(), p5) self.assertEqual(p6.expanduser(), p6) - # Test the first lookup key in the env vars. - env['HOME'] = 'C:\\Users\\alice' - check() - - # Test that HOMEPATH is available instead. - env.pop('HOME', None) env['HOMEPATH'] = 'C:\\Users\\alice' check() @@ -2366,6 +2368,10 @@ def check(): env['USERPROFILE'] = 'C:\\Users\\alice' check() + # bpo-38883: ignore `HOME` when set on windows + env['HOME'] = 'C:\\Users\\eve' + check() + class CompatiblePathTest(unittest.TestCase): """ diff --git a/Misc/NEWS.d/next/Windows/2020-01-11-22-53-55.bpo-38883.X7FRaN.rst b/Misc/NEWS.d/next/Windows/2020-01-11-22-53-55.bpo-38883.X7FRaN.rst new file mode 100644 index 00000000000000..c552e850a36840 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-01-11-22-53-55.bpo-38883.X7FRaN.rst @@ -0,0 +1,5 @@ +:meth:`~pathlib.Path.home()` and :meth:`~pathlib.Path.expanduser()` on Windows +now prefer :envvar:`USERPROFILE` and no longer use :envvar:`HOME`, which is not +normally set for regular user accounts. This makes them again behave like +:func:`os.path.expanduser`, which was changed to ignore :envvar:`HOME` in 3.8, +see :issue:`36264`. From 46735c7e101753769e31c69a36d0030bb056a162 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 28 Jan 2020 02:00:16 -0800 Subject: [PATCH 1138/2163] bpo-39393: Misleading error message on dependent DLL resolution failure (GH-18093) (cherry picked from commit 13c1c3556f2c12d0be2af890fabfbf44280b845c) Co-authored-by: Zackery Spytz --- .../next/Windows/2020-01-20-23-42-53.bpo-39393.gWlJDG.rst | 2 ++ Modules/_ctypes/callproc.c | 5 +++-- 2 files changed, 5 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Windows/2020-01-20-23-42-53.bpo-39393.gWlJDG.rst diff --git a/Misc/NEWS.d/next/Windows/2020-01-20-23-42-53.bpo-39393.gWlJDG.rst b/Misc/NEWS.d/next/Windows/2020-01-20-23-42-53.bpo-39393.gWlJDG.rst new file mode 100644 index 00000000000000..025b7e96a6e743 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-01-20-23-42-53.bpo-39393.gWlJDG.rst @@ -0,0 +1,2 @@ +Improve the error message when attempting to load a DLL with unresolved +dependencies. diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c index 77492e650ea484..4027bdb622750c 100644 --- a/Modules/_ctypes/callproc.c +++ b/Modules/_ctypes/callproc.c @@ -1311,8 +1311,9 @@ static PyObject *load_library(PyObject *self, PyObject *args) if (err == ERROR_MOD_NOT_FOUND) { PyErr_Format(PyExc_FileNotFoundError, - ("Could not find module '%.500S'. Try using " - "the full path with constructor syntax."), + ("Could not find module '%.500S' (or one of its " + "dependencies). Try using the full path with " + "constructor syntax."), nameobj); return NULL; } else if (err) { From cee5da861091cdaca99f0636af7ab8c4b0274e5f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 28 Jan 2020 02:18:09 -0800 Subject: [PATCH 1139/2163] bpo-39287: Doc: Add UTF-8 mode section in using/windows. (GH-17935) Co-Authored-By: Kyle Stanley (cherry picked from commit 148610d88a2785751ed435a4e60f07a9f1bc50a6) Co-authored-by: Inada Naoki --- Doc/using/cmdline.rst | 2 -- Doc/using/windows.rst | 44 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index c30f6fc7cfd87c..53206c7b546492 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -936,8 +936,6 @@ conflict. Also available as the :option:`-X` ``utf8`` option. - .. availability:: \*nix. - .. versionadded:: 3.7 See :pep:`540` for more details. diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index 9dc3e796cc2110..636f48dfb012b0 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -602,6 +602,50 @@ existed):: C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Python 3.8 +.. _win-utf8-mode: + +UTF-8 mode +========== + +.. versionadded:: 3.7 + +Windows still uses legacy encodings for the system encoding (the ANSI Code +Page). Python uses it for the default encoding of text files (e.g. +:func:`locale.getpreferredencoding`). + +This may cause issues because UTF-8 is widely used on the internet +and most Unix systems, including WSL (Windows Subsystem for Linux). + +You can use UTF-8 mode to change the default text encoding to UTF-8. +You can enable UTF-8 mode via the ``-X utf8`` command line option, or +the ``PYTHONUTF8=1`` environment variable. See :envvar:`PYTHONUTF8` for +enabling UTF-8 mode, and :ref:`setting-envvars` for how to modify +environment variables. + +When UTF-8 mode is enabled: + +* :func:`locale.getpreferredencoding` returns ``'UTF-8'`` instead of + the system encoding. This function is used for the default text + encoding in many places, including :func:`open`, :class:`Popen`, + :meth:`Path.read_text`, etc. +* :data:`sys.stdin`, :data:`sys.stdout`, and :data:`sys.stderr` + all use UTF-8 as their text encoding. +* You can still use the system encoding via the "mbcs" codec. + +Note that adding ``PYTHONUTF8=1`` to the default environment variables +will affect all Python 3.7+ applications on your system. +If you have any Python 3.7+ applications which rely on the legacy +system encoding, it is recommended to set the environment variable +temporarily or use the ``-X utf8`` command line option. + +.. note:: + Even when UTF-8 mode is disabled, Python uses UTF-8 by default + on Windows for: + + * Console I/O including standard I/O (see :pep:`528` for details). + * The filesystem encoding (see :pep:`529` for details). + + .. _launcher: Python Launcher for Windows From dbb37aac1428ddcba784d9ddc77c1708c391aa80 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 28 Jan 2020 02:52:47 -0800 Subject: [PATCH 1140/2163] bpo-39439: Fix multiprocessing spawn path in a venv on Windows (GH-18158) (cherry picked from commit 0be3246d4f9c8eddcd55491901d95b09fe163f15) Co-authored-by: Adam Meily --- Lib/multiprocessing/spawn.py | 2 +- .../next/Windows/2020-01-24-03-15-05.bpo-39439.sFxGfR.rst | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Windows/2020-01-24-03-15-05.bpo-39439.sFxGfR.rst diff --git a/Lib/multiprocessing/spawn.py b/Lib/multiprocessing/spawn.py index 075f3455478b52..7cc129e2610761 100644 --- a/Lib/multiprocessing/spawn.py +++ b/Lib/multiprocessing/spawn.py @@ -36,7 +36,7 @@ if WINSERVICE: _python_exe = os.path.join(sys.exec_prefix, 'python.exe') else: - _python_exe = sys._base_executable + _python_exe = sys.executable def set_executable(exe): global _python_exe diff --git a/Misc/NEWS.d/next/Windows/2020-01-24-03-15-05.bpo-39439.sFxGfR.rst b/Misc/NEWS.d/next/Windows/2020-01-24-03-15-05.bpo-39439.sFxGfR.rst new file mode 100644 index 00000000000000..d677c4c3e02d50 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-01-24-03-15-05.bpo-39439.sFxGfR.rst @@ -0,0 +1 @@ +Honor the Python path when a virtualenv is active on Windows. \ No newline at end of file From 526523c19322169a7f7507d9da291053df979412 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 29 Jan 2020 03:29:35 -0800 Subject: [PATCH 1141/2163] bpo-39153: Clarify C API *SetItem refcounting semantics (GH-18220) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Some of the *SetItem methods in the C API steal a reference to the given value. This annotates the better behaved ones to assure the reader that these are not the ones with the inconsistent behaviour. * 📜🤖 Added by blurb_it. * make docs consistent with signature Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> (cherry picked from commit e1e80002e28e1055f399a20918c49d50d093709e) Co-authored-by: Joannah Nanjekye <33177550+nanjekyejoannah@users.noreply.github.com> --- Doc/c-api/dict.rst | 9 +++++---- Doc/c-api/mapping.rst | 3 ++- Doc/c-api/object.rst | 3 ++- .../2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst | 5 +++++ 4 files changed, 14 insertions(+), 6 deletions(-) create mode 100644 Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst index e7922dc0c73f2a..e48c11d336b8ce 100644 --- a/Doc/c-api/dict.rst +++ b/Doc/c-api/dict.rst @@ -62,19 +62,20 @@ Dictionary Objects .. c:function:: int PyDict_SetItem(PyObject *p, PyObject *key, PyObject *val) - Insert *value* into the dictionary *p* with a key of *key*. *key* must be + Insert *val* into the dictionary *p* with a key of *key*. *key* must be :term:`hashable`; if it isn't, :exc:`TypeError` will be raised. Return - ``0`` on success or ``-1`` on failure. + ``0`` on success or ``-1`` on failure. This function *does not* steal a + reference to *val*. .. c:function:: int PyDict_SetItemString(PyObject *p, const char *key, PyObject *val) .. index:: single: PyUnicode_FromString() - Insert *value* into the dictionary *p* using *key* as a key. *key* should + Insert *val* into the dictionary *p* using *key* as a key. *key* should be a :c:type:`const char\*`. The key object is created using ``PyUnicode_FromString(key)``. Return ``0`` on success or ``-1`` on - failure. + failure. This function *does not* steal a reference to *val*. .. c:function:: int PyDict_DelItem(PyObject *p, PyObject *key) diff --git a/Doc/c-api/mapping.rst b/Doc/c-api/mapping.rst index 6a80b033b651ec..682160d1475c1c 100644 --- a/Doc/c-api/mapping.rst +++ b/Doc/c-api/mapping.rst @@ -37,7 +37,8 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and Map the string *key* to the value *v* in object *o*. Returns ``-1`` on failure. This is the equivalent of the Python statement ``o[key] = v``. - See also :c:func:`PyObject_SetItem`. + See also :c:func:`PyObject_SetItem`. This function *does not* steal a + reference to *v*. .. c:function:: int PyMapping_DelItem(PyObject *o, PyObject *key) diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst index 404a98fb417bb1..db815ae288b869 100644 --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -503,7 +503,8 @@ Object Protocol Map the object *key* to the value *v*. Raise an exception and return ``-1`` on failure; return ``0`` on success. This is the - equivalent of the Python statement ``o[key] = v``. + equivalent of the Python statement ``o[key] = v``. This function *does + not* steal a reference to *v*. .. c:function:: int PyObject_DelItem(PyObject *o, PyObject *key) diff --git a/Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst b/Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst new file mode 100644 index 00000000000000..95be00b4b777f6 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst @@ -0,0 +1,5 @@ +Clarify refcounting semantics for the following functions: +- PyObject_SetItem +- PyMapping_SetItemString +- PyDict_SetItem +- PyDict_SetItemString \ No newline at end of file From d1fa90584f5f4bb0c47743d88a6520ba40e846a0 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 29 Jan 2020 05:17:40 -0800 Subject: [PATCH 1142/2163] Doc: Fix external links to functional programming tutorial. (GH-18249) (cherry picked from commit 35eac4500a8bd89b087407f59ba337343b22d403) Co-authored-by: Julien Palard --- Doc/howto/functional.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index f8f2aac70f9b06..74e861480d2ff8 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -1229,9 +1229,9 @@ Text Processing". Mertz also wrote a 3-part series of articles on functional programming for IBM's DeveloperWorks site; see -`part 1 `__, -`part 2 `__, and -`part 3 `__, +`part 1 `__, +`part 2 `__, and +`part 3 `__, Python documentation From 2b675f0c8fd96f61977f6dc636f44fbd5587b6b3 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 29 Jan 2020 06:41:49 -0800 Subject: [PATCH 1143/2163] bpo-39460: Fix test_zipfile.test_add_file_after_2107() (GH-18247) XFS filesystem is limited to 32-bit timestamp, but the utimensat() syscall doesn't fail. Moreover, there is a VFS bug which returns a cached timestamp which is different than the value on disk. https://bugzilla.redhat.com/show_bug.cgi?id=1795576 https://bugs.python.org/issue39460GH-msg360952 (cherry picked from commit 3cb49b62e61208efcefbc04414e769fc173f205d) Co-authored-by: Victor Stinner --- Lib/test/test_zipfile.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py index f03c044eae3a60..0737c343cbf5fc 100644 --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -602,6 +602,18 @@ def test_add_file_after_2107(self): except OverflowError: self.skipTest('Host fs cannot set timestamp to required value.') + mtime_ns = os.stat(TESTFN).st_mtime_ns + if mtime_ns != (4386268800 * 10**9): + # XFS filesystem is limited to 32-bit timestamp, but the syscall + # didn't fail. Moreover, there is a VFS bug which returns + # a cached timestamp which is different than the value on disk. + # + # Test st_mtime_ns rather than st_mtime to avoid rounding issues. + # + # https://bugzilla.redhat.com/show_bug.cgi?id=1795576 + # https://bugs.python.org/issue39460#msg360952 + self.skipTest(f"Linux VFS/XFS kernel bug detected: {mtime_ns=}") + with zipfile.ZipFile(TESTFN2, "w") as zipfp: self.assertRaises(struct.error, zipfp.write, TESTFN) From 696d2324cf2a54e20e8d6a6739fa97ba815a8be9 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 29 Jan 2020 08:15:36 -0800 Subject: [PATCH 1144/2163] bpo-39485: fix corner-case in method-detection of mock (GH-18255) Replace check for whether something is a method in the mock module. The previous version fails on PyPy, because there no method wrappers exist (everything looks like a regular Python-defined function). Thus the isinstance(getattr(result, '__get__', None), MethodWrapperTypes) check returns True for any descriptor, not just methods. This condition could also return erroneously True in CPython for C-defined descriptors. Instead to decide whether something is a method, just check directly whether it's a function defined on the class. This passes all tests on CPython and fixes the bug on PyPy. (cherry picked from commit a327677905956ae0b239ff430a1346dfe265709e) Co-authored-by: Carl Friedrich Bolz-Tereick Co-authored-by: Carl Friedrich Bolz-Tereick --- Lib/unittest/mock.py | 6 +----- .../next/Library/2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst | 3 +++ 2 files changed, 4 insertions(+), 5 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py index 66ace80cb5f9c3..204b3e7789b333 100644 --- a/Lib/unittest/mock.py +++ b/Lib/unittest/mock.py @@ -2715,7 +2715,7 @@ def _must_skip(spec, entry, is_type): continue if isinstance(result, (staticmethod, classmethod)): return False - elif isinstance(getattr(result, '__get__', None), MethodWrapperTypes): + elif isinstance(result, FunctionTypes): # Normal method => skip if looked up on type # (if looked up on instance, self is already skipped) return is_type @@ -2745,10 +2745,6 @@ def __init__(self, spec, spec_set=False, parent=None, type(ANY.__eq__), ) -MethodWrapperTypes = ( - type(ANY.__eq__.__get__), -) - file_spec = None diff --git a/Misc/NEWS.d/next/Library/2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst b/Misc/NEWS.d/next/Library/2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst new file mode 100644 index 00000000000000..f62c31fc686ad8 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst @@ -0,0 +1,3 @@ +Fix a bug in :func:`unittest.mock.create_autospec` that would complain about +the wrong number of arguments for custom descriptors defined in an extension +module returning functions. From 58076df0c59677111dc77b72852cb2a313a2ef91 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 29 Jan 2020 21:42:38 -0800 Subject: [PATCH 1145/2163] bpo-39493: Fix definition of IO.closed in typing.py (GH-18265) (cherry picked from commit 2e6569b6692298fcc9aae0df3eb3181adb2a5099) Co-authored-by: Shantanu --- Lib/typing.py | 1 + .../NEWS.d/next/Library/2020-01-30-01-13-19.bpo-39493.CbFRi7.rst | 1 + 2 files changed, 2 insertions(+) create mode 100644 Misc/NEWS.d/next/Library/2020-01-30-01-13-19.bpo-39493.CbFRi7.rst diff --git a/Lib/typing.py b/Lib/typing.py index 69f6d9873490a9..83d310f3c0dd33 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -1848,6 +1848,7 @@ def name(self) -> str: def close(self) -> None: pass + @property @abstractmethod def closed(self) -> bool: pass diff --git a/Misc/NEWS.d/next/Library/2020-01-30-01-13-19.bpo-39493.CbFRi7.rst b/Misc/NEWS.d/next/Library/2020-01-30-01-13-19.bpo-39493.CbFRi7.rst new file mode 100644 index 00000000000000..b676629a4414a2 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-30-01-13-19.bpo-39493.CbFRi7.rst @@ -0,0 +1 @@ +Mark ``typing.IO.closed`` as a property \ No newline at end of file From ad4a20b87d79a619ffbdea3f26848780899494e5 Mon Sep 17 00:00:00 2001 From: Steve Dower Date: Thu, 30 Jan 2020 17:18:25 +1100 Subject: [PATCH 1146/2163] [3.8] bpo-39401: Avoid unsafe DLL load on Windows 7 and earlier (GH-18231) (GH-18234) https://bugs.python.org/issue39401 Automerge-Triggered-By: @zooba --- .../next/Security/2020-01-28-20-54-09.bpo-39401.he7h_A.rst | 1 + PC/getpathp.c | 6 ++++-- 2 files changed, 5 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Security/2020-01-28-20-54-09.bpo-39401.he7h_A.rst diff --git a/Misc/NEWS.d/next/Security/2020-01-28-20-54-09.bpo-39401.he7h_A.rst b/Misc/NEWS.d/next/Security/2020-01-28-20-54-09.bpo-39401.he7h_A.rst new file mode 100644 index 00000000000000..5071e126b70d02 --- /dev/null +++ b/Misc/NEWS.d/next/Security/2020-01-28-20-54-09.bpo-39401.he7h_A.rst @@ -0,0 +1 @@ +Avoid unsafe load of ``api-ms-win-core-path-l1-1-0.dll`` at startup on Windows 7. diff --git a/PC/getpathp.c b/PC/getpathp.c index 04f24d986f667c..3747ffb2d89f09 100644 --- a/PC/getpathp.c +++ b/PC/getpathp.c @@ -251,7 +251,8 @@ static void join(wchar_t *buffer, const wchar_t *stuff) { if (_PathCchCombineEx_Initialized == 0) { - HMODULE pathapi = LoadLibraryW(L"api-ms-win-core-path-l1-1-0.dll"); + HMODULE pathapi = LoadLibraryExW(L"api-ms-win-core-path-l1-1-0.dll", NULL, + LOAD_LIBRARY_SEARCH_SYSTEM32); if (pathapi) { _PathCchCombineEx = (PPathCchCombineEx)GetProcAddress(pathapi, "PathCchCombineEx"); } @@ -287,7 +288,8 @@ canonicalize(wchar_t *buffer, const wchar_t *path) } if (_PathCchCanonicalizeEx_Initialized == 0) { - HMODULE pathapi = LoadLibraryW(L"api-ms-win-core-path-l1-1-0.dll"); + HMODULE pathapi = LoadLibraryExW(L"api-ms-win-core-path-l1-1-0.dll", NULL, + LOAD_LIBRARY_SEARCH_SYSTEM32); if (pathapi) { _PathCchCanonicalizeEx = (PPathCchCanonicalizeEx)GetProcAddress(pathapi, "PathCchCanonicalizeEx"); } From b841633cc2d7619cf4a7db108d91b14926450a6e Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 30 Jan 2020 07:05:08 -0800 Subject: [PATCH 1147/2163] bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX (GH-18282) Skip test_zipfile.test_add_file_after_2107() if time.localtime() fails with OverflowError. It is the case on AIX 6.1 for example. (cherry picked from commit c232c9110cfefa0935cbf158e35e91746a8a9361) Co-authored-by: Victor Stinner --- Lib/test/test_zipfile.py | 7 ++++++- .../next/Tests/2020-01-30-15-04-54.bpo-39502.chbpII.rst | 2 ++ 2 files changed, 8 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Tests/2020-01-30-15-04-54.bpo-39502.chbpII.rst diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py index 0737c343cbf5fc..c65de9202c0c4b 100644 --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -597,8 +597,13 @@ def test_add_file_before_1980(self): def test_add_file_after_2107(self): # Set atime and mtime to 2108-12-30 + ts = 4386268800 try: - os.utime(TESTFN, (4386268800, 4386268800)) + time.localtime(ts) + except OverflowError: + self.skipTest(f'time.localtime({ts}) raises OverflowError') + try: + os.utime(TESTFN, (ts, ts)) except OverflowError: self.skipTest('Host fs cannot set timestamp to required value.') diff --git a/Misc/NEWS.d/next/Tests/2020-01-30-15-04-54.bpo-39502.chbpII.rst b/Misc/NEWS.d/next/Tests/2020-01-30-15-04-54.bpo-39502.chbpII.rst new file mode 100644 index 00000000000000..0a13746e34759b --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2020-01-30-15-04-54.bpo-39502.chbpII.rst @@ -0,0 +1,2 @@ +Skip test_zipfile.test_add_file_after_2107() if :func:`time.localtime` fails +with :exc:`OverflowError`. It is the case on AIX 6.1 for example. From 83d3202b92fb4c2fc6df5b035d57f3a1cf715f20 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 30 Jan 2020 18:14:16 -0800 Subject: [PATCH 1148/2163] bpo-38792: Remove IDLE shell calltip before new prompt. (GH-17150) Previously, a calltip might be left after SyntaxError, KeyboardInterrupt, or Shell Restart. Co-authored-by: Terry Jan Reedy Co-authored-by: Tal Einat (cherry picked from commit bfdeaa37b3df7466624c17f9450d2bd1c3d95edf) Co-authored-by: Zackery Spytz --- Lib/idlelib/NEWS.txt | 3 +++ Lib/idlelib/calltip.py | 4 ++-- Lib/idlelib/editor.py | 2 +- Lib/idlelib/pyshell.py | 1 + .../NEWS.d/next/IDLE/2019-11-13-23-51-39.bpo-38792.xhTC5a.rst | 2 ++ 5 files changed, 9 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/IDLE/2019-11-13-23-51-39.bpo-38792.xhTC5a.rst diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 4d2dd6d7ff2b95..1fc9e0f3090f65 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,9 @@ Released on 2019-12-16? ====================================== +bpo-38792: Close a shell calltip if a :exc:`KeyboardInterrupt` +or shell restart occurs. Patch by Zackery Spytz. + bpo-30780: Add remaining configdialog tests for buttons and highlights and keys tabs. diff --git a/Lib/idlelib/calltip.py b/Lib/idlelib/calltip.py index a3dda2678bd4da..2e0db60d476aef 100644 --- a/Lib/idlelib/calltip.py +++ b/Lib/idlelib/calltip.py @@ -33,7 +33,7 @@ def _make_tk_calltip_window(self): # See __init__ for usage return calltip_w.CalltipWindow(self.text) - def _remove_calltip_window(self, event=None): + def remove_calltip_window(self, event=None): if self.active_calltip: self.active_calltip.hidetip() self.active_calltip = None @@ -55,7 +55,7 @@ def refresh_calltip_event(self, event): self.open_calltip(False) def open_calltip(self, evalfuncs): - self._remove_calltip_window() + self.remove_calltip_window() hp = HyperParser(self.editwin, "insert") sur_paren = hp.get_surrounding_brackets('(') diff --git a/Lib/idlelib/editor.py b/Lib/idlelib/editor.py index c9f1a1625ca5ea..04c786dc5234c2 100644 --- a/Lib/idlelib/editor.py +++ b/Lib/idlelib/editor.py @@ -328,7 +328,7 @@ def __init__(self, flist=None, filename=None, key=None, root=None): text.bind("<>", scriptbinding.run_module_event) text.bind("<>", scriptbinding.run_custom_event) text.bind("<>", self.Rstrip(self).do_rstrip) - ctip = self.Calltip(self) + self.ctip = ctip = self.Calltip(self) text.bind("<>", ctip.try_open_calltip_event) #refresh-calltip must come after paren-closed to work right text.bind("<>", ctip.refresh_calltip_event) diff --git a/Lib/idlelib/pyshell.py b/Lib/idlelib/pyshell.py index 065122dec7a73d..d5b310ffd7a9be 100755 --- a/Lib/idlelib/pyshell.py +++ b/Lib/idlelib/pyshell.py @@ -1292,6 +1292,7 @@ def resetoutput(self): self.text.insert("end-1c", "\n") self.text.mark_set("iomark", "end-1c") self.set_line_and_column() + self.ctip.remove_calltip_window() def write(self, s, tags=()): try: diff --git a/Misc/NEWS.d/next/IDLE/2019-11-13-23-51-39.bpo-38792.xhTC5a.rst b/Misc/NEWS.d/next/IDLE/2019-11-13-23-51-39.bpo-38792.xhTC5a.rst new file mode 100644 index 00000000000000..9aa2f0ffddfafd --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2019-11-13-23-51-39.bpo-38792.xhTC5a.rst @@ -0,0 +1,2 @@ +Close an IDLE shell calltip if a :exc:`KeyboardInterrupt` +or shell restart occurs. Patch by Zackery Spytz. From 17236873392533ce0c5d7bbf52cbd61bca171c59 Mon Sep 17 00:00:00 2001 From: Antoine Pitrou Date: Sun, 2 Feb 2020 21:22:57 +0100 Subject: [PATCH 1149/2163] [3.8] bpo-39492: Fix a reference cycle between reducer_override and a Pickler instance (GH-18266) (#18316) https://bugs.python.org/issue39492 Automerge-Triggered-By: @pitrou (cherry picked from commit 0f2f35e) Co-authored-by: Pierre Glaser --- Lib/test/pickletester.py | 24 +++++++++++++++++++ .../2020-01-30-01-14-42.bpo-39492.eTuy0F.rst | 1 + Modules/_pickle.c | 22 +++++++++++++---- 3 files changed, 43 insertions(+), 4 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index c9f374678ae35a..3a8aee4320c65a 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -3497,6 +3497,30 @@ class MyClass: ValueError, 'The reducer just failed'): p.dump(h) + @support.cpython_only + def test_reducer_override_no_reference_cycle(self): + # bpo-39492: reducer_override used to induce a spurious reference cycle + # inside the Pickler object, that could prevent all serialized objects + # from being garbage-collected without explicity invoking gc.collect. + + for proto in range(0, pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + def f(): + pass + + wr = weakref.ref(f) + + bio = io.BytesIO() + p = self.pickler_class(bio, proto) + p.dump(f) + new_f = pickle.loads(bio.getvalue()) + assert new_f == 5 + + del p + del f + + self.assertIsNone(wr()) + class AbstractDispatchTableTests(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst new file mode 100644 index 00000000000000..6e8b715c46365b --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst @@ -0,0 +1 @@ +Fix a reference cycle in the C Pickler that was preventing the garbage collection of deleted, pickled objects. \ No newline at end of file diff --git a/Modules/_pickle.c b/Modules/_pickle.c index 8150bf3b33dc38..9f6e66f70a5462 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -4457,12 +4457,13 @@ static int dump(PicklerObject *self, PyObject *obj) { const char stop_op = STOP; + int status = -1; PyObject *tmp; _Py_IDENTIFIER(reducer_override); if (_PyObject_LookupAttrId((PyObject *)self, &PyId_reducer_override, &tmp) < 0) { - return -1; + goto error; } /* Cache the reducer_override method, if it exists. */ if (tmp != NULL) { @@ -4479,7 +4480,7 @@ dump(PicklerObject *self, PyObject *obj) assert(self->proto >= 0 && self->proto < 256); header[1] = (unsigned char)self->proto; if (_Pickler_Write(self, header, 2) < 0) - return -1; + goto error; if (self->proto >= 4) self->framing = 1; } @@ -4487,9 +4488,22 @@ dump(PicklerObject *self, PyObject *obj) if (save(self, obj, 0) < 0 || _Pickler_Write(self, &stop_op, 1) < 0 || _Pickler_CommitFrame(self) < 0) - return -1; + goto error; + + // Success + status = 0; + + error: self->framing = 0; - return 0; + + /* Break the reference cycle we generated at the beginning this function + * call when setting the reducer_override attribute of the Pickler instance + * to a bound method of the same instance. This is important as the Pickler + * instance holds a reference to each object it has pickled (through its + * memo): thus, these objects wont be garbage-collected as long as the + * Pickler itself is not collected. */ + Py_CLEAR(self->reducer_override); + return status; } /*[clinic input] From 02395fad8e3a35ef00fa31c308693844013a1dd4 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 3 Feb 2020 00:20:41 -0800 Subject: [PATCH 1150/2163] bpo-39450 Stripped whitespace before parsing the docstring in TestCase.shortDescription (GH-18175) (#18323) (cherry picked from commit 032de7324e30c6b44ef272cea3be205a3d768759) Co-authored-by: Steve Cirelli --- Lib/unittest/case.py | 2 +- Lib/unittest/test/test_case.py | 9 +++++++++ .../Library/2020-02-02-14-46-34.bpo-39450.48R274.rst | 2 ++ 3 files changed, 12 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py index b639c64d02a7aa..e5734b6b7a298b 100644 --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -529,7 +529,7 @@ def shortDescription(self): the specified test method's docstring. """ doc = self._testMethodDoc - return doc and doc.split("\n")[0].strip() or None + return doc.strip().split("\n")[0].strip() if doc else None def id(self): diff --git a/Lib/unittest/test/test_case.py b/Lib/unittest/test/test_case.py index c2401c39b917e3..f855c4dc00b316 100644 --- a/Lib/unittest/test/test_case.py +++ b/Lib/unittest/test/test_case.py @@ -610,6 +610,15 @@ def testShortDescriptionWithMultiLineDocstring(self): 'Tests shortDescription() for a method with a longer ' 'docstring.') + def testShortDescriptionWhitespaceTrimming(self): + """ + Tests shortDescription() whitespace is trimmed, so that the first + line of nonwhite-space text becomes the docstring. + """ + self.assertEqual( + self.shortDescription(), + 'Tests shortDescription() whitespace is trimmed, so that the first') + def testAddTypeEqualityFunc(self): class SadSnake(object): """Dummy class for test_addTypeEqualityFunc.""" diff --git a/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst b/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst new file mode 100644 index 00000000000000..55fed519a2d806 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst @@ -0,0 +1,2 @@ +Striped whitespace from docstring before returning it from +:func:`unittest.case.shortDescription`. From db2f3114b2be4f3ee81a3258a979327d539ab51a Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 3 Feb 2020 04:07:19 -0800 Subject: [PATCH 1151/2163] fixes typos in http.client documentation (GH-18300) (cherry picked from commit b94737a4af96b29bd4c025724f671e7bc0f6b6f1) Co-authored-by: James Corbett --- Doc/library/http.client.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index 9cdabc2a03a751..be31c3c07154f4 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -563,8 +563,8 @@ Here is an example session that shows how to ``POST`` requests:: Client side ``HTTP PUT`` requests are very similar to ``POST`` requests. The difference lies only the server side where HTTP server will allow resources to be created via ``PUT`` request. It should be noted that custom HTTP methods -+are also handled in :class:`urllib.request.Request` by sending the appropriate -+method attribute.Here is an example session that shows how to do ``PUT`` +are also handled in :class:`urllib.request.Request` by setting the appropriate +method attribute. Here is an example session that shows how to send a ``PUT`` request using http.client:: >>> # This creates an HTTP message From d01ae1b22330992eadc7b2a0842ead544f7e507d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 3 Feb 2020 09:17:17 -0800 Subject: [PATCH 1152/2163] bpo-38558: Link to further docs from walrus operator mention in tutorial (GH-16973) (cherry picked from commit 5807efd4c396d5718325e21f5a14e324a77ff77c) Co-authored-by: Adorilson Bezerra --- Doc/faq/design.rst | 2 ++ Doc/tutorial/datastructures.rst | 7 ++++--- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst index 81c0f474ac1624..e7921baf242f52 100644 --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -146,6 +146,8 @@ variables and instance variables live in two different namespaces, and you need to tell Python which namespace to use. +.. _why-can-t-i-use-an-assignment-in-an-expression: + Why can't I use an assignment in an expression? ----------------------------------------------- diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst index 2f7afb088f3bbb..0edb73ad736919 100644 --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -676,9 +676,10 @@ to a variable. For example, :: 'Trondheim' Note that in Python, unlike C, assignment inside expressions must be done -explicitly with the walrus operator ``:=``. This avoids a common class of -problems encountered in C programs: typing ``=`` in an expression when ``==`` -was intended. +explicitly with the +:ref:`walrus operator ` ``:=``. +This avoids a common class of problems encountered in C programs: typing ``=`` +in an expression when ``==`` was intended. .. _tut-comparing: From a0389ba84bb2dbee7a59e25163e514059afc166d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 3 Feb 2020 16:50:29 -0800 Subject: [PATCH 1153/2163] Fixes in sorting descriptions (GH-18317) Improvements in listsort.txt and a comment in sortperf.py. Automerge-Triggered-By: @csabella (cherry picked from commit 24e5ad4689de9adc8e4a7d8c08fe400dcea668e6) Co-authored-by: Stefan Pochmann --- Lib/test/sortperf.py | 2 +- Objects/listsort.txt | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/Lib/test/sortperf.py b/Lib/test/sortperf.py index 171e5cef5e29fd..14a9d827ed57c5 100644 --- a/Lib/test/sortperf.py +++ b/Lib/test/sortperf.py @@ -134,7 +134,7 @@ def tabulate(r): L = list(range(half - 1, -1, -1)) L.extend(range(half)) # Force to float, so that the timings are comparable. This is - # significantly faster if we leave tham as ints. + # significantly faster if we leave them as ints. L = list(map(float, L)) doit(L) # !sort print() diff --git a/Objects/listsort.txt b/Objects/listsort.txt index 43fe1574c32393..174777a2658dc6 100644 --- a/Objects/listsort.txt +++ b/Objects/listsort.txt @@ -319,13 +319,13 @@ So merging is always done on two consecutive runs at a time, and in-place, although this may require some temp memory (more on that later). When a run is identified, its base address and length are pushed on a stack -in the MergeState struct. merge_collapse() is then called to see whether it -should merge it with preceding run(s). We would like to delay merging as -long as possible in order to exploit patterns that may come up later, but we -like even more to do merging as soon as possible to exploit that the run just -found is still high in the memory hierarchy. We also can't delay merging -"too long" because it consumes memory to remember the runs that are still -unmerged, and the stack has a fixed size. +in the MergeState struct. merge_collapse() is then called to potentially +merge runs on that stack. We would like to delay merging as long as possible +in order to exploit patterns that may come up later, but we like even more to +do merging as soon as possible to exploit that the run just found is still +high in the memory hierarchy. We also can't delay merging "too long" because +it consumes memory to remember the runs that are still unmerged, and the +stack has a fixed size. What turned out to be a good compromise maintains two invariants on the stack entries, where A, B and C are the lengths of the three rightmost not-yet @@ -739,7 +739,7 @@ slice (leaving off both endpoints) (2**(k-1)-1)+1 through (2**k-1)-1 inclusive = 2**(k-1) through (2**k-1)-1 inclusive, which has (2**k-1)-1 - 2**(k-1) + 1 = 2**k-1 - 2**(k-1) = - 2*2**k-1 - 2**(k-1) = + 2*2**(k-1)-1 - 2**(k-1) = (2-1)*2**(k-1) - 1 = 2**(k-1) - 1 elements. From 9a740b6c7e7a88185d79128b8a1993ac387d5091 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 4 Feb 2020 07:31:19 -0800 Subject: [PATCH 1154/2163] bpo-37224: Improve test__xxsubinterpreters.DestroyTests (GH-18058) Adds an additional assertion check based on a race condition for `test__xxsubinterpreters.DestroyTests.test_still_running` discovered in the bpo issue. https://bugs.python.org/issue37224 (cherry picked from commit f03a8f8d5001963ad5b5b28dbd95497e9cc15596) Co-authored-by: Kyle Stanley --- Lib/test/test__xxsubinterpreters.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/Lib/test/test__xxsubinterpreters.py b/Lib/test/test__xxsubinterpreters.py index 207b5db5d8fb9b..30f8f98acc9dd3 100644 --- a/Lib/test/test__xxsubinterpreters.py +++ b/Lib/test/test__xxsubinterpreters.py @@ -759,7 +759,11 @@ def test_still_running(self): main, = interpreters.list_all() interp = interpreters.create() with _running(interp): - with self.assertRaises(RuntimeError): + self.assertTrue(interpreters.is_running(interp), + msg=f"Interp {interp} should be running before destruction.") + + with self.assertRaises(RuntimeError, + msg=f"Should not be able to destroy interp {interp} while it's still running."): interpreters.destroy(interp) self.assertTrue(interpreters.is_running(interp)) From 30e769382dfb67a68fe8e6bfe8509addb4aa9514 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 4 Feb 2020 13:41:55 -0800 Subject: [PATCH 1155/2163] closes bpo-39510: Fix use-after-free in BufferedReader.readinto() (GH-18295) When called on a closed object, readinto() segfaults on account of a write to a freed buffer: ==220553== Process terminating with default action of signal 11 (SIGSEGV): dumping core ==220553== Access not within mapped region at address 0x2A ==220553== at 0x48408A0: memmove (vg_replace_strmem.c:1272) ==220553== by 0x58DB0C: _buffered_readinto_generic (bufferedio.c:972) ==220553== by 0x58DCBA: _io__Buffered_readinto_impl (bufferedio.c:1053) ==220553== by 0x58DCBA: _io__Buffered_readinto (bufferedio.c.h:253) Reproducer: reader = open ("/dev/zero", "rb") _void = reader.read (42) reader.close () reader.readinto (bytearray (42)) GH-GH-GH- BANG! The problem exists since 2012 when commit dc469454ec added code to free the read buffer on close(). Signed-off-by: Philipp Gesang (cherry picked from commit cb1c0746f277052e45a60d6c436a765e34722821) Co-authored-by: Philipp Gesang --- Lib/test/test_io.py | 5 +++++ .../2020-02-04-10-27-41.bpo-39510.PMIh-f.rst | 1 + Modules/_io/bufferedio.c | 1 + 3 files changed, 7 insertions(+) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-02-04-10-27-41.bpo-39510.PMIh-f.rst diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py index 50459e07929622..bea4342e7f0f9e 100644 --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -735,6 +735,11 @@ def test_read_closed(self): file.seek(0) file.close() self.assertRaises(ValueError, file.read) + with self.open(support.TESTFN, "rb") as f: + file = self.open(f.fileno(), "rb", closefd=False) + self.assertEqual(file.read()[:3], b"egg") + file.close() + self.assertRaises(ValueError, file.readinto, bytearray(1)) def test_no_closefd_with_filename(self): # can't use closefd in combination with a file name diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-02-04-10-27-41.bpo-39510.PMIh-f.rst b/Misc/NEWS.d/next/Core and Builtins/2020-02-04-10-27-41.bpo-39510.PMIh-f.rst new file mode 100644 index 00000000000000..9a38e4ab762287 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-02-04-10-27-41.bpo-39510.PMIh-f.rst @@ -0,0 +1 @@ +Fix segfault in ``readinto()`` method on closed BufferedReader. diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c index 8e8ff97ff8c676..ad7a8c9d264270 100644 --- a/Modules/_io/bufferedio.c +++ b/Modules/_io/bufferedio.c @@ -965,6 +965,7 @@ _buffered_readinto_generic(buffered *self, Py_buffer *buffer, char readinto1) PyObject *res = NULL; CHECK_INITIALIZED(self) + CHECK_CLOSED(self, "readinto of closed file") n = Py_SAFE_DOWNCAST(READAHEAD(self), Py_off_t, Py_ssize_t); if (n > 0) { From 3498ac55bcfc18d698ea605424ec65a6e1457a39 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 4 Feb 2020 16:32:32 -0800 Subject: [PATCH 1156/2163] bpo-39184: Add audit events to command execution functions in os and pty modules (GH-17824) (cherry picked from commit 95f60010219e142a436fae18e1695cbc45407afe) Co-authored-by: Saiyang Gou --- Doc/library/os.rst | 10 ++++ Doc/library/pty.rst | 1 + Lib/pty.py | 2 + .../2020-01-07-00-42-08.bpo-39184.fe7NgK.rst | 1 + Modules/posixmodule.c | 49 ++++++++++++++++--- 5 files changed, 56 insertions(+), 7 deletions(-) create mode 100644 Misc/NEWS.d/next/Security/2020-01-07-00-42-08.bpo-39184.fe7NgK.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 7aadbcf773b6a7..3a828c5e08563a 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -3313,6 +3313,8 @@ to be ignored. you can check whether or not it is available using :data:`os.supports_fd`. If it is unavailable, using it will raise a :exc:`NotImplementedError`. + .. audit-event:: os.exec path,args,env os.execl + .. availability:: Unix, Windows. .. versionadded:: 3.3 @@ -3656,6 +3658,8 @@ written in Python, such as a mail server's external command delivery program. :c:data:`POSIX_SPAWN_SETSCHEDPARAM` and :c:data:`POSIX_SPAWN_SETSCHEDULER` flags. + .. audit-event:: os.posix_spawn path,argv,env os.posix_spawn + .. versionadded:: 3.8 .. availability:: Unix. @@ -3670,6 +3674,8 @@ written in Python, such as a mail server's external command delivery program. for the *executable* file in the list of directories specified by the :envvar:`PATH` environment variable (in the same way as for ``execvp(3)``). + .. audit-event:: os.posix_spawn path,argv,env os.posix_spawnp + .. versionadded:: 3.8 .. availability:: See :func:`posix_spawn` documentation. @@ -3770,6 +3776,8 @@ written in Python, such as a mail server's external command delivery program. L = ['cp', 'index.html', '/dev/null'] os.spawnvpe(os.P_WAIT, 'cp', L, os.environ) + .. audit-event:: os.spawn mode,path,args,env os.spawnl + .. availability:: Unix, Windows. :func:`spawnlp`, :func:`spawnlpe`, :func:`spawnvp` and :func:`spawnvpe` are not available on Windows. :func:`spawnle` and :func:`spawnve` are not thread-safe on Windows; we advise you to use the @@ -3839,6 +3847,8 @@ written in Python, such as a mail server's external command delivery program. function is not resolved until this function is first called. If the function cannot be resolved, :exc:`NotImplementedError` will be raised. + .. audit-event:: os.startfile path,operation os.startfile + .. availability:: Windows. diff --git a/Doc/library/pty.rst b/Doc/library/pty.rst index 12268437d07e98..e85d2e239fdbdb 100644 --- a/Doc/library/pty.rst +++ b/Doc/library/pty.rst @@ -69,6 +69,7 @@ The :mod:`pty` module defines the following functions: *select* throws an error on your platform when passed three empty lists. This is a bug, documented in `issue 26228 `_. + .. audit-event:: pty.spawn argv pty.spawn .. versionchanged:: 3.4 :func:`spawn` now returns the status value from :func:`os.waitpid` diff --git a/Lib/pty.py b/Lib/pty.py index e841f12f3edb9b..a32432041fa1db 100644 --- a/Lib/pty.py +++ b/Lib/pty.py @@ -8,6 +8,7 @@ from select import select import os +import sys import tty __all__ = ["openpty","fork","spawn"] @@ -151,6 +152,7 @@ def spawn(argv, master_read=_read, stdin_read=_read): """Create a spawned process.""" if type(argv) == type(''): argv = (argv,) + sys.audit('pty.spawn', argv) pid, master_fd = fork() if pid == CHILD: os.execlp(argv[0], *argv) diff --git a/Misc/NEWS.d/next/Security/2020-01-07-00-42-08.bpo-39184.fe7NgK.rst b/Misc/NEWS.d/next/Security/2020-01-07-00-42-08.bpo-39184.fe7NgK.rst new file mode 100644 index 00000000000000..1ab5d4d70eec5a --- /dev/null +++ b/Misc/NEWS.d/next/Security/2020-01-07-00-42-08.bpo-39184.fe7NgK.rst @@ -0,0 +1 @@ +Add audit events to command execution functions in os and pty modules. \ No newline at end of file diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 850769fd95eef7..88b47164bca09c 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -5156,6 +5156,12 @@ os_execv_impl(PyObject *module, path_t *path, PyObject *argv) return NULL; } + if (PySys_Audit("os.exec", "OOO", path->object ? path->object : Py_None, + argv, Py_None) < 0) { + free_string_array(argvlist, argc); + return NULL; + } + _Py_BEGIN_SUPPRESS_IPH #ifdef HAVE_WEXECV _wexecv(path->wide, argvlist); @@ -5199,7 +5205,7 @@ os_execve_impl(PyObject *module, path_t *path, PyObject *argv, PyObject *env) if (!PyList_Check(argv) && !PyTuple_Check(argv)) { PyErr_SetString(PyExc_TypeError, "execve: argv must be a tuple or list"); - goto fail; + goto fail_0; } argc = PySequence_Size(argv); if (argc < 1) { @@ -5210,22 +5216,27 @@ os_execve_impl(PyObject *module, path_t *path, PyObject *argv, PyObject *env) if (!PyMapping_Check(env)) { PyErr_SetString(PyExc_TypeError, "execve: environment must be a mapping object"); - goto fail; + goto fail_0; } argvlist = parse_arglist(argv, &argc); if (argvlist == NULL) { - goto fail; + goto fail_0; } if (!argvlist[0][0]) { PyErr_SetString(PyExc_ValueError, "execve: argv first element cannot be empty"); - goto fail; + goto fail_0; } envlist = parse_envlist(env, &envc); if (envlist == NULL) - goto fail; + goto fail_0; + + if (PySys_Audit("os.exec", "OOO", path->object ? path->object : Py_None, + argv, env) < 0) { + goto fail_1; + } _Py_BEGIN_SUPPRESS_IPH #ifdef HAVE_FEXECVE @@ -5243,9 +5254,9 @@ os_execve_impl(PyObject *module, path_t *path, PyObject *argv, PyObject *env) /* If we get here it's definitely an error */ posix_path_error(path); - + fail_1: free_string_array(envlist, envc); - fail: + fail_0: if (argvlist) free_string_array(argvlist, argc); return NULL; @@ -5576,6 +5587,11 @@ py_posix_spawn(int use_posix_spawnp, PyObject *module, path_t *path, PyObject *a } attrp = &attr; + if (PySys_Audit("os.posix_spawn", "OOO", + path->object ? path->object : Py_None, argv, env) < 0) { + goto exit; + } + _Py_BEGIN_SUPPRESS_IPH #ifdef HAVE_POSIX_SPAWNP if (use_posix_spawnp) { @@ -5816,6 +5832,13 @@ os_spawnv_impl(PyObject *module, int mode, path_t *path, PyObject *argv) mode = _P_OVERLAY; #endif + if (PySys_Audit("os.spawn", "iOOO", mode, + path->object ? path->object : Py_None, argv, + Py_None) < 0) { + free_string_array(argvlist, argc); + return NULL; + } + Py_BEGIN_ALLOW_THREADS _Py_BEGIN_SUPPRESS_IPH #ifdef HAVE_WSPAWNV @@ -5925,6 +5948,11 @@ os_spawnve_impl(PyObject *module, int mode, path_t *path, PyObject *argv, mode = _P_OVERLAY; #endif + if (PySys_Audit("os.spawn", "iOOO", mode, + path->object ? path->object : Py_None, argv, env) < 0) { + goto fail_2; + } + Py_BEGIN_ALLOW_THREADS _Py_BEGIN_SUPPRESS_IPH #ifdef HAVE_WSPAWNV @@ -5943,6 +5971,7 @@ os_spawnve_impl(PyObject *module, int mode, path_t *path, PyObject *argv, else res = Py_BuildValue(_Py_PARSE_INTPTR, spawnval); + fail_2: while (--envc >= 0) PyMem_DEL(envlist[envc]); PyMem_DEL(envlist); @@ -11608,6 +11637,12 @@ os_startfile_impl(PyObject *module, path_t *filepath, "startfile not available on this platform"); } + if (PySys_Audit("os.startfile", "Ou", + filepath->object ? filepath->object : Py_None, + operation) < 0) { + return NULL; + } + Py_BEGIN_ALLOW_THREADS rc = Py_ShellExecuteW((HWND)0, operation, filepath->wide, NULL, NULL, SW_SHOWNORMAL); From 6470a7643018b357adcd88f77ae2a196f41cb351 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 4 Feb 2020 16:48:01 -0800 Subject: [PATCH 1157/2163] bpo-39185 Add the d[etailed] and q[uiet] verbosity levels for msbuild (GH-17791) (cherry picked from commit 89ae20b30e4543f379ee647c965eb46200556496) Co-authored-by: Anthony Shaw --- .../next/Windows/2020-01-02-01-11-53.bpo-39185.T4herN.rst | 1 + PCbuild/build.bat | 4 ++++ 2 files changed, 5 insertions(+) create mode 100644 Misc/NEWS.d/next/Windows/2020-01-02-01-11-53.bpo-39185.T4herN.rst diff --git a/Misc/NEWS.d/next/Windows/2020-01-02-01-11-53.bpo-39185.T4herN.rst b/Misc/NEWS.d/next/Windows/2020-01-02-01-11-53.bpo-39185.T4herN.rst new file mode 100644 index 00000000000000..3b84bd52172648 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-01-02-01-11-53.bpo-39185.T4herN.rst @@ -0,0 +1 @@ +The build.bat script has additional options for very-quiet output (-q) and very-verbose output (-vv) \ No newline at end of file diff --git a/PCbuild/build.bat b/PCbuild/build.bat index 623409c24ec04a..7f9de1f5b3f557 100644 --- a/PCbuild/build.bat +++ b/PCbuild/build.bat @@ -27,6 +27,8 @@ echo. building externals. echo. -m Enable parallel build (enabled by default) echo. -M Disable parallel build echo. -v Increased output messages +echo. -vv Verbose output messages +echo. -q Quiet output messages (errors and warnings only) echo. -k Attempt to kill any running Pythons before building (usually done echo. automatically by the pythoncore project) echo. --pgo Build with Profile-Guided Optimization. This flag @@ -72,6 +74,8 @@ if "%~1"=="-d" (set conf=Debug) & shift & goto CheckOpts if "%~1"=="-m" (set parallel=/m) & shift & goto CheckOpts if "%~1"=="-M" (set parallel=) & shift & goto CheckOpts if "%~1"=="-v" (set verbose=/v:n) & shift & goto CheckOpts +if "%~1"=="-vv" (set verbose=/v:d /ds) & shift & goto CheckOpts +if "%~1"=="-q" (set verbose=/v:q /nologo /clp:summary) & shift & goto CheckOpts if "%~1"=="-k" (set kill=true) & shift & goto CheckOpts if "%~1"=="--pgo" (set do_pgo=true) & shift & goto CheckOpts if "%~1"=="--pgo-job" (set do_pgo=true) & (set pgo_job=%~2) & shift & shift & goto CheckOpts From 927d3aab1c7874b5705fcc8269ea608315434e66 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 5 Feb 2020 00:39:36 -0800 Subject: [PATCH 1158/2163] bpo-39505: delete the redundant '/' in $env:VIRTUAL_ENV (GH-18290) (GH-18359) (cherry picked from commit 787b6d548c250f36df6d3f3179f60d754c8aa5e3) Co-authored-by: schwarzichet <15522755+schwarzichet@users.noreply.github.com> --- Lib/venv/scripts/common/Activate.ps1 | 1 - 1 file changed, 1 deletion(-) diff --git a/Lib/venv/scripts/common/Activate.ps1 b/Lib/venv/scripts/common/Activate.ps1 index 699c84097f1ac9..98cb1b85d11c61 100644 --- a/Lib/venv/scripts/common/Activate.ps1 +++ b/Lib/venv/scripts/common/Activate.ps1 @@ -168,7 +168,6 @@ if ($VenvDir) { } else { Write-Verbose "VenvDir not given as a parameter, using parent directory name as VenvDir." $VenvDir = $VenvExecDir.Parent.FullName.TrimEnd("\\/") - $VenvDir = $VenvDir.Insert($VenvDir.Length, "/") Write-Verbose "VenvDir=$VenvDir" } From 708f472dd92f4f46c27ace710492da65da4a3319 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 6 Feb 2020 00:45:18 -0800 Subject: [PATCH 1159/2163] bpo-38149: Call sys.audit() only once per call for glob.glob(). (GH-18360) (cherry picked from commit 54b4f14712b9350f11c983f1c8ac47a3716958a7) Co-authored-by: Serhiy Storchaka --- Lib/glob.py | 2 +- .../next/Library/2020-02-05-11-24-16.bpo-38149.GWsjHE.rst | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2020-02-05-11-24-16.bpo-38149.GWsjHE.rst diff --git a/Lib/glob.py b/Lib/glob.py index 0b3fcc6bbb9af3..0dd2f8be661094 100644 --- a/Lib/glob.py +++ b/Lib/glob.py @@ -31,6 +31,7 @@ def iglob(pathname, *, recursive=False): If recursive is true, the pattern '**' will match any files and zero or more directories and subdirectories. """ + sys.audit("glob.glob", pathname, recursive) it = _iglob(pathname, recursive, False) if recursive and _isrecursive(pathname): s = next(it) # skip empty string @@ -38,7 +39,6 @@ def iglob(pathname, *, recursive=False): return it def _iglob(pathname, recursive, dironly): - sys.audit("glob.glob", pathname, recursive) dirname, basename = os.path.split(pathname) if not has_magic(pathname): assert not dironly diff --git a/Misc/NEWS.d/next/Library/2020-02-05-11-24-16.bpo-38149.GWsjHE.rst b/Misc/NEWS.d/next/Library/2020-02-05-11-24-16.bpo-38149.GWsjHE.rst new file mode 100644 index 00000000000000..b4ec60b2abad14 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-05-11-24-16.bpo-38149.GWsjHE.rst @@ -0,0 +1,2 @@ +:func:`sys.audit` is now called only once per call of :func:`glob.glob` and +:func:`glob.iglob`. From 0d03a1028200646479ef9bb0ad8973d0e73f9525 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 6 Feb 2020 07:13:38 -0800 Subject: [PATCH 1160/2163] bpo-39274: Ensure Fraction.__bool__() returns a bool (GH-18017) Some numerator types used (specifically NumPy) decides to not return a Python boolean for the "a != b" operation. Using the equivalent call to bool() guarantees a bool return also for such types. (cherry picked from commit 427c84f13f7719e6014a21bd1b81efdc02a046fb) Co-authored-by: Sebastian Berg --- Lib/fractions.py | 4 +- Lib/test/test_fractions.py | 37 +++++++++++++++++++ .../2020-01-15-23-13-03.bpo-39274.lpc0-n.rst | 1 + 3 files changed, 41 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2020-01-15-23-13-03.bpo-39274.lpc0-n.rst diff --git a/Lib/fractions.py b/Lib/fractions.py index e774d58e403539..e4fcc8901bbf40 100644 --- a/Lib/fractions.py +++ b/Lib/fractions.py @@ -636,7 +636,9 @@ def __ge__(a, b): def __bool__(a): """a != 0""" - return a._numerator != 0 + # bpo-39274: Use bool() because (a._numerator != 0) can return an + # object which is not a bool. + return bool(a._numerator) # support for pickling, copy, and deepcopy diff --git a/Lib/test/test_fractions.py b/Lib/test/test_fractions.py index 18ab28cfebe0c8..86b49f30f549fb 100644 --- a/Lib/test/test_fractions.py +++ b/Lib/test/test_fractions.py @@ -6,6 +6,7 @@ import numbers import operator import fractions +import functools import sys import unittest import warnings @@ -346,6 +347,42 @@ def testConversions(self): self.assertTypedEquals(0.1+0j, complex(F(1,10))) + def testBoolGuarateesBoolReturn(self): + # Ensure that __bool__ is used on numerator which guarantees a bool + # return. See also bpo-39274. + @functools.total_ordering + class CustomValue: + denominator = 1 + + def __init__(self, value): + self.value = value + + def __bool__(self): + return bool(self.value) + + @property + def numerator(self): + # required to preserve `self` during instantiation + return self + + def __eq__(self, other): + raise AssertionError("Avoid comparisons in Fraction.__bool__") + + __lt__ = __eq__ + + # We did not implement all abstract methods, so register: + numbers.Rational.register(CustomValue) + + numerator = CustomValue(1) + r = F(numerator) + # ensure the numerator was not lost during instantiation: + self.assertIs(r.numerator, numerator) + self.assertIs(bool(r), True) + + numerator = CustomValue(0) + r = F(numerator) + self.assertIs(bool(r), False) + def testRound(self): self.assertTypedEquals(F(-200), round(F(-150), -2)) self.assertTypedEquals(F(-200), round(F(-250), -2)) diff --git a/Misc/NEWS.d/next/Library/2020-01-15-23-13-03.bpo-39274.lpc0-n.rst b/Misc/NEWS.d/next/Library/2020-01-15-23-13-03.bpo-39274.lpc0-n.rst new file mode 100644 index 00000000000000..4c398682b98ab1 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-15-23-13-03.bpo-39274.lpc0-n.rst @@ -0,0 +1 @@ +``bool(fraction.Fraction)`` now returns a boolean even if (numerator != 0) does not return a boolean (ex: numpy number). From 30096c9365fcb7cc3c38c4d161eaf8f61ae5cfea Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 6 Feb 2020 13:54:54 -0800 Subject: [PATCH 1161/2163] Fix MinGW library generation command (GH-17917) To print the exports to stdout, the gendef command requires the option "-". Without this option, no output is generated. (cherry picked from commit 2545fa87628b4caca519da8aeb0eeef368b9dc0d) Co-authored-by: Baljak --- Doc/whatsnew/3.8.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index 007e3228f9ae6d..9b61e1f05542df 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -2111,7 +2111,7 @@ Changes in the C API .. code-block:: shell - gendef python38.dll > tmp.def + gendef - python38.dll > tmp.def dlltool --dllname python38.dll --def tmp.def --output-lib libpython38.a The location of an installed :file:`pythonXY.dll` will depend on the From 97e00b3c52796cb54dd0a50548760579b9cb7b3a Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 6 Feb 2020 14:23:04 -0800 Subject: [PATCH 1162/2163] bpo-39534: Doc: Clarify return in finally (GH-18324) (cherry picked from commit 446463f8dbce0556be8020914f37089b63bb8ab6) Co-authored-by: Julien Palard --- Doc/tutorial/errors.rst | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/Doc/tutorial/errors.rst b/Doc/tutorial/errors.rst index 4bc7184d1078c5..27c67c6e015af5 100644 --- a/Doc/tutorial/errors.rst +++ b/Doc/tutorial/errors.rst @@ -341,15 +341,33 @@ example:: File "", line 2, in KeyboardInterrupt -If a :keyword:`finally` clause is present, the :keyword:`finally` clause will execute as the last task before the :keyword:`try` statement completes. The :keyword:`finally` clause runs whether or not the :keyword:`try` statement produces an exception. The following points discuss more complex cases when an exception occurs: - -* If an exception occurs during execution of the :keyword:`!try` clause, the exception may be handled by an :keyword:`except` clause. If the exception is not handled by an :keyword:`except` clause, the exception is re-raised after the :keyword:`!finally` clause has been executed. - -* An exception could occur during execution of an :keyword:`!except` or :keyword:`!else` clause. Again, the exception is re-raised after the :keyword:`!finally` clause has been executed. - -* If the :keyword:`!try` statement reaches a :keyword:`break`, :keyword:`continue` or :keyword:`return` statement, the :keyword:`finally` clause will execute just prior to the :keyword:`break`, :keyword:`continue` or :keyword:`return` statement's execution. - -* If a :keyword:`finally` clause includes a :keyword:`return` statement, the :keyword:`finally` clause's :keyword:`return` statement will execute before, and instead of, the :keyword:`return` statement in a :keyword:`try` clause. +If a :keyword:`finally` clause is present, the :keyword:`!finally` +clause will execute as the last task before the :keyword:`try` +statement completes. The :keyword:`!finally` clause runs whether or +not the :keyword:`!try` statement produces an exception. The following +points discuss more complex cases when an exception occurs: + +* If an exception occurs during execution of the :keyword:`!try` + clause, the exception may be handled by an :keyword:`except` + clause. If the exception is not handled by an :keyword:`!except` + clause, the exception is re-raised after the :keyword:`!finally` + clause has been executed. + +* An exception could occur during execution of an :keyword:`!except` + or :keyword:`!else` clause. Again, the exception is re-raised after + the :keyword:`!finally` clause has been executed. + +* If the :keyword:`!try` statement reaches a :keyword:`break`, + :keyword:`continue` or :keyword:`return` statement, the + :keyword:`!finally` clause will execute just prior to the + :keyword:`!break`, :keyword:`!continue` or :keyword:`!return` + statement's execution. + +* If a :keyword:`!finally` clause includes a :keyword:`!return` + statement, the returned value will be the one from the + :keyword:`!finally` clause's :keyword:`!return` statement, not the + value from the :keyword:`!try` clause's :keyword:`!return` + statement. For example:: From 8b9cebce09cb6919fdb97d8e608288a503681d13 Mon Sep 17 00:00:00 2001 From: Lysandros Nikolaou Date: Sat, 8 Feb 2020 01:21:38 +0100 Subject: [PATCH 1163/2163] [3.8] bpo-39579: Fix Attribute end_col_offset to point at the current node (GH-18405) (GH-18408) (cherry picked from commit d2e1098641f98594702ef29049c3c4a3f394786f) https://bugs.python.org/issue39579 Automerge-Triggered-By: @gvanrossum --- Lib/test/test_ast.py | 8 ++++++++ .../2020-02-07-15-18-35.bpo-39579.itNmC0.rst | 1 + Python/ast.c | 5 +++-- 3 files changed, 12 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-02-07-15-18-35.bpo-39579.itNmC0.rst diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py index 69ed83cd8b68b6..e843d53781d25d 100644 --- a/Lib/test/test_ast.py +++ b/Lib/test/test_ast.py @@ -140,6 +140,8 @@ def to_tuple(t): "@deco1\n@deco2()\n@deco3(1)\nclass C: pass", # Decorator with generator argument "@deco(a for a in b)\ndef f(): pass", + # Decorator with attribute + "@a.b.c\ndef f(): pass", # Simple assignment expression "(a := 1)", # Positional-only arguments @@ -614,6 +616,11 @@ def test_issue18374_binop_col_offset(self): self.assertEqual(grandchild_binop.end_col_offset, 3) self.assertEqual(grandchild_binop.end_lineno, 1) + def test_issue39579_dotted_name_end_col_offset(self): + tree = ast.parse('@a.b.c\ndef f(): pass') + attr_b = tree.body[0].decorator_list[0].value + self.assertEqual(attr_b.end_col_offset, 4) + class ASTHelpers_Test(unittest.TestCase): maxDiff = None @@ -1838,6 +1845,7 @@ def main(): ('Module', [('AsyncFunctionDef', (4, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (4, 15))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Constant', (3, 7), 1, None)], [])], None, None)], []), ('Module', [('ClassDef', (4, 0), 'C', [], [], [('Pass', (4, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Constant', (3, 7), 1, None)], [])])], []), ('Module', [('FunctionDef', (2, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (2, 9))], [('Call', (1, 1), ('Name', (1, 1), 'deco', ('Load',)), [('GeneratorExp', (1, 5), ('Name', (1, 6), 'a', ('Load',)), [('comprehension', ('Name', (1, 12), 'a', ('Store',)), ('Name', (1, 17), 'b', ('Load',)), [], 0)])], [])], None, None)], []), +('Module', [('FunctionDef', (2, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (2, 9))], [('Attribute', (1, 1), ('Attribute', (1, 1), ('Name', (1, 1), 'a', ('Load',)), 'b', ('Load',)), 'c', ('Load',))], None, None)], []), ('Module', [('Expr', (1, 0), ('NamedExpr', (1, 1), ('Name', (1, 1), 'a', ('Store',)), ('Constant', (1, 6), 1, None)))], []), ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None, None)], [], None, [], [], None, []), [('Pass', (1, 14))], [], None, None)], []), ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None, None)], [('arg', (1, 12), 'c', None, None), ('arg', (1, 15), 'd', None, None), ('arg', (1, 18), 'e', None, None)], None, [], [], None, []), [('Pass', (1, 22))], [], None, None)], []), diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-02-07-15-18-35.bpo-39579.itNmC0.rst b/Misc/NEWS.d/next/Core and Builtins/2020-02-07-15-18-35.bpo-39579.itNmC0.rst new file mode 100644 index 00000000000000..36d5c425670c25 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-02-07-15-18-35.bpo-39579.itNmC0.rst @@ -0,0 +1 @@ +Change the ending column offset of `Attribute` nodes constructed in `ast_for_dotted_name` to point at the end of the current node and not at the end of the last `NAME` node. \ No newline at end of file diff --git a/Python/ast.c b/Python/ast.c index f3263c1e3fcdce..12f24f2c22ab96 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -1715,11 +1715,12 @@ ast_for_dotted_name(struct compiling *c, const node *n) return NULL; for (i = 2; i < NCH(n); i+=2) { - id = NEW_IDENTIFIER(CHILD(n, i)); + const node *child = CHILD(n, i); + id = NEW_IDENTIFIER(child); if (!id) return NULL; e = Attribute(e, id, Load, lineno, col_offset, - n->n_end_lineno, n->n_end_col_offset, c->c_arena); + child->n_end_lineno, child->n_end_col_offset, c->c_arena); if (!e) return NULL; } From 0c915e620d049558bacc78cf25c1560f55d1fb82 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 7 Feb 2020 16:54:06 -0800 Subject: [PATCH 1164/2163] Doc: sys.__unraisablehook__ and bytearray.hex separators are new in 3.8 (GH-17884) Minor fix in documentation: - `sys.__unraisablehook__` is new in version 3.8 - Optional `sep` and `bytes_per_sep` parameters for `bytearray.hex` is also supported in Python 3.8 (just like `bytes.hex`) (cherry picked from commit 0edc2c7678266c39a7ceb2df885cb050f887e32b) Co-authored-by: Saiyang Gou --- Doc/library/stdtypes.rst | 16 +++++++++++++--- Doc/library/sys.rst | 2 ++ 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index d6db9b5411d8a6..d369b38b31bc90 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -2399,7 +2399,7 @@ data and are closely related to string objects in a variety of other ways. A reverse conversion function exists to transform a bytes object into its hexadecimal representation. - .. method:: hex() + .. method:: hex([sep[, bytes_per_sep]]) Return a string object containing two hexadecimal digits for each byte in the instance. @@ -2493,7 +2493,7 @@ objects. A reverse conversion function exists to transform a bytearray object into its hexadecimal representation. - .. method:: hex() + .. method:: hex([sep[, bytes_per_sep]]) Return a string object containing two hexadecimal digits for each byte in the instance. @@ -2503,6 +2503,11 @@ objects. .. versionadded:: 3.5 + .. versionchanged:: 3.8 + Similar to :meth:`bytes.hex`, :meth:`bytearray.hex` now supports + optional *sep* and *bytes_per_sep* parameters to insert separators + between bytes in the hex output. + Since bytearray objects are sequences of integers (akin to a list), for a bytearray object *b*, ``b[0]`` will be an integer, while ``b[0:1]`` will be a bytearray object of length 1. (This contrasts with text strings, where @@ -3649,7 +3654,7 @@ copying. in-memory Fortran order is preserved. For non-contiguous views, the data is converted to C first. *order=None* is the same as *order='C'*. - .. method:: hex() + .. method:: hex([sep[, bytes_per_sep]]) Return a string object containing two hexadecimal digits for each byte in the buffer. :: @@ -3660,6 +3665,11 @@ copying. .. versionadded:: 3.5 + .. versionchanged:: 3.8 + Similar to :meth:`bytes.hex`, :meth:`memoryview.hex` now supports + optional *sep* and *bytes_per_sep* parameters to insert separators + between bytes in the hex output. + .. method:: tolist() Return the data in the buffer as a list of elements. :: diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index e1d93f85bd4071..d3473de1292eab 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -343,6 +343,8 @@ always available. .. versionadded:: 3.7 __breakpointhook__ + .. versionadded:: 3.8 + __unraisablehook__ .. function:: exc_info() From dc56f5f48866bf3c5412642bba00890d9a090cfb Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 9 Feb 2020 00:39:28 -0800 Subject: [PATCH 1165/2163] bpo-39590: make deque.__contains__ and deque.count hold strong references (GH-18421) (GH-18423) (cherry picked from commit c6dedde160a9fce5d049e860f586ad8f93aec822) Co-authored-by: sweeneyde <36520290+sweeneyde@users.noreply.github.com> Co-authored-by: sweeneyde <36520290+sweeneyde@users.noreply.github.com> --- Lib/test/test_deque.py | 12 ++++++++++++ .../Library/2020-02-09-05-51-05.bpo-39590.rf98GU.rst | 1 + Modules/_collectionsmodule.c | 4 ++++ 3 files changed, 17 insertions(+) create mode 100644 Misc/NEWS.d/next/Library/2020-02-09-05-51-05.bpo-39590.rf98GU.rst diff --git a/Lib/test/test_deque.py b/Lib/test/test_deque.py index 51b66b76aca91d..c0f7138254f3f6 100644 --- a/Lib/test/test_deque.py +++ b/Lib/test/test_deque.py @@ -183,6 +183,18 @@ def test_contains(self): with self.assertRaises(RuntimeError): n in d + def test_contains_count_stop_crashes(self): + class A: + def __eq__(self, other): + d.clear() + return NotImplemented + d = deque([A(), A()]) + with self.assertRaises(RuntimeError): + _ = 3 in d + d = deque([A(), A()]) + with self.assertRaises(RuntimeError): + _ = d.count(3) + def test_extend(self): d = deque('a') self.assertRaises(TypeError, d.extend, 1) diff --git a/Misc/NEWS.d/next/Library/2020-02-09-05-51-05.bpo-39590.rf98GU.rst b/Misc/NEWS.d/next/Library/2020-02-09-05-51-05.bpo-39590.rf98GU.rst new file mode 100644 index 00000000000000..68625028fb7afc --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-09-05-51-05.bpo-39590.rf98GU.rst @@ -0,0 +1 @@ +Collections.deque now holds strong references during deque.__contains__ and deque.count, fixing crashes. \ No newline at end of file diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index 45169ecd11af00..cc2b90eaa283ea 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -966,7 +966,9 @@ deque_count(dequeobject *deque, PyObject *v) while (--n >= 0) { CHECK_NOT_END(b); item = b->data[index]; + Py_INCREF(item); cmp = PyObject_RichCompareBool(item, v, Py_EQ); + Py_DECREF(item); if (cmp < 0) return NULL; count += cmp; @@ -1003,7 +1005,9 @@ deque_contains(dequeobject *deque, PyObject *v) while (--n >= 0) { CHECK_NOT_END(b); item = b->data[index]; + Py_INCREF(item); cmp = PyObject_RichCompareBool(item, v, Py_EQ); + Py_DECREF(item); if (cmp) { return cmp; } From af95d790a86fc46739badfa9edbaeb264ee96600 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 10 Feb 2020 01:54:38 -0800 Subject: [PATCH 1166/2163] bpo-39128: Added happy_eyeballs_delay, interleave to function signature (GH-18315) (cherry picked from commit 5305cc9dbfe8a5a0ab666511f3ba7f026c8983f8) Co-authored-by: idomic --- Doc/library/asyncio-eventloop.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 24f621e2d9fffa..4ea8521eb0bf36 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -347,7 +347,8 @@ Opening network connections host=None, port=None, \*, ssl=None, \ family=0, proto=0, flags=0, sock=None, \ local_addr=None, server_hostname=None, \ - ssl_handshake_timeout=None) + ssl_handshake_timeout=None, \ + happy_eyeballs_delay=None, interleave=None) Open a streaming transport connection to a given address specified by *host* and *port*. @@ -436,7 +437,7 @@ Opening network connections .. versionadded:: 3.8 - The *happy_eyeballs_delay* and *interleave* parameters. + Added the *happy_eyeballs_delay* and *interleave* parameters. .. versionadded:: 3.7 From b086ea5edc44cf64ecfb645d578927aa96e8c355 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 10 Feb 2020 03:40:15 -0800 Subject: [PATCH 1167/2163] Grammar fix in tutorial (GH-18425) (GH-18426) (cherry picked from commit 3ed4d251587c36c3853daf42602eaad121b59bba) Co-authored-by: Don Kirkby --- Doc/tutorial/controlflow.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/tutorial/controlflow.rst b/Doc/tutorial/controlflow.rst index 7dfd33af258867..f05f5edd5ccc40 100644 --- a/Doc/tutorial/controlflow.rst +++ b/Doc/tutorial/controlflow.rst @@ -142,7 +142,7 @@ the list, thus saving space. We say such an object is :term:`iterable`, that is, suitable as a target for functions and constructs that expect something from which they can obtain successive items until the supply is exhausted. We have seen that -the :keyword:`for` statement is such a construct, while an example of function +the :keyword:`for` statement is such a construct, while an example of a function that takes an iterable is :func:`sum`:: >>> sum(range(4)) # 0 + 1 + 2 + 3 From 8623e68ea856830e084839e1d726c1f5be727203 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Mon, 10 Feb 2020 20:08:24 +0100 Subject: [PATCH 1168/2163] Python 3.8.2rc1 --- Include/patchlevel.h | 8 +- Lib/pydoc_data/topics.py | 171 ++++-- Misc/NEWS.d/3.8.2rc1.rst | 580 ++++++++++++++++++ .../2019-12-27-22-18-26.bpo-39144.dwHMlR.rst | 1 - .../2019-10-31-14-30-39.bpo-38610.fHdVMS.rst | 2 - .../2019-12-29-19-13-54.bpo-38588.pgXnNS.rst | 2 - .../2020-01-04-17-25-34.bpo-39215.xiqiIz.rst | 2 - .../2020-01-05-06-55-52.bpo-39216.74jLh9.rst | 2 - .../2020-01-06-10-29-16.bpo-39209.QHAONe.rst | 2 - .../2020-01-09-10-01-18.bpo-39235.RYwjoc.rst | 2 - .../2020-01-20-21-40-57.bpo-39386.ULqD8t.rst | 1 - .../2020-01-22-15-53-37.bpo-39421.O3nG7u.rst | 2 - .../2020-01-30-01-14-42.bpo-39492.eTuy0F.rst | 1 - .../2020-02-04-10-27-41.bpo-39510.PMIh-f.rst | 1 - .../2020-02-07-15-18-35.bpo-39579.itNmC0.rst | 1 - .../2019-11-17-11-53-10.bpo-3530.8zFUMc.rst | 2 - .../2019-12-15-22-04-41.bpo-38918.8JnDTS.rst | 3 - .../2020-01-18-15-37-56.bpo-39381.wTWe8d.rst | 2 - .../2020-01-27-18-18-42.bpo-39392.oiqcLO.rst | 1 - .../2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst | 5 - .../2018-03-03-12-56-26.bpo-32989.FVhmhH.rst | 2 - .../2019-11-13-23-51-39.bpo-38792.xhTC5a.rst | 2 - .../2019-12-30-16-44-07.bpo-34118.FaNW0a.rst | 2 - .../2020-01-22-22-28-06.bpo-39050.zkn0FO.rst | 1 - .../2020-01-25-02-26-45.bpo-39388.x4TQNh.rst | 1 - .../2020-01-27-16-44-29.bpo-30780.nR80qu.rst | 1 - .../2019-09-29-08-17-03.bpo-38293.wls5s3.rst | 1 - .../2019-10-14-21-14-55.bpo-38473.uXpVld.rst | 2 - .../2019-10-31-19-23-25.bpo-35182.hzeNl9.rst | 3 - .../2019-11-22-12-08-52.bpo-38878.EJ0cFf.rst | 2 - .../2019-12-13-18-54-49.bpo-39033.cepuyD.rst | 1 - .../2019-12-15-19-23-23.bpo-39055.FmN3un.rst | 2 - .../2019-12-15-21-05-16.bpo-39056.nEfUM9.rst | 2 - .../2019-12-15-21-47-54.bpo-39057.FOxn-w.rst | 2 - .../2019-12-24-10-43-13.bpo-39129.jVx5rW.rst | 1 - .../2019-12-31-19-27-23.bpo-39142.oqU5iD.rst | 5 - .../2020-01-01-18-44-52.bpo-38871.3EEOLg.rst | 2 - .../2020-01-02-17-28-03.bpo-39191.ur_scy.rst | 3 - .../2020-01-02-20-21-03.bpo-39198.nzwGyG.rst | 1 - .../2020-01-03-18-02-50.bpo-39152.JgPjCC.rst | 2 - .../2020-01-06-02-14-38.bpo-38907.F1RkCR.rst | 1 - .../2020-01-08-23-25-27.bpo-39242.bnL65N.rst | 3 - .../2020-01-11-01-15-37.bpo-39297.y98Z6Q.rst | 1 - .../2020-01-15-23-13-03.bpo-39274.lpc0-n.rst | 1 - .../2020-01-20-00-56-01.bpo-39389.fEirIS.rst | 2 - .../2020-01-23-21-34-29.bpo-39390.D2tSXk.rst | 2 - .../2020-01-24-11-05-21.bpo-39430.I0UQzM.rst | 1 - .../2020-01-24-13-24-35.bpo-39082.qKgrq_.rst | 1 - .../2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst | 3 - .../2020-01-30-01-13-19.bpo-39493.CbFRi7.rst | 1 - .../2020-02-02-14-46-34.bpo-39450.48R274.rst | 2 - .../2020-02-05-11-24-16.bpo-38149.GWsjHE.rst | 2 - .../2020-02-09-05-51-05.bpo-39590.rf98GU.rst | 1 - .../2020-01-07-00-42-08.bpo-39184.fe7NgK.rst | 1 - .../2020-01-28-20-54-09.bpo-39401.he7h_A.rst | 1 - .../2019-12-18-14-52-08.bpo-38546.2kxNuM.rst | 3 - .../2020-01-30-15-04-54.bpo-39502.chbpII.rst | 2 - .../2020-01-02-01-11-53.bpo-39185.T4herN.rst | 1 - .../2020-01-11-22-53-55.bpo-38883.X7FRaN.rst | 5 - .../2020-01-20-23-42-53.bpo-39393.gWlJDG.rst | 2 - .../2020-01-24-03-15-05.bpo-39439.sFxGfR.rst | 1 - README.rst | 4 +- 62 files changed, 705 insertions(+), 165 deletions(-) create mode 100644 Misc/NEWS.d/3.8.2rc1.rst delete mode 100644 Misc/NEWS.d/next/Build/2019-12-27-22-18-26.bpo-39144.dwHMlR.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-10-31-14-30-39.bpo-38610.fHdVMS.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-12-29-19-13-54.bpo-38588.pgXnNS.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-04-17-25-34.bpo-39215.xiqiIz.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-05-06-55-52.bpo-39216.74jLh9.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-06-10-29-16.bpo-39209.QHAONe.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-09-10-01-18.bpo-39235.RYwjoc.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-20-21-40-57.bpo-39386.ULqD8t.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-22-15-53-37.bpo-39421.O3nG7u.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-02-04-10-27-41.bpo-39510.PMIh-f.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-02-07-15-18-35.bpo-39579.itNmC0.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2019-11-17-11-53-10.bpo-3530.8zFUMc.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2019-12-15-22-04-41.bpo-38918.8JnDTS.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2020-01-18-15-37-56.bpo-39381.wTWe8d.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2020-01-27-18-18-42.bpo-39392.oiqcLO.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2018-03-03-12-56-26.bpo-32989.FVhmhH.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2019-11-13-23-51-39.bpo-38792.xhTC5a.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2019-12-30-16-44-07.bpo-34118.FaNW0a.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2020-01-22-22-28-06.bpo-39050.zkn0FO.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2020-01-25-02-26-45.bpo-39388.x4TQNh.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2020-01-27-16-44-29.bpo-30780.nR80qu.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-09-29-08-17-03.bpo-38293.wls5s3.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-10-14-21-14-55.bpo-38473.uXpVld.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-10-31-19-23-25.bpo-35182.hzeNl9.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-22-12-08-52.bpo-38878.EJ0cFf.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-13-18-54-49.bpo-39033.cepuyD.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-15-19-23-23.bpo-39055.FmN3un.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-15-21-05-16.bpo-39056.nEfUM9.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-15-21-47-54.bpo-39057.FOxn-w.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-24-10-43-13.bpo-39129.jVx5rW.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-31-19-27-23.bpo-39142.oqU5iD.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-01-18-44-52.bpo-38871.3EEOLg.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-02-17-28-03.bpo-39191.ur_scy.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-02-20-21-03.bpo-39198.nzwGyG.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-03-18-02-50.bpo-39152.JgPjCC.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-06-02-14-38.bpo-38907.F1RkCR.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-08-23-25-27.bpo-39242.bnL65N.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-11-01-15-37.bpo-39297.y98Z6Q.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-15-23-13-03.bpo-39274.lpc0-n.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-20-00-56-01.bpo-39389.fEirIS.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-23-21-34-29.bpo-39390.D2tSXk.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-24-11-05-21.bpo-39430.I0UQzM.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-24-13-24-35.bpo-39082.qKgrq_.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-30-01-13-19.bpo-39493.CbFRi7.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-02-05-11-24-16.bpo-38149.GWsjHE.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-02-09-05-51-05.bpo-39590.rf98GU.rst delete mode 100644 Misc/NEWS.d/next/Security/2020-01-07-00-42-08.bpo-39184.fe7NgK.rst delete mode 100644 Misc/NEWS.d/next/Security/2020-01-28-20-54-09.bpo-39401.he7h_A.rst delete mode 100644 Misc/NEWS.d/next/Tests/2019-12-18-14-52-08.bpo-38546.2kxNuM.rst delete mode 100644 Misc/NEWS.d/next/Tests/2020-01-30-15-04-54.bpo-39502.chbpII.rst delete mode 100644 Misc/NEWS.d/next/Windows/2020-01-02-01-11-53.bpo-39185.T4herN.rst delete mode 100644 Misc/NEWS.d/next/Windows/2020-01-11-22-53-55.bpo-38883.X7FRaN.rst delete mode 100644 Misc/NEWS.d/next/Windows/2020-01-20-23-42-53.bpo-39393.gWlJDG.rst delete mode 100644 Misc/NEWS.d/next/Windows/2020-01-24-03-15-05.bpo-39439.sFxGfR.rst diff --git a/Include/patchlevel.h b/Include/patchlevel.h index 503d3aa85676f0..a40ef8232ece9b 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -18,12 +18,12 @@ /*--start constants--*/ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 8 -#define PY_MICRO_VERSION 1 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL -#define PY_RELEASE_SERIAL 0 +#define PY_MICRO_VERSION 2 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA +#define PY_RELEASE_SERIAL 1 /* Version as a string */ -#define PY_VERSION "3.8.1+" +#define PY_VERSION "3.8.2rc1" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index 9d779d1e69f23b..b9e741707cba5b 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Wed Dec 18 18:17:58 2019 +# Autogenerated by Sphinx on Mon Feb 10 19:25:27 2020 topics = {'assert': 'The "assert" statement\n' '**********************\n' '\n' @@ -470,24 +470,25 @@ 'The following code:\n' '\n' ' async for TARGET in ITER:\n' - ' BLOCK\n' + ' SUITE\n' ' else:\n' - ' BLOCK2\n' + ' SUITE2\n' '\n' 'Is semantically equivalent to:\n' '\n' ' iter = (ITER)\n' ' iter = type(iter).__aiter__(iter)\n' ' running = True\n' + '\n' ' while running:\n' ' try:\n' ' TARGET = await type(iter).__anext__(iter)\n' ' except StopAsyncIteration:\n' ' running = False\n' ' else:\n' - ' BLOCK\n' + ' SUITE\n' ' else:\n' - ' BLOCK2\n' + ' SUITE2\n' '\n' 'See also "__aiter__()" and "__anext__()" for details.\n' '\n' @@ -507,23 +508,27 @@ '\n' 'The following code:\n' '\n' - ' async with EXPR as VAR:\n' - ' BLOCK\n' + ' async with EXPRESSION as TARGET:\n' + ' SUITE\n' '\n' - 'Is semantically equivalent to:\n' + 'is semantically equivalent to:\n' '\n' - ' mgr = (EXPR)\n' - ' aexit = type(mgr).__aexit__\n' - ' aenter = type(mgr).__aenter__(mgr)\n' + ' manager = (EXPRESSION)\n' + ' aexit = type(manager).__aexit__\n' + ' aenter = type(manager).__aenter__\n' + ' value = await aenter(manager)\n' + ' hit_except = False\n' '\n' - ' VAR = await aenter\n' ' try:\n' - ' BLOCK\n' + ' TARGET = value\n' + ' SUITE\n' ' except:\n' - ' if not await aexit(mgr, *sys.exc_info()):\n' + ' hit_except = True\n' + ' if not await aexit(manager, *sys.exc_info()):\n' ' raise\n' - ' else:\n' - ' await aexit(mgr, None, None, None)\n' + ' finally:\n' + ' if not hit_except:\n' + ' await aexit(manager, None, None, None)\n' '\n' 'See also "__aenter__()" and "__aexit__()" for details.\n' '\n' @@ -2518,11 +2523,13 @@ '"with_item")\n' ' is evaluated to obtain a context manager.\n' '\n' - '2. The context manager’s "__exit__()" is loaded for later use.\n' + '2. The context manager’s "__enter__()" is loaded for later use.\n' + '\n' + '3. The context manager’s "__exit__()" is loaded for later use.\n' '\n' - '3. The context manager’s "__enter__()" method is invoked.\n' + '4. The context manager’s "__enter__()" method is invoked.\n' '\n' - '4. If a target was included in the "with" statement, the return\n' + '5. If a target was included in the "with" statement, the return\n' ' value from "__enter__()" is assigned to it.\n' '\n' ' Note: The "with" statement guarantees that if the ' @@ -2535,9 +2542,9 @@ 'occurring\n' ' within the suite would be. See step 6 below.\n' '\n' - '5. The suite is executed.\n' + '6. The suite is executed.\n' '\n' - '6. The context manager’s "__exit__()" method is invoked. If an\n' + '7. The context manager’s "__exit__()" method is invoked. If an\n' ' exception caused the suite to be exited, its type, value, ' 'and\n' ' traceback are passed as arguments to "__exit__()". Otherwise, ' @@ -2559,18 +2566,42 @@ 'proceeds\n' ' at the normal location for the kind of exit that was taken.\n' '\n' + 'The following code:\n' + '\n' + ' with EXPRESSION as TARGET:\n' + ' SUITE\n' + '\n' + 'is semantically equivalent to:\n' + '\n' + ' manager = (EXPRESSION)\n' + ' enter = type(manager).__enter__\n' + ' exit = type(manager).__exit__\n' + ' value = enter(manager)\n' + ' hit_except = False\n' + '\n' + ' try:\n' + ' TARGET = value\n' + ' SUITE\n' + ' except:\n' + ' hit_except = True\n' + ' if not exit(manager, *sys.exc_info()):\n' + ' raise\n' + ' finally:\n' + ' if not hit_except:\n' + ' exit(manager, None, None, None)\n' + '\n' 'With more than one item, the context managers are processed as ' 'if\n' 'multiple "with" statements were nested:\n' '\n' ' with A() as a, B() as b:\n' - ' suite\n' + ' SUITE\n' '\n' - 'is equivalent to\n' + 'is semantically equivalent to:\n' '\n' ' with A() as a:\n' ' with B() as b:\n' - ' suite\n' + ' SUITE\n' '\n' 'Changed in version 3.1: Support for multiple context ' 'expressions.\n' @@ -2934,24 +2965,25 @@ 'The following code:\n' '\n' ' async for TARGET in ITER:\n' - ' BLOCK\n' + ' SUITE\n' ' else:\n' - ' BLOCK2\n' + ' SUITE2\n' '\n' 'Is semantically equivalent to:\n' '\n' ' iter = (ITER)\n' ' iter = type(iter).__aiter__(iter)\n' ' running = True\n' + '\n' ' while running:\n' ' try:\n' ' TARGET = await type(iter).__anext__(iter)\n' ' except StopAsyncIteration:\n' ' running = False\n' ' else:\n' - ' BLOCK\n' + ' SUITE\n' ' else:\n' - ' BLOCK2\n' + ' SUITE2\n' '\n' 'See also "__aiter__()" and "__anext__()" for details.\n' '\n' @@ -2971,23 +3003,27 @@ '\n' 'The following code:\n' '\n' - ' async with EXPR as VAR:\n' - ' BLOCK\n' + ' async with EXPRESSION as TARGET:\n' + ' SUITE\n' '\n' - 'Is semantically equivalent to:\n' + 'is semantically equivalent to:\n' '\n' - ' mgr = (EXPR)\n' - ' aexit = type(mgr).__aexit__\n' - ' aenter = type(mgr).__aenter__(mgr)\n' + ' manager = (EXPRESSION)\n' + ' aexit = type(manager).__aexit__\n' + ' aenter = type(manager).__aenter__\n' + ' value = await aenter(manager)\n' + ' hit_except = False\n' '\n' - ' VAR = await aenter\n' ' try:\n' - ' BLOCK\n' + ' TARGET = value\n' + ' SUITE\n' ' except:\n' - ' if not await aexit(mgr, *sys.exc_info()):\n' + ' hit_except = True\n' + ' if not await aexit(manager, *sys.exc_info()):\n' ' raise\n' - ' else:\n' - ' await aexit(mgr, None, None, None)\n' + ' finally:\n' + ' if not hit_except:\n' + ' await aexit(manager, None, None, None)\n' '\n' 'See also "__aenter__()" and "__aexit__()" for details.\n' '\n' @@ -6803,7 +6839,7 @@ 'object.__rfloordiv__(self, other)\n' 'object.__rmod__(self, other)\n' 'object.__rdivmod__(self, other)\n' - 'object.__rpow__(self, other)\n' + 'object.__rpow__(self, other[, modulo])\n' 'object.__rlshift__(self, other)\n' 'object.__rrshift__(self, other)\n' 'object.__rand__(self, other)\n' @@ -8963,7 +8999,9 @@ 'bases,\n' '**kwds)" (where the additional keyword arguments, if any, ' 'come from\n' - 'the class definition).\n' + 'the class definition). The "__prepare__" method should be ' + 'implemented\n' + 'as a "classmethod()".\n' '\n' 'If the metaclass has no "__prepare__" attribute, then the ' 'class\n' @@ -9477,7 +9515,7 @@ 'object.__rfloordiv__(self, other)\n' 'object.__rmod__(self, other)\n' 'object.__rdivmod__(self, other)\n' - 'object.__rpow__(self, other)\n' + 'object.__rpow__(self, other[, modulo])\n' 'object.__rlshift__(self, other)\n' 'object.__rrshift__(self, other)\n' 'object.__rand__(self, other)\n' @@ -11918,8 +11956,9 @@ ' bytecode offsets to line numbers (for details see the source\n' ' code of the interpreter); "co_stacksize" is the required ' 'stack\n' - ' size (including local variables); "co_flags" is an integer\n' - ' encoding a number of flags for the interpreter.\n' + ' size; "co_flags" is an integer encoding a number of flags ' + 'for\n' + ' the interpreter.\n' '\n' ' The following flag bits are defined for "co_flags": bit ' '"0x04"\n' @@ -12372,6 +12411,8 @@ 'dictionary. This\n' ' is a shortcut for "reversed(d.keys())".\n' '\n' + ' New in version 3.8.\n' + '\n' ' setdefault(key[, default])\n' '\n' ' If *key* is in the dictionary, return its value. If ' @@ -13577,11 +13618,13 @@ '1. The context expression (the expression given in the "with_item")\n' ' is evaluated to obtain a context manager.\n' '\n' - '2. The context manager’s "__exit__()" is loaded for later use.\n' + '2. The context manager’s "__enter__()" is loaded for later use.\n' '\n' - '3. The context manager’s "__enter__()" method is invoked.\n' + '3. The context manager’s "__exit__()" is loaded for later use.\n' '\n' - '4. If a target was included in the "with" statement, the return\n' + '4. The context manager’s "__enter__()" method is invoked.\n' + '\n' + '5. If a target was included in the "with" statement, the return\n' ' value from "__enter__()" is assigned to it.\n' '\n' ' Note: The "with" statement guarantees that if the "__enter__()"\n' @@ -13591,9 +13634,9 @@ ' target list, it will be treated the same as an error occurring\n' ' within the suite would be. See step 6 below.\n' '\n' - '5. The suite is executed.\n' + '6. The suite is executed.\n' '\n' - '6. The context manager’s "__exit__()" method is invoked. If an\n' + '7. The context manager’s "__exit__()" method is invoked. If an\n' ' exception caused the suite to be exited, its type, value, and\n' ' traceback are passed as arguments to "__exit__()". Otherwise, ' 'three\n' @@ -13613,17 +13656,41 @@ 'proceeds\n' ' at the normal location for the kind of exit that was taken.\n' '\n' + 'The following code:\n' + '\n' + ' with EXPRESSION as TARGET:\n' + ' SUITE\n' + '\n' + 'is semantically equivalent to:\n' + '\n' + ' manager = (EXPRESSION)\n' + ' enter = type(manager).__enter__\n' + ' exit = type(manager).__exit__\n' + ' value = enter(manager)\n' + ' hit_except = False\n' + '\n' + ' try:\n' + ' TARGET = value\n' + ' SUITE\n' + ' except:\n' + ' hit_except = True\n' + ' if not exit(manager, *sys.exc_info()):\n' + ' raise\n' + ' finally:\n' + ' if not hit_except:\n' + ' exit(manager, None, None, None)\n' + '\n' 'With more than one item, the context managers are processed as if\n' 'multiple "with" statements were nested:\n' '\n' ' with A() as a, B() as b:\n' - ' suite\n' + ' SUITE\n' '\n' - 'is equivalent to\n' + 'is semantically equivalent to:\n' '\n' ' with A() as a:\n' ' with B() as b:\n' - ' suite\n' + ' SUITE\n' '\n' 'Changed in version 3.1: Support for multiple context expressions.\n' '\n' diff --git a/Misc/NEWS.d/3.8.2rc1.rst b/Misc/NEWS.d/3.8.2rc1.rst new file mode 100644 index 00000000000000..a4b8a1ed8fb4f9 --- /dev/null +++ b/Misc/NEWS.d/3.8.2rc1.rst @@ -0,0 +1,580 @@ +.. bpo: 39401 +.. date: 2020-01-28-20-54-09 +.. nonce: he7h_A +.. release date: 2020-02-10 +.. section: Security + +Avoid unsafe load of ``api-ms-win-core-path-l1-1-0.dll`` at startup on +Windows 7. + +.. + +.. bpo: 39184 +.. date: 2020-01-07-00-42-08 +.. nonce: fe7NgK +.. section: Security + +Add audit events to command execution functions in os and pty modules. + +.. + +.. bpo: 39579 +.. date: 2020-02-07-15-18-35 +.. nonce: itNmC0 +.. section: Core and Builtins + +Change the ending column offset of `Attribute` nodes constructed in +`ast_for_dotted_name` to point at the end of the current node and not at the +end of the last `NAME` node. + +.. + +.. bpo: 39510 +.. date: 2020-02-04-10-27-41 +.. nonce: PMIh-f +.. section: Core and Builtins + +Fix segfault in ``readinto()`` method on closed BufferedReader. + +.. + +.. bpo: 39492 +.. date: 2020-01-30-01-14-42 +.. nonce: eTuy0F +.. section: Core and Builtins + +Fix a reference cycle in the C Pickler that was preventing the garbage +collection of deleted, pickled objects. + +.. + +.. bpo: 39421 +.. date: 2020-01-22-15-53-37 +.. nonce: O3nG7u +.. section: Core and Builtins + +Fix possible crashes when operating with the functions in the :mod:`heapq` +module and custom comparison operators. + +.. + +.. bpo: 39386 +.. date: 2020-01-20-21-40-57 +.. nonce: ULqD8t +.. section: Core and Builtins + +Prevent double awaiting of async iterator. + +.. + +.. bpo: 39235 +.. date: 2020-01-09-10-01-18 +.. nonce: RYwjoc +.. section: Core and Builtins + +Fix AST end location for lone generator expression in function call, e.g. +f(i for i in a). + +.. + +.. bpo: 39209 +.. date: 2020-01-06-10-29-16 +.. nonce: QHAONe +.. section: Core and Builtins + +Correctly handle multi-line tokens in interactive mode. Patch by Pablo +Galindo. + +.. + +.. bpo: 39216 +.. date: 2020-01-05-06-55-52 +.. nonce: 74jLh9 +.. section: Core and Builtins + +Fix constant folding optimization for positional only arguments - by Anthony +Sottile. + +.. + +.. bpo: 39215 +.. date: 2020-01-04-17-25-34 +.. nonce: xiqiIz +.. section: Core and Builtins + +Fix ``SystemError`` when nested function has annotation on positional-only +argument - by Anthony Sottile. + +.. + +.. bpo: 38588 +.. date: 2019-12-29-19-13-54 +.. nonce: pgXnNS +.. section: Core and Builtins + +Fix possible crashes in dict and list when calling +:c:func:`PyObject_RichCompareBool`. + +.. + +.. bpo: 38610 +.. date: 2019-10-31-14-30-39 +.. nonce: fHdVMS +.. section: Core and Builtins + +Fix possible crashes in several list methods by holding strong references to +list elements when calling :c:func:`PyObject_RichCompareBool`. + +.. + +.. bpo: 39590 +.. date: 2020-02-09-05-51-05 +.. nonce: rf98GU +.. section: Library + +Collections.deque now holds strong references during deque.__contains__ and +deque.count, fixing crashes. + +.. + +.. bpo: 38149 +.. date: 2020-02-05-11-24-16 +.. nonce: GWsjHE +.. section: Library + +:func:`sys.audit` is now called only once per call of :func:`glob.glob` and +:func:`glob.iglob`. + +.. + +.. bpo: 39450 +.. date: 2020-02-02-14-46-34 +.. nonce: 48R274 +.. section: Library + +Striped whitespace from docstring before returning it from +:func:`unittest.case.shortDescription`. + +.. + +.. bpo: 39493 +.. date: 2020-01-30-01-13-19 +.. nonce: CbFRi7 +.. section: Library + +Mark ``typing.IO.closed`` as a property + +.. + +.. bpo: 39485 +.. date: 2020-01-29-14-58-27 +.. nonce: Zy3ot6 +.. section: Library + +Fix a bug in :func:`unittest.mock.create_autospec` that would complain about +the wrong number of arguments for custom descriptors defined in an extension +module returning functions. + +.. + +.. bpo: 39082 +.. date: 2020-01-24-13-24-35 +.. nonce: qKgrq_ +.. section: Library + +Allow AsyncMock to correctly patch static/class methods + +.. + +.. bpo: 39430 +.. date: 2020-01-24-11-05-21 +.. nonce: I0UQzM +.. section: Library + +Fixed race condition in lazy imports in :mod:`tarfile`. + +.. + +.. bpo: 39390 +.. date: 2020-01-23-21-34-29 +.. nonce: D2tSXk +.. section: Library + +Fixed a regression with the `ignore` callback of :func:`shutil.copytree`. +The argument types are now str and List[str] again. + +.. + +.. bpo: 39389 +.. date: 2020-01-20-00-56-01 +.. nonce: fEirIS +.. section: Library + +Write accurate compression level metadata in :mod:`gzip` archives, rather +than always signaling maximum compression. + +.. + +.. bpo: 39274 +.. date: 2020-01-15-23-13-03 +.. nonce: lpc0-n +.. section: Library + +``bool(fraction.Fraction)`` now returns a boolean even if (numerator != 0) +does not return a boolean (ex: numpy number). + +.. + +.. bpo: 39297 +.. date: 2020-01-11-01-15-37 +.. nonce: y98Z6Q +.. section: Library + +Improved performance of importlib.metadata distribution discovery and +resilients to inaccessible sys.path entries (importlib_metadata v1.4.0). + +.. + +.. bpo: 39242 +.. date: 2020-01-08-23-25-27 +.. nonce: bnL65N +.. section: Library + +Updated the Gmane domain from news.gmane.org to news.gmane.io which is used +for examples of :class:`~nntplib.NNTP` news reader server and nntplib tests. + +.. + +.. bpo: 38907 +.. date: 2020-01-06-02-14-38 +.. nonce: F1RkCR +.. section: Library + +In http.server script, restore binding to IPv4 on Windows. + +.. + +.. bpo: 39152 +.. date: 2020-01-03-18-02-50 +.. nonce: JgPjCC +.. section: Library + +Fix ttk.Scale.configure([name]) to return configuration tuple for name or +all options. Giovanni Lombardo contributed part of the patch. + +.. + +.. bpo: 39198 +.. date: 2020-01-02-20-21-03 +.. nonce: nzwGyG +.. section: Library + +If an exception were to be thrown in `Logger.isEnabledFor` (say, by asyncio +timeouts or stopit) , the `logging` global lock may not be released +appropriately, resulting in deadlock. This change wraps that block of code +with `try...finally` to ensure the lock is released. + +.. + +.. bpo: 39191 +.. date: 2020-01-02-17-28-03 +.. nonce: ur_scy +.. section: Library + +Perform a check for running loop before starting a new task in +``loop.run_until_complete()`` to fail fast; it prevents the side effect of +new task spawning before exception raising. + +.. + +.. bpo: 38871 +.. date: 2020-01-01-18-44-52 +.. nonce: 3EEOLg +.. section: Library + +Correctly parenthesize filter-based statements that contain lambda +expressions in mod:`lib2to3`. Patch by Dong-hee Na. + +.. + +.. bpo: 39142 +.. date: 2019-12-31-19-27-23 +.. nonce: oqU5iD +.. section: Library + +A change was made to logging.config.dictConfig to avoid converting instances +of named tuples to ConvertingTuple. It's assumed that named tuples are too +specialised to be treated like ordinary tuples; if a user of named tuples +requires ConvertingTuple functionality, they will have to implement that +themselves in their named tuple class. + +.. + +.. bpo: 39129 +.. date: 2019-12-24-10-43-13 +.. nonce: jVx5rW +.. section: Library + +Fix import path for ``asyncio.TimeoutError`` + +.. + +.. bpo: 39057 +.. date: 2019-12-15-21-47-54 +.. nonce: FOxn-w +.. section: Library + +:func:`urllib.request.proxy_bypass_environment` now ignores leading dots and +no longer ignores a trailing newline. + +.. + +.. bpo: 39056 +.. date: 2019-12-15-21-05-16 +.. nonce: nEfUM9 +.. section: Library + +Fixed handling invalid warning category in the -W option. No longer import +the re module if it is not needed. + +.. + +.. bpo: 39055 +.. date: 2019-12-15-19-23-23 +.. nonce: FmN3un +.. section: Library + +:func:`base64.b64decode` with ``validate=True`` raises now a binascii.Error +if the input ends with a single ``\n``. + +.. + +.. bpo: 39033 +.. date: 2019-12-13-18-54-49 +.. nonce: cepuyD +.. section: Library + +Fix :exc:`NameError` in :mod:`zipimport`. Patch by Karthikeyan Singaravelan. + +.. + +.. bpo: 38878 +.. date: 2019-11-22-12-08-52 +.. nonce: EJ0cFf +.. section: Library + +Fixed __subclasshook__ of :class:`os.PathLike` to return a correct result +upon inheritence. Patch by Bar Harel. + +.. + +.. bpo: 35182 +.. date: 2019-10-31-19-23-25 +.. nonce: hzeNl9 +.. section: Library + +Fixed :func:`Popen.communicate` subsequent call crash when the child process +has already closed any piped standard stream, but still continues to be +running. Patch by Andriy Maletsky. + +.. + +.. bpo: 38473 +.. date: 2019-10-14-21-14-55 +.. nonce: uXpVld +.. section: Library + +Use signature from inner mock for autospecced methods attached with +:func:`unittest.mock.attach_mock`. Patch by Karthikeyan Singaravelan. + +.. + +.. bpo: 38293 +.. date: 2019-09-29-08-17-03 +.. nonce: wls5s3 +.. section: Library + +Add :func:`copy.copy` and :func:`copy.deepcopy` support to :func:`property` +objects. + +.. + +.. bpo: 39153 +.. date: 2020-01-27-22-24-51 +.. nonce: Pjl8jV +.. section: Documentation + +Clarify refcounting semantics for the following functions: - +PyObject_SetItem - PyMapping_SetItemString - PyDict_SetItem - +PyDict_SetItemString + +.. + +.. bpo: 39392 +.. date: 2020-01-27-18-18-42 +.. nonce: oiqcLO +.. section: Documentation + +Explain that when filling with turtle, overlap regions may be left unfilled. + +.. + +.. bpo: 39381 +.. date: 2020-01-18-15-37-56 +.. nonce: wTWe8d +.. section: Documentation + +Mention in docs that :func:`asyncio.get_event_loop` implicitly creates new +event loop only if called from the main thread. + +.. + +.. bpo: 38918 +.. date: 2019-12-15-22-04-41 +.. nonce: 8JnDTS +.. section: Documentation + +Add an entry for ``__module__`` in the "function" & "method" sections of the +`inspect docs types and members table +`_ + +.. + +.. bpo: 3530 +.. date: 2019-11-17-11-53-10 +.. nonce: 8zFUMc +.. section: Documentation + +In the :mod:`ast` module documentation, fix a misleading ``NodeTransformer`` +example and add advice on when to use the ``fix_missing_locations`` +function. + +.. + +.. bpo: 39502 +.. date: 2020-01-30-15-04-54 +.. nonce: chbpII +.. section: Tests + +Skip test_zipfile.test_add_file_after_2107() if :func:`time.localtime` fails +with :exc:`OverflowError`. It is the case on AIX 6.1 for example. + +.. + +.. bpo: 38546 +.. date: 2019-12-18-14-52-08 +.. nonce: 2kxNuM +.. section: Tests + +Fix test_ressources_gced_in_workers() of test_concurrent_futures: explicitly +stop the manager to prevent leaking a child process running in the +background after the test completes. + +.. + +.. bpo: 39144 +.. date: 2019-12-27-22-18-26 +.. nonce: dwHMlR +.. section: Build + +The ctags and etags build targets both include Modules/_ctypes and Python +standard library source files. + +.. + +.. bpo: 39439 +.. date: 2020-01-24-03-15-05 +.. nonce: sFxGfR +.. section: Windows + +Honor the Python path when a virtualenv is active on Windows. + +.. + +.. bpo: 39393 +.. date: 2020-01-20-23-42-53 +.. nonce: gWlJDG +.. section: Windows + +Improve the error message when attempting to load a DLL with unresolved +dependencies. + +.. + +.. bpo: 38883 +.. date: 2020-01-11-22-53-55 +.. nonce: X7FRaN +.. section: Windows + +:meth:`~pathlib.Path.home()` and :meth:`~pathlib.Path.expanduser()` on +Windows now prefer :envvar:`USERPROFILE` and no longer use :envvar:`HOME`, +which is not normally set for regular user accounts. This makes them again +behave like :func:`os.path.expanduser`, which was changed to ignore +:envvar:`HOME` in 3.8, see :issue:`36264`. + +.. + +.. bpo: 39185 +.. date: 2020-01-02-01-11-53 +.. nonce: T4herN +.. section: Windows + +The build.bat script has additional options for very-quiet output (-q) and +very-verbose output (-vv) + +.. + +.. bpo: 30780 +.. date: 2020-01-27-16-44-29 +.. nonce: nR80qu +.. section: IDLE + +Add remaining configdialog tests for buttons and highlights and keys tabs. + +.. + +.. bpo: 39388 +.. date: 2020-01-25-02-26-45 +.. nonce: x4TQNh +.. section: IDLE + +IDLE Settings Cancel button now cancels pending changes + +.. + +.. bpo: 39050 +.. date: 2020-01-22-22-28-06 +.. nonce: zkn0FO +.. section: IDLE + +Make IDLE Settings dialog Help button work again. + +.. + +.. bpo: 34118 +.. date: 2019-12-30-16-44-07 +.. nonce: FaNW0a +.. section: IDLE + +Tag memoryview, range, and tuple as classes, the same as list, etcetera, in +the library manual built-in functions list. + +.. + +.. bpo: 38792 +.. date: 2019-11-13-23-51-39 +.. nonce: xhTC5a +.. section: IDLE + +Close an IDLE shell calltip if a :exc:`KeyboardInterrupt` or shell restart +occurs. Patch by Zackery Spytz. + +.. + +.. bpo: 32989 +.. date: 2018-03-03-12-56-26 +.. nonce: FVhmhH +.. section: IDLE + +Add tests for editor newline_and_indent_event method. Remove dead code from +pyparse find_good_parse_start method. diff --git a/Misc/NEWS.d/next/Build/2019-12-27-22-18-26.bpo-39144.dwHMlR.rst b/Misc/NEWS.d/next/Build/2019-12-27-22-18-26.bpo-39144.dwHMlR.rst deleted file mode 100644 index 8b90da19622e61..00000000000000 --- a/Misc/NEWS.d/next/Build/2019-12-27-22-18-26.bpo-39144.dwHMlR.rst +++ /dev/null @@ -1 +0,0 @@ -The ctags and etags build targets both include Modules/_ctypes and Python standard library source files. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-31-14-30-39.bpo-38610.fHdVMS.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-31-14-30-39.bpo-38610.fHdVMS.rst deleted file mode 100644 index 0ee63bbb40dc66..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-10-31-14-30-39.bpo-38610.fHdVMS.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix possible crashes in several list methods by holding strong references to -list elements when calling :c:func:`PyObject_RichCompareBool`. diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-12-29-19-13-54.bpo-38588.pgXnNS.rst b/Misc/NEWS.d/next/Core and Builtins/2019-12-29-19-13-54.bpo-38588.pgXnNS.rst deleted file mode 100644 index 0b81085a89d254..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-12-29-19-13-54.bpo-38588.pgXnNS.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix possible crashes in dict and list when calling -:c:func:`PyObject_RichCompareBool`. diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-04-17-25-34.bpo-39215.xiqiIz.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-04-17-25-34.bpo-39215.xiqiIz.rst deleted file mode 100644 index 9a3178f9c62189..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-01-04-17-25-34.bpo-39215.xiqiIz.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix ``SystemError`` when nested function has annotation on positional-only -argument - by Anthony Sottile. diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-05-06-55-52.bpo-39216.74jLh9.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-05-06-55-52.bpo-39216.74jLh9.rst deleted file mode 100644 index 971b06552973ea..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-01-05-06-55-52.bpo-39216.74jLh9.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix constant folding optimization for positional only arguments - by Anthony -Sottile. diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-06-10-29-16.bpo-39209.QHAONe.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-06-10-29-16.bpo-39209.QHAONe.rst deleted file mode 100644 index c05b3f8dfa4d41..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-01-06-10-29-16.bpo-39209.QHAONe.rst +++ /dev/null @@ -1,2 +0,0 @@ -Correctly handle multi-line tokens in interactive mode. Patch by Pablo -Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-09-10-01-18.bpo-39235.RYwjoc.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-09-10-01-18.bpo-39235.RYwjoc.rst deleted file mode 100644 index 5fb0d45356badf..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-01-09-10-01-18.bpo-39235.RYwjoc.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix AST end location for lone generator expression in function call, e.g. -f(i for i in a). diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-20-21-40-57.bpo-39386.ULqD8t.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-20-21-40-57.bpo-39386.ULqD8t.rst deleted file mode 100644 index f24e1f4e8a1831..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-01-20-21-40-57.bpo-39386.ULqD8t.rst +++ /dev/null @@ -1 +0,0 @@ -Prevent double awaiting of async iterator. diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-22-15-53-37.bpo-39421.O3nG7u.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-22-15-53-37.bpo-39421.O3nG7u.rst deleted file mode 100644 index bae008150ee127..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-01-22-15-53-37.bpo-39421.O3nG7u.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix possible crashes when operating with the functions in the :mod:`heapq` -module and custom comparison operators. diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst deleted file mode 100644 index 6e8b715c46365b..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-01-30-01-14-42.bpo-39492.eTuy0F.rst +++ /dev/null @@ -1 +0,0 @@ -Fix a reference cycle in the C Pickler that was preventing the garbage collection of deleted, pickled objects. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-02-04-10-27-41.bpo-39510.PMIh-f.rst b/Misc/NEWS.d/next/Core and Builtins/2020-02-04-10-27-41.bpo-39510.PMIh-f.rst deleted file mode 100644 index 9a38e4ab762287..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-02-04-10-27-41.bpo-39510.PMIh-f.rst +++ /dev/null @@ -1 +0,0 @@ -Fix segfault in ``readinto()`` method on closed BufferedReader. diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-02-07-15-18-35.bpo-39579.itNmC0.rst b/Misc/NEWS.d/next/Core and Builtins/2020-02-07-15-18-35.bpo-39579.itNmC0.rst deleted file mode 100644 index 36d5c425670c25..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-02-07-15-18-35.bpo-39579.itNmC0.rst +++ /dev/null @@ -1 +0,0 @@ -Change the ending column offset of `Attribute` nodes constructed in `ast_for_dotted_name` to point at the end of the current node and not at the end of the last `NAME` node. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Documentation/2019-11-17-11-53-10.bpo-3530.8zFUMc.rst b/Misc/NEWS.d/next/Documentation/2019-11-17-11-53-10.bpo-3530.8zFUMc.rst deleted file mode 100644 index 65f1a6d156a120..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2019-11-17-11-53-10.bpo-3530.8zFUMc.rst +++ /dev/null @@ -1,2 +0,0 @@ -In the :mod:`ast` module documentation, fix a misleading ``NodeTransformer`` example and add -advice on when to use the ``fix_missing_locations`` function. diff --git a/Misc/NEWS.d/next/Documentation/2019-12-15-22-04-41.bpo-38918.8JnDTS.rst b/Misc/NEWS.d/next/Documentation/2019-12-15-22-04-41.bpo-38918.8JnDTS.rst deleted file mode 100644 index 5747936dd64d50..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2019-12-15-22-04-41.bpo-38918.8JnDTS.rst +++ /dev/null @@ -1,3 +0,0 @@ -Add an entry for ``__module__`` in the "function" & "method" sections of the -`inspect docs types and members table -`_ diff --git a/Misc/NEWS.d/next/Documentation/2020-01-18-15-37-56.bpo-39381.wTWe8d.rst b/Misc/NEWS.d/next/Documentation/2020-01-18-15-37-56.bpo-39381.wTWe8d.rst deleted file mode 100644 index 37b66ad9dfd178..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-01-18-15-37-56.bpo-39381.wTWe8d.rst +++ /dev/null @@ -1,2 +0,0 @@ -Mention in docs that :func:`asyncio.get_event_loop` implicitly creates new -event loop only if called from the main thread. diff --git a/Misc/NEWS.d/next/Documentation/2020-01-27-18-18-42.bpo-39392.oiqcLO.rst b/Misc/NEWS.d/next/Documentation/2020-01-27-18-18-42.bpo-39392.oiqcLO.rst deleted file mode 100644 index 715874981f7356..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-01-27-18-18-42.bpo-39392.oiqcLO.rst +++ /dev/null @@ -1 +0,0 @@ -Explain that when filling with turtle, overlap regions may be left unfilled. diff --git a/Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst b/Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst deleted file mode 100644 index 95be00b4b777f6..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst +++ /dev/null @@ -1,5 +0,0 @@ -Clarify refcounting semantics for the following functions: -- PyObject_SetItem -- PyMapping_SetItemString -- PyDict_SetItem -- PyDict_SetItemString \ No newline at end of file diff --git a/Misc/NEWS.d/next/IDLE/2018-03-03-12-56-26.bpo-32989.FVhmhH.rst b/Misc/NEWS.d/next/IDLE/2018-03-03-12-56-26.bpo-32989.FVhmhH.rst deleted file mode 100644 index 38f0fb6e104527..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2018-03-03-12-56-26.bpo-32989.FVhmhH.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add tests for editor newline_and_indent_event method. -Remove dead code from pyparse find_good_parse_start method. diff --git a/Misc/NEWS.d/next/IDLE/2019-11-13-23-51-39.bpo-38792.xhTC5a.rst b/Misc/NEWS.d/next/IDLE/2019-11-13-23-51-39.bpo-38792.xhTC5a.rst deleted file mode 100644 index 9aa2f0ffddfafd..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2019-11-13-23-51-39.bpo-38792.xhTC5a.rst +++ /dev/null @@ -1,2 +0,0 @@ -Close an IDLE shell calltip if a :exc:`KeyboardInterrupt` -or shell restart occurs. Patch by Zackery Spytz. diff --git a/Misc/NEWS.d/next/IDLE/2019-12-30-16-44-07.bpo-34118.FaNW0a.rst b/Misc/NEWS.d/next/IDLE/2019-12-30-16-44-07.bpo-34118.FaNW0a.rst deleted file mode 100644 index ce95eb5482f2bf..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2019-12-30-16-44-07.bpo-34118.FaNW0a.rst +++ /dev/null @@ -1,2 +0,0 @@ -Tag memoryview, range, and tuple as classes, the same as list, etcetera, in -the library manual built-in functions list. diff --git a/Misc/NEWS.d/next/IDLE/2020-01-22-22-28-06.bpo-39050.zkn0FO.rst b/Misc/NEWS.d/next/IDLE/2020-01-22-22-28-06.bpo-39050.zkn0FO.rst deleted file mode 100644 index e71265cdf109b6..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2020-01-22-22-28-06.bpo-39050.zkn0FO.rst +++ /dev/null @@ -1 +0,0 @@ -Make IDLE Settings dialog Help button work again. diff --git a/Misc/NEWS.d/next/IDLE/2020-01-25-02-26-45.bpo-39388.x4TQNh.rst b/Misc/NEWS.d/next/IDLE/2020-01-25-02-26-45.bpo-39388.x4TQNh.rst deleted file mode 100644 index 42bbfb168c19de..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2020-01-25-02-26-45.bpo-39388.x4TQNh.rst +++ /dev/null @@ -1 +0,0 @@ -IDLE Settings Cancel button now cancels pending changes diff --git a/Misc/NEWS.d/next/IDLE/2020-01-27-16-44-29.bpo-30780.nR80qu.rst b/Misc/NEWS.d/next/IDLE/2020-01-27-16-44-29.bpo-30780.nR80qu.rst deleted file mode 100644 index 2f65a00a5af3b4..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2020-01-27-16-44-29.bpo-30780.nR80qu.rst +++ /dev/null @@ -1 +0,0 @@ -Add remaining configdialog tests for buttons and highlights and keys tabs. diff --git a/Misc/NEWS.d/next/Library/2019-09-29-08-17-03.bpo-38293.wls5s3.rst b/Misc/NEWS.d/next/Library/2019-09-29-08-17-03.bpo-38293.wls5s3.rst deleted file mode 100644 index 0b19551970eb09..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-09-29-08-17-03.bpo-38293.wls5s3.rst +++ /dev/null @@ -1 +0,0 @@ -Add :func:`copy.copy` and :func:`copy.deepcopy` support to :func:`property` objects. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2019-10-14-21-14-55.bpo-38473.uXpVld.rst b/Misc/NEWS.d/next/Library/2019-10-14-21-14-55.bpo-38473.uXpVld.rst deleted file mode 100644 index de80e89e00e2d3..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-10-14-21-14-55.bpo-38473.uXpVld.rst +++ /dev/null @@ -1,2 +0,0 @@ -Use signature from inner mock for autospecced methods attached with -:func:`unittest.mock.attach_mock`. Patch by Karthikeyan Singaravelan. diff --git a/Misc/NEWS.d/next/Library/2019-10-31-19-23-25.bpo-35182.hzeNl9.rst b/Misc/NEWS.d/next/Library/2019-10-31-19-23-25.bpo-35182.hzeNl9.rst deleted file mode 100644 index 9438cd8f9fd0ba..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-10-31-19-23-25.bpo-35182.hzeNl9.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fixed :func:`Popen.communicate` subsequent call crash when the child process -has already closed any piped standard stream, but still continues to be -running. Patch by Andriy Maletsky. diff --git a/Misc/NEWS.d/next/Library/2019-11-22-12-08-52.bpo-38878.EJ0cFf.rst b/Misc/NEWS.d/next/Library/2019-11-22-12-08-52.bpo-38878.EJ0cFf.rst deleted file mode 100644 index 9cbdf08dd53e31..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-22-12-08-52.bpo-38878.EJ0cFf.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed __subclasshook__ of :class:`os.PathLike` to return a correct result -upon inheritence. Patch by Bar Harel. diff --git a/Misc/NEWS.d/next/Library/2019-12-13-18-54-49.bpo-39033.cepuyD.rst b/Misc/NEWS.d/next/Library/2019-12-13-18-54-49.bpo-39033.cepuyD.rst deleted file mode 100644 index 3dee3c08cc5bea..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-13-18-54-49.bpo-39033.cepuyD.rst +++ /dev/null @@ -1 +0,0 @@ -Fix :exc:`NameError` in :mod:`zipimport`. Patch by Karthikeyan Singaravelan. diff --git a/Misc/NEWS.d/next/Library/2019-12-15-19-23-23.bpo-39055.FmN3un.rst b/Misc/NEWS.d/next/Library/2019-12-15-19-23-23.bpo-39055.FmN3un.rst deleted file mode 100644 index 83b1431e92fcb2..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-15-19-23-23.bpo-39055.FmN3un.rst +++ /dev/null @@ -1,2 +0,0 @@ -:func:`base64.b64decode` with ``validate=True`` raises now a binascii.Error -if the input ends with a single ``\n``. diff --git a/Misc/NEWS.d/next/Library/2019-12-15-21-05-16.bpo-39056.nEfUM9.rst b/Misc/NEWS.d/next/Library/2019-12-15-21-05-16.bpo-39056.nEfUM9.rst deleted file mode 100644 index d5d2b98e9b0b38..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-15-21-05-16.bpo-39056.nEfUM9.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed handling invalid warning category in the -W option. No longer import -the re module if it is not needed. diff --git a/Misc/NEWS.d/next/Library/2019-12-15-21-47-54.bpo-39057.FOxn-w.rst b/Misc/NEWS.d/next/Library/2019-12-15-21-47-54.bpo-39057.FOxn-w.rst deleted file mode 100644 index 24a17444b97daa..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-15-21-47-54.bpo-39057.FOxn-w.rst +++ /dev/null @@ -1,2 +0,0 @@ -:func:`urllib.request.proxy_bypass_environment` now ignores leading dots and -no longer ignores a trailing newline. diff --git a/Misc/NEWS.d/next/Library/2019-12-24-10-43-13.bpo-39129.jVx5rW.rst b/Misc/NEWS.d/next/Library/2019-12-24-10-43-13.bpo-39129.jVx5rW.rst deleted file mode 100644 index 6667697671a280..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-24-10-43-13.bpo-39129.jVx5rW.rst +++ /dev/null @@ -1 +0,0 @@ -Fix import path for ``asyncio.TimeoutError`` diff --git a/Misc/NEWS.d/next/Library/2019-12-31-19-27-23.bpo-39142.oqU5iD.rst b/Misc/NEWS.d/next/Library/2019-12-31-19-27-23.bpo-39142.oqU5iD.rst deleted file mode 100644 index 508d1338d7c317..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-31-19-27-23.bpo-39142.oqU5iD.rst +++ /dev/null @@ -1,5 +0,0 @@ -A change was made to logging.config.dictConfig to avoid converting instances -of named tuples to ConvertingTuple. It's assumed that named tuples are too -specialised to be treated like ordinary tuples; if a user of named tuples -requires ConvertingTuple functionality, they will have to implement that -themselves in their named tuple class. diff --git a/Misc/NEWS.d/next/Library/2020-01-01-18-44-52.bpo-38871.3EEOLg.rst b/Misc/NEWS.d/next/Library/2020-01-01-18-44-52.bpo-38871.3EEOLg.rst deleted file mode 100644 index fe970fd9e3fa1a..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-01-18-44-52.bpo-38871.3EEOLg.rst +++ /dev/null @@ -1,2 +0,0 @@ -Correctly parenthesize filter-based statements that contain lambda -expressions in mod:`lib2to3`. Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/Library/2020-01-02-17-28-03.bpo-39191.ur_scy.rst b/Misc/NEWS.d/next/Library/2020-01-02-17-28-03.bpo-39191.ur_scy.rst deleted file mode 100644 index 138c93c2e4877a..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-02-17-28-03.bpo-39191.ur_scy.rst +++ /dev/null @@ -1,3 +0,0 @@ -Perform a check for running loop before starting a new task in -``loop.run_until_complete()`` to fail fast; it prevents the side effect of -new task spawning before exception raising. diff --git a/Misc/NEWS.d/next/Library/2020-01-02-20-21-03.bpo-39198.nzwGyG.rst b/Misc/NEWS.d/next/Library/2020-01-02-20-21-03.bpo-39198.nzwGyG.rst deleted file mode 100644 index ec4e81e2bbe4a0..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-02-20-21-03.bpo-39198.nzwGyG.rst +++ /dev/null @@ -1 +0,0 @@ -If an exception were to be thrown in `Logger.isEnabledFor` (say, by asyncio timeouts or stopit) , the `logging` global lock may not be released appropriately, resulting in deadlock. This change wraps that block of code with `try...finally` to ensure the lock is released. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2020-01-03-18-02-50.bpo-39152.JgPjCC.rst b/Misc/NEWS.d/next/Library/2020-01-03-18-02-50.bpo-39152.JgPjCC.rst deleted file mode 100644 index abb3df0da0fe46..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-03-18-02-50.bpo-39152.JgPjCC.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix ttk.Scale.configure([name]) to return configuration tuple for name -or all options. Giovanni Lombardo contributed part of the patch. diff --git a/Misc/NEWS.d/next/Library/2020-01-06-02-14-38.bpo-38907.F1RkCR.rst b/Misc/NEWS.d/next/Library/2020-01-06-02-14-38.bpo-38907.F1RkCR.rst deleted file mode 100644 index a6e79f7809521d..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-06-02-14-38.bpo-38907.F1RkCR.rst +++ /dev/null @@ -1 +0,0 @@ -In http.server script, restore binding to IPv4 on Windows. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2020-01-08-23-25-27.bpo-39242.bnL65N.rst b/Misc/NEWS.d/next/Library/2020-01-08-23-25-27.bpo-39242.bnL65N.rst deleted file mode 100644 index a87dddf81dcd5c..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-08-23-25-27.bpo-39242.bnL65N.rst +++ /dev/null @@ -1,3 +0,0 @@ -Updated the Gmane domain from news.gmane.org to news.gmane.io -which is used for examples of :class:`~nntplib.NNTP` news reader server and -nntplib tests. diff --git a/Misc/NEWS.d/next/Library/2020-01-11-01-15-37.bpo-39297.y98Z6Q.rst b/Misc/NEWS.d/next/Library/2020-01-11-01-15-37.bpo-39297.y98Z6Q.rst deleted file mode 100644 index 618f6f9f2b7ff2..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-11-01-15-37.bpo-39297.y98Z6Q.rst +++ /dev/null @@ -1 +0,0 @@ -Improved performance of importlib.metadata distribution discovery and resilients to inaccessible sys.path entries (importlib_metadata v1.4.0). diff --git a/Misc/NEWS.d/next/Library/2020-01-15-23-13-03.bpo-39274.lpc0-n.rst b/Misc/NEWS.d/next/Library/2020-01-15-23-13-03.bpo-39274.lpc0-n.rst deleted file mode 100644 index 4c398682b98ab1..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-15-23-13-03.bpo-39274.lpc0-n.rst +++ /dev/null @@ -1 +0,0 @@ -``bool(fraction.Fraction)`` now returns a boolean even if (numerator != 0) does not return a boolean (ex: numpy number). diff --git a/Misc/NEWS.d/next/Library/2020-01-20-00-56-01.bpo-39389.fEirIS.rst b/Misc/NEWS.d/next/Library/2020-01-20-00-56-01.bpo-39389.fEirIS.rst deleted file mode 100644 index d4c80506f7d6bb..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-20-00-56-01.bpo-39389.fEirIS.rst +++ /dev/null @@ -1,2 +0,0 @@ -Write accurate compression level metadata in :mod:`gzip` archives, rather -than always signaling maximum compression. diff --git a/Misc/NEWS.d/next/Library/2020-01-23-21-34-29.bpo-39390.D2tSXk.rst b/Misc/NEWS.d/next/Library/2020-01-23-21-34-29.bpo-39390.D2tSXk.rst deleted file mode 100644 index ffa961ea4cd22c..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-23-21-34-29.bpo-39390.D2tSXk.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed a regression with the `ignore` callback of :func:`shutil.copytree`. -The argument types are now str and List[str] again. diff --git a/Misc/NEWS.d/next/Library/2020-01-24-11-05-21.bpo-39430.I0UQzM.rst b/Misc/NEWS.d/next/Library/2020-01-24-11-05-21.bpo-39430.I0UQzM.rst deleted file mode 100644 index 712fc5d34bbe08..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-24-11-05-21.bpo-39430.I0UQzM.rst +++ /dev/null @@ -1 +0,0 @@ -Fixed race condition in lazy imports in :mod:`tarfile`. diff --git a/Misc/NEWS.d/next/Library/2020-01-24-13-24-35.bpo-39082.qKgrq_.rst b/Misc/NEWS.d/next/Library/2020-01-24-13-24-35.bpo-39082.qKgrq_.rst deleted file mode 100644 index 52c4ee1b33bdae..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-24-13-24-35.bpo-39082.qKgrq_.rst +++ /dev/null @@ -1 +0,0 @@ -Allow AsyncMock to correctly patch static/class methods diff --git a/Misc/NEWS.d/next/Library/2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst b/Misc/NEWS.d/next/Library/2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst deleted file mode 100644 index f62c31fc686ad8..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix a bug in :func:`unittest.mock.create_autospec` that would complain about -the wrong number of arguments for custom descriptors defined in an extension -module returning functions. diff --git a/Misc/NEWS.d/next/Library/2020-01-30-01-13-19.bpo-39493.CbFRi7.rst b/Misc/NEWS.d/next/Library/2020-01-30-01-13-19.bpo-39493.CbFRi7.rst deleted file mode 100644 index b676629a4414a2..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-30-01-13-19.bpo-39493.CbFRi7.rst +++ /dev/null @@ -1 +0,0 @@ -Mark ``typing.IO.closed`` as a property \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst b/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst deleted file mode 100644 index 55fed519a2d806..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-02-02-14-46-34.bpo-39450.48R274.rst +++ /dev/null @@ -1,2 +0,0 @@ -Striped whitespace from docstring before returning it from -:func:`unittest.case.shortDescription`. diff --git a/Misc/NEWS.d/next/Library/2020-02-05-11-24-16.bpo-38149.GWsjHE.rst b/Misc/NEWS.d/next/Library/2020-02-05-11-24-16.bpo-38149.GWsjHE.rst deleted file mode 100644 index b4ec60b2abad14..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-02-05-11-24-16.bpo-38149.GWsjHE.rst +++ /dev/null @@ -1,2 +0,0 @@ -:func:`sys.audit` is now called only once per call of :func:`glob.glob` and -:func:`glob.iglob`. diff --git a/Misc/NEWS.d/next/Library/2020-02-09-05-51-05.bpo-39590.rf98GU.rst b/Misc/NEWS.d/next/Library/2020-02-09-05-51-05.bpo-39590.rf98GU.rst deleted file mode 100644 index 68625028fb7afc..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-02-09-05-51-05.bpo-39590.rf98GU.rst +++ /dev/null @@ -1 +0,0 @@ -Collections.deque now holds strong references during deque.__contains__ and deque.count, fixing crashes. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Security/2020-01-07-00-42-08.bpo-39184.fe7NgK.rst b/Misc/NEWS.d/next/Security/2020-01-07-00-42-08.bpo-39184.fe7NgK.rst deleted file mode 100644 index 1ab5d4d70eec5a..00000000000000 --- a/Misc/NEWS.d/next/Security/2020-01-07-00-42-08.bpo-39184.fe7NgK.rst +++ /dev/null @@ -1 +0,0 @@ -Add audit events to command execution functions in os and pty modules. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Security/2020-01-28-20-54-09.bpo-39401.he7h_A.rst b/Misc/NEWS.d/next/Security/2020-01-28-20-54-09.bpo-39401.he7h_A.rst deleted file mode 100644 index 5071e126b70d02..00000000000000 --- a/Misc/NEWS.d/next/Security/2020-01-28-20-54-09.bpo-39401.he7h_A.rst +++ /dev/null @@ -1 +0,0 @@ -Avoid unsafe load of ``api-ms-win-core-path-l1-1-0.dll`` at startup on Windows 7. diff --git a/Misc/NEWS.d/next/Tests/2019-12-18-14-52-08.bpo-38546.2kxNuM.rst b/Misc/NEWS.d/next/Tests/2019-12-18-14-52-08.bpo-38546.2kxNuM.rst deleted file mode 100644 index d8ec7cabbbab82..00000000000000 --- a/Misc/NEWS.d/next/Tests/2019-12-18-14-52-08.bpo-38546.2kxNuM.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix test_ressources_gced_in_workers() of test_concurrent_futures: explicitly -stop the manager to prevent leaking a child process running in the background -after the test completes. diff --git a/Misc/NEWS.d/next/Tests/2020-01-30-15-04-54.bpo-39502.chbpII.rst b/Misc/NEWS.d/next/Tests/2020-01-30-15-04-54.bpo-39502.chbpII.rst deleted file mode 100644 index 0a13746e34759b..00000000000000 --- a/Misc/NEWS.d/next/Tests/2020-01-30-15-04-54.bpo-39502.chbpII.rst +++ /dev/null @@ -1,2 +0,0 @@ -Skip test_zipfile.test_add_file_after_2107() if :func:`time.localtime` fails -with :exc:`OverflowError`. It is the case on AIX 6.1 for example. diff --git a/Misc/NEWS.d/next/Windows/2020-01-02-01-11-53.bpo-39185.T4herN.rst b/Misc/NEWS.d/next/Windows/2020-01-02-01-11-53.bpo-39185.T4herN.rst deleted file mode 100644 index 3b84bd52172648..00000000000000 --- a/Misc/NEWS.d/next/Windows/2020-01-02-01-11-53.bpo-39185.T4herN.rst +++ /dev/null @@ -1 +0,0 @@ -The build.bat script has additional options for very-quiet output (-q) and very-verbose output (-vv) \ No newline at end of file diff --git a/Misc/NEWS.d/next/Windows/2020-01-11-22-53-55.bpo-38883.X7FRaN.rst b/Misc/NEWS.d/next/Windows/2020-01-11-22-53-55.bpo-38883.X7FRaN.rst deleted file mode 100644 index c552e850a36840..00000000000000 --- a/Misc/NEWS.d/next/Windows/2020-01-11-22-53-55.bpo-38883.X7FRaN.rst +++ /dev/null @@ -1,5 +0,0 @@ -:meth:`~pathlib.Path.home()` and :meth:`~pathlib.Path.expanduser()` on Windows -now prefer :envvar:`USERPROFILE` and no longer use :envvar:`HOME`, which is not -normally set for regular user accounts. This makes them again behave like -:func:`os.path.expanduser`, which was changed to ignore :envvar:`HOME` in 3.8, -see :issue:`36264`. diff --git a/Misc/NEWS.d/next/Windows/2020-01-20-23-42-53.bpo-39393.gWlJDG.rst b/Misc/NEWS.d/next/Windows/2020-01-20-23-42-53.bpo-39393.gWlJDG.rst deleted file mode 100644 index 025b7e96a6e743..00000000000000 --- a/Misc/NEWS.d/next/Windows/2020-01-20-23-42-53.bpo-39393.gWlJDG.rst +++ /dev/null @@ -1,2 +0,0 @@ -Improve the error message when attempting to load a DLL with unresolved -dependencies. diff --git a/Misc/NEWS.d/next/Windows/2020-01-24-03-15-05.bpo-39439.sFxGfR.rst b/Misc/NEWS.d/next/Windows/2020-01-24-03-15-05.bpo-39439.sFxGfR.rst deleted file mode 100644 index d677c4c3e02d50..00000000000000 --- a/Misc/NEWS.d/next/Windows/2020-01-24-03-15-05.bpo-39439.sFxGfR.rst +++ /dev/null @@ -1 +0,0 @@ -Honor the Python path when a virtualenv is active on Windows. \ No newline at end of file diff --git a/README.rst b/README.rst index be371c64980794..0b4695ce0e509f 100644 --- a/README.rst +++ b/README.rst @@ -1,5 +1,5 @@ -This is Python version 3.8.1 -============================ +This is Python version 3.8.2rc1 +=============================== .. image:: https://travis-ci.org/python/cpython.svg?branch=3.8 :alt: CPython build status on Travis CI From a83d9108066a7a22605ba7ee962a755f4bec1ab1 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 10 Feb 2020 13:24:53 -0800 Subject: [PATCH 1169/2163] Issue3950: Fix docs for default locale used by gettext to match implementation (GH-18435) documentation for default locale directory Doc/library/gettext.rst changed to match gettext implementation line 63. (cherry picked from commit d68e0a8a165761604e820c8cb4f20abc735e717f) Co-authored-by: Carl --- Doc/library/gettext.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/gettext.rst b/Doc/library/gettext.rst index 937330bb201b08..ec2c12806b4160 100644 --- a/Doc/library/gettext.rst +++ b/Doc/library/gettext.rst @@ -724,8 +724,8 @@ implementations, and valuable experience to the creation of this module: .. [#] The default locale directory is system dependent; for example, on RedHat Linux it is :file:`/usr/share/locale`, but on Solaris it is :file:`/usr/lib/locale`. The :mod:`gettext` module does not try to support these system dependent - defaults; instead its default is :file:`{sys.prefix}/share/locale` (see - :data:`sys.prefix`). For this reason, it is always best to call + defaults; instead its default is :file:`{sys.base_prefix}/share/locale` (see + :data:`sys.base_prefix`). For this reason, it is always best to call :func:`bindtextdomain` with an explicit absolute path at the start of your application. From 021a5694ede9d7be119f9ceb3ee7e8e518ec5002 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 10 Feb 2020 13:38:30 -0800 Subject: [PATCH 1170/2163] bpo-39600, IDLE: Remove duplicated font names (GH-18430) In the font configuration window, remove duplicated font names. (cherry picked from commit ed335cf53b5d4bca9a08c9b83ba684ba17be0f10) Co-authored-by: Victor Stinner --- Lib/idlelib/configdialog.py | 5 +++-- .../next/IDLE/2020-02-10-17-09-48.bpo-39600.X6NsyM.rst | 1 + 2 files changed, 4 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/IDLE/2020-02-10-17-09-48.bpo-39600.X6NsyM.rst diff --git a/Lib/idlelib/configdialog.py b/Lib/idlelib/configdialog.py index 22359735874d19..7b844f00e7736d 100644 --- a/Lib/idlelib/configdialog.py +++ b/Lib/idlelib/configdialog.py @@ -607,8 +607,9 @@ def load_font_cfg(self): font_bold = configured_font[2]=='bold' # Set editor font selection list and font_name. - fonts = list(tkFont.families(self)) - fonts.sort() + fonts = tkFont.families(self) + # remove duplicated names and sort + fonts = sorted(set(fonts)) for font in fonts: self.fontlist.insert(END, font) self.font_name.set(font_name) diff --git a/Misc/NEWS.d/next/IDLE/2020-02-10-17-09-48.bpo-39600.X6NsyM.rst b/Misc/NEWS.d/next/IDLE/2020-02-10-17-09-48.bpo-39600.X6NsyM.rst new file mode 100644 index 00000000000000..102aa75f5813e0 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2020-02-10-17-09-48.bpo-39600.X6NsyM.rst @@ -0,0 +1 @@ +In the font configuration window, remove duplicated font names. From 8ef9e6d59aedfd91cbd42b421c34a6c935ef7b25 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 10 Feb 2020 14:37:46 -0800 Subject: [PATCH 1171/2163] Improve grammar in the import system reference documentation (GH-18209) Replaced the period with a comma. Automerge-Triggered-By: @Mariatta (cherry picked from commit d47d0c8e9f2ca0f9f5d1bf0b35006a9a4d5ca684) Co-authored-by: Bonifacio de Oliveira --- Doc/reference/import.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index c6f6d030d2ac3f..1c98aab7d83aad 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -855,7 +855,7 @@ module. ``find_spec()`` returns a fully populated spec for the module. This spec will always have "loader" set (with one exception). To indicate to the import machinery that the spec represents a namespace -:term:`portion`. the path entry finder sets "loader" on the spec to +:term:`portion`, the path entry finder sets "loader" on the spec to ``None`` and "submodule_search_locations" to a list containing the portion. From e6690f6cd1b0c2bd5804bad30239a4070f79102c Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 10 Feb 2020 14:57:06 -0800 Subject: [PATCH 1172/2163] bpo-13826: Clarify Popen constructor example (GH-18438) Clarifies that the use of `shlex.split` is more instructive than normative, and provides a simpler example. https://bugs.python.org/issue13826 (cherry picked from commit 95d024d585bd3ed627437a2f0cbc783c8a014c8a) Co-authored-by: Tim D. Smith --- Doc/library/subprocess.rst | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index ea12cd133a6ffc..cce7da1c9b1669 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -355,14 +355,20 @@ functions. arguments for additional differences from the default behavior. Unless otherwise stated, it is recommended to pass *args* as a sequence. + An example of passing some arguments to an external program + as a sequence is:: + + Popen(["/usr/bin/git", "commit", "-m", "Fixes a bug."]) + On POSIX, if *args* is a string, the string is interpreted as the name or path of the program to execute. However, this can only be done if not passing arguments to the program. .. note:: - :meth:`shlex.split` can be useful when determining the correct - tokenization for *args*, especially in complex cases:: + It may not be obvious how to break a shell command into a sequence of arguments, + especially in complex cases. :meth:`shlex.split` can illustrate how to + determine the correct tokenization for *args*:: >>> import shlex, subprocess >>> command_line = input() From 4d4301782cbc789eedc5b76741d1028df579cfa5 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 10 Feb 2020 14:57:14 -0800 Subject: [PATCH 1173/2163] bpo-39594: Fix typo in os.times documentation (GH-18443) There was an extra space in the url markup, causing the documentation not rendered properly. https://bugs.python.org/issue39594 (cherry picked from commit 37c55b2b49a3acb7c56c9f6a5062bc6e4e35bc1c) Co-authored-by: Roger Hurwitz --- Doc/library/os.rst | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 3a828c5e08563a..0d8df34c345c4d 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -3898,10 +3898,8 @@ written in Python, such as a mail server's external command delivery program. See the Unix manual page :manpage:`times(2)` and :manpage:`times(3)` manual page on Unix or `the GetProcessTimes MSDN - ` - _ on Windows. - On Windows, only :attr:`user` and :attr:`system` are known; the other - attributes are zero. + `_ + on Windows. On Windows, only :attr:`user` and :attr:`system` are known; the other attributes are zero. .. availability:: Unix, Windows. From c372f9b9e758a22608b8df33423b7413d224fdad Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 10 Feb 2020 17:27:31 -0800 Subject: [PATCH 1174/2163] bpo-39600: Adjust code, add idlelib/NEWS item (GH-18449) Complete previous patch. (cherry picked from commit 96ce22706735779cf8cc46eaaa5ac61359364b5a) Co-authored-by: Terry Jan Reedy --- Lib/idlelib/NEWS.txt | 2 ++ Lib/idlelib/configdialog.py | 6 ++---- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 1fc9e0f3090f65..9cf563401259d1 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,8 @@ Released on 2019-12-16? ====================================== +bpo-39600: Remove duplicate font names from configuration list. + bpo-38792: Close a shell calltip if a :exc:`KeyboardInterrupt` or shell restart occurs. Patch by Zackery Spytz. diff --git a/Lib/idlelib/configdialog.py b/Lib/idlelib/configdialog.py index 7b844f00e7736d..9d5c2cde04b240 100644 --- a/Lib/idlelib/configdialog.py +++ b/Lib/idlelib/configdialog.py @@ -606,10 +606,8 @@ def load_font_cfg(self): font_size = configured_font[1] font_bold = configured_font[2]=='bold' - # Set editor font selection list and font_name. - fonts = tkFont.families(self) - # remove duplicated names and sort - fonts = sorted(set(fonts)) + # Set sorted no-duplicate editor font selection list and font_name. + fonts = sorted(set(tkFont.families(self))) for font in fonts: self.fontlist.insert(END, font) self.font_name.set(font_name) From 3f8d181446aa9d87e3773896c2695161ea8f6e42 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 10 Feb 2020 23:02:01 -0800 Subject: [PATCH 1175/2163] bpo-38374: Remove weakref.ReferenceError from docs (GH-18452) Reflecting changes to the code, removed weakref.ReferenceError from weakref.rst and exceptions.rst. Issue submitter provided evidence that the `weakref.ReferenceError` alias for `ReferenceError` was removed from the code in 2007. Working with @gvanrossum at PyCascades CPython sprint we looked at the code and confirmed that `weakref.ReferenceError` was no longer in `weakref.py`. Based on that analysis I removed references `weakref.ReferenceError` from the two documents where it was still being referenced: `weakref.rst` and `exceptions.rst`. https://bugs.python.org/issue38374 (cherry picked from commit 4eb9f4313cfaea6a9611221024a1c54f5662cc37) Co-authored-by: Roger Hurwitz --- Doc/c-api/exceptions.rst | 3 --- Doc/library/weakref.rst | 6 ------ 2 files changed, 9 deletions(-) diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index c7ba74cc8d5874..7300c802e0817f 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -971,9 +971,6 @@ Notes: This is a base class for other standard exceptions. (2) - This is the same as :exc:`weakref.ReferenceError`. - -(3) Only defined on Windows; protect code that uses this by testing that the preprocessor macro ``MS_WINDOWS`` is defined. diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst index b3c8e354553311..2dbe5e33bd0117 100644 --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -327,12 +327,6 @@ objects. types. -.. exception:: ReferenceError - - Exception raised when a proxy object is used but the underlying object has been - collected. This is the same as the standard :exc:`ReferenceError` exception. - - .. seealso:: :pep:`205` - Weak References From 40ef4fc07e64be93522bc705c3d8a954a62b9ded Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Tue, 11 Feb 2020 13:38:43 +0100 Subject: [PATCH 1176/2163] Post 3.8.2rc1 --- Include/patchlevel.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h index a40ef8232ece9b..506f43c3e0adc4 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 1 /* Version as a string */ -#define PY_VERSION "3.8.2rc1" +#define PY_VERSION "3.8.2rc1+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. From d3f9fb2d28ceedb0a17a703338424ff284a578c8 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 11 Feb 2020 07:32:40 -0800 Subject: [PATCH 1177/2163] bpo-39299: Add more tests for mimetypes and its cli. (GH-17949) * Add tests for case insensitive check of types and extensions as fallback. * Add tests for data url with no comma. * Add tests for read_mime_types. * Add tests for the mimetypes cli and refactor __main__ code to private function. * Restore mimetypes.knownfiles value at the end of the test. (cherry picked from commit d8efc1495194228c3a4cd472200275d6491d8e2d) Co-authored-by: Karthikeyan Singaravelan --- Lib/mimetypes.py | 6 ++- Lib/test/test_mimetypes.py | 84 ++++++++++++++++++++++++++++++++++++-- 2 files changed, 85 insertions(+), 5 deletions(-) diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py index 9b42bf6dd2ca74..f33b658f10e5eb 100644 --- a/Lib/mimetypes.py +++ b/Lib/mimetypes.py @@ -563,7 +563,7 @@ def _default_mime_types(): _default_mime_types() -if __name__ == '__main__': +def _main(): import getopt USAGE = """\ @@ -607,3 +607,7 @@ def usage(code, msg=''): guess, encoding = guess_type(gtype, strict) if not guess: print("I don't know anything about type", gtype) else: print('type:', guess, 'encoding:', encoding) + + +if __name__ == '__main__': + _main() diff --git a/Lib/test/test_mimetypes.py b/Lib/test/test_mimetypes.py index a5a06b189dec4f..9cac6ce0225e1c 100644 --- a/Lib/test/test_mimetypes.py +++ b/Lib/test/test_mimetypes.py @@ -8,10 +8,20 @@ from test import support from platform import win32_edition -# Tell it we don't know about external files: -mimetypes.knownfiles = [] -mimetypes.inited = False -mimetypes._default_mime_types() + +def setUpModule(): + global knownfiles + knownfiles = mimetypes.knownfiles + + # Tell it we don't know about external files: + mimetypes.knownfiles = [] + mimetypes.inited = False + mimetypes._default_mime_types() + + +def tearDownModule(): + # Restore knownfiles to its initial state + mimetypes.knownfiles = knownfiles class MimeTypesTestCase(unittest.TestCase): @@ -21,6 +31,7 @@ def setUp(self): def test_default_data(self): eq = self.assertEqual eq(self.db.guess_type("foo.html"), ("text/html", None)) + eq(self.db.guess_type("foo.HTML"), ("text/html", None)) eq(self.db.guess_type("foo.tgz"), ("application/x-tar", "gzip")) eq(self.db.guess_type("foo.tar.gz"), ("application/x-tar", "gzip")) eq(self.db.guess_type("foo.tar.Z"), ("application/x-tar", "compress")) @@ -30,6 +41,7 @@ def test_default_data(self): def test_data_urls(self): eq = self.assertEqual guess_type = self.db.guess_type + eq(guess_type("data:invalidDataWithoutComma"), (None, None)) eq(guess_type("data:,thisIsTextPlain"), ("text/plain", None)) eq(guess_type("data:;base64,thisIsTextPlain"), ("text/plain", None)) eq(guess_type("data:text/x-foo,thisIsTextXFoo"), ("text/x-foo", None)) @@ -42,6 +54,19 @@ def test_file_parsing(self): ("x-application/x-unittest", None)) eq(self.db.guess_extension("x-application/x-unittest"), ".pyunit") + def test_read_mime_types(self): + eq = self.assertEqual + + # Unreadable file returns None + self.assertIsNone(mimetypes.read_mime_types("non-existent")) + + with support.temp_dir() as directory: + data = "x-application/x-unittest pyunit\n" + file = pathlib.Path(directory, "sample.mimetype") + file.write_text(data) + mime_dict = mimetypes.read_mime_types(file) + eq(mime_dict[".pyunit"], "x-application/x-unittest") + def test_non_standard_types(self): eq = self.assertEqual # First try strict @@ -49,7 +74,10 @@ def test_non_standard_types(self): eq(self.db.guess_extension('image/jpg', strict=True), None) # And then non-strict eq(self.db.guess_type('foo.xul', strict=False), ('text/xul', None)) + eq(self.db.guess_type('foo.XUL', strict=False), ('text/xul', None)) + eq(self.db.guess_type('foo.invalid', strict=False), (None, None)) eq(self.db.guess_extension('image/jpg', strict=False), '.jpg') + eq(self.db.guess_extension('image/JPG', strict=False), '.jpg') def test_filename_with_url_delimiters(self): # bpo-38449: URL delimiters cases should be handled also. @@ -200,5 +228,53 @@ def test__all__(self): support.check__all__(self, mimetypes) +class MimetypesCliTestCase(unittest.TestCase): + + def mimetypes_cmd(self, *args, **kwargs): + support.patch(self, sys, "argv", [sys.executable, *args]) + with support.captured_stdout() as output: + mimetypes._main() + return output.getvalue().strip() + + def test_help_option(self): + support.patch(self, sys, "argv", [sys.executable, "-h"]) + with support.captured_stdout() as output: + with self.assertRaises(SystemExit) as cm: + mimetypes._main() + + self.assertIn("Usage: mimetypes.py", output.getvalue()) + self.assertEqual(cm.exception.code, 0) + + def test_invalid_option(self): + support.patch(self, sys, "argv", [sys.executable, "--invalid"]) + with support.captured_stdout() as output: + with self.assertRaises(SystemExit) as cm: + mimetypes._main() + + self.assertIn("Usage: mimetypes.py", output.getvalue()) + self.assertEqual(cm.exception.code, 1) + + def test_guess_extension(self): + eq = self.assertEqual + + extension = self.mimetypes_cmd("-l", "-e", "image/jpg") + eq(extension, ".jpg") + + extension = self.mimetypes_cmd("-e", "image/jpg") + eq(extension, "I don't know anything about type image/jpg") + + extension = self.mimetypes_cmd("-e", "image/jpeg") + eq(extension, ".jpg") + + def test_guess_type(self): + eq = self.assertEqual + + type_info = self.mimetypes_cmd("-l", "foo.pic") + eq(type_info, "type: image/pict encoding: None") + + type_info = self.mimetypes_cmd("foo.pic") + eq(type_info, "I don't know anything about type foo.pic") + + if __name__ == "__main__": unittest.main() From 0f0d2e496205c345b182b6572ee09db23f8f9daf Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 11 Feb 2020 17:39:33 -0800 Subject: [PATCH 1178/2163] docs: macos - change "versiona" to "versions" (GH-18467) (GH-18469) (cherry picked from commit 029e8401b7741cc0964b5f38d2c2264749dbff6b) Co-authored-by: @RandyMcMillan --- Mac/README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Mac/README.rst b/Mac/README.rst index 4f2e2ce6623df7..ec7d873df277d7 100644 --- a/Mac/README.rst +++ b/Mac/README.rst @@ -49,7 +49,7 @@ macOS specific arguments to configure system header files in their traditional locations, like ``/usr/include`` and ``/System/Library/Frameworks``; instead they are found within a MacOSX SDK. The Apple-supplied build tools handle this transparently and current - versiona of Python now handle this as well. So it is no longer necessary, + versions of Python now handle this as well. So it is no longer necessary, and since macOS 10.14, no longer possible to force the installation of system headers with ``xcode-select``. From 190433d8150bf719fa0ba972dbacf2214942f54e Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 11 Feb 2020 18:47:20 -0800 Subject: [PATCH 1179/2163] closes bpo-39605: Fix some casts to not cast away const. (GH-18453) gcc -Wcast-qual turns up a number of instances of casting away constness of pointers. Some of these can be safely modified, by either: Adding the const to the type cast, as in: - return _PyUnicode_FromUCS1((unsigned char*)s, size); + return _PyUnicode_FromUCS1((const unsigned char*)s, size); or, Removing the cast entirely, because it's not necessary (but probably was at one time), as in: - PyDTrace_FUNCTION_ENTRY((char *)filename, (char *)funcname, lineno); + PyDTrace_FUNCTION_ENTRY(filename, funcname, lineno); These changes will not change code, but they will make it much easier to check for errors in consts (cherry picked from commit e6be9b59a911626d6597fe148c32f0342bd2bd24) Co-authored-by: Andy Lester --- Modules/_io/textio.c | 4 ++-- Objects/bytes_methods.c | 16 ++++++++-------- Objects/memoryobject.c | 8 ++++---- Objects/stringlib/asciilib.h | 2 +- Objects/stringlib/codecs.h | 4 ++-- Objects/stringlib/find_max_char.h | 2 +- Objects/unicodeobject.c | 30 +++++++++++++++--------------- Python/marshal.c | 2 +- Python/pyhash.c | 2 +- 9 files changed, 35 insertions(+), 35 deletions(-) diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c index 2f4a5245681f0b..a4bf7cd1076707 100644 --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -1985,7 +1985,7 @@ find_control_char(int kind, const char *s, const char *end, Py_UCS4 ch) { if (kind == PyUnicode_1BYTE_KIND) { assert(ch < 256); - return (char *) memchr((void *) s, (char) ch, end - s); + return (char *) memchr((const void *) s, (char) ch, end - s); } for (;;) { while (PyUnicode_READ(kind, s, 0) > ch) @@ -2003,7 +2003,7 @@ _PyIO_find_line_ending( int translated, int universal, PyObject *readnl, int kind, const char *start, const char *end, Py_ssize_t *consumed) { - Py_ssize_t len = ((char*)end - (char*)start)/kind; + Py_ssize_t len = (end - start)/kind; if (translated) { /* Newlines are already translated, only search for \n */ diff --git a/Objects/bytes_methods.c b/Objects/bytes_methods.c index 7d131842059228..db030be4fe7561 100644 --- a/Objects/bytes_methods.c +++ b/Objects/bytes_methods.c @@ -12,7 +12,7 @@ PyObject* _Py_bytes_isspace(const char *cptr, Py_ssize_t len) { const unsigned char *p - = (unsigned char *) cptr; + = (const unsigned char *) cptr; const unsigned char *e; /* Shortcut for single character strings */ @@ -42,7 +42,7 @@ PyObject* _Py_bytes_isalpha(const char *cptr, Py_ssize_t len) { const unsigned char *p - = (unsigned char *) cptr; + = (const unsigned char *) cptr; const unsigned char *e; /* Shortcut for single character strings */ @@ -72,7 +72,7 @@ PyObject* _Py_bytes_isalnum(const char *cptr, Py_ssize_t len) { const unsigned char *p - = (unsigned char *) cptr; + = (const unsigned char *) cptr; const unsigned char *e; /* Shortcut for single character strings */ @@ -123,7 +123,7 @@ _Py_bytes_isascii(const char *cptr, Py_ssize_t len) /* Help allocation */ const char *_p = p; while (_p < aligned_end) { - unsigned long value = *(unsigned long *) _p; + unsigned long value = *(const unsigned long *) _p; if (value & ASCII_CHAR_MASK) { Py_RETURN_FALSE; } @@ -154,7 +154,7 @@ PyObject* _Py_bytes_isdigit(const char *cptr, Py_ssize_t len) { const unsigned char *p - = (unsigned char *) cptr; + = (const unsigned char *) cptr; const unsigned char *e; /* Shortcut for single character strings */ @@ -184,7 +184,7 @@ PyObject* _Py_bytes_islower(const char *cptr, Py_ssize_t len) { const unsigned char *p - = (unsigned char *) cptr; + = (const unsigned char *) cptr; const unsigned char *e; int cased; @@ -218,7 +218,7 @@ PyObject* _Py_bytes_isupper(const char *cptr, Py_ssize_t len) { const unsigned char *p - = (unsigned char *) cptr; + = (const unsigned char *) cptr; const unsigned char *e; int cased; @@ -254,7 +254,7 @@ PyObject* _Py_bytes_istitle(const char *cptr, Py_ssize_t len) { const unsigned char *p - = (unsigned char *) cptr; + = (const unsigned char *) cptr; const unsigned char *e; int cased, previous_is_cased; diff --git a/Objects/memoryobject.c b/Objects/memoryobject.c index a873ac1ec1ea5f..92050791b5d167 100644 --- a/Objects/memoryobject.c +++ b/Objects/memoryobject.c @@ -1681,8 +1681,8 @@ unpack_single(const char *ptr, const char *fmt) switch (fmt[0]) { /* signed integers and fast path for 'B' */ - case 'B': uc = *((unsigned char *)ptr); goto convert_uc; - case 'b': ld = *((signed char *)ptr); goto convert_ld; + case 'B': uc = *((const unsigned char *)ptr); goto convert_uc; + case 'b': ld = *((const signed char *)ptr); goto convert_ld; case 'h': UNPACK_SINGLE(ld, ptr, short); goto convert_ld; case 'i': UNPACK_SINGLE(ld, ptr, int); goto convert_ld; case 'l': UNPACK_SINGLE(ld, ptr, long); goto convert_ld; @@ -2683,8 +2683,8 @@ unpack_cmp(const char *p, const char *q, char fmt, switch (fmt) { /* signed integers and fast path for 'B' */ - case 'B': return *((unsigned char *)p) == *((unsigned char *)q); - case 'b': return *((signed char *)p) == *((signed char *)q); + case 'B': return *((const unsigned char *)p) == *((const unsigned char *)q); + case 'b': return *((const signed char *)p) == *((const signed char *)q); case 'h': CMP_SINGLE(p, q, short); return equal; case 'i': CMP_SINGLE(p, q, int); return equal; case 'l': CMP_SINGLE(p, q, long); return equal; diff --git a/Objects/stringlib/asciilib.h b/Objects/stringlib/asciilib.h index d0fc18d22fa5f3..8f83614488e844 100644 --- a/Objects/stringlib/asciilib.h +++ b/Objects/stringlib/asciilib.h @@ -18,7 +18,7 @@ #define STRINGLIB_TODECIMAL Py_UNICODE_TODECIMAL #define STRINGLIB_STR PyUnicode_1BYTE_DATA #define STRINGLIB_LEN PyUnicode_GET_LENGTH -#define STRINGLIB_NEW(STR,LEN) _PyUnicode_FromASCII((char*)(STR),(LEN)) +#define STRINGLIB_NEW(STR,LEN) _PyUnicode_FromASCII((const char*)(STR),(LEN)) #define STRINGLIB_CHECK PyUnicode_Check #define STRINGLIB_CHECK_EXACT PyUnicode_CheckExact diff --git a/Objects/stringlib/codecs.h b/Objects/stringlib/codecs.h index d6f2b98f2b30a3..269a5581f70055 100644 --- a/Objects/stringlib/codecs.h +++ b/Objects/stringlib/codecs.h @@ -46,7 +46,7 @@ STRINGLIB(utf8_decode)(const char **inptr, const char *end, /* Read a whole long at a time (either 4 or 8 bytes), and do a fast unrolled copy if it only contains ASCII characters. */ - unsigned long value = *(unsigned long *) _s; + unsigned long value = *(const unsigned long *) _s; if (value & ASCII_CHAR_MASK) break; #if PY_LITTLE_ENDIAN @@ -515,7 +515,7 @@ STRINGLIB(utf16_decode)(const unsigned char **inptr, const unsigned char *e, /* Fast path for runs of in-range non-surrogate chars. */ const unsigned char *_q = q; while (_q < aligned_end) { - unsigned long block = * (unsigned long *) _q; + unsigned long block = * (const unsigned long *) _q; if (native_ordering) { /* Can use buffer directly */ if (block & FAST_CHAR_MASK) diff --git a/Objects/stringlib/find_max_char.h b/Objects/stringlib/find_max_char.h index 8ccbc3094463df..f4e0a7761d3119 100644 --- a/Objects/stringlib/find_max_char.h +++ b/Objects/stringlib/find_max_char.h @@ -28,7 +28,7 @@ STRINGLIB(find_max_char)(const STRINGLIB_CHAR *begin, const STRINGLIB_CHAR *end) /* Help register allocation */ const unsigned char *_p = p; while (_p < aligned_end) { - unsigned long value = *(unsigned long *) _p; + unsigned long value = *(const unsigned long *) _p; if (value & UCS1_ASCII_CHAR_MASK) return 255; _p += SIZEOF_LONG; diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 0556eff8d07132..8ba379e8887e98 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -171,8 +171,8 @@ extern "C" { #define _PyUnicode_CONVERT_BYTES(from_type, to_type, begin, end, to) \ do { \ to_type *_to = (to_type *)(to); \ - const from_type *_iter = (from_type *)(begin); \ - const from_type *_end = (from_type *)(end); \ + const from_type *_iter = (const from_type *)(begin);\ + const from_type *_end = (const from_type *)(end);\ Py_ssize_t n = (_end) - (_iter); \ const from_type *_unrolled_end = \ _iter + _Py_SIZE_ROUND_DOWN(n, 4); \ @@ -919,21 +919,21 @@ findchar(const void *s, int kind, if ((Py_UCS1) ch != ch) return -1; if (direction > 0) - return ucs1lib_find_char((Py_UCS1 *) s, size, (Py_UCS1) ch); + return ucs1lib_find_char((const Py_UCS1 *) s, size, (Py_UCS1) ch); else - return ucs1lib_rfind_char((Py_UCS1 *) s, size, (Py_UCS1) ch); + return ucs1lib_rfind_char((const Py_UCS1 *) s, size, (Py_UCS1) ch); case PyUnicode_2BYTE_KIND: if ((Py_UCS2) ch != ch) return -1; if (direction > 0) - return ucs2lib_find_char((Py_UCS2 *) s, size, (Py_UCS2) ch); + return ucs2lib_find_char((const Py_UCS2 *) s, size, (Py_UCS2) ch); else - return ucs2lib_rfind_char((Py_UCS2 *) s, size, (Py_UCS2) ch); + return ucs2lib_rfind_char((const Py_UCS2 *) s, size, (Py_UCS2) ch); case PyUnicode_4BYTE_KIND: if (direction > 0) - return ucs4lib_find_char((Py_UCS4 *) s, size, ch); + return ucs4lib_find_char((const Py_UCS4 *) s, size, ch); else - return ucs4lib_rfind_char((Py_UCS4 *) s, size, ch); + return ucs4lib_rfind_char((const Py_UCS4 *) s, size, ch); default: Py_UNREACHABLE(); } @@ -3353,7 +3353,7 @@ PyUnicode_Decode(const char *s, /* Decode via the codec registry */ buffer = NULL; - if (PyBuffer_FillInfo(&info, NULL, (void *)s, size, 1, PyBUF_FULL_RO) < 0) + if (PyBuffer_FillInfo(&info, NULL, (const void *)s, size, 1, PyBUF_FULL_RO) < 0) goto onError; buffer = PyMemoryView_FromBuffer(&info); if (buffer == NULL) @@ -4861,7 +4861,7 @@ ascii_decode(const char *start, const char *end, Py_UCS1 *dest) /* Help allocation */ const char *_p = p; while (_p < aligned_end) { - unsigned long value = *(unsigned long *) _p; + unsigned long value = *(const unsigned long *) _p; if (value & ASCII_CHAR_MASK) break; _p += SIZEOF_LONG; @@ -5406,7 +5406,7 @@ PyUnicode_DecodeUTF32Stateful(const char *s, PyObject *errorHandler = NULL; PyObject *exc = NULL; - q = (unsigned char *)s; + q = (const unsigned char *)s; e = q + size; if (byteorder) @@ -5731,7 +5731,7 @@ PyUnicode_DecodeUTF16Stateful(const char *s, PyObject *exc = NULL; const char *encoding; - q = (unsigned char *)s; + q = (const unsigned char *)s; e = q + size; if (byteorder) @@ -6660,7 +6660,7 @@ PyUnicode_DecodeLatin1(const char *s, const char *errors) { /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */ - return _PyUnicode_FromUCS1((unsigned char*)s, size); + return _PyUnicode_FromUCS1((const unsigned char*)s, size); } /* create or adjust a UnicodeEncodeError */ @@ -13708,7 +13708,7 @@ _PyUnicodeWriter_WriteASCIIString(_PyUnicodeWriter *writer, if (len == -1) len = strlen(ascii); - assert(ucs1lib_find_max_char((Py_UCS1*)ascii, (Py_UCS1*)ascii + len) < 128); + assert(ucs1lib_find_max_char((const Py_UCS1*)ascii, (const Py_UCS1*)ascii + len) < 128); if (writer->buffer == NULL && !writer->overallocate) { PyObject *str; @@ -13767,7 +13767,7 @@ _PyUnicodeWriter_WriteLatin1String(_PyUnicodeWriter *writer, { Py_UCS4 maxchar; - maxchar = ucs1lib_find_max_char((Py_UCS1*)str, (Py_UCS1*)str + len); + maxchar = ucs1lib_find_max_char((const Py_UCS1*)str, (const Py_UCS1*)str + len); if (_PyUnicodeWriter_Prepare(writer, len, maxchar) == -1) return -1; unicode_write_cstr(writer->buffer, writer->pos, str, len); diff --git a/Python/marshal.c b/Python/marshal.c index c6a06e8c239764..d3fee32380b58b 100644 --- a/Python/marshal.c +++ b/Python/marshal.c @@ -734,7 +734,7 @@ r_byte(RFILE *p) else { const char *ptr = r_string(1, p); if (ptr != NULL) - c = *(unsigned char *) ptr; + c = *(const unsigned char *) ptr; } return c; } diff --git a/Python/pyhash.c b/Python/pyhash.c index 4c0b929586fc16..ba224ee373631d 100644 --- a/Python/pyhash.c +++ b/Python/pyhash.c @@ -366,7 +366,7 @@ static PyHash_FuncDef PyHash_Func = {fnv, "fnv", 8 * SIZEOF_PY_HASH_T, static uint64_t siphash24(uint64_t k0, uint64_t k1, const void *src, Py_ssize_t src_sz) { uint64_t b = (uint64_t)src_sz << 56; - const uint8_t *in = (uint8_t*)src; + const uint8_t *in = (const uint8_t*)src; uint64_t v0 = k0 ^ 0x736f6d6570736575ULL; uint64_t v1 = k1 ^ 0x646f72616e646f6dULL; From ed4d263e8767b0e4c47df99141b500d36ce0275d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 11 Feb 2020 19:21:32 -0800 Subject: [PATCH 1180/2163] bpo-39595: Improve zipfile.Path performance (GH-18406) (GH-18472) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Improve zipfile.Path performance on zipfiles with a large number of entries. * 📜🤖 Added by blurb_it. * Add bpo to blurb * Sync with importlib_metadata 1.5 (6fe70ca) * Update blurb. * Remove compatibility code * Add stubs module, omitted from earlier commit Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> (cherry picked from commit e5bd73632e77dc5ab0cab77e48e94ca5e354be8a) Co-authored-by: Jason R. Coombs Co-authored-by: Jason R. Coombs --- Lib/importlib/metadata.py | 12 +- Lib/test/test_importlib/fixtures.py | 23 ++- Lib/test/test_importlib/stubs.py | 10 ++ Lib/test/test_importlib/test_main.py | 32 ++++ Lib/test/test_zipfile.py | 142 +++++++++++++----- Lib/zipfile.py | 102 ++++++++++--- .../2020-02-07-23-14-14.bpo-39595.DHwddE.rst | 1 + 7 files changed, 254 insertions(+), 68 deletions(-) create mode 100644 Lib/test/test_importlib/stubs.py create mode 100644 Misc/NEWS.d/next/Library/2020-02-07-23-14-14.bpo-39595.DHwddE.rst diff --git a/Lib/importlib/metadata.py b/Lib/importlib/metadata.py index ae8ecf9b8500cc..831f593277ccd4 100644 --- a/Lib/importlib/metadata.py +++ b/Lib/importlib/metadata.py @@ -391,6 +391,7 @@ class FastPath: def __init__(self, root): self.root = root + self.base = os.path.basename(root).lower() def joinpath(self, child): return pathlib.Path(self.root, child) @@ -413,12 +414,11 @@ def zip_children(self): ) def is_egg(self, search): - root_n_low = os.path.split(self.root)[1].lower() - + base = self.base return ( - root_n_low == search.normalized + '.egg' - or root_n_low.startswith(search.prefix) - and root_n_low.endswith('.egg')) + base == search.versionless_egg_name + or base.startswith(search.prefix) + and base.endswith('.egg')) def search(self, name): for child in self.children(): @@ -439,6 +439,7 @@ class Prepared: prefix = '' suffixes = '.dist-info', '.egg-info' exact_matches = [''][:0] + versionless_egg_name = '' def __init__(self, name): self.name = name @@ -448,6 +449,7 @@ def __init__(self, name): self.prefix = self.normalized + '-' self.exact_matches = [ self.normalized + suffix for suffix in self.suffixes] + self.versionless_egg_name = self.normalized + '.egg' class MetadataPathFinder(DistributionFinder): diff --git a/Lib/test/test_importlib/fixtures.py b/Lib/test/test_importlib/fixtures.py index 0b4ce18d5a6cd7..695c92a786cb0b 100644 --- a/Lib/test/test_importlib/fixtures.py +++ b/Lib/test/test_importlib/fixtures.py @@ -47,14 +47,28 @@ def tempdir_as_cwd(): yield tmp -class SiteDir: +@contextlib.contextmanager +def install_finder(finder): + sys.meta_path.append(finder) + try: + yield + finally: + sys.meta_path.remove(finder) + + +class Fixtures: def setUp(self): self.fixtures = ExitStack() self.addCleanup(self.fixtures.close) + + +class SiteDir(Fixtures): + def setUp(self): + super(SiteDir, self).setUp() self.site_dir = self.fixtures.enter_context(tempdir()) -class OnSysPath: +class OnSysPath(Fixtures): @staticmethod @contextlib.contextmanager def add_sys_path(dir): @@ -198,3 +212,8 @@ def build_files(file_defs, prefix=pathlib.Path()): def DALS(str): "Dedent and left-strip" return textwrap.dedent(str).lstrip() + + +class NullFinder: + def find_module(self, name): + pass diff --git a/Lib/test/test_importlib/stubs.py b/Lib/test/test_importlib/stubs.py new file mode 100644 index 00000000000000..e5b011c399fa96 --- /dev/null +++ b/Lib/test/test_importlib/stubs.py @@ -0,0 +1,10 @@ +import unittest + + +class fake_filesystem_unittest: + """ + Stubbed version of the pyfakefs module + """ + class TestCase(unittest.TestCase): + def setUpPyfakefs(self): + self.skipTest("pyfakefs not available") diff --git a/Lib/test/test_importlib/test_main.py b/Lib/test/test_importlib/test_main.py index c5f1dbbae325ed..42a79992ecc8c0 100644 --- a/Lib/test/test_importlib/test_main.py +++ b/Lib/test/test_importlib/test_main.py @@ -7,6 +7,11 @@ import unittest import importlib.metadata +try: + import pyfakefs.fake_filesystem_unittest as ffs +except ImportError: + from .stubs import fake_filesystem_unittest as ffs + from . import fixtures from importlib.metadata import ( Distribution, EntryPoint, @@ -185,6 +190,33 @@ def test_egg(self): version('foo') +class MissingSysPath(fixtures.OnSysPath, unittest.TestCase): + site_dir = '/does-not-exist' + + def test_discovery(self): + """ + Discovering distributions should succeed even if + there is an invalid path on sys.path. + """ + importlib.metadata.distributions() + + +class InaccessibleSysPath(fixtures.OnSysPath, ffs.TestCase): + site_dir = '/access-denied' + + def setUp(self): + super(InaccessibleSysPath, self).setUp() + self.setUpPyfakefs() + self.fs.create_dir(self.site_dir, perm_bits=000) + + def test_discovery(self): + """ + Discovering distributions should succeed even if + there is an invalid path on sys.path. + """ + list(importlib.metadata.distributions()) + + class TestEntryPoints(unittest.TestCase): def __init__(self, *args): super(TestEntryPoints, self).__init__(*args) diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py index c65de9202c0c4b..61bca8651c02a3 100644 --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -2683,16 +2683,71 @@ def test_extract_command(self): self.assertEqual(f.read(), zf.read(zi)) +class TestExecutablePrependedZip(unittest.TestCase): + """Test our ability to open zip files with an executable prepended.""" + + def setUp(self): + self.exe_zip = findfile('exe_with_zip', subdir='ziptestdata') + self.exe_zip64 = findfile('exe_with_z64', subdir='ziptestdata') + + def _test_zip_works(self, name): + # bpo28494 sanity check: ensure is_zipfile works on these. + self.assertTrue(zipfile.is_zipfile(name), + f'is_zipfile failed on {name}') + # Ensure we can operate on these via ZipFile. + with zipfile.ZipFile(name) as zipfp: + for n in zipfp.namelist(): + data = zipfp.read(n) + self.assertIn(b'FAVORITE_NUMBER', data) + + def test_read_zip_with_exe_prepended(self): + self._test_zip_works(self.exe_zip) + + def test_read_zip64_with_exe_prepended(self): + self._test_zip_works(self.exe_zip64) + + @unittest.skipUnless(sys.executable, 'sys.executable required.') + @unittest.skipUnless(os.access('/bin/bash', os.X_OK), + 'Test relies on #!/bin/bash working.') + def test_execute_zip2(self): + output = subprocess.check_output([self.exe_zip, sys.executable]) + self.assertIn(b'number in executable: 5', output) + + @unittest.skipUnless(sys.executable, 'sys.executable required.') + @unittest.skipUnless(os.access('/bin/bash', os.X_OK), + 'Test relies on #!/bin/bash working.') + def test_execute_zip64(self): + output = subprocess.check_output([self.exe_zip64, sys.executable]) + self.assertIn(b'number in executable: 5', output) + + # Poor man's technique to consume a (smallish) iterable. consume = tuple +# from jaraco.itertools 5.0 +class jaraco: + class itertools: + class Counter: + def __init__(self, i): + self.count = 0 + self._orig_iter = iter(i) + + def __iter__(self): + return self + + def __next__(self): + result = next(self._orig_iter) + self.count += 1 + return result + + def add_dirs(zf): """ Given a writable zip file zf, inject directory entries for any directories implied by the presence of children. """ - for name in zipfile.Path._implied_dirs(zf.namelist()): + for name in zipfile.CompleteDirs._implied_dirs(zf.namelist()): zf.writestr(name, b"") return zf @@ -2733,44 +2788,6 @@ def build_alpharep_fixture(): return zf -class TestExecutablePrependedZip(unittest.TestCase): - """Test our ability to open zip files with an executable prepended.""" - - def setUp(self): - self.exe_zip = findfile('exe_with_zip', subdir='ziptestdata') - self.exe_zip64 = findfile('exe_with_z64', subdir='ziptestdata') - - def _test_zip_works(self, name): - # bpo-28494 sanity check: ensure is_zipfile works on these. - self.assertTrue(zipfile.is_zipfile(name), - f'is_zipfile failed on {name}') - # Ensure we can operate on these via ZipFile. - with zipfile.ZipFile(name) as zipfp: - for n in zipfp.namelist(): - data = zipfp.read(n) - self.assertIn(b'FAVORITE_NUMBER', data) - - def test_read_zip_with_exe_prepended(self): - self._test_zip_works(self.exe_zip) - - def test_read_zip64_with_exe_prepended(self): - self._test_zip_works(self.exe_zip64) - - @unittest.skipUnless(sys.executable, 'sys.executable required.') - @unittest.skipUnless(os.access('/bin/bash', os.X_OK), - 'Test relies on #!/bin/bash working.') - def test_execute_zip2(self): - output = subprocess.check_output([self.exe_zip, sys.executable]) - self.assertIn(b'number in executable: 5', output) - - @unittest.skipUnless(sys.executable, 'sys.executable required.') - @unittest.skipUnless(os.access('/bin/bash', os.X_OK), - 'Test relies on #!/bin/bash working.') - def test_execute_zip64(self): - output = subprocess.check_output([self.exe_zip64, sys.executable]) - self.assertIn(b'number in executable: 5', output) - - class TestPath(unittest.TestCase): def setUp(self): self.fixtures = contextlib.ExitStack() @@ -2808,6 +2825,14 @@ def test_iterdir_and_types(self): i, = h.iterdir() assert i.is_file() + def test_subdir_is_dir(self): + for alpharep in self.zipfile_alpharep(): + root = zipfile.Path(alpharep) + assert (root / 'b').is_dir() + assert (root / 'b/').is_dir() + assert (root / 'g').is_dir() + assert (root / 'g/').is_dir() + def test_open(self): for alpharep in self.zipfile_alpharep(): root = zipfile.Path(alpharep) @@ -2869,6 +2894,45 @@ def test_missing_dir_parent(self): root = zipfile.Path(alpharep) assert (root / 'missing dir/').parent.at == '' + def test_mutability(self): + """ + If the underlying zipfile is changed, the Path object should + reflect that change. + """ + for alpharep in self.zipfile_alpharep(): + root = zipfile.Path(alpharep) + a, b, g = root.iterdir() + alpharep.writestr('foo.txt', 'foo') + alpharep.writestr('bar/baz.txt', 'baz') + assert any( + child.name == 'foo.txt' + for child in root.iterdir()) + assert (root / 'foo.txt').read_text() == 'foo' + baz, = (root / 'bar').iterdir() + assert baz.read_text() == 'baz' + + HUGE_ZIPFILE_NUM_ENTRIES = 2 ** 13 + + def huge_zipfile(self): + """Create a read-only zipfile with a huge number of entries entries.""" + strm = io.BytesIO() + zf = zipfile.ZipFile(strm, "w") + for entry in map(str, range(self.HUGE_ZIPFILE_NUM_ENTRIES)): + zf.writestr(entry, entry) + zf.mode = 'r' + return zf + + def test_joinpath_constant_time(self): + """ + Ensure joinpath on items in zipfile is linear time. + """ + root = zipfile.Path(self.huge_zipfile()) + entries = jaraco.itertools.Counter(root.iterdir()) + for entry in entries: + entry.joinpath('suffix') + # Check the file iterated all items + assert entries.count == self.HUGE_ZIPFILE_NUM_ENTRIES + if __name__ == "__main__": unittest.main() diff --git a/Lib/zipfile.py b/Lib/zipfile.py index 8b99c1189baa8f..5dc6516cc47b7e 100644 --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -16,6 +16,8 @@ import sys import threading import time +import contextlib +from collections import OrderedDict try: import zlib # We may need its compression method @@ -2182,6 +2184,79 @@ def _ancestry(path): path, tail = posixpath.split(path) +class CompleteDirs(ZipFile): + """ + A ZipFile subclass that ensures that implied directories + are always included in the namelist. + """ + + @staticmethod + def _implied_dirs(names): + parents = itertools.chain.from_iterable(map(_parents, names)) + # Deduplicate entries in original order + implied_dirs = OrderedDict.fromkeys( + p + posixpath.sep for p in parents + # Cast names to a set for O(1) lookups + if p + posixpath.sep not in set(names) + ) + return implied_dirs + + def namelist(self): + names = super(CompleteDirs, self).namelist() + return names + list(self._implied_dirs(names)) + + def _name_set(self): + return set(self.namelist()) + + def resolve_dir(self, name): + """ + If the name represents a directory, return that name + as a directory (with the trailing slash). + """ + names = self._name_set() + dirname = name + '/' + dir_match = name not in names and dirname in names + return dirname if dir_match else name + + @classmethod + def make(cls, source): + """ + Given a source (filename or zipfile), return an + appropriate CompleteDirs subclass. + """ + if isinstance(source, CompleteDirs): + return source + + if not isinstance(source, ZipFile): + return cls(source) + + # Only allow for FastPath when supplied zipfile is read-only + if 'r' not in source.mode: + cls = CompleteDirs + + res = cls.__new__(cls) + vars(res).update(vars(source)) + return res + + +class FastLookup(CompleteDirs): + """ + ZipFile subclass to ensure implicit + dirs exist and are resolved rapidly. + """ + def namelist(self): + with contextlib.suppress(AttributeError): + return self.__names + self.__names = super(FastLookup, self).namelist() + return self.__names + + def _name_set(self): + with contextlib.suppress(AttributeError): + return self.__lookup + self.__lookup = super(FastLookup, self)._name_set() + return self.__lookup + + class Path: """ A pathlib-compatible interface for zip files. @@ -2250,7 +2325,7 @@ class Path: __repr = "{self.__class__.__name__}({self.root.filename!r}, {self.at!r})" def __init__(self, root, at=""): - self.root = root if isinstance(root, ZipFile) else ZipFile(root) + self.root = FastLookup.make(root) self.at = at @property @@ -2282,12 +2357,12 @@ def is_file(self): return not self.is_dir() def exists(self): - return self.at in self._names() + return self.at in self.root._name_set() def iterdir(self): if not self.is_dir(): raise ValueError("Can't listdir a file") - subs = map(self._next, self._names()) + subs = map(self._next, self.root.namelist()) return filter(self._is_child, subs) def __str__(self): @@ -2298,25 +2373,10 @@ def __repr__(self): def joinpath(self, add): next = posixpath.join(self.at, add) - next_dir = posixpath.join(self.at, add, "") - names = self._names() - return self._next(next_dir if next not in names and next_dir in names else next) + return self._next(self.root.resolve_dir(next)) __truediv__ = joinpath - @staticmethod - def _implied_dirs(names): - return _unique_everseen( - parent + "/" - for name in names - for parent in _parents(name) - if parent + "/" not in names - ) - - @classmethod - def _add_implied_dirs(cls, names): - return names + list(cls._implied_dirs(names)) - @property def parent(self): parent_at = posixpath.dirname(self.at.rstrip('/')) @@ -2324,9 +2384,6 @@ def parent(self): parent_at += '/' return self._next(parent_at) - def _names(self): - return self._add_implied_dirs(self.root.namelist()) - def main(args=None): import argparse @@ -2388,5 +2445,6 @@ def addToZip(zf, path, zippath): zippath = '' addToZip(zf, path, zippath) + if __name__ == "__main__": main() diff --git a/Misc/NEWS.d/next/Library/2020-02-07-23-14-14.bpo-39595.DHwddE.rst b/Misc/NEWS.d/next/Library/2020-02-07-23-14-14.bpo-39595.DHwddE.rst new file mode 100644 index 00000000000000..3a461389af7d18 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-07-23-14-14.bpo-39595.DHwddE.rst @@ -0,0 +1 @@ +Improved performance of zipfile.Path for files with a large number of entries. Also improved performance and fixed minor issue as published with `importlib_metadata 1.5 `_. From 0b8f738eb3ee0110461e7da28c0b6b452f91999d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 11 Feb 2020 19:52:46 -0800 Subject: [PATCH 1181/2163] bpo-39605: Remove a cast that causes a warning. (GH-18473) (cherry picked from commit 95905ce0f41fd42eb1ef60ddb83f057401c3d52f) Co-authored-by: Benjamin Peterson --- Objects/unicodeobject.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 8ba379e8887e98..4c2b42f959b838 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -3353,7 +3353,7 @@ PyUnicode_Decode(const char *s, /* Decode via the codec registry */ buffer = NULL; - if (PyBuffer_FillInfo(&info, NULL, (const void *)s, size, 1, PyBUF_FULL_RO) < 0) + if (PyBuffer_FillInfo(&info, NULL, (void *)s, size, 1, PyBUF_FULL_RO) < 0) goto onError; buffer = PyMemoryView_FromBuffer(&info); if (buffer == NULL) From efd878cdb46d9c7038d93fb36eb1ff7dc5baf9ec Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 12 Feb 2020 02:35:10 -0800 Subject: [PATCH 1182/2163] bpo-39219: Fix SyntaxError attributes in the tokenizer. (GH-17828) * Always set the text attribute. * Correct the offset attribute for non-ascii sources. (cherry picked from commit 0cc6b5e559b8303b18fdd56c2befd900fe7b5e35) Co-authored-by: Serhiy Storchaka --- Lib/test/test_exceptions.py | 14 +++++++- .../2020-01-05-13-36-08.bpo-39219.uHtKd4.rst | 2 ++ Parser/tokenizer.c | 36 ++++++++++++++++--- 3 files changed, 47 insertions(+), 5 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-05-13-36-08.bpo-39219.uHtKd4.rst diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 10c1e076464e2c..3a322531573693 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -179,17 +179,25 @@ def ckmsg(src, msg, exception=SyntaxError): ckmsg(s, "inconsistent use of tabs and spaces in indentation", TabError) def testSyntaxErrorOffset(self): - def check(src, lineno, offset): + def check(src, lineno, offset, encoding='utf-8'): with self.assertRaises(SyntaxError) as cm: compile(src, '', 'exec') self.assertEqual(cm.exception.lineno, lineno) self.assertEqual(cm.exception.offset, offset) + if cm.exception.text is not None: + if not isinstance(src, str): + src = src.decode(encoding, 'replace') + line = src.split('\n')[lineno-1] + self.assertEqual(cm.exception.text.rstrip('\n'), line) check('def fact(x):\n\treturn x!\n', 2, 10) check('1 +\n', 1, 4) check('def spam():\n print(1)\n print(2)', 3, 10) check('Python = "Python" +', 1, 20) check('Python = "\u1e54\xfd\u0163\u0125\xf2\xf1" +', 1, 20) + check(b'# -*- coding: cp1251 -*-\nPython = "\xcf\xb3\xf2\xee\xed" +', + 2, 19, encoding='cp1251') + check(b'Python = "\xcf\xb3\xf2\xee\xed" +', 1, 18) check('x = "a', 1, 7) check('lambda x: x = 2', 1, 1) @@ -205,6 +213,10 @@ def check(src, lineno, offset): check('0010 + 2', 1, 4) check('x = 32e-+4', 1, 8) check('x = 0o9', 1, 6) + check('\u03b1 = 0xI', 1, 6) + check(b'\xce\xb1 = 0xI', 1, 6) + check(b'# -*- coding: iso8859-7 -*-\n\xe1 = 0xI', 2, 6, + encoding='iso8859-7') # Errors thrown by symtable.c check('x = [(yield i) for i in range(3)]', 1, 5) diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-05-13-36-08.bpo-39219.uHtKd4.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-05-13-36-08.bpo-39219.uHtKd4.rst new file mode 100644 index 00000000000000..dac8360df712ce --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-01-05-13-36-08.bpo-39219.uHtKd4.rst @@ -0,0 +1,2 @@ +Syntax errors raised in the tokenizer now always set correct "text" and +"offset" attributes. diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index f73c32684c7b73..aecbcebb917e83 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -1,6 +1,7 @@ /* Tokenizer implementation */ +#define PY_SSIZE_T_CLEAN #include "Python.h" #include @@ -1034,17 +1035,44 @@ tok_backup(struct tok_state *tok, int c) static int syntaxerror(struct tok_state *tok, const char *format, ...) { + PyObject *errmsg, *errtext, *args; va_list vargs; #ifdef HAVE_STDARG_PROTOTYPES va_start(vargs, format); #else va_start(vargs); #endif - PyErr_FormatV(PyExc_SyntaxError, format, vargs); + errmsg = PyUnicode_FromFormatV(format, vargs); va_end(vargs); - PyErr_SyntaxLocationObject(tok->filename, - tok->lineno, - (int)(tok->cur - tok->line_start)); + if (!errmsg) { + goto error; + } + + errtext = PyUnicode_DecodeUTF8(tok->line_start, tok->cur - tok->line_start, + "replace"); + if (!errtext) { + goto error; + } + int offset = (int)PyUnicode_GET_LENGTH(errtext); + Py_ssize_t line_len = strcspn(tok->line_start, "\n"); + if (line_len != tok->cur - tok->line_start) { + Py_DECREF(errtext); + errtext = PyUnicode_DecodeUTF8(tok->line_start, line_len, + "replace"); + } + if (!errtext) { + goto error; + } + + args = Py_BuildValue("(O(OiiN))", errmsg, + tok->filename, tok->lineno, offset, errtext); + if (args) { + PyErr_SetObject(PyExc_SyntaxError, args); + Py_DECREF(args); + } + +error: + Py_XDECREF(errmsg); tok->done = E_ERROR; return ERRORTOKEN; } From ac6f4d2db703c0ff88e496bcb7b7fe55cf2ac458 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 12 Feb 2020 04:32:52 -0800 Subject: [PATCH 1183/2163] bpo-21016: pydoc and trace use sysconfig (GH-18476) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit bpo-21016, bpo-1294959: The pydoc and trace modules now use the sysconfig module to get the path to the Python standard library, to support uncommon installation path like /usr/lib64/python3.9/ on Fedora. Co-Authored-By: Jan MatÄ›jek (cherry picked from commit 4fac7ed43ebf1771a8fe86fdfe7b9991f3be78cd) Co-authored-by: Victor Stinner --- Lib/pydoc.py | 5 ++--- Lib/trace.py | 6 +++--- .../next/Library/2020-02-12-10-04-39.bpo-21016.bFXPH7.rst | 4 ++++ 3 files changed, 9 insertions(+), 6 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-02-12-10-04-39.bpo-21016.bFXPH7.rst diff --git a/Lib/pydoc.py b/Lib/pydoc.py index 9a22e56686f618..dc3377d68f8caa 100644 --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -66,6 +66,7 @@ class or function within a module or module in a package. If the import platform import re import sys +import sysconfig import time import tokenize import urllib.parse @@ -392,9 +393,7 @@ def fail(self, object, name=None, *args): docmodule = docclass = docroutine = docother = docproperty = docdata = fail - def getdocloc(self, object, - basedir=os.path.join(sys.base_exec_prefix, "lib", - "python%d.%d" % sys.version_info[:2])): + def getdocloc(self, object, basedir=sysconfig.get_path('stdlib')): """Return the location of module docs or None""" try: diff --git a/Lib/trace.py b/Lib/trace.py index 62325d3f238ad6..a44735761df420 100755 --- a/Lib/trace.py +++ b/Lib/trace.py @@ -52,6 +52,7 @@ import linecache import os import sys +import sysconfig import token import tokenize import inspect @@ -676,9 +677,8 @@ def main(): opts = parser.parse_args() if opts.ignore_dir: - rel_path = 'lib', 'python{0.major}.{0.minor}'.format(sys.version_info) - _prefix = os.path.join(sys.base_prefix, *rel_path) - _exec_prefix = os.path.join(sys.base_exec_prefix, *rel_path) + _prefix = sysconfig.get_path("stdlib") + _exec_prefix = sysconfig.get_path("platstdlib") def parse_ignore_dir(s): s = os.path.expanduser(os.path.expandvars(s)) diff --git a/Misc/NEWS.d/next/Library/2020-02-12-10-04-39.bpo-21016.bFXPH7.rst b/Misc/NEWS.d/next/Library/2020-02-12-10-04-39.bpo-21016.bFXPH7.rst new file mode 100644 index 00000000000000..fb91bb38255557 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-12-10-04-39.bpo-21016.bFXPH7.rst @@ -0,0 +1,4 @@ +The :mod:`pydoc` and :mod:`trace` modules now use the :mod:`sysconfig` +module to get the path to the Python standard library, to support uncommon +installation path like ``/usr/lib64/python3.9/`` on Fedora. +Patch by Jan MatÄ›jek. From 2076d4f97ef514bb4dc4ca768fbaa3f538ce7f1f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 12 Feb 2020 12:56:44 -0800 Subject: [PATCH 1184/2163] bpo-39474: Fix AST pos for expressions like (a)(b), (a)[b] and (a).b. (GH-18477) (cherry picked from commit 6e619c48b8e804ece9521453fc8da0640a04d5b1) Co-authored-by: Serhiy Storchaka --- Lib/test/test_ast.py | 27 ++++++++++++++ .../2020-02-12-12-01-26.bpo-39474.RZMEUH.rst | 2 ++ Python/ast.c | 36 +++++++++---------- 3 files changed, 47 insertions(+), 18 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-02-12-12-01-26.bpo-39474.RZMEUH.rst diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py index e843d53781d25d..3e8a39dc41047e 100644 --- a/Lib/test/test_ast.py +++ b/Lib/test/test_ast.py @@ -1642,6 +1642,33 @@ def test_attribute_spaces(self): self._check_content(s, call, s) self._check_content(s, call.args[0], 'x. y .z') + def test_redundant_parenthesis(self): + s = '( ( ( a + b ) ) )' + v = ast.parse(s).body[0].value + self.assertEqual(type(v).__name__, 'BinOp') + self._check_content(s, v, 'a + b') + s2 = 'await ' + s + v = ast.parse(s2).body[0].value.value + self.assertEqual(type(v).__name__, 'BinOp') + self._check_content(s2, v, 'a + b') + + def test_trailers_with_redundant_parenthesis(self): + tests = ( + ('( ( ( a ) ) ) ( )', 'Call'), + ('( ( ( a ) ) ) ( b )', 'Call'), + ('( ( ( a ) ) ) [ b ]', 'Subscript'), + ('( ( ( a ) ) ) . b', 'Attribute'), + ) + for s, t in tests: + with self.subTest(s): + v = ast.parse(s).body[0].value + self.assertEqual(type(v).__name__, t) + self._check_content(s, v, s) + s2 = 'await ' + s + v = ast.parse(s2).body[0].value.value + self.assertEqual(type(v).__name__, t) + self._check_content(s2, v, s) + def test_displays(self): s1 = '[{}, {1, }, {1, 2,} ]' s2 = '{a: b, f (): g () ,}' diff --git a/Misc/NEWS.d/next/Library/2020-02-12-12-01-26.bpo-39474.RZMEUH.rst b/Misc/NEWS.d/next/Library/2020-02-12-12-01-26.bpo-39474.RZMEUH.rst new file mode 100644 index 00000000000000..e990f84a9dbf25 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-12-12-01-26.bpo-39474.RZMEUH.rst @@ -0,0 +1,2 @@ +Fixed starting position of AST for expressions like ``(a)(b)``, ``(a)[b]`` +and ``(a).b``. diff --git a/Python/ast.c b/Python/ast.c index 12f24f2c22ab96..f70d48ba3a15da 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -583,7 +583,7 @@ static stmt_ty ast_for_for_stmt(struct compiling *, const node *, bool); /* Note different signature for ast_for_call */ static expr_ty ast_for_call(struct compiling *, const node *, expr_ty, - const node *, const node *); + const node *, const node *, const node *); static PyObject *parsenumber(struct compiling *, const char *); static expr_ty parsestrplus(struct compiling *, const node *n); @@ -1757,7 +1757,8 @@ ast_for_decorator(struct compiling *c, const node *n) name_expr = NULL; } else { - d = ast_for_call(c, CHILD(n, 3), name_expr, CHILD(n, 2), CHILD(n, 4)); + d = ast_for_call(c, CHILD(n, 3), name_expr, + CHILD(n, 1), CHILD(n, 2), CHILD(n, 4)); if (!d) return NULL; name_expr = NULL; @@ -2658,7 +2659,7 @@ ast_for_binop(struct compiling *c, const node *n) } static expr_ty -ast_for_trailer(struct compiling *c, const node *n, expr_ty left_expr) +ast_for_trailer(struct compiling *c, const node *n, expr_ty left_expr, const node *start) { /* trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME subscriptlist: subscript (',' subscript)* [','] @@ -2668,17 +2669,18 @@ ast_for_trailer(struct compiling *c, const node *n, expr_ty left_expr) REQ(n, trailer); if (TYPE(CHILD(n, 0)) == LPAR) { if (NCH(n) == 2) - return Call(left_expr, NULL, NULL, LINENO(n), n->n_col_offset, + return Call(left_expr, NULL, NULL, LINENO(start), start->n_col_offset, n->n_end_lineno, n->n_end_col_offset, c->c_arena); else - return ast_for_call(c, CHILD(n, 1), left_expr, CHILD(n, 0), CHILD(n, 2)); + return ast_for_call(c, CHILD(n, 1), left_expr, + start, CHILD(n, 0), CHILD(n, 2)); } else if (TYPE(CHILD(n, 0)) == DOT) { PyObject *attr_id = NEW_IDENTIFIER(CHILD(n, 1)); if (!attr_id) return NULL; return Attribute(left_expr, attr_id, Load, - LINENO(n), n->n_col_offset, + LINENO(start), start->n_col_offset, n->n_end_lineno, n->n_end_col_offset, c->c_arena); } else { @@ -2689,7 +2691,7 @@ ast_for_trailer(struct compiling *c, const node *n, expr_ty left_expr) slice_ty slc = ast_for_slice(c, CHILD(n, 0)); if (!slc) return NULL; - return Subscript(left_expr, slc, Load, LINENO(n), n->n_col_offset, + return Subscript(left_expr, slc, Load, LINENO(start), start->n_col_offset, n_copy->n_end_lineno, n_copy->n_end_col_offset, c->c_arena); } @@ -2716,7 +2718,7 @@ ast_for_trailer(struct compiling *c, const node *n, expr_ty left_expr) } if (!simple) { return Subscript(left_expr, ExtSlice(slices, c->c_arena), - Load, LINENO(n), n->n_col_offset, + Load, LINENO(start), start->n_col_offset, n_copy->n_end_lineno, n_copy->n_end_col_offset, c->c_arena); } /* extract Index values and put them in a Tuple */ @@ -2733,7 +2735,7 @@ ast_for_trailer(struct compiling *c, const node *n, expr_ty left_expr) if (!e) return NULL; return Subscript(left_expr, Index(e, c->c_arena), - Load, LINENO(n), n->n_col_offset, + Load, LINENO(start), start->n_col_offset, n_copy->n_end_lineno, n_copy->n_end_col_offset, c->c_arena); } } @@ -2771,7 +2773,7 @@ static expr_ty ast_for_atom_expr(struct compiling *c, const node *n) { int i, nch, start = 0; - expr_ty e, tmp; + expr_ty e; REQ(n, atom_expr); nch = NCH(n); @@ -2800,12 +2802,9 @@ ast_for_atom_expr(struct compiling *c, const node *n) node *ch = CHILD(n, i); if (TYPE(ch) != trailer) break; - tmp = ast_for_trailer(c, ch, e); - if (!tmp) + e = ast_for_trailer(c, ch, e, CHILD(n, start)); + if (!e) return NULL; - tmp->lineno = e->lineno; - tmp->col_offset = e->col_offset; - e = tmp; } if (start) { @@ -3035,7 +3034,7 @@ ast_for_expr(struct compiling *c, const node *n) static expr_ty ast_for_call(struct compiling *c, const node *n, expr_ty func, - const node *maybegenbeg, const node *closepar) + const node *start, const node *maybegenbeg, const node *closepar) { /* arglist: argument (',' argument)* [','] @@ -3239,7 +3238,7 @@ ast_for_call(struct compiling *c, const node *n, expr_ty func, } } - return Call(func, args, keywords, func->lineno, func->col_offset, + return Call(func, args, keywords, LINENO(start), start->n_col_offset, closepar->n_end_lineno, closepar->n_end_col_offset, c->c_arena); } @@ -4489,7 +4488,8 @@ ast_for_classdef(struct compiling *c, const node *n, asdl_seq *decorator_seq) dummy = Name(dummy_name, Load, LINENO(n), n->n_col_offset, CHILD(n, 1)->n_end_lineno, CHILD(n, 1)->n_end_col_offset, c->c_arena); - call = ast_for_call(c, CHILD(n, 3), dummy, NULL, CHILD(n, 4)); + call = ast_for_call(c, CHILD(n, 3), dummy, + CHILD(n, 1), NULL, CHILD(n, 4)); if (!call) return NULL; } From 669981b3b14dd16dec42089d6ac8d6449fde8abd Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 12 Feb 2020 21:12:53 -0800 Subject: [PATCH 1185/2163] closes bpo-39621: Make buf arg to md5_compress be const. (GH-18497) (cherry picked from commit 597ebed748d0b0c061f8c108bd98270d103286c1) Co-authored-by: Andy Lester --- Modules/md5module.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Modules/md5module.c b/Modules/md5module.c index b9a351a8c1cdd2..c2ebaaf61f91c6 100644 --- a/Modules/md5module.c +++ b/Modules/md5module.c @@ -119,7 +119,7 @@ typedef struct { a = (a + I(b,c,d) + M + t); a = ROLc(a, s) + b; -static void md5_compress(struct md5_state *md5, unsigned char *buf) +static void md5_compress(struct md5_state *md5, const unsigned char *buf) { MD5_INT32 i, W[16], a, b, c, d; @@ -242,7 +242,7 @@ md5_process(struct md5_state *md5, const unsigned char *in, Py_ssize_t inlen) while (inlen > 0) { if (md5->curlen == 0 && inlen >= MD5_BLOCKSIZE) { - md5_compress(md5, (unsigned char *)in); + md5_compress(md5, in); md5->length += MD5_BLOCKSIZE * 8; in += MD5_BLOCKSIZE; inlen -= MD5_BLOCKSIZE; From a00b5be5f71b702ab80b0a7c046485046aaae160 Mon Sep 17 00:00:00 2001 From: Steve Dower Date: Thu, 13 Feb 2020 08:30:27 +0000 Subject: [PATCH 1186/2163] bpo-39184: Add audit events to functions in `fcntl`, `msvcrt`, `os`, `resource`, `shutil`, `signal`, `syslog` (GH-18407) Co-authored-by: Saiyang Gou --- Doc/library/fcntl.rst | 8 + Doc/library/msvcrt.rst | 6 + Doc/library/os.rst | 68 ++++++++ Doc/library/resource.rst | 5 + Doc/library/shutil.rst | 20 +++ Doc/library/signal.rst | 2 + Doc/library/syslog.rst | 8 + Lib/shutil.py | 10 ++ .../2020-02-07-23-54-18.bpo-39184.v-ue-v.rst | 1 + Modules/fcntlmodule.c | 18 +++ Modules/posixmodule.c | 145 ++++++++++++++++-- Modules/resource.c | 10 ++ Modules/signalmodule.c | 4 + Modules/syslogmodule.c | 14 ++ PC/msvcrtmodule.c | 12 ++ 15 files changed, 314 insertions(+), 17 deletions(-) create mode 100644 Misc/NEWS.d/next/Security/2020-02-07-23-54-18.bpo-39184.v-ue-v.rst diff --git a/Doc/library/fcntl.rst b/Doc/library/fcntl.rst index a7390150f7910d..69484b647363dd 100644 --- a/Doc/library/fcntl.rst +++ b/Doc/library/fcntl.rst @@ -57,6 +57,8 @@ The module defines the following functions: If the :c:func:`fcntl` fails, an :exc:`OSError` is raised. + .. audit-event:: fcntl.fcntl fd,cmd,arg fcntl.fcntl + .. function:: ioctl(fd, request, arg=0, mutate_flag=True) @@ -106,6 +108,8 @@ The module defines the following functions: >>> buf array('h', [13341]) + .. audit-event:: fcntl.ioctl fd,request,arg fcntl.ioctl + .. function:: flock(fd, operation) @@ -116,6 +120,8 @@ The module defines the following functions: If the :c:func:`flock` fails, an :exc:`OSError` exception is raised. + .. audit-event:: fcntl.flock fd,operation fcntl.flock + .. function:: lockf(fd, cmd, len=0, start=0, whence=0) @@ -149,6 +155,8 @@ The module defines the following functions: The default for *len* is 0 which means to lock to the end of the file. The default for *whence* is also 0. + .. audit-event:: fcntl.lockf fd,cmd,len,start,whence fcntl.lockf + Examples (all on a SVR4 compliant system):: import struct, fcntl, os diff --git a/Doc/library/msvcrt.rst b/Doc/library/msvcrt.rst index 14ad2cd4373afe..42fffee6a0f449 100644 --- a/Doc/library/msvcrt.rst +++ b/Doc/library/msvcrt.rst @@ -42,6 +42,8 @@ File Operations regions in a file may be locked at the same time, but may not overlap. Adjacent regions are not merged; they must be unlocked individually. + .. audit-event:: msvcrt.locking fd,mode,nbytes msvcrt.locking + .. data:: LK_LOCK LK_RLCK @@ -77,12 +79,16 @@ File Operations and :const:`os.O_TEXT`. The returned file descriptor may be used as a parameter to :func:`os.fdopen` to create a file object. + .. audit-event:: msvcrt.open_osfhandle handle,flags msvcrt.open_osfhandle + .. function:: get_osfhandle(fd) Return the file handle for the file descriptor *fd*. Raises :exc:`OSError` if *fd* is not recognized. + .. audit-event:: msvcrt.get_osfhandle fd msvcrt.get_osfhandle + .. _msvcrt-console: diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 0d8df34c345c4d..3157b887b535c7 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -451,6 +451,8 @@ process and user. calls to :func:`putenv` don't update ``os.environ``, so it is actually preferable to assign to items of ``os.environ``. + .. audit-event:: os.putenv key,value os.putenv + .. function:: setegid(egid) @@ -643,6 +645,8 @@ process and user. calls to :func:`unsetenv` don't update ``os.environ``, so it is actually preferable to delete items of ``os.environ``. + .. audit-event:: os.unsetenv key os.unsetenv + .. availability:: most flavors of Unix. @@ -768,6 +772,8 @@ as internal buffering of data. docs for :func:`chmod` for possible values of *mode*. As of Python 3.3, this is equivalent to ``os.chmod(fd, mode)``. + .. audit-event:: os.chmod path,mode,dir_fd os.fchmod + .. availability:: Unix. @@ -778,6 +784,8 @@ as internal buffering of data. :func:`chown`. As of Python 3.3, this is equivalent to ``os.chown(fd, uid, gid)``. + .. audit-event:: os.chown path,uid,gid,dir_fd os.fchown + .. availability:: Unix. @@ -885,6 +893,8 @@ as internal buffering of data. :data:`F_ULOCK` or :data:`F_TEST`. *len* specifies the section of the file to lock. + .. audit-event:: os.lockf fd,cmd,len os.lockf + .. availability:: Unix. .. versionadded:: 3.3 @@ -1602,6 +1612,8 @@ features: This function can raise :exc:`OSError` and subclasses such as :exc:`FileNotFoundError`, :exc:`PermissionError`, and :exc:`NotADirectoryError`. + .. audit-event:: os.chdir path os.chdir + .. versionadded:: 3.3 Added support for specifying *path* as a file descriptor on some platforms. @@ -1630,6 +1642,8 @@ features: This function can support :ref:`not following symlinks `. + .. audit-event:: os.chflags path,flags os.chflags + .. availability:: Unix. .. versionadded:: 3.3 @@ -1675,6 +1689,8 @@ features: read-only flag with it (via the ``stat.S_IWRITE`` and ``stat.S_IREAD`` constants or a corresponding integer value). All other bits are ignored. + .. audit-event:: os.chmod path,mode,dir_fd os.chmod + .. versionadded:: 3.3 Added support for specifying *path* as an open file descriptor, and the *dir_fd* and *follow_symlinks* arguments. @@ -1695,6 +1711,8 @@ features: See :func:`shutil.chown` for a higher-level function that accepts names in addition to numeric ids. + .. audit-event:: os.chown path,uid,gid,dir_fd os.chown + .. availability:: Unix. .. versionadded:: 3.3 @@ -1721,6 +1739,8 @@ features: descriptor *fd*. The descriptor must refer to an opened directory, not an open file. As of Python 3.3, this is equivalent to ``os.chdir(fd)``. + .. audit-event:: os.chdir path os.fchdir + .. availability:: Unix. @@ -1745,6 +1765,8 @@ features: not follow symbolic links. As of Python 3.3, this is equivalent to ``os.chflags(path, flags, follow_symlinks=False)``. + .. audit-event:: os.chflags path,flags os.lchflags + .. availability:: Unix. .. versionchanged:: 3.6 @@ -1758,6 +1780,8 @@ features: for possible values of *mode*. As of Python 3.3, this is equivalent to ``os.chmod(path, mode, follow_symlinks=False)``. + .. audit-event:: os.chmod path,mode,dir_fd os.lchmod + .. availability:: Unix. .. versionchanged:: 3.6 @@ -1769,6 +1793,8 @@ features: function will not follow symbolic links. As of Python 3.3, this is equivalent to ``os.chown(path, uid, gid, follow_symlinks=False)``. + .. audit-event:: os.chown path,uid,gid,dir_fd os.lchown + .. availability:: Unix. .. versionchanged:: 3.6 @@ -1783,6 +1809,8 @@ features: supply :ref:`paths relative to directory descriptors `, and :ref:`not following symlinks `. + .. audit-event:: os.link src,dst,src_dir_fd,dst_dir_fd os.link + .. availability:: Unix, Windows. .. versionchanged:: 3.2 @@ -1885,6 +1913,8 @@ features: It is also possible to create temporary directories; see the :mod:`tempfile` module's :func:`tempfile.mkdtemp` function. + .. audit-event:: os.mkdir path,mode,dir_fd os.mkdir + .. versionadded:: 3.3 The *dir_fd* argument. @@ -1917,6 +1947,8 @@ features: This function handles UNC paths correctly. + .. audit-event:: os.mkdir path,mode,dir_fd os.makedirs + .. versionadded:: 3.2 The *exist_ok* parameter. @@ -2082,6 +2114,8 @@ features: This function is semantically identical to :func:`unlink`. + .. audit-event:: os.remove path,dir_fd os.remove + .. versionadded:: 3.3 The *dir_fd* argument. @@ -2102,6 +2136,8 @@ features: they are empty. Raises :exc:`OSError` if the leaf directory could not be successfully removed. + .. audit-event:: os.remove path,dir_fd os.removedirs + .. versionchanged:: 3.6 Accepts a :term:`path-like object`. @@ -2127,6 +2163,8 @@ features: If you want cross-platform overwriting of the destination, use :func:`replace`. + .. audit-event:: os.rename src,dst,src_dir_fd,dst_dir_fd os.rename + .. versionadded:: 3.3 The *src_dir_fd* and *dst_dir_fd* arguments. @@ -2146,6 +2184,8 @@ features: This function can fail with the new directory structure made if you lack permissions needed to remove the leaf directory or file. + .. audit-event:: os.rename src,dst,src_dir_fd,dst_dir_fd os.renames + .. versionchanged:: 3.6 Accepts a :term:`path-like object` for *old* and *new*. @@ -2161,6 +2201,8 @@ features: This function can support specifying *src_dir_fd* and/or *dst_dir_fd* to supply :ref:`paths relative to directory descriptors `. + .. audit-event:: os.rename src,dst,src_dir_fd,dst_dir_fd os.replace + .. versionadded:: 3.3 .. versionchanged:: 3.6 @@ -2177,6 +2219,8 @@ features: This function can support :ref:`paths relative to directory descriptors `. + .. audit-event:: os.rmdir path,dir_fd os.rmdir + .. versionadded:: 3.3 The *dir_fd* parameter. @@ -2820,6 +2864,8 @@ features: :exc:`OSError` is raised when the function is called by an unprivileged user. + .. audit-event:: os.symlink src,dst,dir_fd os.symlink + .. availability:: Unix, Windows. .. versionchanged:: 3.2 @@ -2872,6 +2918,8 @@ features: traditional Unix name. Please see the documentation for :func:`remove` for further information. + .. audit-event:: os.remove path,dir_fd os.unlink + .. versionadded:: 3.3 The *dir_fd* parameter. @@ -2909,6 +2957,8 @@ features: :ref:`paths relative to directory descriptors ` and :ref:`not following symlinks `. + .. audit-event:: os.utime path,times,ns,dir_fd os.utime + .. versionadded:: 3.3 Added support for specifying *path* as an open file descriptor, and the *dir_fd*, *follow_symlinks*, and *ns* parameters. @@ -3134,6 +3184,8 @@ These functions are all available on Linux only. This function can support :ref:`specifying a file descriptor ` and :ref:`not following symlinks `. + .. audit-event:: os.getxattr path,attribute os.getxattr + .. versionchanged:: 3.6 Accepts a :term:`path-like object` for *path* and *attribute*. @@ -3148,6 +3200,8 @@ These functions are all available on Linux only. This function can support :ref:`specifying a file descriptor ` and :ref:`not following symlinks `. + .. audit-event:: os.listxattr path os.listxattr + .. versionchanged:: 3.6 Accepts a :term:`path-like object`. @@ -3162,6 +3216,8 @@ These functions are all available on Linux only. This function can support :ref:`specifying a file descriptor ` and :ref:`not following symlinks `. + .. audit-event:: os.removexattr path,attribute os.removexattr + .. versionchanged:: 3.6 Accepts a :term:`path-like object` for *path* and *attribute*. @@ -3185,6 +3241,8 @@ These functions are all available on Linux only. A bug in Linux kernel versions less than 2.6.39 caused the flags argument to be ignored on some filesystems. + .. audit-event:: os.setxattr path,attribute,value,flags os.setxattr + .. versionchanged:: 3.6 Accepts a :term:`path-like object` for *path* and *attribute*. @@ -3247,6 +3305,8 @@ to be ignored. `_ for more information about how DLLs are loaded. + .. audit-event:: os.add_dll_directory path os.add_dll_directory + .. availability:: Windows. .. versionadded:: 3.8 @@ -3479,6 +3539,8 @@ written in Python, such as a mail server's external command delivery program. Note that some platforms including FreeBSD <= 6.3 and Cygwin have known issues when using ``fork()`` from a thread. + .. audit-event:: os.fork "" os.fork + .. versionchanged:: 3.8 Calling ``fork()`` in a subinterpreter is no longer supported (:exc:`RuntimeError` is raised). @@ -3498,6 +3560,8 @@ written in Python, such as a mail server's external command delivery program. master end of the pseudo-terminal. For a more portable approach, use the :mod:`pty` module. If an error occurs :exc:`OSError` is raised. + .. audit-event:: os.forkpty "" os.forkpty + .. versionchanged:: 3.8 Calling ``forkpty()`` in a subinterpreter is no longer supported (:exc:`RuntimeError` is raised). @@ -3524,6 +3588,8 @@ written in Python, such as a mail server's external command delivery program. See also :func:`signal.pthread_kill`. + .. audit-event:: os.kill pid,sig os.kill + .. versionadded:: 3.2 Windows support. @@ -3536,6 +3602,8 @@ written in Python, such as a mail server's external command delivery program. Send the signal *sig* to the process group *pgid*. + .. audit-event:: os.killpg pgid,sig os.killpg + .. availability:: Unix. diff --git a/Doc/library/resource.rst b/Doc/library/resource.rst index 3573da7ea2d716..e4eac43642d14d 100644 --- a/Doc/library/resource.rst +++ b/Doc/library/resource.rst @@ -78,6 +78,9 @@ this module for those platforms. VxWorks only supports setting :data:`RLIMIT_NOFILE`. + .. audit-event:: resource.setrlimit resource,limits resource.setrlimit + + .. function:: prlimit(pid, resource[, limits]) Combines :func:`setrlimit` and :func:`getrlimit` in one function and @@ -94,6 +97,8 @@ this module for those platforms. :exc:`PermissionError` when the user doesn't have ``CAP_SYS_RESOURCE`` for the process. + .. audit-event:: resource.prlimit pid,resource,limits resource.prlimit + .. availability:: Linux 2.6.36 or later with glibc 2.13 or later. .. versionadded:: 3.4 diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 174b7e875a38a4..bd24de72023214 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -67,6 +67,8 @@ Directory and files operations a new symbolic link will be created instead of copying the file *src* points to. + .. audit-event:: shutil.copyfile src,dst shutil.copyfile + .. versionchanged:: 3.3 :exc:`IOError` used to be raised instead of :exc:`OSError`. Added *follow_symlinks* argument. @@ -101,6 +103,8 @@ Directory and files operations :func:`copymode` cannot modify symbolic links on the local platform, and it is asked to do so, it will do nothing and return. + .. audit-event:: shutil.copymode src,dst shutil.copymode + .. versionchanged:: 3.3 Added *follow_symlinks* argument. @@ -146,6 +150,8 @@ Directory and files operations Please see :data:`os.supports_follow_symlinks` for more information. + .. audit-event:: shutil.copystat src,dst shutil.copystat + .. versionchanged:: 3.3 Added *follow_symlinks* argument and support for Linux extended attributes. @@ -167,6 +173,10 @@ Directory and files operations To preserve all file metadata from the original, use :func:`~shutil.copy2` instead. + .. audit-event:: shutil.copyfile src,dst shutil.copy + + .. audit-event:: shutil.copymode src,dst shutil.copy + .. versionchanged:: 3.3 Added *follow_symlinks* argument. Now returns path to the newly created file. @@ -194,6 +204,10 @@ Directory and files operations Please see :func:`copystat` for more information about platform support for modifying symbolic link metadata. + .. audit-event:: shutil.copyfile src,dst shutil.copy2 + + .. audit-event:: shutil.copystat src,dst shutil.copy2 + .. versionchanged:: 3.3 Added *follow_symlinks* argument, try to copy extended file system attributes too (currently Linux only). @@ -342,6 +356,8 @@ Directory and files operations *copy_function* allows the move to succeed when it is not possible to also copy the metadata, at the expense of not copying any of the metadata. + .. audit-event:: shutil.move src,dst shutil.move + .. versionchanged:: 3.3 Added explicit symlink handling for foreign filesystems, thus adapting it to the behavior of GNU's :program:`mv`. @@ -378,6 +394,8 @@ Directory and files operations See also :func:`os.chown`, the underlying function. + .. audit-event:: shutil.chown path,user,group shutil.chown + .. availability:: Unix. .. versionadded:: 3.3 @@ -629,6 +647,8 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. registered for that extension. In case none is found, a :exc:`ValueError` is raised. + .. audit-event:: shutil.unpack_archive filename,extract_dir,format shutil.unpack_archive + .. versionchanged:: 3.7 Accepts a :term:`path-like object` for *filename* and *extract_dir*. diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 8fecc2b7eed0e0..932201b0e9d7d3 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -264,6 +264,8 @@ The :mod:`signal` module defines the following functions: If *signalnum* is 0, then no signal is sent, but error checking is still performed; this can be used to check if the target thread is still running. + .. audit-event:: signal.pthread_kill thread_id,signalnum signal.pthread_kill + .. availability:: Unix. See the man page :manpage:`pthread_kill(3)` for further information. diff --git a/Doc/library/syslog.rst b/Doc/library/syslog.rst index 7151527ce57a54..d264a3340c98b0 100644 --- a/Doc/library/syslog.rst +++ b/Doc/library/syslog.rst @@ -31,6 +31,8 @@ The module defines the following functions: If :func:`openlog` has not been called prior to the call to :func:`syslog`, ``openlog()`` will be called with no arguments. + .. audit-event:: syslog.syslog priority,message syslog.syslog + .. function:: openlog([ident[, logoption[, facility]]]) @@ -45,6 +47,8 @@ The module defines the following functions: keyword argument (default is :const:`LOG_USER`) sets the default facility for messages which do not have a facility explicitly encoded. + .. audit-event:: syslog.openlog ident,logoption,facility syslog.openlog + .. versionchanged:: 3.2 In previous versions, keyword arguments were not allowed, and *ident* was required. The default for *ident* was dependent on the system libraries, @@ -60,6 +64,8 @@ The module defines the following functions: :func:`openlog` hasn't already been called), and *ident* and other :func:`openlog` parameters are reset to defaults. + .. audit-event:: syslog.closelog "" syslog.closelog + .. function:: setlogmask(maskpri) @@ -70,6 +76,8 @@ The module defines the following functions: ``LOG_UPTO(pri)`` calculates the mask for all priorities up to and including *pri*. + .. audit-event:: syslog.setlogmask maskpri syslog.setlogmask + The module defines the following constants: Priority levels (high to low): diff --git a/Lib/shutil.py b/Lib/shutil.py index cde7b860050a26..1f05d80f32a8fc 100644 --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -235,6 +235,8 @@ def copyfile(src, dst, *, follow_symlinks=True): symlink will be created instead of copying the file it points to. """ + sys.audit("shutil.copyfile", src, dst) + if _samefile(src, dst): raise SameFileError("{!r} and {!r} are the same file".format(src, dst)) @@ -289,6 +291,8 @@ def copymode(src, dst, *, follow_symlinks=True): (e.g. Linux) this method does nothing. """ + sys.audit("shutil.copymode", src, dst) + if not follow_symlinks and _islink(src) and os.path.islink(dst): if hasattr(os, 'lchmod'): stat_func, chmod_func = os.lstat, os.lchmod @@ -340,6 +344,8 @@ def copystat(src, dst, *, follow_symlinks=True): If the optional flag `follow_symlinks` is not set, symlinks aren't followed if and only if both `src` and `dst` are symlinks. """ + sys.audit("shutil.copystat", src, dst) + def _nop(*args, ns=None, follow_symlinks=None): pass @@ -766,6 +772,7 @@ def move(src, dst, copy_function=copy2): the issues this implementation glosses over. """ + sys.audit("shutil.move", src, dst) real_dst = dst if os.path.isdir(dst): if _samefile(src, dst): @@ -1193,6 +1200,8 @@ def unpack_archive(filename, extract_dir=None, format=None): In case none is found, a ValueError is raised. """ + sys.audit("shutil.unpack_archive", filename, extract_dir, format) + if extract_dir is None: extract_dir = os.getcwd() @@ -1260,6 +1269,7 @@ def chown(path, user=None, group=None): user and group can be the uid/gid or the user/group names, and in that case, they are converted to their respective uid/gid. """ + sys.audit('shutil.chown', path, user, group) if user is None and group is None: raise ValueError("user and/or group must be set") diff --git a/Misc/NEWS.d/next/Security/2020-02-07-23-54-18.bpo-39184.v-ue-v.rst b/Misc/NEWS.d/next/Security/2020-02-07-23-54-18.bpo-39184.v-ue-v.rst new file mode 100644 index 00000000000000..cf25c24d58788f --- /dev/null +++ b/Misc/NEWS.d/next/Security/2020-02-07-23-54-18.bpo-39184.v-ue-v.rst @@ -0,0 +1 @@ +Add audit events to functions in `fcntl`, `msvcrt`, `os`, `resource`, `shutil`, `signal` and `syslog`. \ No newline at end of file diff --git a/Modules/fcntlmodule.c b/Modules/fcntlmodule.c index 0fbf7876c3e207..a7d2193022e982 100644 --- a/Modules/fcntlmodule.c +++ b/Modules/fcntlmodule.c @@ -66,6 +66,10 @@ fcntl_fcntl_impl(PyObject *module, int fd, int code, PyObject *arg) char buf[1024]; int async_err = 0; + if (PySys_Audit("fcntl.fcntl", "iiO", fd, code, arg ? arg : Py_None) < 0) { + return NULL; + } + if (arg != NULL) { int parse_result; @@ -171,6 +175,11 @@ fcntl_ioctl_impl(PyObject *module, int fd, unsigned int code, Py_ssize_t len; char buf[IOCTL_BUFSZ+1]; /* argument plus NUL byte */ + if (PySys_Audit("fcntl.ioctl", "iIO", fd, code, + ob_arg ? ob_arg : Py_None) < 0) { + return NULL; + } + if (ob_arg != NULL) { if (PyArg_Parse(ob_arg, "w*:ioctl", &pstr)) { char *arg; @@ -288,6 +297,10 @@ fcntl_flock_impl(PyObject *module, int fd, int code) int ret; int async_err = 0; + if (PySys_Audit("fcntl.flock", "ii", fd, code) < 0) { + return NULL; + } + #ifdef HAVE_FLOCK do { Py_BEGIN_ALLOW_THREADS @@ -372,6 +385,11 @@ fcntl_lockf_impl(PyObject *module, int fd, int code, PyObject *lenobj, int ret; int async_err = 0; + if (PySys_Audit("fcntl.lockf", "iiOOi", fd, code, lenobj ? lenobj : Py_None, + startobj ? startobj : Py_None, whence) < 0) { + return NULL; + } + #ifndef LOCK_SH #define LOCK_SH 1 /* shared lock */ #define LOCK_EX 2 /* exclusive lock */ diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 88b47164bca09c..2f791d1df953a0 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -2827,6 +2827,10 @@ os_chdir_impl(PyObject *module, path_t *path) { int result; + if (PySys_Audit("os.chdir", "(O)", path->object) < 0) { + return NULL; + } + Py_BEGIN_ALLOW_THREADS #ifdef MS_WINDOWS /* on unix, success = 0, on windows, success = !0 */ @@ -2866,6 +2870,9 @@ static PyObject * os_fchdir_impl(PyObject *module, int fd) /*[clinic end generated code: output=42e064ec4dc00ab0 input=18e816479a2fa985]*/ { + if (PySys_Audit("os.chdir", "(i)", fd) < 0) { + return NULL; + } return posix_fildes_fd(fd, fchdir); } #endif /* HAVE_FCHDIR */ @@ -2923,6 +2930,11 @@ os_chmod_impl(PyObject *module, path_t *path, int mode, int dir_fd, return NULL; #endif + if (PySys_Audit("os.chmod", "Oii", path->object, mode, + dir_fd == DEFAULT_DIR_FD ? -1 : dir_fd) < 0) { + return NULL; + } + #ifdef MS_WINDOWS Py_BEGIN_ALLOW_THREADS attr = GetFileAttributesW(path->wide); @@ -3019,6 +3031,10 @@ os_fchmod_impl(PyObject *module, int fd, int mode) int res; int async_err = 0; + if (PySys_Audit("os.chmod", "iii", fd, mode, -1) < 0) { + return NULL; + } + do { Py_BEGIN_ALLOW_THREADS res = fchmod(fd, mode); @@ -3050,6 +3066,9 @@ os_lchmod_impl(PyObject *module, path_t *path, int mode) /*[clinic end generated code: output=082344022b51a1d5 input=90c5663c7465d24f]*/ { int res; + if (PySys_Audit("os.chmod", "Oii", path->object, mode, -1) < 0) { + return NULL; + } Py_BEGIN_ALLOW_THREADS res = lchmod(path->narrow, mode); Py_END_ALLOW_THREADS @@ -3092,6 +3111,10 @@ os_chflags_impl(PyObject *module, path_t *path, unsigned long flags, return NULL; #endif + if (PySys_Audit("os.chflags", "Ok", path->object, flags) < 0) { + return NULL; + } + Py_BEGIN_ALLOW_THREADS #ifdef HAVE_LCHFLAGS if (!follow_symlinks) @@ -3127,6 +3150,9 @@ os_lchflags_impl(PyObject *module, path_t *path, unsigned long flags) /*[clinic end generated code: output=30ae958695c07316 input=f9f82ea8b585ca9d]*/ { int res; + if (PySys_Audit("os.chflags", "Ok", path->object, flags) < 0) { + return NULL; + } Py_BEGIN_ALLOW_THREADS res = lchflags(path->narrow, flags); Py_END_ALLOW_THREADS @@ -3289,6 +3315,11 @@ os_chown_impl(PyObject *module, path_t *path, uid_t uid, gid_t gid, } #endif + if (PySys_Audit("os.chown", "OIIi", path->object, uid, gid, + dir_fd == DEFAULT_DIR_FD ? -1 : dir_fd) < 0) { + return NULL; + } + Py_BEGIN_ALLOW_THREADS #ifdef HAVE_FCHOWN if (path->fd != -1) @@ -3338,6 +3369,10 @@ os_fchown_impl(PyObject *module, int fd, uid_t uid, gid_t gid) int res; int async_err = 0; + if (PySys_Audit("os.chown", "iIIi", fd, uid, gid, -1) < 0) { + return NULL; + } + do { Py_BEGIN_ALLOW_THREADS res = fchown(fd, uid, gid); @@ -3370,6 +3405,9 @@ os_lchown_impl(PyObject *module, path_t *path, uid_t uid, gid_t gid) /*[clinic end generated code: output=25eaf6af412fdf2f input=b1c6014d563a7161]*/ { int res; + if (PySys_Audit("os.chown", "OIIi", path->object, uid, gid, -1) < 0) { + return NULL; + } Py_BEGIN_ALLOW_THREADS res = lchown(path->narrow, uid, gid); Py_END_ALLOW_THREADS @@ -3563,6 +3601,12 @@ os_link_impl(PyObject *module, path_t *src, path_t *dst, int src_dir_fd, } #endif + if (PySys_Audit("os.link", "OOii", src->object, dst->object, + src_dir_fd == DEFAULT_DIR_FD ? -1 : src_dir_fd, + dst_dir_fd == DEFAULT_DIR_FD ? -1 : dst_dir_fd) < 0) { + return NULL; + } + #ifdef MS_WINDOWS Py_BEGIN_ALLOW_THREADS result = CreateHardLinkW(dst->wide, src->wide, NULL); @@ -4034,6 +4078,11 @@ os_mkdir_impl(PyObject *module, path_t *path, int mode, int dir_fd) { int result; + if (PySys_Audit("os.mkdir", "Oii", path->object, mode, + dir_fd == DEFAULT_DIR_FD ? -1 : dir_fd) < 0) { + return NULL; + } + #ifdef MS_WINDOWS Py_BEGIN_ALLOW_THREADS result = CreateDirectoryW(path->wide, NULL); @@ -4179,6 +4228,12 @@ internal_rename(path_t *src, path_t *dst, int src_dir_fd, int dst_dir_fd, int is } #endif + if (PySys_Audit("os.rename", "OOii", src->object, dst->object, + src_dir_fd == DEFAULT_DIR_FD ? -1 : src_dir_fd, + dst_dir_fd == DEFAULT_DIR_FD ? -1 : dst_dir_fd) < 0) { + return NULL; + } + #ifdef MS_WINDOWS Py_BEGIN_ALLOW_THREADS result = MoveFileExW(src->wide, dst->wide, flags); @@ -4279,6 +4334,11 @@ os_rmdir_impl(PyObject *module, path_t *path, int dir_fd) { int result; + if (PySys_Audit("os.rmdir", "Oi", path->object, + dir_fd == DEFAULT_DIR_FD ? -1 : dir_fd) < 0) { + return NULL; + } + Py_BEGIN_ALLOW_THREADS #ifdef MS_WINDOWS /* Windows, success=1, UNIX, success=0 */ @@ -4437,6 +4497,11 @@ os_unlink_impl(PyObject *module, path_t *path, int dir_fd) { int result; + if (PySys_Audit("os.remove", "Oi", path->object, + dir_fd == DEFAULT_DIR_FD ? -1 : dir_fd) < 0) { + return NULL; + } + Py_BEGIN_ALLOW_THREADS _Py_BEGIN_SUPPRESS_IPH #ifdef MS_WINDOWS @@ -4855,6 +4920,11 @@ os_utime_impl(PyObject *module, path_t *path, PyObject *times, PyObject *ns, } #endif + if (PySys_Audit("os.utime", "OOOi", path->object, times, ns ? ns : Py_None, + dir_fd == DEFAULT_DIR_FD ? -1 : dir_fd) < 0) { + return NULL; + } + #ifdef MS_WINDOWS Py_BEGIN_ALLOW_THREADS hFile = CreateFileW(path->wide, FILE_WRITE_ATTRIBUTES, 0, @@ -5156,8 +5226,7 @@ os_execv_impl(PyObject *module, path_t *path, PyObject *argv) return NULL; } - if (PySys_Audit("os.exec", "OOO", path->object ? path->object : Py_None, - argv, Py_None) < 0) { + if (PySys_Audit("os.exec", "OOO", path->object, argv, Py_None) < 0) { free_string_array(argvlist, argc); return NULL; } @@ -5233,8 +5302,7 @@ os_execve_impl(PyObject *module, path_t *path, PyObject *argv, PyObject *env) if (envlist == NULL) goto fail_0; - if (PySys_Audit("os.exec", "OOO", path->object ? path->object : Py_None, - argv, env) < 0) { + if (PySys_Audit("os.exec", "OOO", path->object, argv, env) < 0) { goto fail_1; } @@ -5587,8 +5655,7 @@ py_posix_spawn(int use_posix_spawnp, PyObject *module, path_t *path, PyObject *a } attrp = &attr; - if (PySys_Audit("os.posix_spawn", "OOO", - path->object ? path->object : Py_None, argv, env) < 0) { + if (PySys_Audit("os.posix_spawn", "OOO", path->object, argv, env) < 0) { goto exit; } @@ -5832,8 +5899,7 @@ os_spawnv_impl(PyObject *module, int mode, path_t *path, PyObject *argv) mode = _P_OVERLAY; #endif - if (PySys_Audit("os.spawn", "iOOO", mode, - path->object ? path->object : Py_None, argv, + if (PySys_Audit("os.spawn", "iOOO", mode, path->object, argv, Py_None) < 0) { free_string_array(argvlist, argc); return NULL; @@ -5948,8 +6014,7 @@ os_spawnve_impl(PyObject *module, int mode, path_t *path, PyObject *argv, mode = _P_OVERLAY; #endif - if (PySys_Audit("os.spawn", "iOOO", mode, - path->object ? path->object : Py_None, argv, env) < 0) { + if (PySys_Audit("os.spawn", "iOOO", mode, path->object, argv, env) < 0) { goto fail_2; } @@ -6104,6 +6169,9 @@ os_fork_impl(PyObject *module) PyErr_SetString(PyExc_RuntimeError, "fork not supported for subinterpreters"); return NULL; } + if (PySys_Audit("os.fork", NULL) < 0) { + return NULL; + } PyOS_BeforeFork(); pid = fork(); if (pid == 0) { @@ -6709,6 +6777,9 @@ os_forkpty_impl(PyObject *module) PyErr_SetString(PyExc_RuntimeError, "fork not supported for subinterpreters"); return NULL; } + if (PySys_Audit("os.forkpty", NULL) < 0) { + return NULL; + } PyOS_BeforeFork(); pid = forkpty(&master_fd, NULL, NULL, NULL); if (pid == 0) { @@ -7230,14 +7301,15 @@ Kill a process with a signal. static PyObject * os_kill_impl(PyObject *module, pid_t pid, Py_ssize_t signal) /*[clinic end generated code: output=8e346a6701c88568 input=61a36b86ca275ab9]*/ -#ifndef MS_WINDOWS { + if (PySys_Audit("os.kill", "in", pid, signal) < 0) { + return NULL; + } +#ifndef MS_WINDOWS if (kill(pid, (int)signal) == -1) return posix_error(); Py_RETURN_NONE; -} #else /* !MS_WINDOWS */ -{ PyObject *result; DWORD sig = (DWORD)signal; DWORD err; @@ -7272,8 +7344,8 @@ os_kill_impl(PyObject *module, pid_t pid, Py_ssize_t signal) CloseHandle(handle); return result; -} #endif /* !MS_WINDOWS */ +} #endif /* HAVE_KILL */ @@ -7292,6 +7364,9 @@ static PyObject * os_killpg_impl(PyObject *module, pid_t pgid, int signal) /*[clinic end generated code: output=6dbcd2f1fdf5fdba input=38b5449eb8faec19]*/ { + if (PySys_Audit("os.killpg", "ii", pgid, signal) < 0) { + return NULL; + } /* XXX some man pages make the `pgid` parameter an int, others a pid_t. Since getpgrp() returns a pid_t, we assume killpg should take the same type. Moreover, pid_t is always at least as wide as @@ -8035,6 +8110,11 @@ os_symlink_impl(PyObject *module, path_t *src, path_t *dst, int result; #endif + if (PySys_Audit("os.symlink", "OOi", src->object, dst->object, + dir_fd == DEFAULT_DIR_FD ? -1 : dir_fd) < 0) { + return NULL; + } + #ifdef MS_WINDOWS if (windows_has_symlink_unprivileged_flag) { @@ -8638,6 +8718,10 @@ os_lockf_impl(PyObject *module, int fd, int command, Py_off_t length) { int res; + if (PySys_Audit("os.lockf", "iiL", fd, command, length) < 0) { + return NULL; + } + Py_BEGIN_ALLOW_THREADS res = lockf(fd, command, length); Py_END_ALLOW_THREADS @@ -10070,6 +10154,9 @@ os_putenv_impl(PyObject *module, PyObject *name, PyObject *value) PyErr_SetString(PyExc_ValueError, "illegal environment variable name"); return NULL; } + if (PySys_Audit("os.putenv", "OO", name, value) < 0) { + return NULL; + } bytes = PyBytes_FromFormat("%s=%s", name_string, value_string); if (bytes == NULL) { return NULL; @@ -10105,6 +10192,10 @@ os_unsetenv_impl(PyObject *module, PyObject *name) int err; #endif + if (PySys_Audit("os.unsetenv", "(O)", name) < 0) { + return NULL; + } + #ifdef HAVE_BROKEN_UNSETENV unsetenv(PyBytes_AS_STRING(name)); #else @@ -11637,9 +11728,7 @@ os_startfile_impl(PyObject *module, path_t *filepath, "startfile not available on this platform"); } - if (PySys_Audit("os.startfile", "Ou", - filepath->object ? filepath->object : Py_None, - operation) < 0) { + if (PySys_Audit("os.startfile", "Ou", filepath->object, operation) < 0) { return NULL; } @@ -11817,6 +11906,10 @@ os_getxattr_impl(PyObject *module, path_t *path, path_t *attribute, if (fd_and_follow_symlinks_invalid("getxattr", path->fd, follow_symlinks)) return NULL; + if (PySys_Audit("os.getxattr", "OO", path->object, attribute->object) < 0) { + return NULL; + } + for (i = 0; ; i++) { void *ptr; ssize_t result; @@ -11888,6 +11981,11 @@ os_setxattr_impl(PyObject *module, path_t *path, path_t *attribute, if (fd_and_follow_symlinks_invalid("setxattr", path->fd, follow_symlinks)) return NULL; + if (PySys_Audit("os.setxattr", "OOy#i", path->object, attribute->object, + value->buf, value->len, flags) < 0) { + return NULL; + } + Py_BEGIN_ALLOW_THREADS; if (path->fd > -1) result = fsetxattr(path->fd, attribute->narrow, @@ -11936,6 +12034,10 @@ os_removexattr_impl(PyObject *module, path_t *path, path_t *attribute, if (fd_and_follow_symlinks_invalid("removexattr", path->fd, follow_symlinks)) return NULL; + if (PySys_Audit("os.removexattr", "OO", path->object, attribute->object) < 0) { + return NULL; + } + Py_BEGIN_ALLOW_THREADS; if (path->fd > -1) result = fremovexattr(path->fd, attribute->narrow); @@ -11981,6 +12083,11 @@ os_listxattr_impl(PyObject *module, path_t *path, int follow_symlinks) if (fd_and_follow_symlinks_invalid("listxattr", path->fd, follow_symlinks)) goto exit; + if (PySys_Audit("os.listxattr", "(O)", + path->object ? path->object : Py_None) < 0) { + return NULL; + } + name = path->narrow ? path->narrow : "."; for (i = 0; ; i++) { @@ -13494,6 +13601,10 @@ os__add_dll_directory_impl(PyObject *module, path_t *path) DLL_DIRECTORY_COOKIE cookie = 0; DWORD err = 0; + if (PySys_Audit("os.add_dll_directory", "(O)", path->object) < 0) { + return NULL; + } + /* For Windows 7, we have to load this. As this will be a fairly infrequent operation, just do it each time. Kernel32 is always loaded. */ diff --git a/Modules/resource.c b/Modules/resource.c index 87c72e7409895c..afde03c6c7e559 100644 --- a/Modules/resource.c +++ b/Modules/resource.c @@ -224,6 +224,11 @@ resource_setrlimit_impl(PyObject *module, int resource, PyObject *limits) return NULL; } + if (PySys_Audit("resource.setrlimit", "iO", resource, + limits ? limits : Py_None) < 0) { + return NULL; + } + if (py2rlimit(limits, &rl) < 0) { return NULL; } @@ -269,6 +274,11 @@ resource_prlimit_impl(PyObject *module, pid_t pid, int resource, return NULL; } + if (PySys_Audit("resource.prlimit", "iiO", pid, resource, + limits ? limits : Py_None) < 0) { + return NULL; + } + if (group_right_1) { if (py2rlimit(limits, &new_limit) < 0) { return NULL; diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c index 9aca70599689b6..0c9a2671fe19b2 100644 --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -1233,6 +1233,10 @@ signal_pthread_kill_impl(PyObject *module, unsigned long thread_id, { int err; + if (PySys_Audit("signal.pthread_kill", "ki", thread_id, signalnum) < 0) { + return NULL; + } + err = pthread_kill((pthread_t)thread_id, signalnum); if (err != 0) { errno = err; diff --git a/Modules/syslogmodule.c b/Modules/syslogmodule.c index b2ea73baa1be48..66ea2703fc84a7 100644 --- a/Modules/syslogmodule.c +++ b/Modules/syslogmodule.c @@ -144,6 +144,10 @@ syslog_openlog(PyObject * self, PyObject * args, PyObject *kwds) return NULL; } + if (PySys_Audit("syslog.openlog", "sll", ident, logopt, facility) < 0) { + return NULL; + } + openlog(ident, logopt, facility); S_log_open = 1; @@ -170,6 +174,10 @@ syslog_syslog(PyObject * self, PyObject * args) if (message == NULL) return NULL; + if (PySys_Audit("syslog.syslog", "is", priority, message) < 0) { + return NULL; + } + /* if log is not opened, open it now */ if (!S_log_open) { PyObject *openargs; @@ -194,6 +202,9 @@ syslog_syslog(PyObject * self, PyObject * args) static PyObject * syslog_closelog(PyObject *self, PyObject *unused) { + if (PySys_Audit("syslog.closelog", NULL) < 0) { + return NULL; + } if (S_log_open) { closelog(); Py_CLEAR(S_ident_o); @@ -209,6 +220,9 @@ syslog_setlogmask(PyObject *self, PyObject *args) if (!PyArg_ParseTuple(args, "l;mask for priority", &maskpri)) return NULL; + if (PySys_Audit("syslog.setlogmask", "(O)", args ? args : Py_None) < 0) { + return NULL; + } omaskpri = setlogmask(maskpri); return PyLong_FromLong(omaskpri); } diff --git a/PC/msvcrtmodule.c b/PC/msvcrtmodule.c index c4113e54c2b8fe..5c06ec2621ea6c 100644 --- a/PC/msvcrtmodule.c +++ b/PC/msvcrtmodule.c @@ -116,6 +116,10 @@ msvcrt_locking_impl(PyObject *module, int fd, int mode, long nbytes) { int err; + if (PySys_Audit("msvcrt.locking", "iil", fd, mode, nbytes) < 0) { + return NULL; + } + Py_BEGIN_ALLOW_THREADS _Py_BEGIN_SUPPRESS_IPH err = _locking(fd, mode, nbytes); @@ -175,6 +179,10 @@ msvcrt_open_osfhandle_impl(PyObject *module, void *handle, int flags) { int fd; + if (PySys_Audit("msvcrt.open_osfhandle", "Ki", handle, flags) < 0) { + return NULL; + } + _Py_BEGIN_SUPPRESS_IPH fd = _open_osfhandle((intptr_t)handle, flags); _Py_END_SUPPRESS_IPH @@ -201,6 +209,10 @@ msvcrt_get_osfhandle_impl(PyObject *module, int fd) { intptr_t handle = -1; + if (PySys_Audit("msvcrt.get_osfhandle", "(i)", fd) < 0) { + return NULL; + } + _Py_BEGIN_SUPPRESS_IPH handle = _get_osfhandle(fd); _Py_END_SUPPRESS_IPH From 8dbdf5f275c6462bb522bcf3a29054239d72989d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 13 Feb 2020 00:43:23 -0800 Subject: [PATCH 1187/2163] [3.8] bpo-39606: allow closing async generators that are already closed (GH-18475) (GH-18501) The fix for [bpo-39386](https://bugs.python.org/issue39386) attempted to make it so you couldn't reuse a agen.aclose() coroutine object. It accidentally also prevented you from calling aclose() at all on an async generator that was already closed or exhausted. This commit fixes it so we're only blocking the actually illegal cases, while allowing the legal cases. The new tests failed before this patch. Also confirmed that this fixes the test failures we were seeing in Trio with Python dev builds: https://github.com/python-trio/trio/pull/1396 https://bugs.python.org/issue39606 (cherry picked from commit 925dc7fb1d0db85dc137afa4cd14211bf0d67414) Co-authored-by: Nathaniel J. Smith https://bugs.python.org/issue39606 Automerge-Triggered-By: @njsmith --- Lib/test/test_asyncgen.py | 30 +++++++++++++++++-- .../2020-02-11-23-59-07.bpo-39606.a72Sxc.rst | 2 ++ Objects/genobject.c | 15 +++++++--- 3 files changed, 41 insertions(+), 6 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-02-11-23-59-07.bpo-39606.a72Sxc.rst diff --git a/Lib/test/test_asyncgen.py b/Lib/test/test_asyncgen.py index 24b20bec2b2d1e..fb6321d2264f31 100644 --- a/Lib/test/test_asyncgen.py +++ b/Lib/test/test_asyncgen.py @@ -1128,7 +1128,7 @@ async def main(): self.assertEqual([], messages) - def test_async_gen_await_anext_twice(self): + def test_async_gen_await_same_anext_coro_twice(self): async def async_iterate(): yield 1 yield 2 @@ -1147,7 +1147,7 @@ async def run(): self.loop.run_until_complete(run()) - def test_async_gen_await_aclose_twice(self): + def test_async_gen_await_same_aclose_coro_twice(self): async def async_iterate(): yield 1 yield 2 @@ -1164,6 +1164,32 @@ async def run(): self.loop.run_until_complete(run()) + def test_async_gen_aclose_twice_with_different_coros(self): + # Regression test for https://bugs.python.org/issue39606 + async def async_iterate(): + yield 1 + yield 2 + + async def run(): + it = async_iterate() + await it.aclose() + await it.aclose() + + self.loop.run_until_complete(run()) + + def test_async_gen_aclose_after_exhaustion(self): + # Regression test for https://bugs.python.org/issue39606 + async def async_iterate(): + yield 1 + yield 2 + + async def run(): + it = async_iterate() + async for _ in it: + pass + await it.aclose() + + self.loop.run_until_complete(run()) if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-02-11-23-59-07.bpo-39606.a72Sxc.rst b/Misc/NEWS.d/next/Core and Builtins/2020-02-11-23-59-07.bpo-39606.a72Sxc.rst new file mode 100644 index 00000000000000..b7cbe4e91f59c9 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-02-11-23-59-07.bpo-39606.a72Sxc.rst @@ -0,0 +1,2 @@ +Fix regression caused by fix for bpo-39386, that prevented calling +``aclose`` on an async generator that had already been closed or exhausted. diff --git a/Objects/genobject.c b/Objects/genobject.c index 72aa872c6b59f5..2c06bdcc72642f 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -1812,16 +1812,22 @@ async_gen_athrow_send(PyAsyncGenAThrow *o, PyObject *arg) PyFrameObject *f = gen->gi_frame; PyObject *retval; - if (f == NULL || f->f_stacktop == NULL || - o->agt_state == AWAITABLE_STATE_CLOSED) { + if (o->agt_state == AWAITABLE_STATE_CLOSED) { PyErr_SetString( PyExc_RuntimeError, "cannot reuse already awaited aclose()/athrow()"); return NULL; } + if (f == NULL || f->f_stacktop == NULL) { + o->agt_state = AWAITABLE_STATE_CLOSED; + PyErr_SetNone(PyExc_StopIteration); + return NULL; + } + if (o->agt_state == AWAITABLE_STATE_INIT) { if (o->agt_gen->ag_running_async) { + o->agt_state = AWAITABLE_STATE_CLOSED; if (o->agt_args == NULL) { PyErr_SetString( PyExc_RuntimeError, @@ -1893,7 +1899,6 @@ async_gen_athrow_send(PyAsyncGenAThrow *o, PyObject *arg) /* aclose() mode */ if (retval) { if (_PyAsyncGenWrappedValue_CheckExact(retval)) { - o->agt_gen->ag_running_async = 0; Py_DECREF(retval); goto yield_close; } @@ -1908,16 +1913,17 @@ async_gen_athrow_send(PyAsyncGenAThrow *o, PyObject *arg) yield_close: o->agt_gen->ag_running_async = 0; + o->agt_state = AWAITABLE_STATE_CLOSED; PyErr_SetString( PyExc_RuntimeError, ASYNC_GEN_IGNORED_EXIT_MSG); return NULL; check_error: o->agt_gen->ag_running_async = 0; + o->agt_state = AWAITABLE_STATE_CLOSED; if (PyErr_ExceptionMatches(PyExc_StopAsyncIteration) || PyErr_ExceptionMatches(PyExc_GeneratorExit)) { - o->agt_state = AWAITABLE_STATE_CLOSED; if (o->agt_args == NULL) { /* when aclose() is called we don't want to propagate StopAsyncIteration or GeneratorExit; just raise @@ -1951,6 +1957,7 @@ async_gen_athrow_throw(PyAsyncGenAThrow *o, PyObject *args) /* aclose() mode */ if (retval && _PyAsyncGenWrappedValue_CheckExact(retval)) { o->agt_gen->ag_running_async = 0; + o->agt_state = AWAITABLE_STATE_CLOSED; Py_DECREF(retval); PyErr_SetString(PyExc_RuntimeError, ASYNC_GEN_IGNORED_EXIT_MSG); return NULL; From 581b8606ca0609cf36c4eb9a5bb025eb77540e5e Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 13 Feb 2020 16:03:59 -0800 Subject: [PATCH 1188/2163] bpo-39545: Document changes in the support of await in f-strings. (GH-18456) https://bugs.python.org/issue39545 (cherry picked from commit f632736023502816f2e6bd714d1b48c81aa2ccc1) Co-authored-by: Serhiy Storchaka --- Doc/reference/lexical_analysis.rst | 5 +++++ Doc/whatsnew/3.7.rst | 5 +++++ 2 files changed, 10 insertions(+) diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index c0e13b53698e6b..3d4b03e6bd459c 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -685,6 +685,11 @@ strings), but they cannot contain comments. Each expression is evaluated in the context where the formatted string literal appears, in order from left to right. +.. versionchanged:: 3.7 + Prior to Python 3.7, an :keyword:`await` expression and comprehensions + containing an :keyword:`async for` clause were illegal in the expressions + in formatted string literals due to a problem with the implementation. + If a conversion is specified, the result of evaluating the expression is converted before formatting. Conversion ``'!s'`` calls :func:`str` on the result, ``'!r'`` calls :func:`repr`, and ``'!a'`` calls :func:`ascii`. diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 8a70fe22d52bdc..b9b50216d23a63 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -493,6 +493,11 @@ of this mode. Other Language Changes ====================== +* An :keyword:`await` expression and comprehensions containing an + :keyword:`async for` clause were illegal in the expressions in + :ref:`formatted string literals ` due to a problem with the + implementation. In Python 3.7 this restriction was lifted. + * More than 255 arguments can now be passed to a function, and a function can now have more than 255 parameters. (Contributed by Serhiy Storchaka in :issue:`12844` and :issue:`18896`.) From 28fc1bac0fbe1f4ae2e3dcba1dee38d2c063a539 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 13 Feb 2020 19:27:28 -0800 Subject: [PATCH 1189/2163] closes bpo-39619 Fix os.chroot on HP-UX 11.31 (GH-18495) Setting `-D_XOPEN_SOURCE=700` on HP-UX causes system functions such as chroot to be undefined. This change stops `_XOPEN_SOURCE` begin set on HP-UX Co-authored-by: Benjamin Peterson (cherry picked from commit a9edf44a2de9b23a1690b36cdfeed7b41ab763bd) Co-authored-by: Ian Norton --- ...3-07-35-00.bpo-39619.inb_master_chroot.rst | 1 + configure | 20 ++++++++++++++++++- configure.ac | 6 ++++++ 3 files changed, 26 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-02-13-07-35-00.bpo-39619.inb_master_chroot.rst diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-02-13-07-35-00.bpo-39619.inb_master_chroot.rst b/Misc/NEWS.d/next/Core and Builtins/2020-02-13-07-35-00.bpo-39619.inb_master_chroot.rst new file mode 100644 index 00000000000000..18f32f7e804bdc --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-02-13-07-35-00.bpo-39619.inb_master_chroot.rst @@ -0,0 +1 @@ +Enable use of :func:`os.chroot` on HP-UX systems. diff --git a/configure b/configure index 0914e247043832..a979363acffc41 100755 --- a/configure +++ b/configure @@ -782,6 +782,7 @@ infodir docdir oldincludedir includedir +runstatedir localstatedir sharedstatedir sysconfdir @@ -895,6 +896,7 @@ datadir='${datarootdir}' sysconfdir='${prefix}/etc' sharedstatedir='${prefix}/com' localstatedir='${prefix}/var' +runstatedir='${localstatedir}/run' includedir='${prefix}/include' oldincludedir='/usr/include' docdir='${datarootdir}/doc/${PACKAGE_TARNAME}' @@ -1147,6 +1149,15 @@ do | -silent | --silent | --silen | --sile | --sil) silent=yes ;; + -runstatedir | --runstatedir | --runstatedi | --runstated \ + | --runstate | --runstat | --runsta | --runst | --runs \ + | --run | --ru | --r) + ac_prev=runstatedir ;; + -runstatedir=* | --runstatedir=* | --runstatedi=* | --runstated=* \ + | --runstate=* | --runstat=* | --runsta=* | --runst=* | --runs=* \ + | --run=* | --ru=* | --r=*) + runstatedir=$ac_optarg ;; + -sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb) ac_prev=sbindir ;; -sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \ @@ -1284,7 +1295,7 @@ fi for ac_var in exec_prefix prefix bindir sbindir libexecdir datarootdir \ datadir sysconfdir sharedstatedir localstatedir includedir \ oldincludedir docdir infodir htmldir dvidir pdfdir psdir \ - libdir localedir mandir + libdir localedir mandir runstatedir do eval ac_val=\$$ac_var # Remove trailing slashes. @@ -1437,6 +1448,7 @@ Fine tuning of the installation directories: --sysconfdir=DIR read-only single-machine data [PREFIX/etc] --sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com] --localstatedir=DIR modifiable single-machine data [PREFIX/var] + --runstatedir=DIR modifiable per-process data [LOCALSTATEDIR/run] --libdir=DIR object code libraries [EPREFIX/lib] --includedir=DIR C header files [PREFIX/include] --oldincludedir=DIR C header files for non-gcc [/usr/include] @@ -3405,6 +3417,12 @@ $as_echo "#define _BSD_SOURCE 1" >>confdefs.h define_xopen_source=no ;; + # On HP-UX, defining _XOPEN_SOURCE to 600 or greater hides + # chroot() and other functions + hp*|HP*) + define_xopen_source=no + ;; + esac if test $define_xopen_source = yes diff --git a/configure.ac b/configure.ac index 7051dc109a725d..e57ef7c38bf7f3 100644 --- a/configure.ac +++ b/configure.ac @@ -521,6 +521,12 @@ case $ac_sys_system/$ac_sys_release in define_xopen_source=no ;; + # On HP-UX, defining _XOPEN_SOURCE to 600 or greater hides + # chroot() and other functions + hp*|HP*) + define_xopen_source=no + ;; + esac if test $define_xopen_source = yes From 0d860dd43c72dc7046a5d18fc72d495cadd4a2df Mon Sep 17 00:00:00 2001 From: Benjamin Peterson Date: Thu, 13 Feb 2020 21:05:00 -0800 Subject: [PATCH 1190/2163] [3.8] closes bpo-39630: Update pointers to string literals to be const char *. (GH-18511) (cherry picked from commit 7386a70746cf9aaf2d95db75d9201fb124f085df) Co-authored-by: Andy Lester --- Objects/genobject.c | 4 ++-- Python/codecs.c | 2 +- Python/errors.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Objects/genobject.c b/Objects/genobject.c index 2c06bdcc72642f..ce7dd48a17cfb6 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -11,10 +11,10 @@ static PyObject *gen_close(PyGenObject *, PyObject *); static PyObject *async_gen_asend_new(PyAsyncGenObject *, PyObject *); static PyObject *async_gen_athrow_new(PyAsyncGenObject *, PyObject *); -static char *NON_INIT_CORO_MSG = "can't send non-None value to a " +static const char *NON_INIT_CORO_MSG = "can't send non-None value to a " "just-started coroutine"; -static char *ASYNC_GEN_IGNORED_EXIT_MSG = +static const char *ASYNC_GEN_IGNORED_EXIT_MSG = "async generator ignored GeneratorExit"; static inline int diff --git a/Python/codecs.c b/Python/codecs.c index d4b34f8397f05a..4bd28ec9c761ca 100644 --- a/Python/codecs.c +++ b/Python/codecs.c @@ -1415,7 +1415,7 @@ static PyObject *surrogateescape_errors(PyObject *self, PyObject *exc) static int _PyCodecRegistry_Init(void) { static struct { - char *name; + const char *name; PyMethodDef def; } methods[] = { diff --git a/Python/errors.c b/Python/errors.c index 197d9779b390c7..1360c0d91a33fa 100644 --- a/Python/errors.c +++ b/Python/errors.c @@ -584,7 +584,7 @@ PyErr_SetFromErrnoWithFilenameObjects(PyObject *exc, PyObject *filenameObject, P #ifndef MS_WINDOWS if (i != 0) { - char *s = strerror(i); + const char *s = strerror(i); message = PyUnicode_DecodeLocale(s, "surrogateescape"); } else { From ea316fd21527dec53e704a5b04833ac462ce3863 Mon Sep 17 00:00:00 2001 From: Senthil Kumaran Date: Sun, 16 Feb 2020 13:47:21 -0800 Subject: [PATCH 1191/2163] Revert "[3.8] bpo-27657: Fix urlparse() with numeric paths (GH-16839)" (GH-18525) This reverts commit 0f3187c1ce3b3ace60f6c1691dfa3d4e744f0384. The change broke the backwards compatibility of parsing behavior in a patch release of Python (3.8.1). A decision was taken to revert this patch in 3.8.2. In https://bugs.python.org/issue27657 it was decided that the previous behavior like >>> urlparse('localhost:8080') ParseResult(scheme='', netloc='', path='localhost:8080', params='', query='', fragment='') >>> urlparse('undefined:8080') ParseResult(scheme='', netloc='', path='undefined:8080', params='', query='', fragment='') needs to be preserved in patch releases as number of users rely upon it. Explicitly mention the releases involved with the revert in NEWS. Adopt the wording suggested by @ned-deily. --- Lib/test/test_urlparse.py | 10 ++++----- Lib/urllib/parse.py | 22 ++++++++++++++++++- .../2020-02-16-07-08-54.bpo-27657.9atgcz.rst | 5 +++++ 3 files changed, 30 insertions(+), 7 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-02-16-07-08-54.bpo-27657.9atgcz.rst diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py index 762500789f73ac..4ae6ed33858ce2 100644 --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -709,17 +709,15 @@ def test_withoutscheme(self): def test_portseparator(self): # Issue 754016 makes changes for port separator ':' from scheme separator - self.assertEqual(urllib.parse.urlparse("http:80"), ('http','','80','','','')) - self.assertEqual(urllib.parse.urlparse("https:80"), ('https','','80','','','')) - self.assertEqual(urllib.parse.urlparse("path:80"), ('path','','80','','','')) + self.assertEqual(urllib.parse.urlparse("path:80"), + ('','','path:80','','','')) self.assertEqual(urllib.parse.urlparse("http:"),('http','','','','','')) self.assertEqual(urllib.parse.urlparse("https:"),('https','','','','','')) self.assertEqual(urllib.parse.urlparse("http://www.python.org:80"), ('http','www.python.org:80','','','','')) # As usual, need to check bytes input as well - self.assertEqual(urllib.parse.urlparse(b"http:80"), (b'http',b'',b'80',b'',b'',b'')) - self.assertEqual(urllib.parse.urlparse(b"https:80"), (b'https',b'',b'80',b'',b'',b'')) - self.assertEqual(urllib.parse.urlparse(b"path:80"), (b'path',b'',b'80',b'',b'',b'')) + self.assertEqual(urllib.parse.urlparse(b"path:80"), + (b'',b'',b'path:80',b'',b'',b'')) self.assertEqual(urllib.parse.urlparse(b"http:"),(b'http',b'',b'',b'',b'',b'')) self.assertEqual(urllib.parse.urlparse(b"https:"),(b'https',b'',b'',b'',b'',b'')) self.assertEqual(urllib.parse.urlparse(b"http://www.python.org:80"), diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py index 0b39b6eaf7ded1..e2b6f133e1cd9c 100644 --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -431,11 +431,31 @@ def urlsplit(url, scheme='', allow_fragments=True): netloc = query = fragment = '' i = url.find(':') if i > 0: + if url[:i] == 'http': # optimize the common case + url = url[i+1:] + if url[:2] == '//': + netloc, url = _splitnetloc(url, 2) + if (('[' in netloc and ']' not in netloc) or + (']' in netloc and '[' not in netloc)): + raise ValueError("Invalid IPv6 URL") + if allow_fragments and '#' in url: + url, fragment = url.split('#', 1) + if '?' in url: + url, query = url.split('?', 1) + _checknetloc(netloc) + v = SplitResult('http', netloc, url, query, fragment) + _parse_cache[key] = v + return _coerce_result(v) for c in url[:i]: if c not in scheme_chars: break else: - scheme, url = url[:i].lower(), url[i+1:] + # make sure "url" is not actually a port number (in which case + # "scheme" is really part of the path) + rest = url[i+1:] + if not rest or any(c not in '0123456789' for c in rest): + # not a port number + scheme, url = url[:i].lower(), rest if url[:2] == '//': netloc, url = _splitnetloc(url, 2) diff --git a/Misc/NEWS.d/next/Library/2020-02-16-07-08-54.bpo-27657.9atgcz.rst b/Misc/NEWS.d/next/Library/2020-02-16-07-08-54.bpo-27657.9atgcz.rst new file mode 100644 index 00000000000000..50bf8b28217b18 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-16-07-08-54.bpo-27657.9atgcz.rst @@ -0,0 +1,5 @@ +The original fix for bpo-27657, "Fix urlparse() with numeric paths" (GH-16839) +included in 3.8.1, inadvertently introduced a behavior change that broke +several third-party packages relying on the original undefined parsing +behavior. The change is reverted in 3.8.2, restoring the behavior of 3.8.0 and +earlier releases. From 988aeba94bf1dab81dd52fc7b02dca7a57ea8ba0 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 17 Feb 2020 01:09:49 -0800 Subject: [PATCH 1192/2163] bpo-32892: Update the documentation for handling constants in AST. (GH-18514) (cherry picked from commit 85a2eef473a2c9ed3ab9c6ee339891fe99adbbc9) Co-authored-by: Serhiy Storchaka --- Doc/library/ast.rst | 10 +++++++--- Doc/whatsnew/3.8.rst | 6 ++++++ 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index a5dd0e1cc70dd6..80afbbce4167eb 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -101,12 +101,16 @@ Node classes node = ast.UnaryOp(ast.USub(), ast.Constant(5, lineno=0, col_offset=0), lineno=0, col_offset=0) +.. versionchanged:: 3.8 + + Class :class:`ast.Constant` is now used for all constants. + .. deprecated:: 3.8 - Class :class:`ast.Constant` is now used for all constants. Old classes - :class:`ast.Num`, :class:`ast.Str`, :class:`ast.Bytes`, + Old classes :class:`ast.Num`, :class:`ast.Str`, :class:`ast.Bytes`, :class:`ast.NameConstant` and :class:`ast.Ellipsis` are still available, - but they will be removed in future Python releases. + but they will be removed in future Python releases. In the meanwhile, + instantiating them will return an instance of a different class. .. _abstract-grammar: diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index 9b61e1f05542df..3ef97c98ca9077 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -1942,6 +1942,12 @@ Changes in the Python API :exc:`dbm.gnu.error` or :exc:`dbm.ndbm.error`) instead of :exc:`KeyError`. (Contributed by Xiang Zhang in :issue:`33106`.) +* Simplified AST for literals. All constants will be represented as + :class:`ast.Constant` instances. Instantiating old classes ``Num``, + ``Str``, ``Bytes``, ``NameConstant`` and ``Ellipsis`` will return + an instance of ``Constant``. + (Contributed by Serhiy Storchaka in :issue:`32892`.) + * :func:`~os.path.expanduser` on Windows now prefers the :envvar:`USERPROFILE` environment variable and does not use :envvar:`HOME`, which is not normally set for regular user accounts. From f64abd10563c25a94011f9e3335fd8a1cf47c205 Mon Sep 17 00:00:00 2001 From: Dong-hee Na Date: Mon, 17 Feb 2020 18:13:52 +0900 Subject: [PATCH 1193/2163] [3.8] bpo-39453: Fix contains method of list to hold strong references (GH-18204) --- Lib/test/test_list.py | 7 +++++++ .../2020-01-25-23-51-17.bpo-39453.xCOkYk.rst | 2 ++ Objects/listobject.c | 10 +++++++--- 3 files changed, 16 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-25-23-51-17.bpo-39453.xCOkYk.rst diff --git a/Lib/test/test_list.py b/Lib/test/test_list.py index f4dcced9c167f8..105ef650eee99e 100644 --- a/Lib/test/test_list.py +++ b/Lib/test/test_list.py @@ -216,6 +216,13 @@ def __eq__(self, other): with self.assertRaises(ValueError): lst.remove(lst) + # bpo-39453: list.__contains__ was not holding strong references + # to list elements while calling PyObject_RichCompareBool(). + lst = [X(), X()] + 3 in lst + lst = [X(), X()] + X() in lst + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-25-23-51-17.bpo-39453.xCOkYk.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-25-23-51-17.bpo-39453.xCOkYk.rst new file mode 100644 index 00000000000000..8c2e49f9474c42 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-01-25-23-51-17.bpo-39453.xCOkYk.rst @@ -0,0 +1,2 @@ +Fixed a possible crash in :meth:`list.__contains__` when a list is changed +during comparing items. Patch by Dong-hee Na. diff --git a/Objects/listobject.c b/Objects/listobject.c index 73afc44c39e36e..30444089ffaf96 100644 --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -445,12 +445,16 @@ list_length(PyListObject *a) static int list_contains(PyListObject *a, PyObject *el) { + PyObject *item; Py_ssize_t i; int cmp; - for (i = 0, cmp = 0 ; cmp == 0 && i < Py_SIZE(a); ++i) - cmp = PyObject_RichCompareBool(el, PyList_GET_ITEM(a, i), - Py_EQ); + for (i = 0, cmp = 0 ; cmp == 0 && i < Py_SIZE(a); ++i) { + item = PyList_GET_ITEM(a, i); + Py_INCREF(item); + cmp = PyObject_RichCompareBool(el, item, Py_EQ); + Py_DECREF(item); + } return cmp; } From 777ba072d61f50cc39427ae87ab63eb3d3d88542 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Mon, 17 Feb 2020 23:45:14 +0100 Subject: [PATCH 1194/2163] Python 3.8.2rc2 --- Include/patchlevel.h | 4 +- Lib/pydoc_data/topics.py | 2 +- Misc/NEWS.d/3.8.2rc2.rst | 103 ++++++++++++++++++ .../2020-01-05-13-36-08.bpo-39219.uHtKd4.rst | 2 - .../2020-01-25-23-51-17.bpo-39453.xCOkYk.rst | 2 - .../2020-02-11-23-59-07.bpo-39606.a72Sxc.rst | 2 - ...3-07-35-00.bpo-39619.inb_master_chroot.rst | 1 - .../2020-02-10-17-09-48.bpo-39600.X6NsyM.rst | 1 - .../2020-02-07-23-14-14.bpo-39595.DHwddE.rst | 1 - .../2020-02-12-10-04-39.bpo-21016.bFXPH7.rst | 4 - .../2020-02-12-12-01-26.bpo-39474.RZMEUH.rst | 2 - .../2020-02-16-07-08-54.bpo-27657.9atgcz.rst | 5 - .../2020-02-07-23-54-18.bpo-39184.v-ue-v.rst | 1 - README.rst | 2 +- 14 files changed, 107 insertions(+), 25 deletions(-) create mode 100644 Misc/NEWS.d/3.8.2rc2.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-05-13-36-08.bpo-39219.uHtKd4.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-25-23-51-17.bpo-39453.xCOkYk.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-02-11-23-59-07.bpo-39606.a72Sxc.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-02-13-07-35-00.bpo-39619.inb_master_chroot.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2020-02-10-17-09-48.bpo-39600.X6NsyM.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-02-07-23-14-14.bpo-39595.DHwddE.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-02-12-10-04-39.bpo-21016.bFXPH7.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-02-12-12-01-26.bpo-39474.RZMEUH.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-02-16-07-08-54.bpo-27657.9atgcz.rst delete mode 100644 Misc/NEWS.d/next/Security/2020-02-07-23-54-18.bpo-39184.v-ue-v.rst diff --git a/Include/patchlevel.h b/Include/patchlevel.h index 506f43c3e0adc4..fc1c0f591f8d47 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -20,10 +20,10 @@ #define PY_MINOR_VERSION 8 #define PY_MICRO_VERSION 2 #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA -#define PY_RELEASE_SERIAL 1 +#define PY_RELEASE_SERIAL 2 /* Version as a string */ -#define PY_VERSION "3.8.2rc1+" +#define PY_VERSION "3.8.2rc2" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index b9e741707cba5b..3c1b780384570c 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Mon Feb 10 19:25:27 2020 +# Autogenerated by Sphinx on Mon Feb 17 23:43:37 2020 topics = {'assert': 'The "assert" statement\n' '**********************\n' '\n' diff --git a/Misc/NEWS.d/3.8.2rc2.rst b/Misc/NEWS.d/3.8.2rc2.rst new file mode 100644 index 00000000000000..fed6ceadb28c55 --- /dev/null +++ b/Misc/NEWS.d/3.8.2rc2.rst @@ -0,0 +1,103 @@ +.. bpo: 39184 +.. date: 2020-02-07-23-54-18 +.. nonce: v-ue-v +.. release date: 2020-02-17 +.. section: Security + +Add audit events to functions in `fcntl`, `msvcrt`, `os`, `resource`, +`shutil`, `signal` and `syslog`. + +.. + +.. bpo: 39619 +.. date: 2020-02-13-07-35-00 +.. nonce: inb_master_chroot +.. section: Core and Builtins + +Enable use of :func:`os.chroot` on HP-UX systems. + +.. + +.. bpo: 39606 +.. date: 2020-02-11-23-59-07 +.. nonce: a72Sxc +.. section: Core and Builtins + +Fix regression caused by fix for bpo-39386, that prevented calling +``aclose`` on an async generator that had already been closed or exhausted. + +.. + +.. bpo: 39453 +.. date: 2020-01-25-23-51-17 +.. nonce: xCOkYk +.. section: Core and Builtins + +Fixed a possible crash in :meth:`list.__contains__` when a list is changed +during comparing items. Patch by Dong-hee Na. + +.. + +.. bpo: 39219 +.. date: 2020-01-05-13-36-08 +.. nonce: uHtKd4 +.. section: Core and Builtins + +Syntax errors raised in the tokenizer now always set correct "text" and +"offset" attributes. + +.. + +.. bpo: 27657 +.. date: 2020-02-16-07-08-54 +.. nonce: 9atgcz +.. section: Library + +The original fix for bpo-27657, "Fix urlparse() with numeric paths" +(GH-16839) included in 3.8.1, inadvertently introduced a behavior change +that broke several third-party packages relying on the original undefined +parsing behavior. The change is reverted in 3.8.2, restoring the behavior of +3.8.0 and earlier releases. + +.. + +.. bpo: 39474 +.. date: 2020-02-12-12-01-26 +.. nonce: RZMEUH +.. section: Library + +Fixed starting position of AST for expressions like ``(a)(b)``, ``(a)[b]`` +and ``(a).b``. + +.. + +.. bpo: 21016 +.. date: 2020-02-12-10-04-39 +.. nonce: bFXPH7 +.. section: Library + +The :mod:`pydoc` and :mod:`trace` modules now use the :mod:`sysconfig` +module to get the path to the Python standard library, to support uncommon +installation path like ``/usr/lib64/python3.9/`` on Fedora. Patch by Jan +MatÄ›jek. + +.. + +.. bpo: 39595 +.. date: 2020-02-07-23-14-14 +.. nonce: DHwddE +.. section: Library + +Improved performance of zipfile.Path for files with a large number of +entries. Also improved performance and fixed minor issue as published with +`importlib_metadata 1.5 +`_. + +.. + +.. bpo: 39600 +.. date: 2020-02-10-17-09-48 +.. nonce: X6NsyM +.. section: IDLE + +In the font configuration window, remove duplicated font names. diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-05-13-36-08.bpo-39219.uHtKd4.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-05-13-36-08.bpo-39219.uHtKd4.rst deleted file mode 100644 index dac8360df712ce..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-01-05-13-36-08.bpo-39219.uHtKd4.rst +++ /dev/null @@ -1,2 +0,0 @@ -Syntax errors raised in the tokenizer now always set correct "text" and -"offset" attributes. diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-25-23-51-17.bpo-39453.xCOkYk.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-25-23-51-17.bpo-39453.xCOkYk.rst deleted file mode 100644 index 8c2e49f9474c42..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-01-25-23-51-17.bpo-39453.xCOkYk.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed a possible crash in :meth:`list.__contains__` when a list is changed -during comparing items. Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-02-11-23-59-07.bpo-39606.a72Sxc.rst b/Misc/NEWS.d/next/Core and Builtins/2020-02-11-23-59-07.bpo-39606.a72Sxc.rst deleted file mode 100644 index b7cbe4e91f59c9..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-02-11-23-59-07.bpo-39606.a72Sxc.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix regression caused by fix for bpo-39386, that prevented calling -``aclose`` on an async generator that had already been closed or exhausted. diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-02-13-07-35-00.bpo-39619.inb_master_chroot.rst b/Misc/NEWS.d/next/Core and Builtins/2020-02-13-07-35-00.bpo-39619.inb_master_chroot.rst deleted file mode 100644 index 18f32f7e804bdc..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-02-13-07-35-00.bpo-39619.inb_master_chroot.rst +++ /dev/null @@ -1 +0,0 @@ -Enable use of :func:`os.chroot` on HP-UX systems. diff --git a/Misc/NEWS.d/next/IDLE/2020-02-10-17-09-48.bpo-39600.X6NsyM.rst b/Misc/NEWS.d/next/IDLE/2020-02-10-17-09-48.bpo-39600.X6NsyM.rst deleted file mode 100644 index 102aa75f5813e0..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2020-02-10-17-09-48.bpo-39600.X6NsyM.rst +++ /dev/null @@ -1 +0,0 @@ -In the font configuration window, remove duplicated font names. diff --git a/Misc/NEWS.d/next/Library/2020-02-07-23-14-14.bpo-39595.DHwddE.rst b/Misc/NEWS.d/next/Library/2020-02-07-23-14-14.bpo-39595.DHwddE.rst deleted file mode 100644 index 3a461389af7d18..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-02-07-23-14-14.bpo-39595.DHwddE.rst +++ /dev/null @@ -1 +0,0 @@ -Improved performance of zipfile.Path for files with a large number of entries. Also improved performance and fixed minor issue as published with `importlib_metadata 1.5 `_. diff --git a/Misc/NEWS.d/next/Library/2020-02-12-10-04-39.bpo-21016.bFXPH7.rst b/Misc/NEWS.d/next/Library/2020-02-12-10-04-39.bpo-21016.bFXPH7.rst deleted file mode 100644 index fb91bb38255557..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-02-12-10-04-39.bpo-21016.bFXPH7.rst +++ /dev/null @@ -1,4 +0,0 @@ -The :mod:`pydoc` and :mod:`trace` modules now use the :mod:`sysconfig` -module to get the path to the Python standard library, to support uncommon -installation path like ``/usr/lib64/python3.9/`` on Fedora. -Patch by Jan MatÄ›jek. diff --git a/Misc/NEWS.d/next/Library/2020-02-12-12-01-26.bpo-39474.RZMEUH.rst b/Misc/NEWS.d/next/Library/2020-02-12-12-01-26.bpo-39474.RZMEUH.rst deleted file mode 100644 index e990f84a9dbf25..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-02-12-12-01-26.bpo-39474.RZMEUH.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed starting position of AST for expressions like ``(a)(b)``, ``(a)[b]`` -and ``(a).b``. diff --git a/Misc/NEWS.d/next/Library/2020-02-16-07-08-54.bpo-27657.9atgcz.rst b/Misc/NEWS.d/next/Library/2020-02-16-07-08-54.bpo-27657.9atgcz.rst deleted file mode 100644 index 50bf8b28217b18..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-02-16-07-08-54.bpo-27657.9atgcz.rst +++ /dev/null @@ -1,5 +0,0 @@ -The original fix for bpo-27657, "Fix urlparse() with numeric paths" (GH-16839) -included in 3.8.1, inadvertently introduced a behavior change that broke -several third-party packages relying on the original undefined parsing -behavior. The change is reverted in 3.8.2, restoring the behavior of 3.8.0 and -earlier releases. diff --git a/Misc/NEWS.d/next/Security/2020-02-07-23-54-18.bpo-39184.v-ue-v.rst b/Misc/NEWS.d/next/Security/2020-02-07-23-54-18.bpo-39184.v-ue-v.rst deleted file mode 100644 index cf25c24d58788f..00000000000000 --- a/Misc/NEWS.d/next/Security/2020-02-07-23-54-18.bpo-39184.v-ue-v.rst +++ /dev/null @@ -1 +0,0 @@ -Add audit events to functions in `fcntl`, `msvcrt`, `os`, `resource`, `shutil`, `signal` and `syslog`. \ No newline at end of file diff --git a/README.rst b/README.rst index 0b4695ce0e509f..5858120b429290 100644 --- a/README.rst +++ b/README.rst @@ -1,4 +1,4 @@ -This is Python version 3.8.2rc1 +This is Python version 3.8.2rc2 =============================== .. image:: https://travis-ci.org/python/cpython.svg?branch=3.8 From 7fd752c1bc637aeca2bd122d07c0ebc379d0d8ca Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 17 Feb 2020 19:05:39 -0800 Subject: [PATCH 1195/2163] bpo-39663: IDLE: Add additional tests for pyparse (GH-18536) Test when find_good_parse_start should return 0. Co-authored-by: Terry Jan Reedy (cherry picked from commit ffda25f6b825f3dee493b6f0746266a4dd6989f0) Co-authored-by: Cheryl Sabella --- Lib/idlelib/NEWS.txt | 2 ++ Lib/idlelib/idle_test/test_pyparse.py | 16 ++++++++++++++-- .../2020-02-17-21-09-03.bpo-39663.wexcsH.rst | 1 + 3 files changed, 17 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/IDLE/2020-02-17-21-09-03.bpo-39663.wexcsH.rst diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 9cf563401259d1..fe1706ed5a8be0 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,8 @@ Released on 2019-12-16? ====================================== +bpo-39663: Add tests for pyparse find_good_parse_start(). + bpo-39600: Remove duplicate font names from configuration list. bpo-38792: Close a shell calltip if a :exc:`KeyboardInterrupt` diff --git a/Lib/idlelib/idle_test/test_pyparse.py b/Lib/idlelib/idle_test/test_pyparse.py index a2b13c38d80d5f..f21baf7534420a 100644 --- a/Lib/idlelib/idle_test/test_pyparse.py +++ b/Lib/idlelib/idle_test/test_pyparse.py @@ -58,6 +58,18 @@ def test_find_good_parse_start(self): p = self.parser setcode = p.set_code start = p.find_good_parse_start + def char_in_string_false(index): return False + + # First line starts with 'def' and ends with ':', then 0 is the pos. + setcode('def spam():\n') + eq(start(char_in_string_false), 0) + + # First line begins with a keyword in the list and ends + # with an open brace, then 0 is the pos. This is how + # hyperparser calls this function as the newline is not added + # in the editor, but rather on the call to setcode. + setcode('class spam( ' + ' \n') + eq(start(char_in_string_false), 0) # Split def across lines. setcode('"""This is a module docstring"""\n' @@ -79,7 +91,7 @@ def test_find_good_parse_start(self): # Make all text look like it's not in a string. This means that it # found a good start position. - eq(start(is_char_in_string=lambda index: False), 44) + eq(start(char_in_string_false), 44) # If the beginning of the def line is not in a string, then it # returns that as the index. @@ -98,7 +110,7 @@ def test_find_good_parse_start(self): ' def __init__(self, a, b=True):\n' ' pass\n' ) - eq(start(is_char_in_string=lambda index: False), 44) + eq(start(char_in_string_false), 44) eq(start(is_char_in_string=lambda index: index > 44), 44) eq(start(is_char_in_string=lambda index: index >= 44), 33) # When the def line isn't split, this returns which doesn't match the diff --git a/Misc/NEWS.d/next/IDLE/2020-02-17-21-09-03.bpo-39663.wexcsH.rst b/Misc/NEWS.d/next/IDLE/2020-02-17-21-09-03.bpo-39663.wexcsH.rst new file mode 100644 index 00000000000000..19e16329ce0a01 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2020-02-17-21-09-03.bpo-39663.wexcsH.rst @@ -0,0 +1 @@ +Add tests for pyparse find_good_parse_start(). From e412cbba52e7cf6699720d99a4b88baef92db7b2 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 18 Feb 2020 03:14:07 -0800 Subject: [PATCH 1196/2163] [3.8] bpo-39546: argparse: Honor allow_abbrev=False for specified prefix_chars (GH-18337) (GH-18543) When `allow_abbrev` was first added, disabling the abbreviation of long options broke the grouping of short flags ([bpo-26967](https://bugs.python.org/issue26967)). As a fix, b1e4d1b603 (contained in v3.8) ignores `allow_abbrev=False` for a given argument string if the string does _not_ start with "--" (i.e. it doesn't look like a long option). This fix, however, doesn't take into account that long options can start with alternative characters specified via `prefix_chars`, introducing a regression: `allow_abbrev=False` has no effect on long options that start with an alternative prefix character. The most minimal fix would be to replace the "starts with --" check with a "starts with two prefix_chars characters". But `_get_option_tuples` already distinguishes between long and short options, so let's instead piggyback off of that check by moving the `allow_abbrev` condition into `_get_option_tuples`. https://bugs.python.org/issue39546 (cherry picked from commit 8edfc47baec7ff4cb1b9db83dd35c8ffc1d498a4) Co-authored-by: Kyle Meyer https://bugs.python.org/issue39546 Automerge-Triggered-By: @encukou --- Lib/argparse.py | 56 +++++++++---------- Lib/test/test_argparse.py | 37 ++++++++++++ Misc/ACKS | 1 + .../2020-02-03-15-12-51.bpo-39546._Kj0Pn.rst | 3 + 4 files changed, 69 insertions(+), 28 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-02-03-15-12-51.bpo-39546._Kj0Pn.rst diff --git a/Lib/argparse.py b/Lib/argparse.py index fd61bc78c2c3dd..2dad5f123f3903 100644 --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -2144,24 +2144,23 @@ def _parse_optional(self, arg_string): action = self._option_string_actions[option_string] return action, option_string, explicit_arg - if self.allow_abbrev or not arg_string.startswith('--'): - # search through all possible prefixes of the option string - # and all actions in the parser for possible interpretations - option_tuples = self._get_option_tuples(arg_string) - - # if multiple actions match, the option string was ambiguous - if len(option_tuples) > 1: - options = ', '.join([option_string - for action, option_string, explicit_arg in option_tuples]) - args = {'option': arg_string, 'matches': options} - msg = _('ambiguous option: %(option)s could match %(matches)s') - self.error(msg % args) - - # if exactly one action matched, this segmentation is good, - # so return the parsed action - elif len(option_tuples) == 1: - option_tuple, = option_tuples - return option_tuple + # search through all possible prefixes of the option string + # and all actions in the parser for possible interpretations + option_tuples = self._get_option_tuples(arg_string) + + # if multiple actions match, the option string was ambiguous + if len(option_tuples) > 1: + options = ', '.join([option_string + for action, option_string, explicit_arg in option_tuples]) + args = {'option': arg_string, 'matches': options} + msg = _('ambiguous option: %(option)s could match %(matches)s') + self.error(msg % args) + + # if exactly one action matched, this segmentation is good, + # so return the parsed action + elif len(option_tuples) == 1: + option_tuple, = option_tuples + return option_tuple # if it was not found as an option, but it looks like a negative # number, it was meant to be positional @@ -2185,16 +2184,17 @@ def _get_option_tuples(self, option_string): # split at the '=' chars = self.prefix_chars if option_string[0] in chars and option_string[1] in chars: - if '=' in option_string: - option_prefix, explicit_arg = option_string.split('=', 1) - else: - option_prefix = option_string - explicit_arg = None - for option_string in self._option_string_actions: - if option_string.startswith(option_prefix): - action = self._option_string_actions[option_string] - tup = action, option_string, explicit_arg - result.append(tup) + if self.allow_abbrev: + if '=' in option_string: + option_prefix, explicit_arg = option_string.split('=', 1) + else: + option_prefix = option_string + explicit_arg = None + for option_string in self._option_string_actions: + if option_string.startswith(option_prefix): + action = self._option_string_actions[option_string] + tup = action, option_string, explicit_arg + result.append(tup) # single character options can be concatenated with their arguments # but multiple character options always have to have their argument diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index 86ec6cca51acc4..0753a4b59652dd 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -786,6 +786,23 @@ class TestOptionalsDisallowLongAbbreviation(ParserTestCase): ] +class TestOptionalsDisallowLongAbbreviationPrefixChars(ParserTestCase): + """Disallowing abbreviations works with alternative prefix characters""" + + parser_signature = Sig(prefix_chars='+', allow_abbrev=False) + argument_signatures = [ + Sig('++foo'), + Sig('++foodle', action='store_true'), + Sig('++foonly'), + ] + failures = ['+foon 3', '++foon 3', '++food', '++food ++foo 2'] + successes = [ + ('', NS(foo=None, foodle=False, foonly=None)), + ('++foo 3', NS(foo='3', foodle=False, foonly=None)), + ('++foonly 7 ++foodle ++foo 2', NS(foo='2', foodle=True, foonly='7')), + ] + + class TestDisallowLongAbbreviationAllowsShortGrouping(ParserTestCase): """Do not allow abbreviations of long options at all""" @@ -804,6 +821,26 @@ class TestDisallowLongAbbreviationAllowsShortGrouping(ParserTestCase): ('-ccrcc', NS(r='cc', c=2)), ] + +class TestDisallowLongAbbreviationAllowsShortGroupingPrefix(ParserTestCase): + """Short option grouping works with custom prefix and allow_abbrev=False""" + + parser_signature = Sig(prefix_chars='+', allow_abbrev=False) + argument_signatures = [ + Sig('+r'), + Sig('+c', action='count'), + ] + failures = ['+r', '+c +r'] + successes = [ + ('', NS(r=None, c=None)), + ('+ra', NS(r='a', c=None)), + ('+rcc', NS(r='cc', c=None)), + ('+cc', NS(r=None, c=2)), + ('+cc +ra', NS(r='a', c=2)), + ('+ccrcc', NS(r='cc', c=2)), + ] + + # ================ # Positional tests # ================ diff --git a/Misc/ACKS b/Misc/ACKS index 21ee85b4558ed1..69ed251b988db7 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1096,6 +1096,7 @@ Brian Merrell Alexis MĂ©taireau Luke Mewburn Carl Meyer +Kyle Meyer Mike Meyer Piotr Meyer Steven Miale diff --git a/Misc/NEWS.d/next/Library/2020-02-03-15-12-51.bpo-39546._Kj0Pn.rst b/Misc/NEWS.d/next/Library/2020-02-03-15-12-51.bpo-39546._Kj0Pn.rst new file mode 100644 index 00000000000000..8f035e79963e00 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-03-15-12-51.bpo-39546._Kj0Pn.rst @@ -0,0 +1,3 @@ +Fix a regression in :class:`~argparse.ArgumentParser` where +``allow_abbrev=False`` was ignored for long options that used a prefix +character other than "-". From 9d839840853bc8b3d665e46a8e06b64ac1924f47 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Tue, 18 Feb 2020 13:01:01 +0100 Subject: [PATCH 1197/2163] Post 3.8.2rc2 --- Include/patchlevel.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h index fc1c0f591f8d47..bbdb7a2aa6a0dc 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 2 /* Version as a string */ -#define PY_VERSION "3.8.2rc2" +#define PY_VERSION "3.8.2rc2+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. From 5bf58cef151249f1cca92166d1b70693348da9d8 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 18 Feb 2020 05:14:43 -0800 Subject: [PATCH 1198/2163] bpo-39432: Implement PEP-489 algorithm for non-ascii "PyInit_*" symbol names in distutils (GH-18150) (GH-18546) Make it export the correct init symbol also on Windows. https://bugs.python.org/issue39432 (cherry picked from commit 9538bc9185e934bee2bd5ae2cda2b2e92a61906d) Co-authored-by: Stefan Behnel --- Lib/distutils/command/build_ext.py | 10 +++++++++- Lib/distutils/tests/test_build_ext.py | 13 +++++++++++++ .../2020-01-23-16-08-58.bpo-39432.Cee6mi.rst | 1 + 3 files changed, 23 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2020-01-23-16-08-58.bpo-39432.Cee6mi.rst diff --git a/Lib/distutils/command/build_ext.py b/Lib/distutils/command/build_ext.py index 2d7cdf063f01db..dbcd9d16f28c59 100644 --- a/Lib/distutils/command/build_ext.py +++ b/Lib/distutils/command/build_ext.py @@ -688,7 +688,15 @@ def get_export_symbols(self, ext): provided, "PyInit_" + module_name. Only relevant on Windows, where the .pyd file (DLL) must export the module "PyInit_" function. """ - initfunc_name = "PyInit_" + ext.name.split('.')[-1] + suffix = '_' + ext.name.split('.')[-1] + try: + # Unicode module name support as defined in PEP-489 + # https://www.python.org/dev/peps/pep-0489/#export-hook-name + suffix.encode('ascii') + except UnicodeEncodeError: + suffix = 'U' + suffix.encode('punycode').replace(b'-', b'_').decode('ascii') + + initfunc_name = "PyInit" + suffix if initfunc_name not in ext.export_symbols: ext.export_symbols.append(initfunc_name) return ext.export_symbols diff --git a/Lib/distutils/tests/test_build_ext.py b/Lib/distutils/tests/test_build_ext.py index 52d36b2484f414..7e3eafa8ef231e 100644 --- a/Lib/distutils/tests/test_build_ext.py +++ b/Lib/distutils/tests/test_build_ext.py @@ -304,6 +304,19 @@ def test_get_source_files(self): cmd.ensure_finalized() self.assertEqual(cmd.get_source_files(), ['xxx']) + def test_unicode_module_names(self): + modules = [ + Extension('foo', ['aaa'], optional=False), + Extension('föö', ['uuu'], optional=False), + ] + dist = Distribution({'name': 'xx', 'ext_modules': modules}) + cmd = self.build_ext(dist) + cmd.ensure_finalized() + self.assertRegex(cmd.get_ext_filename(modules[0].name), r'foo\..*') + self.assertRegex(cmd.get_ext_filename(modules[1].name), r'föö\..*') + self.assertEqual(cmd.get_export_symbols(modules[0]), ['PyInit_foo']) + self.assertEqual(cmd.get_export_symbols(modules[1]), ['PyInitU_f_gkaa']) + def test_compiler_option(self): # cmd.compiler is an option and # should not be overridden by a compiler instance diff --git a/Misc/NEWS.d/next/Library/2020-01-23-16-08-58.bpo-39432.Cee6mi.rst b/Misc/NEWS.d/next/Library/2020-01-23-16-08-58.bpo-39432.Cee6mi.rst new file mode 100644 index 00000000000000..21c4ba86207550 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-23-16-08-58.bpo-39432.Cee6mi.rst @@ -0,0 +1 @@ +Implement PEP-489 algorithm for non-ascii "PyInit\_..." symbol names in distutils to make it export the correct init symbol also on Windows. From d77e77116fa7a9fc85be1d9f417c7e9e33fe1296 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 18 Feb 2020 13:39:19 -0800 Subject: [PATCH 1199/2163] bpo-39555: Fix distutils test to handle _d suffix on Windows debug build (GH-18357) (GH-18548) https://bugs.python.org/issue39555 Co-authored-by: Steve Dower (cherry picked from commit ab0d892288f3058856a8213333e8c3e4ed8a562b) --- Lib/distutils/tests/test_build_ext.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/distutils/tests/test_build_ext.py b/Lib/distutils/tests/test_build_ext.py index 7e3eafa8ef231e..5e47e0773a9649 100644 --- a/Lib/distutils/tests/test_build_ext.py +++ b/Lib/distutils/tests/test_build_ext.py @@ -312,8 +312,8 @@ def test_unicode_module_names(self): dist = Distribution({'name': 'xx', 'ext_modules': modules}) cmd = self.build_ext(dist) cmd.ensure_finalized() - self.assertRegex(cmd.get_ext_filename(modules[0].name), r'foo\..*') - self.assertRegex(cmd.get_ext_filename(modules[1].name), r'föö\..*') + self.assertRegex(cmd.get_ext_filename(modules[0].name), r'foo(_d)?\..*') + self.assertRegex(cmd.get_ext_filename(modules[1].name), r'föö(_d)?\..*') self.assertEqual(cmd.get_export_symbols(modules[0]), ['PyInit_foo']) self.assertEqual(cmd.get_export_symbols(modules[1]), ['PyInitU_f_gkaa']) From 44c690112d96a81fe02433de7900a4f8f9457012 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 18 Feb 2020 21:24:51 -0800 Subject: [PATCH 1200/2163] =?UTF-8?q?bpo-39572:=20Document=20=E2=80=99tota?= =?UTF-8?q?l=E2=80=99=20flag=20of=20TypedDict=20(GH-18554)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit (cherry picked from commit ab6423fe2de0ed5f8a0dc86a9c7070229326b0f0) Co-authored-by: ananthan-123 --- Doc/library/typing.rst | 16 ++++++++++++++-- Lib/typing.py | 15 ++++++++++++++- .../2020-02-18-18-37-07.bpo-39572.CCtzy1.rst | 1 + 3 files changed, 29 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Documentation/2020-02-18-18-37-07.bpo-39572.CCtzy1.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 323dac20822010..a9c7c4756dd0d1 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -996,8 +996,20 @@ The module defines the following classes, functions and decorators: Point2D = TypedDict('Point2D', x=int, y=int, label=str) Point2D = TypedDict('Point2D', {'x': int, 'y': int, 'label': str}) - See :pep:`589` for more examples and detailed rules of using ``TypedDict`` - with type checkers. + By default, all keys must be present in a TypedDict. It is possible + to override this by specifying totality. + Usage:: + + class point2D(TypedDict, total=False): + x: int + y: int + + This means that a point2D TypedDict can have any of the keys omitted.A type + checker is only expected to support a literal False or True as the value of + the total argument. True is the default, and makes all items defined in the + class body be required. + + See :pep:`589` for more examples and detailed rules of using ``TypedDict``. .. versionadded:: 3.8 diff --git a/Lib/typing.py b/Lib/typing.py index 83d310f3c0dd33..7aab8db0656708 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -13,7 +13,7 @@ * Public helper functions: get_type_hints, overload, cast, no_type_check, no_type_check_decorator. * Generic aliases for collections.abc ABCs and few additional protocols. -* Special types: NewType, NamedTuple, TypedDict (may be added soon). +* Special types: NewType, NamedTuple, TypedDict. * Wrapper submodules for re and io related types. """ @@ -1779,6 +1779,19 @@ class Point2D(TypedDict): Point2D = TypedDict('Point2D', x=int, y=int, label=str) Point2D = TypedDict('Point2D', {'x': int, 'y': int, 'label': str}) + By default, all keys must be present in a TypedDict. It is possible + to override this by specifying totality. + Usage:: + + class point2D(TypedDict, total=False): + x: int + y: int + + This means that a point2D TypedDict can have any of the keys omitted.A type + checker is only expected to support a literal False or True as the value of + the total argument. True is the default, and makes all items defined in the + class body be required. + The class syntax is only supported in Python 3.6+, while two other syntax forms work for Python 2.7 and 3.2+ """ diff --git a/Misc/NEWS.d/next/Documentation/2020-02-18-18-37-07.bpo-39572.CCtzy1.rst b/Misc/NEWS.d/next/Documentation/2020-02-18-18-37-07.bpo-39572.CCtzy1.rst new file mode 100644 index 00000000000000..d47bb455e71d1e --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-02-18-18-37-07.bpo-39572.CCtzy1.rst @@ -0,0 +1 @@ +Updated documentation of ``total`` flag of TypeDict. From 471d2b9ce071947ea1d6b61618518cbeb9227348 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 20 Feb 2020 10:31:11 -0800 Subject: [PATCH 1201/2163] Update runall.bat to the latest Windows build system. (GH-18571) (#18573) (cherry picked from commit 9b833e00e447a3b8b6966686bff701f549c66263) Co-authored-by: Stefan Krah Co-authored-by: Stefan Krah --- Modules/_decimal/tests/runall.bat | 100 ++++++++++++++++++------------ 1 file changed, 59 insertions(+), 41 deletions(-) diff --git a/Modules/_decimal/tests/runall.bat b/Modules/_decimal/tests/runall.bat index 5bc872a63f83aa..a9d6c6f19c8bb0 100755 --- a/Modules/_decimal/tests/runall.bat +++ b/Modules/_decimal/tests/runall.bat @@ -7,105 +7,123 @@ cd ..\..\..\ echo. echo # ====================================================================== -echo # Building Python +echo # Building Python (Debug^|x64) echo # ====================================================================== echo. -call "%VS100COMNTOOLS%\..\..\VC\vcvarsall.bat" x64 -msbuild /noconsolelogger /target:clean PCbuild\pcbuild.sln /p:Configuration=Release /p:PlatformTarget=x64 -msbuild /noconsolelogger /target:clean PCbuild\pcbuild.sln /p:Configuration=Debug /p:PlatformTarget=x64 -msbuild /noconsolelogger PCbuild\pcbuild.sln /p:Configuration=Release /p:Platform=x64 -msbuild /noconsolelogger PCbuild\pcbuild.sln /p:Configuration=Debug /p:Platform=x64 - -call "%VS100COMNTOOLS%\..\..\VC\vcvarsall.bat" x86 -msbuild /noconsolelogger PCbuild\pcbuild.sln /p:Configuration=Release /p:Platform=Win32 -msbuild /noconsolelogger PCbuild\pcbuild.sln /p:Configuration=Debug /p:Platform=Win32 -echo. -echo. +call .\Tools\buildbot\clean.bat +call .\Tools\buildbot\build.bat -c Debug -p x64 echo. echo # ====================================================================== -echo # test_decimal: platform=x64 +echo # platform=Debug^|x64 echo # ====================================================================== echo. -cd PCbuild\amd64 - echo # ==================== refleak tests ======================= echo. -python_d.exe -m test -uall -R 2:2 test_decimal +call python.bat -m test -uall -R 3:3 test_decimal echo. echo. echo # ==================== regular tests ======================= echo. -python.exe -m test -uall test_decimal +call python.bat -m test -uall test_decimal +echo. +echo. + +echo # ==================== deccheck ======================= +echo. +call python.bat .\Modules\_decimal\tests\deccheck.py echo. echo. -cd .. echo. echo # ====================================================================== -echo # test_decimal: platform=x86 +echo # Building Python (Release^|x64) echo # ====================================================================== echo. -echo # ==================== refleak tests ======================= -echo. -python_d.exe -m test -uall -R 2:2 test_decimal +call .\Tools\buildbot\clean.bat +call .\Tools\buildbot\build.bat -c Release -p x64 + echo. +echo # ====================================================================== +echo # platform=Release^|x64 +echo # ====================================================================== echo. echo # ==================== regular tests ======================= echo. -python.exe -m test -uall test_decimal +call python.bat -m test -uall test_decimal +echo. +echo. + +echo # ==================== deccheck ======================= +echo. +call python.bat .\Modules\_decimal\tests\deccheck.py +echo. +echo. + + echo. +echo # ====================================================================== +echo # Building Python (Debug^|Win32) +echo # ====================================================================== echo. -cd amd64 +call .\Tools\buildbot\clean.bat +call Tools\buildbot\build.bat -c Debug -p Win32 echo. echo # ====================================================================== -echo # deccheck: platform=x64 +echo # platform=Debug^|Win32 echo # ====================================================================== echo. -echo # ==================== debug build ======================= +echo # ==================== refleak tests ======================= +echo. +call python.bat -m test -uall -R 3:3 test_decimal +echo. +echo. + +echo # ==================== regular tests ======================= echo. -python_d.exe ..\..\Modules\_decimal\tests\deccheck.py +call python.bat -m test -uall test_decimal echo. echo. -echo # =================== release build ====================== +echo # ==================== deccheck ======================= echo. -python.exe ..\..\Modules\_decimal\tests\deccheck.py +call python.bat .\Modules\_decimal\tests\deccheck.py echo. echo. -cd .. echo. echo # ====================================================================== -echo # deccheck: platform=x86 +echo # Building Python (Release^|Win32) echo # ====================================================================== echo. + +call .\Tools\buildbot\clean.bat +call .\Tools\buildbot\build.bat -c Release -p Win32 + +echo. +echo # ====================================================================== +echo # platform=Release^|Win32 +echo # ====================================================================== echo. -echo # ==================== debug build ======================= +echo # ==================== regular tests ======================= echo. -python_d.exe ..\Modules\_decimal\tests\deccheck.py +call python.bat -m test -uall test_decimal echo. echo. -echo # =================== release build ====================== +echo # ==================== deccheck ======================= echo. -python.exe ..\Modules\_decimal\tests\deccheck.py +call python.bat .\Modules\_decimal\tests\deccheck.py echo. echo. - - -cd ..\Modules\_decimal\tests - - - From 0721f7afb0a9c0500d2a912a55f2a290ec94ff53 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 20 Feb 2020 10:31:39 -0800 Subject: [PATCH 1202/2163] Use the new recommended number of repetitions in the refleak tests. (GH-18569) (#18574) (cherry picked from commit 1246d892038a693304549f8574e6c2784b91589a) Co-authored-by: Stefan Krah Co-authored-by: Stefan Krah --- Modules/_decimal/tests/runall-memorydebugger.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Modules/_decimal/tests/runall-memorydebugger.sh b/Modules/_decimal/tests/runall-memorydebugger.sh index 77c0c9cd82c13f..18328841afcc1a 100755 --- a/Modules/_decimal/tests/runall-memorydebugger.sh +++ b/Modules/_decimal/tests/runall-memorydebugger.sh @@ -78,7 +78,7 @@ for config in $CONFIGS; do $GMAKE | grep _decimal printf "\n\n# ======================== refleak tests ===========================\n\n" - ./python -m test -uall -R 2:2 test_decimal + ./python -m test -uall -R 3:3 test_decimal ############ regular tests ########### From 9c87fbe54e1c797e3c690c6426bdf5e79c457cf1 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 20 Feb 2020 10:38:19 -0800 Subject: [PATCH 1203/2163] Valgrind no longer supports --db-attach=yes. (GH-18568) (#18576) (cherry picked from commit c0cb8beb389da3ba67ad31b1ecc95e100b6292ab) Co-authored-by: Stefan Krah Co-authored-by: Stefan Krah --- Modules/_decimal/tests/runall-memorydebugger.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Modules/_decimal/tests/runall-memorydebugger.sh b/Modules/_decimal/tests/runall-memorydebugger.sh index 18328841afcc1a..29b7723dd50c05 100755 --- a/Modules/_decimal/tests/runall-memorydebugger.sh +++ b/Modules/_decimal/tests/runall-memorydebugger.sh @@ -18,7 +18,7 @@ CONFIGS_64="x64 uint128 ansi64 universal" CONFIGS_32="ppro ansi32 ansi-legacy universal" VALGRIND="valgrind --tool=memcheck --leak-resolution=high \ - --db-attach=yes --suppressions=Misc/valgrind-python.supp" + --suppressions=Misc/valgrind-python.supp" # Get args case $@ in From d0a464e31ac67685ef8ad35ecd993f17dfd6ab35 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 20 Feb 2020 14:44:47 -0800 Subject: [PATCH 1204/2163] bpo-39184: Fix incorrect return value (GH-18580) https://bugs.python.org/issue39184 Automerge-Triggered-By: @zooba (cherry picked from commit 6c444d0dab8f06cf304263b34beb299101cef3de) Co-authored-by: Steve Dower --- PC/msvcrtmodule.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PC/msvcrtmodule.c b/PC/msvcrtmodule.c index 5c06ec2621ea6c..faceb03fba39d1 100644 --- a/PC/msvcrtmodule.c +++ b/PC/msvcrtmodule.c @@ -180,7 +180,7 @@ msvcrt_open_osfhandle_impl(PyObject *module, void *handle, int flags) int fd; if (PySys_Audit("msvcrt.open_osfhandle", "Ki", handle, flags) < 0) { - return NULL; + return -1; } _Py_BEGIN_SUPPRESS_IPH From b6271025c640c228505dc9f194362a0c2ab81c61 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 20 Feb 2020 17:16:42 -0800 Subject: [PATCH 1205/2163] bpo-39576: Prevent memory error for overly optimistic precisions (GH-18581) (#18584) (cherry picked from commit 90930e65455f60216f09d175586139242dbba260) Authored-by: Stefan Krah --- Lib/test/test_decimal.py | 35 +++++++ Modules/_decimal/libmpdec/mpdecimal.c | 77 +++++++++++++- Modules/_decimal/tests/deccheck.py | 139 +++++++++++++++++++++++++- 3 files changed, 245 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py index 1f37b5372a3e7e..0e9cd3095c85ed 100644 --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -5476,6 +5476,41 @@ def __abs__(self): self.assertEqual(Decimal.from_float(cls(101.1)), Decimal.from_float(101.1)) + def test_maxcontext_exact_arith(self): + + # Make sure that exact operations do not raise MemoryError due + # to huge intermediate values when the context precision is very + # large. + + # The following functions fill the available precision and are + # therefore not suitable for large precisions (by design of the + # specification). + MaxContextSkip = ['logical_invert', 'next_minus', 'next_plus', + 'logical_and', 'logical_or', 'logical_xor', + 'next_toward', 'rotate', 'shift'] + + Decimal = C.Decimal + Context = C.Context + localcontext = C.localcontext + + # Here only some functions that are likely candidates for triggering a + # MemoryError are tested. deccheck.py has an exhaustive test. + maxcontext = Context(prec=C.MAX_PREC, Emin=C.MIN_EMIN, Emax=C.MAX_EMAX) + with localcontext(maxcontext): + self.assertEqual(Decimal(0).exp(), 1) + self.assertEqual(Decimal(1).ln(), 0) + self.assertEqual(Decimal(1).log10(), 0) + self.assertEqual(Decimal(10**2).log10(), 2) + self.assertEqual(Decimal(10**223).log10(), 223) + self.assertEqual(Decimal(10**19).logb(), 19) + self.assertEqual(Decimal(4).sqrt(), 2) + self.assertEqual(Decimal("40E9").sqrt(), Decimal('2.0E+5')) + self.assertEqual(divmod(Decimal(10), 3), (3, 1)) + self.assertEqual(Decimal(10) // 3, 3) + self.assertEqual(Decimal(4) / 2, 2) + self.assertEqual(Decimal(400) ** -1, Decimal('0.0025')) + + @requires_docstrings @unittest.skipUnless(C, "test requires C version") class SignatureTest(unittest.TestCase): diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c index bfa8bb343e60c1..0986edb576a10f 100644 --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -3781,6 +3781,43 @@ mpd_qdiv(mpd_t *q, const mpd_t *a, const mpd_t *b, const mpd_context_t *ctx, uint32_t *status) { _mpd_qdiv(SET_IDEAL_EXP, q, a, b, ctx, status); + + if (*status & MPD_Malloc_error) { + /* Inexact quotients (the usual case) fill the entire context precision, + * which can lead to malloc() failures for very high precisions. Retry + * the operation with a lower precision in case the result is exact. + * + * We need an upper bound for the number of digits of a_coeff / b_coeff + * when the result is exact. If a_coeff' * 1 / b_coeff' is in lowest + * terms, then maxdigits(a_coeff') + maxdigits(1 / b_coeff') is a suitable + * bound. + * + * 1 / b_coeff' is exact iff b_coeff' exclusively has prime factors 2 or 5. + * The largest amount of digits is generated if b_coeff' is a power of 2 or + * a power of 5 and is less than or equal to log5(b_coeff') <= log2(b_coeff'). + * + * We arrive at a total upper bound: + * + * maxdigits(a_coeff') + maxdigits(1 / b_coeff') <= + * a->digits + log2(b_coeff) = + * a->digits + log10(b_coeff) / log10(2) <= + * a->digits + b->digits * 4; + */ + uint32_t workstatus = 0; + mpd_context_t workctx = *ctx; + workctx.prec = a->digits + b->digits * 4; + if (workctx.prec >= ctx->prec) { + return; /* No point in retrying, keep the original error. */ + } + + _mpd_qdiv(SET_IDEAL_EXP, q, a, b, &workctx, &workstatus); + if (workstatus == 0) { /* The result is exact, unrounded, normal etc. */ + *status = 0; + return; + } + + mpd_seterror(q, *status, status); + } } /* Internal function. */ @@ -7702,9 +7739,9 @@ mpd_qinvroot(mpd_t *result, const mpd_t *a, const mpd_context_t *ctx, /* END LIBMPDEC_ONLY */ /* Algorithm from decimal.py */ -void -mpd_qsqrt(mpd_t *result, const mpd_t *a, const mpd_context_t *ctx, - uint32_t *status) +static void +_mpd_qsqrt(mpd_t *result, const mpd_t *a, const mpd_context_t *ctx, + uint32_t *status) { mpd_context_t maxcontext; MPD_NEW_STATIC(c,0,0,0,0); @@ -7836,6 +7873,40 @@ mpd_qsqrt(mpd_t *result, const mpd_t *a, const mpd_context_t *ctx, goto out; } +void +mpd_qsqrt(mpd_t *result, const mpd_t *a, const mpd_context_t *ctx, + uint32_t *status) +{ + _mpd_qsqrt(result, a, ctx, status); + + if (*status & (MPD_Malloc_error|MPD_Division_impossible)) { + /* The above conditions can occur at very high context precisions + * if intermediate values get too large. Retry the operation with + * a lower context precision in case the result is exact. + * + * If the result is exact, an upper bound for the number of digits + * is the number of digits in the input. + * + * NOTE: sqrt(40e9) = 2.0e+5 /\ digits(40e9) = digits(2.0e+5) = 2 + */ + uint32_t workstatus = 0; + mpd_context_t workctx = *ctx; + workctx.prec = a->digits; + + if (workctx.prec >= ctx->prec) { + return; /* No point in repeating this, keep the original error. */ + } + + _mpd_qsqrt(result, a, &workctx, &workstatus); + if (workstatus == 0) { + *status = 0; + return; + } + + mpd_seterror(result, *status, status); + } +} + /******************************************************************************/ /* Base conversions */ diff --git a/Modules/_decimal/tests/deccheck.py b/Modules/_decimal/tests/deccheck.py index f907531e1ffa58..5cd5db5711426c 100644 --- a/Modules/_decimal/tests/deccheck.py +++ b/Modules/_decimal/tests/deccheck.py @@ -125,6 +125,12 @@ 'special': ('context.__reduce_ex__', 'context.create_decimal_from_float') } +# Functions that set no context flags but whose result can differ depending +# on prec, Emin and Emax. +MaxContextSkip = ['is_normal', 'is_subnormal', 'logical_invert', 'next_minus', + 'next_plus', 'number_class', 'logical_and', 'logical_or', + 'logical_xor', 'next_toward', 'rotate', 'shift'] + # Functions that require a restricted exponent range for reasonable runtimes. UnaryRestricted = [ '__ceil__', '__floor__', '__int__', '__trunc__', @@ -344,6 +350,20 @@ def __init__(self, funcname, operands): self.pex = RestrictedList() # Python exceptions for P.Decimal self.presults = RestrictedList() # P.Decimal results + # If the above results are exact, unrounded and not clamped, repeat + # the operation with a maxcontext to ensure that huge intermediate + # values do not cause a MemoryError. + self.with_maxcontext = False + self.maxcontext = context.c.copy() + self.maxcontext.prec = C.MAX_PREC + self.maxcontext.Emax = C.MAX_EMAX + self.maxcontext.Emin = C.MIN_EMIN + self.maxcontext.clear_flags() + + self.maxop = RestrictedList() # converted C.Decimal operands + self.maxex = RestrictedList() # Python exceptions for C.Decimal + self.maxresults = RestrictedList() # C.Decimal results + # ====================================================================== # SkipHandler: skip known discrepancies @@ -545,13 +565,17 @@ def function_as_string(t): if t.contextfunc: cargs = t.cop pargs = t.pop + maxargs = t.maxop cfunc = "c_func: %s(" % t.funcname pfunc = "p_func: %s(" % t.funcname + maxfunc = "max_func: %s(" % t.funcname else: cself, cargs = t.cop[0], t.cop[1:] pself, pargs = t.pop[0], t.pop[1:] + maxself, maxargs = t.maxop[0], t.maxop[1:] cfunc = "c_func: %s.%s(" % (repr(cself), t.funcname) pfunc = "p_func: %s.%s(" % (repr(pself), t.funcname) + maxfunc = "max_func: %s.%s(" % (repr(maxself), t.funcname) err = cfunc for arg in cargs: @@ -565,6 +589,14 @@ def function_as_string(t): err = err.rstrip(", ") err += ")" + if t.with_maxcontext: + err += "\n" + err += maxfunc + for arg in maxargs: + err += "%s, " % repr(arg) + err = err.rstrip(", ") + err += ")" + return err def raise_error(t): @@ -577,9 +609,24 @@ def raise_error(t): err = "Error in %s:\n\n" % t.funcname err += "input operands: %s\n\n" % (t.op,) err += function_as_string(t) - err += "\n\nc_result: %s\np_result: %s\n\n" % (t.cresults, t.presults) - err += "c_exceptions: %s\np_exceptions: %s\n\n" % (t.cex, t.pex) - err += "%s\n\n" % str(t.context) + + err += "\n\nc_result: %s\np_result: %s\n" % (t.cresults, t.presults) + if t.with_maxcontext: + err += "max_result: %s\n\n" % (t.maxresults) + else: + err += "\n" + + err += "c_exceptions: %s\np_exceptions: %s\n" % (t.cex, t.pex) + if t.with_maxcontext: + err += "max_exceptions: %s\n\n" % t.maxex + else: + err += "\n" + + err += "%s\n" % str(t.context) + if t.with_maxcontext: + err += "%s\n" % str(t.maxcontext) + else: + err += "\n" raise VerifyError(err) @@ -603,6 +650,13 @@ def raise_error(t): # are printed to stdout. # ====================================================================== +def all_nan(a): + if isinstance(a, C.Decimal): + return a.is_nan() + elif isinstance(a, tuple): + return all(all_nan(v) for v in a) + return False + def convert(t, convstr=True): """ t is the testset. At this stage the testset contains a tuple of operands t.op of various types. For decimal methods the first @@ -617,10 +671,12 @@ def convert(t, convstr=True): for i, op in enumerate(t.op): context.clear_status() + t.maxcontext.clear_flags() if op in RoundModes: t.cop.append(op) t.pop.append(op) + t.maxop.append(op) elif not t.contextfunc and i == 0 or \ convstr and isinstance(op, str): @@ -638,11 +694,25 @@ def convert(t, convstr=True): p = None pex = e.__class__ + try: + C.setcontext(t.maxcontext) + maxop = C.Decimal(op) + maxex = None + except (TypeError, ValueError, OverflowError) as e: + maxop = None + maxex = e.__class__ + finally: + C.setcontext(context.c) + t.cop.append(c) t.cex.append(cex) + t.pop.append(p) t.pex.append(pex) + t.maxop.append(maxop) + t.maxex.append(maxex) + if cex is pex: if str(c) != str(p) or not context.assert_eq_status(): raise_error(t) @@ -652,14 +722,21 @@ def convert(t, convstr=True): else: raise_error(t) + # The exceptions in the maxcontext operation can legitimately + # differ, only test that maxex implies cex: + if maxex is not None and cex is not maxex: + raise_error(t) + elif isinstance(op, Context): t.context = op t.cop.append(op.c) t.pop.append(op.p) + t.maxop.append(t.maxcontext) else: t.cop.append(op) t.pop.append(op) + t.maxop.append(op) return 1 @@ -673,6 +750,7 @@ def callfuncs(t): t.rc and t.rp are the results of the operation. """ context.clear_status() + t.maxcontext.clear_flags() try: if t.contextfunc: @@ -700,6 +778,35 @@ def callfuncs(t): t.rp = None t.pex.append(e.__class__) + # If the above results are exact, unrounded, normal etc., repeat the + # operation with a maxcontext to ensure that huge intermediate values + # do not cause a MemoryError. + if (t.funcname not in MaxContextSkip and + not context.c.flags[C.InvalidOperation] and + not context.c.flags[C.Inexact] and + not context.c.flags[C.Rounded] and + not context.c.flags[C.Subnormal] and + not context.c.flags[C.Clamped] and + not context.clamp and # results are padded to context.prec if context.clamp==1. + not any(isinstance(v, C.Context) for v in t.cop)): # another context is used. + t.with_maxcontext = True + try: + if t.contextfunc: + maxargs = t.maxop + t.rmax = getattr(t.maxcontext, t.funcname)(*maxargs) + else: + maxself = t.maxop[0] + maxargs = t.maxop[1:] + try: + C.setcontext(t.maxcontext) + t.rmax = getattr(maxself, t.funcname)(*maxargs) + finally: + C.setcontext(context.c) + t.maxex.append(None) + except (TypeError, ValueError, OverflowError, MemoryError) as e: + t.rmax = None + t.maxex.append(e.__class__) + def verify(t, stat): """ t is the testset. At this stage the testset contains the following tuples: @@ -714,6 +821,9 @@ def verify(t, stat): """ t.cresults.append(str(t.rc)) t.presults.append(str(t.rp)) + if t.with_maxcontext: + t.maxresults.append(str(t.rmax)) + if isinstance(t.rc, C.Decimal) and isinstance(t.rp, P.Decimal): # General case: both results are Decimals. t.cresults.append(t.rc.to_eng_string()) @@ -725,6 +835,12 @@ def verify(t, stat): t.presults.append(str(t.rp.imag)) t.presults.append(str(t.rp.real)) + if t.with_maxcontext and isinstance(t.rmax, C.Decimal): + t.maxresults.append(t.rmax.to_eng_string()) + t.maxresults.append(t.rmax.as_tuple()) + t.maxresults.append(str(t.rmax.imag)) + t.maxresults.append(str(t.rmax.real)) + nc = t.rc.number_class().lstrip('+-s') stat[nc] += 1 else: @@ -732,6 +848,9 @@ def verify(t, stat): if not isinstance(t.rc, tuple) and not isinstance(t.rp, tuple): if t.rc != t.rp: raise_error(t) + if t.with_maxcontext and not isinstance(t.rmax, tuple): + if t.rmax != t.rc: + raise_error(t) stat[type(t.rc).__name__] += 1 # The return value lists must be equal. @@ -744,6 +863,20 @@ def verify(t, stat): if not t.context.assert_eq_status(): raise_error(t) + if t.with_maxcontext: + # NaN payloads etc. depend on precision and clamp. + if all_nan(t.rc) and all_nan(t.rmax): + return + # The return value lists must be equal. + if t.maxresults != t.cresults: + raise_error(t) + # The Python exception lists (TypeError, etc.) must be equal. + if t.maxex != t.cex: + raise_error(t) + # The context flags must be equal. + if t.maxcontext.flags != t.context.c.flags: + raise_error(t) + # ====================================================================== # Main test loops From a2075121217e809c9a5511f6ca225c12f340de0c Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 20 Feb 2020 22:06:18 -0800 Subject: [PATCH 1206/2163] bpo-38657: Clarify numeric padding behaviour in string formatting (GH-17036) Make the definition of the width more explicit that it includes any extra signs added by other options. https://bugs.python.org/issue38657 Automerge-Triggered-By: @Mariatta (cherry picked from commit 424e5686d82235e08f8108b8bbe034bc91421689) Co-authored-by: Pete Wicken <2273100+JamoBox@users.noreply.github.com> --- Doc/library/string.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Doc/library/string.rst b/Doc/library/string.rst index e2983db1ac8321..89c169a512b520 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -415,8 +415,9 @@ error. .. versionchanged:: 3.6 Added the ``'_'`` option (see also :pep:`515`). -*width* is a decimal integer defining the minimum field width. If not -specified, then the field width will be determined by the content. +*width* is a decimal integer defining the minimum total field width, +including any prefixes, separators, and other formatting characters. +If not specified, then the field width will be determined by the content. When no explicit alignment is given, preceding the *width* field by a zero (``'0'``) character enables From 9ce361d3bb15cf49b82fa03e3e593d7cbd8ee1ff Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 21 Feb 2020 02:54:00 -0800 Subject: [PATCH 1207/2163] fix(doc): set correct RST syntax for c:function (GH-18589) The current content is not rendered since the syntax is not correct. (cherry picked from commit d4d17fd2cf69e7c8f4cd03fbf2d575370945b952) Co-authored-by: Julien Danjou --- Doc/c-api/memory.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index ba7bd3b9a53878..8a8542f0479ec5 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -533,7 +533,7 @@ tracemalloc C API .. versionadded:: 3.7 -.. c:function: int PyTraceMalloc_Track(unsigned int domain, uintptr_t ptr, size_t size) +.. c:function:: int PyTraceMalloc_Track(unsigned int domain, uintptr_t ptr, size_t size) Track an allocated memory block in the :mod:`tracemalloc` module. @@ -542,7 +542,7 @@ tracemalloc C API If memory block is already tracked, update the existing trace. -.. c:function: int PyTraceMalloc_Untrack(unsigned int domain, uintptr_t ptr) +.. c:function:: int PyTraceMalloc_Untrack(unsigned int domain, uintptr_t ptr) Untrack an allocated memory block in the :mod:`tracemalloc` module. Do nothing if the block was not tracked. From d6965ff026f35498e554bc964ef2be8f4d80eb7f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 21 Feb 2020 12:36:42 -0800 Subject: [PATCH 1208/2163] bpo-39576: docs: set context for decimal arbitrary precision arithmetic (GH-18594) (#18597) (cherry picked from commit a025d4ca99fb4c652465368e0b4eb03cf4b316b9) Authored-by: Stefan Krah --- Doc/library/decimal.rst | 66 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 58 insertions(+), 8 deletions(-) diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index bcae55eb821784..2a51429bdff5c9 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -2121,17 +2121,67 @@ Q. Is the CPython implementation fast for large numbers? A. Yes. In the CPython and PyPy3 implementations, the C/CFFI versions of the decimal module integrate the high speed `libmpdec `_ library for -arbitrary precision correctly-rounded decimal floating point arithmetic. +arbitrary precision correctly-rounded decimal floating point arithmetic [#]_. ``libmpdec`` uses `Karatsuba multiplication `_ for medium-sized numbers and the `Number Theoretic Transform `_ -for very large numbers. However, to realize this performance gain, the -context needs to be set for unrounded calculations. +for very large numbers. - >>> c = getcontext() - >>> c.prec = MAX_PREC - >>> c.Emax = MAX_EMAX - >>> c.Emin = MIN_EMIN +The context must be adapted for exact arbitrary precision arithmetic. :attr:`Emin` +and :attr:`Emax` should always be set to the maximum values, :attr:`clamp` +should always be 0 (the default). Setting :attr:`prec` requires some care. -.. versionadded:: 3.3 \ No newline at end of file +The easiest approach for trying out bignum arithmetic is to use the maximum +value for :attr:`prec` as well [#]_:: + + >>> setcontext(Context(prec=MAX_PREC, Emax=MAX_EMAX, Emin=MIN_EMIN)) + >>> x = Decimal(2) ** 256 + >>> x / 128 + Decimal('904625697166532776746648320380374280103671755200316906558262375061821325312') + + +For inexact results, :attr:`MAX_PREC` is far too large on 64-bit platforms and +the available memory will be insufficient:: + + >>> Decimal(1) / 3 + Traceback (most recent call last): + File "", line 1, in + MemoryError + +On systems with overallocation (e.g. Linux), a more sophisticated approach is to +adjust :attr:`prec` to the amount of available RAM. Suppose that you have 8GB of +RAM and expect 10 simultaneous operands using a maximum of 500MB each:: + + >>> import sys + >>> + >>> # Maximum number of digits for a single operand using 500MB in 8 byte words + >>> # with 19 (9 for the 32-bit version) digits per word: + >>> maxdigits = 19 * ((500 * 1024**2) // 8) + >>> + >>> # Check that this works: + >>> c = Context(prec=maxdigits, Emax=MAX_EMAX, Emin=MIN_EMIN) + >>> c.traps[Inexact] = True + >>> setcontext(c) + >>> + >>> # Fill the available precision with nines: + >>> x = Decimal(0).logical_invert() * 9 + >>> sys.getsizeof(x) + 524288112 + >>> x + 2 + Traceback (most recent call last): + File "", line 1, in + decimal.Inexact: [] + +In general (and especially on systems without overallocation), it is recommended +to estimate even tighter bounds and set the :attr:`Inexact` trap if all calculations +are expected to be exact. + + +.. [#] + .. versionadded:: 3.3 + +.. [#] + .. versionchanged:: 3.9 + This approach now works for all exact results except for non-integer powers. + Also backported to 3.7 and 3.8. From 0c1827e70c1c05ce1982a34380cea7d391904293 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 22 Feb 2020 05:34:06 -0800 Subject: [PATCH 1209/2163] bpo-39382: Avoid dangling object use in abstract_issubclass() (GH-18530) Hold reference of __bases__ tuple until tuple item is done with, because by dropping the reference the item may be destroyed. (cherry picked from commit 1c56f8ffad44478b4214a2bf8eb7cf51c28a347a) Co-authored-by: Yonatan Goldschmidt --- Lib/test/test_isinstance.py | 21 +++++++++++++++++++ Misc/ACKS | 1 + .../2020-02-18-01-40-13.bpo-39382.OLSJu9.rst | 3 +++ Objects/abstract.c | 12 ++++++++--- 4 files changed, 34 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-02-18-01-40-13.bpo-39382.OLSJu9.rst diff --git a/Lib/test/test_isinstance.py b/Lib/test/test_isinstance.py index 65751ab9168550..53639e984e48a7 100644 --- a/Lib/test/test_isinstance.py +++ b/Lib/test/test_isinstance.py @@ -251,6 +251,27 @@ def test_isinstance_recursion_limit(self): # blown self.assertRaises(RecursionError, blowstack, isinstance, '', str) + def test_issubclass_refcount_handling(self): + # bpo-39382: abstract_issubclass() didn't hold item reference while + # peeking in the bases tuple, in the single inheritance case. + class A: + @property + def __bases__(self): + return (int, ) + + class B: + def __init__(self): + # setting this here increases the chances of exhibiting the bug, + # probably due to memory layout changes. + self.x = 1 + + @property + def __bases__(self): + return (A(), ) + + self.assertEqual(True, issubclass(B(), int)) + + def blowstack(fxn, arg, compare_to): # Make sure that calling isinstance with a deeply nested tuple for its # argument will raise RecursionError eventually. diff --git a/Misc/ACKS b/Misc/ACKS index 69ed251b988db7..8fd4df5bf3f199 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -584,6 +584,7 @@ Karan Goel Jeroen Van Goey Christoph Gohlke Tim Golden +Yonatan Goldschmidt Mark Gollahon Guilherme Gonçalves Tiago Gonçalves diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-02-18-01-40-13.bpo-39382.OLSJu9.rst b/Misc/NEWS.d/next/Core and Builtins/2020-02-18-01-40-13.bpo-39382.OLSJu9.rst new file mode 100644 index 00000000000000..605f4c8e5dfd1f --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-02-18-01-40-13.bpo-39382.OLSJu9.rst @@ -0,0 +1,3 @@ +Fix a use-after-free in the single inheritance path of ``issubclass()``, when +the ``__bases__`` of an object has a single reference, and so does its first item. +Patch by Yonatan Goldschmidt. diff --git a/Objects/abstract.c b/Objects/abstract.c index 77d09143aa0764..bc1ebd930c0166 100644 --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -2336,9 +2336,16 @@ abstract_issubclass(PyObject *derived, PyObject *cls) int r = 0; while (1) { - if (derived == cls) + if (derived == cls) { + Py_XDECREF(bases); /* See below comment */ return 1; - bases = abstract_get_bases(derived); + } + /* Use XSETREF to drop bases reference *after* finishing with + derived; bases might be the only reference to it. + XSETREF is used instead of SETREF, because bases is NULL on the + first iteration of the loop. + */ + Py_XSETREF(bases, abstract_get_bases(derived)); if (bases == NULL) { if (PyErr_Occurred()) return -1; @@ -2352,7 +2359,6 @@ abstract_issubclass(PyObject *derived, PyObject *cls) /* Avoid recursivity in the single inheritance case */ if (n == 1) { derived = PyTuple_GET_ITEM(bases, 0); - Py_DECREF(bases); continue; } for (i = 0; i < n; i++) { From 91ba44623e5cb8f4a7b0d3fc1fab8ff8d8a849c2 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 22 Feb 2020 13:16:17 -0800 Subject: [PATCH 1210/2163] bpo-17422: Language reference should specify restrictions on class namespace (GH-18559) The language reference now specifies restrictions on class namespaces. Adapted from a patch by Ethan Furman. (cherry picked from commit fbe2e0bb8a7ee75d0f9d57682436dac7d69e202e) Co-authored-by: ananthan-123 --- Doc/reference/datamodel.rst | 6 ++++-- .../Documentation/2020-02-19-11-13-47.bpo-17422.g7_9zz.rst | 1 + 2 files changed, 5 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Documentation/2020-02-19-11-13-47.bpo-17422.g7_9zz.rst diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 7b3bf0d2f5f366..06f4992cf544a9 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1946,10 +1946,12 @@ Once the appropriate metaclass has been identified, then the class namespace is prepared. If the metaclass has a ``__prepare__`` attribute, it is called as ``namespace = metaclass.__prepare__(name, bases, **kwds)`` (where the additional keyword arguments, if any, come from the class definition). The -``__prepare__`` method should be implemented as a :func:`classmethod`. +``__prepare__`` method should be implemented as a :func:`classmethod`. The +namespace returned by ``__prepare__`` is passed in to ``__new__``, but when +the final class object is created the namespace is copied into a new ``dict``. If the metaclass has no ``__prepare__`` attribute, then the class namespace -is initialised as an empty ordered mapping. +is initialised as an empty :func:`dict`. .. seealso:: diff --git a/Misc/NEWS.d/next/Documentation/2020-02-19-11-13-47.bpo-17422.g7_9zz.rst b/Misc/NEWS.d/next/Documentation/2020-02-19-11-13-47.bpo-17422.g7_9zz.rst new file mode 100644 index 00000000000000..f071d286176aee --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-02-19-11-13-47.bpo-17422.g7_9zz.rst @@ -0,0 +1 @@ +The language reference now specifies restrictions on class namespaces. Adapted from a patch by Ethan Furman. From c6ecd9c14081a787959e13df33e250102a658154 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 23 Feb 2020 05:44:28 -0800 Subject: [PATCH 1211/2163] bpo-39576: Clarify the word size for the 32-bit build. (GH-18616) (#18618) (cherry picked from commit b76518d43fb82ed9e5d27025d18c90a23d525c90) Authored-by: Stefan Krah --- Doc/library/decimal.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index 2a51429bdff5c9..4e640cc695990f 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -2155,8 +2155,8 @@ RAM and expect 10 simultaneous operands using a maximum of 500MB each:: >>> import sys >>> - >>> # Maximum number of digits for a single operand using 500MB in 8 byte words - >>> # with 19 (9 for the 32-bit version) digits per word: + >>> # Maximum number of digits for a single operand using 500MB in 8-byte words + >>> # with 19 digits per word (4-byte and 9 digits for the 32-bit build): >>> maxdigits = 19 * ((500 * 1024**2) // 8) >>> >>> # Check that this works: From 13951c7f25c628ea2dc0a869ebe18e7bf593fa6d Mon Sep 17 00:00:00 2001 From: Pablo Galindo Date: Sun, 23 Feb 2020 20:48:27 +0000 Subject: [PATCH 1212/2163] [3.8] bpo-39427: Document -X opt options in the CLI --help and the man page (GH-18131) (GH-18133) https://bugs.python.org/issue39427 Automerge-Triggered-By: @pablogsal (cherry picked from commit 41f0ef6abbd304409c55612a08788cdd59fbc8a3) Co-authored-by: Pablo Galindo --- .../2020-01-22-22-28-04.bpo-39427.LiO-Eo.rst | 2 + Misc/python.man | 40 ++++++++++++++++++- Python/initconfig.c | 33 ++++++++++++++- 3 files changed, 73 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-22-22-28-04.bpo-39427.LiO-Eo.rst diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-22-22-28-04.bpo-39427.LiO-Eo.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-22-22-28-04.bpo-39427.LiO-Eo.rst new file mode 100644 index 00000000000000..a3915a4d81c79c --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-01-22-22-28-04.bpo-39427.LiO-Eo.rst @@ -0,0 +1,2 @@ +Document all possibilities for the ``-X`` options in the command line help +section. Patch by Pablo Galindo. diff --git a/Misc/python.man b/Misc/python.man index 3aa9f1f9c7eace..3645b0206eb2b5 100644 --- a/Misc/python.man +++ b/Misc/python.man @@ -273,7 +273,45 @@ field matches the line number, where zero matches all line numbers and is thus equivalent to an omitted line number. .TP .BI "\-X " option -Set implementation specific option. +Set implementation specific option. The following options are available: + + -X faulthandler: enable faulthandler + + -X showrefcount: output the total reference count and number of used + memory blocks when the program finishes or after each statement in the + interactive interpreter. This only works on debug builds + + -X tracemalloc: start tracing Python memory allocations using the + tracemalloc module. By default, only the most recent frame is stored in a + traceback of a trace. Use -X tracemalloc=NFRAME to start tracing with a + traceback limit of NFRAME frames + + -X showalloccount: output the total count of allocated objects for each + type when the program finishes. This only works when Python was built with + COUNT_ALLOCS defined + + -X importtime: show how long each import takes. It shows module name, + cumulative time (including nested imports) and self time (excluding + nested imports). Note that its output may be broken in multi-threaded + application. Typical usage is python3 -X importtime -c 'import asyncio' + + -X dev: enable CPython’s “development mode”, introducing additional runtime + checks which are too expensive to be enabled by default. It will not be + more verbose than the default if the code is correct: new warnings are + only emitted when an issue is detected. Effect of the developer mode: + * Add default warning filter, as -W default + * Install debug hooks on memory allocators: see the PyMem_SetupDebugHooks() C function + * Enable the faulthandler module to dump the Python traceback on a crash + * Enable asyncio debug mode + * Set the dev_mode attribute of sys.flags to True + * io.IOBase destructor logs close() exceptions + + -X utf8: enable UTF-8 mode for operating system interfaces, overriding the default + locale-aware mode. -X utf8=0 explicitly disables UTF-8 mode (even when it would + otherwise activate automatically). See PYTHONUTF8 for more details + + -X pycache_prefix=PATH: enable writing .pyc files to a parallel tree rooted at the + given directory instead of to the code tree. .TP .B \-x Skip the first line of the source. This is intended for a DOS diff --git a/Python/initconfig.c b/Python/initconfig.c index a30fdd1bab9542..a0b2691bcd75ca 100644 --- a/Python/initconfig.c +++ b/Python/initconfig.c @@ -63,7 +63,38 @@ static const char usage_3[] = "\ -W arg : warning control; arg is action:message:category:module:lineno\n\ also PYTHONWARNINGS=arg\n\ -x : skip first line of source, allowing use of non-Unix forms of #!cmd\n\ --X opt : set implementation-specific option\n\ +-X opt : set implementation-specific option. The following options are available:\n\ +\n\ + -X faulthandler: enable faulthandler\n\ + -X showrefcount: output the total reference count and number of used\n\ + memory blocks when the program finishes or after each statement in the\n\ + interactive interpreter. This only works on debug builds\n\ + -X tracemalloc: start tracing Python memory allocations using the\n\ + tracemalloc module. By default, only the most recent frame is stored in a\n\ + traceback of a trace. Use -X tracemalloc=NFRAME to start tracing with a\n\ + traceback limit of NFRAME frames\n\ + -X showalloccount: output the total count of allocated objects for each\n\ + type when the program finishes. This only works when Python was built with\n\ + COUNT_ALLOCS defined\n\ + -X importtime: show how long each import takes. It shows module name,\n\ + cumulative time (including nested imports) and self time (excluding\n\ + nested imports). Note that its output may be broken in multi-threaded\n\ + application. Typical usage is python3 -X importtime -c 'import asyncio'\n\ + -X dev: enable CPython’s “development mode”, introducing additional runtime\n\ + checks which are too expensive to be enabled by default. Effect of the\n\ + developer mode:\n\ + * Add default warning filter, as -W default\n\ + * Install debug hooks on memory allocators: see the PyMem_SetupDebugHooks() C function\n\ + * Enable the faulthandler module to dump the Python traceback on a crash\n\ + * Enable asyncio debug mode\n\ + * Set the dev_mode attribute of sys.flags to True\n\ + * io.IOBase destructor logs close() exceptions\n\ + -X utf8: enable UTF-8 mode for operating system interfaces, overriding the default\n\ + locale-aware mode. -X utf8=0 explicitly disables UTF-8 mode (even when it would\n\ + otherwise activate automatically)\n\ + -X pycache_prefix=PATH: enable writing .pyc files to a parallel tree rooted at the\n\ + given directory instead of to the code tree\n\ +\n\ --check-hash-based-pycs always|default|never:\n\ control how Python invalidates hash-based .pyc files\n\ "; From b19f7ecfa3adc6ba1544225317b9473649815b38 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 23 Feb 2020 14:53:24 -0800 Subject: [PATCH 1213/2163] bpo-39681: Fix C pickle regression with minimal file-like objects (GH-18592) (#18630) Fix a regression where the C pickle module wouldn't allow unpickling from a file-like object that doesn't expose a readinto() method. (cherry picked from commit 9f37872e307734666a7169f7be6e3370d3068282) Co-authored-by: Antoine Pitrou Co-authored-by: Antoine Pitrou --- Lib/test/pickletester.py | 25 +++++++++-- .../2020-02-21-13-58-40.bpo-39681.zN8hf0.rst | 2 + Modules/_pickle.c | 41 ++++++++++++++++--- 3 files changed, 59 insertions(+), 9 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-02-21-13-58-40.bpo-39681.zN8hf0.rst diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index 3a8aee4320c65a..7c8383f3eaac92 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -73,6 +73,18 @@ def tell(self): raise io.UnsupportedOperation +class MinimalIO(object): + """ + A file-like object that doesn't support readinto(). + """ + def __init__(self, *args): + self._bio = io.BytesIO(*args) + self.getvalue = self._bio.getvalue + self.read = self._bio.read + self.readline = self._bio.readline + self.write = self._bio.write + + # We can't very well test the extension registry without putting known stuff # in it, but we have to be careful to restore its original state. Code # should do this: @@ -3361,7 +3373,7 @@ def test_reusing_unpickler_objects(self): f.seek(0) self.assertEqual(unpickler.load(), data2) - def _check_multiple_unpicklings(self, ioclass): + def _check_multiple_unpicklings(self, ioclass, *, seekable=True): for proto in protocols: with self.subTest(proto=proto): data1 = [(x, str(x)) for x in range(2000)] + [b"abcde", len] @@ -3374,10 +3386,10 @@ def _check_multiple_unpicklings(self, ioclass): f = ioclass(pickled * N) unpickler = self.unpickler_class(f) for i in range(N): - if f.seekable(): + if seekable: pos = f.tell() self.assertEqual(unpickler.load(), data1) - if f.seekable(): + if seekable: self.assertEqual(f.tell(), pos + len(pickled)) self.assertRaises(EOFError, unpickler.load) @@ -3385,7 +3397,12 @@ def test_multiple_unpicklings_seekable(self): self._check_multiple_unpicklings(io.BytesIO) def test_multiple_unpicklings_unseekable(self): - self._check_multiple_unpicklings(UnseekableIO) + self._check_multiple_unpicklings(UnseekableIO, seekable=False) + + def test_multiple_unpicklings_minimal(self): + # File-like object that doesn't support peek() and readinto() + # (bpo-39681) + self._check_multiple_unpicklings(MinimalIO, seekable=False) def test_unpickling_buffering_readline(self): # Issue #12687: the unpickler's buffering logic could fail with diff --git a/Misc/NEWS.d/next/Library/2020-02-21-13-58-40.bpo-39681.zN8hf0.rst b/Misc/NEWS.d/next/Library/2020-02-21-13-58-40.bpo-39681.zN8hf0.rst new file mode 100644 index 00000000000000..c10e2fd7a4b6d6 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-21-13-58-40.bpo-39681.zN8hf0.rst @@ -0,0 +1,2 @@ +Fix a regression where the C pickle module wouldn't allow unpickling from a +file-like object that doesn't expose a readinto() method. diff --git a/Modules/_pickle.c b/Modules/_pickle.c index 9f6e66f70a5462..55e2734ca2f31d 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -1371,13 +1371,42 @@ _Unpickler_ReadInto(UnpicklerObject *self, char *buf, Py_ssize_t n) } /* Read from file */ - if (!self->readinto) { + if (!self->read) { + /* We're unpickling memory, this means the input is truncated */ return bad_readline(); } if (_Unpickler_SkipConsumed(self) < 0) { return -1; } + if (!self->readinto) { + /* readinto() not supported on file-like object, fall back to read() + * and copy into destination buffer (bpo-39681) */ + PyObject* len = PyLong_FromSsize_t(n); + if (len == NULL) { + return -1; + } + PyObject* data = _Pickle_FastCall(self->read, len); + if (data == NULL) { + return -1; + } + if (!PyBytes_Check(data)) { + PyErr_Format(PyExc_ValueError, + "read() returned non-bytes object (%R)", + Py_TYPE(data)); + Py_DECREF(data); + return -1; + } + Py_ssize_t read_size = PyBytes_GET_SIZE(data); + if (read_size < n) { + Py_DECREF(data); + return bad_readline(); + } + memcpy(buf, PyBytes_AS_STRING(data), n); + Py_DECREF(data); + return n; + } + /* Call readinto() into user buffer */ PyObject *buf_obj = PyMemoryView_FromMemory(buf, n, PyBUF_WRITE); if (buf_obj == NULL) { @@ -1606,17 +1635,19 @@ _Unpickler_SetInputStream(UnpicklerObject *self, PyObject *file) _Py_IDENTIFIER(readinto); _Py_IDENTIFIER(readline); + /* Optional file methods */ if (_PyObject_LookupAttrId(file, &PyId_peek, &self->peek) < 0) { return -1; } + if (_PyObject_LookupAttrId(file, &PyId_readinto, &self->readinto) < 0) { + return -1; + } (void)_PyObject_LookupAttrId(file, &PyId_read, &self->read); - (void)_PyObject_LookupAttrId(file, &PyId_readinto, &self->readinto); (void)_PyObject_LookupAttrId(file, &PyId_readline, &self->readline); - if (!self->readline || !self->readinto || !self->read) { + if (!self->readline || !self->read) { if (!PyErr_Occurred()) { PyErr_SetString(PyExc_TypeError, - "file must have 'read', 'readinto' and " - "'readline' attributes"); + "file must have 'read' and 'readline' attributes"); } Py_CLEAR(self->read); Py_CLEAR(self->readinto); From 973348427e3e987f42ea8a871e18c8d17b5abf52 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 23 Feb 2020 18:07:33 -0800 Subject: [PATCH 1214/2163] bpo-39654: Update pyclbr doc to reflect additional information returned (GH-18528) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full nested function and class info makes it a module browser. Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Terry Jan Reedy (cherry picked from commit aea045adb8c90394264908670cbc495c5a41b65e) Co-authored-by: Hakan Çelik --- Doc/library/pyclbr.rst | 9 ++++++--- .../2020-02-18-07-42-20.bpo-39654.MoT1jI.rst | 2 ++ 2 files changed, 8 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Documentation/2020-02-18-07-42-20.bpo-39654.MoT1jI.rst diff --git a/Doc/library/pyclbr.rst b/Doc/library/pyclbr.rst index b80a2faed9b424..36e83e85c23141 100644 --- a/Doc/library/pyclbr.rst +++ b/Doc/library/pyclbr.rst @@ -1,8 +1,8 @@ -:mod:`pyclbr` --- Python class browser support -============================================== +:mod:`pyclbr` --- Python module browser support +=============================================== .. module:: pyclbr - :synopsis: Supports information extraction for a Python class browser. + :synopsis: Supports information extraction for a Python module browser. .. sectionauthor:: Fred L. Drake, Jr. @@ -29,6 +29,9 @@ modules. *path* is a sequence of directory paths prepended to ``sys.path``, which is used to locate the module source code. + This function is the original interface and is only kept for back + compatibility. It returns a filtered version of the following. + .. function:: readmodule_ex(module, path=None) diff --git a/Misc/NEWS.d/next/Documentation/2020-02-18-07-42-20.bpo-39654.MoT1jI.rst b/Misc/NEWS.d/next/Documentation/2020-02-18-07-42-20.bpo-39654.MoT1jI.rst new file mode 100644 index 00000000000000..cff201d8124769 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-02-18-07-42-20.bpo-39654.MoT1jI.rst @@ -0,0 +1,2 @@ +In pyclbr doc, update 'class' to 'module' where appropriate and add readmodule comment. +Patch by Hakan Çelik. From c97fc564a6c76ba5287f1b16bc841a1765820b0c Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 23 Feb 2020 19:33:07 -0800 Subject: [PATCH 1215/2163] bpo-39649: Remove obsolete check for `__args__` in bdb.Bdb.format_stack_entry (GH-18531) Appears to be obsolete since 75bb54c3d8. Co-authored-by: Terry Jan Reedy (cherry picked from commit 4015d1cda3cdba869103779eb6ff32ad798ff885) Co-authored-by: Daniel Hahler --- Lib/bdb.py | 9 +-------- .../Library/2020-02-23-21-27-10.bpo-39649.qiubSp.rst | 1 + 2 files changed, 2 insertions(+), 8 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-02-23-21-27-10.bpo-39649.qiubSp.rst diff --git a/Lib/bdb.py b/Lib/bdb.py index 50d9eece89ad72..18491da897357e 100644 --- a/Lib/bdb.py +++ b/Lib/bdb.py @@ -548,14 +548,7 @@ def format_stack_entry(self, frame_lineno, lprefix=': '): s += frame.f_code.co_name else: s += "" - if '__args__' in frame.f_locals: - args = frame.f_locals['__args__'] - else: - args = None - if args: - s += reprlib.repr(args) - else: - s += '()' + s += '()' if '__return__' in frame.f_locals: rv = frame.f_locals['__return__'] s += '->' diff --git a/Misc/NEWS.d/next/Library/2020-02-23-21-27-10.bpo-39649.qiubSp.rst b/Misc/NEWS.d/next/Library/2020-02-23-21-27-10.bpo-39649.qiubSp.rst new file mode 100644 index 00000000000000..5a88f79f05f0e4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-23-21-27-10.bpo-39649.qiubSp.rst @@ -0,0 +1 @@ +Remove obsolete check for `__args__` in bdb.Bdb.format_stack_entry. From aa8213486f69c15bb9e07d2232d19bf2e9875070 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 24 Feb 2020 02:52:10 -0800 Subject: [PATCH 1216/2163] Give proper credits for the memoryview implementation. (GH-18626) (#18642) (cherry picked from commit ee3bac4cba56b51ce924f13d77b97131eec1a865) Authored-by: Stefan Krah --- Objects/memoryobject.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/Objects/memoryobject.c b/Objects/memoryobject.c index 92050791b5d167..0bbcbb2e7eefa7 100644 --- a/Objects/memoryobject.c +++ b/Objects/memoryobject.c @@ -1,4 +1,14 @@ -/* Memoryview object implementation */ +/* + * Memoryview object implementation + * -------------------------------- + * + * This implementation is a complete rewrite contributed by Stefan Krah in + * Python 3.3. Substantial credit goes to Antoine Pitrou (who had already + * fortified and rewritten the previous implementation) and Nick Coghlan + * (who came up with the idea of the ManagedBuffer) for analyzing the complex + * ownership rules. + * + */ #include "Python.h" #include "pycore_object.h" From 45cc9ce4de0a333a3beb0111741de0e8ea6b5a26 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 24 Feb 2020 03:44:06 -0800 Subject: [PATCH 1217/2163] Give proper credit for figuring out and writing PEP-3118 tests. (GH-18644) (#18646) (cherry picked from commit b942ba03b8530f26240d4e36567d2ff42d701420) Authored-by: Stefan Krah --- Lib/test/test_buffer.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Lib/test/test_buffer.py b/Lib/test/test_buffer.py index 47413c03d6630e..dd84faf371e7d6 100644 --- a/Lib/test/test_buffer.py +++ b/Lib/test/test_buffer.py @@ -10,6 +10,8 @@ # the same way as the original. Thus, a substantial part of the # memoryview tests is now in this module. # +# Written and designed by Stefan Krah for Python 3.3. +# import contextlib import unittest From a199f48ea7ec22e7753baf52bffa3a89b8784d96 Mon Sep 17 00:00:00 2001 From: Ned Deily Date: Mon, 24 Feb 2020 07:59:13 -0500 Subject: [PATCH 1218/2163] Add note to Mac installer ReadMe about macOS 10.15 Gatekeeper changes. (GH-18647) --- Mac/BuildScript/resources/ReadMe.rtf | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf index d321fcd80c2eb8..4cb0111d83a530 100644 --- a/Mac/BuildScript/resources/ReadMe.rtf +++ b/Mac/BuildScript/resources/ReadMe.rtf @@ -1,5 +1,5 @@ -{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600 -{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fswiss\fcharset0 Helvetica-Oblique; +{\rtf1\ansi\ansicpg1252\cocoartf2511 +\cocoatextscaling0\cocoaplatform0{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fswiss\fcharset0 Helvetica-Oblique; \f3\fmodern\fcharset0 CourierNewPSMT;} {\colortbl;\red255\green255\blue255;} {\*\expandedcolortbl;;} @@ -36,8 +36,7 @@ The bundled \f0\b0 \ulnone \ \ This package includes its own private version of Tcl/Tk 8.6. It does not use any system-supplied or third-party supplied versions of Tcl/Tk.\ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 -\cf0 \ +\ Due to new security checks on macOS 10.15 Catalina, when launching IDLE macOS may open a window with a message \f1\b "Python" would like to access files in your Documents folder \f0\b0 . This is normal as IDLE uses your @@ -49,9 +48,14 @@ Due to new security checks on macOS 10.15 Catalina, when launching IDLE macOS ma \f0\b0 file dialog windows. Click on the \f1\b OK \f0\b0 button to proceed.\ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 -\f1\b \cf0 \ul \ulc0 \ +\f1\b \ul \ +macOS 10.15 (Catalina) Gatekeeper Requirements [changed in 3.8.2]\ + +\f0\b0 \ulnone \ +As of 2020-02-03, Apple has changed how third-party installer packages, like those provided by python.org, are notarized for verification by Gatekeeper and begun enforcing additional requirements such as code signing and use of the hardened runtime. As of 3.8.2, python.org installer packages now meet those additional notarization requirements. The necessary changes in packaging should be transparent to your use of Python but, in the unlikely event that you encounter changes in behavior between 3.8.1 and 3.8.2 in areas like ctypes, importlib, or mmap, please check bugs.python.org for existing reports and, if necessary, open a new issue.\ + +\f1\b \ul \ Other changes\ \f0\b0 \ulnone \ From 1f4cf0c22b00fefb17611546755b65d3cb488330 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 24 Feb 2020 07:06:00 -0800 Subject: [PATCH 1219/2163] bpo-39128: Added algorithm description (GH-18624) GH- [bpo-39128](https://bugs.python.org/issue39128): happy eyeballs description GH- [3.9] 39128 - happy eyeballs description (GH-18624) GH- [3.8] 39128 - happy eyeballs description (GH-18624) https://bugs.python.org/issue39128 (cherry picked from commit 8af4712a16e4b7d1b60f1faec13cd7a88da95f6a) Co-authored-by: idomic --- Doc/library/asyncio-eventloop.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 4ea8521eb0bf36..9022993e619a59 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -439,6 +439,17 @@ Opening network connections Added the *happy_eyeballs_delay* and *interleave* parameters. + Happy Eyeballs Algorithm: Success with Dual-Stack Hosts. + When a server's IPv4 path and protocol are working, but the server's + IPv6 path and protocol are not working, a dual-stack client + application experiences significant connection delay compared to an + IPv4-only client. This is undesirable because it causes the dual- + stack client to have a worse user experience. This document + specifies requirements for algorithms that reduce this user-visible + delay and provides an algorithm. + + For more information: https://tools.ietf.org/html/rfc6555 + .. versionadded:: 3.7 The *ssl_handshake_timeout* parameter. From 7b3ab5921fa25ed8b97b6296f97c5c78aacf5447 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Mon, 24 Feb 2020 22:36:25 +0100 Subject: [PATCH 1220/2163] Python 3.8.2 --- Include/patchlevel.h | 6 +- Lib/pydoc_data/topics.py | 19 ++-- Misc/NEWS.d/3.8.2.rst | 97 +++++++++++++++++++ .../2020-01-22-22-28-04.bpo-39427.LiO-Eo.rst | 2 - .../2020-02-18-01-40-13.bpo-39382.OLSJu9.rst | 3 - .../2020-02-18-07-42-20.bpo-39654.MoT1jI.rst | 2 - .../2020-02-18-18-37-07.bpo-39572.CCtzy1.rst | 1 - .../2020-02-19-11-13-47.bpo-17422.g7_9zz.rst | 1 - .../2020-02-17-21-09-03.bpo-39663.wexcsH.rst | 1 - .../2020-01-23-16-08-58.bpo-39432.Cee6mi.rst | 1 - .../2020-02-03-15-12-51.bpo-39546._Kj0Pn.rst | 3 - .../2020-02-21-13-58-40.bpo-39681.zN8hf0.rst | 2 - .../2020-02-23-21-27-10.bpo-39649.qiubSp.rst | 1 - README.rst | 4 +- 14 files changed, 115 insertions(+), 28 deletions(-) create mode 100644 Misc/NEWS.d/3.8.2.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-01-22-22-28-04.bpo-39427.LiO-Eo.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-02-18-01-40-13.bpo-39382.OLSJu9.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2020-02-18-07-42-20.bpo-39654.MoT1jI.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2020-02-18-18-37-07.bpo-39572.CCtzy1.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2020-02-19-11-13-47.bpo-17422.g7_9zz.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2020-02-17-21-09-03.bpo-39663.wexcsH.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-01-23-16-08-58.bpo-39432.Cee6mi.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-02-03-15-12-51.bpo-39546._Kj0Pn.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-02-21-13-58-40.bpo-39681.zN8hf0.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-02-23-21-27-10.bpo-39649.qiubSp.rst diff --git a/Include/patchlevel.h b/Include/patchlevel.h index bbdb7a2aa6a0dc..3671bb35829cfa 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -19,11 +19,11 @@ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 8 #define PY_MICRO_VERSION 2 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA -#define PY_RELEASE_SERIAL 2 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL +#define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.8.2rc2+" +#define PY_VERSION "3.8.2" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index 3c1b780384570c..f1fdb7fc8617c6 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Mon Feb 17 23:43:37 2020 +# Autogenerated by Sphinx on Mon Feb 24 21:52:17 2020 topics = {'assert': 'The "assert" statement\n' '**********************\n' '\n' @@ -5286,9 +5286,12 @@ 'Changed in version 3.6: Added the "\'_\'" option (see also ' '**PEP 515**).\n' '\n' - '*width* is a decimal integer defining the minimum field ' - 'width. If not\n' - 'specified, then the field width will be determined by the ' + '*width* is a decimal integer defining the minimum total ' + 'field width,\n' + 'including any prefixes, separators, and other formatting ' + 'characters.\n' + 'If not specified, then the field width will be determined ' + 'by the\n' 'content.\n' '\n' 'When no explicit alignment is given, preceding the *width* ' @@ -9001,11 +9004,15 @@ 'come from\n' 'the class definition). The "__prepare__" method should be ' 'implemented\n' - 'as a "classmethod()".\n' + 'as a "classmethod()". The namespace returned by ' + '"__prepare__" is\n' + 'passed in to "__new__", but when the final class object is ' + 'created the\n' + 'namespace is copied into a new "dict".\n' '\n' 'If the metaclass has no "__prepare__" attribute, then the ' 'class\n' - 'namespace is initialised as an empty ordered mapping.\n' + 'namespace is initialised as an empty "dict()".\n' '\n' 'See also:\n' '\n' diff --git a/Misc/NEWS.d/3.8.2.rst b/Misc/NEWS.d/3.8.2.rst new file mode 100644 index 00000000000000..bde681204cb0cb --- /dev/null +++ b/Misc/NEWS.d/3.8.2.rst @@ -0,0 +1,97 @@ +.. bpo: 39382 +.. date: 2020-02-18-01-40-13 +.. nonce: OLSJu9 +.. release date: 2020-02-24 +.. section: Core and Builtins + +Fix a use-after-free in the single inheritance path of ``issubclass()``, +when the ``__bases__`` of an object has a single reference, and so does its +first item. Patch by Yonatan Goldschmidt. + +.. + +.. bpo: 39427 +.. date: 2020-01-22-22-28-04 +.. nonce: LiO-Eo +.. section: Core and Builtins + +Document all possibilities for the ``-X`` options in the command line help +section. Patch by Pablo Galindo. + +.. + +.. bpo: 39649 +.. date: 2020-02-23-21-27-10 +.. nonce: qiubSp +.. section: Library + +Remove obsolete check for `__args__` in bdb.Bdb.format_stack_entry. + +.. + +.. bpo: 39681 +.. date: 2020-02-21-13-58-40 +.. nonce: zN8hf0 +.. section: Library + +Fix a regression where the C pickle module wouldn't allow unpickling from a +file-like object that doesn't expose a readinto() method. + +.. + +.. bpo: 39546 +.. date: 2020-02-03-15-12-51 +.. nonce: _Kj0Pn +.. section: Library + +Fix a regression in :class:`~argparse.ArgumentParser` where +``allow_abbrev=False`` was ignored for long options that used a prefix +character other than "-". + +.. + +.. bpo: 39432 +.. date: 2020-01-23-16-08-58 +.. nonce: Cee6mi +.. section: Library + +Implement PEP-489 algorithm for non-ascii "PyInit\_..." symbol names in +distutils to make it export the correct init symbol also on Windows. + +.. + +.. bpo: 17422 +.. date: 2020-02-19-11-13-47 +.. nonce: g7_9zz +.. section: Documentation + +The language reference now specifies restrictions on class namespaces. +Adapted from a patch by Ethan Furman. + +.. + +.. bpo: 39572 +.. date: 2020-02-18-18-37-07 +.. nonce: CCtzy1 +.. section: Documentation + +Updated documentation of ``total`` flag of TypeDict. + +.. + +.. bpo: 39654 +.. date: 2020-02-18-07-42-20 +.. nonce: MoT1jI +.. section: Documentation + +In pyclbr doc, update 'class' to 'module' where appropriate and add +readmodule comment. Patch by Hakan Çelik. + +.. + +.. bpo: 39663 +.. date: 2020-02-17-21-09-03 +.. nonce: wexcsH +.. section: IDLE + +Add tests for pyparse find_good_parse_start(). diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-01-22-22-28-04.bpo-39427.LiO-Eo.rst b/Misc/NEWS.d/next/Core and Builtins/2020-01-22-22-28-04.bpo-39427.LiO-Eo.rst deleted file mode 100644 index a3915a4d81c79c..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-01-22-22-28-04.bpo-39427.LiO-Eo.rst +++ /dev/null @@ -1,2 +0,0 @@ -Document all possibilities for the ``-X`` options in the command line help -section. Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-02-18-01-40-13.bpo-39382.OLSJu9.rst b/Misc/NEWS.d/next/Core and Builtins/2020-02-18-01-40-13.bpo-39382.OLSJu9.rst deleted file mode 100644 index 605f4c8e5dfd1f..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-02-18-01-40-13.bpo-39382.OLSJu9.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix a use-after-free in the single inheritance path of ``issubclass()``, when -the ``__bases__`` of an object has a single reference, and so does its first item. -Patch by Yonatan Goldschmidt. diff --git a/Misc/NEWS.d/next/Documentation/2020-02-18-07-42-20.bpo-39654.MoT1jI.rst b/Misc/NEWS.d/next/Documentation/2020-02-18-07-42-20.bpo-39654.MoT1jI.rst deleted file mode 100644 index cff201d8124769..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-02-18-07-42-20.bpo-39654.MoT1jI.rst +++ /dev/null @@ -1,2 +0,0 @@ -In pyclbr doc, update 'class' to 'module' where appropriate and add readmodule comment. -Patch by Hakan Çelik. diff --git a/Misc/NEWS.d/next/Documentation/2020-02-18-18-37-07.bpo-39572.CCtzy1.rst b/Misc/NEWS.d/next/Documentation/2020-02-18-18-37-07.bpo-39572.CCtzy1.rst deleted file mode 100644 index d47bb455e71d1e..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-02-18-18-37-07.bpo-39572.CCtzy1.rst +++ /dev/null @@ -1 +0,0 @@ -Updated documentation of ``total`` flag of TypeDict. diff --git a/Misc/NEWS.d/next/Documentation/2020-02-19-11-13-47.bpo-17422.g7_9zz.rst b/Misc/NEWS.d/next/Documentation/2020-02-19-11-13-47.bpo-17422.g7_9zz.rst deleted file mode 100644 index f071d286176aee..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-02-19-11-13-47.bpo-17422.g7_9zz.rst +++ /dev/null @@ -1 +0,0 @@ -The language reference now specifies restrictions on class namespaces. Adapted from a patch by Ethan Furman. diff --git a/Misc/NEWS.d/next/IDLE/2020-02-17-21-09-03.bpo-39663.wexcsH.rst b/Misc/NEWS.d/next/IDLE/2020-02-17-21-09-03.bpo-39663.wexcsH.rst deleted file mode 100644 index 19e16329ce0a01..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2020-02-17-21-09-03.bpo-39663.wexcsH.rst +++ /dev/null @@ -1 +0,0 @@ -Add tests for pyparse find_good_parse_start(). diff --git a/Misc/NEWS.d/next/Library/2020-01-23-16-08-58.bpo-39432.Cee6mi.rst b/Misc/NEWS.d/next/Library/2020-01-23-16-08-58.bpo-39432.Cee6mi.rst deleted file mode 100644 index 21c4ba86207550..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-23-16-08-58.bpo-39432.Cee6mi.rst +++ /dev/null @@ -1 +0,0 @@ -Implement PEP-489 algorithm for non-ascii "PyInit\_..." symbol names in distutils to make it export the correct init symbol also on Windows. diff --git a/Misc/NEWS.d/next/Library/2020-02-03-15-12-51.bpo-39546._Kj0Pn.rst b/Misc/NEWS.d/next/Library/2020-02-03-15-12-51.bpo-39546._Kj0Pn.rst deleted file mode 100644 index 8f035e79963e00..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-02-03-15-12-51.bpo-39546._Kj0Pn.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix a regression in :class:`~argparse.ArgumentParser` where -``allow_abbrev=False`` was ignored for long options that used a prefix -character other than "-". diff --git a/Misc/NEWS.d/next/Library/2020-02-21-13-58-40.bpo-39681.zN8hf0.rst b/Misc/NEWS.d/next/Library/2020-02-21-13-58-40.bpo-39681.zN8hf0.rst deleted file mode 100644 index c10e2fd7a4b6d6..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-02-21-13-58-40.bpo-39681.zN8hf0.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix a regression where the C pickle module wouldn't allow unpickling from a -file-like object that doesn't expose a readinto() method. diff --git a/Misc/NEWS.d/next/Library/2020-02-23-21-27-10.bpo-39649.qiubSp.rst b/Misc/NEWS.d/next/Library/2020-02-23-21-27-10.bpo-39649.qiubSp.rst deleted file mode 100644 index 5a88f79f05f0e4..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-02-23-21-27-10.bpo-39649.qiubSp.rst +++ /dev/null @@ -1 +0,0 @@ -Remove obsolete check for `__args__` in bdb.Bdb.format_stack_entry. diff --git a/README.rst b/README.rst index 5858120b429290..cf74d1d8ee6b32 100644 --- a/README.rst +++ b/README.rst @@ -1,5 +1,5 @@ -This is Python version 3.8.2rc2 -=============================== +This is Python version 3.8.2 +============================ .. image:: https://travis-ci.org/python/cpython.svg?branch=3.8 :alt: CPython build status on Travis CI From daef21ce7dfd3735101d85d6ebf7554187c33ab8 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 24 Feb 2020 19:42:39 -0800 Subject: [PATCH 1221/2163] bpo-30566: Fix IndexError when using punycode codec (GH-18632) Trying to decode an invalid string with the punycode codec shoud raise UnicodeError. (cherry picked from commit ba22e8f174309979d90047c5dc64fcb63bc2c32e) Co-authored-by: Berker Peksag --- Lib/encodings/punycode.py | 2 +- Lib/test/test_codecs.py | 12 ++++++++++++ .../Library/2020-02-24-03-45-28.bpo-30566.qROxty.rst | 2 ++ 3 files changed, 15 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2020-02-24-03-45-28.bpo-30566.qROxty.rst diff --git a/Lib/encodings/punycode.py b/Lib/encodings/punycode.py index 66c51013ea431a..1c5726447077b1 100644 --- a/Lib/encodings/punycode.py +++ b/Lib/encodings/punycode.py @@ -143,7 +143,7 @@ def decode_generalized_number(extended, extpos, bias, errors): digit = char - 22 # 0x30-26 elif errors == "strict": raise UnicodeError("Invalid extended code point '%s'" - % extended[extpos]) + % extended[extpos-1]) else: return extpos, None t = T(j, bias) diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index b37525bf660430..8c10e948e8041d 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -1331,6 +1331,18 @@ def test_decode(self): puny = puny.decode("ascii").encode("ascii") self.assertEqual(uni, puny.decode("punycode")) + def test_decode_invalid(self): + testcases = [ + (b"xn--w&", "strict", UnicodeError()), + (b"xn--w&", "ignore", "xn-"), + ] + for puny, errors, expected in testcases: + with self.subTest(puny=puny, errors=errors): + if isinstance(expected, Exception): + self.assertRaises(UnicodeError, puny.decode, "punycode", errors) + else: + self.assertEqual(puny.decode("punycode", errors), expected) + # From http://www.gnu.org/software/libidn/draft-josefsson-idn-test-vectors.html nameprep_tests = [ diff --git a/Misc/NEWS.d/next/Library/2020-02-24-03-45-28.bpo-30566.qROxty.rst b/Misc/NEWS.d/next/Library/2020-02-24-03-45-28.bpo-30566.qROxty.rst new file mode 100644 index 00000000000000..c780633030090d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-24-03-45-28.bpo-30566.qROxty.rst @@ -0,0 +1,2 @@ +Fix :exc:`IndexError` when trying to decode an invalid string with punycode +codec. From eb4a3dfbb9b797d4b2ecf70cb36ac08219ed065a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Tue, 25 Feb 2020 12:45:02 +0100 Subject: [PATCH 1222/2163] Post 3.8.2 --- Include/patchlevel.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h index 3671bb35829cfa..affbd82a709762 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.8.2" +#define PY_VERSION "3.8.2+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. From 1bbb81b251bcc8b05e0cd33cd36aef55481b13db Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 25 Feb 2020 12:24:48 -0800 Subject: [PATCH 1223/2163] bpo-38403: Update nuspec file for deprecated field and git repository (GH-18657) (cherry picked from commit d6448919702142123d937a54f20a81aeaf8d2acc) Co-authored-by: Steve Dower --- .../2020-02-25-18-43-34.bpo-34803.S3VcS0.rst | 2 ++ PC/icons/logo.svg | 1 + PC/icons/logox128.png | Bin 0 -> 1203 bytes PC/layout/support/nuspec.py | 6 ++++++ 4 files changed, 9 insertions(+) create mode 100644 Misc/NEWS.d/next/Windows/2020-02-25-18-43-34.bpo-34803.S3VcS0.rst create mode 100644 PC/icons/logo.svg create mode 100644 PC/icons/logox128.png diff --git a/Misc/NEWS.d/next/Windows/2020-02-25-18-43-34.bpo-34803.S3VcS0.rst b/Misc/NEWS.d/next/Windows/2020-02-25-18-43-34.bpo-34803.S3VcS0.rst new file mode 100644 index 00000000000000..144ffd50af0e91 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-02-25-18-43-34.bpo-34803.S3VcS0.rst @@ -0,0 +1,2 @@ +Package for nuget.org now includes repository reference and bundled icon +image. diff --git a/PC/icons/logo.svg b/PC/icons/logo.svg new file mode 100644 index 00000000000000..6f521503a38328 --- /dev/null +++ b/PC/icons/logo.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/PC/icons/logox128.png b/PC/icons/logox128.png new file mode 100644 index 0000000000000000000000000000000000000000..d2655c72e7df49618b8c58795eccef7985e3be64 GIT binary patch literal 1203 zcmV;k1WfyhP)Px#1ZP1_K>z@;j|==^1poj5V^B;~MF0Q*FmR9mzc4sxm;c8&HENar#Wgf&lmEvw zHffgs#x^x-mH)>zH))pt#y2)!HEWjt$Tc@=m;cB&Hfomt$Tl@;mjB2#Hfomt z$Tl`=mjB2$Hfomt$Tl`=mjB2$*mnqA0000VbW%=J01yx$ATTgMKu}O%U~q7NfRB%# zpRcdKzt7L#-|z2sM@*Rj000SaNLh0L02dJe02dJf$|mza000B7Nklv|2$xXs`NTm9N8`uzDC~#wt3Uug{`*bIw1Uzl-PndOkz} z{_RWd~L>EMRN`WT7j$^G|!?r0|Kpp2wAs2K}t~UIebb>W848 z75bI_0s2|-FJi@?h_H>E2stO9n->D)oPhs(!BHfLPK|-nft(W<+twXb`AFADJ7lGI zSn~`;S~6tr30x&4B6Yb%cEQHRnJAcw)YVYM!eObAh}0znh?U;)7$6fFtc#iB)gcwB zyFGS7lp}JHo6rcnOqNQd3R{r_f?Sb{)W=pt>kSQ2h&&n%rd9cj(ZWX zP(F~l-m?p$+~8#XGvu6*`neo=#Mlujr#!rz_a_d;wAivG$yq{QjPT{f;SK`Ju2Qrx`EKDOg<{U zMDl(hr1JCo=t*Q$ESQYNk4Po9?>A%P?L{P))QL{_@3ZeNQa#4i8#xG`kgiP_EchJA zIU&LSy?~q({zRmD$(hvYOh8U!a`dwTeU14SvEsKxsx*$1&T#ya$V=Q!@J_!Y(%s{s z7|S9NttAp(hY+urMIu^r*B^0FOY1dl(Av~Kt0;IZx>nS+63`YG8^jx5JtMc{85@P|zz5vs-0Y`~@ RG7|s*002ovPDHLkV1jb|Ev^6n literal 0 HcmV?d00001 diff --git a/PC/layout/support/nuspec.py b/PC/layout/support/nuspec.py index b85095c555fe0a..9c6a9a91595098 100644 --- a/PC/layout/support/nuspec.py +++ b/PC/layout/support/nuspec.py @@ -3,6 +3,7 @@ """ import os +import sys from .constants import * @@ -14,6 +15,7 @@ "PYTHON_TAG": VER_DOT, "PYTHON_VERSION": os.getenv("PYTHON_NUSPEC_VERSION"), "FILELIST": r' ', + "GIT": sys._git, } NUSPEC_PLATFORM_DATA = dict( @@ -42,10 +44,13 @@ tools\LICENSE.txt https://www.python.org/ Installs {PYTHON_BITNESS} Python for use in build scenarios. + images\logox128.png https://www.python.org/static/favicon.ico python + + {FILELIST} @@ -68,5 +73,6 @@ def get_nuspec_layout(ns): data[k] = v if ns.include_all or ns.include_props: data["FILELIST"] = FILELIST_WITH_PROPS + data["LOGO"] = ns.source / "PC" / "icons" / "logox128.png" nuspec = NUSPEC_TEMPLATE.format_map(data) yield "python.nuspec", ("python.nuspec", nuspec.encode("utf-8")) From c3536b79425edc623860073eed4cf8945b2281dc Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 25 Feb 2020 22:11:16 -0800 Subject: [PATCH 1224/2163] Doc: int -> int or Py_ssize_t (GH-18663) (cherry picked from commit 57c7a0bdf4f7da8cf47f797f075950f6b8c98b99) Co-authored-by: Inada Naoki --- Doc/c-api/arg.rst | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst index f17c63d0b9dc93..b7baad589a72c8 100644 --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -105,7 +105,7 @@ which disallows mutable objects such as :class:`bytearray`. Like ``s*``, but the Python object may also be ``None``, in which case the ``buf`` member of the :c:type:`Py_buffer` structure is set to ``NULL``. -``z#`` (:class:`str`, read-only :term:`bytes-like object` or ``None``) [const char \*, int] +``z#`` (:class:`str`, read-only :term:`bytes-like object` or ``None``) [const char \*, int or :c:type:`Py_ssize_t`] Like ``s#``, but the Python object may also be ``None``, in which case the C pointer is set to ``NULL``. @@ -124,7 +124,7 @@ which disallows mutable objects such as :class:`bytearray`. bytes-like objects. **This is the recommended way to accept binary data.** -``y#`` (read-only :term:`bytes-like object`) [const char \*, int] +``y#`` (read-only :term:`bytes-like object`) [const char \*, int or :c:type:`Py_ssize_t`] This variant on ``s#`` doesn't accept Unicode objects, only bytes-like objects. @@ -155,7 +155,7 @@ which disallows mutable objects such as :class:`bytearray`. Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using :c:func:`PyUnicode_AsWideCharString`. -``u#`` (:class:`str`) [const Py_UNICODE \*, int] +``u#`` (:class:`str`) [const Py_UNICODE \*, int or :c:type:`Py_ssize_t`] This variant on ``u`` stores into two C variables, the first one a pointer to a Unicode data buffer, the second one its length. This variant allows null code points. @@ -172,7 +172,7 @@ which disallows mutable objects such as :class:`bytearray`. Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using :c:func:`PyUnicode_AsWideCharString`. -``Z#`` (:class:`str` or ``None``) [const Py_UNICODE \*, int] +``Z#`` (:class:`str` or ``None``) [const Py_UNICODE \*, int or :c:type:`Py_ssize_t`] Like ``u#``, but the Python object may also be ``None``, in which case the :c:type:`Py_UNICODE` pointer is set to ``NULL``. @@ -213,7 +213,7 @@ which disallows mutable objects such as :class:`bytearray`. recoding them. Instead, the implementation assumes that the byte string object uses the encoding passed in as parameter. -``es#`` (:class:`str`) [const char \*encoding, char \*\*buffer, int \*buffer_length] +``es#`` (:class:`str`) [const char \*encoding, char \*\*buffer, int or :c:type:`Py_ssize_t` \*buffer_length] This variant on ``s#`` is used for encoding Unicode into a character buffer. Unlike the ``es`` format, this variant allows input data which contains NUL characters. @@ -244,7 +244,7 @@ which disallows mutable objects such as :class:`bytearray`. In both cases, *\*buffer_length* is set to the length of the encoded data without the trailing NUL byte. -``et#`` (:class:`str`, :class:`bytes` or :class:`bytearray`) [const char \*encoding, char \*\*buffer, int \*buffer_length] +``et#`` (:class:`str`, :class:`bytes` or :class:`bytearray`) [const char \*encoding, char \*\*buffer, int or :c:type:`Py_ssize_t` \*buffer_length] Same as ``es#`` except that byte string objects are passed through without recoding them. Instead, the implementation assumes that the byte string object uses the encoding passed in as parameter. @@ -549,7 +549,7 @@ Building values Convert a null-terminated C string to a Python :class:`str` object using ``'utf-8'`` encoding. If the C string pointer is ``NULL``, ``None`` is used. - ``s#`` (:class:`str` or ``None``) [const char \*, int] + ``s#`` (:class:`str` or ``None``) [const char \*, int or :c:type:`Py_ssize_t`] Convert a C string and its length to a Python :class:`str` object using ``'utf-8'`` encoding. If the C string pointer is ``NULL``, the length is ignored and ``None`` is returned. @@ -558,14 +558,14 @@ Building values This converts a C string to a Python :class:`bytes` object. If the C string pointer is ``NULL``, ``None`` is returned. - ``y#`` (:class:`bytes`) [const char \*, int] + ``y#`` (:class:`bytes`) [const char \*, int or :c:type:`Py_ssize_t`] This converts a C string and its lengths to a Python object. If the C string pointer is ``NULL``, ``None`` is returned. ``z`` (:class:`str` or ``None``) [const char \*] Same as ``s``. - ``z#`` (:class:`str` or ``None``) [const char \*, int] + ``z#`` (:class:`str` or ``None``) [const char \*, int or :c:type:`Py_ssize_t`] Same as ``s#``. ``u`` (:class:`str`) [const wchar_t \*] @@ -573,7 +573,7 @@ Building values data to a Python Unicode object. If the Unicode buffer pointer is ``NULL``, ``None`` is returned. - ``u#`` (:class:`str`) [const wchar_t \*, int] + ``u#`` (:class:`str`) [const wchar_t \*, int or :c:type:`Py_ssize_t`] Convert a Unicode (UTF-16 or UCS-4) data buffer and its length to a Python Unicode object. If the Unicode buffer pointer is ``NULL``, the length is ignored and ``None`` is returned. @@ -581,7 +581,7 @@ Building values ``U`` (:class:`str` or ``None``) [const char \*] Same as ``s``. - ``U#`` (:class:`str` or ``None``) [const char \*, int] + ``U#`` (:class:`str` or ``None``) [const char \*, int or :c:type:`Py_ssize_t`] Same as ``s#``. ``i`` (:class:`int`) [int] From 343bc06d8047e4a2e675394528dbb5155be1b3b5 Mon Sep 17 00:00:00 2001 From: Steve Dower Date: Wed, 26 Feb 2020 19:57:14 +0000 Subject: [PATCH 1225/2163] bpo-39699: Don't silence make on Azure and Github CIs (GH-18583) Co-authored-by: Ammar Askar --- .azure-pipelines/macos-steps.yml | 2 +- .azure-pipelines/posix-steps.yml | 2 +- .github/workflows/build.yml | 4 ++-- .github/workflows/coverage.yml | 2 +- .github/workflows/doc.yml | 18 +++++++++--------- Doc/whatsnew/3.0.rst | 2 ++ 6 files changed, 16 insertions(+), 14 deletions(-) diff --git a/.azure-pipelines/macos-steps.yml b/.azure-pipelines/macos-steps.yml index d2ca580a93d7dd..fa38a0df8c87b8 100644 --- a/.azure-pipelines/macos-steps.yml +++ b/.azure-pipelines/macos-steps.yml @@ -6,7 +6,7 @@ steps: - script: ./configure --with-pydebug --with-openssl=/usr/local/opt/openssl --prefix=/opt/python-azdev displayName: 'Configure CPython (debug)' -- script: make -s -j4 +- script: make -j4 displayName: 'Build CPython' - script: make pythoninfo diff --git a/.azure-pipelines/posix-steps.yml b/.azure-pipelines/posix-steps.yml index 3ed3abd02a7146..a63659fa204910 100644 --- a/.azure-pipelines/posix-steps.yml +++ b/.azure-pipelines/posix-steps.yml @@ -20,7 +20,7 @@ steps: - script: ./configure --with-pydebug displayName: 'Configure CPython (debug)' -- script: make -s -j4 +- script: make -j4 displayName: 'Build CPython' - ${{ if eq(parameters.coverage, 'true') }}: diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 16d6f0db8c908f..2b4aec6e30975e 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -51,7 +51,7 @@ jobs: - name: Configure CPython run: ./configure --with-pydebug --with-openssl=/usr/local/opt/openssl --prefix=/opt/python-dev - name: Build CPython - run: make -s -j4 + run: make -j4 - name: Display build info run: make pythoninfo - name: Tests @@ -78,7 +78,7 @@ jobs: - name: Configure CPython run: ./configure --with-pydebug --with-openssl=$PWD/multissl/openssl/$OPENSSL_VER - name: Build CPython - run: make -s -j4 + run: make -j4 - name: Display build info run: make pythoninfo - name: Tests diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index e8b47b390e5a79..8e1b764ca8df48 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -40,7 +40,7 @@ jobs: - name: Configure CPython run: ./configure --with-openssl=$PWD/multissl/openssl/$OPENSSL_VER - name: Build CPython - run: make -s -j4 + run: make -j4 - name: Display build info run: make pythoninfo - name: 'Coverage Preparation' diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml index 405b12e3d29c95..c8d395cea5156c 100644 --- a/.github/workflows/doc.yml +++ b/.github/workflows/doc.yml @@ -23,17 +23,17 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v1 - - uses: actions/setup-python@v1 - with: - python-version: '3.7' - architecture: 'x64' + - name: 'Install Dependencies' + run: sudo ./.github/workflows/posix-deps-apt.sh && sudo apt-get install wamerican + - name: 'Configure CPython' + run: ./configure --with-pydebug + - name: 'Build CPython' + run: make -j4 - name: 'Install build dependencies' - run: python -m pip install sphinx==2.2.0 blurb python-docs-theme + run: make -C Doc/ PYTHON=../python venv - name: 'Build documentation' - run: | - cd Doc - make check suspicious html PYTHON=python - - name: Upload + run: xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W -j4" doctest suspicious html + - name: 'Upload' uses: actions/upload-artifact@v1 with: name: doc-html diff --git a/Doc/whatsnew/3.0.rst b/Doc/whatsnew/3.0.rst index 880958d3edb900..6b8bd8861fe9b3 100644 --- a/Doc/whatsnew/3.0.rst +++ b/Doc/whatsnew/3.0.rst @@ -2,6 +2,8 @@ What's New In Python 3.0 **************************** +TEST CHANGE TO BE UNDONE + .. XXX Add trademark info for Apple, Microsoft. :Author: Guido van Rossum From 846ca4961da24669e6e0c901986e66ff485917b2 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 28 Feb 2020 10:43:25 -0800 Subject: [PATCH 1226/2163] bpo-39781: Do not jump when select in IDLE codecontext (GH-18683) Previously, the button-up part of selecting with a mouse was treated as a click that meant 'jump' to this line, which modified the context and undid the selection (cherry picked from commit c705fd1e89ccb8f6d414ec817b4616546147d877) Co-authored-by: Terry Jan Reedy --- Lib/idlelib/NEWS.txt | 2 + Lib/idlelib/codecontext.py | 44 +++++++++++-------- Lib/idlelib/idle_test/test_codecontext.py | 8 ++++ .../2020-02-27-22-17-09.bpo-39781.bbYBeL.rst | 1 + 4 files changed, 36 insertions(+), 19 deletions(-) create mode 100644 Misc/NEWS.d/next/IDLE/2020-02-27-22-17-09.bpo-39781.bbYBeL.rst diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index fe1706ed5a8be0..134f843d85e8a6 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,8 @@ Released on 2019-12-16? ====================================== +bpo-39781: Selecting code context lines no longer causes a jump. + bpo-39663: Add tests for pyparse find_good_parse_start(). bpo-39600: Remove duplicate font names from configuration list. diff --git a/Lib/idlelib/codecontext.py b/Lib/idlelib/codecontext.py index 4ce98136fe4175..989b30e5994650 100644 --- a/Lib/idlelib/codecontext.py +++ b/Lib/idlelib/codecontext.py @@ -7,7 +7,6 @@ enclosing block. The number of hint lines is determined by the maxlines variable in the codecontext section of config-extensions.def. Lines which do not open blocks are not shown in the context hints pane. - """ import re from sys import maxsize as INFINITY @@ -17,8 +16,8 @@ from idlelib.config import idleConf -BLOCKOPENERS = {"class", "def", "elif", "else", "except", "finally", "for", - "if", "try", "while", "with", "async"} +BLOCKOPENERS = {'class', 'def', 'if', 'elif', 'else', 'while', 'for', + 'try', 'except', 'finally', 'with', 'async'} def get_spaces_firstword(codeline, c=re.compile(r"^(\s*)(\w*)")): @@ -84,7 +83,7 @@ def __del__(self): if self.t1 is not None: try: self.text.after_cancel(self.t1) - except tkinter.TclError: + except tkinter.TclError: # pragma: no cover pass self.t1 = None @@ -112,7 +111,7 @@ def toggle_code_context_event(self, event=None): padx += widget.tk.getint(info['padx']) padx += widget.tk.getint(widget.cget('padx')) border += widget.tk.getint(widget.cget('border')) - self.context = tkinter.Text( + context = self.context = tkinter.Text( self.editwin.text_frame, height=1, width=1, # Don't request more than we get. @@ -120,11 +119,11 @@ def toggle_code_context_event(self, event=None): padx=padx, border=border, relief=SUNKEN, state='disabled') self.update_font() self.update_highlight_colors() - self.context.bind('', self.jumptoline) + context.bind('', self.jumptoline) # Get the current context and initiate the recurring update event. self.timer_event() # Grid the context widget above the text widget. - self.context.grid(row=0, column=1, sticky=NSEW) + context.grid(row=0, column=1, sticky=NSEW) line_number_colors = idleConf.GetHighlight(idleConf.CurrentTheme(), 'linenumber') @@ -215,18 +214,25 @@ def update_code_context(self): self.context['state'] = 'disabled' def jumptoline(self, event=None): - "Show clicked context line at top of editor." - lines = len(self.info) - if lines == 1: # No context lines are showing. - newtop = 1 - else: - # Line number clicked. - contextline = int(float(self.context.index('insert'))) - # Lines not displayed due to maxlines. - offset = max(1, lines - self.context_depth) - 1 - newtop = self.info[offset + contextline][0] - self.text.yview(f'{newtop}.0') - self.update_code_context() + """ Show clicked context line at top of editor. + + If a selection was made, don't jump; allow copying. + If no visible context, show the top line of the file. + """ + try: + self.context.index("sel.first") + except tkinter.TclError: + lines = len(self.info) + if lines == 1: # No context lines are showing. + newtop = 1 + else: + # Line number clicked. + contextline = int(float(self.context.index('insert'))) + # Lines not displayed due to maxlines. + offset = max(1, lines - self.context_depth) - 1 + newtop = self.info[offset + contextline][0] + self.text.yview(f'{newtop}.0') + self.update_code_context() def timer_event(self): "Event on editor text widget triggered every UPDATEINTERVAL ms." diff --git a/Lib/idlelib/idle_test/test_codecontext.py b/Lib/idlelib/idle_test/test_codecontext.py index 3ec49e97af6f91..9578cc731a6f93 100644 --- a/Lib/idlelib/idle_test/test_codecontext.py +++ b/Lib/idlelib/idle_test/test_codecontext.py @@ -332,6 +332,14 @@ def test_jumptoline(self): jump() eq(cc.topvisible, 8) + # Context selection stops jump. + cc.text.yview('5.0') + cc.update_code_context() + cc.context.tag_add('sel', '1.0', '2.0') + cc.context.mark_set('insert', '1.0') + jump() # Without selection, to line 2. + eq(cc.topvisible, 5) + @mock.patch.object(codecontext.CodeContext, 'update_code_context') def test_timer_event(self, mock_update): # Ensure code context is not active. diff --git a/Misc/NEWS.d/next/IDLE/2020-02-27-22-17-09.bpo-39781.bbYBeL.rst b/Misc/NEWS.d/next/IDLE/2020-02-27-22-17-09.bpo-39781.bbYBeL.rst new file mode 100644 index 00000000000000..4ae0defc2e2179 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2020-02-27-22-17-09.bpo-39781.bbYBeL.rst @@ -0,0 +1 @@ +Selecting code context lines no longer causes a jump. From 445152e0d3ab6e4381aef8d1404c2c17a516070f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 28 Feb 2020 12:06:53 -0800 Subject: [PATCH 1227/2163] bpo-13790: Change 'string' to 'specification' in format doc (GH-18690) (cherry picked from commit 916895f93905f8b8dad677cceff501833f5a633a) Co-authored-by: Terry Jan Reedy --- Doc/library/string.rst | 4 ++-- .../Documentation/2020-02-28-14-39-25.bpo-13790.hvLaRI.rst | 1 + 2 files changed, 3 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Documentation/2020-02-28-14-39-25.bpo-13790.hvLaRI.rst diff --git a/Doc/library/string.rst b/Doc/library/string.rst index 89c169a512b520..fa906f799c1082 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -302,9 +302,9 @@ specification is to be interpreted. Most built-in types implement the following options for format specifications, although some of the formatting options are only supported by the numeric types. -A general convention is that an empty format string (``""``) produces +A general convention is that an empty format specification produces the same result as if you had called :func:`str` on the value. A -non-empty format string typically modifies the result. +non-empty format specification typically modifies the result. The general form of a *standard format specifier* is: diff --git a/Misc/NEWS.d/next/Documentation/2020-02-28-14-39-25.bpo-13790.hvLaRI.rst b/Misc/NEWS.d/next/Documentation/2020-02-28-14-39-25.bpo-13790.hvLaRI.rst new file mode 100644 index 00000000000000..77db173168fc53 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-02-28-14-39-25.bpo-13790.hvLaRI.rst @@ -0,0 +1 @@ +Change 'string' to 'specification' in format doc. From 7f53d87cb0324e391c969855ed8f4b3255f9f744 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 28 Feb 2020 15:31:20 -0800 Subject: [PATCH 1228/2163] bpo-39718: add TYPE_IGNORE, COLONEQUAL to py38 changes in token (GH-18598) (cherry picked from commit c2f7eb254bee036afc8a71437ec6aac82f06a1ce) Co-authored-by: Shantanu --- Doc/library/token.rst | 2 +- .../next/Documentation/2020-02-21-22-05-20.bpo-39718.xtBoSi.rst | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Documentation/2020-02-21-22-05-20.bpo-39718.xtBoSi.rst diff --git a/Doc/library/token.rst b/Doc/library/token.rst index 1777929be739d1..dab8f0fa9b64fc 100644 --- a/Doc/library/token.rst +++ b/Doc/library/token.rst @@ -87,7 +87,7 @@ the :mod:`tokenize` module. now tokenized as :data:`NAME` tokens. .. versionchanged:: 3.8 - Added :data:`TYPE_COMMENT`. + Added :data:`TYPE_COMMENT`, :data:`TYPE_IGNORE`, :data:`COLONEQUAL`. Added :data:`AWAIT` and :data:`ASYNC` tokens back (they're needed to support parsing older Python versions for :func:`ast.parse` with ``feature_version`` set to 6 or lower). diff --git a/Misc/NEWS.d/next/Documentation/2020-02-21-22-05-20.bpo-39718.xtBoSi.rst b/Misc/NEWS.d/next/Documentation/2020-02-21-22-05-20.bpo-39718.xtBoSi.rst new file mode 100644 index 00000000000000..8072f617192b44 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-02-21-22-05-20.bpo-39718.xtBoSi.rst @@ -0,0 +1 @@ +Update :mod:`token` documentation to reflect additions in Python 3.8 \ No newline at end of file From 45c4112b7250cb600aa2f3a42b4e6b5bfd2919c4 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 28 Feb 2020 16:41:03 -0800 Subject: [PATCH 1229/2163] bpo-39789: Update Windows release build machines to VS 2019 (GH-18695) Also fixes some potential Nuget build issues. (cherry picked from commit 03153dd1459fab94f294a118ed1525e34d58601a) Co-authored-by: Steve Dower --- .azure-pipelines/windows-release/stage-build.yml | 8 ++++---- .../windows-release/stage-layout-embed.yml | 2 +- .azure-pipelines/windows-release/stage-layout-full.yml | 2 +- .azure-pipelines/windows-release/stage-layout-msix.yml | 2 +- .../windows-release/stage-layout-nuget.yml | 2 +- .azure-pipelines/windows-release/stage-msi.yml | 2 +- .azure-pipelines/windows-release/stage-pack-msix.yml | 2 +- .azure-pipelines/windows-release/stage-pack-nuget.yml | 2 +- .../windows-release/stage-publish-nugetorg.yml | 2 +- .../windows-release/stage-publish-pythonorg.yml | 2 +- .../windows-release/stage-publish-store.yml | 2 +- .azure-pipelines/windows-release/stage-sign.yml | 2 +- .azure-pipelines/windows-release/stage-test-embed.yml | 2 +- .azure-pipelines/windows-release/stage-test-msi.yml | 2 +- .azure-pipelines/windows-release/stage-test-nuget.yml | 2 +- .../Windows/2020-02-28-22-46-09.bpo-39789.67XRoP.rst | 1 + PC/layout/support/nuspec.py | 10 +++++----- Tools/nuget/make_pkg.proj | 2 +- 18 files changed, 25 insertions(+), 24 deletions(-) create mode 100644 Misc/NEWS.d/next/Windows/2020-02-28-22-46-09.bpo-39789.67XRoP.rst diff --git a/.azure-pipelines/windows-release/stage-build.yml b/.azure-pipelines/windows-release/stage-build.yml index 9391a91e30b5e6..69f3b1e16451ec 100644 --- a/.azure-pipelines/windows-release/stage-build.yml +++ b/.azure-pipelines/windows-release/stage-build.yml @@ -3,7 +3,7 @@ jobs: displayName: Docs build pool: name: 'Windows Release' - #vmName: win2016-vs2017 + #vmImage: windows-2019 workspace: clean: all @@ -45,7 +45,7 @@ jobs: displayName: Python build pool: - vmName: win2016-vs2017 + vmImage: windows-2019 workspace: clean: all @@ -91,7 +91,7 @@ jobs: condition: and(succeeded(), ne(variables['DoPGO'], 'true')) pool: - vmName: win2016-vs2017 + vmImage: windows-2019 workspace: clean: all @@ -141,7 +141,7 @@ jobs: displayName: Publish Tcl/Tk Library pool: - vmName: windows-latest + vmImage: windows-2019 workspace: clean: all diff --git a/.azure-pipelines/windows-release/stage-layout-embed.yml b/.azure-pipelines/windows-release/stage-layout-embed.yml index 3306e1cbc49d98..dbccdead143b21 100644 --- a/.azure-pipelines/windows-release/stage-layout-embed.yml +++ b/.azure-pipelines/windows-release/stage-layout-embed.yml @@ -4,7 +4,7 @@ jobs: condition: and(succeeded(), eq(variables['DoEmbed'], 'true')) pool: - vmName: win2016-vs2017 + vmImage: windows-2019 workspace: clean: all diff --git a/.azure-pipelines/windows-release/stage-layout-full.yml b/.azure-pipelines/windows-release/stage-layout-full.yml index 78bc1b3975e93d..8fc8da3e52fe03 100644 --- a/.azure-pipelines/windows-release/stage-layout-full.yml +++ b/.azure-pipelines/windows-release/stage-layout-full.yml @@ -4,7 +4,7 @@ jobs: condition: and(succeeded(), eq(variables['DoLayout'], 'true')) pool: - vmName: win2016-vs2017 + vmImage: windows-2019 workspace: clean: all diff --git a/.azure-pipelines/windows-release/stage-layout-msix.yml b/.azure-pipelines/windows-release/stage-layout-msix.yml index 60a5c9ea5435c8..def4f7d3c6bee5 100644 --- a/.azure-pipelines/windows-release/stage-layout-msix.yml +++ b/.azure-pipelines/windows-release/stage-layout-msix.yml @@ -3,7 +3,7 @@ jobs: displayName: Make MSIX layout pool: - vmName: win2016-vs2017 + vmImage: windows-2019 workspace: clean: all diff --git a/.azure-pipelines/windows-release/stage-layout-nuget.yml b/.azure-pipelines/windows-release/stage-layout-nuget.yml index 7e20f89530349c..41cdff850e83be 100644 --- a/.azure-pipelines/windows-release/stage-layout-nuget.yml +++ b/.azure-pipelines/windows-release/stage-layout-nuget.yml @@ -4,7 +4,7 @@ jobs: condition: and(succeeded(), eq(variables['DoNuget'], 'true')) pool: - vmName: win2016-vs2017 + vmImage: windows-2019 workspace: clean: all diff --git a/.azure-pipelines/windows-release/stage-msi.yml b/.azure-pipelines/windows-release/stage-msi.yml index 7afc816a0c6e9c..9b965b09c14748 100644 --- a/.azure-pipelines/windows-release/stage-msi.yml +++ b/.azure-pipelines/windows-release/stage-msi.yml @@ -4,7 +4,7 @@ jobs: condition: and(succeeded(), not(variables['SigningCertificate'])) pool: - vmName: win2016-vs2017 + vmImage: windows-2019 variables: ReleaseUri: http://www.python.org/{arch} diff --git a/.azure-pipelines/windows-release/stage-pack-msix.yml b/.azure-pipelines/windows-release/stage-pack-msix.yml index f17ba9628e21b0..07e343a0b4e0c7 100644 --- a/.azure-pipelines/windows-release/stage-pack-msix.yml +++ b/.azure-pipelines/windows-release/stage-pack-msix.yml @@ -3,7 +3,7 @@ jobs: displayName: Pack MSIX bundles pool: - vmName: win2016-vs2017 + vmImage: windows-2019 workspace: clean: all diff --git a/.azure-pipelines/windows-release/stage-pack-nuget.yml b/.azure-pipelines/windows-release/stage-pack-nuget.yml index 34619fc5fdc318..b100364820d95b 100644 --- a/.azure-pipelines/windows-release/stage-pack-nuget.yml +++ b/.azure-pipelines/windows-release/stage-pack-nuget.yml @@ -4,7 +4,7 @@ jobs: condition: and(succeeded(), eq(variables['DoNuget'], 'true')) pool: - vmName: win2016-vs2017 + vmImage: windows-2019 workspace: clean: all diff --git a/.azure-pipelines/windows-release/stage-publish-nugetorg.yml b/.azure-pipelines/windows-release/stage-publish-nugetorg.yml index b78bd493a0fd1b..d5edf44ef5c2ec 100644 --- a/.azure-pipelines/windows-release/stage-publish-nugetorg.yml +++ b/.azure-pipelines/windows-release/stage-publish-nugetorg.yml @@ -4,7 +4,7 @@ jobs: condition: and(succeeded(), eq(variables['DoNuget'], 'true')) pool: - vmName: win2016-vs2017 + vmImage: windows-2019 workspace: clean: all diff --git a/.azure-pipelines/windows-release/stage-publish-pythonorg.yml b/.azure-pipelines/windows-release/stage-publish-pythonorg.yml index 8c95f1b950cd75..b4710d8c248e23 100644 --- a/.azure-pipelines/windows-release/stage-publish-pythonorg.yml +++ b/.azure-pipelines/windows-release/stage-publish-pythonorg.yml @@ -4,7 +4,7 @@ jobs: condition: and(succeeded(), and(eq(variables['DoMSI'], 'true'), eq(variables['DoEmbed'], 'true'))) pool: - #vmName: win2016-vs2017 + #vmImage: windows-2019 name: 'Windows Release' workspace: diff --git a/.azure-pipelines/windows-release/stage-publish-store.yml b/.azure-pipelines/windows-release/stage-publish-store.yml index b22147b1ab45e7..e0512b95f27da8 100644 --- a/.azure-pipelines/windows-release/stage-publish-store.yml +++ b/.azure-pipelines/windows-release/stage-publish-store.yml @@ -4,7 +4,7 @@ jobs: condition: and(succeeded(), eq(variables['DoMSIX'], 'true')) pool: - vmName: win2016-vs2017 + vmImage: windows-2019 workspace: clean: all diff --git a/.azure-pipelines/windows-release/stage-sign.yml b/.azure-pipelines/windows-release/stage-sign.yml index a0adc0581229d9..4d757ae8fca032 100644 --- a/.azure-pipelines/windows-release/stage-sign.yml +++ b/.azure-pipelines/windows-release/stage-sign.yml @@ -114,7 +114,7 @@ jobs: condition: and(succeeded(), not(variables['SigningCertificate'])) pool: - vmName: win2016-vs2017 + vmImage: windows-2019 steps: - checkout: none diff --git a/.azure-pipelines/windows-release/stage-test-embed.yml b/.azure-pipelines/windows-release/stage-test-embed.yml index b33176266a20da..d99bd74722bacb 100644 --- a/.azure-pipelines/windows-release/stage-test-embed.yml +++ b/.azure-pipelines/windows-release/stage-test-embed.yml @@ -4,7 +4,7 @@ jobs: condition: and(succeeded(), eq(variables['DoEmbed'], 'true')) pool: - vmName: win2016-vs2017 + vmImage: windows-2019 workspace: clean: all diff --git a/.azure-pipelines/windows-release/stage-test-msi.yml b/.azure-pipelines/windows-release/stage-test-msi.yml index 27b0c96987a7a0..21e38c39590f70 100644 --- a/.azure-pipelines/windows-release/stage-test-msi.yml +++ b/.azure-pipelines/windows-release/stage-test-msi.yml @@ -3,7 +3,7 @@ jobs: displayName: Test MSI pool: - vmName: win2016-vs2017 + vmImage: windows-2019 workspace: clean: all diff --git a/.azure-pipelines/windows-release/stage-test-nuget.yml b/.azure-pipelines/windows-release/stage-test-nuget.yml index 1f8b601d0d023b..94d815e95226ef 100644 --- a/.azure-pipelines/windows-release/stage-test-nuget.yml +++ b/.azure-pipelines/windows-release/stage-test-nuget.yml @@ -4,7 +4,7 @@ jobs: condition: and(succeeded(), eq(variables['DoNuget'], 'true')) pool: - vmName: win2016-vs2017 + vmImage: windows-2019 workspace: clean: all diff --git a/Misc/NEWS.d/next/Windows/2020-02-28-22-46-09.bpo-39789.67XRoP.rst b/Misc/NEWS.d/next/Windows/2020-02-28-22-46-09.bpo-39789.67XRoP.rst new file mode 100644 index 00000000000000..077b0afcba3c10 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-02-28-22-46-09.bpo-39789.67XRoP.rst @@ -0,0 +1 @@ +Update Windows release build machines to Visual Studio 2019 (MSVC 14.2). diff --git a/PC/layout/support/nuspec.py b/PC/layout/support/nuspec.py index 9c6a9a91595098..dbcb713ef9d0c0 100644 --- a/PC/layout/support/nuspec.py +++ b/PC/layout/support/nuspec.py @@ -14,7 +14,7 @@ NUSPEC_DATA = { "PYTHON_TAG": VER_DOT, "PYTHON_VERSION": os.getenv("PYTHON_NUSPEC_VERSION"), - "FILELIST": r' ', + "FILELIST": r' ', "GIT": sys._git, } @@ -31,7 +31,7 @@ VER_DOT, VER_MICRO, "-" if VER_SUFFIX else "", VER_SUFFIX ) -FILELIST_WITH_PROPS = r""" +FILELIST_WITH_PROPS = r""" """ NUSPEC_TEMPLATE = r""" @@ -44,13 +44,13 @@ tools\LICENSE.txt https://www.python.org/ Installs {PYTHON_BITNESS} Python for use in build scenarios. - images\logox128.png + images\python.png https://www.python.org/static/favicon.ico python - + {FILELIST} @@ -73,6 +73,6 @@ def get_nuspec_layout(ns): data[k] = v if ns.include_all or ns.include_props: data["FILELIST"] = FILELIST_WITH_PROPS - data["LOGO"] = ns.source / "PC" / "icons" / "logox128.png" nuspec = NUSPEC_TEMPLATE.format_map(data) yield "python.nuspec", ("python.nuspec", nuspec.encode("utf-8")) + yield "python.png", ns.source / "PC" / "icons" / "logox128.png" diff --git a/Tools/nuget/make_pkg.proj b/Tools/nuget/make_pkg.proj index b387b8eef5423c..710ef3dcb5c01d 100644 --- a/Tools/nuget/make_pkg.proj +++ b/Tools/nuget/make_pkg.proj @@ -33,7 +33,7 @@ "$(IntermediateOutputPath)pkg\pip.exe" -B -m pip install -U $(Packages) - "$(Nuget)" pack "$(MSBuildThisFileDirectory)\$(OutputName).nuspec" -BasePath "$(IntermediateOutputPath)pkg" + "$(Nuget)" pack "$(IntermediateOutputPath)pkg\python.nuspec" -BasePath "$(IntermediateOutputPath)pkg" "$(Nuget)" pack "$(MSBuildThisFileDirectory)\$(OutputName).symbols.nuspec" -BasePath "$(BuildPath.TrimEnd(`\`))" $(NugetArguments) -OutputDirectory "$(OutputPath.Trim(`\`))" $(NugetArguments) -Version "$(NuspecVersion)" From e4686b79798f7a492dcbaa62cf51f4d07fd5ae78 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 29 Feb 2020 13:05:23 -0800 Subject: [PATCH 1230/2163] bpo-39548: Fix handling of 'WWW-Authenticate' header for Digest Auth (GH-18338) * bpo-39548: Fix handling of 'WWW-Authenticate' header for Digest authentication - The 'qop' value in the 'WWW-Authenticate' header is optional. The presence of 'qop' in the header should be checked before its value is parsed with 'split'. Signed-off-by: Stephen Balousek * bpo-39548: Fix handling of 'WWW-Authenticate' header for Digest authentication - Add NEWS item Signed-off-by: Stephen Balousek * Update Misc/NEWS.d/next/Library/2020-02-06-05-33-52.bpo-39548.DF4FFe.rst Co-Authored-By: Brandt Bucher Co-authored-by: Brandt Bucher (cherry picked from commit 5e260e0fde211829fcb67060cfd602f4b679f802) Co-authored-by: Stephen Balousek --- Lib/urllib/request.py | 6 +++--- .../next/Library/2020-02-06-05-33-52.bpo-39548.DF4FFe.rst | 2 ++ 2 files changed, 5 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-02-06-05-33-52.bpo-39548.DF4FFe.rst diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py index 6f6577bf1d9029..fd91b9d1d0beb2 100644 --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -1146,7 +1146,9 @@ def get_authorization(self, req, chal): req.selector) # NOTE: As per RFC 2617, when server sends "auth,auth-int", the client could use either `auth` # or `auth-int` to the response back. we use `auth` to send the response back. - if 'auth' in qop.split(','): + if qop is None: + respdig = KD(H(A1), "%s:%s" % (nonce, H(A2))) + elif 'auth' in qop.split(','): if nonce == self.last_nonce: self.nonce_count += 1 else: @@ -1156,8 +1158,6 @@ def get_authorization(self, req, chal): cnonce = self.get_cnonce(nonce) noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, 'auth', H(A2)) respdig = KD(H(A1), noncebit) - elif qop is None: - respdig = KD(H(A1), "%s:%s" % (nonce, H(A2))) else: # XXX handle auth-int. raise URLError("qop '%s' is not supported." % qop) diff --git a/Misc/NEWS.d/next/Library/2020-02-06-05-33-52.bpo-39548.DF4FFe.rst b/Misc/NEWS.d/next/Library/2020-02-06-05-33-52.bpo-39548.DF4FFe.rst new file mode 100644 index 00000000000000..4cf32487b1f7a0 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-06-05-33-52.bpo-39548.DF4FFe.rst @@ -0,0 +1,2 @@ +Fix handling of header in :class:`urllib.request.AbstractDigestAuthHandler` when the optional ``qop`` parameter +is not present. From 4d7012410cf4f91cbca4c406f4747289c2802333 Mon Sep 17 00:00:00 2001 From: Stefan Krah Date: Sat, 29 Feb 2020 22:16:32 +0100 Subject: [PATCH 1231/2163] [3.8] bpo-39794: Add --without-decimal-contextvar (GH-18702) (cherry picked from commit 815280eb160af637e1347213659f9236adf78f80) --- Doc/library/decimal.rst | 15 +- Lib/_pydecimal.py | 8 +- Lib/test/test_asyncio/test_context.py | 1 + .../2020-02-29-19-17-39.bpo-39794.7VjatS.rst | 2 + Modules/_decimal/_decimal.c | 169 ++++++++++++++++-- .../_decimal/tests/runall-memorydebugger.sh | 36 ++-- PC/pyconfig.h | 4 + aclocal.m4 | 4 +- configure | 26 +++ configure.ac | 15 ++ pyconfig.h.in | 4 + 11 files changed, 248 insertions(+), 36 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-02-29-19-17-39.bpo-39794.7VjatS.rst diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index 4e640cc695990f..c8a4bcfdc79a05 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -1475,9 +1475,18 @@ are also included in the pure Python version for compatibility. .. data:: HAVE_THREADS - The default value is ``True``. If Python is compiled without threads, the - C version automatically disables the expensive thread local context - machinery. In this case, the value is ``False``. + The value is ``True``. Deprecated, because Python now always has threads. + +.. deprecated:: 3.9 + +.. data:: HAVE_CONTEXTVAR + + The default value is ``True``. If Python is compiled ``--without-decimal-contextvar``, + the C version uses a thread-local rather than a coroutine-local context and the value + is ``False``. This is slightly faster in some nested context scenarios. + +.. versionadded:: 3.9, backported to 3.7 and 3.8 + Rounding modes -------------- diff --git a/Lib/_pydecimal.py b/Lib/_pydecimal.py index c14d8ca86a1181..ab989e5206a9e9 100644 --- a/Lib/_pydecimal.py +++ b/Lib/_pydecimal.py @@ -140,8 +140,11 @@ # Limits for the C version for compatibility 'MAX_PREC', 'MAX_EMAX', 'MIN_EMIN', 'MIN_ETINY', - # C version: compile time choice that enables the thread local context - 'HAVE_THREADS' + # C version: compile time choice that enables the thread local context (deprecated, now always true) + 'HAVE_THREADS', + + # C version: compile time choice that enables the coroutine local context + 'HAVE_CONTEXTVAR' ] __xname__ = __name__ # sys.modules lookup (--without-threads) @@ -172,6 +175,7 @@ # Compatibility with the C version HAVE_THREADS = True +HAVE_CONTEXTVAR = True if sys.maxsize == 2**63-1: MAX_PREC = 999999999999999999 MAX_EMAX = 999999999999999999 diff --git a/Lib/test/test_asyncio/test_context.py b/Lib/test/test_asyncio/test_context.py index c309faa90062e1..63b1eb320ce16b 100644 --- a/Lib/test/test_asyncio/test_context.py +++ b/Lib/test/test_asyncio/test_context.py @@ -7,6 +7,7 @@ def tearDownModule(): asyncio.set_event_loop_policy(None) +@unittest.skipUnless(decimal.HAVE_CONTEXTVAR, "decimal is built with a thread-local context") class DecimalContextTest(unittest.TestCase): def test_asyncio_task_decimal_context(self): diff --git a/Misc/NEWS.d/next/Library/2020-02-29-19-17-39.bpo-39794.7VjatS.rst b/Misc/NEWS.d/next/Library/2020-02-29-19-17-39.bpo-39794.7VjatS.rst new file mode 100644 index 00000000000000..b2a4726068af95 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-29-19-17-39.bpo-39794.7VjatS.rst @@ -0,0 +1,2 @@ +Add --without-decimal-contextvar build option. This enables a thread-local +rather than a coroutine local context. diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c index e2ac19800315ce..4358c4d686758d 100644 --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -122,7 +122,14 @@ incr_false(void) } +#ifndef WITH_DECIMAL_CONTEXTVAR +/* Key for thread state dictionary */ +static PyObject *tls_context_key = NULL; +/* Invariant: NULL or the most recently accessed thread local context */ +static PyDecContextObject *cached_context = NULL; +#else static PyObject *current_context_var; +#endif /* Template for creating new thread contexts, calling Context() without * arguments and initializing the module_context on first access. */ @@ -1217,6 +1224,12 @@ context_new(PyTypeObject *type, PyObject *args UNUSED, PyObject *kwds UNUSED) static void context_dealloc(PyDecContextObject *self) { +#ifndef WITH_DECIMAL_CONTEXTVAR + if (self == cached_context) { + cached_context = NULL; + } +#endif + Py_XDECREF(self->traps); Py_XDECREF(self->flags); Py_TYPE(self)->tp_free(self); @@ -1491,6 +1504,134 @@ static PyGetSetDef context_getsets [] = * operation. */ +#ifndef WITH_DECIMAL_CONTEXTVAR +/* Get the context from the thread state dictionary. */ +static PyObject * +current_context_from_dict(void) +{ + PyObject *dict; + PyObject *tl_context; + PyThreadState *tstate; + + dict = PyThreadState_GetDict(); + if (dict == NULL) { + PyErr_SetString(PyExc_RuntimeError, + "cannot get thread state"); + return NULL; + } + + tl_context = PyDict_GetItemWithError(dict, tls_context_key); + if (tl_context != NULL) { + /* We already have a thread local context. */ + CONTEXT_CHECK(tl_context); + } + else { + if (PyErr_Occurred()) { + return NULL; + } + + /* Set up a new thread local context. */ + tl_context = context_copy(default_context_template, NULL); + if (tl_context == NULL) { + return NULL; + } + CTX(tl_context)->status = 0; + + if (PyDict_SetItem(dict, tls_context_key, tl_context) < 0) { + Py_DECREF(tl_context); + return NULL; + } + Py_DECREF(tl_context); + } + + /* Cache the context of the current thread, assuming that it + * will be accessed several times before a thread switch. */ + tstate = PyThreadState_GET(); + if (tstate) { + cached_context = (PyDecContextObject *)tl_context; + cached_context->tstate = tstate; + } + + /* Borrowed reference with refcount==1 */ + return tl_context; +} + +/* Return borrowed reference to thread local context. */ +static PyObject * +current_context(void) +{ + PyThreadState *tstate; + + tstate = PyThreadState_GET(); + if (cached_context && cached_context->tstate == tstate) { + return (PyObject *)cached_context; + } + + return current_context_from_dict(); +} + +/* ctxobj := borrowed reference to the current context */ +#define CURRENT_CONTEXT(ctxobj) \ + ctxobj = current_context(); \ + if (ctxobj == NULL) { \ + return NULL; \ + } + +/* Return a new reference to the current context */ +static PyObject * +PyDec_GetCurrentContext(PyObject *self UNUSED, PyObject *args UNUSED) +{ + PyObject *context; + + context = current_context(); + if (context == NULL) { + return NULL; + } + + Py_INCREF(context); + return context; +} + +/* Set the thread local context to a new context, decrement old reference */ +static PyObject * +PyDec_SetCurrentContext(PyObject *self UNUSED, PyObject *v) +{ + PyObject *dict; + + CONTEXT_CHECK(v); + + dict = PyThreadState_GetDict(); + if (dict == NULL) { + PyErr_SetString(PyExc_RuntimeError, + "cannot get thread state"); + return NULL; + } + + /* If the new context is one of the templates, make a copy. + * This is the current behavior of decimal.py. */ + if (v == default_context_template || + v == basic_context_template || + v == extended_context_template) { + v = context_copy(v, NULL); + if (v == NULL) { + return NULL; + } + CTX(v)->status = 0; + } + else { + Py_INCREF(v); + } + + cached_context = NULL; + if (PyDict_SetItem(dict, tls_context_key, v) < 0) { + Py_DECREF(v); + return NULL; + } + + Py_DECREF(v); + Py_RETURN_NONE; +} +#else static PyObject * init_current_context(void) { @@ -1570,6 +1711,7 @@ PyDec_SetCurrentContext(PyObject *self UNUSED, PyObject *v) Py_RETURN_NONE; } +#endif /* Context manager object for the 'with' statement. The manager * owns one reference to the global (outer) context and one @@ -4388,15 +4530,8 @@ _dec_hash(PyDecObject *v) mpd_ssize_t exp; uint32_t status = 0; mpd_context_t maxctx; - PyObject *context; - context = current_context(); - if (context == NULL) { - return -1; - } - Py_DECREF(context); - if (mpd_isspecial(MPD(v))) { if (mpd_issnan(MPD(v))) { PyErr_SetString(PyExc_TypeError, @@ -5538,11 +5673,6 @@ PyInit__decimal(void) mpd_free = PyMem_Free; mpd_setminalloc(_Py_DEC_MINALLOC); - /* Init context variable */ - current_context_var = PyContextVar_New("decimal_context", NULL); - if (current_context_var == NULL) { - goto error; - } /* Init external C-API functions */ _py_long_multiply = PyLong_Type.tp_as_number->nb_multiply; @@ -5714,6 +5844,15 @@ PyInit__decimal(void) CHECK_INT(PyModule_AddObject(m, "DefaultContext", default_context_template)); +#ifndef WITH_DECIMAL_CONTEXTVAR + ASSIGN_PTR(tls_context_key, PyUnicode_FromString("___DECIMAL_CTX__")); + Py_INCREF(Py_False); + CHECK_INT(PyModule_AddObject(m, "HAVE_CONTEXTVAR", Py_False)); +#else + ASSIGN_PTR(current_context_var, PyContextVar_New("decimal_context", NULL)); + Py_INCREF(Py_True); + CHECK_INT(PyModule_AddObject(m, "HAVE_CONTEXTVAR", Py_True)); +#endif Py_INCREF(Py_True); CHECK_INT(PyModule_AddObject(m, "HAVE_THREADS", Py_True)); @@ -5773,9 +5912,13 @@ PyInit__decimal(void) Py_CLEAR(SignalTuple); /* GCOV_NOT_REACHED */ Py_CLEAR(DecimalTuple); /* GCOV_NOT_REACHED */ Py_CLEAR(default_context_template); /* GCOV_NOT_REACHED */ +#ifndef WITH_DECIMAL_CONTEXTVAR + Py_CLEAR(tls_context_key); /* GCOV_NOT_REACHED */ +#else + Py_CLEAR(current_context_var); /* GCOV_NOT_REACHED */ +#endif Py_CLEAR(basic_context_template); /* GCOV_NOT_REACHED */ Py_CLEAR(extended_context_template); /* GCOV_NOT_REACHED */ - Py_CLEAR(current_context_var); /* GCOV_NOT_REACHED */ Py_CLEAR(m); /* GCOV_NOT_REACHED */ return NULL; /* GCOV_NOT_REACHED */ diff --git a/Modules/_decimal/tests/runall-memorydebugger.sh b/Modules/_decimal/tests/runall-memorydebugger.sh index 29b7723dd50c05..7f3e527461871a 100755 --- a/Modules/_decimal/tests/runall-memorydebugger.sh +++ b/Modules/_decimal/tests/runall-memorydebugger.sh @@ -1,8 +1,8 @@ #!/bin/sh # -# Purpose: test all machine configurations, pydebug, refleaks, release build -# and release build with valgrind. +# Purpose: test with and without contextvar, all machine configurations, pydebug, +# refleaks, release build and release build with valgrind. # # Synopsis: ./runall-memorydebugger.sh [--all-configs64 | --all-configs32] # @@ -57,7 +57,8 @@ print_config () cd .. # test_decimal: refleak, regular and Valgrind tests -for config in $CONFIGS; do +for args in "--without-decimal-contextvar" ""; do + for config in $CONFIGS; do unset PYTHON_DECIMAL_WITH_MACHINE libmpdec_config=$config @@ -69,12 +70,12 @@ for config in $CONFIGS; do fi ############ refleak tests ########### - print_config "refleak tests: config=$config" + print_config "refleak tests: config=$config" $args printf "\nbuilding python ...\n\n" cd ../../ $GMAKE distclean > /dev/null 2>&1 - ./configure CFLAGS="$ADD_CFLAGS" LDFLAGS="$ADD_LDFLAGS" --with-pydebug > /dev/null 2>&1 + ./configure CFLAGS="$ADD_CFLAGS" LDFLAGS="$ADD_LDFLAGS" --with-pydebug $args > /dev/null 2>&1 $GMAKE | grep _decimal printf "\n\n# ======================== refleak tests ===========================\n\n" @@ -82,11 +83,11 @@ for config in $CONFIGS; do ############ regular tests ########### - print_config "regular tests: config=$config" + print_config "regular tests: config=$config" $args printf "\nbuilding python ...\n\n" $GMAKE distclean > /dev/null 2>&1 - ./configure CFLAGS="$ADD_CFLAGS" LDFLAGS="$ADD_LDFLAGS" > /dev/null 2>&1 + ./configure CFLAGS="$ADD_CFLAGS" LDFLAGS="$ADD_LDFLAGS" $args > /dev/null 2>&1 $GMAKE | grep _decimal printf "\n\n# ======================== regular tests ===========================\n\n" @@ -103,21 +104,23 @@ for config in $CONFIGS; do esac esac - print_config "valgrind tests: config=$config" + print_config "valgrind tests: config=$config" $args printf "\nbuilding python ...\n\n" $GMAKE distclean > /dev/null 2>&1 - ./configure CFLAGS="$ADD_CFLAGS" LDFLAGS="$ADD_LDFLAGS" --without-pymalloc > /dev/null 2>&1 + ./configure CFLAGS="$ADD_CFLAGS" LDFLAGS="$ADD_LDFLAGS" --without-pymalloc $args > /dev/null 2>&1 $GMAKE | grep _decimal printf "\n\n# ======================== valgrind tests ===========================\n\n" $valgrind ./python -m test -uall test_decimal cd Modules/_decimal + done done # deccheck cd ../../ -for config in $CONFIGS; do +for args in "--without-decimal-contextvar" ""; do + for config in $CONFIGS; do unset PYTHON_DECIMAL_WITH_MACHINE if [ X"$config" != X"auto" ]; then @@ -126,22 +129,22 @@ for config in $CONFIGS; do fi ############ debug ############ - print_config "deccheck: config=$config --with-pydebug" + print_config "deccheck: config=$config --with-pydebug" $args printf "\nbuilding python ...\n\n" $GMAKE distclean > /dev/null 2>&1 - ./configure CFLAGS="$ADD_CFLAGS" LDFLAGS="$ADD_LDFLAGS" --with-pydebug > /dev/null 2>&1 + ./configure CFLAGS="$ADD_CFLAGS" LDFLAGS="$ADD_LDFLAGS" --with-pydebug $args > /dev/null 2>&1 $GMAKE | grep _decimal printf "\n\n# ========================== debug ===========================\n\n" ./python Modules/_decimal/tests/deccheck.py ########### regular ########### - print_config "deccheck: config=$config " + print_config "deccheck: config=$config" $args printf "\nbuilding python ...\n\n" $GMAKE distclean > /dev/null 2>&1 - ./configure CFLAGS="$ADD_CFLAGS" LDFLAGS="$ADD_LDFLAGS" > /dev/null 2>&1 + ./configure CFLAGS="$ADD_CFLAGS" LDFLAGS="$ADD_LDFLAGS" $args > /dev/null 2>&1 $GMAKE | grep _decimal printf "\n\n# ======================== regular ===========================\n\n" @@ -157,15 +160,16 @@ for config in $CONFIGS; do esac esac - print_config "valgrind deccheck: config=$config " + print_config "valgrind deccheck: config=$config" $args printf "\nbuilding python ...\n\n" $GMAKE distclean > /dev/null 2>&1 - ./configure CFLAGS="$ADD_CFLAGS" LDFLAGS="$ADD_LDFLAGS" --without-pymalloc > /dev/null 2>&1 + ./configure CFLAGS="$ADD_CFLAGS" LDFLAGS="$ADD_LDFLAGS" --without-pymalloc $args > /dev/null 2>&1 $GMAKE | grep _decimal printf "\n\n# ======================== valgrind ==========================\n\n" $valgrind ./python Modules/_decimal/tests/deccheck.py + done done diff --git a/PC/pyconfig.h b/PC/pyconfig.h index b40e24f438ef60..b6b8d445869bcf 100644 --- a/PC/pyconfig.h +++ b/PC/pyconfig.h @@ -470,6 +470,10 @@ Py_NO_ENABLE_SHARED to find out. Also support MS_NO_COREDLL for b/w compat */ (which you can't on SCO ODT 3.0). */ /* #undef SYS_SELECT_WITH_SYS_TIME */ +/* Define if you want build the _decimal module using a coroutine-local rather + than a thread-local context */ +#define WITH_DECIMAL_CONTEXTVAR 1 + /* Define if you want documentation strings in extension modules */ #define WITH_DOC_STRINGS 1 diff --git a/aclocal.m4 b/aclocal.m4 index 85f00dd5fac7f2..f98db73656d305 100644 --- a/aclocal.m4 +++ b/aclocal.m4 @@ -1,6 +1,6 @@ -# generated automatically by aclocal 1.16.1 -*- Autoconf -*- +# generated automatically by aclocal 1.15 -*- Autoconf -*- -# Copyright (C) 1996-2018 Free Software Foundation, Inc. +# Copyright (C) 1996-2014 Free Software Foundation, Inc. # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, diff --git a/configure b/configure index a979363acffc41..3dfd6cce69e4e8 100755 --- a/configure +++ b/configure @@ -826,6 +826,7 @@ with_libs with_system_expat with_system_ffi with_system_libmpdec +with_decimal_contextvar enable_loadable_sqlite_extensions with_tcltk_includes with_tcltk_libs @@ -1529,6 +1530,9 @@ Optional Packages: --with-system-ffi build _ctypes module using an installed ffi library --with-system-libmpdec build _decimal module using an installed libmpdec library + --with-decimal-contextvar + build _decimal module using a coroutine-local rather + than a thread-local context (default is yes) --with-tcltk-includes='-I...' override search for Tcl and Tk include files --with-tcltk-libs='-L...' @@ -10419,6 +10423,28 @@ fi { $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_system_libmpdec" >&5 $as_echo "$with_system_libmpdec" >&6; } +# Check whether _decimal should use a coroutine-local or thread-local context +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-decimal-contextvar" >&5 +$as_echo_n "checking for --with-decimal-contextvar... " >&6; } + +# Check whether --with-decimal_contextvar was given. +if test "${with_decimal_contextvar+set}" = set; then : + withval=$with_decimal_contextvar; +else + with_decimal_contextvar="yes" +fi + + +if test "$with_decimal_contextvar" != "no" +then + +$as_echo "#define WITH_DECIMAL_CONTEXTVAR 1" >>confdefs.h + +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_decimal_contextvar" >&5 +$as_echo "$with_decimal_contextvar" >&6; } + # Check for support for loadable sqlite extensions { $as_echo "$as_me:${as_lineno-$LINENO}: checking for --enable-loadable-sqlite-extensions" >&5 $as_echo_n "checking for --enable-loadable-sqlite-extensions... " >&6; } diff --git a/configure.ac b/configure.ac index e57ef7c38bf7f3..d8de9d49439c31 100644 --- a/configure.ac +++ b/configure.ac @@ -3004,6 +3004,21 @@ AC_ARG_WITH(system_libmpdec, AC_MSG_RESULT($with_system_libmpdec) +# Check whether _decimal should use a coroutine-local or thread-local context +AC_MSG_CHECKING(for --with-decimal-contextvar) +AC_ARG_WITH(decimal_contextvar, + AS_HELP_STRING([--with-decimal-contextvar], [build _decimal module using a coroutine-local rather than a thread-local context (default is yes)]), + [], + [with_decimal_contextvar="yes"]) + +if test "$with_decimal_contextvar" != "no" +then + AC_DEFINE(WITH_DECIMAL_CONTEXTVAR, 1, + [Define if you want build the _decimal module using a coroutine-local rather than a thread-local context]) +fi + +AC_MSG_RESULT($with_decimal_contextvar) + # Check for support for loadable sqlite extensions AC_MSG_CHECKING(for --enable-loadable-sqlite-extensions) AC_ARG_ENABLE(loadable-sqlite-extensions, diff --git a/pyconfig.h.in b/pyconfig.h.in index aaa7b72cac8840..4263a712c3278f 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -1518,6 +1518,10 @@ /* Define if WINDOW in curses.h offers a field _flags. */ #undef WINDOW_HAS_FLAGS +/* Define if you want build the _decimal module using a coroutine-local rather + than a thread-local context */ +#undef WITH_DECIMAL_CONTEXTVAR + /* Define if you want documentation strings in extension modules */ #undef WITH_DOC_STRINGS From 70d7a62c7c8fd6baabf1e13c33773db79db7a9f4 Mon Sep 17 00:00:00 2001 From: Stefan Krah Date: Sat, 29 Feb 2020 22:42:06 +0100 Subject: [PATCH 1232/2163] [3.8] Fix syntax (GH-18716) --- Doc/library/decimal.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index c8a4bcfdc79a05..3dda35fbd35dbd 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -1485,7 +1485,7 @@ are also included in the pure Python version for compatibility. the C version uses a thread-local rather than a coroutine-local context and the value is ``False``. This is slightly faster in some nested context scenarios. -.. versionadded:: 3.9, backported to 3.7 and 3.8 +.. versionadded:: 3.9 backported to 3.7 and 3.8 Rounding modes From fec6681f7ae3e8867bd0446aa993a0b5f23045f9 Mon Sep 17 00:00:00 2001 From: Stefan Krah Date: Sat, 29 Feb 2020 23:08:04 +0100 Subject: [PATCH 1233/2163] [3.8] Explicitly initialize like the surrounding code (GH-18717) --- Modules/_decimal/_decimal.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c index 4358c4d686758d..df7c6e254bcf29 100644 --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -128,7 +128,7 @@ static PyObject *tls_context_key = NULL; /* Invariant: NULL or the most recently accessed thread local context */ static PyDecContextObject *cached_context = NULL; #else -static PyObject *current_context_var; +static PyObject *current_context_var = NULL; #endif /* Template for creating new thread contexts, calling Context() without From ce720d3e0674d6ac6f1b950c20a89be4cfde7853 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith" Date: Sun, 1 Mar 2020 10:42:56 -0800 Subject: [PATCH 1234/2163] bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) (GH-18718) Fix compileall.compile_dir() ddir= behavior on sub-packages. Fixes compileall.compile_dir's ddir parameter and compileall command line flag `-d` to no longer write the wrong pathname to the generated pyc file for submodules beneath the root of the directory tree being compiled. This fixes a regression introduced with Python 3.5. Tests backported from GH 02673352b5db6ca4d3dc804965facbedfe66425d, the implementation is different due to intervening code changes. But still quiet simple. Why was the bug ever introduced? The refactoring to add parallel execution kept the ddir -> dfile computations but discarded the results instead of sending them to compile_file(). This fixes that. Lack of tests meant this went unnoticed. --- Lib/compileall.py | 29 +++++++------ Lib/test/test_compileall.py | 41 +++++++++++++++++++ Lib/test/test_importlib/util.py | 11 +++++ .../2020-02-29-13-20-33.bpo-39769.hJmxu4.rst | 4 ++ 4 files changed, 73 insertions(+), 12 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-02-29-13-20-33.bpo-39769.hJmxu4.rst diff --git a/Lib/compileall.py b/Lib/compileall.py index 49306d9dabbc51..bfac8efc804d06 100644 --- a/Lib/compileall.py +++ b/Lib/compileall.py @@ -41,7 +41,7 @@ def _walk_dir(dir, ddir=None, maxlevels=10, quiet=0): else: dfile = None if not os.path.isdir(fullname): - yield fullname + yield fullname, ddir elif (maxlevels > 0 and name != os.curdir and name != os.pardir and os.path.isdir(fullname) and not os.path.islink(fullname)): yield from _walk_dir(fullname, ddir=dfile, @@ -76,28 +76,33 @@ def compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, from concurrent.futures import ProcessPoolExecutor except ImportError: workers = 1 - files = _walk_dir(dir, quiet=quiet, maxlevels=maxlevels, - ddir=ddir) + files_and_ddirs = _walk_dir(dir, quiet=quiet, maxlevels=maxlevels, + ddir=ddir) success = True if workers != 1 and ProcessPoolExecutor is not None: # If workers == 0, let ProcessPoolExecutor choose workers = workers or None with ProcessPoolExecutor(max_workers=workers) as executor: - results = executor.map(partial(compile_file, - ddir=ddir, force=force, - rx=rx, quiet=quiet, - legacy=legacy, - optimize=optimize, - invalidation_mode=invalidation_mode), - files) + results = executor.map( + partial(_compile_file_tuple, + force=force, rx=rx, quiet=quiet, + legacy=legacy, optimize=optimize, + invalidation_mode=invalidation_mode, + ), + files_and_ddirs) success = min(results, default=True) else: - for file in files: - if not compile_file(file, ddir, force, rx, quiet, + for file, dfile in files_and_ddirs: + if not compile_file(file, dfile, force, rx, quiet, legacy, optimize, invalidation_mode): success = False return success +def _compile_file_tuple(file_and_dfile, **kwargs): + """Needs to be toplevel for ProcessPoolExecutor.""" + file, dfile = file_and_dfile + return compile_file(file, dfile, **kwargs) + def compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=None): diff --git a/Lib/test/test_compileall.py b/Lib/test/test_compileall.py index 04f6e1e049dde1..64f092b53b5000 100644 --- a/Lib/test/test_compileall.py +++ b/Lib/test/test_compileall.py @@ -577,6 +577,47 @@ def test_workers_available_cores(self, compile_dir): self.assertTrue(compile_dir.called) self.assertEqual(compile_dir.call_args[-1]['workers'], 0) + def _test_ddir_only(self, *, ddir, parallel=True): + """Recursive compile_dir ddir must contain package paths; bpo39769.""" + fullpath = ["test", "foo"] + path = self.directory + mods = [] + for subdir in fullpath: + path = os.path.join(path, subdir) + os.mkdir(path) + script_helper.make_script(path, "__init__", "") + mods.append(script_helper.make_script(path, "mod", + "def fn(): 1/0\nfn()\n")) + compileall.compile_dir( + self.directory, quiet=True, ddir=ddir, + workers=2 if parallel else 1) + self.assertTrue(mods) + for mod in mods: + self.assertTrue(mod.startswith(self.directory), mod) + modcode = importlib.util.cache_from_source(mod) + modpath = mod[len(self.directory+os.sep):] + _, _, err = script_helper.assert_python_failure(modcode) + expected_in = os.path.join(ddir, modpath) + mod_code_obj = test.test_importlib.util._get_code_from_pyc(modcode) + self.assertEqual(mod_code_obj.co_filename, expected_in) + self.assertIn(f'"{expected_in}"', os.fsdecode(err)) + + def test_ddir_only_one_worker(self): + """Recursive compile_dir ddir= contains package paths; bpo39769.""" + return self._test_ddir_only(ddir="", parallel=False) + + def test_ddir_multiple_workers(self): + """Recursive compile_dir ddir= contains package paths; bpo39769.""" + return self._test_ddir_only(ddir="", parallel=True) + + def test_ddir_empty_only_one_worker(self): + """Recursive compile_dir ddir='' contains package paths; bpo39769.""" + return self._test_ddir_only(ddir="", parallel=False) + + def test_ddir_empty_multiple_workers(self): + """Recursive compile_dir ddir='' contains package paths; bpo39769.""" + return self._test_ddir_only(ddir="", parallel=True) + class CommmandLineTestsWithSourceEpoch(CommandLineTestsBase, unittest.TestCase, diff --git a/Lib/test/test_importlib/util.py b/Lib/test/test_importlib/util.py index e016ea49119ae9..e6a1476875b5e3 100644 --- a/Lib/test/test_importlib/util.py +++ b/Lib/test/test_importlib/util.py @@ -7,6 +7,7 @@ from importlib import machinery, util, invalidate_caches from importlib.abc import ResourceReader import io +import marshal import os import os.path from pathlib import Path, PurePath @@ -118,6 +119,16 @@ def submodule(parent, name, pkg_dir, content=''): return '{}.{}'.format(parent, name), path +def _get_code_from_pyc(pyc_path): + """Reads a pyc file and returns the unmarshalled code object within. + + No header validation is performed. + """ + with open(pyc_path, 'rb') as pyc_f: + pyc_f.seek(16) + return marshal.load(pyc_f) + + @contextlib.contextmanager def uncache(*names): """Uncache a module from sys.modules. diff --git a/Misc/NEWS.d/next/Library/2020-02-29-13-20-33.bpo-39769.hJmxu4.rst b/Misc/NEWS.d/next/Library/2020-02-29-13-20-33.bpo-39769.hJmxu4.rst new file mode 100644 index 00000000000000..9b564bd10d3b3b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-29-13-20-33.bpo-39769.hJmxu4.rst @@ -0,0 +1,4 @@ +The :func:`compileall.compile_dir` function's *ddir* parameter and the +compileall command line flag `-d` no longer write the wrong pathname to the +generated pyc file for submodules beneath the root of the directory tree +being compiled. This fixes a regression introduced with Python 3.5. From 12b714391e485d0150b343b114999bae4a0d34dd Mon Sep 17 00:00:00 2001 From: Pablo Galindo Date: Mon, 2 Mar 2020 00:08:29 +0000 Subject: [PATCH 1235/2163] [3.8] bpo-39815: add cached_property to all (GH-18726) (GH-18728) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Automerge-Triggered-By: @pablogsal. (cherry picked from commit 217dce9ee6e3cf27a0cedbe1e4a6455776373ec2) Co-authored-by: Hakan Çelik --- Lib/functools.py | 3 ++- Misc/ACKS | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/Lib/functools.py b/Lib/functools.py index b41dea79083fc0..4cde5f590cf290 100644 --- a/Lib/functools.py +++ b/Lib/functools.py @@ -11,7 +11,8 @@ __all__ = ['update_wrapper', 'wraps', 'WRAPPER_ASSIGNMENTS', 'WRAPPER_UPDATES', 'total_ordering', 'cmp_to_key', 'lru_cache', 'reduce', 'partial', - 'partialmethod', 'singledispatch', 'singledispatchmethod'] + 'partialmethod', 'singledispatch', 'singledispatchmethod', + "cached_property"] from abc import get_cache_token from collections import namedtuple diff --git a/Misc/ACKS b/Misc/ACKS index 8fd4df5bf3f199..ef13d55e756bd4 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -362,6 +362,7 @@ Tom Culliton RaĂşl Cumplido Antonio Cuni Brian Curtin +Hakan Celik Paul Dagnelie Lisandro Dalcin Darren Dale From 5f2ade20a556f8c20555c7032436477d6dc86d4f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 1 Mar 2020 22:47:23 -0800 Subject: [PATCH 1236/2163] bpo-39378: partial of PickleState struct should be traversed. (GH-18046) (cherry picked from commit 1f577ce363121d590b51abf5c41d1bcf3d751436) Co-authored-by: Hai Shi --- Modules/_pickle.c | 1 + 1 file changed, 1 insertion(+) diff --git a/Modules/_pickle.c b/Modules/_pickle.c index 55e2734ca2f31d..55affb2c7c4796 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -7961,6 +7961,7 @@ pickle_traverse(PyObject *m, visitproc visit, void *arg) Py_VISIT(st->import_mapping_3to2); Py_VISIT(st->codecs_encode); Py_VISIT(st->getattr); + Py_VISIT(st->partial); return 0; } From a7b8a969eb3daacb1fcb029a8c5fecb5d09c964b Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Mon, 2 Mar 2020 09:54:43 +0200 Subject: [PATCH 1237/2163] [3.8] bpo-38913: Fix segfault in Py_BuildValue("(sGH-O)", ...) if entered with exception raised. (GH-18656). (GH-18732) (cherry picked from commit 28d0bcac8b7e6dbd28311f1283dabb6a4d649fcb) --- .../2020-02-25-20-10-34.bpo-38913.siF1lS.rst | 2 + Modules/_testcapimodule.c | 42 +++++++++++++++++++ Python/modsupport.c | 6 +-- 3 files changed, 47 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/C API/2020-02-25-20-10-34.bpo-38913.siF1lS.rst diff --git a/Misc/NEWS.d/next/C API/2020-02-25-20-10-34.bpo-38913.siF1lS.rst b/Misc/NEWS.d/next/C API/2020-02-25-20-10-34.bpo-38913.siF1lS.rst new file mode 100644 index 00000000000000..0e4d1210315d89 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2020-02-25-20-10-34.bpo-38913.siF1lS.rst @@ -0,0 +1,2 @@ +Fixed segfault in ``Py_BuildValue()`` called with a format containing "#" +and undefined PY_SSIZE_T_CLEAN whwn an exception is set. diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index b54071669cf997..a8f8e683b2013a 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -5063,6 +5063,8 @@ test_write_unraisable_exc(PyObject *self, PyObject *args) } +static PyObject *test_buildvalue_issue38913(PyObject *, PyObject *); + static PyMethodDef TestMethods[] = { {"raise_exception", raise_exception, METH_VARARGS}, {"raise_memoryerror", raise_memoryerror, METH_NOARGS}, @@ -5122,6 +5124,7 @@ static PyMethodDef TestMethods[] = { #endif {"getbuffer_with_null_view", getbuffer_with_null_view, METH_O}, {"test_buildvalue_N", test_buildvalue_N, METH_NOARGS}, + {"test_buildvalue_issue38913", test_buildvalue_issue38913, METH_NOARGS}, {"get_args", get_args, METH_VARARGS}, {"get_kwargs", (PyCFunction)(void(*)(void))get_kwargs, METH_VARARGS|METH_KEYWORDS}, {"getargs_tuple", getargs_tuple, METH_VARARGS}, @@ -6332,3 +6335,42 @@ PyInit__testcapi(void) PyState_AddModule(m, &_testcapimodule); return m; } + + +/* Test the C API exposed when PY_SSIZE_T_CLEAN is not defined */ + +#undef Py_BuildValue +PyAPI_FUNC(PyObject *) Py_BuildValue(const char *, ...); + +static PyObject * +test_buildvalue_issue38913(PyObject *self, PyObject *Py_UNUSED(ignored)) +{ + PyObject *res; + const char str[] = "string"; + const Py_UNICODE unicode[] = L"unicode"; + PyErr_SetNone(PyExc_ZeroDivisionError); + + res = Py_BuildValue("(s#O)", str, 1, Py_None); + assert(res == NULL); + if (!PyErr_ExceptionMatches(PyExc_ZeroDivisionError)) { + return NULL; + } + res = Py_BuildValue("(z#O)", str, 1, Py_None); + assert(res == NULL); + if (!PyErr_ExceptionMatches(PyExc_ZeroDivisionError)) { + return NULL; + } + res = Py_BuildValue("(y#O)", str, 1, Py_None); + assert(res == NULL); + if (!PyErr_ExceptionMatches(PyExc_ZeroDivisionError)) { + return NULL; + } + res = Py_BuildValue("(u#O)", unicode, 1, Py_None); + assert(res == NULL); + if (!PyErr_ExceptionMatches(PyExc_ZeroDivisionError)) { + return NULL; + } + + PyErr_Clear(); + Py_RETURN_NONE; +} diff --git a/Python/modsupport.c b/Python/modsupport.c index 62558221077468..7271af3ac5332f 100644 --- a/Python/modsupport.c +++ b/Python/modsupport.c @@ -343,11 +343,11 @@ do_mkvalue(const char **p_format, va_list *p_va, int flags) if (flags & FLAG_SIZE_T) n = va_arg(*p_va, Py_ssize_t); else { + n = va_arg(*p_va, int); if (PyErr_WarnEx(PyExc_DeprecationWarning, "PY_SSIZE_T_CLEAN will be required for '#' formats", 1)) { return NULL; } - n = va_arg(*p_va, int); } } else @@ -396,11 +396,11 @@ do_mkvalue(const char **p_format, va_list *p_va, int flags) if (flags & FLAG_SIZE_T) n = va_arg(*p_va, Py_ssize_t); else { + n = va_arg(*p_va, int); if (PyErr_WarnEx(PyExc_DeprecationWarning, "PY_SSIZE_T_CLEAN will be required for '#' formats", 1)) { return NULL; } - n = va_arg(*p_va, int); } } else @@ -434,11 +434,11 @@ do_mkvalue(const char **p_format, va_list *p_va, int flags) if (flags & FLAG_SIZE_T) n = va_arg(*p_va, Py_ssize_t); else { + n = va_arg(*p_va, int); if (PyErr_WarnEx(PyExc_DeprecationWarning, "PY_SSIZE_T_CLEAN will be required for '#' formats", 1)) { return NULL; } - n = va_arg(*p_va, int); } } else From 87a4cd5fbebdd0e6166b421d2c3706bc2f2e5a11 Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Mon, 2 Mar 2020 09:58:48 +0200 Subject: [PATCH 1238/2163] bpo-38410: Properly handle PySys_Audit() failures (GH-18658) _PyEval_SetAsyncGenFinalizer() and _PyEval_SetAsyncGenFirstiter() didn't include proper error handling for their PySys_Audit() calls. Co-authored-by: Zackery Spytz --- .../2019-10-09-08-14-25.bpo-38410._YyoMV.rst | 2 + Python/ceval.c | 26 -------- Python/sysmodule.c | 61 +++++++++++++++++-- 3 files changed, 57 insertions(+), 32 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-10-09-08-14-25.bpo-38410._YyoMV.rst diff --git a/Misc/NEWS.d/next/Library/2019-10-09-08-14-25.bpo-38410._YyoMV.rst b/Misc/NEWS.d/next/Library/2019-10-09-08-14-25.bpo-38410._YyoMV.rst new file mode 100644 index 00000000000000..fcfd7936e63ac0 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-10-09-08-14-25.bpo-38410._YyoMV.rst @@ -0,0 +1,2 @@ +Properly handle :func:`sys.audit` failures in +:func:`sys.set_asyncgen_hooks`. Based on patch by Zackery Spytz. diff --git a/Python/ceval.c b/Python/ceval.c index 3306fb9728e8cd..5d78d5a39357dc 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -4742,19 +4742,6 @@ _PyEval_GetCoroutineOriginTrackingDepth(void) return tstate->coroutine_origin_tracking_depth; } -void -_PyEval_SetAsyncGenFirstiter(PyObject *firstiter) -{ - PyThreadState *tstate = _PyThreadState_GET(); - - if (PySys_Audit("sys.set_asyncgen_hook_firstiter", NULL) < 0) { - return; - } - - Py_XINCREF(firstiter); - Py_XSETREF(tstate->async_gen_firstiter, firstiter); -} - PyObject * _PyEval_GetAsyncGenFirstiter(void) { @@ -4762,19 +4749,6 @@ _PyEval_GetAsyncGenFirstiter(void) return tstate->async_gen_firstiter; } -void -_PyEval_SetAsyncGenFinalizer(PyObject *finalizer) -{ - PyThreadState *tstate = _PyThreadState_GET(); - - if (PySys_Audit("sys.set_asyncgen_hook_finalizer", NULL) < 0) { - return; - } - - Py_XINCREF(finalizer); - Py_XSETREF(tstate->async_gen_finalizer, finalizer); -} - PyObject * _PyEval_GetAsyncGenFinalizer(void) { diff --git a/Python/sysmodule.c b/Python/sysmodule.c index da4b6e1a7806b0..b544f2b793ec99 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1178,6 +1178,51 @@ static PyStructSequence_Desc asyncgen_hooks_desc = { 2 }; +static int +set_async_gen_firstiter(PyObject *firstiter) +{ + PyThreadState *tstate = _PyThreadState_GET(); + + if (PySys_Audit("sys.set_asyncgen_hook_firstiter", NULL) < 0) { + return -1; + } + + Py_XINCREF(firstiter); + Py_XSETREF(tstate->async_gen_firstiter, firstiter); + return 0; +} + +void +_PyEval_SetAsyncGenFirstiter(PyObject *firstiter) +{ + if (set_async_gen_firstiter(firstiter) < 0) { + PyErr_WriteUnraisable(NULL); + } +} + +static int +set_async_gen_finalizer(PyObject *finalizer) +{ + PyThreadState *tstate = _PyThreadState_GET(); + + if (PySys_Audit("sys.set_asyncgen_hook_finalizer", NULL) < 0) { + return -1; + } + + Py_XINCREF(finalizer); + Py_XSETREF(tstate->async_gen_finalizer, finalizer); + return 0; +} + +void +_PyEval_SetAsyncGenFinalizer(PyObject *finalizer) +{ + if (set_async_gen_finalizer(finalizer) < 0) { + PyErr_WriteUnraisable(NULL); + } +} + + static PyObject * sys_set_asyncgen_hooks(PyObject *self, PyObject *args, PyObject *kw) { @@ -1198,10 +1243,12 @@ sys_set_asyncgen_hooks(PyObject *self, PyObject *args, PyObject *kw) Py_TYPE(finalizer)->tp_name); return NULL; } - _PyEval_SetAsyncGenFinalizer(finalizer); + if (set_async_gen_finalizer(finalizer) < 0) { + return NULL; + } } - else if (finalizer == Py_None) { - _PyEval_SetAsyncGenFinalizer(NULL); + else if (finalizer == Py_None && set_async_gen_finalizer(NULL) < 0) { + return NULL; } if (firstiter && firstiter != Py_None) { @@ -1211,10 +1258,12 @@ sys_set_asyncgen_hooks(PyObject *self, PyObject *args, PyObject *kw) Py_TYPE(firstiter)->tp_name); return NULL; } - _PyEval_SetAsyncGenFirstiter(firstiter); + if (set_async_gen_firstiter(firstiter) < 0) { + return NULL; + } } - else if (firstiter == Py_None) { - _PyEval_SetAsyncGenFirstiter(NULL); + else if (firstiter == Py_None && set_async_gen_firstiter(NULL) < 0) { + return NULL; } Py_RETURN_NONE; From f28b0c74e54a133cb0287b4297af67d0d7c14d6e Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 2 Mar 2020 00:03:08 -0800 Subject: [PATCH 1239/2163] bpo-38971: Open file in codecs.open() closes if exception raised. (GH-17666) Open issue in the BPO indicated a desire to make the implementation of codecs.open() at parity with io.open(), which implements a try/except to assure file stream gets closed before an exception is raised. (cherry picked from commit 2565edec2c974b2acca03b4cc5025e83f903ddd7) Co-authored-by: Chris A --- Lib/codecs.py | 15 ++++++++++----- Lib/test/test_codecs.py | 9 +++++++++ .../2019-12-20-16-06-28.bpo-38971.fKRYlF.rst | 3 +++ 3 files changed, 22 insertions(+), 5 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-20-16-06-28.bpo-38971.fKRYlF.rst diff --git a/Lib/codecs.py b/Lib/codecs.py index 21c45a7d10a4c9..7f23e9775df804 100644 --- a/Lib/codecs.py +++ b/Lib/codecs.py @@ -905,11 +905,16 @@ def open(filename, mode='r', encoding=None, errors='strict', buffering=-1): file = builtins.open(filename, mode, buffering) if encoding is None: return file - info = lookup(encoding) - srw = StreamReaderWriter(file, info.streamreader, info.streamwriter, errors) - # Add attributes to simplify introspection - srw.encoding = encoding - return srw + + try: + info = lookup(encoding) + srw = StreamReaderWriter(file, info.streamreader, info.streamwriter, errors) + # Add attributes to simplify introspection + srw.encoding = encoding + return srw + except: + file.close() + raise def EncodedFile(file, data_encoding, file_encoding=None, errors='strict'): diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index 8c10e948e8041d..0fd258ca8abf6f 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -1142,6 +1142,7 @@ def test_stream_bare(self): got = ostream.getvalue() self.assertEqual(got, unistring) + class EscapeDecodeTest(unittest.TestCase): def test_empty(self): self.assertEqual(codecs.escape_decode(b""), (b"", 0)) @@ -1713,6 +1714,14 @@ def test_undefined(self): self.assertRaises(UnicodeError, codecs.decode, b'abc', 'undefined', errors) + def test_file_closes_if_lookup_error_raised(self): + mock_open = mock.mock_open() + with mock.patch('builtins.open', mock_open) as file: + with self.assertRaises(LookupError): + codecs.open(support.TESTFN, 'wt', 'invalid-encoding') + + file().close.assert_called() + class StreamReaderTest(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Library/2019-12-20-16-06-28.bpo-38971.fKRYlF.rst b/Misc/NEWS.d/next/Library/2019-12-20-16-06-28.bpo-38971.fKRYlF.rst new file mode 100644 index 00000000000000..9676d72b44abc9 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-20-16-06-28.bpo-38971.fKRYlF.rst @@ -0,0 +1,3 @@ +Open issue in the BPO indicated a desire to make the implementation of +codecs.open() at parity with io.open(), which implements a try/except to +assure file stream gets closed before an exception is raised. \ No newline at end of file From 43932dc1eaf36d75b2ee0420d8be747001315c26 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 2 Mar 2020 05:03:51 -0800 Subject: [PATCH 1240/2163] bpo-39764: Make Task.get_stack accept ag_frame (GH-18669) Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> (cherry picked from commit 4482337decdbd0c6e2150346a68b3616bda664aa) Co-authored-by: Lidi Zheng --- Lib/asyncio/base_tasks.py | 13 ++++++++++--- Lib/test/test_asyncgen.py | 15 +++++++++++++++ .../2020-02-27-18-21-07.bpo-39764.wqPk68.rst | 1 + 3 files changed, 26 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-02-27-18-21-07.bpo-39764.wqPk68.rst diff --git a/Lib/asyncio/base_tasks.py b/Lib/asyncio/base_tasks.py index e2da462fde7400..09bb171a2ce750 100644 --- a/Lib/asyncio/base_tasks.py +++ b/Lib/asyncio/base_tasks.py @@ -24,11 +24,18 @@ def _task_repr_info(task): def _task_get_stack(task, limit): frames = [] - try: - # 'async def' coroutines + if hasattr(task._coro, 'cr_frame'): + # case 1: 'async def' coroutines f = task._coro.cr_frame - except AttributeError: + elif hasattr(task._coro, 'gi_frame'): + # case 2: legacy coroutines f = task._coro.gi_frame + elif hasattr(task._coro, 'ag_frame'): + # case 3: async generators + f = task._coro.ag_frame + else: + # case 4: unknown objects + f = None if f is not None: while f is not None: if limit is not None: diff --git a/Lib/test/test_asyncgen.py b/Lib/test/test_asyncgen.py index fb6321d2264f31..62bf8774166529 100644 --- a/Lib/test/test_asyncgen.py +++ b/Lib/test/test_asyncgen.py @@ -1191,5 +1191,20 @@ async def run(): self.loop.run_until_complete(run()) + def test_async_gen_aclose_compatible_with_get_stack(self): + async def async_generator(): + yield object() + + async def run(): + ag = async_generator() + asyncio.create_task(ag.aclose()) + tasks = asyncio.all_tasks() + for task in tasks: + # No AttributeError raised + task.get_stack() + + self.loop.run_until_complete(run()) + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Library/2020-02-27-18-21-07.bpo-39764.wqPk68.rst b/Misc/NEWS.d/next/Library/2020-02-27-18-21-07.bpo-39764.wqPk68.rst new file mode 100644 index 00000000000000..d61db2ea221f26 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-02-27-18-21-07.bpo-39764.wqPk68.rst @@ -0,0 +1 @@ +Fix AttributeError when calling get_stack on a PyAsyncGenObject Task \ No newline at end of file From 7ad99821d8ae75222c50e69194a39f535bb058f5 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 2 Mar 2020 07:02:03 -0800 Subject: [PATCH 1241/2163] bpo-38380: Update macOS & Windows builds to SQLite v3.31.1 (GH-18678) Automerge-Triggered-By: @zooba (cherry picked from commit 1382c3289bcfd34ac6811fdf9aa5bc09ca8c320e) Co-authored-by: Erlend Egeberg Aasland --- Mac/BuildScript/build-installer.py | 6 +++--- .../next/Windows/2020-02-28-23-51-27.bpo-38380.TpOBCj.rst | 1 + .../next/macOS/2020-02-28-23-51-47.bpo-38380.u-ySyA.rst | 1 + PCbuild/get_externals.bat | 2 +- PCbuild/python.props | 2 +- PCbuild/readme.txt | 2 +- 6 files changed, 8 insertions(+), 6 deletions(-) create mode 100644 Misc/NEWS.d/next/Windows/2020-02-28-23-51-27.bpo-38380.TpOBCj.rst create mode 100644 Misc/NEWS.d/next/macOS/2020-02-28-23-51-47.bpo-38380.u-ySyA.rst diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index 0dd53647dd33e3..31e80637d8a853 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -313,9 +313,9 @@ def library_recipes(): ), ), dict( - name="SQLite 3.28.0", - url="https://www.sqlite.org/2019/sqlite-autoconf-3280000.tar.gz", - checksum='3c68eb400f8354605736cd55400e1572', + name="SQLite 3.31.1", + url="https://sqlite.org/2020/sqlite-autoconf-3310100.tar.gz", + checksum='2d0a553534c521504e3ac3ad3b90f125', extra_cflags=('-Os ' '-DSQLITE_ENABLE_FTS5 ' '-DSQLITE_ENABLE_FTS4 ' diff --git a/Misc/NEWS.d/next/Windows/2020-02-28-23-51-27.bpo-38380.TpOBCj.rst b/Misc/NEWS.d/next/Windows/2020-02-28-23-51-27.bpo-38380.TpOBCj.rst new file mode 100644 index 00000000000000..521075d628f425 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-02-28-23-51-27.bpo-38380.TpOBCj.rst @@ -0,0 +1 @@ +Update Windows builds to use SQLite 3.31.1 diff --git a/Misc/NEWS.d/next/macOS/2020-02-28-23-51-47.bpo-38380.u-ySyA.rst b/Misc/NEWS.d/next/macOS/2020-02-28-23-51-47.bpo-38380.u-ySyA.rst new file mode 100644 index 00000000000000..908281b5d172f0 --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2020-02-28-23-51-47.bpo-38380.u-ySyA.rst @@ -0,0 +1 @@ +Update macOS builds to use SQLite 3.31.1 diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index 100149e83c1645..36ccbd38ad102a 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -54,7 +54,7 @@ set libraries= set libraries=%libraries% bzip2-1.0.6 if NOT "%IncludeLibffiSrc%"=="false" set libraries=%libraries% libffi-3.3.0-rc0-r1 if NOT "%IncludeSSLSrc%"=="false" set libraries=%libraries% openssl-1.1.1d -set libraries=%libraries% sqlite-3.28.0.0 +set libraries=%libraries% sqlite-3.31.1.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tcl-core-8.6.9.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tk-8.6.9.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tix-8.4.3.6 diff --git a/PCbuild/python.props b/PCbuild/python.props index 8583138feb6be7..4cba669a427bcd 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -56,7 +56,7 @@ $(EXTERNALS_DIR) $([System.IO.Path]::GetFullPath(`$(PySourcePath)externals`)) $(ExternalsDir)\ - $(ExternalsDir)sqlite-3.28.0.0\ + $(ExternalsDir)sqlite-3.31.1.0\ $(ExternalsDir)bzip2-1.0.6\ $(ExternalsDir)xz-5.2.2\ $(ExternalsDir)libffi\ diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt index 80e3359df3ddbe..ceed5dc9c3e72c 100644 --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -184,7 +184,7 @@ _ssl again when building. _sqlite3 - Wraps SQLite 3.28.0.0, which is itself built by sqlite3.vcxproj + Wraps SQLite 3.31.1.0, which is itself built by sqlite3.vcxproj Homepage: http://www.sqlite.org/ _tkinter From 9ddcb914f9c2debe7c1359b2450cd1573e86b91c Mon Sep 17 00:00:00 2001 From: Pablo Galindo Date: Mon, 2 Mar 2020 23:55:20 +0000 Subject: [PATCH 1242/2163] [3.8] bpo-39778: Don't traverse weak-reference lists OrderedDict's tp_traverse and tp_clear (GH-18749) (GH-18756) Objects do not own weak references to them directly through the __weakref__ list so these do not need to be traversed by the GC. (cherry picked from commit 0c2b509) --- Lib/test/test_ordered_dict.py | 20 +++++++++++++++++++ .../2020-03-02-19-21-21.bpo-39778._YGLEc.rst | 2 ++ Objects/odictobject.c | 2 -- 3 files changed, 22 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-03-02-19-21-21.bpo-39778._YGLEc.rst diff --git a/Lib/test/test_ordered_dict.py b/Lib/test/test_ordered_dict.py index 148a9bdc35ee3e..085e5f60ed93b6 100644 --- a/Lib/test/test_ordered_dict.py +++ b/Lib/test/test_ordered_dict.py @@ -753,6 +753,26 @@ def test_iterators_pickling(self): self.assertEqual(list(unpickled), expected) self.assertEqual(list(it), expected) + @support.cpython_only + def test_weakref_list_is_not_traversed(self): + # Check that the weakref list is not traversed when collecting + # OrderedDict objects. See bpo-39778 for more information. + + gc.collect() + + x = self.OrderedDict() + x.cycle = x + + cycle = [] + cycle.append(cycle) + + x_ref = weakref.ref(x) + cycle.append(x_ref) + + del x, cycle, x_ref + + gc.collect() + class PurePythonOrderedDictSubclassTests(PurePythonOrderedDictTests): diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-03-02-19-21-21.bpo-39778._YGLEc.rst b/Misc/NEWS.d/next/Core and Builtins/2020-03-02-19-21-21.bpo-39778._YGLEc.rst new file mode 100644 index 00000000000000..dc49512167365d --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-03-02-19-21-21.bpo-39778._YGLEc.rst @@ -0,0 +1,2 @@ +Fixed a crash due to incorrect handling of weak references in +``collections.OrderedDict`` classes. Patch by Pablo Galindo. diff --git a/Objects/odictobject.c b/Objects/odictobject.c index 4c9ae3bc934686..ac0da9bd5bae30 100644 --- a/Objects/odictobject.c +++ b/Objects/odictobject.c @@ -1453,7 +1453,6 @@ odict_traverse(PyODictObject *od, visitproc visit, void *arg) _ODictNode *node; Py_VISIT(od->od_inst_dict); - Py_VISIT(od->od_weakreflist); _odict_FOREACH(od, node) { Py_VISIT(_odictnode_KEY(node)); } @@ -1466,7 +1465,6 @@ static int odict_tp_clear(PyODictObject *od) { Py_CLEAR(od->od_inst_dict); - Py_CLEAR(od->od_weakreflist); PyDict_Clear((PyObject *)od); _odict_clear_nodes(od); return 0; From 0d20364b132014eec609b900997c34779a4d548c Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 2 Mar 2020 16:30:18 -0800 Subject: [PATCH 1243/2163] bpo-38597: Never statically link extension initialization code on Windows (GH-18724) (cherry picked from commit ce3a4984089b8e0ce5422ca32d75ad057b008074) Co-authored-by: Steve Dower --- Lib/distutils/_msvccompiler.py | 60 +++---------------- Lib/distutils/tests/test_msvccompiler.py | 51 ---------------- .../2020-03-01-15-04-54.bpo-38597.MnHdYl.rst | 4 ++ PCbuild/pythoncore.vcxproj | 2 + 4 files changed, 13 insertions(+), 104 deletions(-) create mode 100644 Misc/NEWS.d/next/Windows/2020-03-01-15-04-54.bpo-38597.MnHdYl.rst diff --git a/Lib/distutils/_msvccompiler.py b/Lib/distutils/_msvccompiler.py index e8e4b717b9736f..03a5986d984dc3 100644 --- a/Lib/distutils/_msvccompiler.py +++ b/Lib/distutils/_msvccompiler.py @@ -97,28 +97,11 @@ def _find_vc2017(): } def _find_vcvarsall(plat_spec): + # bpo-38597: Removed vcruntime return value _, best_dir = _find_vc2017() - vcruntime = None - - if plat_spec in PLAT_SPEC_TO_RUNTIME: - vcruntime_plat = PLAT_SPEC_TO_RUNTIME[plat_spec] - else: - vcruntime_plat = 'x64' if 'amd64' in plat_spec else 'x86' - - if best_dir: - vcredist = os.path.join(best_dir, "..", "..", "redist", "MSVC", "**", - vcruntime_plat, "Microsoft.VC14*.CRT", "vcruntime140.dll") - try: - import glob - vcruntime = glob.glob(vcredist, recursive=True)[-1] - except (ImportError, OSError, LookupError): - vcruntime = None if not best_dir: best_version, best_dir = _find_vc2015() - if best_version: - vcruntime = os.path.join(best_dir, 'redist', vcruntime_plat, - "Microsoft.VC140.CRT", "vcruntime140.dll") if not best_dir: log.debug("No suitable Visual C++ version found") @@ -129,11 +112,7 @@ def _find_vcvarsall(plat_spec): log.debug("%s cannot be found", vcvarsall) return None, None - if not vcruntime or not os.path.isfile(vcruntime): - log.debug("%s cannot be found", vcruntime) - vcruntime = None - - return vcvarsall, vcruntime + return vcvarsall, None def _get_vc_env(plat_spec): if os.getenv("DISTUTILS_USE_SDK"): @@ -142,7 +121,7 @@ def _get_vc_env(plat_spec): for key, value in os.environ.items() } - vcvarsall, vcruntime = _find_vcvarsall(plat_spec) + vcvarsall, _ = _find_vcvarsall(plat_spec) if not vcvarsall: raise DistutilsPlatformError("Unable to find vcvarsall.bat") @@ -163,8 +142,6 @@ def _get_vc_env(plat_spec): if key and value } - if vcruntime: - env['py_vcruntime_redist'] = vcruntime return env def _find_exe(exe, paths=None): @@ -194,12 +171,6 @@ def _find_exe(exe, paths=None): 'win-arm64' : 'x86_arm64' } -# A set containing the DLLs that are guaranteed to be available for -# all micro versions of this Python version. Known extension -# dependencies that are not in this set will be copied to the output -# path. -_BUNDLED_DLLS = frozenset(['vcruntime140.dll']) - class MSVCCompiler(CCompiler) : """Concrete class that implements an interface to Microsoft Visual C++, as defined by the CCompiler abstract class.""" @@ -263,7 +234,6 @@ def initialize(self, plat_name=None): self.rc = _find_exe("rc.exe", paths) # resource compiler self.mc = _find_exe("mc.exe", paths) # message compiler self.mt = _find_exe("mt.exe", paths) # message compiler - self._vcruntime_redist = vc_env.get('py_vcruntime_redist', '') for dir in vc_env.get('include', '').split(os.pathsep): if dir: @@ -274,13 +244,12 @@ def initialize(self, plat_name=None): self.add_library_dir(dir.rstrip(os.sep)) self.preprocess_options = None - # If vcruntime_redist is available, link against it dynamically. Otherwise, - # use /MT[d] to build statically, then switch from libucrt[d].lib to ucrt[d].lib - # later to dynamically link to ucrtbase but not vcruntime. + # bpo-38597: Always compile with dynamic linking + # Future releases of Python 3.x will include all past + # versions of vcruntime*.dll for compatibility. self.compile_options = [ - '/nologo', '/Ox', '/W3', '/GL', '/DNDEBUG' + '/nologo', '/Ox', '/W3', '/GL', '/DNDEBUG', '/MD' ] - self.compile_options.append('/MD' if self._vcruntime_redist else '/MT') self.compile_options_debug = [ '/nologo', '/Od', '/MDd', '/Zi', '/W3', '/D_DEBUG' @@ -289,8 +258,6 @@ def initialize(self, plat_name=None): ldflags = [ '/nologo', '/INCREMENTAL:NO', '/LTCG' ] - if not self._vcruntime_redist: - ldflags.extend(('/nodefaultlib:libucrt.lib', 'ucrt.lib')) ldflags_debug = [ '/nologo', '/INCREMENTAL:NO', '/LTCG', '/DEBUG:FULL' @@ -532,24 +499,11 @@ def link(self, try: log.debug('Executing "%s" %s', self.linker, ' '.join(ld_args)) self.spawn([self.linker] + ld_args) - self._copy_vcruntime(output_dir) except DistutilsExecError as msg: raise LinkError(msg) else: log.debug("skipping %s (up-to-date)", output_filename) - def _copy_vcruntime(self, output_dir): - vcruntime = self._vcruntime_redist - if not vcruntime or not os.path.isfile(vcruntime): - return - - if os.path.basename(vcruntime).lower() in _BUNDLED_DLLS: - return - - log.debug('Copying "%s"', vcruntime) - vcruntime = shutil.copy(vcruntime, output_dir) - os.chmod(vcruntime, stat.S_IWRITE) - def spawn(self, cmd): old_path = os.getenv('path') try: diff --git a/Lib/distutils/tests/test_msvccompiler.py b/Lib/distutils/tests/test_msvccompiler.py index 70a9c93a4e8805..b518d6a78b3326 100644 --- a/Lib/distutils/tests/test_msvccompiler.py +++ b/Lib/distutils/tests/test_msvccompiler.py @@ -32,57 +32,6 @@ def _find_vcvarsall(plat_spec): finally: _msvccompiler._find_vcvarsall = old_find_vcvarsall - def test_compiler_options(self): - import distutils._msvccompiler as _msvccompiler - # suppress path to vcruntime from _find_vcvarsall to - # check that /MT is added to compile options - old_find_vcvarsall = _msvccompiler._find_vcvarsall - def _find_vcvarsall(plat_spec): - return old_find_vcvarsall(plat_spec)[0], None - _msvccompiler._find_vcvarsall = _find_vcvarsall - try: - compiler = _msvccompiler.MSVCCompiler() - compiler.initialize() - - self.assertIn('/MT', compiler.compile_options) - self.assertNotIn('/MD', compiler.compile_options) - finally: - _msvccompiler._find_vcvarsall = old_find_vcvarsall - - def test_vcruntime_copy(self): - import distutils._msvccompiler as _msvccompiler - # force path to a known file - it doesn't matter - # what we copy as long as its name is not in - # _msvccompiler._BUNDLED_DLLS - old_find_vcvarsall = _msvccompiler._find_vcvarsall - def _find_vcvarsall(plat_spec): - return old_find_vcvarsall(plat_spec)[0], __file__ - _msvccompiler._find_vcvarsall = _find_vcvarsall - try: - tempdir = self.mkdtemp() - compiler = _msvccompiler.MSVCCompiler() - compiler.initialize() - compiler._copy_vcruntime(tempdir) - - self.assertTrue(os.path.isfile(os.path.join( - tempdir, os.path.basename(__file__)))) - finally: - _msvccompiler._find_vcvarsall = old_find_vcvarsall - - def test_vcruntime_skip_copy(self): - import distutils._msvccompiler as _msvccompiler - - tempdir = self.mkdtemp() - compiler = _msvccompiler.MSVCCompiler() - compiler.initialize() - dll = compiler._vcruntime_redist - self.assertTrue(os.path.isfile(dll), dll or "") - - compiler._copy_vcruntime(tempdir) - - self.assertFalse(os.path.isfile(os.path.join( - tempdir, os.path.basename(dll))), dll or "") - def test_get_vc_env_unicode(self): import distutils._msvccompiler as _msvccompiler diff --git a/Misc/NEWS.d/next/Windows/2020-03-01-15-04-54.bpo-38597.MnHdYl.rst b/Misc/NEWS.d/next/Windows/2020-03-01-15-04-54.bpo-38597.MnHdYl.rst new file mode 100644 index 00000000000000..7f3a2e756c5a13 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-03-01-15-04-54.bpo-38597.MnHdYl.rst @@ -0,0 +1,4 @@ +:mod:`distutils` will no longer statically link :file:`vcruntime140.dll` +when a redistributable version is unavailable. All future releases of +CPython will include a copy of this DLL to ensure distributed extensions can +continue to load. diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 13c3b594e44314..bd16c1e070bc24 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -528,6 +528,8 @@ + + From 394dc0db878c08d003772de163a57ac12046d865 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Date: Tue, 3 Mar 2020 01:13:10 +0000 Subject: [PATCH 1244/2163] [3.8] bpo-39831: Fix a reference leak in PyErr_WarnEx(). (GH-18750) (GH-18761) (cherry picked from commit 2d2f855) Co-authored-by: Serhiy Storchaka --- Python/_warnings.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Python/_warnings.c b/Python/_warnings.c index e02d28305126df..87269dd4dbc9ee 100644 --- a/Python/_warnings.c +++ b/Python/_warnings.c @@ -856,11 +856,11 @@ setup_context(Py_ssize_t stack_level, PyObject **filename, int *lineno, int rc; if (PyErr_Occurred()) { - return 0; + goto handle_error; } *registry = PyDict_New(); if (*registry == NULL) - return 0; + goto handle_error; rc = _PyDict_SetItemId(globals, &PyId___warningregistry__, *registry); if (rc < 0) @@ -890,6 +890,7 @@ setup_context(Py_ssize_t stack_level, PyObject **filename, int *lineno, dangling reference. */ Py_XDECREF(*registry); Py_XDECREF(*module); + Py_XDECREF(*filename); return 0; } From 1827fc30f463786ebff13752e35c3224652bc94e Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 2 Mar 2020 19:04:13 -0800 Subject: [PATCH 1245/2163] bpo-39778: Add clarification about tp_traverse and ownership (GH-18754) Automerge-Triggered-By: @pablogsal (cherry picked from commit 6df421fe87a9418d6c59f89dbc5d5573b6826855) Co-authored-by: Pablo Galindo --- Doc/c-api/typeobj.rst | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index bff5abfea48ee9..163f599d1c2641 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -1218,7 +1218,7 @@ and :c:type:`PyType_Type` effectively act as defaults.) The :c:member:`~PyTypeObject.tp_traverse` pointer is used by the garbage collector to detect reference cycles. A typical implementation of a :c:member:`~PyTypeObject.tp_traverse` function simply calls :c:func:`Py_VISIT` on each of the instance's members that are Python - objects. For example, this is function :c:func:`local_traverse` from the + objects that the instance owns. For example, this is function :c:func:`local_traverse` from the :mod:`_thread` extension module:: static int @@ -1238,6 +1238,18 @@ and :c:type:`PyType_Type` effectively act as defaults.) debugging aid you may want to visit it anyway just so the :mod:`gc` module's :func:`~gc.get_referents` function will include it. + .. warning:: + When implementing :c:member:`~PyTypeObject.tp_traverse`, only the members + that the instance *owns* (by having strong references to them) must be + visited. For instance, if an object supports weak references via the + :c:member:`~PyTypeObject.tp_weaklist` slot, the pointer supporting + the linked list (what *tp_weaklist* points to) must **not** be + visited as the instance does not directly own the weak references to itself + (the weakreference list is there to support the weak reference machinery, + but the instance has no strong reference to the elements inside it, as they + are allowed to be removed even if the instance is still alive). + + Note that :c:func:`Py_VISIT` requires the *visit* and *arg* parameters to :c:func:`local_traverse` to have these specific names; don't name them just anything. From 5a92f42d8723ee865be80f028d402204649da15d Mon Sep 17 00:00:00 2001 From: Stefan Krah Date: Tue, 3 Mar 2020 09:18:55 +0100 Subject: [PATCH 1246/2163] bpo-39776: Lock ++interp->tstate_next_unique_id. (GH-18746) (#18746) (#18752) - Threads created by PyGILState_Ensure() could have a duplicate tstate->id. (cherry picked from commit b3b9ade4a3d3fe00d933bcd8fc5c5c755d1024f9) --- .../2020-03-02-20-12-33.bpo-39776.fNaxi_.rst | 6 ++++++ Python/pystate.c | 3 +-- 2 files changed, 7 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-03-02-20-12-33.bpo-39776.fNaxi_.rst diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-03-02-20-12-33.bpo-39776.fNaxi_.rst b/Misc/NEWS.d/next/Core and Builtins/2020-03-02-20-12-33.bpo-39776.fNaxi_.rst new file mode 100644 index 00000000000000..e5a00bd96ae47a --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-03-02-20-12-33.bpo-39776.fNaxi_.rst @@ -0,0 +1,6 @@ +Fix race condition where threads created by PyGILState_Ensure() could get a +duplicate id. + +This affects consumers of tstate->id like the contextvar caching machinery, +which could return invalid cached objects under heavy thread load (observed +in embedded scenarios). diff --git a/Python/pystate.c b/Python/pystate.c index aba673c00a4322..9f99060feceb21 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -606,13 +606,12 @@ new_threadstate(PyInterpreterState *interp, int init) tstate->context = NULL; tstate->context_ver = 1; - tstate->id = ++interp->tstate_next_unique_id; - if (init) { _PyThreadState_Init(runtime, tstate); } HEAD_LOCK(runtime); + tstate->id = ++interp->tstate_next_unique_id; tstate->prev = NULL; tstate->next = interp->tstate_head; if (tstate->next) From 2aa694dd55202eb6d062fdf49c07cf95f8520f2d Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 3 Mar 2020 17:31:24 +0100 Subject: [PATCH 1247/2163] bpo-39674: Update collections ABC deprecation doc (GH-18748) --- Doc/library/collections.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index a5e8d04099b22f..8dcf9451d72bfe 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -33,10 +33,10 @@ Python's general purpose built-in containers, :class:`dict`, :class:`list`, :class:`UserString` wrapper around string objects for easier string subclassing ===================== ==================================================================== -.. deprecated-removed:: 3.3 3.9 +.. deprecated-removed:: 3.3 3.10 Moved :ref:`collections-abstract-base-classes` to the :mod:`collections.abc` module. For backwards compatibility, they continue to be visible in this module through - Python 3.8. + Python 3.9. :class:`ChainMap` objects From 57fb29219f09ee705065e39ad488334a9c7e3d71 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 3 Mar 2020 10:01:09 -0800 Subject: [PATCH 1248/2163] bpo-39831: Remove outdated comment. (GH-18764) (cherry picked from commit ae75a294352e9b9487f5dc8e88f068e7e6974dc2) Co-authored-by: Serhiy Storchaka --- Python/_warnings.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/Python/_warnings.c b/Python/_warnings.c index 87269dd4dbc9ee..52a13dfc62ccca 100644 --- a/Python/_warnings.c +++ b/Python/_warnings.c @@ -886,11 +886,9 @@ setup_context(Py_ssize_t stack_level, PyObject **filename, int *lineno, return 1; handle_error: - /* filename not XDECREF'ed here as there is no way to jump here with a - dangling reference. */ Py_XDECREF(*registry); Py_XDECREF(*module); - Py_XDECREF(*filename); + Py_DECREF(*filename); return 0; } From 6c8cc0dbfd69984a4e8381388704dc78261e1d59 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 4 Mar 2020 03:52:56 -0800 Subject: [PATCH 1249/2163] Fix misleading statement about mixed-type numeric comparisons (GH-18615) (#18773) (cherry picked from commit 9f1cb1bb49476246de5d9ed5fe680301cf7f7571) Co-authored-by: Mark Dickinson Co-authored-by: Mark Dickinson --- Doc/library/stdtypes.rst | 6 ++++-- .../Documentation/2020-02-23-13-26-40.bpo-39530._bCvzQ.rst | 1 + 2 files changed, 5 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Documentation/2020-02-23-13-26-40.bpo-39530._bCvzQ.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index d369b38b31bc90..8cc57c30c322cb 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -261,8 +261,10 @@ and imaginary parts. Python fully supports mixed arithmetic: when a binary arithmetic operator has operands of different numeric types, the operand with the "narrower" type is widened to that of the other, where integer is narrower than floating point, -which is narrower than complex. Comparisons between numbers of mixed type use -the same rule. [2]_ The constructors :func:`int`, :func:`float`, and +which is narrower than complex. A comparison between numbers of different types +behaves as though the exact values of those numbers were being compared. [2]_ + +The constructors :func:`int`, :func:`float`, and :func:`complex` can be used to produce numbers of a specific type. All numeric types (except complex) support the following operations (for priorities of diff --git a/Misc/NEWS.d/next/Documentation/2020-02-23-13-26-40.bpo-39530._bCvzQ.rst b/Misc/NEWS.d/next/Documentation/2020-02-23-13-26-40.bpo-39530._bCvzQ.rst new file mode 100644 index 00000000000000..b7a02522bbb1ca --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-02-23-13-26-40.bpo-39530._bCvzQ.rst @@ -0,0 +1 @@ +Fix misleading documentation about mixed-type numeric comparisons. From 5a3926d55922bf62faaeab664844dd69df04ab42 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 4 Mar 2020 16:18:22 -0800 Subject: [PATCH 1250/2163] Add a missing space after a period in 'typing.TypedDict' documentation (GH-18784) (cherry picked from commit d4a09c13ddd91a9bc1b4ba76ff4e8a153334a1e2) Co-authored-by: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com> --- Doc/library/typing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index a9c7c4756dd0d1..beb00fc8b61c23 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -1004,7 +1004,7 @@ The module defines the following classes, functions and decorators: x: int y: int - This means that a point2D TypedDict can have any of the keys omitted.A type + This means that a point2D TypedDict can have any of the keys omitted. A type checker is only expected to support a literal False or True as the value of the total argument. True is the default, and makes all items defined in the class body be required. From 6b452ff97f70eca79ab956987cc04b6586feca00 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 4 Mar 2020 17:04:38 -0800 Subject: [PATCH 1251/2163] bpo-13487: Use sys.modules.copy() in inspect.getmodule() for thread safety. (GH-18786) `list(sys.modules.items())` was apparently not immune to "dictionary changed size during iteration" errors. Tested internally using an integration test that has run into this a couple of times in the past two years. With this patch applied, the test is no longer flaky. (cherry picked from commit 85cf1d514b84dc9a4bcb40e20a12e1d82ff19f20) Co-authored-by: Gregory P. Smith --- Lib/inspect.py | 2 +- .../next/Library/2020-03-04-16-10-59.bpo-13487.gqe4Fb.rst | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2020-03-04-16-10-59.bpo-13487.gqe4Fb.rst diff --git a/Lib/inspect.py b/Lib/inspect.py index 3ff395ca333778..e8ea8c221f8288 100644 --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -741,7 +741,7 @@ def getmodule(object, _filename=None): return sys.modules.get(modulesbyfile[file]) # Update the filename to module name cache and check yet again # Copy sys.modules in order to cope with changes while iterating - for modname, module in list(sys.modules.items()): + for modname, module in sys.modules.copy().items(): if ismodule(module) and hasattr(module, '__file__'): f = module.__file__ if f == _filesbymodname.get(modname, None): diff --git a/Misc/NEWS.d/next/Library/2020-03-04-16-10-59.bpo-13487.gqe4Fb.rst b/Misc/NEWS.d/next/Library/2020-03-04-16-10-59.bpo-13487.gqe4Fb.rst new file mode 100644 index 00000000000000..5a1f02a7bdf377 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-03-04-16-10-59.bpo-13487.gqe4Fb.rst @@ -0,0 +1,3 @@ +Avoid a possible *"RuntimeError: dictionary changed size during iteration"* +from :func:`inspect.getmodule` when it tried to loop through +:attr:`sys.modules`. From 7b39c474e4ce7057a9e16b06d40261ff563b1e9d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 5 Mar 2020 10:36:54 -0800 Subject: [PATCH 1252/2163] [3.8] bpo-39808: Improve docs for pathlib.Path.stat() (GH-18719) (GH-18783) (cherry picked from commit 67152d0ed670227b61b5df683655b196ab04ca1a) Co-authored-by: Brett Cannon <54418+brettcannon@users.noreply.github.com> Automerge-Triggered-By: @brettcannon --- Doc/library/pathlib.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 326bb2e933cdb2..03408145cf5722 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -672,7 +672,7 @@ call fails (for example because the path doesn't exist). .. method:: Path.stat() - Return information about this path (similarly to :func:`os.stat`). + Return a :class:`os.stat_result` object containing information about this path, like :func:`os.stat`. The result is looked up at each call to this method. :: From da9ee00ffea9120bea4b5f0cbed20f9fb584d2a2 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 5 Mar 2020 17:35:43 -0800 Subject: [PATCH 1253/2163] IDLE doc: improve Startup failure subsection. (GH-18771) Eliminate repeat of 'Options', reported by Jules Lasne, and improve wording elsewhere. Co-authored-by: Terry Jan Reedy (cherry picked from commit ce305d641074931e4e790f7a83e28f74910644e5) Co-authored-by: Jules Lasne (jlasne) --- Doc/library/idle.rst | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst index f15f46b788b36a..fd6e309567de39 100644 --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -681,19 +681,22 @@ clash, or cannot or does not want to run as admin, it might be easiest to completely remove Python and start over. A zombie pythonw.exe process could be a problem. On Windows, use Task -Manager to detect and stop one. Sometimes a restart initiated by a program -crash or Keyboard Interrupt (control-C) may fail to connect. Dismissing -the error box or Restart Shell on the Shell menu may fix a temporary problem. +Manager to check for one and stop it if there is. Sometimes a restart +initiated by a program crash or Keyboard Interrupt (control-C) may fail +to connect. Dismissing the error box or using Restart Shell on the Shell +menu may fix a temporary problem. When IDLE first starts, it attempts to read user configuration files in ``~/.idlerc/`` (~ is one's home directory). If there is a problem, an error message should be displayed. Leaving aside random disk glitches, this can -be prevented by never editing the files by hand, using the configuration -dialog, under Options, instead Options. Once it happens, the solution may -be to delete one or more of the configuration files. +be prevented by never editing the files by hand. Instead, use the +configuration dialog, under Options. Once there is an error in a user +configuration file, the best solution may be to delete it and start over +with the settings dialog. If IDLE quits with no message, and it was not started from a console, try -starting from a console (``python -m idlelib``) and see if a message appears. +starting it from a console or terminal (``python -m idlelib``) and see if +this results in an error message. Running user code ^^^^^^^^^^^^^^^^^ From 0c71770eeee9c1b19b6f146b56db5f10bab3f09c Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 5 Mar 2020 21:25:15 -0800 Subject: [PATCH 1254/2163] bpo-39868: Update Language Reference for PEP 572. (GH-18793) (cherry picked from commit 8bae21962bab2fac7630982abd73676b89930902) Co-authored-by: Brandt Bucher --- Doc/reference/compound_stmts.rst | 6 +++--- Doc/reference/expressions.rst | 18 +++++++++++++++--- .../2020-03-05-16-29-03.bpo-39868.JQoHhO.rst | 1 + 3 files changed, 19 insertions(+), 6 deletions(-) create mode 100644 Misc/NEWS.d/next/Documentation/2020-03-05-16-29-03.bpo-39868.JQoHhO.rst diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 564d6cc42136da..feaf3109d5a24e 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -90,8 +90,8 @@ The :keyword:`!if` statement The :keyword:`if` statement is used for conditional execution: .. productionlist:: - if_stmt: "if" `expression` ":" `suite` - : ("elif" `expression` ":" `suite`)* + if_stmt: "if" `assignment_expression` ":" `suite` + : ("elif" `assignment_expression` ":" `suite`)* : ["else" ":" `suite`] It selects exactly one of the suites by evaluating the expressions one by one @@ -116,7 +116,7 @@ The :keyword:`while` statement is used for repeated execution as long as an expression is true: .. productionlist:: - while_stmt: "while" `expression` ":" `suite` + while_stmt: "while" `assignment_expression` ":" `suite` : ["else" ":" `suite`] This repeatedly tests the expression and, if it is true, executes the first diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index d9db33a78568e4..3fcc5e17d9a7cd 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -178,7 +178,7 @@ called "displays", each of them in two flavors: Common syntax elements for comprehensions are: .. productionlist:: - comprehension: `expression` `comp_for` + comprehension: `assignment_expression` `comp_for` comp_for: ["async"] "for" `target_list` "in" `or_test` [`comp_iter`] comp_iter: `comp_for` | `comp_if` comp_if: "if" `expression_nocond` [`comp_iter`] @@ -911,7 +911,8 @@ series of :term:`arguments `: : ["," `keywords_arguments`] : | `starred_and_keywords` ["," `keywords_arguments`] : | `keywords_arguments` - positional_arguments: ["*"] `expression` ("," ["*"] `expression`)* + positional_arguments: positional_item ("," positional_item)* + positional_item: `assignment_expression` | "*" `expression` starred_and_keywords: ("*" `expression` | `keyword_item`) : ("," "*" `expression` | "," `keyword_item`)* keywords_arguments: (`keyword_item` | "**" `expression`) @@ -1642,6 +1643,17 @@ returns a boolean value regardless of the type of its argument (for example, ``not 'foo'`` produces ``False`` rather than ``''``.) +Assignment expressions +====================== + +.. productionlist:: + assignment_expression: [`identifier` ":="] `expression` + +.. TODO: BPO-39868 + +See :pep:`572` for more details about assignment expressions. + + .. _if_expr: Conditional expressions @@ -1711,7 +1723,7 @@ Expression lists expression_list: `expression` ("," `expression`)* [","] starred_list: `starred_item` ("," `starred_item`)* [","] starred_expression: `expression` | (`starred_item` ",")* [`starred_item`] - starred_item: `expression` | "*" `or_expr` + starred_item: `assignment_expression` | "*" `or_expr` .. index:: object: tuple diff --git a/Misc/NEWS.d/next/Documentation/2020-03-05-16-29-03.bpo-39868.JQoHhO.rst b/Misc/NEWS.d/next/Documentation/2020-03-05-16-29-03.bpo-39868.JQoHhO.rst new file mode 100644 index 00000000000000..9fa8bfd04f7dbb --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-03-05-16-29-03.bpo-39868.JQoHhO.rst @@ -0,0 +1 @@ +Updated the Language Reference for :pep:`572`. From 6df0c47669031c6a8e9b1a07ec2c5813e4c10ee0 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 6 Mar 2020 10:26:52 -0800 Subject: [PATCH 1255/2163] bpo-17422: slightly more precise language (GH-18682) (cherry picked from commit e59334ebc9308b0f3ad048ef293c6b49e6456d1a) Co-authored-by: Caleb Donovick --- Doc/reference/datamodel.rst | 2 +- .../next/Documentation/2020-02-27-17-35-27.bpo-17422.eS1hVh.rst | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Documentation/2020-02-27-17-35-27.bpo-17422.eS1hVh.rst diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 06f4992cf544a9..1530775839bb6c 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1951,7 +1951,7 @@ namespace returned by ``__prepare__`` is passed in to ``__new__``, but when the final class object is created the namespace is copied into a new ``dict``. If the metaclass has no ``__prepare__`` attribute, then the class namespace -is initialised as an empty :func:`dict`. +is initialised as an empty ordered mapping. .. seealso:: diff --git a/Misc/NEWS.d/next/Documentation/2020-02-27-17-35-27.bpo-17422.eS1hVh.rst b/Misc/NEWS.d/next/Documentation/2020-02-27-17-35-27.bpo-17422.eS1hVh.rst new file mode 100644 index 00000000000000..bbec5ec0eee65f --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-02-27-17-35-27.bpo-17422.eS1hVh.rst @@ -0,0 +1,2 @@ +The language reference no longer restricts default class namespaces to dicts +only. From d692d52f4aaeb6feaabb18f18d49907dd578fbeb Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 6 Mar 2020 16:30:09 -0800 Subject: [PATCH 1256/2163] bpo-39837: Disable macOS tests on Azure Pipelines (GH-18818) (cherry picked from commit 31350f9af09dcff7cf6ff4b0a0a7ea595942372e) Co-authored-by: Steve Dower --- .azure-pipelines/ci.yml | 4 +++- .azure-pipelines/pr.yml | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/.azure-pipelines/ci.yml b/.azure-pipelines/ci.yml index 4c2f115cd9b600..6c2c1acc286c08 100644 --- a/.azure-pipelines/ci.yml +++ b/.azure-pipelines/ci.yml @@ -35,7 +35,9 @@ jobs: - job: macOS_CI_Tests displayName: macOS CI Tests dependsOn: Prebuild - condition: and(succeeded(), eq(dependencies.Prebuild.outputs['tests.run'], 'true')) + #condition: and(succeeded(), eq(dependencies.Prebuild.outputs['tests.run'], 'true')) + # bpo-39837: macOS tests on Azure Pipelines are disabled + condition: false variables: testRunTitle: '$(build.sourceBranchName)-macos' diff --git a/.azure-pipelines/pr.yml b/.azure-pipelines/pr.yml index 73d4f55b864500..0cc764d025f40b 100644 --- a/.azure-pipelines/pr.yml +++ b/.azure-pipelines/pr.yml @@ -33,7 +33,9 @@ jobs: - job: macOS_PR_Tests displayName: macOS PR Tests dependsOn: Prebuild - condition: and(succeeded(), eq(dependencies.Prebuild.outputs['tests.run'], 'true')) + #condition: and(succeeded(), eq(dependencies.Prebuild.outputs['tests.run'], 'true')) + # bpo-39837: macOS tests on Azure Pipelines are disabled + condition: false variables: testRunTitle: '$(system.pullRequest.TargetBranch)-macos' From 92b72788ecf2ee5dfac780c7dfb5ee5350fc641d Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Sat, 7 Mar 2020 18:55:32 +0200 Subject: [PATCH 1257/2163] [3.8] bpo-39889: Fix unparse.py for subscript. (GH-18824). (GH-18826) (cherry picked from commit c4928fc1a853f3f84e2b4ec1253d0349137745e5) --- Lib/test/test_tools/test_unparse.py | 14 ++++++++++++++ .../2020-03-07-18-01-30.bpo-39889.l1czT6.rst | 3 +++ Tools/parser/unparse.py | 19 +++++++++++++++++-- 3 files changed, 34 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Tools-Demos/2020-03-07-18-01-30.bpo-39889.l1czT6.rst diff --git a/Lib/test/test_tools/test_unparse.py b/Lib/test/test_tools/test_unparse.py index a958ebb51cc3d2..d89cb09776756b 100644 --- a/Lib/test/test_tools/test_unparse.py +++ b/Lib/test/test_tools/test_unparse.py @@ -265,6 +265,20 @@ def test_dict_unpacking_in_dict(self): self.check_roundtrip(r"""{**{'y': 2}, 'x': 1}""") self.check_roundtrip(r"""{**{'y': 2}, **{'x': 1}}""") + def test_subscript(self): + self.check_roundtrip("a[i]") + self.check_roundtrip("a[i,]") + self.check_roundtrip("a[i, j]") + self.check_roundtrip("a[()]") + self.check_roundtrip("a[i:j]") + self.check_roundtrip("a[:j]") + self.check_roundtrip("a[i:]") + self.check_roundtrip("a[i:j:k]") + self.check_roundtrip("a[:j:k]") + self.check_roundtrip("a[i::k]") + self.check_roundtrip("a[i:j,]") + self.check_roundtrip("a[i:j, k]") + class DirectoryTestCase(ASTTestCase): """Test roundtrip behaviour on all files in Lib and Lib/test.""" diff --git a/Misc/NEWS.d/next/Tools-Demos/2020-03-07-18-01-30.bpo-39889.l1czT6.rst b/Misc/NEWS.d/next/Tools-Demos/2020-03-07-18-01-30.bpo-39889.l1czT6.rst new file mode 100644 index 00000000000000..1202cb5fa064b1 --- /dev/null +++ b/Misc/NEWS.d/next/Tools-Demos/2020-03-07-18-01-30.bpo-39889.l1czT6.rst @@ -0,0 +1,3 @@ +Fixed ``unparse.py`` for extended slices containing a single element (e.g. +``a[i:j,]``). Remove redundant tuples when index with a tuple (e.g. ``a[i, +j]``). diff --git a/Tools/parser/unparse.py b/Tools/parser/unparse.py index a5cc000676b022..0642b84dcfc828 100644 --- a/Tools/parser/unparse.py +++ b/Tools/parser/unparse.py @@ -556,7 +556,17 @@ def _Call(self, t): def _Subscript(self, t): self.dispatch(t.value) self.write("[") - self.dispatch(t.slice) + if (isinstance(t.slice, ast.Index) + and isinstance(t.slice.value, ast.Tuple) + and t.slice.value.elts): + if len(t.slice.value.elts) == 1: + elt = t.slice.value.elts[0] + self.dispatch(elt) + self.write(",") + else: + interleave(lambda: self.write(", "), self.dispatch, t.slice.value.elts) + else: + self.dispatch(t.slice) self.write("]") def _Starred(self, t): @@ -581,7 +591,12 @@ def _Slice(self, t): self.dispatch(t.step) def _ExtSlice(self, t): - interleave(lambda: self.write(', '), self.dispatch, t.dims) + if len(t.dims) == 1: + elt = t.dims[0] + self.dispatch(elt) + self.write(",") + else: + interleave(lambda: self.write(', '), self.dispatch, t.dims) # argument def _arg(self, t): From 928b4dd0edf0022190a8a296c8ea65e7ef55c694 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 7 Mar 2020 10:11:24 -0800 Subject: [PATCH 1258/2163] bpo-38894: Fix pathlib.Path.glob in the presence of symlinks and insufficient permissions (GH-18815) Co-authored-by: Matt Wozniski (cherry picked from commit eb7560a73d46800e4ade4a8869139b48e6c92811) Co-authored-by: Pablo Galindo --- Lib/pathlib.py | 29 ++++++++------- Lib/test/test_pathlib.py | 36 +++++++++++++++++++ .../2020-03-06-21-04-39.bpo-38894.nfcGKv.rst | 4 +++ 3 files changed, 56 insertions(+), 13 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-03-06-21-04-39.bpo-38894.nfcGKv.rst diff --git a/Lib/pathlib.py b/Lib/pathlib.py index 015370a860e656..d188026bcde840 100644 --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -529,23 +529,26 @@ def _select_from(self, parent_path, is_dir, exists, scandir): try: entries = list(scandir(parent_path)) for entry in entries: - entry_is_dir = False - try: - entry_is_dir = entry.is_dir() - except OSError as e: - if not _ignore_error(e): - raise - if not self.dironly or entry_is_dir: - name = entry.name - if self.match(name): - path = parent_path._make_child_relpath(name) - for p in self.successor._select_from(path, is_dir, exists, scandir): - yield p + if self.dironly: + try: + # "entry.is_dir()" can raise PermissionError + # in some cases (see bpo-38894), which is not + # among the errors ignored by _ignore_error() + if not entry.is_dir(): + continue + except OSError as e: + if not _ignore_error(e): + raise + continue + name = entry.name + if self.match(name): + path = parent_path._make_child_relpath(name) + for p in self.successor._select_from(path, is_dir, exists, scandir): + yield p except PermissionError: return - class _RecursiveWildcardSelector(_Selector): def __init__(self, pat, child_parts, flavour): diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index 97fc5d8ad78382..36226948222d64 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -1508,6 +1508,42 @@ def test_glob_dotdot(self): self.assertEqual(set(p.glob("dirA/../file*")), { P(BASE, "dirA/../fileA") }) self.assertEqual(set(p.glob("../xyzzy")), set()) + @support.skip_unless_symlink + def test_glob_permissions(self): + # See bpo-38894 + P = self.cls + base = P(BASE) / 'permissions' + base.mkdir() + + file1 = base / "file1" + file1.touch() + file2 = base / "file2" + file2.touch() + + subdir = base / "subdir" + + file3 = base / "file3" + file3.symlink_to(subdir / "other") + + # Patching is needed to avoid relying on the filesystem + # to return the order of the files as the error will not + # happen if the symlink is the last item. + + with mock.patch("os.scandir") as scandir: + scandir.return_value = sorted(os.scandir(base)) + self.assertEqual(len(set(base.glob("*"))), 3) + + subdir.mkdir() + + with mock.patch("os.scandir") as scandir: + scandir.return_value = sorted(os.scandir(base)) + self.assertEqual(len(set(base.glob("*"))), 4) + + subdir.chmod(000) + + with mock.patch("os.scandir") as scandir: + scandir.return_value = sorted(os.scandir(base)) + self.assertEqual(len(set(base.glob("*"))), 4) def _check_resolve(self, p, expected, strict=True): q = p.resolve(strict) diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-03-06-21-04-39.bpo-38894.nfcGKv.rst b/Misc/NEWS.d/next/Core and Builtins/2020-03-06-21-04-39.bpo-38894.nfcGKv.rst new file mode 100644 index 00000000000000..a937b8ecc626f8 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-03-06-21-04-39.bpo-38894.nfcGKv.rst @@ -0,0 +1,4 @@ +Fix a bug that was causing incomplete results when calling +``pathlib.Path.glob`` in the presence of symlinks that point +to files where the user does not have read access. Patch by Pablo +Galindo and Matt Wozniski. From 765117dc161d7bd8317e9df7fb3f3b5caca281c2 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 8 Mar 2020 10:58:19 -0700 Subject: [PATCH 1259/2163] fix typo: add space (GH-18853) Fix typo in cmdline.rst Add space between the `-m` option and the module name (`timeit`). (cherry picked from commit c580981ba01c4d9f721dbdd88208ba37704e0217) Co-authored-by: Julin S <48789920+ju-sh@users.noreply.github.com> --- Doc/using/cmdline.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 53206c7b546492..d3f04004efc750 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -109,8 +109,8 @@ source. Many standard library modules contain code that is invoked on their execution as a script. An example is the :mod:`timeit` module:: - python -mtimeit -s 'setup here' 'benchmarked code here' - python -mtimeit -h # for details + python -m timeit -s 'setup here' 'benchmarked code here' + python -m timeit -h # for details .. audit-event:: cpython.run_module module-name cmdoption-m From a5e821c7269ebed6e8b4b4eee3f2247ec41a9788 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 8 Mar 2020 11:49:44 -0700 Subject: [PATCH 1260/2163] bpo-39852: IDLE 'Go to line' deletes selection, updates status (GH-18801) It appears standard that moving the text insert cursor away from a selection clears the selection. Clearing prevents accidental deletion of a possibly off-screen bit of text. The update is for Ln and Col on the status bar. (cherry picked from commit 2522db11df102be3baf25ce9e816ebe8ffdb7fcc) Co-authored-by: Terry Jan Reedy --- Doc/library/idle.rst | 4 ++- Lib/idlelib/NEWS.txt | 3 ++ Lib/idlelib/editor.py | 5 ++- Lib/idlelib/help.html | 33 +++++++++++-------- .../2020-03-06-01-55-14.bpo-39852.QjA1qF.rst | 2 ++ 5 files changed, 31 insertions(+), 16 deletions(-) create mode 100644 Misc/NEWS.d/next/IDLE/2020-03-06-01-55-14.bpo-39852.QjA1qF.rst diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst index fd6e309567de39..b1192e7bb46552 100644 --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -142,7 +142,9 @@ Replace... Open a search-and-replace dialog. Go to Line - Move cursor to the line number requested and make that line visible. + Move the cursor to the beginning of the line requested and make that + line visible. A request past the end of the file goes to the end. + Clear any selection and update the line and column status. Show Completions Open a scrollable list allowing selection of keywords and attributes. See diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 134f843d85e8a6..8493192be9a57f 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,9 @@ Released on 2019-12-16? ====================================== +bpo-39852: Edit "Go to line" now clears any selection, preventing +accidental deletion. It also updates Ln and Col on the status bar. + bpo-39781: Selecting code context lines no longer causes a jump. bpo-39663: Add tests for pyparse find_good_parse_start(). diff --git a/Lib/idlelib/editor.py b/Lib/idlelib/editor.py index 04c786dc5234c2..5b81b52f9196c4 100644 --- a/Lib/idlelib/editor.py +++ b/Lib/idlelib/editor.py @@ -678,8 +678,11 @@ def goto_line_event(self, event): if lineno <= 0: text.bell() return "break" - text.mark_set("insert", "%d.0" % lineno) + + text.tag_remove("sel", "1.0", "end") + text.mark_set("insert", f'{lineno}.0') text.see("insert") + self.set_line_and_column() return "break" def open_module(self): diff --git a/Lib/idlelib/help.html b/Lib/idlelib/help.html index 0b2bdd2e174ccf..424c6b50f339e1 100644 --- a/Lib/idlelib/help.html +++ b/Lib/idlelib/help.html @@ -4,7 +4,7 @@ - IDLE — Python 3.9.0a1 documentation + IDLE — Python 3.9.0a4 documentation @@ -17,7 +17,7 @@ @@ -71,7 +71,7 @@

    Navigation

  • - 3.9.0a1 Documentation » + 3.9.0a4 Documentation »
  • @@ -197,7 +197,9 @@

    Edit menu (Shell and Editor)Completions in the Editing and navigation section below.

    @@ -635,17 +637,20 @@

    Startup failure~/.idlerc/ (~ is one’s home directory). If there is a problem, an error message should be displayed. Leaving aside random disk glitches, this can -be prevented by never editing the files by hand, using the configuration -dialog, under Options, instead Options. Once it happens, the solution may -be to delete one or more of the configuration files.

    +be prevented by never editing the files by hand. Instead, use the +configuration dialog, under Options. Once there is an error in a user +configuration file, the best solution may be to delete it and start over +with the settings dialog.

    If IDLE quits with no message, and it was not started from a console, try -starting from a console (python -m idlelib) and see if a message appears.

    +starting it from a console or terminal (python -m idlelib) and see if +this results in an error message.

    Running user code¶

    @@ -930,7 +935,7 @@

    Navigation

  • - 3.9.0a1 Documentation » + 3.9.0a4 Documentation »
  • @@ -953,7 +958,7 @@

    Navigation

    @@ -947,11 +950,12 @@

    Navigation

  • - 3.10.0a0 Documentation » + 3.10.0a1 Documentation »
  • +
  • @@ -978,11 +982,11 @@

    Navigation



    - Last updated on Sep 22, 2020. + Last updated on Oct 20, 2020. Found a bug?
    - Created using Sphinx 2.1.1. + Created using Sphinx 3.2.1. From aedc94b8e9d0f7700c665d8d1ba9c93df33e63a8 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 15 Dec 2020 19:18:17 -0800 Subject: [PATCH 1880/2163] [3.8] bpo-39416: change word case to not imply ABC (GH-22867) (GH-22869) follow-up to bpo-39416 Co-authored-by: kpinc --- Doc/reference/datamodel.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 083d38cc9f1604..332051e396994e 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -182,7 +182,7 @@ Ellipsis related to mathematical numbers, but subject to the limitations of numerical representation in computers. - The string representations of the Numeric classes, computed by + The string representations of the numeric classes, computed by :meth:`__repr__` and :meth:`__str__`, have the following properties: From 0a24a57888798905e3b8891c59e61ed4f1bfc5a8 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 16 Dec 2020 02:10:32 -0800 Subject: [PATCH 1881/2163] bpo-42644: Validate values in logging.disable() (GH-23786) * bpo-42644: Validate values in logging.disable() Technically make the value of manager a property that checks and convert values assigned to it properly. This has the side effect of making `logging.disable` also accept strings representing the various level of warnings. We want to validate the type of the disable attribute at assignment time, as it is later compared to other levels when emitting warnings and would generate a `TypeError: '>=' not supported between ....` in a different part of the code base, which can make it difficult to track down. When assigned an incorrect value; it will raise a TypeError when the wrong type, or ValueError if an invalid str. Co-authored-by: Andrew Svetlov (cherry picked from commit b32d8b4f9bcd2e7d11240b6b9de0262cf8f5e09d) Co-authored-by: Matthias Bussonnier --- Lib/logging/__init__.py | 8 ++++++++ Lib/test/test_logging.py | 9 +++++++++ .../Library/2020-12-15-10-00-04.bpo-42644.XgLCNx.rst | 3 +++ 3 files changed, 20 insertions(+) create mode 100644 Misc/NEWS.d/next/Library/2020-12-15-10-00-04.bpo-42644.XgLCNx.rst diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py index 79e0787af85968..da3f7b1f73422c 100644 --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -1269,6 +1269,14 @@ def __init__(self, rootnode): self.loggerClass = None self.logRecordFactory = None + @property + def disable(self): + return self._disable + + @disable.setter + def disable(self, value): + self._disable = _checkLevel(value) + def getLogger(self, name): """ Get a logger with the specified name (channel name), creating it diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index b9392aefd44762..3e15db03a43a57 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -4170,6 +4170,15 @@ def test_disable(self): logging.disable(83) self.assertEqual(logging.root.manager.disable, 83) + self.assertRaises(ValueError, logging.disable, "doesnotexists") + + class _NotAnIntOrString: + pass + + self.assertRaises(TypeError, logging.disable, _NotAnIntOrString()) + + logging.disable("WARN") + # test the default value introduced in 3.7 # (Issue #28524) logging.disable() diff --git a/Misc/NEWS.d/next/Library/2020-12-15-10-00-04.bpo-42644.XgLCNx.rst b/Misc/NEWS.d/next/Library/2020-12-15-10-00-04.bpo-42644.XgLCNx.rst new file mode 100644 index 00000000000000..f58b58e4002ada --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-12-15-10-00-04.bpo-42644.XgLCNx.rst @@ -0,0 +1,3 @@ +`logging.disable` will now validate the types and value of its parameter. It +also now accepts strings representing the levels (as does `loging.setLevel`) +instead of only the numerical values. From d549d0b5575b390431b7b26999151f79f74d4516 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 16 Dec 2020 09:57:23 -0800 Subject: [PATCH 1882/2163] bpo-39101: Fixes BaseException hang in IsolatedAsyncioTestCase. (GH-22654) (cherry picked from commit 8374d2ee1589791be8892b00f4bbf8121dde24bd) Co-authored-by: Lisa Roach --- Lib/unittest/async_case.py | 4 +-- Lib/unittest/test/test_async_case.py | 27 +++++++++++++++++++ .../2020-10-11-21-43-03.bpo-39101.-I49Pm.rst | 1 + 3 files changed, 30 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-10-11-21-43-03.bpo-39101.-I49Pm.rst diff --git a/Lib/unittest/async_case.py b/Lib/unittest/async_case.py index 1bc1312c8c2ee9..520213c3727555 100644 --- a/Lib/unittest/async_case.py +++ b/Lib/unittest/async_case.py @@ -102,9 +102,9 @@ async def _asyncioLoopRunner(self, fut): ret = await awaitable if not fut.cancelled(): fut.set_result(ret) - except asyncio.CancelledError: + except (SystemExit, KeyboardInterrupt): raise - except Exception as ex: + except (BaseException, asyncio.CancelledError) as ex: if not fut.cancelled(): fut.set_exception(ex) diff --git a/Lib/unittest/test/test_async_case.py b/Lib/unittest/test/test_async_case.py index 2db441da202a01..d01864b6936ca8 100644 --- a/Lib/unittest/test/test_async_case.py +++ b/Lib/unittest/test/test_async_case.py @@ -190,6 +190,33 @@ async def on_async_cleanup(self, val): 'async_cleanup 2', 'sync_cleanup 1']) + def test_base_exception_from_async_method(self): + events = [] + class Test(unittest.IsolatedAsyncioTestCase): + async def test_base(self): + events.append("test_base") + raise BaseException() + events.append("not it") + + async def test_no_err(self): + events.append("test_no_err") + + async def test_cancel(self): + raise asyncio.CancelledError() + + test = Test("test_base") + output = test.run() + self.assertFalse(output.wasSuccessful()) + + test = Test("test_no_err") + test.run() + self.assertEqual(events, ['test_base', 'test_no_err']) + + test = Test("test_cancel") + output = test.run() + self.assertFalse(output.wasSuccessful()) + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Library/2020-10-11-21-43-03.bpo-39101.-I49Pm.rst b/Misc/NEWS.d/next/Library/2020-10-11-21-43-03.bpo-39101.-I49Pm.rst new file mode 100644 index 00000000000000..a571e8343cde12 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-10-11-21-43-03.bpo-39101.-I49Pm.rst @@ -0,0 +1 @@ +Fixed tests using IsolatedAsyncioTestCase from hanging on BaseExceptions. \ No newline at end of file From dd262ef46d2e2fff4c413cfa5faa9e72cd36220e Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 16 Dec 2020 10:11:51 -0800 Subject: [PATCH 1883/2163] bpo-38323: Add guard clauses in MultiLoopChildWatcher. (GH-22756) This is a trivial refactor in preparation for a fix for bpo-38323. (cherry picked from commit 66d3b589c44fcbcf9afe1e442d9beac3bd8bcd34) Co-authored-by: Chris Jerdonek --- Lib/asyncio/unix_events.py | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py index 1ff8c427da4a66..0420529c5b822a 100644 --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -1152,13 +1152,15 @@ def is_active(self): def close(self): self._callbacks.clear() - if self._saved_sighandler is not None: - handler = signal.getsignal(signal.SIGCHLD) - if handler != self._sig_chld: - logger.warning("SIGCHLD handler was changed by outside code") - else: - signal.signal(signal.SIGCHLD, self._saved_sighandler) - self._saved_sighandler = None + if self._saved_sighandler is None: + return + + handler = signal.getsignal(signal.SIGCHLD) + if handler != self._sig_chld: + logger.warning("SIGCHLD handler was changed by outside code") + else: + signal.signal(signal.SIGCHLD, self._saved_sighandler) + self._saved_sighandler = None def __enter__(self): return self @@ -1185,15 +1187,17 @@ def attach_loop(self, loop): # The reason to do it here is that attach_loop() is called from # unix policy only for the main thread. # Main thread is required for subscription on SIGCHLD signal + if self._saved_sighandler is not None: + return + + self._saved_sighandler = signal.signal(signal.SIGCHLD, self._sig_chld) if self._saved_sighandler is None: - self._saved_sighandler = signal.signal(signal.SIGCHLD, self._sig_chld) - if self._saved_sighandler is None: - logger.warning("Previous SIGCHLD handler was set by non-Python code, " - "restore to default handler on watcher close.") - self._saved_sighandler = signal.SIG_DFL + logger.warning("Previous SIGCHLD handler was set by non-Python code, " + "restore to default handler on watcher close.") + self._saved_sighandler = signal.SIG_DFL - # Set SA_RESTART to limit EINTR occurrences. - signal.siginterrupt(signal.SIGCHLD, False) + # Set SA_RESTART to limit EINTR occurrences. + signal.siginterrupt(signal.SIGCHLD, False) def _do_waitpid_all(self): for pid in list(self._callbacks): From 718bf1a7a1a39ca6f2381a299d00d8318732104a Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 17 Dec 2020 03:14:25 -0800 Subject: [PATCH 1884/2163] bpo-41804: Enhance test_epoll.test_control_and_wait() (GH-23795) (GH-23814) Use shorter timeout and replace send() with sendall(). (cherry picked from commit 79782fe4f8cf73d7fdf8db02073bbadf7ff817b6) Co-authored-by: Victor Stinner Co-authored-by: Victor Stinner --- Lib/test/test_epoll.py | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/Lib/test/test_epoll.py b/Lib/test/test_epoll.py index 8ac0f31d8051c6..e7d58350df86fd 100644 --- a/Lib/test/test_epoll.py +++ b/Lib/test/test_epoll.py @@ -160,44 +160,42 @@ def test_fromfd(self): self.fail("epoll on closed fd didn't raise EBADF") def test_control_and_wait(self): + # create the epoll object client, server = self._connected_pair() - ep = select.epoll(16) ep.register(server.fileno(), select.EPOLLIN | select.EPOLLOUT | select.EPOLLET) ep.register(client.fileno(), select.EPOLLIN | select.EPOLLOUT | select.EPOLLET) + # EPOLLOUT now = time.monotonic() events = ep.poll(1, 4) then = time.monotonic() self.assertFalse(then - now > 0.1, then - now) - events.sort() expected = [(client.fileno(), select.EPOLLOUT), (server.fileno(), select.EPOLLOUT)] - expected.sort() - - self.assertEqual(events, expected) + self.assertEqual(sorted(events), sorted(expected)) - events = ep.poll(timeout=2.1, maxevents=4) + # no event + events = ep.poll(timeout=0.1, maxevents=4) self.assertFalse(events) - client.send(b"Hello!") - server.send(b"world!!!") + # send: EPOLLIN and EPOLLOUT + client.sendall(b"Hello!") + server.sendall(b"world!!!") now = time.monotonic() - events = ep.poll(1, 4) + events = ep.poll(1.0, 4) then = time.monotonic() self.assertFalse(then - now > 0.01) - events.sort() expected = [(client.fileno(), select.EPOLLIN | select.EPOLLOUT), (server.fileno(), select.EPOLLIN | select.EPOLLOUT)] - expected.sort() - - self.assertEqual(events, expected) + self.assertEqual(sorted(events), sorted(expected)) + # unregister, modify ep.unregister(client.fileno()) ep.modify(server.fileno(), select.EPOLLOUT) now = time.monotonic() From 4f65907f38339809532bc06ffc1382c2c09180e2 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 17 Dec 2020 03:16:00 -0800 Subject: [PATCH 1885/2163] bpo-42375: subprocess DragonFlyBSD build update. (GH-23320) (GH-23389) Same as FreeBSD, file descriptors in /dev/fd id from 0 to 63. (cherry picked from commit 13b865f0e17c88b081c23f7f05cf91166d220a50) Co-authored-by: David CARLIER Co-authored-by: David CARLIER --- .../next/Library/2020-11-19-04-13-53.bpo-42375.U8bp4s.rst | 1 + Modules/_posixsubprocess.c | 8 ++++---- 2 files changed, 5 insertions(+), 4 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-11-19-04-13-53.bpo-42375.U8bp4s.rst diff --git a/Misc/NEWS.d/next/Library/2020-11-19-04-13-53.bpo-42375.U8bp4s.rst b/Misc/NEWS.d/next/Library/2020-11-19-04-13-53.bpo-42375.U8bp4s.rst new file mode 100644 index 00000000000000..6d8c80c2f2c0ae --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-11-19-04-13-53.bpo-42375.U8bp4s.rst @@ -0,0 +1 @@ +subprocess module update for DragonFlyBSD support. \ No newline at end of file diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c index e693e532069079..05c051c6125f40 100644 --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -41,7 +41,7 @@ # endif #endif -#if defined(__FreeBSD__) || (defined(__APPLE__) && defined(__MACH__)) +#if defined(__FreeBSD__) || (defined(__APPLE__) && defined(__MACH__)) || defined(__DragonFly__) # define FD_DIR "/dev/fd" #else # define FD_DIR "/proc/self/fd" @@ -88,9 +88,9 @@ _pos_int_from_ascii(const char *name) } -#if defined(__FreeBSD__) +#if defined(__FreeBSD__) || defined(__DragonFly__) /* When /dev/fd isn't mounted it is often a static directory populated - * with 0 1 2 or entries for 0 .. 63 on FreeBSD, NetBSD and OpenBSD. + * with 0 1 2 or entries for 0 .. 63 on FreeBSD, NetBSD, OpenBSD and DragonFlyBSD. * NetBSD and OpenBSD have a /proc fs available (though not necessarily * mounted) and do not have fdescfs for /dev/fd. MacOS X has a devfs * that properly supports /dev/fd. @@ -343,7 +343,7 @@ _close_open_fds_maybe_unsafe(long start_fd, PyObject* py_fds_to_keep) ++start_fd; #endif -#if defined(__FreeBSD__) +#if defined(__FreeBSD__) || defined(__DragonFly__) if (!_is_fdescfs_mounted_on_dev_fd()) proc_fd_dir = NULL; else From 829272e67bbd4b2cc76c01cd20265eb114b392a2 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 17 Dec 2020 03:40:26 -0800 Subject: [PATCH 1886/2163] bpo-42613: Fix freeze.py config directory (GH-23792) Fix freeze.py tool to use the prope config and library directories. (cherry picked from commit 1c653f17cb84d81df3a74ab0b42140d2bb68c5c4) Co-authored-by: Victor Stinner --- .../2020-12-16-09-10-32.bpo-42613.J-jnm5.rst | 2 ++ Tools/freeze/freeze.py | 20 +++++++++---------- 2 files changed, 12 insertions(+), 10 deletions(-) create mode 100644 Misc/NEWS.d/next/Tools-Demos/2020-12-16-09-10-32.bpo-42613.J-jnm5.rst diff --git a/Misc/NEWS.d/next/Tools-Demos/2020-12-16-09-10-32.bpo-42613.J-jnm5.rst b/Misc/NEWS.d/next/Tools-Demos/2020-12-16-09-10-32.bpo-42613.J-jnm5.rst new file mode 100644 index 00000000000000..140ff8255b96b1 --- /dev/null +++ b/Misc/NEWS.d/next/Tools-Demos/2020-12-16-09-10-32.bpo-42613.J-jnm5.rst @@ -0,0 +1,2 @@ +Fix ``freeze.py`` tool to use the prope config and library directories. +Patch by Victor Stinner. diff --git a/Tools/freeze/freeze.py b/Tools/freeze/freeze.py index 83aa508a46a93e..d66e1e2708e758 100755 --- a/Tools/freeze/freeze.py +++ b/Tools/freeze/freeze.py @@ -93,6 +93,7 @@ import getopt import os import sys +import sysconfig # Import the freeze-private modules @@ -226,7 +227,7 @@ def main(): extensions_c = 'frozen_extensions.c' if ishome: print("(Using Python source directory)") - binlib = exec_prefix + configdir = exec_prefix incldir = os.path.join(prefix, 'Include') config_h_dir = exec_prefix config_c_in = os.path.join(prefix, 'Modules', 'config.c.in') @@ -235,22 +236,21 @@ def main(): if win: frozendllmain_c = os.path.join(exec_prefix, 'Pc\\frozen_dllmain.c') else: - binlib = os.path.join(exec_prefix, - 'lib', 'python%s' % version, - 'config-%s' % flagged_version) + configdir = sysconfig.get_config_var('LIBPL') incldir = os.path.join(prefix, 'include', 'python%s' % flagged_version) config_h_dir = os.path.join(exec_prefix, 'include', 'python%s' % flagged_version) - config_c_in = os.path.join(binlib, 'config.c.in') - frozenmain_c = os.path.join(binlib, 'frozenmain.c') - makefile_in = os.path.join(binlib, 'Makefile') - frozendllmain_c = os.path.join(binlib, 'frozen_dllmain.c') + config_c_in = os.path.join(configdir, 'config.c.in') + frozenmain_c = os.path.join(configdir, 'frozenmain.c') + makefile_in = os.path.join(configdir, 'Makefile') + frozendllmain_c = os.path.join(configdir, 'frozen_dllmain.c') + libdir = sysconfig.get_config_var('LIBDIR') supp_sources = [] defines = [] includes = ['-I' + incldir, '-I' + config_h_dir] # sanity check of directories and files - check_dirs = [prefix, exec_prefix, binlib, incldir] + check_dirs = [prefix, exec_prefix, configdir, incldir] if not win: # These are not directories on Windows. check_dirs = check_dirs + extensions @@ -457,7 +457,7 @@ def main(): cflags = ['$(OPT)'] cppflags = defines + includes - libs = [os.path.join(binlib, '$(LDLIBRARY)')] + libs = [os.path.join(libdir, '$(LDLIBRARY)')] somevars = {} if os.path.exists(makefile_in): From cecbaa3a80d5ae28cdd4129d6d2c4395c12a89e4 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 17 Dec 2020 05:19:49 -0800 Subject: [PATCH 1887/2163] bpo-26564: fix obsolete comment in traceback.c (GH-23819) (cherry picked from commit 40125ab3252453bf205ed906e46bf9741c27bf9d) Co-authored-by: Irit Katriel --- Python/traceback.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Python/traceback.c b/Python/traceback.c index 8e2f15e85d6b5d..0aa51ad37f49e2 100644 --- a/Python/traceback.c +++ b/Python/traceback.c @@ -623,7 +623,8 @@ PyTraceBack_Print(PyObject *v, PyObject *f) return err; } -/* Reverse a string. For example, "abcd" becomes "dcba". +/* Format an integer in range [0; 0xffffffff] to decimal and write it + into the file fd. This function is signal safe. */ From 12032cdec5ae1e56d4e4b8206fb2e9cccc5a9f58 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 18 Dec 2020 01:34:24 -0800 Subject: [PATCH 1888/2163] bpo-39096: Format specification documentation fixes for numeric types (GH-23575) (cherry picked from commit 886b2e5c7a2caf87070728dba8f18c3d65e51071) Co-authored-by: Mark Dickinson --- Doc/library/string.rst | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/Doc/library/string.rst b/Doc/library/string.rst index 5542e9b727a6b8..54786d0c2ab0df 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -514,6 +514,8 @@ The available presentation types for :class:`float` and | | this rounds the number to ``p`` significant digits and | | | then formats the result in either fixed-point format | | | or in scientific notation, depending on its magnitude. | + | | A precision of ``0`` is treated as equivalent to a | + | | precision of ``1``. | | | | | | The precise rules are as follows: suppose that the | | | result formatted with presentation type ``'e'`` and | @@ -528,16 +530,19 @@ The available presentation types for :class:`float` and | | removed if there are no remaining digits following it, | | | unless the ``'#'`` option is used. | | | | + | | With no precision given, uses a precision of ``6`` | + | | significant digits for :class:`float`. For | + | | :class:`~decimal.Decimal`, the coefficient of the result | + | | is formed from the coefficient digits of the value; | + | | scientific notation is used for values smaller than | + | | ``1e-6`` in absolute value and values where the place | + | | value of the least significant digit is larger than 1, | + | | and fixed-point notation is used otherwise. | + | | | | | Positive and negative infinity, positive and negative | | | zero, and nans, are formatted as ``inf``, ``-inf``, | | | ``0``, ``-0`` and ``nan`` respectively, regardless of | | | the precision. | - | | | - | | A precision of ``0`` is treated as equivalent to a | - | | precision of ``1``. With no precision given, uses a | - | | precision of ``6`` significant digits for | - | | :class:`float`, and shows all coefficient digits | - | | for :class:`~decimal.Decimal`. | +---------+----------------------------------------------------------+ | ``'G'`` | General format. Same as ``'g'`` except switches to | | | ``'E'`` if the number gets too large. The | @@ -550,12 +555,18 @@ The available presentation types for :class:`float` and | ``'%'`` | Percentage. Multiplies the number by 100 and displays | | | in fixed (``'f'``) format, followed by a percent sign. | +---------+----------------------------------------------------------+ - | None | Similar to ``'g'``, except that fixed-point notation, | - | | when used, has at least one digit past the decimal point.| - | | The default precision is as high as needed to represent | - | | the particular value. The overall effect is to match the | - | | output of :func:`str` as altered by the other format | - | | modifiers. | + | None | For :class:`float` this is the same as ``'g'``, except | + | | that when fixed-point notation is used to format the | + | | result, it always includes at least one digit past the | + | | decimal point. The precision used is as large as needed | + | | to represent the given value faithfully. | + | | | + | | For :class:`~decimal.Decimal`, this is the same as | + | | either ``'g'`` or ``'G'`` depending on the value of | + | | ``context.capitals`` for the current decimal context. | + | | | + | | The overall effect is to match the output of :func:`str` | + | | as altered by the other format modifiers. | +---------+----------------------------------------------------------+ From a3dec9d8ec5ae142946ff6b94947a183d7c48f35 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 18 Dec 2020 09:45:24 -0800 Subject: [PATCH 1889/2163] bpo-41891: ensure asyncio.wait_for waits for task completion (GH-22461) (cherry picked from commit 17ef4319a34f5a2f95e7823dfb5f5b8cff11882d) Co-authored-by: Richard Kojedzinszky --- Lib/asyncio/tasks.py | 5 +- Lib/test/test_asyncio/test_asyncio_waitfor.py | 61 +++++++++++++++++++ .../2020-09-30-13-35-29.bpo-41891.pNAeYI.rst | 1 + 3 files changed, 66 insertions(+), 1 deletion(-) create mode 100644 Lib/test/test_asyncio/test_asyncio_waitfor.py create mode 100644 Misc/NEWS.d/next/Library/2020-09-30-13-35-29.bpo-41891.pNAeYI.rst diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py index 1fa76a161e1e4c..70941325838140 100644 --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -484,7 +484,10 @@ async def wait_for(fut, timeout, *, loop=None): return fut.result() else: fut.remove_done_callback(cb) - fut.cancel() + # We must ensure that the task is not running + # after wait_for() returns. + # See https://bugs.python.org/issue32751 + await _cancel_and_wait(fut, loop=loop) raise if fut.done(): diff --git a/Lib/test/test_asyncio/test_asyncio_waitfor.py b/Lib/test/test_asyncio/test_asyncio_waitfor.py new file mode 100644 index 00000000000000..2ca64abbeb527c --- /dev/null +++ b/Lib/test/test_asyncio/test_asyncio_waitfor.py @@ -0,0 +1,61 @@ +import asyncio +import unittest +import time + +def tearDownModule(): + asyncio.set_event_loop_policy(None) + + +class SlowTask: + """ Task will run for this defined time, ignoring cancel requests """ + TASK_TIMEOUT = 0.2 + + def __init__(self): + self.exited = False + + async def run(self): + exitat = time.monotonic() + self.TASK_TIMEOUT + + while True: + tosleep = exitat - time.monotonic() + if tosleep <= 0: + break + + try: + await asyncio.sleep(tosleep) + except asyncio.CancelledError: + pass + + self.exited = True + +class AsyncioWaitForTest(unittest.TestCase): + + async def atest_asyncio_wait_for_cancelled(self): + t = SlowTask() + + waitfortask = asyncio.create_task(asyncio.wait_for(t.run(), t.TASK_TIMEOUT * 2)) + await asyncio.sleep(0) + waitfortask.cancel() + await asyncio.wait({waitfortask}) + + self.assertTrue(t.exited) + + def test_asyncio_wait_for_cancelled(self): + asyncio.run(self.atest_asyncio_wait_for_cancelled()) + + async def atest_asyncio_wait_for_timeout(self): + t = SlowTask() + + try: + await asyncio.wait_for(t.run(), t.TASK_TIMEOUT / 2) + except asyncio.TimeoutError: + pass + + self.assertTrue(t.exited) + + def test_asyncio_wait_for_timeout(self): + asyncio.run(self.atest_asyncio_wait_for_timeout()) + + +if __name__ == '__main__': + unittest.main() diff --git a/Misc/NEWS.d/next/Library/2020-09-30-13-35-29.bpo-41891.pNAeYI.rst b/Misc/NEWS.d/next/Library/2020-09-30-13-35-29.bpo-41891.pNAeYI.rst new file mode 100644 index 00000000000000..75c25127803153 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-09-30-13-35-29.bpo-41891.pNAeYI.rst @@ -0,0 +1 @@ +Ensure asyncio.wait_for waits for task completion From d21d29ab5b8741da056ac09c49c759b6ccbf264a Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 18 Dec 2020 10:37:57 -0800 Subject: [PATCH 1890/2163] [3.8] bpo-17140: Document multiprocessing's ThreadPool (GH-23812) (GH-23835) Up until now, the `multiprocessing.pool.ThreadPool` class has gone undocumented, despite being a public class in multiprocessing that is included in `multiprocessing.pool.__all__`. (cherry picked from commit 84ebcf271a2cc8bfd1762acb279502b8b6ef236e) Co-authored-by: Matt Wozniski --- Doc/library/multiprocessing.rst | 40 +++++++++++++++++++ .../2020-12-16-21-06-16.bpo-17140.1leSEg.rst | 1 + 2 files changed, 41 insertions(+) create mode 100644 Misc/NEWS.d/next/Documentation/2020-12-16-21-06-16.bpo-17140.1leSEg.rst diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 80bb0857949dc6..041410900dff64 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -2651,6 +2651,46 @@ The :mod:`multiprocessing.dummy` module :mod:`multiprocessing.dummy` replicates the API of :mod:`multiprocessing` but is no more than a wrapper around the :mod:`threading` module. +.. currentmodule:: multiprocessing.pool + +In particular, the ``Pool`` function provided by :mod:`multiprocessing.dummy` +returns an instance of :class:`ThreadPool`, which is a subclass of +:class:`Pool` that supports all the same method calls but uses a pool of +worker threads rather than worker processes. + + +.. class:: ThreadPool([processes[, initializer[, initargs]]]) + + A thread pool object which controls a pool of worker threads to which jobs + can be submitted. :class:`ThreadPool` instances are fully interface + compatible with :class:`Pool` instances, and their resources must also be + properly managed, either by using the pool as a context manager or by + calling :meth:`~multiprocessing.pool.Pool.close` and + :meth:`~multiprocessing.pool.Pool.terminate` manually. + + *processes* is the number of worker threads to use. If *processes* is + ``None`` then the number returned by :func:`os.cpu_count` is used. + + If *initializer* is not ``None`` then each worker process will call + ``initializer(*initargs)`` when it starts. + + Unlike :class:`Pool`, *maxtasksperchild* and *context* cannot be provided. + + .. note:: + + A :class:`ThreadPool` shares the same interface as :class:`Pool`, which + is designed around a pool of processes and predates the introduction of + the :class:`concurrent.futures` module. As such, it inherits some + operations that don't make sense for a pool backed by threads, and it + has its own type for representing the status of asynchronous jobs, + :class:`AsyncResult`, that is not understood by any other libraries. + + Users should generally prefer to use + :class:`concurrent.futures.ThreadPoolExecutor`, which has a simpler + interface that was designed around threads from the start, and which + returns :class:`concurrent.futures.Future` instances that are + compatible with many other libraries, including :mod:`asyncio`. + .. _multiprocessing-programming: diff --git a/Misc/NEWS.d/next/Documentation/2020-12-16-21-06-16.bpo-17140.1leSEg.rst b/Misc/NEWS.d/next/Documentation/2020-12-16-21-06-16.bpo-17140.1leSEg.rst new file mode 100644 index 00000000000000..cb1fd23a56e639 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-12-16-21-06-16.bpo-17140.1leSEg.rst @@ -0,0 +1 @@ +Add documentation for the :class:`multiprocessing.pool.ThreadPool` class. From 80c445cafbdfb16c4a882e3ff6fe28b471aacdfc Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Sat, 19 Dec 2020 16:38:37 +0200 Subject: [PATCH 1891/2163] [3.8] bpo-42630: Improve error reporting in Tkinter for absent default root (GH-23781) (GH-23854) * Tkinter functions and constructors which need a default root window raise now RuntimeError with descriptive message instead of obscure AttributeError or NameError if it is not created yet or cannot be created automatically. * Add tests for all functions which use default root window. * Fix import in the pynche script. (cherry picked from commit 3d569fd6dccf9f582bafaca04d3535094cae393e) --- Lib/idlelib/pyshell.py | 4 +- Lib/test/test_idle.py | 2 +- Lib/tkinter/__init__.py | 51 +++++++----- Lib/tkinter/commondialog.py | 6 +- Lib/tkinter/font.py | 6 +- Lib/tkinter/simpledialog.py | 19 ++--- Lib/tkinter/test/support.py | 27 ++++++ Lib/tkinter/test/test_tkinter/test_font.py | 35 +++++++- Lib/tkinter/test/test_tkinter/test_images.py | 45 +++++++++- Lib/tkinter/test/test_tkinter/test_misc.py | 82 ++++++++++++++++++- .../test/test_tkinter/test_simpledialog.py | 25 ++++++ .../test/test_tkinter/test_variables.py | 17 +++- Lib/tkinter/test/test_tkinter/test_widgets.py | 14 +++- Lib/tkinter/test/test_ttk/test_extensions.py | 26 ++---- Lib/tkinter/test/test_ttk/test_widgets.py | 14 +++- Lib/tkinter/tix.py | 9 +- Lib/tkinter/ttk.py | 7 +- .../2020-12-15-17-51-27.bpo-42630.jf4jBl.rst | 4 + Tools/pynche/PyncheWidget.py | 12 +-- 19 files changed, 317 insertions(+), 88 deletions(-) create mode 100644 Lib/tkinter/test/test_tkinter/test_simpledialog.py create mode 100644 Misc/NEWS.d/next/Library/2020-12-15-17-51-27.bpo-42630.jf4jBl.rst diff --git a/Lib/idlelib/pyshell.py b/Lib/idlelib/pyshell.py index adc302883ae669..6fa138219a24da 100755 --- a/Lib/idlelib/pyshell.py +++ b/Lib/idlelib/pyshell.py @@ -1061,8 +1061,10 @@ def begin(self): (sys.version, sys.platform, self.COPYRIGHT, nosub)) self.text.focus_force() self.showprompt() + # User code should use separate default Tk root window import tkinter - tkinter._default_root = None # 03Jan04 KBK What's this? + tkinter._support_default_root = True + tkinter._default_root = None return True def stop_readline(self): diff --git a/Lib/test/test_idle.py b/Lib/test/test_idle.py index 8bc01deaa33841..310b72c1d72e74 100644 --- a/Lib/test/test_idle.py +++ b/Lib/test/test_idle.py @@ -20,5 +20,5 @@ if __name__ == '__main__': tk.NoDefaultRoot() unittest.main(exit=False) - tk._support_default_root = 1 + tk._support_default_root = True tk._default_root = None diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py index b685901d4df83f..f9ece257841a96 100644 --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -270,7 +270,7 @@ def __repr__(self): ) -_support_default_root = 1 +_support_default_root = True _default_root = None @@ -280,13 +280,26 @@ def NoDefaultRoot(): Call this function to inhibit that the first instance of Tk is used for windows without an explicit parent window. """ - global _support_default_root - _support_default_root = 0 - global _default_root + global _support_default_root, _default_root + _support_default_root = False + # Delete, so any use of _default_root will immediately raise an exception. + # Rebind before deletion, so repeated calls will not fail. _default_root = None del _default_root +def _get_default_root(what=None): + if not _support_default_root: + raise RuntimeError("No master specified and tkinter is " + "configured to not support default root") + if not _default_root: + if what: + raise RuntimeError(f"Too early to {what}: no default root window") + root = Tk() + assert _default_root is root + return _default_root + + def _tkerror(err): """Internal function.""" pass @@ -330,7 +343,7 @@ def __init__(self, master=None, value=None, name=None): raise TypeError("name must be a string") global _varnum if not master: - master = _default_root + master = _get_default_root('create variable') self._root = master._root() self._tk = master.tk if name: @@ -589,7 +602,7 @@ def get(self): def mainloop(n=0): """Run the main loop of Tcl.""" - _default_root.tk.mainloop(n) + _get_default_root('run the main loop').tk.mainloop(n) getint = int @@ -598,9 +611,9 @@ def mainloop(n=0): def getboolean(s): - """Convert true and false to integer values 1 and 0.""" + """Convert Tcl object to True or False.""" try: - return _default_root.tk.getboolean(s) + return _get_default_root('use getboolean()').tk.getboolean(s) except TclError: raise ValueError("invalid literal for getboolean()") @@ -2246,7 +2259,7 @@ def __init__(self, screenName=None, baseName=None, className='Tk', is the name of the widget class.""" self.master = None self.children = {} - self._tkloaded = 0 + self._tkloaded = False # to avoid recursions in the getattr code in case of failure, we # ensure that self.tk is always _something_. self.tk = None @@ -2270,7 +2283,7 @@ def loadtk(self): self._loadtk() def _loadtk(self): - self._tkloaded = 1 + self._tkloaded = True global _default_root # Version sanity checks tk_version = self.tk.getvar('tk_version') @@ -2519,12 +2532,8 @@ class BaseWidget(Misc): def _setup(self, master, cnf): """Internal function. Sets up information about children.""" - if _support_default_root: - global _default_root - if not master: - if not _default_root: - _default_root = Tk() - master = _default_root + if not master: + master = _get_default_root() self.master = master self.tk = master.tk name = None @@ -3988,9 +3997,7 @@ class Image: def __init__(self, imgtype, name=None, cnf={}, master=None, **kw): self.name = None if not master: - master = _default_root - if not master: - raise RuntimeError('Too early to create image') + master = _get_default_root('create image') self.tk = getattr(master, 'tk', master) if not name: Image._last_id += 1 @@ -4144,11 +4151,13 @@ def __init__(self, name=None, cnf={}, master=None, **kw): def image_names(): - return _default_root.tk.splitlist(_default_root.tk.call('image', 'names')) + tk = _get_default_root('use image_names()').tk + return tk.splitlist(tk.call('image', 'names')) def image_types(): - return _default_root.tk.splitlist(_default_root.tk.call('image', 'types')) + tk = _get_default_root('use image_types()').tk + return tk.splitlist(tk.call('image', 'types')) class Spinbox(Widget, XView): diff --git a/Lib/tkinter/commondialog.py b/Lib/tkinter/commondialog.py index c4ec010ee6b34e..b9516c2352613d 100644 --- a/Lib/tkinter/commondialog.py +++ b/Lib/tkinter/commondialog.py @@ -16,10 +16,10 @@ class Dialog: command = None def __init__(self, master=None, **options): - self.master = master + if not master: + master = options.get('parent') + self.master = master self.options = options - if not master and options.get('parent'): - self.master = options['parent'] def _fixoptions(self): pass # hook diff --git a/Lib/tkinter/font.py b/Lib/tkinter/font.py index 136425726ab8ce..31d9afd8f866ca 100644 --- a/Lib/tkinter/font.py +++ b/Lib/tkinter/font.py @@ -68,7 +68,7 @@ def _mkdict(self, args): def __init__(self, root=None, font=None, name=None, exists=False, **options): if not root: - root = tkinter._default_root + root = tkinter._get_default_root('use font') tk = getattr(root, 'tk', root) if font: # get actual settings corresponding to the given font @@ -177,7 +177,7 @@ def metrics(self, *options, **kw): def families(root=None, displayof=None): "Get font families (as a tuple)" if not root: - root = tkinter._default_root + root = tkinter._get_default_root('use font.families()') args = () if displayof: args = ('-displayof', displayof) @@ -187,7 +187,7 @@ def families(root=None, displayof=None): def names(root=None): "Get names of defined fonts (as a tuple)" if not root: - root = tkinter._default_root + root = tkinter._get_default_root('use font.names()') return root.tk.splitlist(root.tk.call("font", "names")) diff --git a/Lib/tkinter/simpledialog.py b/Lib/tkinter/simpledialog.py index 85244171117b61..b882d47c961bdb 100644 --- a/Lib/tkinter/simpledialog.py +++ b/Lib/tkinter/simpledialog.py @@ -24,9 +24,7 @@ """ from tkinter import * -from tkinter import messagebox - -import tkinter # used at _QueryDialog for tkinter._default_root +from tkinter import messagebox, _get_default_root class SimpleDialog: @@ -128,13 +126,17 @@ def __init__(self, parent, title = None): title -- the dialog title ''' - Toplevel.__init__(self, parent) + master = parent + if not master: + master = _get_default_root('create dialog window') + + Toplevel.__init__(self, master) self.withdraw() # remain invisible for now - # If the master is not viewable, don't + # If the parent is not viewable, don't # make the child transient, or else it # would be opened withdrawn - if parent.winfo_viewable(): + if parent is not None and parent.winfo_viewable(): self.transient(parent) if title: @@ -155,7 +157,7 @@ def __init__(self, parent, title = None): self.protocol("WM_DELETE_WINDOW", self.cancel) - if self.parent is not None: + if parent is not None: self.geometry("+%d+%d" % (parent.winfo_rootx()+50, parent.winfo_rooty()+50)) @@ -259,9 +261,6 @@ def __init__(self, title, prompt, minvalue = None, maxvalue = None, parent = None): - if not parent: - parent = tkinter._default_root - self.prompt = prompt self.minvalue = minvalue self.maxvalue = maxvalue diff --git a/Lib/tkinter/test/support.py b/Lib/tkinter/test/support.py index 467a0b66c265c0..dbc47a81e65157 100644 --- a/Lib/tkinter/test/support.py +++ b/Lib/tkinter/test/support.py @@ -36,6 +36,33 @@ def tearDown(self): w.destroy() self.root.withdraw() + +class AbstractDefaultRootTest: + + def setUp(self): + self._old_support_default_root = tkinter._support_default_root + destroy_default_root() + tkinter._support_default_root = True + self.wantobjects = tkinter.wantobjects + + def tearDown(self): + destroy_default_root() + tkinter._default_root = None + tkinter._support_default_root = self._old_support_default_root + + def _test_widget(self, constructor): + # no master passing + x = constructor() + self.assertIsNotNone(tkinter._default_root) + self.assertIs(x.master, tkinter._default_root) + self.assertIs(x.tk, tkinter._default_root.tk) + x.destroy() + destroy_default_root() + tkinter.NoDefaultRoot() + self.assertRaises(RuntimeError, constructor) + self.assertFalse(hasattr(tkinter, '_default_root')) + + def destroy_default_root(): if getattr(tkinter, '_default_root', None): tkinter._default_root.update_idletasks() diff --git a/Lib/tkinter/test/test_tkinter/test_font.py b/Lib/tkinter/test/test_tkinter/test_font.py index 97cd87ccdc81dd..2ea59f17483423 100644 --- a/Lib/tkinter/test/test_tkinter/test_font.py +++ b/Lib/tkinter/test/test_tkinter/test_font.py @@ -2,7 +2,7 @@ import tkinter from tkinter import font from test.support import requires, run_unittest, gc_collect -from tkinter.test.support import AbstractTkTest +from tkinter.test.support import AbstractTkTest, AbstractDefaultRootTest requires('gui') @@ -100,7 +100,38 @@ def test_names(self): self.assertTrue(name) self.assertIn(fontname, names) -tests_gui = (FontTest, ) + +class DefaultRootTest(AbstractDefaultRootTest, unittest.TestCase): + + def test_families(self): + self.assertRaises(RuntimeError, font.families) + root = tkinter.Tk() + families = font.families() + self.assertIsInstance(families, tuple) + self.assertTrue(families) + for family in families: + self.assertIsInstance(family, str) + self.assertTrue(family) + root.destroy() + tkinter.NoDefaultRoot() + self.assertRaises(RuntimeError, font.families) + + def test_names(self): + self.assertRaises(RuntimeError, font.names) + root = tkinter.Tk() + names = font.names() + self.assertIsInstance(names, tuple) + self.assertTrue(names) + for name in names: + self.assertIsInstance(name, str) + self.assertTrue(name) + self.assertIn(fontname, names) + root.destroy() + tkinter.NoDefaultRoot() + self.assertRaises(RuntimeError, font.names) + + +tests_gui = (FontTest, DefaultRootTest) if __name__ == "__main__": run_unittest(*tests_gui) diff --git a/Lib/tkinter/test/test_tkinter/test_images.py b/Lib/tkinter/test/test_tkinter/test_images.py index 2805d35a1f5b1b..94bba8518fdaa6 100644 --- a/Lib/tkinter/test/test_tkinter/test_images.py +++ b/Lib/tkinter/test/test_tkinter/test_images.py @@ -1,7 +1,7 @@ import unittest import tkinter from test import support -from tkinter.test.support import AbstractTkTest, requires_tcl +from tkinter.test.support import AbstractTkTest, AbstractDefaultRootTest, requires_tcl support.requires('gui') @@ -19,6 +19,47 @@ def test_image_names(self): self.assertIsInstance(image_names, tuple) +class DefaultRootTest(AbstractDefaultRootTest, unittest.TestCase): + + def test_image_types(self): + self.assertRaises(RuntimeError, tkinter.image_types) + root = tkinter.Tk() + image_types = tkinter.image_types() + self.assertIsInstance(image_types, tuple) + self.assertIn('photo', image_types) + self.assertIn('bitmap', image_types) + root.destroy() + tkinter.NoDefaultRoot() + self.assertRaises(RuntimeError, tkinter.image_types) + + def test_image_names(self): + self.assertRaises(RuntimeError, tkinter.image_names) + root = tkinter.Tk() + image_names = tkinter.image_names() + self.assertIsInstance(image_names, tuple) + root.destroy() + tkinter.NoDefaultRoot() + self.assertRaises(RuntimeError, tkinter.image_names) + + def test_image_create_bitmap(self): + self.assertRaises(RuntimeError, tkinter.BitmapImage) + root = tkinter.Tk() + image = tkinter.BitmapImage() + self.assertIn(image.name, tkinter.image_names()) + root.destroy() + tkinter.NoDefaultRoot() + self.assertRaises(RuntimeError, tkinter.BitmapImage) + + def test_image_create_photo(self): + self.assertRaises(RuntimeError, tkinter.PhotoImage) + root = tkinter.Tk() + image = tkinter.PhotoImage() + self.assertIn(image.name, tkinter.image_names()) + root.destroy() + tkinter.NoDefaultRoot() + self.assertRaises(RuntimeError, tkinter.PhotoImage) + + class BitmapImageTest(AbstractTkTest, unittest.TestCase): @classmethod @@ -330,7 +371,7 @@ def test_transparency(self): self.assertEqual(image.transparency_get(4, 6), False) -tests_gui = (MiscTest, BitmapImageTest, PhotoImageTest,) +tests_gui = (MiscTest, DefaultRootTest, BitmapImageTest, PhotoImageTest,) if __name__ == "__main__": support.run_unittest(*tests_gui) diff --git a/Lib/tkinter/test/test_tkinter/test_misc.py b/Lib/tkinter/test/test_tkinter/test_misc.py index 350f561bab4065..49fb6b16de09eb 100644 --- a/Lib/tkinter/test/test_tkinter/test_misc.py +++ b/Lib/tkinter/test/test_tkinter/test_misc.py @@ -1,7 +1,7 @@ import unittest import tkinter from test import support -from tkinter.test.support import AbstractTkTest +from tkinter.test.support import AbstractTkTest, AbstractDefaultRootTest support.requires('gui') @@ -227,7 +227,85 @@ def test_event_repr(self): " num=3 delta=-1 focus=True" " x=10 y=20 width=300 height=200>") -tests_gui = (MiscTest, ) + def test_getboolean(self): + for v in 'true', 'yes', 'on', '1', 't', 'y', 1, True: + self.assertIs(self.root.getboolean(v), True) + for v in 'false', 'no', 'off', '0', 'f', 'n', 0, False: + self.assertIs(self.root.getboolean(v), False) + self.assertRaises(ValueError, self.root.getboolean, 'yea') + self.assertRaises(ValueError, self.root.getboolean, '') + self.assertRaises(TypeError, self.root.getboolean, None) + self.assertRaises(TypeError, self.root.getboolean, ()) + + def test_mainloop(self): + log = [] + def callback(): + log.append(1) + self.root.after(100, self.root.quit) + self.root.after(100, callback) + self.root.mainloop(1) + self.assertEqual(log, []) + self.root.mainloop(0) + self.assertEqual(log, [1]) + self.assertTrue(self.root.winfo_exists()) + + +class DefaultRootTest(AbstractDefaultRootTest, unittest.TestCase): + + def test_default_root(self): + self.assertIs(tkinter._support_default_root, True) + self.assertIsNone(tkinter._default_root) + root = tkinter.Tk() + root2 = tkinter.Tk() + root3 = tkinter.Tk() + self.assertIs(tkinter._default_root, root) + root2.destroy() + self.assertIs(tkinter._default_root, root) + root.destroy() + self.assertIsNone(tkinter._default_root) + root3.destroy() + self.assertIsNone(tkinter._default_root) + + def test_no_default_root(self): + self.assertIs(tkinter._support_default_root, True) + self.assertIsNone(tkinter._default_root) + root = tkinter.Tk() + self.assertIs(tkinter._default_root, root) + tkinter.NoDefaultRoot() + self.assertIs(tkinter._support_default_root, False) + self.assertFalse(hasattr(tkinter, '_default_root')) + # repeated call is no-op + tkinter.NoDefaultRoot() + self.assertIs(tkinter._support_default_root, False) + self.assertFalse(hasattr(tkinter, '_default_root')) + root.destroy() + self.assertIs(tkinter._support_default_root, False) + self.assertFalse(hasattr(tkinter, '_default_root')) + root = tkinter.Tk() + self.assertIs(tkinter._support_default_root, False) + self.assertFalse(hasattr(tkinter, '_default_root')) + root.destroy() + + def test_getboolean(self): + self.assertRaises(RuntimeError, tkinter.getboolean, '1') + root = tkinter.Tk() + self.assertIs(tkinter.getboolean('1'), True) + self.assertRaises(ValueError, tkinter.getboolean, 'yea') + root.destroy() + tkinter.NoDefaultRoot() + self.assertRaises(RuntimeError, tkinter.getboolean, '1') + + def test_mainloop(self): + self.assertRaises(RuntimeError, tkinter.mainloop) + root = tkinter.Tk() + root.after_idle(root.quit) + tkinter.mainloop() + root.destroy() + tkinter.NoDefaultRoot() + self.assertRaises(RuntimeError, tkinter.mainloop) + + +tests_gui = (MiscTest, DefaultRootTest) if __name__ == "__main__": support.run_unittest(*tests_gui) diff --git a/Lib/tkinter/test/test_tkinter/test_simpledialog.py b/Lib/tkinter/test/test_tkinter/test_simpledialog.py new file mode 100644 index 00000000000000..911917258806d1 --- /dev/null +++ b/Lib/tkinter/test/test_tkinter/test_simpledialog.py @@ -0,0 +1,25 @@ +import unittest +import tkinter +from test.support import requires, run_unittest, swap_attr +from tkinter.test.support import AbstractDefaultRootTest +from tkinter.simpledialog import Dialog, askinteger + +requires('gui') + + +class DefaultRootTest(AbstractDefaultRootTest, unittest.TestCase): + + def test_askinteger(self): + self.assertRaises(RuntimeError, askinteger, "Go To Line", "Line number") + root = tkinter.Tk() + with swap_attr(Dialog, 'wait_window', lambda self, w: w.destroy()): + askinteger("Go To Line", "Line number") + root.destroy() + tkinter.NoDefaultRoot() + self.assertRaises(RuntimeError, askinteger, "Go To Line", "Line number") + + +tests_gui = (DefaultRootTest,) + +if __name__ == "__main__": + run_unittest(*tests_gui) diff --git a/Lib/tkinter/test/test_tkinter/test_variables.py b/Lib/tkinter/test/test_tkinter/test_variables.py index 2eb1e12671d22e..e7b24a818f1501 100644 --- a/Lib/tkinter/test/test_tkinter/test_variables.py +++ b/Lib/tkinter/test/test_tkinter/test_variables.py @@ -1,7 +1,9 @@ import unittest import gc +import tkinter from tkinter import (Variable, StringVar, IntVar, DoubleVar, BooleanVar, Tcl, TclError) +from tkinter.test.support import AbstractDefaultRootTest class Var(Variable): @@ -301,8 +303,21 @@ def test_invalid_value_domain(self): v.get() +class DefaultRootTest(AbstractDefaultRootTest, unittest.TestCase): + + def test_variable(self): + self.assertRaises(RuntimeError, Variable) + root = tkinter.Tk() + v = Variable() + v.set("value") + self.assertEqual(v.get(), "value") + root.destroy() + tkinter.NoDefaultRoot() + self.assertRaises(RuntimeError, Variable) + + tests_gui = (TestVariable, TestStringVar, TestIntVar, - TestDoubleVar, TestBooleanVar) + TestDoubleVar, TestBooleanVar, DefaultRootTest) if __name__ == "__main__": diff --git a/Lib/tkinter/test/test_tkinter/test_widgets.py b/Lib/tkinter/test/test_tkinter/test_widgets.py index b6f792d6c2cf85..3659b9457623b4 100644 --- a/Lib/tkinter/test/test_tkinter/test_widgets.py +++ b/Lib/tkinter/test/test_tkinter/test_widgets.py @@ -6,7 +6,8 @@ from test.support import requires from tkinter.test.support import (tcl_version, requires_tcl, - get_tk_patchlevel, widget_eq) + get_tk_patchlevel, widget_eq, + AbstractDefaultRootTest) from tkinter.test.widget_tests import ( add_standard_options, noconv, pixels_round, AbstractWidgetTest, StandardOptionsTests, IntegerSizeTests, PixelSizeTests, @@ -1298,12 +1299,21 @@ def test_aspect(self): self.checkIntegerParam(widget, 'aspect', 250, 0, -300) +class DefaultRootTest(AbstractDefaultRootTest, unittest.TestCase): + + def test_frame(self): + self._test_widget(tkinter.Frame) + + def test_label(self): + self._test_widget(tkinter.Label) + + tests_gui = ( ButtonTest, CanvasTest, CheckbuttonTest, EntryTest, FrameTest, LabelFrameTest,LabelTest, ListboxTest, MenubuttonTest, MenuTest, MessageTest, OptionMenuTest, PanedWindowTest, RadiobuttonTest, ScaleTest, ScrollbarTest, - SpinboxTest, TextTest, ToplevelTest, + SpinboxTest, TextTest, ToplevelTest, DefaultRootTest, ) if __name__ == '__main__': diff --git a/Lib/tkinter/test/test_ttk/test_extensions.py b/Lib/tkinter/test/test_ttk/test_extensions.py index 6937ba1ca9be41..1a70e0befe6234 100644 --- a/Lib/tkinter/test/test_ttk/test_extensions.py +++ b/Lib/tkinter/test/test_ttk/test_extensions.py @@ -2,8 +2,8 @@ import unittest import tkinter from tkinter import ttk -from test.support import requires, run_unittest, swap_attr -from tkinter.test.support import AbstractTkTest, destroy_default_root +from test.support import requires, run_unittest +from tkinter.test.support import AbstractTkTest, AbstractDefaultRootTest requires('gui') @@ -46,20 +46,6 @@ def test_widget_destroy(self): if hasattr(sys, 'last_type'): self.assertNotEqual(sys.last_type, tkinter.TclError) - - def test_initialization_no_master(self): - # no master passing - with swap_attr(tkinter, '_default_root', None), \ - swap_attr(tkinter, '_support_default_root', True): - try: - x = ttk.LabeledScale() - self.assertIsNotNone(tkinter._default_root) - self.assertEqual(x.master, tkinter._default_root) - self.assertEqual(x.tk, tkinter._default_root.tk) - x.destroy() - finally: - destroy_default_root() - def test_initialization(self): # master passing master = tkinter.Frame(self.root) @@ -311,7 +297,13 @@ def test_unique_radiobuttons(self): optmenu2.destroy() -tests_gui = (LabeledScaleTest, OptionMenuTest) +class DefaultRootTest(AbstractDefaultRootTest, unittest.TestCase): + + def test_labeledscale(self): + self._test_widget(ttk.LabeledScale) + + +tests_gui = (LabeledScaleTest, OptionMenuTest, DefaultRootTest) if __name__ == "__main__": run_unittest(*tests_gui) diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index 157ef0e8f87bb5..de30e2476b4eb9 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -6,7 +6,7 @@ from tkinter.test.test_ttk.test_functions import MockTclObj from tkinter.test.support import (AbstractTkTest, tcl_version, get_tk_patchlevel, - simulate_mouse_click) + simulate_mouse_click, AbstractDefaultRootTest) from tkinter.test.widget_tests import (add_standard_options, noconv, AbstractWidgetTest, StandardOptionsTests, IntegerSizeTests, PixelSizeTests, setUpModule) @@ -1860,12 +1860,22 @@ class SizegripTest(AbstractWidgetTest, unittest.TestCase): def create(self, **kwargs): return ttk.Sizegrip(self.root, **kwargs) + +class DefaultRootTest(AbstractDefaultRootTest, unittest.TestCase): + + def test_frame(self): + self._test_widget(ttk.Frame) + + def test_label(self): + self._test_widget(ttk.Label) + + tests_gui = ( ButtonTest, CheckbuttonTest, ComboboxTest, EntryTest, FrameTest, LabelFrameTest, LabelTest, MenubuttonTest, NotebookTest, PanedWindowTest, ProgressbarTest, RadiobuttonTest, ScaleTest, ScrollbarTest, SeparatorTest, - SizegripTest, SpinboxTest, TreeviewTest, WidgetTest, + SizegripTest, SpinboxTest, TreeviewTest, WidgetTest, DefaultRootTest, ) if __name__ == "__main__": diff --git a/Lib/tkinter/tix.py b/Lib/tkinter/tix.py index ac545502e45c30..ef1e7406bc1ae0 100644 --- a/Lib/tkinter/tix.py +++ b/Lib/tkinter/tix.py @@ -387,9 +387,7 @@ def config_all(self, option, value): # These are missing from Tkinter def image_create(self, imgtype, cnf={}, master=None, **kw): if not master: - master = tkinter._default_root - if not master: - raise RuntimeError('Too early to create image') + master = self if kw and cnf: cnf = _cnfmerge((cnf, kw)) elif kw: cnf = kw options = () @@ -475,10 +473,7 @@ def __init__(self, itemtype, cnf={}, *, master=None, **kw): elif 'refwindow' in cnf: master = cnf['refwindow'] else: - master = tkinter._default_root - if not master: - raise RuntimeError("Too early to create display style: " - "no root window") + master = tkinter._get_default_root('create display style') self.tk = master.tk self.stylename = self.tk.call('tixDisplayStyle', itemtype, *self._options(cnf,kw) ) diff --git a/Lib/tkinter/ttk.py b/Lib/tkinter/ttk.py index f3a2f7660f30b8..ab7aeb15e8ff22 100644 --- a/Lib/tkinter/ttk.py +++ b/Lib/tkinter/ttk.py @@ -349,12 +349,7 @@ def setup_master(master=None): If it is not allowed to use the default root and master is None, RuntimeError is raised.""" if master is None: - if tkinter._support_default_root: - master = tkinter._default_root or tkinter.Tk() - else: - raise RuntimeError( - "No master specified and tkinter is " - "configured to not support default root") + master = tkinter._get_default_root() return master diff --git a/Misc/NEWS.d/next/Library/2020-12-15-17-51-27.bpo-42630.jf4jBl.rst b/Misc/NEWS.d/next/Library/2020-12-15-17-51-27.bpo-42630.jf4jBl.rst new file mode 100644 index 00000000000000..4b4a520931fda4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-12-15-17-51-27.bpo-42630.jf4jBl.rst @@ -0,0 +1,4 @@ +:mod:`tkinter` functions and constructors which need a default root window +raise now :exc:`RuntimeError` with descriptive message instead of obscure +:exc:`AttributeError` or :exc:`NameError` if it is not created yet or cannot +be created automatically. diff --git a/Tools/pynche/PyncheWidget.py b/Tools/pynche/PyncheWidget.py index 364f22b0b2095c..59b41373f64795 100644 --- a/Tools/pynche/PyncheWidget.py +++ b/Tools/pynche/PyncheWidget.py @@ -36,15 +36,11 @@ def __init__(self, version, switchboard, master=None, extrapath=[]): else: # Is there already a default root for Tk, say because we're # running under Guido's IDE? :-) Two conditions say no, either the - # import fails or _default_root is None. - tkroot = None - try: - from Tkinter import _default_root - tkroot = self.__tkroot = _default_root - except ImportError: - pass + # _default_root is None or it is unset. + tkroot = getattr(tkinter, '_default_root', None) if not tkroot: - tkroot = self.__tkroot = Tk(className='Pynche') + tkroot = Tk(className='Pynche') + self.__tkroot = tkroot # but this isn't our top level widget, so make it invisible tkroot.withdraw() # create the menubar From 24862b02dfd1e2843727f28fa2ba05828fdfa8de Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 19 Dec 2020 16:02:58 -0800 Subject: [PATCH 1892/2163] bpo-41724: Explain when the conversion is not possible with detect_types enabled (GH-23855) (GH-23863) * Explain when the conversion is not possible with detect_types enabled (cherry picked from commit 09a36cdfb7c22f44df45b44e5561776206bcedfb) Co-authored-by: sblondon --- Doc/library/sqlite3.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 314d3a58e2759a..e493324495890e 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -197,7 +197,9 @@ Module functions and constants *detect_types* defaults to 0 (i. e. off, no type detection), you can set it to any combination of :const:`PARSE_DECLTYPES` and :const:`PARSE_COLNAMES` to turn - type detection on. + type detection on. Due to SQLite behaviour, types can't be detected for generated + fields (for example ``max(data)``), even when *detect_types* parameter is set. In + such case, the returned type is :class:`str`. By default, *check_same_thread* is :const:`True` and only the creating thread may use the connection. If set :const:`False`, the returned connection may be shared From b01091a3e71e6636d2df4db45920e820cdf7df3b Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 19 Dec 2020 19:17:42 -0800 Subject: [PATCH 1893/2163] bpo-42604: always set EXT_SUFFIX=${SOABI}${SHLIB_SUFFIX} when using configure (GH-23708) Now all platforms use a value for the "EXT_SUFFIX" build variable derived from SOABI (for instance in FreeBSD, "EXT_SUFFIX" is now ".cpython-310d.so" instead of ".so"). Previously only Linux, Mac and VxWorks were using a value for "EXT_SUFFIX" that included "SOABI". Co-authored-by: Pablo Galindo (cherry picked from commit a44ce6c9f725d336aea51a946b42769f29fed613) Co-authored-by: Matti Picus --- .../next/Build/2020-12-20-02-35-28.bpo-42604.gRd89w.rst | 4 ++++ configure | 8 +------- configure.ac | 7 +------ 3 files changed, 6 insertions(+), 13 deletions(-) create mode 100644 Misc/NEWS.d/next/Build/2020-12-20-02-35-28.bpo-42604.gRd89w.rst diff --git a/Misc/NEWS.d/next/Build/2020-12-20-02-35-28.bpo-42604.gRd89w.rst b/Misc/NEWS.d/next/Build/2020-12-20-02-35-28.bpo-42604.gRd89w.rst new file mode 100644 index 00000000000000..caaada41cf9bad --- /dev/null +++ b/Misc/NEWS.d/next/Build/2020-12-20-02-35-28.bpo-42604.gRd89w.rst @@ -0,0 +1,4 @@ +Now all platforms use a value for the "EXT_SUFFIX" build variable derived +from SOABI (for instance in freeBSD, "EXT_SUFFIX" is now ".cpython-310d.so" +instead of ".so"). Previosuly only Linux, Mac and VxWorks were using a value +for "EXT_SUFFIX" that included "SOABI". diff --git a/configure b/configure index 15ea8cdfdc0f06..c164d68c4e5023 100755 --- a/configure +++ b/configure @@ -15215,13 +15215,7 @@ _ACEOF fi - -case $ac_sys_system in - Linux*|GNU*|Darwin|VxWorks) - EXT_SUFFIX=.${SOABI}${SHLIB_SUFFIX};; - *) - EXT_SUFFIX=${SHLIB_SUFFIX};; -esac +EXT_SUFFIX=.${SOABI}${SHLIB_SUFFIX} { $as_echo "$as_me:${as_lineno-$LINENO}: checking LDVERSION" >&5 $as_echo_n "checking LDVERSION... " >&6; } diff --git a/configure.ac b/configure.ac index c93ba0ee7dc0e0..fc082a3cd28319 100644 --- a/configure.ac +++ b/configure.ac @@ -4674,12 +4674,7 @@ if test "$Py_DEBUG" = 'true' -a "$with_trace_refs" != "yes"; then fi AC_SUBST(EXT_SUFFIX) -case $ac_sys_system in - Linux*|GNU*|Darwin|VxWorks) - EXT_SUFFIX=.${SOABI}${SHLIB_SUFFIX};; - *) - EXT_SUFFIX=${SHLIB_SUFFIX};; -esac +EXT_SUFFIX=.${SOABI}${SHLIB_SUFFIX} AC_MSG_CHECKING(LDVERSION) LDVERSION='$(VERSION)$(ABIFLAGS)' From 81f706d2db0f57c4fdd747df6e0a4cffcbc54704 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 20 Dec 2020 13:18:40 -0800 Subject: [PATCH 1894/2163] bpo-42669: Document that `except` rejects nested tuples (GH-23822) (GH-23871) In Python 2, it was possible to use `except` with a nested tuple, and occasionally natural. For example, `zope.formlib.interfaces.InputErrors` is a tuple of several exception classes, and one might reasonably think to do something like this: try: self.getInputValue() return True except (InputErrors, SomethingElse): return False As of Python 3.0, this raises `TypeError: catching classes that do not inherit from BaseException is not allowed` instead: one must instead either break it up into multiple `except` clauses or flatten the tuple. However, the reference documentation was never updated to match this new restriction. Make it clear that the definition is no longer recursive. Automerge-Triggered-By: GH:ericvsmith (cherry picked from commit c95f8bc2700b42f4568886505a819816c9b0ba28) Co-authored-by: Colin Watson Co-authored-by: Colin Watson --- Doc/reference/compound_stmts.rst | 3 ++- Misc/ACKS | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 6900c7ac7d0714..61037d97a0c3cc 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -254,7 +254,8 @@ present, must be last; it matches any exception. For an except clause with an expression, that expression is evaluated, and the clause matches the exception if the resulting object is "compatible" with the exception. An object is compatible with an exception if it is the class or a base class of the exception -object or a tuple containing an item compatible with the exception. +object, or a tuple containing an item that is the class or a base class of +the exception object. If no except clause matches the exception, the search for an exception handler continues in the surrounding code and on the invocation stack. [#]_ diff --git a/Misc/ACKS b/Misc/ACKS index 8afc0a8e46d571..6ae882479d4363 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1792,6 +1792,7 @@ Zachary Ware Barry Warsaw Steve Waterbury Bob Watson +Colin Watson David Watson Aaron Watters Henrik Weber From a3537716a582a3fd98bc2cd20dc6ec287a4730e8 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 21 Dec 2020 06:28:47 -0800 Subject: [PATCH 1895/2163] Fix typo in docstring (GH-23515) (cherry picked from commit 711381dfb09fbd434cc3b404656f7fd306161a64) Co-authored-by: Fernando Toledo <42938011+fernandohtr@users.noreply.github.com> --- Lib/http/client.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Lib/http/client.py b/Lib/http/client.py index c2ad0471bfee50..16afc871ea6d7a 100644 --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -846,7 +846,7 @@ def set_tunnel(self, host, port=None, headers=None): the endpoint passed to `set_tunnel`. This done by sending an HTTP CONNECT request to the proxy server when the connection is established. - This method must be called before the HTML connection has been + This method must be called before the HTTP connection has been established. The headers argument should be a mapping of extra HTTP headers to send From 6503f05dd59e26a9986bdea097b3da9b3546f45b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Mon, 21 Dec 2020 17:25:24 +0100 Subject: [PATCH 1896/2163] Python 3.8.7 --- Include/patchlevel.h | 6 +- Lib/pydoc_data/topics.py | 122 ++++++----- Misc/NEWS.d/3.8.7.rst | 195 ++++++++++++++++++ .../2020-12-13-14-43-10.bpo-42598.7ipr5H.rst | 2 - .../2020-12-20-02-35-28.bpo-42604.gRd89w.rst | 4 - .../2020-12-02-20-23-31.bpo-42536.Kx3ZOu.rst | 26 --- .../2020-12-04-17-17-44.bpo-32381.NY5t2S.rst | 3 - .../2020-12-16-21-06-16.bpo-17140.1leSEg.rst | 1 - .../2020-03-29-21-32-00.bpo-40084.MCYwcv.rst | 1 - .../2020-07-13-19-43-11.bpo-40219.MUoJEP.rst | 1 - .../2020-09-30-13-35-29.bpo-41891.pNAeYI.rst | 1 - .../2020-10-01-16-17-11.bpo-41889.qLkNh8.rst | 1 - .../2020-10-02-10-19-49.bpo-41907.wiIEsz.rst | 1 - .../2020-10-11-21-43-03.bpo-39101.-I49Pm.rst | 1 - .../2020-10-20-08-28-26.bpo-39825.n6KnG0.rst | 5 - .../2020-11-19-04-13-53.bpo-42375.U8bp4s.rst | 1 - .../2020-12-14-08-23-57.bpo-36541.qdEtZv.rst | 2 - .../2020-12-15-10-00-04.bpo-42644.XgLCNx.rst | 3 - .../2020-12-15-17-51-27.bpo-42630.jf4jBl.rst | 4 - .../2020-12-16-09-10-32.bpo-42613.J-jnm5.rst | 2 - README.rst | 4 +- 21 files changed, 268 insertions(+), 118 deletions(-) create mode 100644 Misc/NEWS.d/3.8.7.rst delete mode 100644 Misc/NEWS.d/next/Build/2020-12-13-14-43-10.bpo-42598.7ipr5H.rst delete mode 100644 Misc/NEWS.d/next/Build/2020-12-20-02-35-28.bpo-42604.gRd89w.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-12-02-20-23-31.bpo-42536.Kx3ZOu.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-12-04-17-17-44.bpo-32381.NY5t2S.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2020-12-16-21-06-16.bpo-17140.1leSEg.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-03-29-21-32-00.bpo-40084.MCYwcv.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-07-13-19-43-11.bpo-40219.MUoJEP.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-09-30-13-35-29.bpo-41891.pNAeYI.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-10-01-16-17-11.bpo-41889.qLkNh8.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-10-02-10-19-49.bpo-41907.wiIEsz.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-10-11-21-43-03.bpo-39101.-I49Pm.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-10-20-08-28-26.bpo-39825.n6KnG0.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-11-19-04-13-53.bpo-42375.U8bp4s.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-12-14-08-23-57.bpo-36541.qdEtZv.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-12-15-10-00-04.bpo-42644.XgLCNx.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-12-15-17-51-27.bpo-42630.jf4jBl.rst delete mode 100644 Misc/NEWS.d/next/Tools-Demos/2020-12-16-09-10-32.bpo-42613.J-jnm5.rst diff --git a/Include/patchlevel.h b/Include/patchlevel.h index 0fe82eec524351..de3694e4821d71 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -19,11 +19,11 @@ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 8 #define PY_MICRO_VERSION 7 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA -#define PY_RELEASE_SERIAL 1 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL +#define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.8.7rc1+" +#define PY_VERSION "3.8.7" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index 93b54c8ef08ede..723a9b8ac57bc1 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Mon Dec 7 15:10:25 2020 +# Autogenerated by Sphinx on Mon Dec 21 17:22:46 2020 topics = {'assert': 'The "assert" statement\n' '**********************\n' '\n' @@ -2376,8 +2376,9 @@ 'compatible\n' 'with an exception if it is the class or a base class of the ' 'exception\n' - 'object or a tuple containing an item compatible with the ' - 'exception.\n' + 'object, or a tuple containing an item that is the class or a ' + 'base\n' + 'class of the exception object.\n' '\n' 'If no except clause matches the exception, the search for an ' 'exception\n' @@ -5512,44 +5513,51 @@ ' | | formats the result in either fixed-point ' 'format or in |\n' ' | | scientific notation, depending on its ' - 'magnitude. The |\n' - ' | | precise rules are as follows: suppose that ' - 'the result |\n' + 'magnitude. A |\n' + ' | | precision of "0" is treated as equivalent ' + 'to a precision |\n' + ' | | of "1". The precise rules are as follows: ' + 'suppose that |\n' + ' | | the result formatted with presentation ' + 'type "\'e\'" and |\n' + ' | | precision "p-1" would have exponent ' + '"exp". Then, if "m <= |\n' + ' | | exp < p", where "m" is -4 for floats and ' + '-6 for |\n' + ' | | "Decimals", the number is formatted with ' + 'presentation type |\n' + ' | | "\'f\'" and precision "p-1-exp". ' + 'Otherwise, the number is |\n' ' | | formatted with presentation type "\'e\'" ' - 'and precision "p-1" |\n' - ' | | would have exponent "exp". Then, if "m <= ' - 'exp < p", where |\n' - ' | | "m" is -4 for floats and -6 for ' - '"Decimals", the number is |\n' - ' | | formatted with presentation type "\'f\'" ' 'and precision |\n' - ' | | "p-1-exp". Otherwise, the number is ' - 'formatted with |\n' - ' | | presentation type "\'e\'" and precision ' - '"p-1". In both cases |\n' - ' | | insignificant trailing zeros are removed ' - 'from the |\n' - ' | | significand, and the decimal point is also ' - 'removed if |\n' - ' | | there are no remaining digits following ' - 'it, unless the |\n' - ' | | "\'#\'" option is used. Positive and ' - 'negative infinity, |\n' - ' | | positive and negative zero, and nans, are ' - 'formatted as |\n' - ' | | "inf", "-inf", "0", "-0" and "nan" ' - 'respectively, |\n' - ' | | regardless of the precision. A precision ' - 'of "0" is |\n' - ' | | treated as equivalent to a precision of ' - '"1". With no |\n' - ' | | precision given, uses a precision of "6" ' - 'significant |\n' - ' | | digits for "float", and shows all ' - 'coefficient digits for |\n' - ' | | ' - '"Decimal". ' - '|\n' + ' | | "p-1". In both cases insignificant ' + 'trailing zeros are |\n' + ' | | removed from the significand, and the ' + 'decimal point is |\n' + ' | | also removed if there are no remaining ' + 'digits following |\n' + ' | | it, unless the "\'#\'" option is used. ' + 'With no precision |\n' + ' | | given, uses a precision of "6" significant ' + 'digits for |\n' + ' | | "float". For "Decimal", the coefficient of ' + 'the result is |\n' + ' | | formed from the coefficient digits of the ' + 'value; |\n' + ' | | scientific notation is used for values ' + 'smaller than "1e-6" |\n' + ' | | in absolute value and values where the ' + 'place value of the |\n' + ' | | least significant digit is larger than 1, ' + 'and fixed-point |\n' + ' | | notation is used otherwise. Positive and ' + 'negative |\n' + ' | | infinity, positive and negative zero, and ' + 'nans, are |\n' + ' | | formatted as "inf", "-inf", "0", "-0" and ' + '"nan" |\n' + ' | | respectively, regardless of the ' + 'precision. |\n' ' ' '+-----------+------------------------------------------------------------+\n' ' | "\'G\'" | General format. Same as "\'g\'" except ' @@ -5574,19 +5582,24 @@ 'percent sign. |\n' ' ' '+-----------+------------------------------------------------------------+\n' - ' | None | Similar to "\'g\'", except that ' - 'fixed-point notation, when |\n' - ' | | used, has at least one digit past the ' - 'decimal point. The |\n' - ' | | default precision is as high as needed to ' - 'represent the |\n' - ' | | particular value. The overall effect is to ' - 'match the |\n' - ' | | output of "str()" as altered by the other ' - 'format |\n' - ' | | ' - 'modifiers. ' - '|\n' + ' | None | For "float" this is the same as "\'g\'", ' + 'except that when |\n' + ' | | fixed-point notation is used to format the ' + 'result, it |\n' + ' | | always includes at least one digit past ' + 'the decimal point. |\n' + ' | | The precision used is as large as needed ' + 'to represent the |\n' + ' | | given value faithfully. For "Decimal", ' + 'this is the same |\n' + ' | | as either "\'g\'" or "\'G\'" depending on ' + 'the value of |\n' + ' | | "context.capitals" for the current decimal ' + 'context. The |\n' + ' | | overall effect is to match the output of ' + '"str()" as |\n' + ' | | altered by the other format ' + 'modifiers. |\n' ' ' '+-----------+------------------------------------------------------------+\n' '\n' @@ -11159,7 +11172,8 @@ 'object is “compatible” with the exception. An object is compatible\n' 'with an exception if it is the class or a base class of the ' 'exception\n' - 'object or a tuple containing an item compatible with the exception.\n' + 'object, or a tuple containing an item that is the class or a base\n' + 'class of the exception object.\n' '\n' 'If no except clause matches the exception, the search for an ' 'exception\n' @@ -11340,7 +11354,7 @@ 'representation\n' ' in computers.\n' '\n' - ' The string representations of the Numeric classes, computed by\n' + ' The string representations of the numeric classes, computed by\n' ' "__repr__()" and "__str__()", have the following properties:\n' '\n' ' * They are valid numeric literals which, when passed to their ' diff --git a/Misc/NEWS.d/3.8.7.rst b/Misc/NEWS.d/3.8.7.rst new file mode 100644 index 00000000000000..0688e480ba6fd0 --- /dev/null +++ b/Misc/NEWS.d/3.8.7.rst @@ -0,0 +1,195 @@ +.. bpo: 32381 +.. date: 2020-12-04-17-17-44 +.. nonce: NY5t2S +.. release date: 2020-12-21 +.. section: Core and Builtins + +Fix encoding name when running a ``.pyc`` file on Windows: +:c:func:`PyRun_SimpleFileExFlags()` now uses the correct encoding to decode +the filename. + +.. + +.. bpo: 42536 +.. date: 2020-12-02-20-23-31 +.. nonce: Kx3ZOu +.. section: Core and Builtins + +Several built-in and standard library types now ensure that their internal +result tuples are always tracked by the :term:`garbage collector `: + +- :meth:`collections.OrderedDict.items() ` + +- :meth:`dict.items` + +- :func:`enumerate` + +- :func:`functools.reduce` + +- :func:`itertools.combinations` + +- :func:`itertools.combinations_with_replacement` + +- :func:`itertools.permutations` + +- :func:`itertools.product` + +- :func:`itertools.zip_longest` + +- :func:`zip` + +Previously, they could have become untracked by a prior garbage collection. +Patch by Brandt Bucher. + +.. + +.. bpo: 42630 +.. date: 2020-12-15-17-51-27 +.. nonce: jf4jBl +.. section: Library + +:mod:`tkinter` functions and constructors which need a default root window +raise now :exc:`RuntimeError` with descriptive message instead of obscure +:exc:`AttributeError` or :exc:`NameError` if it is not created yet or cannot +be created automatically. + +.. + +.. bpo: 42644 +.. date: 2020-12-15-10-00-04 +.. nonce: XgLCNx +.. section: Library + +`logging.disable` will now validate the types and value of its parameter. It +also now accepts strings representing the levels (as does `loging.setLevel`) +instead of only the numerical values. + +.. + +.. bpo: 36541 +.. date: 2020-12-14-08-23-57 +.. nonce: qdEtZv +.. section: Library + +Fixed lib2to3.pgen2 to be able to parse PEP-570 positional only argument +syntax. + +.. + +.. bpo: 42375 +.. date: 2020-11-19-04-13-53 +.. nonce: U8bp4s +.. section: Library + +subprocess module update for DragonFlyBSD support. + +.. + +.. bpo: 39825 +.. date: 2020-10-20-08-28-26 +.. nonce: n6KnG0 +.. section: Library + +Windows: Change ``sysconfig.get_config_var('EXT_SUFFIX')`` to the expected +full ``platform_tag.extension`` format. Previously it was hard-coded to +``.pyd``, now it is compatible with ``distutils.sysconfig`` and will result +in something like ``.cp38-win_amd64.pyd``. This brings windows into +conformance with the other platforms. + +.. + +.. bpo: 39101 +.. date: 2020-10-11-21-43-03 +.. nonce: -I49Pm +.. section: Library + +Fixed tests using IsolatedAsyncioTestCase from hanging on BaseExceptions. + +.. + +.. bpo: 41907 +.. date: 2020-10-02-10-19-49 +.. nonce: wiIEsz +.. section: Library + +fix `format()` behavior for `IntFlag` + +.. + +.. bpo: 41889 +.. date: 2020-10-01-16-17-11 +.. nonce: qLkNh8 +.. section: Library + +Enum: fix regression involving inheriting a multiply-inherited enum + +.. + +.. bpo: 41891 +.. date: 2020-09-30-13-35-29 +.. nonce: pNAeYI +.. section: Library + +Ensure asyncio.wait_for waits for task completion + +.. + +.. bpo: 40219 +.. date: 2020-07-13-19-43-11 +.. nonce: MUoJEP +.. section: Library + +Lowered :class:`tkinter.ttk.LabeledScale` dummy widget to prevent hiding +part of the content label. + +.. + +.. bpo: 40084 +.. date: 2020-03-29-21-32-00 +.. nonce: MCYwcv +.. section: Library + +Fix ``Enum.__dir__``: dir(Enum.member) now includes attributes as well as +methods. + +.. + +.. bpo: 17140 +.. date: 2020-12-16-21-06-16 +.. nonce: 1leSEg +.. section: Documentation + +Add documentation for the :class:`multiprocessing.pool.ThreadPool` class. + +.. + +.. bpo: 42604 +.. date: 2020-12-20-02-35-28 +.. nonce: gRd89w +.. section: Build + +Now all platforms use a value for the "EXT_SUFFIX" build variable derived +from SOABI (for instance in freeBSD, "EXT_SUFFIX" is now ".cpython-310d.so" +instead of ".so"). Previosuly only Linux, Mac and VxWorks were using a value +for "EXT_SUFFIX" that included "SOABI". + +.. + +.. bpo: 42598 +.. date: 2020-12-13-14-43-10 +.. nonce: 7ipr5H +.. section: Build + +Fix implicit function declarations in configure which could have resulted in +incorrect configuration checks. Patch contributed by Joshua Root. + +.. + +.. bpo: 42613 +.. date: 2020-12-16-09-10-32 +.. nonce: J-jnm5 +.. section: Tools/Demos + +Fix ``freeze.py`` tool to use the prope config and library directories. +Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Build/2020-12-13-14-43-10.bpo-42598.7ipr5H.rst b/Misc/NEWS.d/next/Build/2020-12-13-14-43-10.bpo-42598.7ipr5H.rst deleted file mode 100644 index 7dafc105c45ea9..00000000000000 --- a/Misc/NEWS.d/next/Build/2020-12-13-14-43-10.bpo-42598.7ipr5H.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix implicit function declarations in configure which could have resulted in -incorrect configuration checks. Patch contributed by Joshua Root. diff --git a/Misc/NEWS.d/next/Build/2020-12-20-02-35-28.bpo-42604.gRd89w.rst b/Misc/NEWS.d/next/Build/2020-12-20-02-35-28.bpo-42604.gRd89w.rst deleted file mode 100644 index caaada41cf9bad..00000000000000 --- a/Misc/NEWS.d/next/Build/2020-12-20-02-35-28.bpo-42604.gRd89w.rst +++ /dev/null @@ -1,4 +0,0 @@ -Now all platforms use a value for the "EXT_SUFFIX" build variable derived -from SOABI (for instance in freeBSD, "EXT_SUFFIX" is now ".cpython-310d.so" -instead of ".so"). Previosuly only Linux, Mac and VxWorks were using a value -for "EXT_SUFFIX" that included "SOABI". diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-12-02-20-23-31.bpo-42536.Kx3ZOu.rst b/Misc/NEWS.d/next/Core and Builtins/2020-12-02-20-23-31.bpo-42536.Kx3ZOu.rst deleted file mode 100644 index 6ccacab1f64f68..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-12-02-20-23-31.bpo-42536.Kx3ZOu.rst +++ /dev/null @@ -1,26 +0,0 @@ -Several built-in and standard library types now ensure that their internal -result tuples are always tracked by the :term:`garbage collector -`: - -- :meth:`collections.OrderedDict.items() ` - -- :meth:`dict.items` - -- :func:`enumerate` - -- :func:`functools.reduce` - -- :func:`itertools.combinations` - -- :func:`itertools.combinations_with_replacement` - -- :func:`itertools.permutations` - -- :func:`itertools.product` - -- :func:`itertools.zip_longest` - -- :func:`zip` - -Previously, they could have become untracked by a prior garbage collection. -Patch by Brandt Bucher. diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-12-04-17-17-44.bpo-32381.NY5t2S.rst b/Misc/NEWS.d/next/Core and Builtins/2020-12-04-17-17-44.bpo-32381.NY5t2S.rst deleted file mode 100644 index f4d84f9d848d4f..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-12-04-17-17-44.bpo-32381.NY5t2S.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix encoding name when running a ``.pyc`` file on Windows: -:c:func:`PyRun_SimpleFileExFlags()` now uses the correct encoding to decode -the filename. diff --git a/Misc/NEWS.d/next/Documentation/2020-12-16-21-06-16.bpo-17140.1leSEg.rst b/Misc/NEWS.d/next/Documentation/2020-12-16-21-06-16.bpo-17140.1leSEg.rst deleted file mode 100644 index cb1fd23a56e639..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-12-16-21-06-16.bpo-17140.1leSEg.rst +++ /dev/null @@ -1 +0,0 @@ -Add documentation for the :class:`multiprocessing.pool.ThreadPool` class. diff --git a/Misc/NEWS.d/next/Library/2020-03-29-21-32-00.bpo-40084.MCYwcv.rst b/Misc/NEWS.d/next/Library/2020-03-29-21-32-00.bpo-40084.MCYwcv.rst deleted file mode 100644 index 65ff4ce36e82ea..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-03-29-21-32-00.bpo-40084.MCYwcv.rst +++ /dev/null @@ -1 +0,0 @@ -Fix ``Enum.__dir__``: dir(Enum.member) now includes attributes as well as methods. diff --git a/Misc/NEWS.d/next/Library/2020-07-13-19-43-11.bpo-40219.MUoJEP.rst b/Misc/NEWS.d/next/Library/2020-07-13-19-43-11.bpo-40219.MUoJEP.rst deleted file mode 100644 index aedc5c49b4087a..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-07-13-19-43-11.bpo-40219.MUoJEP.rst +++ /dev/null @@ -1 +0,0 @@ -Lowered :class:`tkinter.ttk.LabeledScale` dummy widget to prevent hiding part of the content label. diff --git a/Misc/NEWS.d/next/Library/2020-09-30-13-35-29.bpo-41891.pNAeYI.rst b/Misc/NEWS.d/next/Library/2020-09-30-13-35-29.bpo-41891.pNAeYI.rst deleted file mode 100644 index 75c25127803153..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-09-30-13-35-29.bpo-41891.pNAeYI.rst +++ /dev/null @@ -1 +0,0 @@ -Ensure asyncio.wait_for waits for task completion diff --git a/Misc/NEWS.d/next/Library/2020-10-01-16-17-11.bpo-41889.qLkNh8.rst b/Misc/NEWS.d/next/Library/2020-10-01-16-17-11.bpo-41889.qLkNh8.rst deleted file mode 100644 index 768865ae62116d..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-10-01-16-17-11.bpo-41889.qLkNh8.rst +++ /dev/null @@ -1 +0,0 @@ -Enum: fix regression involving inheriting a multiply-inherited enum diff --git a/Misc/NEWS.d/next/Library/2020-10-02-10-19-49.bpo-41907.wiIEsz.rst b/Misc/NEWS.d/next/Library/2020-10-02-10-19-49.bpo-41907.wiIEsz.rst deleted file mode 100644 index aa337b38046e61..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-10-02-10-19-49.bpo-41907.wiIEsz.rst +++ /dev/null @@ -1 +0,0 @@ -fix `format()` behavior for `IntFlag` diff --git a/Misc/NEWS.d/next/Library/2020-10-11-21-43-03.bpo-39101.-I49Pm.rst b/Misc/NEWS.d/next/Library/2020-10-11-21-43-03.bpo-39101.-I49Pm.rst deleted file mode 100644 index a571e8343cde12..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-10-11-21-43-03.bpo-39101.-I49Pm.rst +++ /dev/null @@ -1 +0,0 @@ -Fixed tests using IsolatedAsyncioTestCase from hanging on BaseExceptions. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2020-10-20-08-28-26.bpo-39825.n6KnG0.rst b/Misc/NEWS.d/next/Library/2020-10-20-08-28-26.bpo-39825.n6KnG0.rst deleted file mode 100644 index c337731f435843..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-10-20-08-28-26.bpo-39825.n6KnG0.rst +++ /dev/null @@ -1,5 +0,0 @@ -Windows: Change ``sysconfig.get_config_var('EXT_SUFFIX')`` to the expected -full ``platform_tag.extension`` format. Previously it was hard-coded to -``.pyd``, now it is compatible with ``distutils.sysconfig`` and will result -in something like ``.cp38-win_amd64.pyd``. This brings windows into -conformance with the other platforms. diff --git a/Misc/NEWS.d/next/Library/2020-11-19-04-13-53.bpo-42375.U8bp4s.rst b/Misc/NEWS.d/next/Library/2020-11-19-04-13-53.bpo-42375.U8bp4s.rst deleted file mode 100644 index 6d8c80c2f2c0ae..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-11-19-04-13-53.bpo-42375.U8bp4s.rst +++ /dev/null @@ -1 +0,0 @@ -subprocess module update for DragonFlyBSD support. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2020-12-14-08-23-57.bpo-36541.qdEtZv.rst b/Misc/NEWS.d/next/Library/2020-12-14-08-23-57.bpo-36541.qdEtZv.rst deleted file mode 100644 index 5678d8c595ca08..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-12-14-08-23-57.bpo-36541.qdEtZv.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed lib2to3.pgen2 to be able to parse PEP-570 positional only argument -syntax. diff --git a/Misc/NEWS.d/next/Library/2020-12-15-10-00-04.bpo-42644.XgLCNx.rst b/Misc/NEWS.d/next/Library/2020-12-15-10-00-04.bpo-42644.XgLCNx.rst deleted file mode 100644 index f58b58e4002ada..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-12-15-10-00-04.bpo-42644.XgLCNx.rst +++ /dev/null @@ -1,3 +0,0 @@ -`logging.disable` will now validate the types and value of its parameter. It -also now accepts strings representing the levels (as does `loging.setLevel`) -instead of only the numerical values. diff --git a/Misc/NEWS.d/next/Library/2020-12-15-17-51-27.bpo-42630.jf4jBl.rst b/Misc/NEWS.d/next/Library/2020-12-15-17-51-27.bpo-42630.jf4jBl.rst deleted file mode 100644 index 4b4a520931fda4..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-12-15-17-51-27.bpo-42630.jf4jBl.rst +++ /dev/null @@ -1,4 +0,0 @@ -:mod:`tkinter` functions and constructors which need a default root window -raise now :exc:`RuntimeError` with descriptive message instead of obscure -:exc:`AttributeError` or :exc:`NameError` if it is not created yet or cannot -be created automatically. diff --git a/Misc/NEWS.d/next/Tools-Demos/2020-12-16-09-10-32.bpo-42613.J-jnm5.rst b/Misc/NEWS.d/next/Tools-Demos/2020-12-16-09-10-32.bpo-42613.J-jnm5.rst deleted file mode 100644 index 140ff8255b96b1..00000000000000 --- a/Misc/NEWS.d/next/Tools-Demos/2020-12-16-09-10-32.bpo-42613.J-jnm5.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix ``freeze.py`` tool to use the prope config and library directories. -Patch by Victor Stinner. diff --git a/README.rst b/README.rst index 9ad101c39b2c56..f6fd4824f0a9b0 100644 --- a/README.rst +++ b/README.rst @@ -1,5 +1,5 @@ -This is Python version 3.8.7rc1 -=============================== +This is Python version 3.8.7 +============================ .. image:: https://travis-ci.org/python/cpython.svg?branch=3.8 :alt: CPython build status on Travis CI From 305eae8ebff047039469cd162fef7eab3aa04d77 Mon Sep 17 00:00:00 2001 From: Ned Deily Date: Mon, 21 Dec 2020 12:12:25 -0500 Subject: [PATCH 1897/2163] Update macOS installer ReadMe for 3.8.7 and Big Sur (GH-23882) --- Mac/BuildScript/resources/ReadMe.rtf | 48 +++++++++++++++++----------- 1 file changed, 30 insertions(+), 18 deletions(-) diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf index b1e972eec2d6e2..b070506abd448e 100644 --- a/Mac/BuildScript/resources/ReadMe.rtf +++ b/Mac/BuildScript/resources/ReadMe.rtf @@ -1,6 +1,6 @@ {\rtf1\ansi\ansicpg1252\cocoartf2513 -\cocoatextscaling0\cocoaplatform0{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fswiss\fcharset0 Helvetica-Oblique; -\f3\fmodern\fcharset0 CourierNewPSMT;} +\cocoatextscaling0\cocoaplatform0{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fmodern\fcharset0 CourierNewPSMT; +\f3\fswiss\fcharset0 Helvetica-Oblique;} {\colortbl;\red255\green255\blue255;} {\*\expandedcolortbl;;} \margl1440\margr1440\vieww13380\viewh14600\viewkind0 @@ -10,25 +10,37 @@ \ \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 -\f1\b \cf0 \ul \ulc0 Certificate verification and OpenSSL\ +\f1\b \cf0 \ul \ulc0 macOS 11 Big Sur not fully supported\ + +\f0\b0 \ulnone \ +Python 3.8.7 is not yet fully supported on macOS 11 Big Sur. It will install on macOS 11 Big Sur and will run on Apple Silicon Macs using Rosetta 2 translation. But a few features do not work correctly, most noticeably those involving searching for system libraries such as +\f2 ctypes.util.find_library() +\f0 and in +\f2 Distutils +\f0 . Python 3.9.1 or later provides full support for Big Sur and Apple Silicon Macs, including building natively on Apple Silicon Macs and support for +\f2 universal2 +\f0 binaries\ +\ + +\f1\b \ul Certificate verification and OpenSSL\ \f0\b0 \ulnone \ This package includes its own private copy of OpenSSL 1.1.1. The trust certificates in system and user keychains managed by the -\f2\i Keychain Access +\f3\i Keychain Access \f0\i0 application and the -\f2\i security +\f3\i security \f0\i0 command line utility are not used as defaults by the Python -\f3 ssl +\f2 ssl \f0 module. A sample command script is included in -\f3 /Applications/Python 3.8 +\f2 /Applications/Python 3.8 \f0 to install a curated bundle of default root certificates from the third-party -\f3 certifi +\f2 certifi \f0 package ({\field{\*\fldinst{HYPERLINK "https://pypi.org/project/certifi/"}}{\fldrslt https://pypi.org/project/certifi/}}). Double-click on -\f3 Install Certificates +\f2 Install Certificates \f0 to run it.\ \ The bundled -\f3 pip +\f2 pip \f0 has its own default certificate store for verifying download connections.\ \ @@ -60,17 +72,17 @@ Python 2.7 end-of-life [changed in 3.8.4]\ \ \f0\b0 \ulnone Python 2.7 has now reached end-of-life. As of Python 3.8.4, the -\f3 Python Launcher +\f2 Python Launcher \f0 app now has -\f3 python3 +\f2 python3 \f0 factory defaults. Also, the -\f3 Current +\f2 Current \f0 link in the -\f3 /Library/Frameworks/Python.framework/Versions +\f2 /Library/Frameworks/Python.framework/Versions \f0 directory is now updated to point to the Python 3 being installed; previously, only Python 2 installs updated -\f3 Current +\f2 Current \f0 . This change might affect developers using the framework to embed Python in their applications. If another version is desired for embedding, the -\f3 Current +\f2 Current \f0 symlink can be changed manually without affecting 3.8.x behavior.\ \f1\b \ul \ @@ -78,8 +90,8 @@ Other changes\ \f0\b0 \ulnone \ For other changes in this release, see the -\f2\i What's new +\f3\i What's new \f0\i0 section in the {\field{\*\fldinst{HYPERLINK "https://www.python.org/doc/"}}{\fldrslt Documentation Set}} for this release and its -\f2\i Release Notes +\f3\i Release Notes \f0\i0 link at {\field{\*\fldinst{HYPERLINK "https://www.python.org/downloads/"}}{\fldrslt https://www.python.org/downloads/}}.\ } \ No newline at end of file From 285b38c3a85486a2dee0b675cb5953d5da1442a0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Mon, 21 Dec 2020 18:46:59 +0100 Subject: [PATCH 1898/2163] Post 3.8.7 --- Include/patchlevel.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h index de3694e4821d71..e73cadbc6fef67 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.8.7" +#define PY_VERSION "3.8.7+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. From d3ab4c82a9b594d3cd98062eece19672fe42bce2 Mon Sep 17 00:00:00 2001 From: Andre Delfino Date: Mon, 21 Dec 2020 23:55:19 -0300 Subject: [PATCH 1899/2163] [3.8] [doc] Fix a few margins due to bad markup (GH-23619). (GH-23860) (cherry picked from commit 96a09df64483b70c4215c7025a19b9d2f1636c55) Co-authored-by: Andre Delfino --- Doc/library/enum.rst | 2 +- Doc/library/logging.config.rst | 78 +++++++++++++++++----------------- Doc/library/socket.rst | 10 ++--- Doc/library/trace.rst | 50 +++++++++++----------- Doc/library/turtle.rst | 12 +++--- Doc/reference/datamodel.rst | 1 - 6 files changed, 76 insertions(+), 77 deletions(-) diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index bd10016b937b41..c7e2b109c35132 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -56,7 +56,7 @@ helper, :class:`auto`. the bitwise operations without losing their :class:`Flag` membership. .. function:: unique - :noindex: + :noindex: Enum class decorator that ensures only one name is bound to any one value. diff --git a/Doc/library/logging.config.rst b/Doc/library/logging.config.rst index 683d6ed5e8ba52..0b5e2fc2a658d5 100644 --- a/Doc/library/logging.config.rst +++ b/Doc/library/logging.config.rst @@ -35,45 +35,45 @@ in :mod:`logging` itself) and defining handlers which are declared either in .. function:: dictConfig(config) - Takes the logging configuration from a dictionary. The contents of - this dictionary are described in :ref:`logging-config-dictschema` - below. - - If an error is encountered during configuration, this function will - raise a :exc:`ValueError`, :exc:`TypeError`, :exc:`AttributeError` - or :exc:`ImportError` with a suitably descriptive message. The - following is a (possibly incomplete) list of conditions which will - raise an error: - - * A ``level`` which is not a string or which is a string not - corresponding to an actual logging level. - * A ``propagate`` value which is not a boolean. - * An id which does not have a corresponding destination. - * A non-existent handler id found during an incremental call. - * An invalid logger name. - * Inability to resolve to an internal or external object. - - Parsing is performed by the :class:`DictConfigurator` class, whose - constructor is passed the dictionary used for configuration, and - has a :meth:`configure` method. The :mod:`logging.config` module - has a callable attribute :attr:`dictConfigClass` - which is initially set to :class:`DictConfigurator`. - You can replace the value of :attr:`dictConfigClass` with a - suitable implementation of your own. - - :func:`dictConfig` calls :attr:`dictConfigClass` passing - the specified dictionary, and then calls the :meth:`configure` method on - the returned object to put the configuration into effect:: - - def dictConfig(config): - dictConfigClass(config).configure() - - For example, a subclass of :class:`DictConfigurator` could call - ``DictConfigurator.__init__()`` in its own :meth:`__init__()`, then - set up custom prefixes which would be usable in the subsequent - :meth:`configure` call. :attr:`dictConfigClass` would be bound to - this new subclass, and then :func:`dictConfig` could be called exactly as - in the default, uncustomized state. + Takes the logging configuration from a dictionary. The contents of + this dictionary are described in :ref:`logging-config-dictschema` + below. + + If an error is encountered during configuration, this function will + raise a :exc:`ValueError`, :exc:`TypeError`, :exc:`AttributeError` + or :exc:`ImportError` with a suitably descriptive message. The + following is a (possibly incomplete) list of conditions which will + raise an error: + + * A ``level`` which is not a string or which is a string not + corresponding to an actual logging level. + * A ``propagate`` value which is not a boolean. + * An id which does not have a corresponding destination. + * A non-existent handler id found during an incremental call. + * An invalid logger name. + * Inability to resolve to an internal or external object. + + Parsing is performed by the :class:`DictConfigurator` class, whose + constructor is passed the dictionary used for configuration, and + has a :meth:`configure` method. The :mod:`logging.config` module + has a callable attribute :attr:`dictConfigClass` + which is initially set to :class:`DictConfigurator`. + You can replace the value of :attr:`dictConfigClass` with a + suitable implementation of your own. + + :func:`dictConfig` calls :attr:`dictConfigClass` passing + the specified dictionary, and then calls the :meth:`configure` method on + the returned object to put the configuration into effect:: + + def dictConfig(config): + dictConfigClass(config).configure() + + For example, a subclass of :class:`DictConfigurator` could call + ``DictConfigurator.__init__()`` in its own :meth:`__init__()`, then + set up custom prefixes which would be usable in the subsequent + :meth:`configure` call. :attr:`dictConfigClass` would be bound to + this new subclass, and then :func:`dictConfig` could be called exactly as + in the default, uncustomized state. .. versionadded:: 3.2 diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 5e496ca4d6a9b9..b5a4b0cde066fb 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -56,12 +56,12 @@ created. Socket addresses are represented as follows: bytes-like object can be used for either type of address when passing it as an argument. - .. versionchanged:: 3.3 - Previously, :const:`AF_UNIX` socket paths were assumed to use UTF-8 - encoding. + .. versionchanged:: 3.3 + Previously, :const:`AF_UNIX` socket paths were assumed to use UTF-8 + encoding. - .. versionchanged:: 3.5 - Writable :term:`bytes-like object` is now accepted. + .. versionchanged:: 3.5 + Writable :term:`bytes-like object` is now accepted. .. _host_port: diff --git a/Doc/library/trace.rst b/Doc/library/trace.rst index 85fec6830006c7..09bc1441487e03 100644 --- a/Doc/library/trace.rst +++ b/Doc/library/trace.rst @@ -153,47 +153,47 @@ Programmatic Interface count information. *timing* enables a timestamp relative to when tracing was started to be displayed. - .. method:: run(cmd) + .. method:: run(cmd) - Execute the command and gather statistics from the execution with - the current tracing parameters. *cmd* must be a string or code object, - suitable for passing into :func:`exec`. + Execute the command and gather statistics from the execution with + the current tracing parameters. *cmd* must be a string or code object, + suitable for passing into :func:`exec`. - .. method:: runctx(cmd, globals=None, locals=None) + .. method:: runctx(cmd, globals=None, locals=None) - Execute the command and gather statistics from the execution with the - current tracing parameters, in the defined global and local - environments. If not defined, *globals* and *locals* default to empty - dictionaries. + Execute the command and gather statistics from the execution with the + current tracing parameters, in the defined global and local + environments. If not defined, *globals* and *locals* default to empty + dictionaries. - .. method:: runfunc(func, *args, **kwds) + .. method:: runfunc(func, *args, **kwds) - Call *func* with the given arguments under control of the :class:`Trace` - object with the current tracing parameters. + Call *func* with the given arguments under control of the :class:`Trace` + object with the current tracing parameters. - .. method:: results() + .. method:: results() - Return a :class:`CoverageResults` object that contains the cumulative - results of all previous calls to ``run``, ``runctx`` and ``runfunc`` - for the given :class:`Trace` instance. Does not reset the accumulated - trace results. + Return a :class:`CoverageResults` object that contains the cumulative + results of all previous calls to ``run``, ``runctx`` and ``runfunc`` + for the given :class:`Trace` instance. Does not reset the accumulated + trace results. .. class:: CoverageResults A container for coverage results, created by :meth:`Trace.results`. Should not be created directly by the user. - .. method:: update(other) + .. method:: update(other) - Merge in data from another :class:`CoverageResults` object. + Merge in data from another :class:`CoverageResults` object. - .. method:: write_results(show_missing=True, summary=False, coverdir=None) + .. method:: write_results(show_missing=True, summary=False, coverdir=None) - Write coverage results. Set *show_missing* to show lines that had no - hits. Set *summary* to include in the output the coverage summary per - module. *coverdir* specifies the directory into which the coverage - result files will be output. If ``None``, the results for each source - file are placed in its directory. + Write coverage results. Set *show_missing* to show lines that had no + hits. Set *summary* to include in the output the coverage summary per + module. *coverdir* specifies the directory into which the coverage + result files will be output. If ``None``, the results for each source + file are placed in its directory. A simple example demonstrating the use of the programmatic interface:: diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index d3487537df99a9..2084d75b3a57a0 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -913,8 +913,8 @@ Color control Set pencolor to the RGB color represented by *r*, *g*, and *b*. Each of *r*, *g*, and *b* must be in the range 0..colormode. - If turtleshape is a polygon, the outline of that polygon is drawn with the - newly set pencolor. + If turtleshape is a polygon, the outline of that polygon is drawn with the + newly set pencolor. .. doctest:: :skipif: _tkinter is None @@ -962,8 +962,8 @@ Color control Set fillcolor to the RGB color represented by *r*, *g*, and *b*. Each of *r*, *g*, and *b* must be in the range 0..colormode. - If turtleshape is a polygon, the interior of that polygon is drawn - with the newly set fillcolor. + If turtleshape is a polygon, the interior of that polygon is drawn + with the newly set fillcolor. .. doctest:: :skipif: _tkinter is None @@ -1001,8 +1001,8 @@ Color control Equivalent to ``pencolor(colorstring1)`` and ``fillcolor(colorstring2)`` and analogously if the other input format is used. - If turtleshape is a polygon, outline and interior of that polygon is drawn - with the newly set colors. + If turtleshape is a polygon, outline and interior of that polygon is drawn + with the newly set colors. .. doctest:: :skipif: _tkinter is None diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 332051e396994e..a2b3d4e6b0c183 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -212,7 +212,6 @@ Ellipsis There are two types of integers: Integers (:class:`int`) - These represent numbers in an unlimited range, subject to available (virtual) memory only. For the purpose of shift and mask operations, a binary representation is assumed, and negative numbers are represented in a variant of From 02349e2dc9d93202c658ae383b2de2e36b2366f8 Mon Sep 17 00:00:00 2001 From: Andre Delfino Date: Mon, 21 Dec 2020 23:56:55 -0300 Subject: [PATCH 1900/2163] [3.8] [doc] Fix erroneous backslashes in signatures and names (GH-23658) (GH-23828) The issue being resolved is shown in the 3.10 docs (if you select docs for older versions you won't see a visual glitch). The newer sphinx version that produces the 3.10 docs doesn't treat the backslash to escape things in some situations it previously did.. (cherry picked from commit dcc997cd28ab33ebac44182ee55533c1b37689f7) Co-authored-by: Andre Delfino --- Doc/library/asyncio-eventloop.rst | 36 +++++++++++++++--------------- Doc/library/asyncio-future.rst | 6 ++--- Doc/library/asyncio-policy.rst | 2 +- Doc/library/asyncio-stream.rst | 10 ++++----- Doc/library/asyncio-subprocess.rst | 6 ++--- Doc/library/asyncio-task.rst | 16 ++++++------- Doc/library/base64.rst | 2 +- Doc/library/contextvars.rst | 4 ++-- Doc/library/ctypes.rst | 2 +- Doc/library/difflib.rst | 6 ++--- Doc/library/email.header.rst | 2 +- Doc/library/email.policy.rst | 2 +- Doc/library/functions.rst | 2 +- Doc/library/http.cookies.rst | 2 +- Doc/library/importlib.rst | 4 ++-- Doc/library/inspect.rst | 8 +++---- Doc/library/io.rst | 2 +- Doc/library/lzma.rst | 4 ++-- Doc/library/os.rst | 10 ++++----- Doc/library/pickle.rst | 12 +++++----- Doc/library/plistlib.rst | 8 +++---- Doc/library/shutil.rst | 2 +- Doc/library/stdtypes.rst | 4 ++-- Doc/library/sysconfig.rst | 2 +- Doc/library/tarfile.rst | 2 +- Doc/library/test.rst | 4 ++-- Doc/library/warnings.rst | 2 +- Doc/library/winreg.rst | 2 +- Doc/library/xml.dom.minidom.rst | 2 +- Doc/requirements.txt | 2 +- 30 files changed, 84 insertions(+), 84 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 32bc219cf5c376..dde57e295aa688 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -309,7 +309,7 @@ Creating Futures and Tasks .. versionadded:: 3.5.2 -.. method:: loop.create_task(coro, \*, name=None) +.. method:: loop.create_task(coro, *, name=None) Schedule the execution of a :ref:`coroutine`. Return a :class:`Task` object. @@ -344,7 +344,7 @@ Opening network connections ^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. coroutinemethod:: loop.create_connection(protocol_factory, \ - host=None, port=None, \*, ssl=None, \ + host=None, port=None, *, ssl=None, \ family=0, proto=0, flags=0, sock=None, \ local_addr=None, server_hostname=None, \ ssl_handshake_timeout=None, \ @@ -470,7 +470,7 @@ Opening network connections that can be used directly in async/await code. .. coroutinemethod:: loop.create_datagram_endpoint(protocol_factory, \ - local_addr=None, remote_addr=None, \*, \ + local_addr=None, remote_addr=None, *, \ family=0, proto=0, flags=0, \ reuse_address=None, reuse_port=None, \ allow_broadcast=None, sock=None) @@ -547,7 +547,7 @@ Opening network connections Added support for Windows. .. coroutinemethod:: loop.create_unix_connection(protocol_factory, \ - path=None, \*, ssl=None, sock=None, \ + path=None, *, ssl=None, sock=None, \ server_hostname=None, ssl_handshake_timeout=None) Create a Unix connection. @@ -580,7 +580,7 @@ Creating network servers ^^^^^^^^^^^^^^^^^^^^^^^^ .. coroutinemethod:: loop.create_server(protocol_factory, \ - host=None, port=None, \*, \ + host=None, port=None, *, \ family=socket.AF_UNSPEC, \ flags=socket.AI_PASSIVE, \ sock=None, backlog=100, ssl=None, \ @@ -671,7 +671,7 @@ Creating network servers .. coroutinemethod:: loop.create_unix_server(protocol_factory, path=None, \ - \*, sock=None, backlog=100, ssl=None, \ + *, sock=None, backlog=100, ssl=None, \ ssl_handshake_timeout=None, start_serving=True) Similar to :meth:`loop.create_server` but works with the @@ -696,7 +696,7 @@ Creating network servers The *path* parameter can now be a :class:`~pathlib.Path` object. .. coroutinemethod:: loop.connect_accepted_socket(protocol_factory, \ - sock, \*, ssl=None, ssl_handshake_timeout=None) + sock, *, ssl=None, ssl_handshake_timeout=None) Wrap an already accepted connection into a transport/protocol pair. @@ -761,7 +761,7 @@ TLS Upgrade ^^^^^^^^^^^ .. coroutinemethod:: loop.start_tls(transport, protocol, \ - sslcontext, \*, server_side=False, \ + sslcontext, *, server_side=False, \ server_hostname=None, ssl_handshake_timeout=None) Upgrade an existing transport-based connection to TLS. @@ -794,7 +794,7 @@ TLS Upgrade Watching file descriptors ^^^^^^^^^^^^^^^^^^^^^^^^^ -.. method:: loop.add_reader(fd, callback, \*args) +.. method:: loop.add_reader(fd, callback, *args) Start monitoring the *fd* file descriptor for read availability and invoke *callback* with the specified arguments once *fd* is available for @@ -804,7 +804,7 @@ Watching file descriptors Stop monitoring the *fd* file descriptor for read availability. -.. method:: loop.add_writer(fd, callback, \*args) +.. method:: loop.add_writer(fd, callback, *args) Start monitoring the *fd* file descriptor for write availability and invoke *callback* with the specified arguments once *fd* is available for @@ -918,7 +918,7 @@ convenient. :meth:`loop.create_server` and :func:`start_server`. .. coroutinemethod:: loop.sock_sendfile(sock, file, offset=0, count=None, \ - \*, fallback=True) + *, fallback=True) Send a file using high-performance :mod:`os.sendfile` if possible. Return the total number of bytes sent. @@ -952,7 +952,7 @@ convenient. DNS ^^^ -.. coroutinemethod:: loop.getaddrinfo(host, port, \*, family=0, \ +.. coroutinemethod:: loop.getaddrinfo(host, port, *, family=0, \ type=0, proto=0, flags=0) Asynchronous version of :meth:`socket.getaddrinfo`. @@ -1017,7 +1017,7 @@ Working with pipes Unix signals ^^^^^^^^^^^^ -.. method:: loop.add_signal_handler(signum, callback, \*args) +.. method:: loop.add_signal_handler(signum, callback, *args) Set *callback* as the handler for the *signum* signal. @@ -1052,7 +1052,7 @@ Unix signals Executing code in thread or process pools ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. awaitablemethod:: loop.run_in_executor(executor, func, \*args) +.. awaitablemethod:: loop.run_in_executor(executor, func, *args) Arrange for *func* to be called in the specified executor. @@ -1222,9 +1222,9 @@ async/await code consider using the high-level subprocesses. See :ref:`Subprocess Support on Windows ` for details. -.. coroutinemethod:: loop.subprocess_exec(protocol_factory, \*args, \ +.. coroutinemethod:: loop.subprocess_exec(protocol_factory, *args, \ stdin=subprocess.PIPE, stdout=subprocess.PIPE, \ - stderr=subprocess.PIPE, \*\*kwargs) + stderr=subprocess.PIPE, **kwargs) Create a subprocess from one or more string arguments specified by *args*. @@ -1304,9 +1304,9 @@ async/await code consider using the high-level conforms to the :class:`asyncio.SubprocessTransport` base class and *protocol* is an object instantiated by the *protocol_factory*. -.. coroutinemethod:: loop.subprocess_shell(protocol_factory, cmd, \*, \ +.. coroutinemethod:: loop.subprocess_shell(protocol_factory, cmd, *, \ stdin=subprocess.PIPE, stdout=subprocess.PIPE, \ - stderr=subprocess.PIPE, \*\*kwargs) + stderr=subprocess.PIPE, **kwargs) Create a subprocess from *cmd*, which can be a :class:`str` or a :class:`bytes` string encoded to the diff --git a/Doc/library/asyncio-future.rst b/Doc/library/asyncio-future.rst index 832d58119b7b07..45a2754e9658d9 100644 --- a/Doc/library/asyncio-future.rst +++ b/Doc/library/asyncio-future.rst @@ -31,7 +31,7 @@ Future Functions .. versionadded:: 3.5 -.. function:: ensure_future(obj, \*, loop=None) +.. function:: ensure_future(obj, *, loop=None) Return: @@ -58,7 +58,7 @@ Future Functions The function accepts any :term:`awaitable` object. -.. function:: wrap_future(future, \*, loop=None) +.. function:: wrap_future(future, *, loop=None) Wrap a :class:`concurrent.futures.Future` object in a :class:`asyncio.Future` object. @@ -67,7 +67,7 @@ Future Functions Future Object ============= -.. class:: Future(\*, loop=None) +.. class:: Future(*, loop=None) A Future represents an eventual result of an asynchronous operation. Not thread-safe. diff --git a/Doc/library/asyncio-policy.rst b/Doc/library/asyncio-policy.rst index 91f9822599d482..ceb2323c54586f 100644 --- a/Doc/library/asyncio-policy.rst +++ b/Doc/library/asyncio-policy.rst @@ -159,7 +159,7 @@ implementation used by the asyncio event loop: .. class:: AbstractChildWatcher - .. method:: add_child_handler(pid, callback, \*args) + .. method:: add_child_handler(pid, callback, *args) Register a new child handler. diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst index b76ed379c7f4c8..584bf10fc042bc 100644 --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -48,7 +48,7 @@ The following top-level asyncio functions can be used to create and work with streams: -.. coroutinefunction:: open_connection(host=None, port=None, \*, \ +.. coroutinefunction:: open_connection(host=None, port=None, *, \ loop=None, limit=None, ssl=None, family=0, \ proto=0, flags=0, sock=None, local_addr=None, \ server_hostname=None, ssl_handshake_timeout=None) @@ -74,7 +74,7 @@ and work with streams: The *ssl_handshake_timeout* parameter. .. coroutinefunction:: start_server(client_connected_cb, host=None, \ - port=None, \*, loop=None, limit=None, \ + port=None, *, loop=None, limit=None, \ family=socket.AF_UNSPEC, \ flags=socket.AI_PASSIVE, sock=None, \ backlog=100, ssl=None, reuse_address=None, \ @@ -109,7 +109,7 @@ and work with streams: .. rubric:: Unix Sockets -.. coroutinefunction:: open_unix_connection(path=None, \*, loop=None, \ +.. coroutinefunction:: open_unix_connection(path=None, *, loop=None, \ limit=None, ssl=None, sock=None, \ server_hostname=None, ssl_handshake_timeout=None) @@ -132,7 +132,7 @@ and work with streams: .. coroutinefunction:: start_unix_server(client_connected_cb, path=None, \ - \*, loop=None, limit=None, sock=None, \ + *, loop=None, limit=None, sock=None, \ backlog=100, ssl=None, ssl_handshake_timeout=None, \ start_serving=True) @@ -192,7 +192,7 @@ StreamReader can be read. Use the :attr:`IncompleteReadError.partial` attribute to get the partially read data. - .. coroutinemethod:: readuntil(separator=b'\\n') + .. coroutinemethod:: readuntil(separator=b'\n') Read data from the stream until *separator* is found. diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst index b0330349dfb651..fb98552c86d4c1 100644 --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -61,9 +61,9 @@ See also the `Examples`_ subsection. Creating Subprocesses ===================== -.. coroutinefunction:: create_subprocess_exec(program, \*args, stdin=None, \ +.. coroutinefunction:: create_subprocess_exec(program, *args, stdin=None, \ stdout=None, stderr=None, loop=None, \ - limit=None, \*\*kwds) + limit=None, **kwds) Create a subprocess. @@ -82,7 +82,7 @@ Creating Subprocesses .. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, \ stdout=None, stderr=None, loop=None, \ - limit=None, \*\*kwds) + limit=None, **kwds) Run the *cmd* shell command. diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst index 6acec30558cba6..246d5b9a73c548 100644 --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -210,7 +210,7 @@ is :meth:`loop.run_in_executor`. Running an asyncio Program ========================== -.. function:: run(coro, \*, debug=False) +.. function:: run(coro, *, debug=False) Execute the :term:`coroutine` *coro* and return the result. @@ -245,7 +245,7 @@ Running an asyncio Program Creating Tasks ============== -.. function:: create_task(coro, \*, name=None) +.. function:: create_task(coro, *, name=None) Wrap the *coro* :ref:`coroutine ` into a :class:`Task` and schedule its execution. Return the Task object. @@ -317,7 +317,7 @@ Sleeping Running Tasks Concurrently ========================== -.. awaitablefunction:: gather(\*aws, loop=None, return_exceptions=False) +.. awaitablefunction:: gather(*aws, loop=None, return_exceptions=False) Run :ref:`awaitable objects ` in the *aws* sequence *concurrently*. @@ -497,7 +497,7 @@ Timeouts Waiting Primitives ================== -.. coroutinefunction:: wait(aws, \*, loop=None, timeout=None,\ +.. coroutinefunction:: wait(aws, *, loop=None, timeout=None,\ return_when=ALL_COMPLETED) Run :ref:`awaitable objects ` in the *aws* @@ -585,7 +585,7 @@ Waiting Primitives deprecated. -.. function:: as_completed(aws, \*, loop=None, timeout=None) +.. function:: as_completed(aws, *, loop=None, timeout=None) Run :ref:`awaitable objects ` in the *aws* iterable concurrently. Return an iterator of coroutines. @@ -679,7 +679,7 @@ Introspection Task Object =========== -.. class:: Task(coro, \*, loop=None, name=None) +.. class:: Task(coro, *, loop=None, name=None) A :class:`Future-like ` object that runs a Python :ref:`coroutine `. Not thread-safe. @@ -842,7 +842,7 @@ Task Object See the documentation of :meth:`Future.remove_done_callback` for more details. - .. method:: get_stack(\*, limit=None) + .. method:: get_stack(*, limit=None) Return the list of stack frames for this Task. @@ -863,7 +863,7 @@ Task Object stack are returned, but the oldest frames of a traceback are returned. (This matches the behavior of the traceback module.) - .. method:: print_stack(\*, limit=None, file=None) + .. method:: print_stack(*, limit=None, file=None) Print the stack or traceback for this Task. diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst index ad9f5f58bee2aa..3c63c15ad401e9 100644 --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -178,7 +178,7 @@ The modern interface provides: .. versionadded:: 3.4 -.. function:: a85decode(b, *, foldspaces=False, adobe=False, ignorechars=b' \\t\\n\\r\\v') +.. function:: a85decode(b, *, foldspaces=False, adobe=False, ignorechars=b' \t\n\r\v') Decode the Ascii85 encoded :term:`bytes-like object` or ASCII string *b* and return the decoded :class:`bytes`. diff --git a/Doc/library/contextvars.rst b/Doc/library/contextvars.rst index 8805661c456edb..14ac47f4c9eb16 100644 --- a/Doc/library/contextvars.rst +++ b/Doc/library/contextvars.rst @@ -26,7 +26,7 @@ See also :pep:`567` for additional details. Context Variables ----------------- -.. class:: ContextVar(name, [\*, default]) +.. class:: ContextVar(name, [*, default]) This class is used to declare a new Context Variable, e.g.:: @@ -146,7 +146,7 @@ Manual Context Management Context implements the :class:`collections.abc.Mapping` interface. - .. method:: run(callable, \*args, \*\*kwargs) + .. method:: run(callable, *args, **kwargs) Execute ``callable(*args, **kwargs)`` code in the context object the *run* method is called on. Return the result of the execution diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index bf32d3e549b480..7313148721dac6 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -2508,7 +2508,7 @@ other data types containing pointer type fields. Arrays and pointers ^^^^^^^^^^^^^^^^^^^ -.. class:: Array(\*args) +.. class:: Array(*args) Abstract base class for arrays. diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index a8543b38c197ec..e6dd1dd7a54f03 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -149,7 +149,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. contains a good example of its use. -.. function:: context_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\\n') +.. function:: context_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\n') Compare *a* and *b* (lists of strings); return a delta (a :term:`generator` generating the delta lines) in context diff format. @@ -279,7 +279,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. emu -.. function:: unified_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\\n') +.. function:: unified_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\n') Compare *a* and *b* (lists of strings); return a delta (a :term:`generator` generating the delta lines) in unified diff format. @@ -321,7 +321,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. See :ref:`difflib-interface` for a more detailed example. -.. function:: diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\\n') +.. function:: diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\n') Compare *a* and *b* (lists of bytes objects) using *dfunc*; yield a sequence of delta lines (also bytes) in the format returned by *dfunc*. diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst index 07152c224f2ff0..e093f138936b36 100644 --- a/Doc/library/email.header.rst +++ b/Doc/library/email.header.rst @@ -116,7 +116,7 @@ Here is the :class:`Header` class description: if *s* is a byte string. - .. method:: encode(splitchars=';, \\t', maxlinelen=None, linesep='\\n') + .. method:: encode(splitchars=';, \t', maxlinelen=None, linesep='\n') Encode a message header into an RFC-compliant format, possibly wrapping long lines and encapsulating non-ASCII parts in base64 or quoted-printable diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst index 8e7076259810f5..bf53b9520fc723 100644 --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -210,7 +210,7 @@ added matters. To illustrate:: :meth:`register_defect` method. - .. attribute:: mangle_from\_ + .. attribute:: mangle_from_ If :const:`True`, lines starting with *"From "* in the body are escaped by putting a ``>`` in front of them. This parameter is used when diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 4f5fe6bc230697..be98d30488ab8a 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1312,7 +1312,7 @@ are always available. They are listed here in alphabetical order. supported. -.. function:: print(*objects, sep=' ', end='\\n', file=sys.stdout, flush=False) +.. function:: print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False) Print *objects* to the text stream *file*, separated by *sep* and followed by *end*. *sep*, *end*, *file* and *flush*, if present, must be given as keyword diff --git a/Doc/library/http.cookies.rst b/Doc/library/http.cookies.rst index 17792b200599bd..a2c1eb00d8b33d 100644 --- a/Doc/library/http.cookies.rst +++ b/Doc/library/http.cookies.rst @@ -93,7 +93,7 @@ Cookie Objects :meth:`value_decode` are inverses on the range of *value_decode*. -.. method:: BaseCookie.output(attrs=None, header='Set-Cookie:', sep='\\r\\n') +.. method:: BaseCookie.output(attrs=None, header='Set-Cookie:', sep='\r\n') Return a string representation suitable to be sent as HTTP headers. *attrs* and *header* are sent to each :class:`Morsel`'s :meth:`output` method. *sep* is used diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index be0dab3d057cab..8635e2bc66f3a8 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -1101,7 +1101,7 @@ find and load modules. directory for ``''`` (i.e. the empty string). -.. class:: FileFinder(path, \*loader_details) +.. class:: FileFinder(path, *loader_details) A concrete implementation of :class:`importlib.abc.PathEntryFinder` which caches results from the file system. @@ -1144,7 +1144,7 @@ find and load modules. Clear out the internal cache. - .. classmethod:: path_hook(\*loader_details) + .. classmethod:: path_hook(*loader_details) A class method which returns a closure for use on :attr:`sys.path_hooks`. An instance of :class:`FileFinder` is returned by the closure using the diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index bab2c41e4e2242..ffa0d9840cd3cf 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -556,7 +556,7 @@ The Signature object represents the call signature of a callable object and its return annotation. To retrieve a Signature object, use the :func:`signature` function. -.. function:: signature(callable, \*, follow_wrapped=True) +.. function:: signature(callable, *, follow_wrapped=True) Return a :class:`Signature` object for the given ``callable``:: @@ -597,7 +597,7 @@ function. C provide no metadata about their arguments. -.. class:: Signature(parameters=None, \*, return_annotation=Signature.empty) +.. class:: Signature(parameters=None, *, return_annotation=Signature.empty) A Signature object represents the call signature of a function and its return annotation. For each parameter accepted by the function it stores a @@ -668,7 +668,7 @@ function. >>> str(new_sig) "(a, b) -> 'new return anno'" - .. classmethod:: Signature.from_callable(obj, \*, follow_wrapped=True) + .. classmethod:: Signature.from_callable(obj, *, follow_wrapped=True) Return a :class:`Signature` (or its subclass) object for a given callable ``obj``. Pass ``follow_wrapped=False`` to get a signature of ``obj`` @@ -684,7 +684,7 @@ function. .. versionadded:: 3.5 -.. class:: Parameter(name, kind, \*, default=Parameter.empty, annotation=Parameter.empty) +.. class:: Parameter(name, kind, *, default=Parameter.empty, annotation=Parameter.empty) Parameter objects are *immutable*. Instead of modifying a Parameter object, you can use :meth:`Parameter.replace` to create a modified copy. diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 32151a0ace458e..667e4c654ff235 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -959,7 +959,7 @@ Text I/O .. versionadded:: 3.7 -.. class:: StringIO(initial_value='', newline='\\n') +.. class:: StringIO(initial_value='', newline='\n') An in-memory stream for text I/O. The text buffer is discarded when the :meth:`~IOBase.close` method is called. diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst index 4bfff9c6147ed4..633c87873cd8ce 100644 --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -33,7 +33,7 @@ from multiple threads, it is necessary to protect it with a lock. Reading and writing compressed files ------------------------------------ -.. function:: open(filename, mode="rb", \*, format=None, check=-1, preset=None, filters=None, encoding=None, errors=None, newline=None) +.. function:: open(filename, mode="rb", *, format=None, check=-1, preset=None, filters=None, encoding=None, errors=None, newline=None) Open an LZMA-compressed file in binary or text mode, returning a :term:`file object`. @@ -69,7 +69,7 @@ Reading and writing compressed files Accepts a :term:`path-like object`. -.. class:: LZMAFile(filename=None, mode="r", \*, format=None, check=-1, preset=None, filters=None) +.. class:: LZMAFile(filename=None, mode="r", *, format=None, check=-1, preset=None, filters=None) Open an LZMA-compressed file in binary mode. diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 1a6ff40bf09e65..45213a664e9f22 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1860,7 +1860,7 @@ features: Accepts a :term:`path-like object`. -.. function:: lstat(path, \*, dir_fd=None) +.. function:: lstat(path, *, dir_fd=None) Perform the equivalent of an :c:func:`lstat` system call on the given path. Similar to :func:`~os.stat`, but does not follow symbolic links. Return a @@ -2367,7 +2367,7 @@ features: On the first, uncached call, a system call is required on Windows but not on Unix. - .. method:: is_dir(\*, follow_symlinks=True) + .. method:: is_dir(*, follow_symlinks=True) Return ``True`` if this entry is a directory or a symbolic link pointing to a directory; return ``False`` if the entry is or points to any other @@ -2391,7 +2391,7 @@ features: This method can raise :exc:`OSError`, such as :exc:`PermissionError`, but :exc:`FileNotFoundError` is caught and not raised. - .. method:: is_file(\*, follow_symlinks=True) + .. method:: is_file(*, follow_symlinks=True) Return ``True`` if this entry is a file or a symbolic link pointing to a file; return ``False`` if the entry is or points to a directory or other @@ -2421,7 +2421,7 @@ features: This method can raise :exc:`OSError`, such as :exc:`PermissionError`, but :exc:`FileNotFoundError` is caught and not raised. - .. method:: stat(\*, follow_symlinks=True) + .. method:: stat(*, follow_symlinks=True) Return a :class:`stat_result` object for this entry. This method follows symbolic links by default; to stat a symbolic link add the @@ -2453,7 +2453,7 @@ features: for :class:`bytes` paths on Windows. -.. function:: stat(path, \*, dir_fd=None, follow_symlinks=True) +.. function:: stat(path, *, dir_fd=None, follow_symlinks=True) Get the status of a file or a file descriptor. Perform the equivalent of a :c:func:`stat` system call on the given path. *path* may be specified as diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index d92e947a76403e..ba41d5f7d09184 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -213,7 +213,7 @@ The :mod:`pickle` module provides the following constants: The :mod:`pickle` module provides the following functions to make the pickling process more convenient: -.. function:: dump(obj, file, protocol=None, \*, fix_imports=True, buffer_callback=None) +.. function:: dump(obj, file, protocol=None, *, fix_imports=True, buffer_callback=None) Write the pickled representation of the object *obj* to the open :term:`file object` *file*. This is equivalent to @@ -225,7 +225,7 @@ process more convenient: .. versionchanged:: 3.8 The *buffer_callback* argument was added. -.. function:: dumps(obj, protocol=None, \*, fix_imports=True, buffer_callback=None) +.. function:: dumps(obj, protocol=None, *, fix_imports=True, buffer_callback=None) Return the pickled representation of the object *obj* as a :class:`bytes` object, instead of writing it to a file. @@ -236,7 +236,7 @@ process more convenient: .. versionchanged:: 3.8 The *buffer_callback* argument was added. -.. function:: load(file, \*, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) +.. function:: load(file, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) Read the pickled representation of an object from the open :term:`file object` *file* and return the reconstituted object hierarchy specified therein. @@ -252,7 +252,7 @@ process more convenient: .. versionchanged:: 3.8 The *buffers* argument was added. -.. function:: loads(data, \*, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) +.. function:: loads(data, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) Return the reconstituted object hierarchy of the pickled representation *data* of an object. *data* must be a :term:`bytes-like object`. @@ -296,7 +296,7 @@ The :mod:`pickle` module defines three exceptions: The :mod:`pickle` module exports three classes, :class:`Pickler`, :class:`Unpickler` and :class:`PickleBuffer`: -.. class:: Pickler(file, protocol=None, \*, fix_imports=True, buffer_callback=None) +.. class:: Pickler(file, protocol=None, *, fix_imports=True, buffer_callback=None) This takes a binary file for writing a pickle data stream. @@ -391,7 +391,7 @@ The :mod:`pickle` module exports three classes, :class:`Pickler`, Use :func:`pickletools.optimize` if you need more compact pickles. -.. class:: Unpickler(file, \*, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) +.. class:: Unpickler(file, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) This takes a binary file for reading a pickle data stream. diff --git a/Doc/library/plistlib.rst b/Doc/library/plistlib.rst index 9dfe3c8a0dc984..98befabb3f9ab9 100644 --- a/Doc/library/plistlib.rst +++ b/Doc/library/plistlib.rst @@ -48,7 +48,7 @@ or :class:`datetime.datetime` objects. This module defines the following functions: -.. function:: load(fp, \*, fmt=None, use_builtin_types=True, dict_type=dict) +.. function:: load(fp, *, fmt=None, use_builtin_types=True, dict_type=dict) Read a plist file. *fp* should be a readable and binary file object. Return the unpacked root object (which usually is a @@ -80,7 +80,7 @@ This module defines the following functions: .. versionadded:: 3.4 -.. function:: loads(data, \*, fmt=None, use_builtin_types=True, dict_type=dict) +.. function:: loads(data, *, fmt=None, use_builtin_types=True, dict_type=dict) Load a plist from a bytes object. See :func:`load` for an explanation of the keyword arguments. @@ -88,7 +88,7 @@ This module defines the following functions: .. versionadded:: 3.4 -.. function:: dump(value, fp, \*, fmt=FMT_XML, sort_keys=True, skipkeys=False) +.. function:: dump(value, fp, *, fmt=FMT_XML, sort_keys=True, skipkeys=False) Write *value* to a plist file. *Fp* should be a writable, binary file object. @@ -116,7 +116,7 @@ This module defines the following functions: .. versionadded:: 3.4 -.. function:: dumps(value, \*, fmt=FMT_XML, sort_keys=True, skipkeys=False) +.. function:: dumps(value, *, fmt=FMT_XML, sort_keys=True, skipkeys=False) Return *value* as a plist-formatted bytes object. See the documentation for :func:`dump` for an explanation of the keyword diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index b5eeec85fb85df..578fcc6e293d9c 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -218,7 +218,7 @@ Directory and files operations copy the file more efficiently. See :ref:`shutil-platform-dependent-efficient-copy-operations` section. -.. function:: ignore_patterns(\*patterns) +.. function:: ignore_patterns(*patterns) This factory function creates a function that can be used as a callable for :func:`copytree`\'s *ignore* argument, ignoring files and directories that diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 78c7cc76d7e7bb..fb500501c8f9ae 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -478,7 +478,7 @@ class`. In addition, it provides a few more methods: .. versionadded:: 3.1 -.. method:: int.to_bytes(length, byteorder, \*, signed=False) +.. method:: int.to_bytes(length, byteorder, *, signed=False) Return an array of bytes representing an integer. @@ -510,7 +510,7 @@ class`. In addition, it provides a few more methods: .. versionadded:: 3.2 -.. classmethod:: int.from_bytes(bytes, byteorder, \*, signed=False) +.. classmethod:: int.from_bytes(bytes, byteorder, *, signed=False) Return the integer represented by the given array of bytes. diff --git a/Doc/library/sysconfig.rst b/Doc/library/sysconfig.rst index 78a1dfce9ae05c..c9306e9bf9de16 100644 --- a/Doc/library/sysconfig.rst +++ b/Doc/library/sysconfig.rst @@ -32,7 +32,7 @@ can be accessed using :func:`get_config_vars` or :func:`get_config_var`. Notice that on Windows, it's a much smaller set. -.. function:: get_config_vars(\*args) +.. function:: get_config_vars(*args) With no arguments, return a dictionary of all configuration variables relevant for the current platform. diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index 87c36aa4a2cbcf..ef81d9fa37f567 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -37,7 +37,7 @@ Some facts and figures: Added support for :mod:`lzma` compression. -.. function:: open(name=None, mode='r', fileobj=None, bufsize=10240, \*\*kwargs) +.. function:: open(name=None, mode='r', fileobj=None, bufsize=10240, **kwargs) Return a :class:`TarFile` object for the pathname *name*. For detailed information on :class:`TarFile` objects and the keyword arguments that are diff --git a/Doc/library/test.rst b/Doc/library/test.rst index da6a85d340be34..6c99f39076bdbf 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -484,7 +484,7 @@ The :mod:`test.support` module defines the following functions: Define match test with regular expression *patterns*. -.. function:: run_unittest(\*classes) +.. function:: run_unittest(*classes) Execute :class:`unittest.TestCase` subclasses passed to the function. The function scans the classes for methods starting with the prefix ``test_`` @@ -528,7 +528,7 @@ The :mod:`test.support` module defines the following functions: check_impl_detail(cpython=False) # Everywhere except CPython. -.. function:: check_warnings(\*filters, quiet=True) +.. function:: check_warnings(*filters, quiet=True) A convenience wrapper for :func:`warnings.catch_warnings()` that makes it easier to test that a warning was correctly raised. It is approximately diff --git a/Doc/library/warnings.rst b/Doc/library/warnings.rst index a481a3509d4ec8..9c1743cad23cb7 100644 --- a/Doc/library/warnings.rst +++ b/Doc/library/warnings.rst @@ -491,7 +491,7 @@ Available Functions Available Context Managers -------------------------- -.. class:: catch_warnings(\*, record=False, module=None) +.. class:: catch_warnings(*, record=False, module=None) A context manager that copies and, upon exit, restores the warnings filter and the :func:`showwarning` function. diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst index dccb7db27e90cc..487856a3ac6c60 100644 --- a/Doc/library/winreg.rst +++ b/Doc/library/winreg.rst @@ -791,7 +791,7 @@ integer handle, and also disconnect the Windows handle from the handle object. .. method:: PyHKEY.__enter__() - PyHKEY.__exit__(\*exc_info) + PyHKEY.__exit__(*exc_info) The HKEY object implements :meth:`~object.__enter__` and :meth:`~object.__exit__` and thus supports the context protocol for the diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst index 8711242d95d741..673af8326a80e3 100644 --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -164,7 +164,7 @@ module documentation. This section lists the differences between the API and The :meth:`toxml` method now preserves the attribute order specified by the user. -.. method:: Node.toprettyxml(indent="\\t", newl="\\n", encoding=None) +.. method:: Node.toprettyxml(indent="\t", newl="\n", encoding=None) Return a pretty-printed version of the document. *indent* specifies the indentation string and defaults to a tabulator; *newl* specifies the string diff --git a/Doc/requirements.txt b/Doc/requirements.txt index 47b78eeac817e0..d30ff39c8c9a33 100644 --- a/Doc/requirements.txt +++ b/Doc/requirements.txt @@ -3,7 +3,7 @@ # Sphinx version is pinned so that new versions that introduce new warnings # won't suddenly cause build failures. Updating the version is fine as long # as no warnings are raised by doing so. -sphinx==2.4.4 +sphinx blurb From b20494618c6ac6ebcd354e0b16a6645fe47acbee Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 22 Dec 2020 23:44:00 -0800 Subject: [PATCH 1901/2163] bpo-42620: Improve socket.getsockname doc string (GH-23742) Signed-off-by: Christian Heimes (cherry picked from commit cf3565ca9a7ed0f7decd000e41fa3de400986e4d) Co-authored-by: Christian Heimes --- Modules/socketmodule.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index 5dc5f4e0d397b4..3e65dc027a38ca 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -3324,8 +3324,9 @@ sock_getsockname(PySocketSockObject *s, PyObject *Py_UNUSED(ignored)) PyDoc_STRVAR(getsockname_doc, "getsockname() -> address info\n\ \n\ -Return the address of the local endpoint. For IP sockets, the address\n\ -info is a pair (hostaddr, port)."); +Return the address of the local endpoint. The format depends on the\n\ +address family. For IPv4 sockets, the address info is a pair\n\ +(hostaddr, port)."); #ifdef HAVE_GETPEERNAME /* Cray APP doesn't have this :-( */ From 412c935a37eb8fe290b684c44c80cf361f6b814b Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 23 Dec 2020 03:14:27 -0800 Subject: [PATCH 1902/2163] BPO-42703: Fix incorrect documentation links for asyncio.Event (GH-23881) (cherry picked from commit d90ff376813843310a6f9ccc96551fa1521e8fef) Co-authored-by: Matt Fowler --- Doc/library/asyncio-sync.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst index f080b03bc7c51c..c401fb50a50751 100644 --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -104,8 +104,8 @@ Event that some event has happened. An Event object manages an internal flag that can be set to *true* - with the :meth:`set` method and reset to *false* with the - :meth:`clear` method. The :meth:`wait` method blocks until the + with the :meth:`~Event.set` method and reset to *false* with the + :meth:`clear` method. The :meth:`~Event.wait` method blocks until the flag is set to *true*. The flag is set to *false* initially. @@ -142,7 +142,7 @@ Event Wait until the event is set. If the event is set, return ``True`` immediately. - Otherwise block until another task calls :meth:`set`. + Otherwise block until another task calls :meth:`~Event.set`. .. method:: set() @@ -155,8 +155,8 @@ Event Clear (unset) the event. - Tasks awaiting on :meth:`wait` will now block until the - :meth:`set` method is called again. + Tasks awaiting on :meth:`~Event.wait` will now block until the + :meth:`~Event.set` method is called again. .. method:: is_set() From efd64c8ea0fed1c13839cec0feea450820da34f8 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 24 Dec 2020 09:34:28 -0800 Subject: [PATCH 1903/2163] closes bpo-42726: gdb libpython: InstanceProxy support for py3 (GH-23912) On Fedora 31 gdb is using python 3.7.9, calling `proxyval` on an instance with a dictionary fails because of the `dict.iteritems` usage. This PR changes the code to be compatible with py2 and py3. This changed seemed small enough to not need an issue and news blurb, if one is required please let me know. Automerge-Triggered-By: GH:benjaminp (cherry picked from commit b57ada98da0d5b0cf1ebc2c9c5502d04aa962042) Co-authored-by: Augusto Hack --- .../next/Tools-Demos/2020-12-23-19-42-11.bpo-42726.a5EkTv.rst | 2 ++ Tools/gdb/libpython.py | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Tools-Demos/2020-12-23-19-42-11.bpo-42726.a5EkTv.rst diff --git a/Misc/NEWS.d/next/Tools-Demos/2020-12-23-19-42-11.bpo-42726.a5EkTv.rst b/Misc/NEWS.d/next/Tools-Demos/2020-12-23-19-42-11.bpo-42726.a5EkTv.rst new file mode 100644 index 00000000000000..01a6e7fe55f5b3 --- /dev/null +++ b/Misc/NEWS.d/next/Tools-Demos/2020-12-23-19-42-11.bpo-42726.a5EkTv.rst @@ -0,0 +1,2 @@ +Fixed Python 3 compatibility issue with gdb/libpython.py handling of attribute +dictionaries. diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py index ffb1813720e45b..aa674f6be60197 100755 --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -468,7 +468,7 @@ def __init__(self, cl_name, attrdict, address): def __repr__(self): if isinstance(self.attrdict, dict): kwargs = ', '.join(["%s=%r" % (arg, val) - for arg, val in self.attrdict.iteritems()]) + for arg, val in self.attrdict.items()]) return '<%s(%s) at remote 0x%x>' % (self.cl_name, kwargs, self.address) else: From d5aadb28545fd15cd3517b604a8c7a520abd09c6 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 24 Dec 2020 21:18:13 -0800 Subject: [PATCH 1904/2163] bpo-42388: Fix subprocess.check_output input=None when text=True (GH-23467) When the modern text= spelling of the universal_newlines= parameter was added for Python 3.7, check_output's special case around input=None was overlooked. So it behaved differently with universal_newlines=True vs text=True. This reconciles the behavior to be consistent and adds a test to guarantee it. Also clarifies the existing check_output documentation. Co-authored-by: Alexey Izbyshev (cherry picked from commit 64abf373444944a240274a9b6d66d1cb01ecfcdd) Co-authored-by: Gregory P. Smith --- Doc/library/subprocess.rst | 5 +++-- Lib/subprocess.py | 6 ++++- Lib/test/test_subprocess.py | 22 +++++++++++++++++++ .../2020-11-22-11-22-28.bpo-42388.LMgM6B.rst | 2 ++ 4 files changed, 32 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-11-22-11-22-28.bpo-42388.LMgM6B.rst diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index a93a6c186f8d80..7ce274652d394a 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -1127,8 +1127,9 @@ calls these functions. The arguments shown above are merely some common ones. The full function signature is largely the same as that of :func:`run` - most arguments are passed directly through to that interface. - However, explicitly passing ``input=None`` to inherit the parent's - standard input file handle is not supported. + One API deviation from :func:`run` behavior exists: passing ``input=None`` + will behave the same as ``input=b''`` (or ``input=''``, depending on other + arguments) rather than using the parent's standard input file handle. By default, this function will return the data as encoded bytes. The actual encoding of the output data may depend on the command being invoked, so the diff --git a/Lib/subprocess.py b/Lib/subprocess.py index 5c2c2f05093f7e..eecb1e7f253fba 100644 --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -406,7 +406,11 @@ def check_output(*popenargs, timeout=None, **kwargs): if 'input' in kwargs and kwargs['input'] is None: # Explicitly passing input=None was previously equivalent to passing an # empty string. That is maintained here for backwards compatibility. - kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b'' + if kwargs.get('universal_newlines') or kwargs.get('text'): + empty = '' + else: + empty = b'' + kwargs['input'] = empty return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, **kwargs).stdout diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index 2d3ab93555b4c3..4661d1e25280af 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -195,6 +195,28 @@ def test_check_output_input_arg(self): input=b'pear') self.assertIn(b'PEAR', output) + def test_check_output_input_none(self): + """input=None has a legacy meaning of input='' on check_output.""" + output = subprocess.check_output( + [sys.executable, "-c", + "import sys; print('XX' if sys.stdin.read() else '')"], + input=None) + self.assertNotIn(b'XX', output) + + def test_check_output_input_none_text(self): + output = subprocess.check_output( + [sys.executable, "-c", + "import sys; print('XX' if sys.stdin.read() else '')"], + input=None, text=True) + self.assertNotIn('XX', output) + + def test_check_output_input_none_universal_newlines(self): + output = subprocess.check_output( + [sys.executable, "-c", + "import sys; print('XX' if sys.stdin.read() else '')"], + input=None, universal_newlines=True) + self.assertNotIn('XX', output) + def test_check_output_stdout_arg(self): # check_output() refuses to accept 'stdout' argument with self.assertRaises(ValueError) as c: diff --git a/Misc/NEWS.d/next/Library/2020-11-22-11-22-28.bpo-42388.LMgM6B.rst b/Misc/NEWS.d/next/Library/2020-11-22-11-22-28.bpo-42388.LMgM6B.rst new file mode 100644 index 00000000000000..1b19247e841487 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-11-22-11-22-28.bpo-42388.LMgM6B.rst @@ -0,0 +1,2 @@ +Fix subprocess.check_output(..., input=None) behavior when text=True to be +consistent with that of the documentation and universal_newlines=True. From 0178a6b67ca3e782443f311e953509ca3eb4aacf Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 25 Dec 2020 07:20:35 -0800 Subject: [PATCH 1905/2163] bpo-42734: Fix crasher bogus_code_obj.py (GH-23939) It did not work because the signature of code object constructor was changed. Also, it used old format of bytecode (pre-wordcode). (cherry picked from commit 954a7427ba9c2d02faed32c02090caeca873aeca) Co-authored-by: Serhiy Storchaka --- Lib/test/crashers/bogus_code_obj.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Lib/test/crashers/bogus_code_obj.py b/Lib/test/crashers/bogus_code_obj.py index 198d229491b143..e71b3582cf2d76 100644 --- a/Lib/test/crashers/bogus_code_obj.py +++ b/Lib/test/crashers/bogus_code_obj.py @@ -14,6 +14,6 @@ import types -co = types.CodeType(0, 0, 0, 0, 0, b'\x04\x71\x00\x00', +co = types.CodeType(0, 0, 0, 0, 0, 0, b'\x04\x00\x71\x00', (), (), (), '', '', 1, b'') exec(co) From 4d840e428ab1a2712f219c5e4008658cbe15892e Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 25 Dec 2020 14:35:46 -0800 Subject: [PATCH 1906/2163] [3.8] bpo-42318: Fix support of non-BMP characters in Tkinter on macOS (GH-23281). (GH-23784) (GH-23787) (cherry picked from commit a26215db11cfcf7b5f55cab9e91396761a0e0bcf) (cherry picked from commit 28bf6ab61f77c69b732a211c398ac882bf3f65f4) --- Lib/test/test_tcl.py | 46 +++++++++++++--- .../2020-11-14-13-46-27.bpo-42318.wYAcBD.rst | 1 + Modules/_tkinter.c | 54 ++++++++++++++++++- 3 files changed, 94 insertions(+), 7 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-11-14-13-46-27.bpo-42318.wYAcBD.rst diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py index 3183ea850f73b7..6005349eaac452 100644 --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -1,4 +1,5 @@ import unittest +import locale import re import subprocess import sys @@ -58,6 +59,10 @@ def test_eval_null_in_result(self): tcl = self.interp self.assertEqual(tcl.eval('set a "a\\0b"'), 'a\x00b') + def test_eval_surrogates_in_result(self): + tcl = self.interp + self.assertIn(tcl.eval(r'set a "<\ud83d\udcbb>"'), '<\U0001f4bb>') + def testEvalException(self): tcl = self.interp self.assertRaises(TclError,tcl.eval,'set a') @@ -190,29 +195,48 @@ def test_getboolean(self): def testEvalFile(self): tcl = self.interp - with open(support.TESTFN, 'w') as f: - self.addCleanup(support.unlink, support.TESTFN) + filename = support.TESTFN + self.addCleanup(support.unlink, filename) + with open(filename, 'w') as f: f.write("""set a 1 set b 2 set c [ expr $a + $b ] """) - tcl.evalfile(support.TESTFN) + tcl.evalfile(filename) self.assertEqual(tcl.eval('set a'),'1') self.assertEqual(tcl.eval('set b'),'2') self.assertEqual(tcl.eval('set c'),'3') def test_evalfile_null_in_result(self): tcl = self.interp - with open(support.TESTFN, 'w') as f: - self.addCleanup(support.unlink, support.TESTFN) + filename = support.TESTFN + self.addCleanup(support.unlink, filename) + with open(filename, 'w') as f: f.write(""" set a "a\0b" set b "a\\0b" """) - tcl.evalfile(support.TESTFN) + tcl.evalfile(filename) self.assertEqual(tcl.eval('set a'), 'a\x00b') self.assertEqual(tcl.eval('set b'), 'a\x00b') + def test_evalfile_surrogates_in_result(self): + tcl = self.interp + encoding = tcl.call('encoding', 'system') + self.addCleanup(tcl.call, 'encoding', 'system', encoding) + tcl.call('encoding', 'system', 'utf-8') + + filename = support.TESTFN + self.addCleanup(support.unlink, filename) + with open(filename, 'wb') as f: + f.write(b""" + set a "<\xed\xa0\xbd\xed\xb2\xbb>" + set b "<\\ud83d\\udcbb>" + """) + tcl.evalfile(filename) + self.assertEqual(tcl.eval('set a'), '<\U0001f4bb>') + self.assertEqual(tcl.eval('set b'), '<\U0001f4bb>') + def testEvalFileException(self): tcl = self.interp filename = "doesnotexists" @@ -435,6 +459,11 @@ def passValue(value): self.assertEqual(passValue('str\x00ing\u20ac'), 'str\x00ing\u20ac') self.assertEqual(passValue('str\x00ing\U0001f4bb'), 'str\x00ing\U0001f4bb') + if sys.platform != 'win32': + self.assertEqual(passValue('<\udce2\udc82\udcac>'), + '<\u20ac>') + self.assertEqual(passValue('<\udced\udca0\udcbd\udced\udcb2\udcbb>'), + '<\U0001f4bb>') self.assertEqual(passValue(b'str\x00ing'), b'str\x00ing' if self.wantobjects else 'str\x00ing') self.assertEqual(passValue(b'str\xc0\x80ing'), @@ -494,6 +523,9 @@ def float_eq(actual, expected): check('string\xbd') check('string\u20ac') check('string\U0001f4bb') + if sys.platform != 'win32': + check('<\udce2\udc82\udcac>', '<\u20ac>') + check('<\udced\udca0\udcbd\udced\udcb2\udcbb>', '<\U0001f4bb>') check('') check(b'string', 'string') check(b'string\xe2\x82\xac', 'string\xe2\x82\xac') @@ -537,6 +569,8 @@ def test_splitlist(self): ('a \u20ac', ('a', '\u20ac')), ('a \U0001f4bb', ('a', '\U0001f4bb')), (b'a \xe2\x82\xac', ('a', '\u20ac')), + (b'a \xf0\x9f\x92\xbb', ('a', '\U0001f4bb')), + (b'a \xed\xa0\xbd\xed\xb2\xbb', ('a', '\U0001f4bb')), (b'a\xc0\x80b c\xc0\x80d', ('a\x00b', 'c\x00d')), ('a {b c}', ('a', 'b c')), (r'a b\ c', ('a', 'b c')), diff --git a/Misc/NEWS.d/next/Library/2020-11-14-13-46-27.bpo-42318.wYAcBD.rst b/Misc/NEWS.d/next/Library/2020-11-14-13-46-27.bpo-42318.wYAcBD.rst new file mode 100644 index 00000000000000..e72daebb2f152a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-11-14-13-46-27.bpo-42318.wYAcBD.rst @@ -0,0 +1 @@ +Fixed support of non-BMP characters in :mod:`tkinter` on macOS. diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c index a1071e5a5812f2..c1eb6e10c7f213 100644 --- a/Modules/_tkinter.c +++ b/Modules/_tkinter.c @@ -397,7 +397,8 @@ unicodeFromTclStringAndSize(const char *s, Py_ssize_t size) char *buf = NULL; PyErr_Clear(); - /* Tcl encodes null character as \xc0\x80 */ + /* Tcl encodes null character as \xc0\x80. + https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8 */ if (memchr(s, '\xc0', size)) { char *q; const char *e = s + size; @@ -421,6 +422,57 @@ unicodeFromTclStringAndSize(const char *s, Py_ssize_t size) if (buf != NULL) { PyMem_Free(buf); } + if (r == NULL || PyUnicode_KIND(r) == PyUnicode_1BYTE_KIND) { + return r; + } + + /* In CESU-8 non-BMP characters are represented as a surrogate pair, + like in UTF-16, and then each surrogate code point is encoded in UTF-8. + https://en.wikipedia.org/wiki/CESU-8 */ + Py_ssize_t len = PyUnicode_GET_LENGTH(r); + Py_ssize_t i, j; + /* All encoded surrogate characters start with \xED. */ + i = PyUnicode_FindChar(r, 0xdcED, 0, len, 1); + if (i == -2) { + Py_DECREF(r); + return NULL; + } + if (i == -1) { + return r; + } + Py_UCS4 *u = PyUnicode_AsUCS4Copy(r); + Py_DECREF(r); + if (u == NULL) { + return NULL; + } + Py_UCS4 ch; + for (j = i; i < len; i++, u[j++] = ch) { + Py_UCS4 ch1, ch2, ch3, high, low; + /* Low surrogates U+D800 - U+DBFF are encoded as + \xED\xA0\x80 - \xED\xAF\xBF. */ + ch1 = ch = u[i]; + if (ch1 != 0xdcED) continue; + ch2 = u[i + 1]; + if (!(0xdcA0 <= ch2 && ch2 <= 0xdcAF)) continue; + ch3 = u[i + 2]; + if (!(0xdc80 <= ch3 && ch3 <= 0xdcBF)) continue; + high = 0xD000 | ((ch2 & 0x3F) << 6) | (ch3 & 0x3F); + assert(Py_UNICODE_IS_HIGH_SURROGATE(high)); + /* High surrogates U+DC00 - U+DFFF are encoded as + \xED\xB0\x80 - \xED\xBF\xBF. */ + ch1 = u[i + 3]; + if (ch1 != 0xdcED) continue; + ch2 = u[i + 4]; + if (!(0xdcB0 <= ch2 && ch2 <= 0xdcBF)) continue; + ch3 = u[i + 5]; + if (!(0xdc80 <= ch3 && ch3 <= 0xdcBF)) continue; + low = 0xD000 | ((ch2 & 0x3F) << 6) | (ch3 & 0x3F); + assert(Py_UNICODE_IS_HIGH_SURROGATE(high)); + ch = Py_UNICODE_JOIN_SURROGATES(high, low); + i += 5; + } + r = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, u, j); + PyMem_Free(u); return r; } From fc6534f205d6b483298c69e7a302ccf475878a4e Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Sat, 26 Dec 2020 12:38:13 +0200 Subject: [PATCH 1907/2163] [3.8] Rename Tkinter tests for widget options (GH-23944) (GH-23946) Every test for widget option starts now with "test_configure_" to distinguish it from tests for widget commands. (cherry picked from commit c1ae21c965cb4d0566df2095e4bcb274d0bd9353) --- Lib/tkinter/test/test_tkinter/test_widgets.py | 211 +++++++++--------- Lib/tkinter/test/test_ttk/test_widgets.py | 92 ++++---- Lib/tkinter/test/widget_tests.py | 119 +++++----- 3 files changed, 212 insertions(+), 210 deletions(-) diff --git a/Lib/tkinter/test/test_tkinter/test_widgets.py b/Lib/tkinter/test/test_tkinter/test_widgets.py index 3659b9457623b4..cbb5d634e3d348 100644 --- a/Lib/tkinter/test/test_tkinter/test_widgets.py +++ b/Lib/tkinter/test/test_tkinter/test_widgets.py @@ -23,7 +23,7 @@ def float_round(x): class AbstractToplevelTest(AbstractWidgetTest, PixelSizeTests): _conv_pad_pixels = noconv - def test_class(self): + def test_configure_class(self): widget = self.create() self.assertEqual(widget['class'], widget.__class__.__name__.title()) @@ -32,7 +32,7 @@ def test_class(self): widget2 = self.create(class_='Foo') self.assertEqual(widget2['class'], 'Foo') - def test_colormap(self): + def test_configure_colormap(self): widget = self.create() self.assertEqual(widget['colormap'], '') self.checkInvalidParam(widget, 'colormap', 'new', @@ -40,7 +40,7 @@ def test_colormap(self): widget2 = self.create(colormap='new') self.assertEqual(widget2['colormap'], 'new') - def test_container(self): + def test_configure_container(self): widget = self.create() self.assertEqual(widget['container'], 0 if self.wantobjects else '0') self.checkInvalidParam(widget, 'container', 1, @@ -48,7 +48,7 @@ def test_container(self): widget2 = self.create(container=True) self.assertEqual(widget2['container'], 1 if self.wantobjects else '1') - def test_visual(self): + def test_configure_visual(self): widget = self.create() self.assertEqual(widget['visual'], '') self.checkInvalidParam(widget, 'visual', 'default', @@ -70,13 +70,13 @@ class ToplevelTest(AbstractToplevelTest, unittest.TestCase): def create(self, **kwargs): return tkinter.Toplevel(self.root, **kwargs) - def test_menu(self): + def test_configure_menu(self): widget = self.create() menu = tkinter.Menu(self.root) self.checkParam(widget, 'menu', menu, eq=widget_eq) self.checkParam(widget, 'menu', '') - def test_screen(self): + def test_configure_screen(self): widget = self.create() self.assertEqual(widget['screen'], '') try: @@ -88,7 +88,7 @@ def test_screen(self): widget2 = self.create(screen=display) self.assertEqual(widget2['screen'], display) - def test_use(self): + def test_configure_use(self): widget = self.create() self.assertEqual(widget['use'], '') parent = self.create(container=True) @@ -125,14 +125,14 @@ class LabelFrameTest(AbstractToplevelTest, unittest.TestCase): def create(self, **kwargs): return tkinter.LabelFrame(self.root, **kwargs) - def test_labelanchor(self): + def test_configure_labelanchor(self): widget = self.create() self.checkEnumParam(widget, 'labelanchor', 'e', 'en', 'es', 'n', 'ne', 'nw', 's', 'se', 'sw', 'w', 'wn', 'ws') self.checkInvalidParam(widget, 'labelanchor', 'center') - def test_labelwidget(self): + def test_configure_labelwidget(self): widget = self.create() label = tkinter.Label(self.root, text='Mupp', name='foo') self.checkParam(widget, 'labelwidget', label, expected='.foo') @@ -142,7 +142,7 @@ def test_labelwidget(self): class AbstractLabelTest(AbstractWidgetTest, IntegerSizeTests): _conv_pixels = noconv - def test_highlightthickness(self): + def test_configure_highlightthickness(self): widget = self.create() self.checkPixelsParam(widget, 'highlightthickness', 0, 1.3, 2.6, 6, -2, '10p') @@ -180,7 +180,7 @@ class ButtonTest(AbstractLabelTest, unittest.TestCase): def create(self, **kwargs): return tkinter.Button(self.root, **kwargs) - def test_default(self): + def test_configure_default(self): widget = self.create() self.checkEnumParam(widget, 'default', 'active', 'disabled', 'normal') @@ -205,11 +205,11 @@ def create(self, **kwargs): return tkinter.Checkbutton(self.root, **kwargs) - def test_offvalue(self): + def test_configure_offvalue(self): widget = self.create() self.checkParams(widget, 'offvalue', 1, 2.3, '', 'any string') - def test_onvalue(self): + def test_configure_onvalue(self): widget = self.create() self.checkParams(widget, 'onvalue', 1, 2.3, '', 'any string') @@ -232,7 +232,7 @@ class RadiobuttonTest(AbstractLabelTest, unittest.TestCase): def create(self, **kwargs): return tkinter.Radiobutton(self.root, **kwargs) - def test_value(self): + def test_configure_value(self): widget = self.create() self.checkParams(widget, 'value', 1, 2.3, '', 'any string') @@ -255,20 +255,21 @@ class MenubuttonTest(AbstractLabelTest, unittest.TestCase): def create(self, **kwargs): return tkinter.Menubutton(self.root, **kwargs) - def test_direction(self): + def test_configure_direction(self): widget = self.create() self.checkEnumParam(widget, 'direction', 'above', 'below', 'flush', 'left', 'right') - def test_height(self): + def test_configure_height(self): widget = self.create() self.checkIntegerParam(widget, 'height', 100, -100, 0, conv=str) - test_highlightthickness = StandardOptionsTests.test_highlightthickness + test_configure_highlightthickness = \ + StandardOptionsTests.test_configure_highlightthickness @unittest.skipIf(sys.platform == 'darwin', 'crashes with Cocoa Tk (issue19733)') - def test_image(self): + def test_configure_image(self): widget = self.create() image = tkinter.PhotoImage(master=self.root, name='image1') self.checkParam(widget, 'image', image, conv=str) @@ -282,23 +283,23 @@ def test_image(self): if errmsg is not None: self.assertEqual(str(cm.exception), errmsg) - def test_menu(self): + def test_configure_menu(self): widget = self.create() menu = tkinter.Menu(widget, name='menu') self.checkParam(widget, 'menu', menu, eq=widget_eq) menu.destroy() - def test_padx(self): + def test_configure_padx(self): widget = self.create() self.checkPixelsParam(widget, 'padx', 3, 4.4, 5.6, '12m') self.checkParam(widget, 'padx', -2, expected=0) - def test_pady(self): + def test_configure_pady(self): widget = self.create() self.checkPixelsParam(widget, 'pady', 3, 4.4, 5.6, '12m') self.checkParam(widget, 'pady', -2, expected=0) - def test_width(self): + def test_configure_width(self): widget = self.create() self.checkIntegerParam(widget, 'width', 402, -402, 0, conv=str) @@ -331,18 +332,18 @@ class EntryTest(AbstractWidgetTest, unittest.TestCase): def create(self, **kwargs): return tkinter.Entry(self.root, **kwargs) - def test_disabledbackground(self): + def test_configure_disabledbackground(self): widget = self.create() self.checkColorParam(widget, 'disabledbackground') - def test_insertborderwidth(self): + def test_configure_insertborderwidth(self): widget = self.create(insertwidth=100) self.checkPixelsParam(widget, 'insertborderwidth', 0, 1.3, 2.6, 6, -2, '10p') # insertborderwidth is bounded above by a half of insertwidth. self.checkParam(widget, 'insertborderwidth', 60, expected=100//2) - def test_insertwidth(self): + def test_configure_insertwidth(self): widget = self.create() self.checkPixelsParam(widget, 'insertwidth', 1.3, 3.6, '10p') self.checkParam(widget, 'insertwidth', 0.1, expected=2) @@ -352,32 +353,32 @@ def test_insertwidth(self): else: self.checkParam(widget, 'insertwidth', 0.9, expected=1) - def test_invalidcommand(self): + def test_configure_invalidcommand(self): widget = self.create() self.checkCommandParam(widget, 'invalidcommand') self.checkCommandParam(widget, 'invcmd') - def test_readonlybackground(self): + def test_configure_readonlybackground(self): widget = self.create() self.checkColorParam(widget, 'readonlybackground') - def test_show(self): + def test_configure_show(self): widget = self.create() self.checkParam(widget, 'show', '*') self.checkParam(widget, 'show', '') self.checkParam(widget, 'show', ' ') - def test_state(self): + def test_configure_state(self): widget = self.create() self.checkEnumParam(widget, 'state', 'disabled', 'normal', 'readonly') - def test_validate(self): + def test_configure_validate(self): widget = self.create() self.checkEnumParam(widget, 'validate', 'all', 'key', 'focus', 'focusin', 'focusout', 'none') - def test_validatecommand(self): + def test_configure_validatecommand(self): widget = self.create() self.checkCommandParam(widget, 'validatecommand') self.checkCommandParam(widget, 'vcmd') @@ -430,25 +431,25 @@ class SpinboxTest(EntryTest, unittest.TestCase): def create(self, **kwargs): return tkinter.Spinbox(self.root, **kwargs) - test_show = None + test_configure_show = None - def test_buttonbackground(self): + def test_configure_buttonbackground(self): widget = self.create() self.checkColorParam(widget, 'buttonbackground') - def test_buttoncursor(self): + def test_configure_buttoncursor(self): widget = self.create() self.checkCursorParam(widget, 'buttoncursor') - def test_buttondownrelief(self): + def test_configure_buttondownrelief(self): widget = self.create() self.checkReliefParam(widget, 'buttondownrelief') - def test_buttonuprelief(self): + def test_configure_buttonuprelief(self): widget = self.create() self.checkReliefParam(widget, 'buttonuprelief') - def test_format(self): + def test_configure_format(self): widget = self.create() self.checkParam(widget, 'format', '%2f') self.checkParam(widget, 'format', '%2.2f') @@ -463,25 +464,25 @@ def test_format(self): self.checkParam(widget, 'format', '%09.200f') self.checkInvalidParam(widget, 'format', '%d') - def test_from(self): + def test_configure_from(self): widget = self.create() self.checkParam(widget, 'to', 100.0) self.checkFloatParam(widget, 'from', -10, 10.2, 11.7) self.checkInvalidParam(widget, 'from', 200, errmsg='-to value must be greater than -from value') - def test_increment(self): + def test_configure_increment(self): widget = self.create() self.checkFloatParam(widget, 'increment', -1, 1, 10.2, 12.8, 0) - def test_to(self): + def test_configure_to(self): widget = self.create() self.checkParam(widget, 'from', -100.0) self.checkFloatParam(widget, 'to', -10, 10.2, 11.7) self.checkInvalidParam(widget, 'to', -200, errmsg='-to value must be greater than -from value') - def test_values(self): + def test_configure_values(self): # XXX widget = self.create() self.assertEqual(widget['values'], '') @@ -492,7 +493,7 @@ def test_values(self): expected='42 3.14 {} {any string}') self.checkParam(widget, 'values', '') - def test_wrap(self): + def test_configure_wrap(self): widget = self.create() self.checkBooleanParam(widget, 'wrap') @@ -558,17 +559,17 @@ class TextTest(AbstractWidgetTest, unittest.TestCase): def create(self, **kwargs): return tkinter.Text(self.root, **kwargs) - def test_autoseparators(self): + def test_configure_autoseparators(self): widget = self.create() self.checkBooleanParam(widget, 'autoseparators') @requires_tcl(8, 5) - def test_blockcursor(self): + def test_configure_blockcursor(self): widget = self.create() self.checkBooleanParam(widget, 'blockcursor') @requires_tcl(8, 5) - def test_endline(self): + def test_configure_endline(self): widget = self.create() text = '\n'.join('Line %d' for i in range(100)) widget.insert('end', text) @@ -581,50 +582,50 @@ def test_endline(self): self.checkInvalidParam(widget, 'endline', 10, errmsg='-startline must be less than or equal to -endline') - def test_height(self): + def test_configure_height(self): widget = self.create() self.checkPixelsParam(widget, 'height', 100, 101.2, 102.6, '3c') self.checkParam(widget, 'height', -100, expected=1) self.checkParam(widget, 'height', 0, expected=1) - def test_maxundo(self): + def test_configure_maxundo(self): widget = self.create() self.checkIntegerParam(widget, 'maxundo', 0, 5, -1) @requires_tcl(8, 5) - def test_inactiveselectbackground(self): + def test_configure_inactiveselectbackground(self): widget = self.create() self.checkColorParam(widget, 'inactiveselectbackground') @requires_tcl(8, 6) - def test_insertunfocussed(self): + def test_configure_insertunfocussed(self): widget = self.create() self.checkEnumParam(widget, 'insertunfocussed', 'hollow', 'none', 'solid') - def test_selectborderwidth(self): + def test_configure_selectborderwidth(self): widget = self.create() self.checkPixelsParam(widget, 'selectborderwidth', 1.3, 2.6, -2, '10p', conv=noconv, keep_orig=tcl_version >= (8, 5)) - def test_spacing1(self): + def test_configure_spacing1(self): widget = self.create() self.checkPixelsParam(widget, 'spacing1', 20, 21.4, 22.6, '0.5c') self.checkParam(widget, 'spacing1', -5, expected=0) - def test_spacing2(self): + def test_configure_spacing2(self): widget = self.create() self.checkPixelsParam(widget, 'spacing2', 5, 6.4, 7.6, '0.1c') self.checkParam(widget, 'spacing2', -1, expected=0) - def test_spacing3(self): + def test_configure_spacing3(self): widget = self.create() self.checkPixelsParam(widget, 'spacing3', 20, 21.4, 22.6, '0.5c') self.checkParam(widget, 'spacing3', -10, expected=0) @requires_tcl(8, 5) - def test_startline(self): + def test_configure_startline(self): widget = self.create() text = '\n'.join('Line %d' for i in range(100)) widget.insert('end', text) @@ -637,14 +638,14 @@ def test_startline(self): self.checkInvalidParam(widget, 'startline', 70, errmsg='-startline must be less than or equal to -endline') - def test_state(self): + def test_configure_state(self): widget = self.create() if tcl_version < (8, 5): self.checkParams(widget, 'state', 'disabled', 'normal') else: self.checkEnumParam(widget, 'state', 'disabled', 'normal') - def test_tabs(self): + def test_configure_tabs(self): widget = self.create() if get_tk_patchlevel() < (8, 5, 11): self.checkParam(widget, 'tabs', (10.2, 20.7, '1i', '2i'), @@ -660,21 +661,21 @@ def test_tabs(self): keep_orig=tcl_version >= (8, 5)) @requires_tcl(8, 5) - def test_tabstyle(self): + def test_configure_tabstyle(self): widget = self.create() self.checkEnumParam(widget, 'tabstyle', 'tabular', 'wordprocessor') - def test_undo(self): + def test_configure_undo(self): widget = self.create() self.checkBooleanParam(widget, 'undo') - def test_width(self): + def test_configure_width(self): widget = self.create() self.checkIntegerParam(widget, 'width', 402) self.checkParam(widget, 'width', -402, expected=1) self.checkParam(widget, 'width', 0, expected=1) - def test_wrap(self): + def test_configure_wrap(self): widget = self.create() if tcl_version < (8, 5): self.checkParams(widget, 'wrap', 'char', 'none', 'word') @@ -712,16 +713,16 @@ class CanvasTest(AbstractWidgetTest, unittest.TestCase): def create(self, **kwargs): return tkinter.Canvas(self.root, **kwargs) - def test_closeenough(self): + def test_configure_closeenough(self): widget = self.create() self.checkFloatParam(widget, 'closeenough', 24, 2.4, 3.6, -3, conv=float) - def test_confine(self): + def test_configure_confine(self): widget = self.create() self.checkBooleanParam(widget, 'confine') - def test_offset(self): + def test_configure_offset(self): widget = self.create() self.assertEqual(widget['offset'], '0,0') self.checkParams(widget, 'offset', @@ -730,7 +731,7 @@ def test_offset(self): self.checkParam(widget, 'offset', '#5,6') self.checkInvalidParam(widget, 'offset', 'spam') - def test_scrollregion(self): + def test_configure_scrollregion(self): widget = self.create() self.checkParam(widget, 'scrollregion', '0 0 200 150') self.checkParam(widget, 'scrollregion', (0, 0, 200, 150), @@ -742,17 +743,17 @@ def test_scrollregion(self): self.checkInvalidParam(widget, 'scrollregion', (0, 0, 200)) self.checkInvalidParam(widget, 'scrollregion', (0, 0, 200, 150, 0)) - def test_state(self): + def test_configure_state(self): widget = self.create() self.checkEnumParam(widget, 'state', 'disabled', 'normal', errmsg='bad state value "{}": must be normal or disabled') - def test_xscrollincrement(self): + def test_configure_xscrollincrement(self): widget = self.create() self.checkPixelsParam(widget, 'xscrollincrement', 40, 0, 41.2, 43.6, -40, '0.5i') - def test_yscrollincrement(self): + def test_configure_yscrollincrement(self): widget = self.create() self.checkPixelsParam(widget, 'yscrollincrement', 10, 0, 11.2, 13.6, -10, '0.1i') @@ -797,26 +798,26 @@ class ListboxTest(AbstractWidgetTest, unittest.TestCase): def create(self, **kwargs): return tkinter.Listbox(self.root, **kwargs) - def test_activestyle(self): + def test_configure_activestyle(self): widget = self.create() self.checkEnumParam(widget, 'activestyle', 'dotbox', 'none', 'underline') - test_justify = requires_tcl(8, 6, 5)(StandardOptionsTests.test_justify) + test_justify = requires_tcl(8, 6, 5)(StandardOptionsTests.test_configure_justify) - def test_listvariable(self): + def test_configure_listvariable(self): widget = self.create() var = tkinter.DoubleVar(self.root) self.checkVariableParam(widget, 'listvariable', var) - def test_selectmode(self): + def test_configure_selectmode(self): widget = self.create() self.checkParam(widget, 'selectmode', 'single') self.checkParam(widget, 'selectmode', 'browse') self.checkParam(widget, 'selectmode', 'multiple') self.checkParam(widget, 'selectmode', 'extended') - def test_state(self): + def test_configure_state(self): widget = self.create() self.checkEnumParam(widget, 'state', 'disabled', 'normal') @@ -931,53 +932,53 @@ class ScaleTest(AbstractWidgetTest, unittest.TestCase): def create(self, **kwargs): return tkinter.Scale(self.root, **kwargs) - def test_bigincrement(self): + def test_configure_bigincrement(self): widget = self.create() self.checkFloatParam(widget, 'bigincrement', 12.4, 23.6, -5) - def test_digits(self): + def test_configure_digits(self): widget = self.create() self.checkIntegerParam(widget, 'digits', 5, 0) - def test_from(self): + def test_configure_from(self): widget = self.create() conv = False if get_tk_patchlevel() >= (8, 6, 10) else float_round self.checkFloatParam(widget, 'from', 100, 14.9, 15.1, conv=conv) - def test_label(self): + def test_configure_label(self): widget = self.create() self.checkParam(widget, 'label', 'any string') self.checkParam(widget, 'label', '') - def test_length(self): + def test_configure_length(self): widget = self.create() self.checkPixelsParam(widget, 'length', 130, 131.2, 135.6, '5i') - def test_resolution(self): + def test_configure_resolution(self): widget = self.create() self.checkFloatParam(widget, 'resolution', 4.2, 0, 6.7, -2) - def test_showvalue(self): + def test_configure_showvalue(self): widget = self.create() self.checkBooleanParam(widget, 'showvalue') - def test_sliderlength(self): + def test_configure_sliderlength(self): widget = self.create() self.checkPixelsParam(widget, 'sliderlength', 10, 11.2, 15.6, -3, '3m') - def test_sliderrelief(self): + def test_configure_sliderrelief(self): widget = self.create() self.checkReliefParam(widget, 'sliderrelief') - def test_tickinterval(self): + def test_configure_tickinterval(self): widget = self.create() self.checkFloatParam(widget, 'tickinterval', 1, 4.3, 7.6, 0, conv=float_round) self.checkParam(widget, 'tickinterval', -2, expected=2, conv=float_round) - def test_to(self): + def test_configure_to(self): widget = self.create() self.checkFloatParam(widget, 'to', 300, 14.9, 15.1, -10, conv=float_round) @@ -1001,15 +1002,15 @@ class ScrollbarTest(AbstractWidgetTest, unittest.TestCase): def create(self, **kwargs): return tkinter.Scrollbar(self.root, **kwargs) - def test_activerelief(self): + def test_configure_activerelief(self): widget = self.create() self.checkReliefParam(widget, 'activerelief') - def test_elementborderwidth(self): + def test_configure_elementborderwidth(self): widget = self.create() self.checkPixelsParam(widget, 'elementborderwidth', 4.3, 5.6, -2, '1m') - def test_orient(self): + def test_configure_orient(self): widget = self.create() self.checkEnumParam(widget, 'orient', 'vertical', 'horizontal', errmsg='bad orientation "{}": must be vertical or horizontal') @@ -1050,63 +1051,63 @@ class PanedWindowTest(AbstractWidgetTest, unittest.TestCase): def create(self, **kwargs): return tkinter.PanedWindow(self.root, **kwargs) - def test_handlepad(self): + def test_configure_handlepad(self): widget = self.create() self.checkPixelsParam(widget, 'handlepad', 5, 6.4, 7.6, -3, '1m') - def test_handlesize(self): + def test_configure_handlesize(self): widget = self.create() self.checkPixelsParam(widget, 'handlesize', 8, 9.4, 10.6, -3, '2m', conv=noconv) - def test_height(self): + def test_configure_height(self): widget = self.create() self.checkPixelsParam(widget, 'height', 100, 101.2, 102.6, -100, 0, '1i', conv=noconv) - def test_opaqueresize(self): + def test_configure_opaqueresize(self): widget = self.create() self.checkBooleanParam(widget, 'opaqueresize') @requires_tcl(8, 6, 5) - def test_proxybackground(self): + def test_configure_proxybackground(self): widget = self.create() self.checkColorParam(widget, 'proxybackground') @requires_tcl(8, 6, 5) - def test_proxyborderwidth(self): + def test_configure_proxyborderwidth(self): widget = self.create() self.checkPixelsParam(widget, 'proxyborderwidth', 0, 1.3, 2.9, 6, -2, '10p', conv=noconv) @requires_tcl(8, 6, 5) - def test_proxyrelief(self): + def test_configure_proxyrelief(self): widget = self.create() self.checkReliefParam(widget, 'proxyrelief') - def test_sashcursor(self): + def test_configure_sashcursor(self): widget = self.create() self.checkCursorParam(widget, 'sashcursor') - def test_sashpad(self): + def test_configure_sashpad(self): widget = self.create() self.checkPixelsParam(widget, 'sashpad', 8, 1.3, 2.6, -2, '2m') - def test_sashrelief(self): + def test_configure_sashrelief(self): widget = self.create() self.checkReliefParam(widget, 'sashrelief') - def test_sashwidth(self): + def test_configure_sashwidth(self): widget = self.create() self.checkPixelsParam(widget, 'sashwidth', 10, 11.1, 15.6, -3, '1m', conv=noconv) - def test_showhandle(self): + def test_configure_showhandle(self): widget = self.create() self.checkBooleanParam(widget, 'showhandle') - def test_width(self): + def test_configure_width(self): widget = self.create() self.checkPixelsParam(widget, 'width', 402, 403.4, 404.6, -402, 0, '5i', conv=noconv) @@ -1225,23 +1226,23 @@ class MenuTest(AbstractWidgetTest, unittest.TestCase): def create(self, **kwargs): return tkinter.Menu(self.root, **kwargs) - def test_postcommand(self): + def test_configure_postcommand(self): widget = self.create() self.checkCommandParam(widget, 'postcommand') - def test_tearoff(self): + def test_configure_tearoff(self): widget = self.create() self.checkBooleanParam(widget, 'tearoff') - def test_tearoffcommand(self): + def test_configure_tearoffcommand(self): widget = self.create() self.checkCommandParam(widget, 'tearoffcommand') - def test_title(self): + def test_configure_title(self): widget = self.create() self.checkParam(widget, 'title', 'any string') - def test_type(self): + def test_configure_type(self): widget = self.create() self.checkEnumParam(widget, 'type', 'normal', 'tearoff', 'menubar') @@ -1294,7 +1295,7 @@ class MessageTest(AbstractWidgetTest, unittest.TestCase): def create(self, **kwargs): return tkinter.Message(self.root, **kwargs) - def test_aspect(self): + def test_configure_aspect(self): widget = self.create() self.checkIntegerParam(widget, 'aspect', 250, 0, -300) diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index de30e2476b4eb9..1fac83a004a6d0 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -16,7 +16,7 @@ class StandardTtkOptionsTests(StandardOptionsTests): - def test_class(self): + def test_configure_class(self): widget = self.create() self.assertEqual(widget['class'], '') errmsg='attempt to change read-only option' @@ -26,7 +26,7 @@ def test_class(self): widget2 = self.create(class_='Foo') self.assertEqual(widget2['class'], 'Foo') - def test_padding(self): + def test_configure_padding(self): widget = self.create() self.checkParam(widget, 'padding', 0, expected=('0',)) self.checkParam(widget, 'padding', 5, expected=('5',)) @@ -38,7 +38,7 @@ def test_padding(self): self.checkParam(widget, 'padding', ('5p', '6p', '7p', '8p')) self.checkParam(widget, 'padding', (), expected='') - def test_style(self): + def test_configure_style(self): widget = self.create() self.assertEqual(widget['style'], '') errmsg = 'Layout Foo not found' @@ -139,14 +139,14 @@ class LabelFrameTest(AbstractToplevelTest, unittest.TestCase): def create(self, **kwargs): return ttk.LabelFrame(self.root, **kwargs) - def test_labelanchor(self): + def test_configure_labelanchor(self): widget = self.create() self.checkEnumParam(widget, 'labelanchor', 'e', 'en', 'es', 'n', 'ne', 'nw', 's', 'se', 'sw', 'w', 'wn', 'ws', errmsg='Bad label anchor specification {}') self.checkInvalidParam(widget, 'labelanchor', 'center') - def test_labelwidget(self): + def test_configure_labelwidget(self): widget = self.create() label = ttk.Label(self.root, text='Mupp', name='foo') self.checkParam(widget, 'labelwidget', label, expected='.foo') @@ -168,17 +168,17 @@ def checkImageParam(self, widget, name): self.checkInvalidParam(widget, name, 'spam', errmsg='image "spam" doesn\'t exist') - def test_compound(self): + def test_configure_compound(self): widget = self.create() self.checkEnumParam(widget, 'compound', 'none', 'text', 'image', 'center', 'top', 'bottom', 'left', 'right') - def test_state(self): + def test_configure_state(self): widget = self.create() self.checkParams(widget, 'state', 'active', 'disabled', 'normal') - def test_width(self): + def test_configure_width(self): widget = self.create() self.checkParams(widget, 'width', 402, -402, 0) @@ -197,7 +197,7 @@ class LabelTest(AbstractLabelTest, unittest.TestCase): def create(self, **kwargs): return ttk.Label(self.root, **kwargs) - def test_font(self): + def test_configure_font(self): widget = self.create() self.checkParam(widget, 'font', '-Adobe-Helvetica-Medium-R-Normal--*-120-*-*-*-*-*-*') @@ -215,7 +215,7 @@ class ButtonTest(AbstractLabelTest, unittest.TestCase): def create(self, **kwargs): return ttk.Button(self.root, **kwargs) - def test_default(self): + def test_configure_default(self): widget = self.create() self.checkEnumParam(widget, 'default', 'normal', 'active', 'disabled') @@ -240,11 +240,11 @@ class CheckbuttonTest(AbstractLabelTest, unittest.TestCase): def create(self, **kwargs): return ttk.Checkbutton(self.root, **kwargs) - def test_offvalue(self): + def test_configure_offvalue(self): widget = self.create() self.checkParams(widget, 'offvalue', 1, 2.3, '', 'any string') - def test_onvalue(self): + def test_configure_onvalue(self): widget = self.create() self.checkParams(widget, 'onvalue', 1, 2.3, '', 'any string') @@ -292,27 +292,27 @@ def setUp(self): def create(self, **kwargs): return ttk.Entry(self.root, **kwargs) - def test_invalidcommand(self): + def test_configure_invalidcommand(self): widget = self.create() self.checkCommandParam(widget, 'invalidcommand') - def test_show(self): + def test_configure_show(self): widget = self.create() self.checkParam(widget, 'show', '*') self.checkParam(widget, 'show', '') self.checkParam(widget, 'show', ' ') - def test_state(self): + def test_configure_state(self): widget = self.create() self.checkParams(widget, 'state', 'disabled', 'normal', 'readonly') - def test_validate(self): + def test_configure_validate(self): widget = self.create() self.checkEnumParam(widget, 'validate', 'all', 'key', 'focus', 'focusin', 'focusout', 'none') - def test_validatecommand(self): + def test_configure_validatecommand(self): widget = self.create() self.checkCommandParam(widget, 'validatecommand') @@ -429,7 +429,7 @@ def setUp(self): def create(self, **kwargs): return ttk.Combobox(self.root, **kwargs) - def test_height(self): + def test_configure_height(self): widget = self.create() self.checkParams(widget, 'height', 100, 101.2, 102.6, -100, 0, '1i') @@ -459,7 +459,7 @@ def test_virtual_event(self): self.assertTrue(success) - def test_postcommand(self): + def test_configure_postcommand(self): success = [] self.combo['postcommand'] = lambda: success.append(True) @@ -475,7 +475,7 @@ def test_postcommand(self): self.assertEqual(len(success), 1) - def test_values(self): + def test_configure_values(self): def check_get_current(getval, currval): self.assertEqual(self.combo.get(), getval) self.assertEqual(self.combo.current(), currval) @@ -551,7 +551,7 @@ def setUp(self): def create(self, **kwargs): return ttk.PanedWindow(self.root, **kwargs) - def test_orient(self): + def test_configure_orient(self): widget = self.create() self.assertEqual(str(widget['orient']), 'vertical') errmsg='attempt to change read-only option' @@ -684,11 +684,11 @@ class RadiobuttonTest(AbstractLabelTest, unittest.TestCase): def create(self, **kwargs): return ttk.Radiobutton(self.root, **kwargs) - def test_value(self): + def test_configure_value(self): widget = self.create() self.checkParams(widget, 'value', 1, 2.3, '', 'any string') - def test_invoke(self): + def test_configure_invoke(self): success = [] def cb_test(): success.append(1) @@ -739,7 +739,7 @@ def test_direction(self): self.checkEnumParam(widget, 'direction', 'above', 'below', 'left', 'right', 'flush') - def test_menu(self): + def test_configure_menu(self): widget = self.create() menu = tkinter.Menu(widget, name='menu') self.checkParam(widget, 'menu', menu, conv=str) @@ -764,19 +764,19 @@ def setUp(self): def create(self, **kwargs): return ttk.Scale(self.root, **kwargs) - def test_from(self): + def test_configure_from(self): widget = self.create() self.checkFloatParam(widget, 'from', 100, 14.9, 15.1, conv=False) - def test_length(self): + def test_configure_length(self): widget = self.create() self.checkPixelsParam(widget, 'length', 130, 131.2, 135.6, '5i') - def test_to(self): + def test_configure_to(self): widget = self.create() self.checkFloatParam(widget, 'to', 300, 14.9, 15.1, -10, conv=False) - def test_value(self): + def test_configure_value(self): widget = self.create() self.checkFloatParam(widget, 'value', 300, 14.9, 15.1, -10, conv=False) @@ -866,23 +866,23 @@ class ProgressbarTest(AbstractWidgetTest, unittest.TestCase): def create(self, **kwargs): return ttk.Progressbar(self.root, **kwargs) - def test_length(self): + def test_configure_length(self): widget = self.create() self.checkPixelsParam(widget, 'length', 100.1, 56.7, '2i') - def test_maximum(self): + def test_configure_maximum(self): widget = self.create() self.checkFloatParam(widget, 'maximum', 150.2, 77.7, 0, -10, conv=False) - def test_mode(self): + def test_configure_mode(self): widget = self.create() self.checkEnumParam(widget, 'mode', 'determinate', 'indeterminate') - def test_phase(self): + def test_configure_phase(self): # XXX pass - def test_value(self): + def test_configure_value(self): widget = self.create() self.checkFloatParam(widget, 'value', 150.2, 77.7, 0, -10, conv=False) @@ -1071,7 +1071,7 @@ def test_tab(self): self.assertEqual(self.nb.tab(self.child1, 'text'), 'abc') - def test_tabs(self): + def test_configure_tabs(self): self.assertEqual(len(self.nb.tabs()), 2) self.nb.forget(self.child1) @@ -1147,7 +1147,7 @@ def _click_decrement_arrow(self): self.spin.event_generate('', x=x, y=y) self.spin.update_idletasks() - def test_command(self): + def test_configure_command(self): success = [] self.spin['command'] = lambda: success.append(True) @@ -1167,7 +1167,7 @@ def test_command(self): self.spin.update() self.assertEqual(len(success), 2) - def test_to(self): + def test_configure_to(self): self.spin['from'] = 0 self.spin['to'] = 5 self.spin.set(4) @@ -1179,7 +1179,7 @@ def test_to(self): self._click_increment_arrow() # 5 self.assertEqual(self.spin.get(), '5') - def test_from(self): + def test_configure_from(self): self.spin['from'] = 1 self.spin['to'] = 10 self.spin.set(2) @@ -1189,7 +1189,7 @@ def test_from(self): self._click_decrement_arrow() # 1 self.assertEqual(self.spin.get(), '1') - def test_increment(self): + def test_configure_increment(self): self.spin['from'] = 0 self.spin['to'] = 10 self.spin['increment'] = 4 @@ -1203,7 +1203,7 @@ def test_increment(self): self._click_decrement_arrow() # 3 self.assertEqual(self.spin.get(), '3') - def test_format(self): + def test_configure_format(self): self.spin.set(1) self.spin['format'] = '%10.3f' self.spin.update() @@ -1220,7 +1220,7 @@ def test_format(self): self.assertTrue('.' not in value) self.assertEqual(len(value), 1) - def test_wrap(self): + def test_configure_wrap(self): self.spin['to'] = 10 self.spin['from'] = 1 self.spin.set(1) @@ -1239,7 +1239,7 @@ def test_wrap(self): self._click_decrement_arrow() self.assertEqual(self.spin.get(), '1') - def test_values(self): + def test_configure_values(self): self.assertEqual(self.spin['values'], () if tcl_version < (8, 5) else '') self.checkParam(self.spin, 'values', 'mon tue wed thur', @@ -1299,14 +1299,14 @@ def setUp(self): def create(self, **kwargs): return ttk.Treeview(self.root, **kwargs) - def test_columns(self): + def test_configure_columns(self): widget = self.create() self.checkParam(widget, 'columns', 'a b c', expected=('a', 'b', 'c')) self.checkParam(widget, 'columns', ('a', 'b', 'c')) self.checkParam(widget, 'columns', '') - def test_displaycolumns(self): + def test_configure_displaycolumns(self): widget = self.create() widget['columns'] = ('a', 'b', 'c') self.checkParam(widget, 'displaycolumns', 'b a c', @@ -1322,17 +1322,17 @@ def test_displaycolumns(self): self.checkInvalidParam(widget, 'displaycolumns', (1, -2), errmsg='Column index -2 out of bounds') - def test_height(self): + def test_configure_height(self): widget = self.create() self.checkPixelsParam(widget, 'height', 100, -100, 0, '3c', conv=False) self.checkPixelsParam(widget, 'height', 101.2, 102.6, conv=noconv) - def test_selectmode(self): + def test_configure_selectmode(self): widget = self.create() self.checkEnumParam(widget, 'selectmode', 'none', 'browse', 'extended') - def test_show(self): + def test_configure_show(self): widget = self.create() self.checkParam(widget, 'show', 'tree headings', expected=('tree', 'headings')) diff --git a/Lib/tkinter/test/widget_tests.py b/Lib/tkinter/test/widget_tests.py index b42ff52178f29e..1224bd2e0e9d95 100644 --- a/Lib/tkinter/test/widget_tests.py +++ b/Lib/tkinter/test/widget_tests.py @@ -243,31 +243,31 @@ class StandardOptionsTests: 'underline', 'wraplength', 'xscrollcommand', 'yscrollcommand', ) - def test_activebackground(self): + def test_configure_activebackground(self): widget = self.create() self.checkColorParam(widget, 'activebackground') - def test_activeborderwidth(self): + def test_configure_activeborderwidth(self): widget = self.create() self.checkPixelsParam(widget, 'activeborderwidth', 0, 1.3, 2.9, 6, -2, '10p') - def test_activeforeground(self): + def test_configure_activeforeground(self): widget = self.create() self.checkColorParam(widget, 'activeforeground') - def test_anchor(self): + def test_configure_anchor(self): widget = self.create() self.checkEnumParam(widget, 'anchor', 'n', 'ne', 'e', 'se', 's', 'sw', 'w', 'nw', 'center') - def test_background(self): + def test_configure_background(self): widget = self.create() self.checkColorParam(widget, 'background') if 'bg' in self.OPTIONS: self.checkColorParam(widget, 'bg') - def test_bitmap(self): + def test_configure_bitmap(self): widget = self.create() self.checkParam(widget, 'bitmap', 'questhead') self.checkParam(widget, 'bitmap', 'gray50') @@ -280,52 +280,52 @@ def test_bitmap(self): self.checkInvalidParam(widget, 'bitmap', 'spam', errmsg='bitmap "spam" not defined') - def test_borderwidth(self): + def test_configure_borderwidth(self): widget = self.create() self.checkPixelsParam(widget, 'borderwidth', 0, 1.3, 2.6, 6, -2, '10p') if 'bd' in self.OPTIONS: self.checkPixelsParam(widget, 'bd', 0, 1.3, 2.6, 6, -2, '10p') - def test_compound(self): + def test_configure_compound(self): widget = self.create() self.checkEnumParam(widget, 'compound', 'bottom', 'center', 'left', 'none', 'right', 'top') - def test_cursor(self): + def test_configure_cursor(self): widget = self.create() self.checkCursorParam(widget, 'cursor') - def test_disabledforeground(self): + def test_configure_disabledforeground(self): widget = self.create() self.checkColorParam(widget, 'disabledforeground') - def test_exportselection(self): + def test_configure_exportselection(self): widget = self.create() self.checkBooleanParam(widget, 'exportselection') - def test_font(self): + def test_configure_font(self): widget = self.create() self.checkParam(widget, 'font', '-Adobe-Helvetica-Medium-R-Normal--*-120-*-*-*-*-*-*') self.checkInvalidParam(widget, 'font', '', errmsg='font "" doesn\'t exist') - def test_foreground(self): + def test_configure_foreground(self): widget = self.create() self.checkColorParam(widget, 'foreground') if 'fg' in self.OPTIONS: self.checkColorParam(widget, 'fg') - def test_highlightbackground(self): + def test_configure_highlightbackground(self): widget = self.create() self.checkColorParam(widget, 'highlightbackground') - def test_highlightcolor(self): + def test_configure_highlightcolor(self): widget = self.create() self.checkColorParam(widget, 'highlightcolor') - def test_highlightthickness(self): + def test_configure_highlightthickness(self): widget = self.create() self.checkPixelsParam(widget, 'highlightthickness', 0, 1.3, 2.6, 6, '10p') @@ -334,36 +334,36 @@ def test_highlightthickness(self): @unittest.skipIf(sys.platform == 'darwin', 'crashes with Cocoa Tk (issue19733)') - def test_image(self): + def test_configure_image(self): widget = self.create() self.checkImageParam(widget, 'image') - def test_insertbackground(self): + def test_configure_insertbackground(self): widget = self.create() self.checkColorParam(widget, 'insertbackground') - def test_insertborderwidth(self): + def test_configure_insertborderwidth(self): widget = self.create() self.checkPixelsParam(widget, 'insertborderwidth', 0, 1.3, 2.6, 6, -2, '10p') - def test_insertofftime(self): + def test_configure_insertofftime(self): widget = self.create() self.checkIntegerParam(widget, 'insertofftime', 100) - def test_insertontime(self): + def test_configure_insertontime(self): widget = self.create() self.checkIntegerParam(widget, 'insertontime', 100) - def test_insertwidth(self): + def test_configure_insertwidth(self): widget = self.create() self.checkPixelsParam(widget, 'insertwidth', 1.3, 2.6, -2, '10p') - def test_jump(self): + def test_configure_jump(self): widget = self.create() self.checkBooleanParam(widget, 'jump') - def test_justify(self): + def test_configure_justify(self): widget = self.create() self.checkEnumParam(widget, 'justify', 'left', 'right', 'center', errmsg='bad justification "{}": must be ' @@ -372,154 +372,155 @@ def test_justify(self): errmsg='ambiguous justification "": must be ' 'left, right, or center') - def test_orient(self): + def test_configure_orient(self): widget = self.create() self.assertEqual(str(widget['orient']), self.default_orient) self.checkEnumParam(widget, 'orient', 'horizontal', 'vertical') - def test_padx(self): + def test_configure_padx(self): widget = self.create() self.checkPixelsParam(widget, 'padx', 3, 4.4, 5.6, -2, '12m', conv=self._conv_pad_pixels) - def test_pady(self): + def test_configure_pady(self): widget = self.create() self.checkPixelsParam(widget, 'pady', 3, 4.4, 5.6, -2, '12m', conv=self._conv_pad_pixels) - def test_relief(self): + def test_configure_relief(self): widget = self.create() self.checkReliefParam(widget, 'relief') - def test_repeatdelay(self): + def test_configure_repeatdelay(self): widget = self.create() self.checkIntegerParam(widget, 'repeatdelay', -500, 500) - def test_repeatinterval(self): + def test_configure_repeatinterval(self): widget = self.create() self.checkIntegerParam(widget, 'repeatinterval', -500, 500) - def test_selectbackground(self): + def test_configure_selectbackground(self): widget = self.create() self.checkColorParam(widget, 'selectbackground') - def test_selectborderwidth(self): + def test_configure_selectborderwidth(self): widget = self.create() self.checkPixelsParam(widget, 'selectborderwidth', 1.3, 2.6, -2, '10p') - def test_selectforeground(self): + def test_configure_selectforeground(self): widget = self.create() self.checkColorParam(widget, 'selectforeground') - def test_setgrid(self): + def test_configure_setgrid(self): widget = self.create() self.checkBooleanParam(widget, 'setgrid') - def test_state(self): + def test_configure_state(self): widget = self.create() self.checkEnumParam(widget, 'state', 'active', 'disabled', 'normal') - def test_takefocus(self): + def test_configure_takefocus(self): widget = self.create() self.checkParams(widget, 'takefocus', '0', '1', '') - def test_text(self): + def test_configure_text(self): widget = self.create() self.checkParams(widget, 'text', '', 'any string') - def test_textvariable(self): + def test_configure_textvariable(self): widget = self.create() var = tkinter.StringVar(self.root) self.checkVariableParam(widget, 'textvariable', var) - def test_troughcolor(self): + def test_configure_troughcolor(self): widget = self.create() self.checkColorParam(widget, 'troughcolor') - def test_underline(self): + def test_configure_underline(self): widget = self.create() self.checkIntegerParam(widget, 'underline', 0, 1, 10) - def test_wraplength(self): + def test_configure_wraplength(self): widget = self.create() self.checkPixelsParam(widget, 'wraplength', 100) - def test_xscrollcommand(self): + def test_configure_xscrollcommand(self): widget = self.create() self.checkCommandParam(widget, 'xscrollcommand') - def test_yscrollcommand(self): + def test_configure_yscrollcommand(self): widget = self.create() self.checkCommandParam(widget, 'yscrollcommand') # non-standard but common options - def test_command(self): + def test_configure_command(self): widget = self.create() self.checkCommandParam(widget, 'command') - def test_indicatoron(self): + def test_configure_indicatoron(self): widget = self.create() self.checkBooleanParam(widget, 'indicatoron') - def test_offrelief(self): + def test_configure_offrelief(self): widget = self.create() self.checkReliefParam(widget, 'offrelief') - def test_overrelief(self): + def test_configure_overrelief(self): widget = self.create() self.checkReliefParam(widget, 'overrelief') - def test_selectcolor(self): + def test_configure_selectcolor(self): widget = self.create() self.checkColorParam(widget, 'selectcolor') - def test_selectimage(self): + def test_configure_selectimage(self): widget = self.create() self.checkImageParam(widget, 'selectimage') @requires_tcl(8, 5) - def test_tristateimage(self): + def test_configure_tristateimage(self): widget = self.create() self.checkImageParam(widget, 'tristateimage') @requires_tcl(8, 5) - def test_tristatevalue(self): + def test_configure_tristatevalue(self): widget = self.create() self.checkParam(widget, 'tristatevalue', 'unknowable') - def test_variable(self): + def test_configure_variable(self): widget = self.create() var = tkinter.DoubleVar(self.root) self.checkVariableParam(widget, 'variable', var) class IntegerSizeTests: - def test_height(self): + def test_configure_height(self): widget = self.create() self.checkIntegerParam(widget, 'height', 100, -100, 0) - def test_width(self): + def test_configure_width(self): widget = self.create() self.checkIntegerParam(widget, 'width', 402, -402, 0) class PixelSizeTests: - def test_height(self): + def test_configure_height(self): widget = self.create() self.checkPixelsParam(widget, 'height', 100, 101.2, 102.6, -100, 0, '3c') - def test_width(self): + def test_configure_width(self): widget = self.create() self.checkPixelsParam(widget, 'width', 402, 403.4, 404.6, -402, 0, '5i') def add_standard_options(*source_classes): - # This decorator adds test_xxx methods from source classes for every xxx - # option in the OPTIONS class attribute if they are not defined explicitly. + # This decorator adds test_configure_xxx methods from source classes for + # every xxx option in the OPTIONS class attribute if they are not defined + # explicitly. def decorator(cls): for option in cls.OPTIONS: - methodname = 'test_' + option + methodname = 'test_configure_' + option if not hasattr(cls, methodname): for source_class in source_classes: if hasattr(source_class, methodname): From 9d7c5ab6a4ae9d69b39bd16c3195bf6405ddc2d1 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 27 Dec 2020 00:56:40 -0800 Subject: [PATCH 1908/2163] bpo-42749: Fix testing bignum if Tkinter is compiled with Tk 8.4 and dynamic linked with Tk >= 8.5 (GH-23955) (GH-23962) (cherry picked from commit b02ad2458bc127a7afdeef414fa68c9a7f1f32af) Co-authored-by: Serhiy Storchaka --- Lib/test/test_tcl.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py index 6005349eaac452..6ac0db4dc4a9c2 100644 --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -135,10 +135,14 @@ def testUnsetVarException(self): def get_integers(self): integers = (0, 1, -1, 2**31-1, -2**31, 2**31, -2**31-1, 2**63-1, -2**63) - # bignum was added in Tcl 8.5, but its support is able only since 8.5.8 - if (get_tk_patchlevel() >= (8, 6, 0, 'final') or - (8, 5, 8) <= get_tk_patchlevel() < (8, 6)): - integers += (2**63, -2**63-1, 2**1000, -2**1000) + # bignum was added in Tcl 8.5, but its support is able only since 8.5.8. + # Actually it is determined at compile time, so using get_tk_patchlevel() + # is not reliable. + # TODO: expose full static version. + if tcl_version >= (8, 5): + v = get_tk_patchlevel() + if v >= (8, 6, 0, 'final') or (8, 5, 8) <= v < (8, 6): + integers += (2**63, -2**63-1, 2**1000, -2**1000) return integers def test_getint(self): From e11639880a73f30b4009efa8d14c350932e35332 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 28 Dec 2020 20:21:43 -0800 Subject: [PATCH 1909/2163] bpo-42770: Fix a typo in the email.headerregistry docs (GH-23982) Automerge-Triggered-By: GH:zware (cherry picked from commit c56988b88fecf6dc70f039704fda6051a0754db1) Co-authored-by: Zackery Spytz --- Doc/library/email.headerregistry.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/email.headerregistry.rst b/Doc/library/email.headerregistry.rst index 9376da2b8d39ce..3e1d97a03264b2 100644 --- a/Doc/library/email.headerregistry.rst +++ b/Doc/library/email.headerregistry.rst @@ -289,7 +289,7 @@ variant, :attr:`~.BaseHeader.max_count` is set to 1. A :class:`ParameterizedMIMEHeader` class that handles the :mailheader:`Content-Disposition` header. - .. attribute:: content-disposition + .. attribute:: content_disposition ``inline`` and ``attachment`` are the only valid values in common use. From 323cbb5531eb72527d0e3b02f28c1cdc5b7fce92 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 29 Dec 2020 03:16:43 -0800 Subject: [PATCH 1910/2163] bpo-42749: Use dynamic version to test for unsupported bignum in Tk (GH-23966) Tk can internally support bignum even if Tkinter is built without support of bignum. (cherry picked from commit 156b7f7052102ee1633a18e9a136ad8c38f66db0) Co-authored-by: Serhiy Storchaka --- Lib/test/test_tcl.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py index 6ac0db4dc4a9c2..c41235c1971a27 100644 --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -446,7 +446,7 @@ def test_expr_bignum(self): else: self.assertEqual(result, str(i)) self.assertIsInstance(result, str) - if tcl_version < (8, 5): # bignum was added in Tcl 8.5 + if get_tk_patchlevel() < (8, 5): # bignum was added in Tcl 8.5 self.assertRaises(TclError, tcl.call, 'expr', str(2**1000)) def test_passing_values(self): From 8f9313c83f74ff662985085cd5e7a8e4b3d8df3b Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 29 Dec 2020 04:28:47 -0800 Subject: [PATCH 1911/2163] [doc] Fix missing commas in signatures (GH-23693) * Fix star in signatures * Fix comma in signatures (cherry picked from commit 60eccd095624f39195cc5ae0b49a59022bbbb028) Co-authored-by: Andre Delfino --- Doc/library/datetime.rst | 4 ++-- Doc/library/email.contentmanager.rst | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index b733c4406af712..5ab3cc020c3f61 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -1215,7 +1215,7 @@ Instance methods: .. method:: datetime.replace(year=self.year, month=self.month, day=self.day, \ hour=self.hour, minute=self.minute, second=self.second, microsecond=self.microsecond, \ - tzinfo=self.tzinfo, * fold=0) + tzinfo=self.tzinfo, *, fold=0) Return a datetime with the same attributes, except for those attributes given new values by whichever keyword arguments are specified. Note that @@ -1779,7 +1779,7 @@ Other constructor: Instance methods: .. method:: time.replace(hour=self.hour, minute=self.minute, second=self.second, \ - microsecond=self.microsecond, tzinfo=self.tzinfo, * fold=0) + microsecond=self.microsecond, tzinfo=self.tzinfo, *, fold=0) Return a :class:`.time` with the same value, except for those attributes given new values by whichever keyword arguments are specified. Note that diff --git a/Doc/library/email.contentmanager.rst b/Doc/library/email.contentmanager.rst index e09c7c0e402bbc..918fc55677e723 100644 --- a/Doc/library/email.contentmanager.rst +++ b/Doc/library/email.contentmanager.rst @@ -116,7 +116,7 @@ Currently the email package provides only one concrete content manager, decoding the payload to unicode. The default error handler is ``replace``. - .. method:: set_content(msg, <'str'>, subtype="plain", charset='utf-8' \ + .. method:: set_content(msg, <'str'>, subtype="plain", charset='utf-8', \ cte=None, \ disposition=None, filename=None, cid=None, \ params=None, headers=None) From 70ced2dd27ee59abee9af6926e9ce7d93f06f885 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 29 Dec 2020 04:42:05 -0800 Subject: [PATCH 1912/2163] bpo-42700: Swap descriptions in pyexpat.errors (GH-23876) The descriptions of the `codes` and `messages` dictionaries in `xml.parsers.expat.errors` were swapped, and this commit swaps them back. For example, `codes` maps string descriptions of errors to numeric error codes, not the other way around. (cherry picked from commit 84402eb11086f97d31164aaa23e7238da3464f41) Co-authored-by: Michael Wayne Goodman --- Doc/library/pyexpat.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst index e43b9aecd86835..034e579315de00 100644 --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -665,14 +665,14 @@ The ``errors`` module has the following attributes: .. data:: codes - A dictionary mapping numeric error codes to their string descriptions. + A dictionary mapping string descriptions to their error codes. .. versionadded:: 3.2 .. data:: messages - A dictionary mapping string descriptions to their error codes. + A dictionary mapping numeric error codes to their string descriptions. .. versionadded:: 3.2 From 741f22df24ca61db38b5a7a2a58b5939b7154a01 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 29 Dec 2020 05:15:14 -0800 Subject: [PATCH 1913/2163] Allow / character in username,password fields in _PROXY envvars. (GH-23973) (#23992) (cherry picked from commit 030a713183084594659aefd77b76fe30178e23c8) Co-authored-by: Senthil Kumaran --- Lib/test/test_urllib2.py | 10 +++++++++- Lib/urllib/request.py | 6 +++++- .../Library/2020-12-27-18-47-01.bpo-23328._xqepZ.rst | 1 + 3 files changed, 15 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-12-27-18-47-01.bpo-23328._xqepZ.rst diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py index 0132059fdb0ad3..160315394094fa 100644 --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1846,9 +1846,17 @@ def test_parse_proxy(self): ('ftp', 'joe', 'password', 'proxy.example.com')), # Test for no trailing '/' case ('http://joe:password@proxy.example.com', - ('http', 'joe', 'password', 'proxy.example.com')) + ('http', 'joe', 'password', 'proxy.example.com')), + # Testcases with '/' character in username, password + ('http://user/name:password@localhost:22', + ('http', 'user/name', 'password', 'localhost:22')), + ('http://username:pass/word@localhost:22', + ('http', 'username', 'pass/word', 'localhost:22')), + ('http://user/name:pass/word@localhost:22', + ('http', 'user/name', 'pass/word', 'localhost:22')), ] + for tc, expected in parse_proxy_test_cases: self.assertEqual(_parse_proxy(tc), expected) diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py index afd634159fa82c..550a2732697d94 100644 --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -779,7 +779,11 @@ def _parse_proxy(proxy): raise ValueError("proxy URL with no authority: %r" % proxy) # We have an authority, so for RFC 3986-compliant URLs (by ss 3. # and 3.3.), path is empty or starts with '/' - end = r_scheme.find("/", 2) + if '@' in r_scheme: + host_separator = r_scheme.find('@') + end = r_scheme.find("/", host_separator) + else: + end = r_scheme.find("/", 2) if end == -1: end = None authority = r_scheme[2:end] diff --git a/Misc/NEWS.d/next/Library/2020-12-27-18-47-01.bpo-23328._xqepZ.rst b/Misc/NEWS.d/next/Library/2020-12-27-18-47-01.bpo-23328._xqepZ.rst new file mode 100644 index 00000000000000..07b15d34e8d56b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-12-27-18-47-01.bpo-23328._xqepZ.rst @@ -0,0 +1 @@ +Allow / character in username, password fields on _PROXY envars. From bc15cdbc6eb112cb72acf189769ecd539dd45652 Mon Sep 17 00:00:00 2001 From: Andre Delfino Date: Thu, 31 Dec 2020 10:10:46 -0300 Subject: [PATCH 1914/2163] [3.8] bpo-41224: Add versionadded for Symbol.is_annotated (GH-23861). (GH-24016) (cherry picked from commit 2edfc86f69d8a74f4821974678f664ff94a9dc22) --- Doc/library/symtable.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Doc/library/symtable.rst b/Doc/library/symtable.rst index 7c6ac4dccf8b76..56cb23db1d0c91 100644 --- a/Doc/library/symtable.rst +++ b/Doc/library/symtable.rst @@ -160,6 +160,12 @@ Examining Symbol Tables Return ``True`` if the symbol is local to its block. + .. method:: is_annotated() + + Return ``True`` if the symbol is annotated. + + .. versionadded:: 3.6 + .. method:: is_free() Return ``True`` if the symbol is referenced in its block, but not assigned From 82f24ff344056198ff283039c3e9fedac1d7fc2d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 31 Dec 2020 12:27:04 -0800 Subject: [PATCH 1915/2163] Fixes a typo in importlib.metadata. (GH-23921) (#24030) Signed-off-by: Tao He (cherry picked from commit 3631d6deab064de0bb286ef2943885dca3c3075e) Co-authored-by: Tao He Co-authored-by: Tao He --- Doc/library/importlib.metadata.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/importlib.metadata.rst b/Doc/library/importlib.metadata.rst index 15e58b860d97d0..c8b4d4173de03c 100644 --- a/Doc/library/importlib.metadata.rst +++ b/Doc/library/importlib.metadata.rst @@ -198,9 +198,9 @@ Thus, an alternative way to get the version number is through the There are all kinds of additional metadata available on the ``Distribution`` instance:: - >>> d.metadata['Requires-Python'] # doctest: +SKIP + >>> dist.metadata['Requires-Python'] # doctest: +SKIP '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*' - >>> d.metadata['License'] # doctest: +SKIP + >>> dist.metadata['License'] # doctest: +SKIP 'MIT' The full set of available metadata is not described here. See :pep:`566` From 381f3e4bfd4b1c440f7cb3025972fe0acd0406fc Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 1 Jan 2021 06:40:18 -0800 Subject: [PATCH 1916/2163] bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037) (cherry picked from commit ec3165320e81ac87edcb85c86c452528ddbaec1c) Co-authored-by: Dong-hee Na --- Lib/test/test_nntplib.py | 10 +++++++--- .../Tests/2021-01-01-08-52-36.bpo-42794.-7-XGz.rst | 2 ++ 2 files changed, 9 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Tests/2021-01-01-08-52-36.bpo-42794.-7-XGz.rst diff --git a/Lib/test/test_nntplib.py b/Lib/test/test_nntplib.py index fbd7db03defb1d..89a2004dfb1393 100644 --- a/Lib/test/test_nntplib.py +++ b/Lib/test/test_nntplib.py @@ -82,7 +82,7 @@ def _check_desc(desc): desc = self.server.description(self.GROUP_NAME) _check_desc(desc) # Another sanity check - self.assertIn("Python", desc) + self.assertIn(self.DESC, desc) # With a pattern desc = self.server.description(self.GROUP_PAT) _check_desc(desc) @@ -299,6 +299,7 @@ class NetworkedNNTPTests(NetworkedNNTPTestsMixin, unittest.TestCase): NNTP_HOST = 'news.trigofacile.com' GROUP_NAME = 'fr.comp.lang.python' GROUP_PAT = 'fr.comp.lang.*' + DESC = 'Python' NNTP_CLASS = NNTP @@ -332,8 +333,11 @@ class NetworkedNNTP_SSLTests(NetworkedNNTPTests): # 400 connections per day are accepted from each IP address." NNTP_HOST = 'nntp.aioe.org' - GROUP_NAME = 'comp.lang.python' - GROUP_PAT = 'comp.lang.*' + # bpo-42794: aioe.test is one of the official groups on this server + # used for testing: https://news.aioe.org/manual/aioe-hierarchy/ + GROUP_NAME = 'aioe.test' + GROUP_PAT = 'aioe.*' + DESC = 'test' NNTP_CLASS = getattr(nntplib, 'NNTP_SSL', None) diff --git a/Misc/NEWS.d/next/Tests/2021-01-01-08-52-36.bpo-42794.-7-XGz.rst b/Misc/NEWS.d/next/Tests/2021-01-01-08-52-36.bpo-42794.-7-XGz.rst new file mode 100644 index 00000000000000..577f2259e1f00c --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-01-01-08-52-36.bpo-42794.-7-XGz.rst @@ -0,0 +1,2 @@ +Update test_nntplib to use offical group name of news.aioe.org for testing. +Patch by Dong-hee Na. From 02639c3bc6a9bae9de635356c289cbae674aa61c Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 1 Jan 2021 09:28:23 -0800 Subject: [PATCH 1917/2163] Bring Python into the new year. (GH-24036) (cherry picked from commit de6f20a6de48d63066b2cf5b317f50629f01d74a) Co-authored-by: Dong-hee Na --- Doc/copyright.rst | 2 +- Doc/license.rst | 2 +- LICENSE | 2 +- Mac/IDLE/IDLE.app/Contents/Info.plist | 2 +- Mac/PythonLauncher/Info.plist.in | 2 +- Mac/Resources/app/Info.plist.in | 2 +- PC/python_ver_rc.h | 2 +- Python/getcopyright.c | 2 +- README.rst | 4 ++-- 9 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Doc/copyright.rst b/Doc/copyright.rst index 1b90d9f172c992..4191c0bb63a2c1 100644 --- a/Doc/copyright.rst +++ b/Doc/copyright.rst @@ -4,7 +4,7 @@ Copyright Python and this documentation is: -Copyright © 2001-2020 Python Software Foundation. All rights reserved. +Copyright © 2001-2021 Python Software Foundation. All rights reserved. Copyright © 2000 BeOpen.com. All rights reserved. diff --git a/Doc/license.rst b/Doc/license.rst index f70c34d3fde605..ceb946e8bb11f3 100644 --- a/Doc/license.rst +++ b/Doc/license.rst @@ -100,7 +100,7 @@ PSF LICENSE AGREEMENT FOR PYTHON |release| analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python |release| alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of - copyright, i.e., "Copyright © 2001-2020 Python Software Foundation; All Rights + copyright, i.e., "Copyright © 2001-2021 Python Software Foundation; All Rights Reserved" are retained in Python |release| alone or in any derivative version prepared by Licensee. diff --git a/LICENSE b/LICENSE index f42f8adbed845d..473861da1be7c5 100644 --- a/LICENSE +++ b/LICENSE @@ -84,7 +84,7 @@ analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, -2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Python Software Foundation; +2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee. diff --git a/Mac/IDLE/IDLE.app/Contents/Info.plist b/Mac/IDLE/IDLE.app/Contents/Info.plist index dcc48abdd2a395..f6b5cfe8d5451f 100644 --- a/Mac/IDLE/IDLE.app/Contents/Info.plist +++ b/Mac/IDLE/IDLE.app/Contents/Info.plist @@ -36,7 +36,7 @@ CFBundleExecutable IDLE CFBundleGetInfoString - %version%, © 2001-2020 Python Software Foundation + %version%, © 2001-2021 Python Software Foundation CFBundleIconFile IDLE.icns CFBundleIdentifier diff --git a/Mac/PythonLauncher/Info.plist.in b/Mac/PythonLauncher/Info.plist.in index 21a051535fb925..3d8bc3e4154ee2 100644 --- a/Mac/PythonLauncher/Info.plist.in +++ b/Mac/PythonLauncher/Info.plist.in @@ -40,7 +40,7 @@ CFBundleExecutable Python Launcher CFBundleGetInfoString - %VERSION%, © 2001-2020 Python Software Foundation + %VERSION%, © 2001-2021 Python Software Foundation CFBundleIconFile PythonLauncher.icns CFBundleIdentifier diff --git a/Mac/Resources/app/Info.plist.in b/Mac/Resources/app/Info.plist.in index 1d624984a85203..2c801332332b30 100644 --- a/Mac/Resources/app/Info.plist.in +++ b/Mac/Resources/app/Info.plist.in @@ -37,7 +37,7 @@ CFBundleInfoDictionaryVersion 6.0 CFBundleLongVersionString - %version%, (c) 2001-2020 Python Software Foundation. + %version%, (c) 2001-2021 Python Software Foundation. CFBundleName Python CFBundlePackageType diff --git a/PC/python_ver_rc.h b/PC/python_ver_rc.h index d725a9ba06ebd9..81b89fe9d79039 100644 --- a/PC/python_ver_rc.h +++ b/PC/python_ver_rc.h @@ -5,7 +5,7 @@ #include "winver.h" #define PYTHON_COMPANY "Python Software Foundation" -#define PYTHON_COPYRIGHT "Copyright \xA9 2001-2016 Python Software Foundation. Copyright \xA9 2000 BeOpen.com. Copyright \xA9 1995-2001 CNRI. Copyright \xA9 1991-1995 SMC." +#define PYTHON_COPYRIGHT "Copyright \xA9 2001-2021 Python Software Foundation. Copyright \xA9 2000 BeOpen.com. Copyright \xA9 1995-2001 CNRI. Copyright \xA9 1991-1995 SMC." #define MS_WINDOWS #include "modsupport.h" diff --git a/Python/getcopyright.c b/Python/getcopyright.c index 299ccc08c44f83..7fdeb314d5261b 100644 --- a/Python/getcopyright.c +++ b/Python/getcopyright.c @@ -4,7 +4,7 @@ static const char cprt[] = "\ -Copyright (c) 2001-2020 Python Software Foundation.\n\ +Copyright (c) 2001-2021 Python Software Foundation.\n\ All Rights Reserved.\n\ \n\ Copyright (c) 2000 BeOpen.com.\n\ diff --git a/README.rst b/README.rst index f6fd4824f0a9b0..4907629a0f69d6 100644 --- a/README.rst +++ b/README.rst @@ -18,7 +18,7 @@ This is Python version 3.8.7 :target: https://python.zulipchat.com -Copyright (c) 2001-2020 Python Software Foundation. All rights reserved. +Copyright (c) 2001-2021 Python Software Foundation. All rights reserved. See the end of this file for further copyright and license information. @@ -246,7 +246,7 @@ See :pep:`569` for Python 3.8 release details. Copyright and License Information --------------------------------- -Copyright (c) 2001-2020 Python Software Foundation. All rights reserved. +Copyright (c) 2001-2021 Python Software Foundation. All rights reserved. Copyright (c) 2000 BeOpen.com. All rights reserved. From e3a9adba329681b1b73b7223515e71e94fc35e12 Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Fri, 1 Jan 2021 19:39:36 +0200 Subject: [PATCH 1918/2163] [3.8] bpo-42759: Fix equality comparison of Variable and Font in Tkinter (GH-23968) (GH-24026) Objects which belong to different Tcl interpreters are now always different, even if they have the same name. (cherry picked from commit 1df56bc0597a051c13d53514e120e9b6764185f8) --- Lib/tkinter/__init__.py | 12 +++++----- Lib/tkinter/font.py | 4 +++- Lib/tkinter/test/test_tkinter/test_font.py | 9 +++++++- .../test/test_tkinter/test_variables.py | 22 ++++++++++++++++--- .../2020-12-27-22-19-26.bpo-42759.lGi_03.rst | 3 +++ 5 files changed, 38 insertions(+), 12 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-12-27-22-19-26.bpo-42759.lGi_03.rst diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py index f9ece257841a96..91a6d56480e630 100644 --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -491,13 +491,11 @@ def trace_vinfo(self): self._tk.call("trace", "vinfo", self._name))] def __eq__(self, other): - """Comparison for equality (==). - - Note: if the Variable's master matters to behavior - also compare self._master == other._master - """ - return self.__class__.__name__ == other.__class__.__name__ \ - and self._name == other._name + if not isinstance(other, Variable): + return NotImplemented + return (self._name == other._name + and self.__class__.__name__ == other.__class__.__name__ + and self._tk == other._tk) class StringVar(Variable): diff --git a/Lib/tkinter/font.py b/Lib/tkinter/font.py index 31d9afd8f866ca..225ec6d34def96 100644 --- a/Lib/tkinter/font.py +++ b/Lib/tkinter/font.py @@ -100,7 +100,9 @@ def __str__(self): return self.name def __eq__(self, other): - return isinstance(other, Font) and self.name == other.name + if not isinstance(other, Font): + return NotImplemented + return self.name == other.name and self._tk == other._tk def __getitem__(self, key): return self.cget(key) diff --git a/Lib/tkinter/test/test_tkinter/test_font.py b/Lib/tkinter/test/test_tkinter/test_font.py index 2ea59f17483423..5c6f048512850a 100644 --- a/Lib/tkinter/test/test_tkinter/test_font.py +++ b/Lib/tkinter/test/test_tkinter/test_font.py @@ -63,14 +63,21 @@ def test_name(self): self.assertEqual(self.font.name, fontname) self.assertEqual(str(self.font), fontname) - def test_eq(self): + def test_equality(self): font1 = font.Font(root=self.root, name=fontname, exists=True) font2 = font.Font(root=self.root, name=fontname, exists=True) self.assertIsNot(font1, font2) self.assertEqual(font1, font2) self.assertNotEqual(font1, font1.copy()) + self.assertNotEqual(font1, 0) + root2 = tkinter.Tk() + self.addCleanup(root2.destroy) + font3 = font.Font(root=root2, name=fontname, exists=True) + self.assertEqual(str(font1), str(font3)) + self.assertNotEqual(font1, font3) + def test_measure(self): self.assertIsInstance(self.font.measure('abc'), int) diff --git a/Lib/tkinter/test/test_tkinter/test_variables.py b/Lib/tkinter/test/test_tkinter/test_variables.py index e7b24a818f1501..cccec62941353f 100644 --- a/Lib/tkinter/test/test_tkinter/test_variables.py +++ b/Lib/tkinter/test/test_tkinter/test_variables.py @@ -1,6 +1,7 @@ import unittest import gc import tkinter +from test.support import ALWAYS_EQ from tkinter import (Variable, StringVar, IntVar, DoubleVar, BooleanVar, Tcl, TclError) from tkinter.test.support import AbstractDefaultRootTest @@ -57,15 +58,30 @@ def test_dont_unset_not_existing(self): del v2 self.assertFalse(self.info_exists("name")) - def test___eq__(self): + def test_equality(self): # values doesn't matter, only class and name are checked v1 = Variable(self.root, name="abc") v2 = Variable(self.root, name="abc") self.assertEqual(v1, v2) - v3 = Variable(self.root, name="abc") + v3 = Variable(self.root, name="cba") + self.assertNotEqual(v1, v3) + v4 = StringVar(self.root, name="abc") - self.assertNotEqual(v3, v4) + self.assertEqual(str(v1), str(v4)) + self.assertNotEqual(v1, v4) + + V = type('Variable', (), {}) + self.assertNotEqual(v1, V()) + + self.assertNotEqual(v1, object()) + self.assertEqual(v1, ALWAYS_EQ) + + root2 = tkinter.Tk() + self.addCleanup(root2.destroy) + v5 = Variable(root2, name="abc") + self.assertEqual(str(v1), str(v5)) + self.assertNotEqual(v1, v5) def test_invalid_name(self): with self.assertRaises(TypeError): diff --git a/Misc/NEWS.d/next/Library/2020-12-27-22-19-26.bpo-42759.lGi_03.rst b/Misc/NEWS.d/next/Library/2020-12-27-22-19-26.bpo-42759.lGi_03.rst new file mode 100644 index 00000000000000..a5ec7d5820336d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-12-27-22-19-26.bpo-42759.lGi_03.rst @@ -0,0 +1,3 @@ +Fixed equality comparison of :class:`tkinter.Variable` and +:class:`tkinter.font.Font`. Objects which belong to different Tcl +interpreters are now always different, even if they have the same name. From 1a544e1dcf8e0880ee35ce78a7f90fc4e5c98c4f Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Fri, 1 Jan 2021 19:40:11 +0200 Subject: [PATCH 1919/2163] [3.8] bpo-42425: Fix possible leak in initialization of errmap for OSError (GH-23446). (GH-24025) (cherry picked from commit ed1007c0d74e658d1e6c9b51b12ce7501eb8cbf9) --- Objects/exceptions.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Objects/exceptions.c b/Objects/exceptions.c index 8bcf76ff860a22..d22ed6f039ac20 100644 --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -2518,8 +2518,10 @@ _PyExc_Init(void) do { \ PyObject *_code = PyLong_FromLong(CODE); \ assert(_PyObject_RealIsSubclass(PyExc_ ## TYPE, PyExc_OSError)); \ - if (!_code || PyDict_SetItem(errnomap, _code, PyExc_ ## TYPE)) \ + if (!_code || PyDict_SetItem(errnomap, _code, PyExc_ ## TYPE)) { \ + Py_XDECREF(_code); \ return _PyStatus_ERR("errmap insertion problem."); \ + } \ Py_DECREF(_code); \ } while (0) From 187785e2fa2050156a6eda93fb2da31db13f07a6 Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Fri, 1 Jan 2021 19:40:38 +0200 Subject: [PATCH 1920/2163] [3.8] bpo-36589: Fix the error handling in curses.update_lines_cols(). (GH-12766) (GH-24023) Return None instead of 1. (cherry picked from commit 2bc343417a4de83fa6998ff91303877734ecd366) Co-authored-by: Zackery Spytz --- .../2019-11-16-22-56-51.bpo-36589.0Io76D.rst | 2 ++ Modules/_cursesmodule.c | 29 ++++++++++++++----- Modules/clinic/_cursesmodule.c.h | 16 ++-------- 3 files changed, 26 insertions(+), 21 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-11-16-22-56-51.bpo-36589.0Io76D.rst diff --git a/Misc/NEWS.d/next/Library/2019-11-16-22-56-51.bpo-36589.0Io76D.rst b/Misc/NEWS.d/next/Library/2019-11-16-22-56-51.bpo-36589.0Io76D.rst new file mode 100644 index 00000000000000..3c1221b2034947 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-11-16-22-56-51.bpo-36589.0Io76D.rst @@ -0,0 +1,2 @@ +The :func:`curses.update_lines_cols` function now returns ``None`` instead +of ``1`` on success. diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c index b2b1117fb0934e..ac23d5d7474a0e 100644 --- a/Modules/_cursesmodule.c +++ b/Modules/_cursesmodule.c @@ -3757,15 +3757,18 @@ update_lines_cols(void) } /*[clinic input] -_curses.update_lines_cols -> int +_curses.update_lines_cols [clinic start generated code]*/ -static int +static PyObject * _curses_update_lines_cols_impl(PyObject *module) -/*[clinic end generated code: output=0345e7f072ea711a input=3a87760f7d5197f0]*/ +/*[clinic end generated code: output=423f2b1e63ed0f75 input=5f065ab7a28a5d90]*/ { - return update_lines_cols(); + if (!update_lines_cols()) { + return NULL; + } + Py_RETURN_NONE; } #endif @@ -3849,8 +3852,10 @@ _curses_resizeterm_impl(PyObject *module, int nlines, int ncols) result = PyCursesCheckERR(resizeterm(nlines, ncols), "resizeterm"); if (!result) return NULL; - if (!update_lines_cols()) + if (!update_lines_cols()) { + Py_DECREF(result); return NULL; + } return result; } @@ -3886,8 +3891,10 @@ _curses_resize_term_impl(PyObject *module, int nlines, int ncols) result = PyCursesCheckERR(resize_term(nlines, ncols), "resize_term"); if (!result) return NULL; - if (!update_lines_cols()) + if (!update_lines_cols()) { + Py_DECREF(result); return NULL; + } return result; } #endif /* HAVE_CURSES_RESIZE_TERM */ @@ -3958,12 +3965,18 @@ _curses_start_color_impl(PyObject *module) c = PyLong_FromLong((long) COLORS); if (c == NULL) return NULL; - PyDict_SetItemString(ModDict, "COLORS", c); + if (PyDict_SetItemString(ModDict, "COLORS", c) < 0) { + Py_DECREF(c); + return NULL; + } Py_DECREF(c); cp = PyLong_FromLong((long) COLOR_PAIRS); if (cp == NULL) return NULL; - PyDict_SetItemString(ModDict, "COLOR_PAIRS", cp); + if (PyDict_SetItemString(ModDict, "COLOR_PAIRS", cp) < 0) { + Py_DECREF(cp); + return NULL; + } Py_DECREF(cp); Py_RETURN_NONE; } else { diff --git a/Modules/clinic/_cursesmodule.c.h b/Modules/clinic/_cursesmodule.c.h index ad93e6a0ca0220..437604cb92ae67 100644 --- a/Modules/clinic/_cursesmodule.c.h +++ b/Modules/clinic/_cursesmodule.c.h @@ -3799,23 +3799,13 @@ PyDoc_STRVAR(_curses_update_lines_cols__doc__, #define _CURSES_UPDATE_LINES_COLS_METHODDEF \ {"update_lines_cols", (PyCFunction)_curses_update_lines_cols, METH_NOARGS, _curses_update_lines_cols__doc__}, -static int +static PyObject * _curses_update_lines_cols_impl(PyObject *module); static PyObject * _curses_update_lines_cols(PyObject *module, PyObject *Py_UNUSED(ignored)) { - PyObject *return_value = NULL; - int _return_value; - - _return_value = _curses_update_lines_cols_impl(module); - if ((_return_value == -1) && PyErr_Occurred()) { - goto exit; - } - return_value = PyLong_FromLong((long)_return_value); - -exit: - return return_value; + return _curses_update_lines_cols_impl(module); } #endif /* (defined(HAVE_CURSES_RESIZETERM) || defined(HAVE_CURSES_RESIZE_TERM)) */ @@ -4569,4 +4559,4 @@ _curses_use_default_colors(PyObject *module, PyObject *Py_UNUSED(ignored)) #ifndef _CURSES_USE_DEFAULT_COLORS_METHODDEF #define _CURSES_USE_DEFAULT_COLORS_METHODDEF #endif /* !defined(_CURSES_USE_DEFAULT_COLORS_METHODDEF) */ -/*[clinic end generated code: output=e5b3502f1d38dff0 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=820af7050893ed16 input=a9049054013a1b77]*/ From b863607d303a87e8680149361ac987328b35ca5f Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Fri, 1 Jan 2021 19:41:49 +0200 Subject: [PATCH 1921/2163] [3.8] bpo-39068: Fix race condition in base64 (GH-17627) (GH-24022) There was a race condition in base64 in lazy initialization of multiple globals. (cherry picked from commit 9655434cca5dfbea97bf6d355aec028e840b289c) Co-authored-by: Brandon Stansbury --- Lib/base64.py | 4 ++-- Misc/ACKS | 1 + .../next/Library/2019-12-16-17-55-31.bpo-39068.Ti3f9P.rst | 2 ++ 3 files changed, 5 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-12-16-17-55-31.bpo-39068.Ti3f9P.rst diff --git a/Lib/base64.py b/Lib/base64.py index 2e70223dfe7824..54297668585d84 100755 --- a/Lib/base64.py +++ b/Lib/base64.py @@ -320,7 +320,7 @@ def a85encode(b, *, foldspaces=False, wrapcol=0, pad=False, adobe=False): global _a85chars, _a85chars2 # Delay the initialization of tables to not waste memory # if the function is never called - if _a85chars is None: + if _a85chars2 is None: _a85chars = [bytes((i,)) for i in range(33, 118)] _a85chars2 = [(a + b) for a in _a85chars for b in _a85chars] @@ -428,7 +428,7 @@ def b85encode(b, pad=False): global _b85chars, _b85chars2 # Delay the initialization of tables to not waste memory # if the function is never called - if _b85chars is None: + if _b85chars2 is None: _b85chars = [bytes((i,)) for i in _b85alphabet] _b85chars2 = [(a + b) for a in _b85chars for b in _b85chars] return _85encode(b, _b85chars, _b85chars2, pad) diff --git a/Misc/ACKS b/Misc/ACKS index 6ae882479d4363..8ca1f64c9f5f6a 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1607,6 +1607,7 @@ Tage Stabell-Kulo Quentin Stafford-Fraser Frank Stajano Joel Stanley +Brandon Stansbury Anthony Starks David Steele Oliver Steele diff --git a/Misc/NEWS.d/next/Library/2019-12-16-17-55-31.bpo-39068.Ti3f9P.rst b/Misc/NEWS.d/next/Library/2019-12-16-17-55-31.bpo-39068.Ti3f9P.rst new file mode 100644 index 00000000000000..fe6503fdce6b63 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-12-16-17-55-31.bpo-39068.Ti3f9P.rst @@ -0,0 +1,2 @@ +Fix initialization race condition in :func:`a85encode` and :func:`b85encode` +in :mod:`base64`. Patch by Brandon Stansbury. From 6dffa67b98f78ae41b596f84478f3379f55d4d03 Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Fri, 1 Jan 2021 19:42:23 +0200 Subject: [PATCH 1922/2163] [3.8] bpo-26407: Do not mask errors in csv. (GH-20536) (GH-24021) Unexpected errors in calling the __iter__ method are no longer masked by TypeError in csv.reader(), csv.writer.writerow() and csv.writer.writerows(). (cherry picked from commit c88239f864a27f673c0f0a9e62d2488563f9d081) --- Lib/test/test_csv.py | 20 ++++++++++++++++++- .../2020-05-30-14-19-47.bpo-26407.MjWLO1.rst | 3 +++ Modules/_csv.c | 16 +++++++-------- 3 files changed, 30 insertions(+), 9 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-05-30-14-19-47.bpo-26407.MjWLO1.rst diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py index a16d14019f341f..d421be075ca27f 100644 --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -14,6 +14,12 @@ from textwrap import dedent from collections import OrderedDict + +class BadIterable: + def __iter__(self): + raise OSError + + class Test_Csv(unittest.TestCase): """ Test the underlying C csv parser in ways that are not appropriate @@ -40,9 +46,15 @@ def _test_arg_valid(self, ctor, arg): def test_reader_arg_valid(self): self._test_arg_valid(csv.reader, []) + self.assertRaises(OSError, csv.reader, BadIterable()) def test_writer_arg_valid(self): self._test_arg_valid(csv.writer, StringIO()) + class BadWriter: + @property + def write(self): + raise OSError + self.assertRaises(OSError, csv.writer, BadWriter()) def _test_default_attrs(self, ctor, *args): obj = ctor(*args) @@ -141,6 +153,7 @@ def test_write_arg_valid(self): self._write_test([None], '""') self._write_error_test(csv.Error, [None], quoting = csv.QUOTE_NONE) # Check that exceptions are passed up the chain + self._write_error_test(OSError, BadIterable()) class BadList: def __len__(self): return 10; @@ -230,6 +243,12 @@ def test_writerows_with_none(self): fileobj.seek(0) self.assertEqual(fileobj.read(), 'a\r\n""\r\n') + def test_writerows_errors(self): + with TemporaryFile("w+", newline='') as fileobj: + writer = csv.writer(fileobj) + self.assertRaises(TypeError, writer.writerows, None) + self.assertRaises(OSError, writer.writerows, BadIterable()) + @support.cpython_only def test_writerows_legacy_strings(self): import _testcapi @@ -334,7 +353,6 @@ def test_read_linenum(self): def test_roundtrip_quoteed_newlines(self): with TemporaryFile("w+", newline='') as fileobj: writer = csv.writer(fileobj) - self.assertRaises(TypeError, writer.writerows, None) rows = [['a\nb','b'],['c','x\r\nd']] writer.writerows(rows) fileobj.seek(0) diff --git a/Misc/NEWS.d/next/Library/2020-05-30-14-19-47.bpo-26407.MjWLO1.rst b/Misc/NEWS.d/next/Library/2020-05-30-14-19-47.bpo-26407.MjWLO1.rst new file mode 100644 index 00000000000000..d0e45cf1b1f2f4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-05-30-14-19-47.bpo-26407.MjWLO1.rst @@ -0,0 +1,3 @@ +Unexpected errors in calling the ``__iter__`` method are no longer masked +by ``TypeError`` in :func:`csv.reader`, :func:`csv.writer.writerow` and +:meth:`csv.writer.writerows`. diff --git a/Modules/_csv.c b/Modules/_csv.c index 46d414383cbd20..069ec9602cc9ce 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -958,8 +958,6 @@ csv_reader(PyObject *module, PyObject *args, PyObject *keyword_args) } self->input_iter = PyObject_GetIter(iterator); if (self->input_iter == NULL) { - PyErr_SetString(PyExc_TypeError, - "argument 1 must be an iterator"); Py_DECREF(self); return NULL; } @@ -1165,10 +1163,14 @@ csv_writerow(WriterObj *self, PyObject *seq) PyObject *iter, *field, *line, *result; iter = PyObject_GetIter(seq); - if (iter == NULL) - return PyErr_Format(_csvstate_global->error_obj, - "iterable expected, not %.200s", - seq->ob_type->tp_name); + if (iter == NULL) { + if (PyErr_ExceptionMatches(PyExc_TypeError)) { + PyErr_Format(_csvstate_global->error_obj, + "iterable expected, not %.200s", + Py_TYPE(seq)->tp_name); + } + return NULL; + } /* Join all fields in internal buffer. */ @@ -1258,8 +1260,6 @@ csv_writerows(WriterObj *self, PyObject *seqseq) row_iter = PyObject_GetIter(seqseq); if (row_iter == NULL) { - PyErr_SetString(PyExc_TypeError, - "writerows() argument must be iterable"); return NULL; } while ((row_obj = PyIter_Next(row_iter))) { From 2e8b1c9e9b2d2e011bf35f77cd611843bac7f3dd Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 2 Jan 2021 09:50:28 -0800 Subject: [PATCH 1923/2163] bpo-42809: Improve pickle tests for recursive data. (GH-24060) (cherry picked from commit a25011be8c6f62cb3333903befe6295d57f0bd30) Co-authored-by: Serhiy Storchaka --- Lib/test/pickletester.py | 340 +++++++++++++++++++++++++++++---------- 1 file changed, 253 insertions(+), 87 deletions(-) diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index ff7bbb0c8a9bff..dd41f6ecacd27c 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -65,6 +65,10 @@ def count_opcode(code, pickle): return n +def identity(x): + return x + + class UnseekableIO(io.BytesIO): def peek(self, *args): raise NotImplementedError @@ -134,11 +138,12 @@ class E(C): def __getinitargs__(self): return () -class H(object): +# Simple mutable object. +class Object: pass -# Hashable mutable key -class K(object): +# Hashable immutable key object containing unheshable mutable data. +class K: def __init__(self, value): self.value = value @@ -153,10 +158,6 @@ def __reduce__(self): D.__module__ = "__main__" __main__.E = E E.__module__ = "__main__" -__main__.H = H -H.__module__ = "__main__" -__main__.K = K -K.__module__ = "__main__" class myint(int): def __init__(self, x): @@ -1490,54 +1491,182 @@ def dont_test_disassembly(self): got = filelike.getvalue() self.assertEqual(expected, got) - def test_recursive_list(self): - l = [] + def _test_recursive_list(self, cls, aslist=identity, minprotocol=0): + # List containing itself. + l = cls() l.append(l) - for proto in protocols: + for proto in range(minprotocol, pickle.HIGHEST_PROTOCOL + 1): s = self.dumps(l, proto) x = self.loads(s) - self.assertIsInstance(x, list) - self.assertEqual(len(x), 1) - self.assertIs(x[0], x) + self.assertIsInstance(x, cls) + y = aslist(x) + self.assertEqual(len(y), 1) + self.assertIs(y[0], x) - def test_recursive_tuple_and_list(self): - t = ([],) + def test_recursive_list(self): + self._test_recursive_list(list) + + def test_recursive_list_subclass(self): + self._test_recursive_list(MyList, minprotocol=2) + + def test_recursive_list_like(self): + self._test_recursive_list(REX_six, aslist=lambda x: x.items) + + def _test_recursive_tuple_and_list(self, cls, aslist=identity, minprotocol=0): + # Tuple containing a list containing the original tuple. + t = (cls(),) t[0].append(t) - for proto in protocols: + for proto in range(minprotocol, pickle.HIGHEST_PROTOCOL + 1): s = self.dumps(t, proto) x = self.loads(s) self.assertIsInstance(x, tuple) self.assertEqual(len(x), 1) - self.assertIsInstance(x[0], list) - self.assertEqual(len(x[0]), 1) - self.assertIs(x[0][0], x) + self.assertIsInstance(x[0], cls) + y = aslist(x[0]) + self.assertEqual(len(y), 1) + self.assertIs(y[0], x) + + # List containing a tuple containing the original list. + t, = t + for proto in range(minprotocol, pickle.HIGHEST_PROTOCOL + 1): + s = self.dumps(t, proto) + x = self.loads(s) + self.assertIsInstance(x, cls) + y = aslist(x) + self.assertEqual(len(y), 1) + self.assertIsInstance(y[0], tuple) + self.assertEqual(len(y[0]), 1) + self.assertIs(y[0][0], x) - def test_recursive_dict(self): - d = {} + def test_recursive_tuple_and_list(self): + self._test_recursive_tuple_and_list(list) + + def test_recursive_tuple_and_list_subclass(self): + self._test_recursive_tuple_and_list(MyList, minprotocol=2) + + def test_recursive_tuple_and_list_like(self): + self._test_recursive_tuple_and_list(REX_six, aslist=lambda x: x.items) + + def _test_recursive_dict(self, cls, asdict=identity, minprotocol=0): + # Dict containing itself. + d = cls() d[1] = d - for proto in protocols: + for proto in range(minprotocol, pickle.HIGHEST_PROTOCOL + 1): s = self.dumps(d, proto) x = self.loads(s) - self.assertIsInstance(x, dict) - self.assertEqual(list(x.keys()), [1]) - self.assertIs(x[1], x) + self.assertIsInstance(x, cls) + y = asdict(x) + self.assertEqual(list(y.keys()), [1]) + self.assertIs(y[1], x) - def test_recursive_dict_key(self): - d = {} - k = K(d) - d[k] = 1 - for proto in protocols: + def test_recursive_dict(self): + self._test_recursive_dict(dict) + + def test_recursive_dict_subclass(self): + self._test_recursive_dict(MyDict, minprotocol=2) + + def test_recursive_dict_like(self): + self._test_recursive_dict(REX_seven, asdict=lambda x: x.table) + + def _test_recursive_tuple_and_dict(self, cls, asdict=identity, minprotocol=0): + # Tuple containing a dict containing the original tuple. + t = (cls(),) + t[0][1] = t + for proto in range(minprotocol, pickle.HIGHEST_PROTOCOL + 1): + s = self.dumps(t, proto) + x = self.loads(s) + self.assertIsInstance(x, tuple) + self.assertEqual(len(x), 1) + self.assertIsInstance(x[0], cls) + y = asdict(x[0]) + self.assertEqual(list(y), [1]) + self.assertIs(y[1], x) + + # Dict containing a tuple containing the original dict. + t, = t + for proto in range(minprotocol, pickle.HIGHEST_PROTOCOL + 1): + s = self.dumps(t, proto) + x = self.loads(s) + self.assertIsInstance(x, cls) + y = asdict(x) + self.assertEqual(list(y), [1]) + self.assertIsInstance(y[1], tuple) + self.assertEqual(len(y[1]), 1) + self.assertIs(y[1][0], x) + + def test_recursive_tuple_and_dict(self): + self._test_recursive_tuple_and_dict(dict) + + def test_recursive_tuple_and_dict_subclass(self): + self._test_recursive_tuple_and_dict(MyDict, minprotocol=2) + + def test_recursive_tuple_and_dict_like(self): + self._test_recursive_tuple_and_dict(REX_seven, asdict=lambda x: x.table) + + def _test_recursive_dict_key(self, cls, asdict=identity, minprotocol=0): + # Dict containing an immutable object (as key) containing the original + # dict. + d = cls() + d[K(d)] = 1 + for proto in range(minprotocol, pickle.HIGHEST_PROTOCOL + 1): s = self.dumps(d, proto) x = self.loads(s) - self.assertIsInstance(x, dict) - self.assertEqual(len(x.keys()), 1) - self.assertIsInstance(list(x.keys())[0], K) - self.assertIs(list(x.keys())[0].value, x) + self.assertIsInstance(x, cls) + y = asdict(x) + self.assertEqual(len(y.keys()), 1) + self.assertIsInstance(list(y.keys())[0], K) + self.assertIs(list(y.keys())[0].value, x) + + def test_recursive_dict_key(self): + self._test_recursive_dict_key(dict) + + def test_recursive_dict_subclass_key(self): + self._test_recursive_dict_key(MyDict, minprotocol=2) + + def test_recursive_dict_like_key(self): + self._test_recursive_dict_key(REX_seven, asdict=lambda x: x.table) + + def _test_recursive_tuple_and_dict_key(self, cls, asdict=identity, minprotocol=0): + # Tuple containing a dict containing an immutable object (as key) + # containing the original tuple. + t = (cls(),) + t[0][K(t)] = 1 + for proto in range(minprotocol, pickle.HIGHEST_PROTOCOL + 1): + s = self.dumps(t, proto) + x = self.loads(s) + self.assertIsInstance(x, tuple) + self.assertEqual(len(x), 1) + self.assertIsInstance(x[0], cls) + y = asdict(x[0]) + self.assertEqual(len(y), 1) + self.assertIsInstance(list(y.keys())[0], K) + self.assertIs(list(y.keys())[0].value, x) + + # Dict containing an immutable object (as key) containing a tuple + # containing the original dict. + t, = t + for proto in range(minprotocol, pickle.HIGHEST_PROTOCOL + 1): + s = self.dumps(t, proto) + x = self.loads(s) + self.assertIsInstance(x, cls) + y = asdict(x) + self.assertEqual(len(y), 1) + self.assertIsInstance(list(y.keys())[0], K) + self.assertIs(list(y.keys())[0].value[0], x) + + def test_recursive_tuple_and_dict_key(self): + self._test_recursive_tuple_and_dict_key(dict) + + def test_recursive_tuple_and_dict_subclass_key(self): + self._test_recursive_tuple_and_dict_key(MyDict, minprotocol=2) + + def test_recursive_tuple_and_dict_like_key(self): + self._test_recursive_tuple_and_dict_key(REX_seven, asdict=lambda x: x.table) def test_recursive_set(self): + # Set containing an immutable object containing the original set. y = set() - k = K(y) - y.add(k) + y.add(K(y)) for proto in range(4, pickle.HIGHEST_PROTOCOL + 1): s = self.dumps(y, proto) x = self.loads(s) @@ -1546,52 +1675,31 @@ def test_recursive_set(self): self.assertIsInstance(list(x)[0], K) self.assertIs(list(x)[0].value, x) - def test_recursive_list_subclass(self): - y = MyList() - y.append(y) - for proto in range(2, pickle.HIGHEST_PROTOCOL + 1): + # Immutable object containing a set containing the original object. + y, = y + for proto in range(4, pickle.HIGHEST_PROTOCOL + 1): s = self.dumps(y, proto) x = self.loads(s) - self.assertIsInstance(x, MyList) - self.assertEqual(len(x), 1) - self.assertIs(x[0], x) - - def test_recursive_dict_subclass(self): - d = MyDict() - d[1] = d - for proto in range(2, pickle.HIGHEST_PROTOCOL + 1): - s = self.dumps(d, proto) - x = self.loads(s) - self.assertIsInstance(x, MyDict) - self.assertEqual(list(x.keys()), [1]) - self.assertIs(x[1], x) - - def test_recursive_dict_subclass_key(self): - d = MyDict() - k = K(d) - d[k] = 1 - for proto in range(2, pickle.HIGHEST_PROTOCOL + 1): - s = self.dumps(d, proto) - x = self.loads(s) - self.assertIsInstance(x, MyDict) - self.assertEqual(len(list(x.keys())), 1) - self.assertIsInstance(list(x.keys())[0], K) - self.assertIs(list(x.keys())[0].value, x) + self.assertIsInstance(x, K) + self.assertIsInstance(x.value, set) + self.assertEqual(len(x.value), 1) + self.assertIs(list(x.value)[0], x) def test_recursive_inst(self): - i = C() + # Mutable object containing itself. + i = Object() i.attr = i for proto in protocols: s = self.dumps(i, proto) x = self.loads(s) - self.assertIsInstance(x, C) + self.assertIsInstance(x, Object) self.assertEqual(dir(x), dir(i)) self.assertIs(x.attr, x) def test_recursive_multi(self): l = [] d = {1:l} - i = C() + i = Object() i.attr = d l.append(i) for proto in protocols: @@ -1601,49 +1709,94 @@ def test_recursive_multi(self): self.assertEqual(len(x), 1) self.assertEqual(dir(x[0]), dir(i)) self.assertEqual(list(x[0].attr.keys()), [1]) - self.assertTrue(x[0].attr[1] is x) - - def check_recursive_collection_and_inst(self, factory): - h = H() - y = factory([h]) - h.attr = y + self.assertIs(x[0].attr[1], x) + + def _test_recursive_collection_and_inst(self, factory): + # Mutable object containing a collection containing the original + # object. + o = Object() + o.attr = factory([o]) + t = type(o.attr) for proto in protocols: - s = self.dumps(y, proto) + s = self.dumps(o, proto) x = self.loads(s) - self.assertIsInstance(x, type(y)) + self.assertIsInstance(x.attr, t) + self.assertEqual(len(x.attr), 1) + self.assertIsInstance(list(x.attr)[0], Object) + self.assertIs(list(x.attr)[0], x) + + # Collection containing a mutable object containing the original + # collection. + o = o.attr + for proto in protocols: + s = self.dumps(o, proto) + x = self.loads(s) + self.assertIsInstance(x, t) self.assertEqual(len(x), 1) - self.assertIsInstance(list(x)[0], H) + self.assertIsInstance(list(x)[0], Object) self.assertIs(list(x)[0].attr, x) def test_recursive_list_and_inst(self): - self.check_recursive_collection_and_inst(list) + self._test_recursive_collection_and_inst(list) def test_recursive_tuple_and_inst(self): - self.check_recursive_collection_and_inst(tuple) + self._test_recursive_collection_and_inst(tuple) def test_recursive_dict_and_inst(self): - self.check_recursive_collection_and_inst(dict.fromkeys) + self._test_recursive_collection_and_inst(dict.fromkeys) def test_recursive_set_and_inst(self): - self.check_recursive_collection_and_inst(set) + self._test_recursive_collection_and_inst(set) def test_recursive_frozenset_and_inst(self): - self.check_recursive_collection_and_inst(frozenset) + self._test_recursive_collection_and_inst(frozenset) def test_recursive_list_subclass_and_inst(self): - self.check_recursive_collection_and_inst(MyList) + self._test_recursive_collection_and_inst(MyList) def test_recursive_tuple_subclass_and_inst(self): - self.check_recursive_collection_and_inst(MyTuple) + self._test_recursive_collection_and_inst(MyTuple) def test_recursive_dict_subclass_and_inst(self): - self.check_recursive_collection_and_inst(MyDict.fromkeys) + self._test_recursive_collection_and_inst(MyDict.fromkeys) def test_recursive_set_subclass_and_inst(self): - self.check_recursive_collection_and_inst(MySet) + self._test_recursive_collection_and_inst(MySet) def test_recursive_frozenset_subclass_and_inst(self): - self.check_recursive_collection_and_inst(MyFrozenSet) + self._test_recursive_collection_and_inst(MyFrozenSet) + + def test_recursive_inst_state(self): + # Mutable object containing itself. + y = REX_state() + y.state = y + for proto in protocols: + s = self.dumps(y, proto) + x = self.loads(s) + self.assertIsInstance(x, REX_state) + self.assertIs(x.state, x) + + def test_recursive_tuple_and_inst_state(self): + # Tuple containing a mutable object containing the original tuple. + t = (REX_state(),) + t[0].state = t + for proto in protocols: + s = self.dumps(t, proto) + x = self.loads(s) + self.assertIsInstance(x, tuple) + self.assertEqual(len(x), 1) + self.assertIsInstance(x[0], REX_state) + self.assertIs(x[0].state, x) + + # Mutable object containing a tuple containing the object. + t, = t + for proto in protocols: + s = self.dumps(t, proto) + x = self.loads(s) + self.assertIsInstance(x, REX_state) + self.assertIsInstance(x.state, tuple) + self.assertEqual(len(x.state), 1) + self.assertIs(x.state[0], x) def test_unicode(self): endcases = ['', '<\\u>', '<\\\u1234>', '<\n>', @@ -3045,6 +3198,19 @@ def __setitem__(self, key, value): def __reduce__(self): return type(self), (), None, None, iter(self.table.items()) +class REX_state(object): + """This class is used to check the 3th argument (state) of + the reduce protocol. + """ + def __init__(self, state=None): + self.state = state + def __eq__(self, other): + return type(self) is type(other) and self.state == other.state + def __setstate__(self, state): + self.state = state + def __reduce__(self): + return type(self), (), self.state + # Test classes for newobj From 645174abe0d13cce2cb339cc80b095ad484428ea Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Mon, 4 Jan 2021 00:54:13 +0200 Subject: [PATCH 1924/2163] [3.8] [3.9] bpo-42789: Don't skip curses tests on non-tty. (GH-24009) (GH-24076) (GH-24078) If __stdout__ is not attached to terminal, try to use __stderr__ if it is attached to terminal, or open the terminal device, or use regular file as terminal, but some functions will be untested in the latter case. (cherry picked from commit 607501abb488fb37e33cf9d35260ab7baefa192f) (cherry picked from commit 0303008ebceb6ac6035cd9722d1393267304171d) --- Lib/test/test_curses.py | 102 +++++++++++++++++++++++++--------------- 1 file changed, 63 insertions(+), 39 deletions(-) diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py index 09738c8a41c942..b7349d9e42a5d3 100644 --- a/Lib/test/test_curses.py +++ b/Lib/test/test_curses.py @@ -47,37 +47,57 @@ class TestCurses(unittest.TestCase): @classmethod def setUpClass(cls): - if not sys.__stdout__.isatty(): - # Temporary skip tests on non-tty - raise unittest.SkipTest('sys.__stdout__ is not a tty') - cls.tmp = tempfile.TemporaryFile() - fd = cls.tmp.fileno() - else: - cls.tmp = None - fd = sys.__stdout__.fileno() # testing setupterm() inside initscr/endwin # causes terminal breakage - curses.setupterm(fd=fd) - - @classmethod - def tearDownClass(cls): - if cls.tmp: - cls.tmp.close() - del cls.tmp + stdout_fd = sys.__stdout__.fileno() + curses.setupterm(fd=stdout_fd) def setUp(self): + self.isatty = True + self.output = sys.__stdout__ + stdout_fd = sys.__stdout__.fileno() + if not sys.__stdout__.isatty(): + # initstr() unconditionally uses C stdout. + # If it is redirected to file or pipe, try to attach it + # to terminal. + # First, save a copy of the file descriptor of stdout, so it + # can be restored after finishing the test. + dup_fd = os.dup(stdout_fd) + self.addCleanup(os.close, dup_fd) + self.addCleanup(os.dup2, dup_fd, stdout_fd) + + if sys.__stderr__.isatty(): + # If stderr is connected to terminal, use it. + tmp = sys.__stderr__ + self.output = sys.__stderr__ + else: + try: + # Try to open the terminal device. + tmp = open('/dev/tty', 'wb', buffering=0) + except OSError: + # As a fallback, use regular file to write control codes. + # Some functions (like savetty) will not work, but at + # least the garbage control sequences will not be mixed + # with the testing report. + tmp = tempfile.TemporaryFile(mode='wb', buffering=0) + self.isatty = False + self.addCleanup(tmp.close) + self.output = None + os.dup2(tmp.fileno(), stdout_fd) + self.save_signals = SaveSignals() self.save_signals.save() - if verbose: + self.addCleanup(self.save_signals.restore) + if verbose and self.output is not None: # just to make the test output a little more readable - print() + sys.stderr.flush() + sys.stdout.flush() + print(file=self.output, flush=True) self.stdscr = curses.initscr() - curses.savetty() - - def tearDown(self): - curses.resetty() - curses.endwin() - self.save_signals.restore() + if self.isatty: + curses.savetty() + self.addCleanup(curses.endwin) + self.addCleanup(curses.resetty) def test_window_funcs(self): "Test the methods of windows" @@ -95,7 +115,7 @@ def test_window_funcs(self): for meth in [stdscr.clear, stdscr.clrtobot, stdscr.clrtoeol, stdscr.cursyncup, stdscr.delch, stdscr.deleteln, stdscr.erase, stdscr.getbegyx, - stdscr.getbkgd, stdscr.getkey, stdscr.getmaxyx, + stdscr.getbkgd, stdscr.getmaxyx, stdscr.getparyx, stdscr.getyx, stdscr.inch, stdscr.insertln, stdscr.instr, stdscr.is_wintouched, win.noutrefresh, stdscr.redrawwin, stdscr.refresh, @@ -206,6 +226,11 @@ def test_window_funcs(self): if hasattr(stdscr, 'enclose'): stdscr.enclose(10, 10) + with tempfile.TemporaryFile() as f: + self.stdscr.putwin(f) + f.seek(0) + curses.getwin(f) + self.assertRaises(ValueError, stdscr.getstr, -400) self.assertRaises(ValueError, stdscr.getstr, 2, 3, -400) self.assertRaises(ValueError, stdscr.instr, -2) @@ -224,16 +249,19 @@ def test_embedded_null_chars(self): def test_module_funcs(self): "Test module-level functions" for func in [curses.baudrate, curses.beep, curses.can_change_color, - curses.cbreak, curses.def_prog_mode, curses.doupdate, - curses.flash, curses.flushinp, + curses.doupdate, curses.flash, curses.flushinp, curses.has_colors, curses.has_ic, curses.has_il, curses.isendwin, curses.killchar, curses.longname, - curses.nocbreak, curses.noecho, curses.nonl, - curses.noqiflush, curses.noraw, - curses.reset_prog_mode, curses.termattrs, - curses.termname, curses.erasechar]: + curses.noecho, curses.nonl, curses.noqiflush, + curses.termattrs, curses.termname, curses.erasechar]: with self.subTest(func=func.__qualname__): func() + if self.isatty: + for func in [curses.cbreak, curses.def_prog_mode, + curses.nocbreak, curses.noraw, + curses.reset_prog_mode]: + with self.subTest(func=func.__qualname__): + func() if hasattr(curses, 'filter'): curses.filter() if hasattr(curses, 'getsyx'): @@ -245,13 +273,9 @@ def test_module_funcs(self): curses.delay_output(1) curses.echo() ; curses.echo(1) - with tempfile.TemporaryFile() as f: - self.stdscr.putwin(f) - f.seek(0) - curses.getwin(f) - curses.halfdelay(1) - curses.intrflush(1) + if self.isatty: + curses.intrflush(1) curses.meta(1) curses.napms(100) curses.newpad(50,50) @@ -260,7 +284,8 @@ def test_module_funcs(self): curses.nl() ; curses.nl(1) curses.putp(b'abc') curses.qiflush() - curses.raw() ; curses.raw(1) + if self.isatty: + curses.raw() ; curses.raw(1) if hasattr(curses, 'setsyx'): curses.setsyx(5,5) curses.tigetflag('hc') @@ -282,7 +307,7 @@ def test_colors_funcs(self): curses.init_pair(2, 1,1) curses.color_content(1) curses.color_pair(2) - curses.pair_content(curses.COLOR_PAIRS - 1) + curses.pair_content(min(curses.COLOR_PAIRS - 1, 0x7fff)) curses.pair_number(0) if hasattr(curses, 'use_default_colors'): @@ -354,7 +379,6 @@ def test_resize_term(self): @requires_curses_func('resizeterm') def test_resizeterm(self): - stdscr = self.stdscr lines, cols = curses.LINES, curses.COLS new_lines = lines - 1 new_cols = cols + 1 From 82794cacc6073af9bc2b792a5ee21397f4c9037f Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Mon, 4 Jan 2021 10:58:09 +0200 Subject: [PATCH 1925/2163] [3.8] [3.9] bpo-42681: Fix range checks for color and pair numbers in curses (GH-23874). (GH-24077) (GH-24079) (cherry picked from commit 1470edd6131c29b8a09ce012cdfee3afa269d553) (cherry picked from commit b0ee2b492dbf550fbd2a63b82de0a4dc9d67f32e) --- Doc/library/curses.rst | 14 +- Lib/test/test_curses.py | 133 ++++++++++++++++-- .../2020-12-20-22-50-15.bpo-42681.lDO6jb.rst | 1 + Modules/_cursesmodule.c | 44 +++--- Modules/clinic/_cursesmodule.c.h | 26 ++-- 5 files changed, 157 insertions(+), 61 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-12-20-22-50-15.bpo-42681.lDO6jb.rst diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index 7cd20253aeea6a..5a7536412e6ce6 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -112,14 +112,15 @@ The module :mod:`curses` defines the following functions: .. function:: color_content(color_number) Return the intensity of the red, green, and blue (RGB) components in the color - *color_number*, which must be between ``0`` and :const:`COLORS`. Return a 3-tuple, + *color_number*, which must be between ``0`` and ``COLORS - 1``. Return a 3-tuple, containing the R,G,B values for the given color, which will be between ``0`` (no component) and ``1000`` (maximum amount of component). -.. function:: color_pair(color_number) +.. function:: color_pair(pair_number) - Return the attribute value for displaying text in the specified color. This + Return the attribute value for displaying text in the specified color pair. + Only the first 256 color pairs are supported. This attribute value can be combined with :const:`A_STANDOUT`, :const:`A_REVERSE`, and the other :const:`A_\*` attributes. :func:`pair_number` is the counterpart to this function. @@ -278,7 +279,7 @@ The module :mod:`curses` defines the following functions: Change the definition of a color, taking the number of the color to be changed followed by three RGB values (for the amounts of red, green, and blue components). The value of *color_number* must be between ``0`` and - :const:`COLORS`. Each of *r*, *g*, *b*, must be a value between ``0`` and + `COLORS - 1`. Each of *r*, *g*, *b*, must be a value between ``0`` and ``1000``. When :func:`init_color` is used, all occurrences of that color on the screen immediately change to the new definition. This function is a no-op on most terminals; it is active only if :func:`can_change_color` returns ``True``. @@ -291,7 +292,8 @@ The module :mod:`curses` defines the following functions: color number. The value of *pair_number* must be between ``1`` and ``COLOR_PAIRS - 1`` (the ``0`` color pair is wired to white on black and cannot be changed). The value of *fg* and *bg* arguments must be between ``0`` and - :const:`COLORS`. If the color-pair was previously initialized, the screen is + ``COLORS - 1``, or, after calling :func:`use_default_colors`, ``-1``. + If the color-pair was previously initialized, the screen is refreshed and all occurrences of that color-pair are changed to the new definition. @@ -441,7 +443,7 @@ The module :mod:`curses` defines the following functions: .. function:: pair_content(pair_number) Return a tuple ``(fg, bg)`` containing the colors for the requested color pair. - The value of *pair_number* must be between ``1`` and ``COLOR_PAIRS - 1``. + The value of *pair_number* must be between ``0`` and ``COLOR_PAIRS - 1``. .. function:: pair_number(attr) diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py index b7349d9e42a5d3..e39161fde311d0 100644 --- a/Lib/test/test_curses.py +++ b/Lib/test/test_curses.py @@ -4,8 +4,7 @@ # This script doesn't actually display anything very coherent. but it # does call (nearly) every method and function. # -# Functions not tested: {def,reset}_{shell,prog}_mode, getch(), getstr(), -# init_color() +# Functions not tested: {def,reset}_{shell,prog}_mode, getch(), getstr() # Only called, not tested: getmouse(), ungetmouse() # @@ -13,6 +12,7 @@ import string import sys import tempfile +import functools import unittest from test.support import requires, import_module, verbose, SaveSignals @@ -36,7 +36,17 @@ def requires_curses_func(name): return unittest.skipUnless(hasattr(curses, name), 'requires curses.%s' % name) +def requires_colors(test): + @functools.wraps(test) + def wrapped(self, *args, **kwargs): + if not curses.has_colors(): + self.skipTest('requires colors support') + curses.start_color() + test(self, *args, **kwargs) + return wrapped + term = os.environ.get('TERM') +SHORT_MAX = 0x7fff # If newterm was supported we could use it instead of initscr and not exit @unittest.skipIf(not term or term == 'unknown', @@ -47,6 +57,8 @@ class TestCurses(unittest.TestCase): @classmethod def setUpClass(cls): + if verbose: + print(f'TERM={term}', file=sys.stderr, flush=True) # testing setupterm() inside initscr/endwin # causes terminal breakage stdout_fd = sys.__stdout__.fileno() @@ -300,18 +312,111 @@ def test_module_funcs(self): curses.use_env(1) # Functions only available on a few platforms - def test_colors_funcs(self): - if not curses.has_colors(): - self.skipTest('requires colors support') - curses.start_color() - curses.init_pair(2, 1,1) - curses.color_content(1) - curses.color_pair(2) - curses.pair_content(min(curses.COLOR_PAIRS - 1, 0x7fff)) - curses.pair_number(0) - - if hasattr(curses, 'use_default_colors'): - curses.use_default_colors() + + def bad_colors(self): + return (-2**31 - 1, 2**31, -2**63 - 1, 2**63, 2**64) + + def bad_pairs(self): + return (-2**31 - 1, 2**31, -2**63 - 1, 2**63, 2**64) + + @requires_colors + def test_color_content(self): + self.assertEqual(curses.color_content(curses.COLOR_BLACK), (0, 0, 0)) + curses.color_content(0) + curses.color_content(min(curses.COLORS - 1, SHORT_MAX)) + + for color in self.bad_colors(): + self.assertRaises(OverflowError, curses.color_content, color) + if curses.COLORS <= SHORT_MAX: + self.assertRaises(curses.error, curses.color_content, curses.COLORS) + self.assertRaises(curses.error, curses.color_content, -1) + + @requires_colors + def test_init_color(self): + if not curses.can_change_color: + self.skipTest('cannot change color') + + old = curses.color_content(0) + try: + curses.init_color(0, *old) + except curses.error: + self.skipTest('cannot change color (init_color() failed)') + self.addCleanup(curses.init_color, 0, *old) + curses.init_color(0, 0, 0, 0) + self.assertEqual(curses.color_content(0), (0, 0, 0)) + curses.init_color(0, 1000, 1000, 1000) + self.assertEqual(curses.color_content(0), (1000, 1000, 1000)) + + maxcolor = min(curses.COLORS - 1, SHORT_MAX) + old = curses.color_content(maxcolor) + curses.init_color(maxcolor, *old) + self.addCleanup(curses.init_color, maxcolor, *old) + curses.init_color(maxcolor, 0, 500, 1000) + self.assertEqual(curses.color_content(maxcolor), (0, 500, 1000)) + + for color in self.bad_colors(): + self.assertRaises(OverflowError, curses.init_color, color, 0, 0, 0) + if curses.COLORS <= SHORT_MAX: + self.assertRaises(curses.error, curses.init_color, curses.COLORS, 0, 0, 0) + self.assertRaises(curses.error, curses.init_color, -1, 0, 0, 0) + for comp in (-1, 1001): + self.assertRaises(curses.error, curses.init_color, 0, comp, 0, 0) + self.assertRaises(curses.error, curses.init_color, 0, 0, comp, 0) + self.assertRaises(curses.error, curses.init_color, 0, 0, 0, comp) + + @requires_colors + def test_pair_content(self): + if not hasattr(curses, 'use_default_colors'): + self.assertEqual(curses.pair_content(0), + (curses.COLOR_WHITE, curses.COLOR_BLACK)) + curses.pair_content(0) + curses.pair_content(min(curses.COLOR_PAIRS - 1, SHORT_MAX)) + + for pair in self.bad_pairs(): + self.assertRaises(OverflowError, curses.pair_content, pair) + self.assertRaises(curses.error, curses.pair_content, -1) + + @requires_colors + def test_init_pair(self): + old = curses.pair_content(1) + curses.init_pair(1, *old) + self.addCleanup(curses.init_pair, 1, *old) + + curses.init_pair(1, 0, 0) + self.assertEqual(curses.pair_content(1), (0, 0)) + maxcolor = min(curses.COLORS - 1, SHORT_MAX) + curses.init_pair(1, maxcolor, maxcolor) + self.assertEqual(curses.pair_content(1), (maxcolor, maxcolor)) + maxpair = min(curses.COLOR_PAIRS - 1, SHORT_MAX) + curses.init_pair(maxpair, 2, 3) + self.assertEqual(curses.pair_content(maxpair), (2, 3)) + + for pair in self.bad_pairs(): + self.assertRaises(OverflowError, curses.init_pair, pair, 0, 0) + self.assertRaises(curses.error, curses.init_pair, -1, 0, 0) + for color in self.bad_colors(): + self.assertRaises(OverflowError, curses.init_pair, 1, color, 0) + self.assertRaises(OverflowError, curses.init_pair, 1, 0, color) + if curses.COLORS <= SHORT_MAX: + self.assertRaises(curses.error, curses.init_pair, 1, curses.COLORS, 0) + self.assertRaises(curses.error, curses.init_pair, 1, 0, curses.COLORS) + + @requires_colors + def test_color_attrs(self): + for pair in 0, 1, 255: + attr = curses.color_pair(pair) + self.assertEqual(curses.pair_number(attr), pair, attr) + self.assertEqual(curses.pair_number(attr | curses.A_BOLD), pair) + self.assertEqual(curses.color_pair(0), 0) + self.assertEqual(curses.pair_number(0), 0) + + @requires_curses_func('use_default_colors') + @requires_colors + def test_use_default_colors(self): + self.assertIn(curses.pair_content(0), + ((curses.COLOR_WHITE, curses.COLOR_BLACK), (-1, -1))) + curses.use_default_colors() + self.assertEqual(curses.pair_content(0), (-1, -1)) @requires_curses_func('keyname') def test_keyname(self): diff --git a/Misc/NEWS.d/next/Library/2020-12-20-22-50-15.bpo-42681.lDO6jb.rst b/Misc/NEWS.d/next/Library/2020-12-20-22-50-15.bpo-42681.lDO6jb.rst new file mode 100644 index 00000000000000..34ea74a5a323d8 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-12-20-22-50-15.bpo-42681.lDO6jb.rst @@ -0,0 +1 @@ +Fixed range checks for color and pair numbers in :mod:`curses`. diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c index ac23d5d7474a0e..47459d65ff6e49 100644 --- a/Modules/_cursesmodule.c +++ b/Modules/_cursesmodule.c @@ -176,18 +176,6 @@ static char *screen_encoding = NULL; /* Utility Functions */ -static inline int -color_pair_to_attr(short color_number) -{ - return ((int)color_number << 8); -} - -static inline short -attr_to_color_pair(int attr) -{ - return (short)((attr & A_COLOR) >> 8); -} - /* * Check the return code from a curses function and return None * or raise an exception as appropriate. These are exported using the @@ -618,7 +606,7 @@ _curses_window_addch_impl(PyCursesWindowObject *self, int group_left_1, if (type == 2) { funcname = "add_wch"; wstr[1] = L'\0'; - setcchar(&wcval, wstr, attr, attr_to_color_pair(attr), NULL); + setcchar(&wcval, wstr, attr, PAIR_NUMBER(attr), NULL); if (coordinates_group) rtn = mvwadd_wch(self->win,y,x, &wcval); else { @@ -2586,7 +2574,7 @@ NoArgOrFlagNoReturnFunctionBody(cbreak, flag) _curses.color_content color_number: short - The number of the color (0 - COLORS). + The number of the color (0 - (COLORS-1)). / Return the red, green, and blue (RGB) components of the specified color. @@ -2597,7 +2585,7 @@ which will be between 0 (no component) and 1000 (maximum amount of component). static PyObject * _curses_color_content_impl(PyObject *module, short color_number) -/*[clinic end generated code: output=cb15cf3120d4bfc1 input=5555abb1c11e11b7]*/ +/*[clinic end generated code: output=cb15cf3120d4bfc1 input=630f6737514db6ad]*/ { short r,g,b; @@ -2616,8 +2604,8 @@ _curses_color_content_impl(PyObject *module, short color_number) /*[clinic input] _curses.color_pair - color_number: short - The number of the color (0 - COLORS). + pair_number: short + The number of the color pair. / Return the attribute value for displaying text in the specified color. @@ -2627,13 +2615,13 @@ other A_* attributes. pair_number() is the counterpart to this function. [clinic start generated code]*/ static PyObject * -_curses_color_pair_impl(PyObject *module, short color_number) -/*[clinic end generated code: output=6a84cb6b29ecaf9a input=a9d3eb6f50e4dc12]*/ +_curses_color_pair_impl(PyObject *module, short pair_number) +/*[clinic end generated code: output=ce609d238b70dc11 input=8dd0d5da94cb15b5]*/ { PyCursesInitialised; PyCursesInitialisedColor; - return PyLong_FromLong(color_pair_to_attr(color_number)); + return PyLong_FromLong(COLOR_PAIR(pair_number)); } /*[clinic input] @@ -3028,7 +3016,7 @@ _curses_has_key_impl(PyObject *module, int key) _curses.init_color color_number: short - The number of the color to be changed (0 - COLORS). + The number of the color to be changed (0 - (COLORS-1)). r: short Red component (0 - 1000). g: short @@ -3041,13 +3029,13 @@ Change the definition of a color. When init_color() is used, all occurrences of that color on the screen immediately change to the new definition. This function is a no-op on -most terminals; it is active only if can_change_color() returns 1. +most terminals; it is active only if can_change_color() returns true. [clinic start generated code]*/ static PyObject * _curses_init_color_impl(PyObject *module, short color_number, short r, short g, short b) -/*[clinic end generated code: output=280236f5efe9776a input=f3a05bd38f619175]*/ +/*[clinic end generated code: output=280236f5efe9776a input=128601b5dc76d548]*/ { PyCursesInitialised; PyCursesInitialisedColor; @@ -3061,9 +3049,9 @@ _curses.init_pair pair_number: short The number of the color-pair to be changed (1 - (COLOR_PAIRS-1)). fg: short - Foreground color number (0 - COLORS). + Foreground color number (-1 - (COLORS-1)). bg: short - Background color number (0 - COLORS). + Background color number (-1 - (COLORS-1)). / Change the definition of a color-pair. @@ -3075,7 +3063,7 @@ all occurrences of that color-pair are changed to the new definition. static PyObject * _curses_init_pair_impl(PyObject *module, short pair_number, short fg, short bg) -/*[clinic end generated code: output=9c2ce39c22f376b6 input=c9f0b11b17a2ac6d]*/ +/*[clinic end generated code: output=9c2ce39c22f376b6 input=12c320ec14396ea2]*/ { PyCursesInitialised; PyCursesInitialisedColor; @@ -3631,7 +3619,7 @@ _curses_pair_content_impl(PyObject *module, short pair_number) if (pair_content(pair_number, &f, &b)==ERR) { PyErr_SetString(PyCursesError, - "Argument 1 was out of range. (1..COLOR_PAIRS-1)"); + "Argument 1 was out of range. (0..COLOR_PAIRS-1)"); return NULL; } @@ -3656,7 +3644,7 @@ _curses_pair_number_impl(PyObject *module, int attr) PyCursesInitialised; PyCursesInitialisedColor; - return PyLong_FromLong(attr_to_color_pair(attr)); + return PyLong_FromLong(PAIR_NUMBER(attr)); } /*[clinic input] diff --git a/Modules/clinic/_cursesmodule.c.h b/Modules/clinic/_cursesmodule.c.h index 437604cb92ae67..889c2f64e1a118 100644 --- a/Modules/clinic/_cursesmodule.c.h +++ b/Modules/clinic/_cursesmodule.c.h @@ -2028,7 +2028,7 @@ PyDoc_STRVAR(_curses_color_content__doc__, "Return the red, green, and blue (RGB) components of the specified color.\n" "\n" " color_number\n" -" The number of the color (0 - COLORS).\n" +" The number of the color (0 - (COLORS-1)).\n" "\n" "A 3-tuple is returned, containing the R, G, B values for the given color,\n" "which will be between 0 (no component) and 1000 (maximum amount of component)."); @@ -2076,13 +2076,13 @@ _curses_color_content(PyObject *module, PyObject *arg) } PyDoc_STRVAR(_curses_color_pair__doc__, -"color_pair($module, color_number, /)\n" +"color_pair($module, pair_number, /)\n" "--\n" "\n" "Return the attribute value for displaying text in the specified color.\n" "\n" -" color_number\n" -" The number of the color (0 - COLORS).\n" +" pair_number\n" +" The number of the color pair.\n" "\n" "This attribute value can be combined with A_STANDOUT, A_REVERSE, and the\n" "other A_* attributes. pair_number() is the counterpart to this function."); @@ -2091,13 +2091,13 @@ PyDoc_STRVAR(_curses_color_pair__doc__, {"color_pair", (PyCFunction)_curses_color_pair, METH_O, _curses_color_pair__doc__}, static PyObject * -_curses_color_pair_impl(PyObject *module, short color_number); +_curses_color_pair_impl(PyObject *module, short pair_number); static PyObject * _curses_color_pair(PyObject *module, PyObject *arg) { PyObject *return_value = NULL; - short color_number; + short pair_number; if (PyFloat_Check(arg)) { PyErr_SetString(PyExc_TypeError, @@ -2120,10 +2120,10 @@ _curses_color_pair(PyObject *module, PyObject *arg) goto exit; } else { - color_number = (short) ival; + pair_number = (short) ival; } } - return_value = _curses_color_pair_impl(module, color_number); + return_value = _curses_color_pair_impl(module, pair_number); exit: return return_value; @@ -2699,7 +2699,7 @@ PyDoc_STRVAR(_curses_init_color__doc__, "Change the definition of a color.\n" "\n" " color_number\n" -" The number of the color to be changed (0 - COLORS).\n" +" The number of the color to be changed (0 - (COLORS-1)).\n" " r\n" " Red component (0 - 1000).\n" " g\n" @@ -2709,7 +2709,7 @@ PyDoc_STRVAR(_curses_init_color__doc__, "\n" "When init_color() is used, all occurrences of that color on the screen\n" "immediately change to the new definition. This function is a no-op on\n" -"most terminals; it is active only if can_change_color() returns 1."); +"most terminals; it is active only if can_change_color() returns true."); #define _CURSES_INIT_COLOR_METHODDEF \ {"init_color", (PyCFunction)(void(*)(void))_curses_init_color, METH_FASTCALL, _curses_init_color__doc__}, @@ -2841,9 +2841,9 @@ PyDoc_STRVAR(_curses_init_pair__doc__, " pair_number\n" " The number of the color-pair to be changed (1 - (COLOR_PAIRS-1)).\n" " fg\n" -" Foreground color number (0 - COLORS).\n" +" Foreground color number (-1 - (COLORS-1)).\n" " bg\n" -" Background color number (0 - COLORS).\n" +" Background color number (-1 - (COLORS-1)).\n" "\n" "If the color-pair was previously initialized, the screen is refreshed and\n" "all occurrences of that color-pair are changed to the new definition."); @@ -4559,4 +4559,4 @@ _curses_use_default_colors(PyObject *module, PyObject *Py_UNUSED(ignored)) #ifndef _CURSES_USE_DEFAULT_COLORS_METHODDEF #define _CURSES_USE_DEFAULT_COLORS_METHODDEF #endif /* !defined(_CURSES_USE_DEFAULT_COLORS_METHODDEF) */ -/*[clinic end generated code: output=820af7050893ed16 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=c5267f2ffe238810 input=a9049054013a1b77]*/ From ed2d3d15b27f3599edbe3bdd278e529e0f0ba566 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 4 Jan 2021 02:21:01 -0800 Subject: [PATCH 1926/2163] Update Sphinx version for macOS installer build. (GH-24082) (GH-24088) (cherry picked from commit 0f3b96b368dd7ebb5dcd3759a30322dbd027f292) Co-authored-by: Ned Deily --- Mac/BuildScript/build-installer.py | 1 - 1 file changed, 1 deletion(-) diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index 2548b212d9ea42..eeb46990814bdb 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -1090,7 +1090,6 @@ def buildPythonDocs(): if not os.path.exists(htmlDir): # Create virtual environment for docs builds with blurb and sphinx runCommand('make venv') - runCommand('venv/bin/python3 -m pip install -U Sphinx==2.3.1') runCommand('make html PYTHON=venv/bin/python') os.rename(htmlDir, docdir) os.chdir(curDir) From f24ac455521e46bf9f6c7971aec0e4abec4451c4 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 4 Jan 2021 02:27:15 -0800 Subject: [PATCH 1927/2163] bpo-41837: Update macOS installer build to use OpenSSL 1.1.1i. (GH-24080) (#24084) (cherry picked from commit 14097a2785414c728d41d8d730a469a8c46ecdb9) Co-authored-by: Ned Deily --- Mac/BuildScript/build-installer.py | 6 +++--- .../next/macOS/2021-01-04-00-48-08.bpo-41837.dX-unJ.rst | 1 + 2 files changed, 4 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/macOS/2021-01-04-00-48-08.bpo-41837.dX-unJ.rst diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index eeb46990814bdb..7ef3753ad57575 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -209,9 +209,9 @@ def library_recipes(): result.extend([ dict( - name="OpenSSL 1.1.1g", - url="https://www.openssl.org/source/openssl-1.1.1g.tar.gz", - checksum='76766e98997660138cdaf13a187bd234', + name="OpenSSL 1.1.1i", + url="https://www.openssl.org/source/openssl-1.1.1i.tar.gz", + checksum='08987c3cf125202e2b0840035efb392c', buildrecipe=build_universal_openssl, configure=None, install=None, diff --git a/Misc/NEWS.d/next/macOS/2021-01-04-00-48-08.bpo-41837.dX-unJ.rst b/Misc/NEWS.d/next/macOS/2021-01-04-00-48-08.bpo-41837.dX-unJ.rst new file mode 100644 index 00000000000000..3f9415f4a36064 --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2021-01-04-00-48-08.bpo-41837.dX-unJ.rst @@ -0,0 +1 @@ +Update macOS installer build to use OpenSSL 1.1.1i. From 218ed0b013377764ba05f0d21e5b733f7bc481ac Mon Sep 17 00:00:00 2001 From: Julien Palard Date: Mon, 4 Jan 2021 17:19:16 +0100 Subject: [PATCH 1928/2163] Revert "[3.8] [doc] Fix erroneous backslashes in signatures and names (GH-23658)" (GH-24093) This partially reverts commit 02349e2dc9d93202c658ae383b2de2e36b2366f8, which was removing backslashes in documentations compiled with Sphinx < 3, used for Python 3.8 and 3.9 docs. --- Doc/library/asyncio-stream.rst | 2 +- Doc/library/base64.rst | 2 +- Doc/library/difflib.rst | 6 +++--- Doc/library/email.header.rst | 2 +- Doc/library/functions.rst | 2 +- Doc/library/http.cookies.rst | 2 +- Doc/library/io.rst | 2 +- Doc/library/xml.dom.minidom.rst | 2 +- Doc/requirements.txt | 2 +- 9 files changed, 11 insertions(+), 11 deletions(-) diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst index 584bf10fc042bc..714de8d41a3503 100644 --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -192,7 +192,7 @@ StreamReader can be read. Use the :attr:`IncompleteReadError.partial` attribute to get the partially read data. - .. coroutinemethod:: readuntil(separator=b'\n') + .. coroutinemethod:: readuntil(separator=b'\\n') Read data from the stream until *separator* is found. diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst index 3c63c15ad401e9..ad9f5f58bee2aa 100644 --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -178,7 +178,7 @@ The modern interface provides: .. versionadded:: 3.4 -.. function:: a85decode(b, *, foldspaces=False, adobe=False, ignorechars=b' \t\n\r\v') +.. function:: a85decode(b, *, foldspaces=False, adobe=False, ignorechars=b' \\t\\n\\r\\v') Decode the Ascii85 encoded :term:`bytes-like object` or ASCII string *b* and return the decoded :class:`bytes`. diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index e6dd1dd7a54f03..a8543b38c197ec 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -149,7 +149,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. contains a good example of its use. -.. function:: context_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\n') +.. function:: context_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\\n') Compare *a* and *b* (lists of strings); return a delta (a :term:`generator` generating the delta lines) in context diff format. @@ -279,7 +279,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. emu -.. function:: unified_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\n') +.. function:: unified_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\\n') Compare *a* and *b* (lists of strings); return a delta (a :term:`generator` generating the delta lines) in unified diff format. @@ -321,7 +321,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. See :ref:`difflib-interface` for a more detailed example. -.. function:: diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\n') +.. function:: diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\\n') Compare *a* and *b* (lists of bytes objects) using *dfunc*; yield a sequence of delta lines (also bytes) in the format returned by *dfunc*. diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst index e093f138936b36..07152c224f2ff0 100644 --- a/Doc/library/email.header.rst +++ b/Doc/library/email.header.rst @@ -116,7 +116,7 @@ Here is the :class:`Header` class description: if *s* is a byte string. - .. method:: encode(splitchars=';, \t', maxlinelen=None, linesep='\n') + .. method:: encode(splitchars=';, \\t', maxlinelen=None, linesep='\\n') Encode a message header into an RFC-compliant format, possibly wrapping long lines and encapsulating non-ASCII parts in base64 or quoted-printable diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index be98d30488ab8a..4f5fe6bc230697 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1312,7 +1312,7 @@ are always available. They are listed here in alphabetical order. supported. -.. function:: print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False) +.. function:: print(*objects, sep=' ', end='\\n', file=sys.stdout, flush=False) Print *objects* to the text stream *file*, separated by *sep* and followed by *end*. *sep*, *end*, *file* and *flush*, if present, must be given as keyword diff --git a/Doc/library/http.cookies.rst b/Doc/library/http.cookies.rst index a2c1eb00d8b33d..17792b200599bd 100644 --- a/Doc/library/http.cookies.rst +++ b/Doc/library/http.cookies.rst @@ -93,7 +93,7 @@ Cookie Objects :meth:`value_decode` are inverses on the range of *value_decode*. -.. method:: BaseCookie.output(attrs=None, header='Set-Cookie:', sep='\r\n') +.. method:: BaseCookie.output(attrs=None, header='Set-Cookie:', sep='\\r\\n') Return a string representation suitable to be sent as HTTP headers. *attrs* and *header* are sent to each :class:`Morsel`'s :meth:`output` method. *sep* is used diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 667e4c654ff235..32151a0ace458e 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -959,7 +959,7 @@ Text I/O .. versionadded:: 3.7 -.. class:: StringIO(initial_value='', newline='\n') +.. class:: StringIO(initial_value='', newline='\\n') An in-memory stream for text I/O. The text buffer is discarded when the :meth:`~IOBase.close` method is called. diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst index 673af8326a80e3..8711242d95d741 100644 --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -164,7 +164,7 @@ module documentation. This section lists the differences between the API and The :meth:`toxml` method now preserves the attribute order specified by the user. -.. method:: Node.toprettyxml(indent="\t", newl="\n", encoding=None) +.. method:: Node.toprettyxml(indent="\\t", newl="\\n", encoding=None) Return a pretty-printed version of the document. *indent* specifies the indentation string and defaults to a tabulator; *newl* specifies the string diff --git a/Doc/requirements.txt b/Doc/requirements.txt index d30ff39c8c9a33..47b78eeac817e0 100644 --- a/Doc/requirements.txt +++ b/Doc/requirements.txt @@ -3,7 +3,7 @@ # Sphinx version is pinned so that new versions that introduce new warnings # won't suddenly cause build failures. Updating the version is fine as long # as no warnings are raised by doing so. -sphinx +sphinx==2.4.4 blurb From a87bf5fd361b61b45bdfe52731c41fa6681a0f7b Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 4 Jan 2021 13:14:51 -0800 Subject: [PATCH 1929/2163] Do not remove x bit from published directories (GH-24101) (cherry picked from commit af4cd16479f2d55bee3db899e7679d7cde1d2ab7) Co-authored-by: Steve Dower --- Tools/msi/uploadrelease.ps1 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Tools/msi/uploadrelease.ps1 b/Tools/msi/uploadrelease.ps1 index d3673b4582983c..7c825c443f2287 100644 --- a/Tools/msi/uploadrelease.ps1 +++ b/Tools/msi/uploadrelease.ps1 @@ -90,7 +90,7 @@ if (-not $skipupload) { $d = "$target/$($p[0])/" & $plink -batch $user@$server mkdir $d & $plink -batch $user@$server chgrp downloads $d - & $plink -batch $user@$server chmod g-x,o+rx $d + & $plink -batch $user@$server chmod o+rx $d & $pscp -batch $chm.FullName "$user@${server}:$d" if (-not $?) { throw "Failed to upload $chm" } @@ -115,7 +115,7 @@ if (-not $skipupload) { $sd = "$d$($a.Name)$($p[1])/" & $plink -batch $user@$server mkdir $sd & $plink -batch $user@$server chgrp downloads $sd - & $plink -batch $user@$server chmod g-x,o+rx $sd + & $plink -batch $user@$server chmod o+rx $sd & $pscp -batch $msi.FullName "$user@${server}:$sd" if (-not $?) { throw "Failed to upload $msi" } & $plink -batch $user@$server chgrp downloads $sd* From 0ccac5ff587d7637854e5d3e75f0f9a8f5528e59 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 4 Jan 2021 15:36:37 -0800 Subject: [PATCH 1930/2163] bpo-40810: Fix CheckTraceCallbackContent for SQLite pre 3.7.15 (GH-20530) Ref. [SQLite 3.7.15 changelog](https://sqlite.org/changes.htmlGH-version_3_7_15): _"Avoid invoking the sqlite3_trace() callback multiple times when a statement is automatically reprepared due to SQLITE_SCHEMA errors."_ (cherry picked from commit f7f0ed59bcc41ed20674d4b2aa443d3b79e725f4) Co-authored-by: Erlend Egeberg Aasland --- Lib/sqlite3/test/hooks.py | 8 ++++++++ .../next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst | 1 + 2 files changed, 9 insertions(+) create mode 100644 Misc/NEWS.d/next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst diff --git a/Lib/sqlite3/test/hooks.py b/Lib/sqlite3/test/hooks.py index d74e74bf272275..214205c1167a41 100644 --- a/Lib/sqlite3/test/hooks.py +++ b/Lib/sqlite3/test/hooks.py @@ -265,6 +265,14 @@ def trace(statement): cur.execute(queries[0]) con2.execute("create table bar(x)") cur.execute(queries[1]) + + # Extract from SQLite 3.7.15 changelog: + # Avoid invoking the sqlite3_trace() callback multiple times when a + # statement is automatically reprepared due to SQLITE_SCHEMA errors. + # + # See bpo-40810 + if sqlite.sqlite_version_info < (3, 7, 15): + queries.append(queries[-1]) self.assertEqual(traced_statements, queries) diff --git a/Misc/NEWS.d/next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst b/Misc/NEWS.d/next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst new file mode 100644 index 00000000000000..1965ecd6ef511f --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst @@ -0,0 +1 @@ +In :mod:`sqlite3`, fix `CheckTraceCallbackContent` for SQLite pre 3.7.15. From d05b470d6ad6dfe0d4933ffc3c5c1e2c03b390bd Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 5 Jan 2021 00:09:30 -0800 Subject: [PATCH 1931/2163] bpo-42508: Keep IDLE running on macOS (GH-23577) (#23670) Remove obsolete workaround that prevented running files with shortcuts when using new universal2 installers built on macOS 11. Ignore buggy 2nd run_module_event call. (cherry picked from commit 57e511361047895231f5ee7abfdfbbc60e11d2db) Co-authored-by: Terry Jan Reedy --- Lib/idlelib/NEWS.txt | 4 +++ Lib/idlelib/runscript.py | 26 +++++-------------- .../2020-11-30-19-46-05.bpo-42508.fE7w4M.rst | 3 +++ 3 files changed, 14 insertions(+), 19 deletions(-) create mode 100644 Misc/NEWS.d/next/IDLE/2020-11-30-19-46-05.bpo-42508.fE7w4M.rst diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 4107ff7f7e9241..27faa70bf72e75 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,10 @@ Released on 2020-12-?? ====================================== +bpo-42508: Keep IDLE running on macOS. Remove obsolete workaround +that prevented running files with shortcuts when using new universal2 +installers built on macOS 11. + bpo-42426: Fix reporting offset of the RE error in searchengine. bpo-42416: Get docstrings for IDLE calltips more often diff --git a/Lib/idlelib/runscript.py b/Lib/idlelib/runscript.py index a54108794ab595..028b0dbd21dfe6 100644 --- a/Lib/idlelib/runscript.py +++ b/Lib/idlelib/runscript.py @@ -11,6 +11,7 @@ """ import os import tabnanny +import time import tokenize import tkinter.messagebox as tkMessageBox @@ -42,9 +43,7 @@ def __init__(self, editwin): self.root = self.editwin.root # cli_args is list of strings that extends sys.argv self.cli_args = [] - - if macosx.isCocoaTk(): - self.editwin.text_frame.bind('<>', self._run_module_event) + self.perf = 0.0 # Workaround for macOS 11 Uni2; see bpo-42508. def check_module_event(self, event): if isinstance(self.editwin, outwin.OutputWindow): @@ -107,24 +106,10 @@ def checksyntax(self, filename): finally: shell.set_warning_stream(saved_stream) - def run_module_event(self, event): - if macosx.isCocoaTk(): - # Tk-Cocoa in MacOSX is broken until at least - # Tk 8.5.9, and without this rather - # crude workaround IDLE would hang when a user - # tries to run a module using the keyboard shortcut - # (the menu item works fine). - self.editwin.text_frame.after(200, - lambda: self.editwin.text_frame.event_generate( - '<>')) - return 'break' - else: - return self._run_module_event(event) - def run_custom_event(self, event): - return self._run_module_event(event, customize=True) + return self.run_module_event(event, customize=True) - def _run_module_event(self, event, *, customize=False): + def run_module_event(self, event, *, customize=False): """Run the module after setting up the environment. First check the syntax. Next get customization. If OK, make @@ -133,6 +118,8 @@ def _run_module_event(self, event, *, customize=False): module being executed and also add that directory to its sys.path if not already included. """ + if macosx.isCocoaTk() and (time.perf_counter() - self.perf < .05): + return 'break' if isinstance(self.editwin, outwin.OutputWindow): self.editwin.text.bell() return 'break' @@ -218,6 +205,7 @@ def errorbox(self, title, message): # XXX This should really be a function of EditorWindow... tkMessageBox.showerror(title, message, parent=self.editwin.text) self.editwin.text.focus_set() + self.perf = time.perf_counter() if __name__ == "__main__": diff --git a/Misc/NEWS.d/next/IDLE/2020-11-30-19-46-05.bpo-42508.fE7w4M.rst b/Misc/NEWS.d/next/IDLE/2020-11-30-19-46-05.bpo-42508.fE7w4M.rst new file mode 100644 index 00000000000000..b449351f7f458e --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2020-11-30-19-46-05.bpo-42508.fE7w4M.rst @@ -0,0 +1,3 @@ +Keep IDLE running on macOS. Remove obsolete workaround that prevented +running files with shortcuts when using new universal2 installers built +on macOS 11. From 3b6dcb86302ddbf48f57f7bd2ffc63a801b11e64 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 5 Jan 2021 00:18:43 -0800 Subject: [PATCH 1932/2163] Fix broken NEWS markup (GH-24110) (cherry picked from commit cde988e893793f58bf87e7a8c014926fd2e32904) Co-authored-by: Brandt Bucher --- Misc/NEWS.d/next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Misc/NEWS.d/next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst b/Misc/NEWS.d/next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst index 1965ecd6ef511f..eafd94cabede93 100644 --- a/Misc/NEWS.d/next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst +++ b/Misc/NEWS.d/next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst @@ -1 +1 @@ -In :mod:`sqlite3`, fix `CheckTraceCallbackContent` for SQLite pre 3.7.15. +In :mod:`sqlite3`, fix ``CheckTraceCallbackContent`` for SQLite pre 3.7.15. From a087a97438e922fcace357ff4c29806ff65838d8 Mon Sep 17 00:00:00 2001 From: Terry Jan Reedy Date: Tue, 5 Jan 2021 04:18:02 -0500 Subject: [PATCH 1933/2163] [3.8] bpo-32631: IDLE: Enable zzdummy example extension module (GH-14491) Make menu items work with formatter, add docstrings, add 100% tests. Co-authored-by: Cheryl Sabella Co-authored-by: Terry Jan Reedy (cherry picked from commit e40e2a2cc94c554e7e245a8ca5a7432d31a95766) --- Lib/idlelib/NEWS.txt | 3 + Lib/idlelib/configdialog.py | 10 +- Lib/idlelib/extend.txt | 18 +-- Lib/idlelib/idle_test/test_zzdummy.py | 152 ++++++++++++++++++ Lib/idlelib/zzdummy.py | 73 ++++++--- .../2019-06-30-20-31-09.bpo-32631.e7_4BG.rst | 2 + 6 files changed, 227 insertions(+), 31 deletions(-) create mode 100644 Lib/idlelib/idle_test/test_zzdummy.py create mode 100644 Misc/NEWS.d/next/IDLE/2019-06-30-20-31-09.bpo-32631.e7_4BG.rst diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 27faa70bf72e75..bc15501a7b062a 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,9 @@ Released on 2020-12-?? ====================================== +bpo-32631: Finish zzdummy example extension module: make menu entries +work; add docstrings and tests with 100% coverage. + bpo-42508: Keep IDLE running on macOS. Remove obsolete workaround that prevented running files with shortcuts when using new universal2 installers built on macOS 11. diff --git a/Lib/idlelib/configdialog.py b/Lib/idlelib/configdialog.py index a84e1c5668f99f..73e64852c69dfd 100644 --- a/Lib/idlelib/configdialog.py +++ b/Lib/idlelib/configdialog.py @@ -2316,7 +2316,15 @@ def detach(self): Shell Preferences: Auto-Squeeze Min. Lines is the minimum number of lines of output to automatically "squeeze". -''' +''', + 'Extensions': ''' +ZzDummy: This extension is provided as an example for how to create and +use an extension. Enable indicates whether the extension is active or +not; likewise enable_editor and enable_shell indicate which windows it +will be active on. For this extension, z-text is the text that will be +inserted at or removed from the beginning of the lines of selected text, +or the current line if no selection. +''', } diff --git a/Lib/idlelib/extend.txt b/Lib/idlelib/extend.txt index c9cb2e8297eb35..b482f76c4fb0f7 100644 --- a/Lib/idlelib/extend.txt +++ b/Lib/idlelib/extend.txt @@ -28,8 +28,8 @@ variables: (There are a few more, but they are rarely useful.) The extension class must not directly bind Window Manager (e.g. X) events. -Rather, it must define one or more virtual events, e.g. <>, and -corresponding methods, e.g. zoom_height_event(). The virtual events will be +Rather, it must define one or more virtual events, e.g. <>, and +corresponding methods, e.g. z_in_event(). The virtual events will be bound to the corresponding methods, and Window Manager events can then be bound to the virtual events. (This indirection is done so that the key bindings can easily be changed, and so that other sources of virtual events can exist, such @@ -54,21 +54,21 @@ Extensions are not required to define menu entries for all the events they implement. (They are also not required to create keybindings, but in that case there must be empty bindings in cofig-extensions.def) -Here is a complete example: +Here is a partial example from zzdummy.py: -class ZoomHeight: +class ZzDummy: menudefs = [ - ('edit', [ - None, # Separator - ('_Zoom Height', '<>'), - ]) + ('format', [ + ('Z in', '<>'), + ('Z out', '<>'), + ] ) ] def __init__(self, editwin): self.editwin = editwin - def zoom_height_event(self, event): + def z_in_event(self, event=None): "...Do what you want here..." The final piece of the puzzle is the file "config-extensions.def", which is diff --git a/Lib/idlelib/idle_test/test_zzdummy.py b/Lib/idlelib/idle_test/test_zzdummy.py new file mode 100644 index 00000000000000..1013cdc3c46f4f --- /dev/null +++ b/Lib/idlelib/idle_test/test_zzdummy.py @@ -0,0 +1,152 @@ +"Test zzdummy, coverage 100%." + +from idlelib import zzdummy +import unittest +from test.support import requires +from tkinter import Tk, Text +from unittest import mock +from idlelib import config +from idlelib import editor +from idlelib import format + + +usercfg = zzdummy.idleConf.userCfg +testcfg = { + 'main': config.IdleUserConfParser(''), + 'highlight': config.IdleUserConfParser(''), + 'keys': config.IdleUserConfParser(''), + 'extensions': config.IdleUserConfParser(''), +} +code_sample = """\ + +class C1(): + # Class comment. + def __init__(self, a, b): + self.a = a + self.b = b +""" + + +class DummyEditwin: + get_selection_indices = editor.EditorWindow.get_selection_indices + def __init__(self, root, text): + self.root = root + self.top = root + self.text = text + self.fregion = format.FormatRegion(self) + self.text.undo_block_start = mock.Mock() + self.text.undo_block_stop = mock.Mock() + + +class ZZDummyTest(unittest.TestCase): + + @classmethod + def setUpClass(cls): + requires('gui') + root = cls.root = Tk() + root.withdraw() + text = cls.text = Text(cls.root) + cls.editor = DummyEditwin(root, text) + zzdummy.idleConf.userCfg = testcfg + + @classmethod + def tearDownClass(cls): + zzdummy.idleConf.userCfg = usercfg + del cls.editor, cls.text + cls.root.update_idletasks() + for id in cls.root.tk.call('after', 'info'): + cls.root.after_cancel(id) # Need for EditorWindow. + cls.root.destroy() + del cls.root + + def setUp(self): + text = self.text + text.insert('1.0', code_sample) + text.undo_block_start.reset_mock() + text.undo_block_stop.reset_mock() + zz = self.zz = zzdummy.ZzDummy(self.editor) + zzdummy.ZzDummy.ztext = '# ignore #' + + def tearDown(self): + self.text.delete('1.0', 'end') + del self.zz + + def checklines(self, text, value): + # Verify that there are lines being checked. + end_line = int(float(text.index('end'))) + + # Check each line for the starting text. + actual = [] + for line in range(1, end_line): + txt = text.get(f'{line}.0', f'{line}.end') + actual.append(txt.startswith(value)) + return actual + + def test_init(self): + zz = self.zz + self.assertEqual(zz.editwin, self.editor) + self.assertEqual(zz.text, self.editor.text) + + def test_reload(self): + self.assertEqual(self.zz.ztext, '# ignore #') + testcfg['extensions'].SetOption('ZzDummy', 'z-text', 'spam') + zzdummy.ZzDummy.reload() + self.assertEqual(self.zz.ztext, 'spam') + + def test_z_in_event(self): + eq = self.assertEqual + zz = self.zz + text = zz.text + eq(self.zz.ztext, '# ignore #') + + # No lines have the leading text. + expected = [False, False, False, False, False, False, False] + actual = self.checklines(text, zz.ztext) + eq(expected, actual) + + text.tag_add('sel', '2.0', '4.end') + eq(zz.z_in_event(), 'break') + expected = [False, True, True, True, False, False, False] + actual = self.checklines(text, zz.ztext) + eq(expected, actual) + + text.undo_block_start.assert_called_once() + text.undo_block_stop.assert_called_once() + + def test_z_out_event(self): + eq = self.assertEqual + zz = self.zz + text = zz.text + eq(self.zz.ztext, '# ignore #') + + # Prepend text. + text.tag_add('sel', '2.0', '5.end') + zz.z_in_event() + text.undo_block_start.reset_mock() + text.undo_block_stop.reset_mock() + + # Select a few lines to remove text. + text.tag_remove('sel', '1.0', 'end') + text.tag_add('sel', '3.0', '4.end') + eq(zz.z_out_event(), 'break') + expected = [False, True, False, False, True, False, False] + actual = self.checklines(text, zz.ztext) + eq(expected, actual) + + text.undo_block_start.assert_called_once() + text.undo_block_stop.assert_called_once() + + def test_roundtrip(self): + # Insert and remove to all code should give back original text. + zz = self.zz + text = zz.text + + text.tag_add('sel', '1.0', 'end-1c') + zz.z_in_event() + zz.z_out_event() + + self.assertEqual(text.get('1.0', 'end-1c'), code_sample) + + +if __name__ == '__main__': + unittest.main(verbosity=2) diff --git a/Lib/idlelib/zzdummy.py b/Lib/idlelib/zzdummy.py index 8084499646653d..1247e8f1cc0528 100644 --- a/Lib/idlelib/zzdummy.py +++ b/Lib/idlelib/zzdummy.py @@ -1,42 +1,73 @@ -"Example extension, also used for testing." +"""Example extension, also used for testing. + +See extend.txt for more details on creating an extension. +See config-extension.def for configuring an extension. +""" from idlelib.config import idleConf +from functools import wraps + + +def format_selection(format_line): + "Apply a formatting function to all of the selected lines." + + @wraps(format_line) + def apply(self, event=None): + head, tail, chars, lines = self.formatter.get_region() + for pos in range(len(lines) - 1): + line = lines[pos] + lines[pos] = format_line(self, line) + self.formatter.set_region(head, tail, chars, lines) + return 'break' -ztext = idleConf.GetOption('extensions', 'ZzDummy', 'z-text') + return apply class ZzDummy: + """Prepend or remove initial text from selected lines.""" -## menudefs = [ -## ('format', [ -## ('Z in', '<>'), -## ('Z out', '<>'), -## ] ) -## ] + # Extend the format menu. + menudefs = [ + ('format', [ + ('Z in', '<>'), + ('Z out', '<>'), + ] ) + ] def __init__(self, editwin): + "Initialize the settings for this extension." + self.editwin = editwin self.text = editwin.text - z_in = False + self.formatter = editwin.fregion @classmethod def reload(cls): + "Load class variables from config." cls.ztext = idleConf.GetOption('extensions', 'ZzDummy', 'z-text') - def z_in_event(self, event): + @format_selection + def z_in_event(self, line): + """Insert text at the beginning of each selected line. + + This is bound to the <> virtual event when the extensions + are loaded. """ + return f'{self.ztext}{line}' + + @format_selection + def z_out_event(self, line): + """Remove specific text from the beginning of each selected line. + + This is bound to the <> virtual event when the extensions + are loaded. """ - text = self.text - text.undo_block_start() - for line in range(1, text.index('end')): - text.insert('%d.0', ztest) - text.undo_block_stop() - return "break" + zlength = 0 if not line.startswith(self.ztext) else len(self.ztext) + return line[zlength:] - def z_out_event(self, event): pass ZzDummy.reload() -##if __name__ == "__main__": -## import unittest -## unittest.main('idlelib.idle_test.test_zzdummy', -## verbosity=2, exit=False) + +if __name__ == "__main__": + import unittest + unittest.main('idlelib.idle_test.test_zzdummy', verbosity=2, exit=False) diff --git a/Misc/NEWS.d/next/IDLE/2019-06-30-20-31-09.bpo-32631.e7_4BG.rst b/Misc/NEWS.d/next/IDLE/2019-06-30-20-31-09.bpo-32631.e7_4BG.rst new file mode 100644 index 00000000000000..c422f43b6d6dd8 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2019-06-30-20-31-09.bpo-32631.e7_4BG.rst @@ -0,0 +1,2 @@ +Finish zzdummy example extension module: make menu entries work; +add docstrings and tests with 100% coverage. From e2c847e0c518dd6c63a898b8265ba54b71548bd6 Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Tue, 5 Jan 2021 16:04:25 +0200 Subject: [PATCH 1934/2163] [3.8] [3.9] bpo-42681: Fix test_curses failures related to color pairs (GH-24089) (GH-24113) (GH-24116) On ncurses 6.1 pair numbers are limited by SHORT_MAX-1. Improve error reporting and tests for color functions. (cherry picked from commit 59f9b4e4509be67494f3d45489fa55523175ff69) (cherry picked from commit 9b3a53a8264d4c469a3f3d8c037e74c010be3e5c) --- Lib/test/test_curses.py | 33 ++++++++++++++++++++++++++------- Modules/_cursesmodule.c | 26 ++++++++++++++++++-------- 2 files changed, 44 insertions(+), 15 deletions(-) diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py index e39161fde311d0..8991134e080635 100644 --- a/Lib/test/test_curses.py +++ b/Lib/test/test_curses.py @@ -319,11 +319,20 @@ def bad_colors(self): def bad_pairs(self): return (-2**31 - 1, 2**31, -2**63 - 1, 2**63, 2**64) + def test_start_color(self): + if not curses.has_colors(): + self.skipTest('requires colors support') + curses.start_color() + if verbose: + print(f'COLORS = {curses.COLORS}', file=sys.stderr) + print(f'COLOR_PAIRS = {curses.COLOR_PAIRS}', file=sys.stderr) + @requires_colors def test_color_content(self): self.assertEqual(curses.color_content(curses.COLOR_BLACK), (0, 0, 0)) curses.color_content(0) - curses.color_content(min(curses.COLORS - 1, SHORT_MAX)) + maxcolor = min(curses.COLORS - 1, SHORT_MAX) + curses.color_content(maxcolor) for color in self.bad_colors(): self.assertRaises(OverflowError, curses.color_content, color) @@ -364,13 +373,18 @@ def test_init_color(self): self.assertRaises(curses.error, curses.init_color, 0, 0, comp, 0) self.assertRaises(curses.error, curses.init_color, 0, 0, 0, comp) + def get_pair_limit(self): + return min(curses.COLOR_PAIRS, SHORT_MAX) + @requires_colors def test_pair_content(self): if not hasattr(curses, 'use_default_colors'): self.assertEqual(curses.pair_content(0), (curses.COLOR_WHITE, curses.COLOR_BLACK)) curses.pair_content(0) - curses.pair_content(min(curses.COLOR_PAIRS - 1, SHORT_MAX)) + maxpair = self.get_pair_limit() - 1 + if maxpair > 0: + curses.pair_content(maxpair) for pair in self.bad_pairs(): self.assertRaises(OverflowError, curses.pair_content, pair) @@ -385,11 +399,14 @@ def test_init_pair(self): curses.init_pair(1, 0, 0) self.assertEqual(curses.pair_content(1), (0, 0)) maxcolor = min(curses.COLORS - 1, SHORT_MAX) - curses.init_pair(1, maxcolor, maxcolor) - self.assertEqual(curses.pair_content(1), (maxcolor, maxcolor)) - maxpair = min(curses.COLOR_PAIRS - 1, SHORT_MAX) - curses.init_pair(maxpair, 2, 3) - self.assertEqual(curses.pair_content(maxpair), (2, 3)) + curses.init_pair(1, maxcolor, 0) + self.assertEqual(curses.pair_content(1), (maxcolor, 0)) + curses.init_pair(1, 0, maxcolor) + self.assertEqual(curses.pair_content(1), (0, maxcolor)) + maxpair = self.get_pair_limit() - 1 + if maxpair > 1: + curses.init_pair(maxpair, 0, 0) + self.assertEqual(curses.pair_content(maxpair), (0, 0)) for pair in self.bad_pairs(): self.assertRaises(OverflowError, curses.init_pair, pair, 0, 0) @@ -587,6 +604,8 @@ def test_update_lines_cols(self): @requires_curses_func('ncurses_version') def test_ncurses_version(self): v = curses.ncurses_version + if verbose: + print(f'ncurses_version = {curses.ncurses_version}', flush=True) self.assertIsInstance(v[:], tuple) self.assertEqual(len(v), 3) self.assertIsInstance(v[0], int) diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c index 47459d65ff6e49..0f35cdd2861217 100644 --- a/Modules/_cursesmodule.c +++ b/Modules/_cursesmodule.c @@ -2592,13 +2592,18 @@ _curses_color_content_impl(PyObject *module, short color_number) PyCursesInitialised; PyCursesInitialisedColor; - if (color_content(color_number, &r, &g, &b) != ERR) - return Py_BuildValue("(iii)", r, g, b); - else { - PyErr_SetString(PyCursesError, - "Argument 1 was out of range. Check value of COLORS."); + if (color_content(color_number, &r, &g, &b) == ERR) { + if (color_number >= COLORS) { + PyErr_SetString(PyCursesError, + "Argument 1 was out of range. Check value of COLORS."); + } + else { + PyErr_SetString(PyCursesError, "color_content() returned ERR"); + } return NULL; } + + return Py_BuildValue("(iii)", r, g, b); } /*[clinic input] @@ -3617,9 +3622,14 @@ _curses_pair_content_impl(PyObject *module, short pair_number) PyCursesInitialised; PyCursesInitialisedColor; - if (pair_content(pair_number, &f, &b)==ERR) { - PyErr_SetString(PyCursesError, - "Argument 1 was out of range. (0..COLOR_PAIRS-1)"); + if (pair_content(pair_number, &f, &b) == ERR) { + if (pair_number >= COLOR_PAIRS) { + PyErr_SetString(PyCursesError, + "Argument 1 was out of range. (0..COLOR_PAIRS-1)"); + } + else { + PyErr_SetString(PyCursesError, "pair_content() returned ERR"); + } return NULL; } From f27451dc35f1c69be0c26e8aeafee7ba6771039a Mon Sep 17 00:00:00 2001 From: Erlend Egeberg Aasland Date: Tue, 5 Jan 2021 23:46:31 +0100 Subject: [PATCH 1935/2163] [3.8] bpo-42584: Update macOS installer to use SQLite 3.34.0 (GH-23674). (GH-24129) (cherry picked from commit c94ee13ad596d26d1859078bc09806aa59bb0000) Co-authored-by: Erlend Egeberg Aasland Automerge-Triggered-By: GH:ned-deily --- Mac/BuildScript/build-installer.py | 6 +++--- .../next/macOS/2020-12-07-11-37-35.bpo-42584.LygmqQ.rst | 1 + 2 files changed, 4 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/macOS/2020-12-07-11-37-35.bpo-42584.LygmqQ.rst diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index 7ef3753ad57575..d25c86bf7b453b 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -307,9 +307,9 @@ def library_recipes(): ), ), dict( - name="SQLite 3.33.0", - url="https://sqlite.org/2020/sqlite-autoconf-3330000.tar.gz", - checksum='842a8a100d7b01b09e543deb2b7951dd', + name="SQLite 3.34.0", + url="https://sqlite.org/2020/sqlite-autoconf-3340000.tar.gz", + checksum='7f33c9db7b713957fcb9271fe9049fef', extra_cflags=('-Os ' '-DSQLITE_ENABLE_FTS5 ' '-DSQLITE_ENABLE_FTS4 ' diff --git a/Misc/NEWS.d/next/macOS/2020-12-07-11-37-35.bpo-42584.LygmqQ.rst b/Misc/NEWS.d/next/macOS/2020-12-07-11-37-35.bpo-42584.LygmqQ.rst new file mode 100644 index 00000000000000..2a625f98e90781 --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2020-12-07-11-37-35.bpo-42584.LygmqQ.rst @@ -0,0 +1 @@ +Update macOS installer to use SQLite 3.34.0. From 86b1207dbb9201d1259d1ec7603e720e29ba9042 Mon Sep 17 00:00:00 2001 From: Steve Dower Date: Wed, 6 Jan 2021 00:07:51 +0000 Subject: [PATCH 1936/2163] bpo-41837: Updated Windows installer to include OpenSSL 1.1.1i (GH-24125) --- .../next/Windows/2021-01-05-20-36-40.bpo-41837.bmS7vB.rst | 1 + PCbuild/get_externals.bat | 4 ++-- PCbuild/python.props | 4 ++-- PCbuild/readme.txt | 2 +- 4 files changed, 6 insertions(+), 5 deletions(-) create mode 100644 Misc/NEWS.d/next/Windows/2021-01-05-20-36-40.bpo-41837.bmS7vB.rst diff --git a/Misc/NEWS.d/next/Windows/2021-01-05-20-36-40.bpo-41837.bmS7vB.rst b/Misc/NEWS.d/next/Windows/2021-01-05-20-36-40.bpo-41837.bmS7vB.rst new file mode 100644 index 00000000000000..8d4bb34ff909c2 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2021-01-05-20-36-40.bpo-41837.bmS7vB.rst @@ -0,0 +1 @@ +Updated Windows installer to include OpenSSL 1.1.1i diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index 24d430a004d3ab..a15277b30b2193 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -53,7 +53,7 @@ echo.Fetching external libraries... set libraries= set libraries=%libraries% bzip2-1.0.6 if NOT "%IncludeLibffiSrc%"=="false" set libraries=%libraries% libffi-3.3.0-rc0-r1 -if NOT "%IncludeSSLSrc%"=="false" set libraries=%libraries% openssl-1.1.1g +if NOT "%IncludeSSLSrc%"=="false" set libraries=%libraries% openssl-1.1.1i set libraries=%libraries% sqlite-3.33.0.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tcl-core-8.6.9.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tk-8.6.9.0 @@ -77,7 +77,7 @@ echo.Fetching external binaries... set binaries= if NOT "%IncludeLibffi%"=="false" set binaries=%binaries% libffi -if NOT "%IncludeSSL%"=="false" set binaries=%binaries% openssl-bin-1.1.1g +if NOT "%IncludeSSL%"=="false" set binaries=%binaries% openssl-bin-1.1.1i if NOT "%IncludeTkinter%"=="false" set binaries=%binaries% tcltk-8.6.9.0 if NOT "%IncludeSSLSrc%"=="false" set binaries=%binaries% nasm-2.11.06 diff --git a/PCbuild/python.props b/PCbuild/python.props index acc41a2c017684..769dabc5641ca9 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -62,8 +62,8 @@ $(ExternalsDir)libffi\ $(ExternalsDir)libffi\$(ArchName)\ $(libffiOutDir)include - $(ExternalsDir)openssl-1.1.1g\ - $(ExternalsDir)openssl-bin-1.1.1g\$(ArchName)\ + $(ExternalsDir)openssl-1.1.1i\ + $(ExternalsDir)openssl-bin-1.1.1i\$(ArchName)\ $(opensslOutDir)include $(ExternalsDir)\nasm-2.11.06\ $(ExternalsDir)\zlib-1.2.11\ diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt index 6e304eae553cd8..dc247c8bcd7064 100644 --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -165,7 +165,7 @@ _lzma Homepage: http://tukaani.org/xz/ _ssl - Python wrapper for version 1.1.1c of the OpenSSL secure sockets + Python wrapper for version 1.1.1i of the OpenSSL secure sockets library, which is downloaded from our binaries repository at https://github.com/python/cpython-bin-deps. From 03a079f7cf8eacc29ea616060916e8a63ba6e5ba Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 5 Jan 2021 16:21:57 -0800 Subject: [PATCH 1937/2163] bpo-38413: Remove outdated section about multithreading in sqlite3 (GH-23159) (cherry picked from commit f9949f82e17c88609adb53eff3a7d5cd63a645bd) Co-authored-by: Vladimir --- Doc/library/sqlite3.rst | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index e493324495890e..9459f9175827b0 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -1093,19 +1093,6 @@ committed: .. literalinclude:: ../includes/sqlite3/ctx_manager.py -Common issues -------------- - -Multithreading -^^^^^^^^^^^^^^ - -Older SQLite versions had issues with sharing connections between threads. -That's why the Python module disallows sharing connections and cursors between -threads. If you still try to do so, you will get an exception at runtime. - -The only exception is calling the :meth:`~Connection.interrupt` method, which -only makes sense to call from a different thread. - .. rubric:: Footnotes .. [#f1] The sqlite3 module is not built with loadable extension support by From 28611c28474a7fafdcf7ea2712f2670dda940eef Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 5 Jan 2021 16:23:37 -0800 Subject: [PATCH 1938/2163] bpo-42755: Fix sqlite3.Connection.backup docs (GH-23965) The `pages` argument default value now reflects the implementation. (cherry picked from commit abba83b4b91f78dc556dc0b7700ecb46cba22c01) Co-authored-by: Erlend Egeberg Aasland --- Doc/library/sqlite3.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 9459f9175827b0..e2e7312c277357 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -545,7 +545,7 @@ Connection Objects con.close() - .. method:: backup(target, *, pages=0, progress=None, name="main", sleep=0.250) + .. method:: backup(target, *, pages=-1, progress=None, name="main", sleep=0.250) This method makes a backup of a SQLite database even while it's being accessed by other clients, or concurrently by the same connection. The copy will be From db91714faa6d3ede59003cdcc719a758160f3970 Mon Sep 17 00:00:00 2001 From: Steve Dower Date: Wed, 6 Jan 2021 00:31:10 +0000 Subject: [PATCH 1939/2163] bpo-42584: Update Windows installer to use SQLite 3.34.0 (GH-23675) Co-authored-by: Erlend Egeberg Aasland --- .../next/Windows/2020-12-07-11-40-52.bpo-42584.AsYnVX.rst | 1 + PCbuild/get_externals.bat | 2 +- PCbuild/python.props | 2 +- PCbuild/readme.txt | 2 +- 4 files changed, 4 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Windows/2020-12-07-11-40-52.bpo-42584.AsYnVX.rst diff --git a/Misc/NEWS.d/next/Windows/2020-12-07-11-40-52.bpo-42584.AsYnVX.rst b/Misc/NEWS.d/next/Windows/2020-12-07-11-40-52.bpo-42584.AsYnVX.rst new file mode 100644 index 00000000000000..afb6530c8f66d7 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-12-07-11-40-52.bpo-42584.AsYnVX.rst @@ -0,0 +1 @@ +Upgrade Windows installer to use SQLite 3.34.0. diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index a15277b30b2193..a1d9a12a362e12 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -54,7 +54,7 @@ set libraries= set libraries=%libraries% bzip2-1.0.6 if NOT "%IncludeLibffiSrc%"=="false" set libraries=%libraries% libffi-3.3.0-rc0-r1 if NOT "%IncludeSSLSrc%"=="false" set libraries=%libraries% openssl-1.1.1i -set libraries=%libraries% sqlite-3.33.0.0 +set libraries=%libraries% sqlite-3.34.0.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tcl-core-8.6.9.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tk-8.6.9.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tix-8.4.3.6 diff --git a/PCbuild/python.props b/PCbuild/python.props index 769dabc5641ca9..3fa774816a7530 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -56,7 +56,7 @@ $(EXTERNALS_DIR) $([System.IO.Path]::GetFullPath(`$(PySourcePath)externals`)) $(ExternalsDir)\ - $(ExternalsDir)sqlite-3.33.0.0\ + $(ExternalsDir)sqlite-3.34.0.0\ $(ExternalsDir)bzip2-1.0.6\ $(ExternalsDir)xz-5.2.2\ $(ExternalsDir)libffi\ diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt index dc247c8bcd7064..0b2aa59ecefabf 100644 --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -184,7 +184,7 @@ _ssl again when building. _sqlite3 - Wraps SQLite 3.33.0, which is itself built by sqlite3.vcxproj + Wraps SQLite 3.34.0, which is itself built by sqlite3.vcxproj Homepage: http://www.sqlite.org/ _tkinter From ca8e96d1edbeb536f58da91e607082463398fce1 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 7 Jan 2021 10:07:48 -0800 Subject: [PATCH 1940/2163] bpo-42811: Update importlib.utils.resolve_name() docs to use __spec__.parent (GH-24100) (GH-24144) Automerge-Triggered-By: GH:brettcannon (cherry picked from commit ff8458b918050168acda1ad6d079f52b8effa821) Co-authored-by: Yair Frid --- Doc/library/importlib.rst | 2 +- .../next/Documentation/2021-01-04-22-14-22.bpo-42811.HY2beA.rst | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Documentation/2021-01-04-22-14-22.bpo-42811.HY2beA.rst diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index 8635e2bc66f3a8..304da984ace6b6 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -1439,7 +1439,7 @@ an :term:`importer`. If **name** has no leading dots, then **name** is simply returned. This allows for usage such as - ``importlib.util.resolve_name('sys', __package__)`` without doing a + ``importlib.util.resolve_name('sys', __spec__.parent)`` without doing a check to see if the **package** argument is needed. :exc:`ValueError` is raised if **name** is a relative module name but diff --git a/Misc/NEWS.d/next/Documentation/2021-01-04-22-14-22.bpo-42811.HY2beA.rst b/Misc/NEWS.d/next/Documentation/2021-01-04-22-14-22.bpo-42811.HY2beA.rst new file mode 100644 index 00000000000000..768508e0ce1c19 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-01-04-22-14-22.bpo-42811.HY2beA.rst @@ -0,0 +1,2 @@ +Updated importlib.utils.resolve_name() doc to use __spec__.parent +instead of __package__. (Thanks Yair Frid.) From 5ded7efa6a7a232dd4a41e6e65e4dae47146514b Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 9 Jan 2021 23:30:59 -0800 Subject: [PATCH 1941/2163] bpo-33065: Fix problem debugging user classes with __repr__ method (GH-24183) If __repr__ uses instance attributes, as normal, and one steps through the __init__ method, debugger may try to get repr before the instance attributes exist. reprlib.repr handles the error. (cherry picked from commit 81f87bbf9f65702062021a78abd9b8f82c98a414) Co-authored-by: Terry Jan Reedy --- Lib/idlelib/NEWS.txt | 2 ++ Lib/idlelib/debugger_r.py | 6 +++--- Lib/idlelib/idle_test/test_debugger_r.py | 14 ++++++++++++++ .../IDLE/2021-01-10-01-25-43.bpo-33065.zmyHYJ.rst | 1 + 4 files changed, 20 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/IDLE/2021-01-10-01-25-43.bpo-33065.zmyHYJ.rst diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index bc15501a7b062a..44782fcb0cecd9 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,8 @@ Released on 2020-12-?? ====================================== +bpo-33065: Fix problem debugging user classes with __repr__ method. + bpo-32631: Finish zzdummy example extension module: make menu entries work; add docstrings and tests with 100% coverage. diff --git a/Lib/idlelib/debugger_r.py b/Lib/idlelib/debugger_r.py index 9dcfc56414c050..26204438858d8a 100644 --- a/Lib/idlelib/debugger_r.py +++ b/Lib/idlelib/debugger_r.py @@ -19,7 +19,7 @@ barrier, in particular frame and traceback objects. """ - +import reprlib import types from idlelib import debugger @@ -170,7 +170,7 @@ def dict_keys_list(self, did): def dict_item(self, did, key): dict = dicttable[did] value = dict[key] - value = repr(value) ### can't pickle module 'builtins' + value = reprlib.repr(value) ### can't pickle module 'builtins' return value #----------end class IdbAdapter---------- @@ -390,4 +390,4 @@ def restart_subprocess_debugger(rpcclt): if __name__ == "__main__": from unittest import main - main('idlelib.idle_test.test_debugger', verbosity=2, exit=False) + main('idlelib.idle_test.test_debugger_r', verbosity=2, exit=False) diff --git a/Lib/idlelib/idle_test/test_debugger_r.py b/Lib/idlelib/idle_test/test_debugger_r.py index 199f63447ce6ca..638ebd36a7405d 100644 --- a/Lib/idlelib/idle_test/test_debugger_r.py +++ b/Lib/idlelib/idle_test/test_debugger_r.py @@ -25,5 +25,19 @@ def test_init(self): # Classes GUIProxy, IdbAdapter, FrameProxy, CodeProxy, DictProxy, # GUIAdapter, IdbProxy plus 7 module functions. +class IdbAdapterTest(unittest.TestCase): + + def test_dict_item_noattr(self): # Issue 33065. + + class BinData: + def __repr__(self): + return self.length + + debugger_r.dicttable[0] = {'BinData': BinData()} + idb = debugger_r.IdbAdapter(None) + self.assertTrue(idb.dict_item(0, 'BinData')) + debugger_r.dicttable.clear() + + if __name__ == '__main__': unittest.main(verbosity=2) diff --git a/Misc/NEWS.d/next/IDLE/2021-01-10-01-25-43.bpo-33065.zmyHYJ.rst b/Misc/NEWS.d/next/IDLE/2021-01-10-01-25-43.bpo-33065.zmyHYJ.rst new file mode 100644 index 00000000000000..87948f3cd1baa1 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2021-01-10-01-25-43.bpo-33065.zmyHYJ.rst @@ -0,0 +1 @@ +Fix problem debugging user classes with __repr__ method. From 187f76def8a5bd0af7ab512575cad30cfe624b05 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 12 Jan 2021 15:45:05 +0100 Subject: [PATCH 1942/2163] [3.8] bpo-40052: Fix alignment issue in PyVectorcall_Function() (GH-23999) (GH-24120) Co-Authored-By: Andreas Schneider Co-Authored-By: Antoine Pitrou . Co-authored-by: Petr Viktorin (cherry picked from commit 056c08211b402b4dbc1530a9de9d00ad5309909f) https://bugs.python.org/issue40052 --- Include/cpython/abstract.h | 6 +++--- .../next/C API/2020-03-24-09-27-10.bpo-40052.27P2KG.rst | 2 ++ Objects/call.c | 3 ++- 3 files changed, 7 insertions(+), 4 deletions(-) create mode 100644 Misc/NEWS.d/next/C API/2020-03-24-09-27-10.bpo-40052.27P2KG.rst diff --git a/Include/cpython/abstract.h b/Include/cpython/abstract.h index 2ea3209bca1098..dbfce2dc9061db 100644 --- a/Include/cpython/abstract.h +++ b/Include/cpython/abstract.h @@ -82,14 +82,14 @@ _PyVectorcall_Function(PyObject *callable) { PyTypeObject *tp = Py_TYPE(callable); Py_ssize_t offset = tp->tp_vectorcall_offset; - vectorcallfunc *ptr; + vectorcallfunc ptr; if (!PyType_HasFeature(tp, _Py_TPFLAGS_HAVE_VECTORCALL)) { return NULL; } assert(PyCallable_Check(callable)); assert(offset > 0); - ptr = (vectorcallfunc*)(((char *)callable) + offset); - return *ptr; + memcpy(&ptr, (char *) callable + offset, sizeof(ptr)); + return ptr; } /* Call the callable object 'callable' with the "vectorcall" calling diff --git a/Misc/NEWS.d/next/C API/2020-03-24-09-27-10.bpo-40052.27P2KG.rst b/Misc/NEWS.d/next/C API/2020-03-24-09-27-10.bpo-40052.27P2KG.rst new file mode 100644 index 00000000000000..538488e2fbaccd --- /dev/null +++ b/Misc/NEWS.d/next/C API/2020-03-24-09-27-10.bpo-40052.27P2KG.rst @@ -0,0 +1,2 @@ +Fix an alignment build warning/error in function ``PyVectorcall_Function()``. +Patch by Andreas Schneider, Antoine Pitrou and Petr Viktorin. diff --git a/Objects/call.c b/Objects/call.c index c66389854d8bf0..9672be01ed0590 100644 --- a/Objects/call.c +++ b/Objects/call.c @@ -175,13 +175,14 @@ PyVectorcall_Call(PyObject *callable, PyObject *tuple, PyObject *kwargs) { /* get vectorcallfunc as in _PyVectorcall_Function, but without * the _Py_TPFLAGS_HAVE_VECTORCALL check */ + vectorcallfunc func; Py_ssize_t offset = Py_TYPE(callable)->tp_vectorcall_offset; if (offset <= 0) { PyErr_Format(PyExc_TypeError, "'%.200s' object does not support vectorcall", Py_TYPE(callable)->tp_name); return NULL; } - vectorcallfunc func = *(vectorcallfunc *)(((char *)callable) + offset); + memcpy(&func, (char *) callable + offset, sizeof(func)); if (func == NULL) { PyErr_Format(PyExc_TypeError, "'%.200s' object does not support vectorcall", Py_TYPE(callable)->tp_name); From f08c66467d56c71f75c6659ede9177449fe9d2e6 Mon Sep 17 00:00:00 2001 From: William Schwartz Date: Sat, 16 Jan 2021 11:22:21 -0600 Subject: [PATCH 1943/2163] [3.8] bpo-42531: Teach importlib.resources.path to handle packages without __file__ (GH-23611) Fixes [bpo-42531]() for Python 3.8. The issue also applies to 3.7. If this PR looks like it'll be accepted, I can cherry-pick it to the 3.7 branch and submit a follow-up PR. Automerge-Triggered-By: GH:jaraco --- Lib/importlib/resources.py | 8 +++++--- Lib/test/test_importlib/test_path.py | 12 ++++++++++++ .../Library/2020-12-02-16-28-04.bpo-42531.2sLlFW.rst | 1 + 3 files changed, 18 insertions(+), 3 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-12-02-16-28-04.bpo-42531.2sLlFW.rst diff --git a/Lib/importlib/resources.py b/Lib/importlib/resources.py index fc3a1c9cabe636..8d37d52cb8d37e 100644 --- a/Lib/importlib/resources.py +++ b/Lib/importlib/resources.py @@ -193,9 +193,11 @@ def path(package: Package, resource: Resource) -> Iterator[Path]: _check_location(package) # Fall-through for both the lack of resource_path() *and* if # resource_path() raises FileNotFoundError. - package_directory = Path(package.__spec__.origin).parent - file_path = package_directory / resource - if file_path.exists(): + file_path = None + if package.__spec__.origin is not None: + package_directory = Path(package.__spec__.origin).parent + file_path = package_directory / resource + if file_path is not None and file_path.exists(): yield file_path else: with open_binary(package, resource) as fp: diff --git a/Lib/test/test_importlib/test_path.py b/Lib/test/test_importlib/test_path.py index 2d3dcda7ed2e79..5c5a8e3d76e79e 100644 --- a/Lib/test/test_importlib/test_path.py +++ b/Lib/test/test_importlib/test_path.py @@ -1,3 +1,4 @@ +import io import unittest from importlib import resources @@ -27,6 +28,17 @@ class PathDiskTests(PathTests, unittest.TestCase): data = data01 +class PathMemoryTests(PathTests, unittest.TestCase): + def setUp(self): + file = io.BytesIO(b'Hello, UTF-8 world!\n') + self.addCleanup(file.close) + self.data = util.create_package( + file=file, path=FileNotFoundError("package exists only in memory") + ) + self.data.__spec__.origin = None + self.data.__spec__.has_location = False + + class PathZipTests(PathTests, util.ZipSetup, unittest.TestCase): def test_remove_in_context_manager(self): # It is not an error if the file that was temporarily stashed on the diff --git a/Misc/NEWS.d/next/Library/2020-12-02-16-28-04.bpo-42531.2sLlFW.rst b/Misc/NEWS.d/next/Library/2020-12-02-16-28-04.bpo-42531.2sLlFW.rst new file mode 100644 index 00000000000000..7927078acda430 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-12-02-16-28-04.bpo-42531.2sLlFW.rst @@ -0,0 +1 @@ +:func:`importlib.resources.path` now works for :term:`package`\ s missing the optional :attr:`__file__` attribute (more specifically, packages whose :attr:`__spec__`\ ``.``\ :attr:`~importlib.machinery.ModuleSpec.origin` :keyword:`is` :data:`None`). \ No newline at end of file From ece5dfd403dac211f8d3c72701fe7ba7b7aa5b5f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 18 Jan 2021 13:28:52 -0800 Subject: [PATCH 1944/2163] closes bpo-42938: Replace snprintf with Python unicode formatting in ctypes param reprs. (GH-24248) (cherry picked from commit 916610ef90a0d0761f08747f7b0905541f0977c7) Co-authored-by: Benjamin Peterson Co-authored-by: Benjamin Peterson --- Lib/ctypes/test/test_parameters.py | 43 ++++++++++++++++ .../2021-01-18-09-27-31.bpo-42938.4Zn4Mp.rst | 2 + Modules/_ctypes/callproc.c | 51 +++++++------------ 3 files changed, 64 insertions(+), 32 deletions(-) create mode 100644 Misc/NEWS.d/next/Security/2021-01-18-09-27-31.bpo-42938.4Zn4Mp.rst diff --git a/Lib/ctypes/test/test_parameters.py b/Lib/ctypes/test/test_parameters.py index e4c25fd880cefb..531894fdec8382 100644 --- a/Lib/ctypes/test/test_parameters.py +++ b/Lib/ctypes/test/test_parameters.py @@ -201,6 +201,49 @@ def __dict__(self): with self.assertRaises(ZeroDivisionError): WorseStruct().__setstate__({}, b'foo') + def test_parameter_repr(self): + from ctypes import ( + c_bool, + c_char, + c_wchar, + c_byte, + c_ubyte, + c_short, + c_ushort, + c_int, + c_uint, + c_long, + c_ulong, + c_longlong, + c_ulonglong, + c_float, + c_double, + c_longdouble, + c_char_p, + c_wchar_p, + c_void_p, + ) + self.assertRegex(repr(c_bool.from_param(True)), r"^$") + self.assertEqual(repr(c_char.from_param(97)), "") + self.assertRegex(repr(c_wchar.from_param('a')), r"^$") + self.assertEqual(repr(c_byte.from_param(98)), "") + self.assertEqual(repr(c_ubyte.from_param(98)), "") + self.assertEqual(repr(c_short.from_param(511)), "") + self.assertEqual(repr(c_ushort.from_param(511)), "") + self.assertRegex(repr(c_int.from_param(20000)), r"^$") + self.assertRegex(repr(c_uint.from_param(20000)), r"^$") + self.assertRegex(repr(c_long.from_param(20000)), r"^$") + self.assertRegex(repr(c_ulong.from_param(20000)), r"^$") + self.assertRegex(repr(c_longlong.from_param(20000)), r"^$") + self.assertRegex(repr(c_ulonglong.from_param(20000)), r"^$") + self.assertEqual(repr(c_float.from_param(1.5)), "") + self.assertEqual(repr(c_double.from_param(1.5)), "") + self.assertEqual(repr(c_double.from_param(1e300)), "") + self.assertRegex(repr(c_longdouble.from_param(1.5)), r"^$") + self.assertRegex(repr(c_char_p.from_param(b'hihi')), "^$") + self.assertRegex(repr(c_wchar_p.from_param('hihi')), "^$") + self.assertRegex(repr(c_void_p.from_param(0x12)), r"^$") + ################################################################ if __name__ == '__main__': diff --git a/Misc/NEWS.d/next/Security/2021-01-18-09-27-31.bpo-42938.4Zn4Mp.rst b/Misc/NEWS.d/next/Security/2021-01-18-09-27-31.bpo-42938.4Zn4Mp.rst new file mode 100644 index 00000000000000..7df65a156feabd --- /dev/null +++ b/Misc/NEWS.d/next/Security/2021-01-18-09-27-31.bpo-42938.4Zn4Mp.rst @@ -0,0 +1,2 @@ +Avoid static buffers when computing the repr of :class:`ctypes.c_double` and +:class:`ctypes.c_longdouble` values. diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c index a9b8675cd951b5..de75918d49f370 100644 --- a/Modules/_ctypes/callproc.c +++ b/Modules/_ctypes/callproc.c @@ -484,58 +484,47 @@ is_literal_char(unsigned char c) static PyObject * PyCArg_repr(PyCArgObject *self) { - char buffer[256]; switch(self->tag) { case 'b': case 'B': - sprintf(buffer, "", + return PyUnicode_FromFormat("", self->tag, self->value.b); - break; case 'h': case 'H': - sprintf(buffer, "", + return PyUnicode_FromFormat("", self->tag, self->value.h); - break; case 'i': case 'I': - sprintf(buffer, "", + return PyUnicode_FromFormat("", self->tag, self->value.i); - break; case 'l': case 'L': - sprintf(buffer, "", + return PyUnicode_FromFormat("", self->tag, self->value.l); - break; case 'q': case 'Q': - sprintf(buffer, -#ifdef MS_WIN32 - "", -#else - "", -#endif + return PyUnicode_FromFormat("", self->tag, self->value.q); - break; case 'd': - sprintf(buffer, "", - self->tag, self->value.d); - break; - case 'f': - sprintf(buffer, "", - self->tag, self->value.f); - break; - + case 'f': { + PyObject *f = PyFloat_FromDouble((self->tag == 'f') ? self->value.f : self->value.d); + if (f == NULL) { + return NULL; + } + PyObject *result = PyUnicode_FromFormat("", self->tag, f); + Py_DECREF(f); + return result; + } case 'c': if (is_literal_char((unsigned char)self->value.c)) { - sprintf(buffer, "", + return PyUnicode_FromFormat("", self->tag, self->value.c); } else { - sprintf(buffer, "", + return PyUnicode_FromFormat("", self->tag, (unsigned char)self->value.c); } - break; /* Hm, are these 'z' and 'Z' codes useful at all? Shouldn't they be replaced by the functionality of c_string @@ -544,22 +533,20 @@ PyCArg_repr(PyCArgObject *self) case 'z': case 'Z': case 'P': - sprintf(buffer, "", + return PyUnicode_FromFormat("", self->tag, self->value.p); break; default: if (is_literal_char((unsigned char)self->tag)) { - sprintf(buffer, "", + return PyUnicode_FromFormat("", (unsigned char)self->tag, (void *)self); } else { - sprintf(buffer, "", + return PyUnicode_FromFormat("", (unsigned char)self->tag, (void *)self); } - break; } - return PyUnicode_FromString(buffer); } static PyMemberDef PyCArgType_members[] = { From 648b72900b5039ab46b8b459f921daecb8db2a6b Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 20 Jan 2021 01:18:07 -0800 Subject: [PATCH 1945/2163] bpo-42005: profile and cProfile catch BrokenPipeError (GH-22643) (cherry picked from commit 3554fa4abecfb77ac5fcaa5ce8310eeca5683960) Co-authored-by: Zhiming Wang --- Lib/cProfile.py | 7 ++++++- Lib/profile.py | 7 ++++++- .../next/Library/2020-10-11-13-48-03.bpo-42005.Jq6Az-.rst | 2 ++ 3 files changed, 14 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2020-10-11-13-48-03.bpo-42005.Jq6Az-.rst diff --git a/Lib/cProfile.py b/Lib/cProfile.py index 47aacf9e2d494e..406a9b7cf11be9 100755 --- a/Lib/cProfile.py +++ b/Lib/cProfile.py @@ -191,7 +191,12 @@ def main(): '__package__': None, '__cached__': None, } - runctx(code, globs, None, options.outfile, options.sort) + try: + runctx(code, globs, None, options.outfile, options.sort) + except BrokenPipeError as exc: + # Prevent "Exception ignored" during interpreter shutdown. + sys.stdout = None + sys.exit(exc.errno) else: parser.print_usage() return parser diff --git a/Lib/profile.py b/Lib/profile.py index 9df4435c5ae837..df4450dac6a1b6 100755 --- a/Lib/profile.py +++ b/Lib/profile.py @@ -611,7 +611,12 @@ def main(): '__package__': None, '__cached__': None, } - runctx(code, globs, None, options.outfile, options.sort) + try: + runctx(code, globs, None, options.outfile, options.sort) + except BrokenPipeError as exc: + # Prevent "Exception ignored" during interpreter shutdown. + sys.stdout = None + sys.exit(exc.errno) else: parser.print_usage() return parser diff --git a/Misc/NEWS.d/next/Library/2020-10-11-13-48-03.bpo-42005.Jq6Az-.rst b/Misc/NEWS.d/next/Library/2020-10-11-13-48-03.bpo-42005.Jq6Az-.rst new file mode 100644 index 00000000000000..be4ed7f55ffded --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-10-11-13-48-03.bpo-42005.Jq6Az-.rst @@ -0,0 +1,2 @@ +Fix CLI of :mod:`cProfile` and :mod:`profile` to catch +:exc:`BrokenPipeError`. From 8f334dbbf04582071d1318e5817e2fe99f1e5169 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 20 Jan 2021 02:43:22 -0800 Subject: [PATCH 1946/2163] bpo-36769: Document that fnmatch.filter supports any kind of iterable (GH-13039) (cherry picked from commit e8d22642105d57007ab1242848a8cbadc7f179df) Co-authored-by: Andre Delfino --- Doc/library/fnmatch.rst | 2 +- Lib/fnmatch.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/fnmatch.rst b/Doc/library/fnmatch.rst index ce07d326b395d8..925f08e914685e 100644 --- a/Doc/library/fnmatch.rst +++ b/Doc/library/fnmatch.rst @@ -75,7 +75,7 @@ patterns. .. function:: filter(names, pattern) - Return the subset of the list of *names* that match *pattern*. It is the same as + Construct a list from those elements of the iterable *names* that match *pattern*. It is the same as ``[n for n in names if fnmatch(n, pattern)]``, but implemented more efficiently. diff --git a/Lib/fnmatch.py b/Lib/fnmatch.py index b98e6413295e1c..7d52871a829a34 100644 --- a/Lib/fnmatch.py +++ b/Lib/fnmatch.py @@ -46,7 +46,7 @@ def _compile_pattern(pat): return re.compile(res).match def filter(names, pat): - """Return the subset of the list NAMES that match PAT.""" + """Construct a list from those elements of the iterable NAMES that match PAT.""" result = [] pat = os.path.normcase(pat) match = _compile_pattern(pat) From ebb2f26cd8b7c0b4919e994a8e62b6e709597939 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 20 Jan 2021 05:18:08 -0800 Subject: [PATCH 1947/2163] Fix typos in unittest documentation (GH-24194) * addCleanupClass -> addClassCleanup * doCleanupsClass -> doClassCleanups (cherry picked from commit e0e398ef1855f3db708c682f70adc412f0877d59) Co-authored-by: Conchylicultor --- Doc/library/unittest.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index a9e679f048472c..af2f46cfe8f6a9 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -1477,11 +1477,11 @@ Test cases after :meth:`setUpClass` if :meth:`setUpClass` raises an exception. It is responsible for calling all the cleanup functions added by - :meth:`addCleanupClass`. If you need cleanup functions to be called + :meth:`addClassCleanup`. If you need cleanup functions to be called *prior* to :meth:`tearDownClass` then you can call - :meth:`doCleanupsClass` yourself. + :meth:`doClassCleanups` yourself. - :meth:`doCleanupsClass` pops methods off the stack of cleanup + :meth:`doClassCleanups` pops methods off the stack of cleanup functions one at a time, so it can be called at any time. .. versionadded:: 3.8 From 844ec0ba6606b60a59b7da82c54c1e646424259c Mon Sep 17 00:00:00 2001 From: cptpcrd <31829097+cptpcrd@users.noreply.github.com> Date: Thu, 21 Jan 2021 05:46:53 -0500 Subject: [PATCH 1948/2163] bpo-42780: Fix set_inheritable() for O_PATH file descriptors on Linux (GH-24172) (GH-24277) (cherry picked from commit 7dc71c425cf6aa6a4070a418dce5d95ca435c79f) --- Lib/test/test_os.py | 27 +++++++++++++++++++ .../2021-01-08-15-49-20.bpo-42780.rtqi6B.rst | 1 + Python/fileutils.c | 7 +++++ 3 files changed, 35 insertions(+) create mode 100644 Misc/NEWS.d/next/Library/2021-01-08-15-49-20.bpo-42780.rtqi6B.rst diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py index 2a4ae1573ef599..5302b1ce575d4e 100644 --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -3513,6 +3513,33 @@ def test_set_inheritable_cloexec(self): self.assertEqual(fcntl.fcntl(fd, fcntl.F_GETFD) & fcntl.FD_CLOEXEC, 0) + @unittest.skipUnless(hasattr(os, 'O_PATH'), "need os.O_PATH") + def test_get_set_inheritable_o_path(self): + fd = os.open(__file__, os.O_PATH) + self.addCleanup(os.close, fd) + self.assertEqual(os.get_inheritable(fd), False) + + os.set_inheritable(fd, True) + self.assertEqual(os.get_inheritable(fd), True) + + os.set_inheritable(fd, False) + self.assertEqual(os.get_inheritable(fd), False) + + def test_get_set_inheritable_badf(self): + fd = support.make_bad_fd() + + with self.assertRaises(OSError) as ctx: + os.get_inheritable(fd) + self.assertEqual(ctx.exception.errno, errno.EBADF) + + with self.assertRaises(OSError) as ctx: + os.set_inheritable(fd, True) + self.assertEqual(ctx.exception.errno, errno.EBADF) + + with self.assertRaises(OSError) as ctx: + os.set_inheritable(fd, False) + self.assertEqual(ctx.exception.errno, errno.EBADF) + def test_open(self): fd = os.open(__file__, os.O_RDONLY) self.addCleanup(os.close, fd) diff --git a/Misc/NEWS.d/next/Library/2021-01-08-15-49-20.bpo-42780.rtqi6B.rst b/Misc/NEWS.d/next/Library/2021-01-08-15-49-20.bpo-42780.rtqi6B.rst new file mode 100644 index 00000000000000..a491690507129e --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-01-08-15-49-20.bpo-42780.rtqi6B.rst @@ -0,0 +1 @@ +Fix os.set_inheritable() for O_PATH file descriptors on Linux. diff --git a/Python/fileutils.c b/Python/fileutils.c index 25516c23bf7ab4..fd2d5faa0fdf7e 100644 --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -1165,6 +1165,13 @@ set_inheritable(int fd, int inheritable, int raise, int *atomic_flag_works) return 0; } +#ifdef __linux__ + if (errno == EBADF) { + // On Linux, ioctl(FIOCLEX) will fail with EBADF for O_PATH file descriptors + // Fall through to the fcntl() path + } + else +#endif if (errno != ENOTTY && errno != EACCES) { if (raise) PyErr_SetFromErrno(PyExc_OSError); From 34f3f4ac70e3ba2e603ba7792addf973c94f90cb Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 21 Jan 2021 21:57:40 -0800 Subject: [PATCH 1949/2163] bpo-40304: Correct type(name, bases, dict) doc (GH-19553) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Éric Araujo Co-authored-by: Terry Jan Reedy Co-authored-by: Tal Einat <532281+taleinat@users.noreply.github.com> (cherry picked from commit 644d52818a6391535e5838fd57d58ffcb1163056) Co-authored-by: Đ‘ĐľŃ€Đ¸Ń Đ’ĐµŃ€Ń…ĐľĐ˛Ńкий --- Doc/library/functions.rst | 17 +++++++++-------- .../2021-01-20-23-03-49.bpo-40304.-LK7Ps.rst | 2 ++ 2 files changed, 11 insertions(+), 8 deletions(-) create mode 100644 Misc/NEWS.d/next/Documentation/2021-01-20-23-03-49.bpo-40304.-LK7Ps.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 4f5fe6bc230697..5a039f7a9476ff 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1683,18 +1683,19 @@ are always available. They are listed here in alphabetical order. With three arguments, return a new type object. This is essentially a - dynamic form of the :keyword:`class` statement. The *name* string is the - class name and becomes the :attr:`~definition.__name__` attribute; the *bases* - tuple itemizes the base classes and becomes the :attr:`~class.__bases__` - attribute; and the *dict* dictionary is the namespace containing definitions - for class body and is copied to a standard dictionary to become the - :attr:`~object.__dict__` attribute. For example, the following two - statements create identical :class:`type` objects: + dynamic form of the :keyword:`class` statement. The *name* string is + the class name and becomes the :attr:`~definition.__name__` attribute. + The *bases* tuple contains the base classes and becomes the + :attr:`~class.__bases__` attribute; if empty, :class:`object`, the + ultimate base of all classes, is added. The *dict* dictionary contains + attribute and method definitions for the class body; it may be copied + or wrapped before becoming the :attr:`~object.__dict__` attribute. + The following two statements create identical :class:`type` objects: >>> class X: ... a = 1 ... - >>> X = type('X', (object,), dict(a=1)) + >>> X = type('X', (), dict(a=1)) See also :ref:`bltin-type-objects`. diff --git a/Misc/NEWS.d/next/Documentation/2021-01-20-23-03-49.bpo-40304.-LK7Ps.rst b/Misc/NEWS.d/next/Documentation/2021-01-20-23-03-49.bpo-40304.-LK7Ps.rst new file mode 100644 index 00000000000000..3f2f14c2d7b893 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-01-20-23-03-49.bpo-40304.-LK7Ps.rst @@ -0,0 +1,2 @@ +Fix doc for type(name, bases, dict). Patch by Boris Verkhovskiy and +Éric Araujo. From dce86c230bb91722e84cd3618c1ee9cb55cc220f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 23 Jan 2021 08:55:20 -0800 Subject: [PATCH 1950/2163] closes bpo-43011: Fix DeprecationWarnings in test_ctypes (GH-24305) (cherry picked from commit f7fa64f0e87edc61d990ed51b4da722906a10928) Co-authored-by: Zackery Spytz --- Lib/ctypes/test/test_parameters.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/ctypes/test/test_parameters.py b/Lib/ctypes/test/test_parameters.py index 531894fdec8382..38af7ac13d756c 100644 --- a/Lib/ctypes/test/test_parameters.py +++ b/Lib/ctypes/test/test_parameters.py @@ -240,8 +240,8 @@ def test_parameter_repr(self): self.assertEqual(repr(c_double.from_param(1.5)), "") self.assertEqual(repr(c_double.from_param(1e300)), "") self.assertRegex(repr(c_longdouble.from_param(1.5)), r"^$") - self.assertRegex(repr(c_char_p.from_param(b'hihi')), "^$") - self.assertRegex(repr(c_wchar_p.from_param('hihi')), "^$") + self.assertRegex(repr(c_char_p.from_param(b'hihi')), r"^$") + self.assertRegex(repr(c_wchar_p.from_param('hihi')), r"^$") self.assertRegex(repr(c_void_p.from_param(0x12)), r"^$") ################################################################ From 63ebba0d9430a0bc18fd8551ad27b9b25709a4be Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 24 Jan 2021 14:16:03 -0800 Subject: [PATCH 1951/2163] bpo-43013: Update idlelib code to 3.x (GH-24315) (#24317) Remove 9 remaining '(object)' occurrences in class headers in idlelib and 25 '()' occurrences in idlelib.idle_test class headers. (cherry picked from commit 8dfe15625e6ea4357a13fec7989a0e6ba2bf1359) Co-authored-by: Terry Jan Reedy --- Lib/idlelib/editor.py | 4 +-- Lib/idlelib/idle_test/test_autocomplete.py | 2 +- Lib/idlelib/idle_test/test_calltip.py | 4 +-- Lib/idlelib/idle_test/test_codecontext.py | 38 +++++++++++----------- Lib/idlelib/idle_test/test_format.py | 2 +- Lib/idlelib/idle_test/test_help_about.py | 2 +- Lib/idlelib/idle_test/test_pyparse.py | 30 +++++++++-------- Lib/idlelib/rpc.py | 10 +++--- Lib/idlelib/run.py | 2 +- Lib/idlelib/tooltip.py | 2 +- 10 files changed, 49 insertions(+), 47 deletions(-) diff --git a/Lib/idlelib/editor.py b/Lib/idlelib/editor.py index a178eaf93c013a..a4d0c95362fa9b 100644 --- a/Lib/idlelib/editor.py +++ b/Lib/idlelib/editor.py @@ -46,7 +46,7 @@ def _sphinx_version(): return release -class EditorWindow(object): +class EditorWindow: from idlelib.percolator import Percolator from idlelib.colorizer import ColorDelegator, color_config from idlelib.undo import UndoDelegator @@ -1546,7 +1546,7 @@ def get_line_indent(line, tabwidth): return m.end(), len(m.group().expandtabs(tabwidth)) -class IndentSearcher(object): +class IndentSearcher: # .run() chews over the Text widget, looking for a block opener # and the stmt following it. Returns a pair, diff --git a/Lib/idlelib/idle_test/test_autocomplete.py b/Lib/idlelib/idle_test/test_autocomplete.py index 9c113bd893f137..642bb5db64dc34 100644 --- a/Lib/idlelib/idle_test/test_autocomplete.py +++ b/Lib/idlelib/idle_test/test_autocomplete.py @@ -195,7 +195,7 @@ def test_open_completions_none(self): self.assertFalse(acp.open_completions(ac.TAB)) self.text.delete('1.0', 'end') - class dummy_acw(): + class dummy_acw: __init__ = Func() show_window = Func(result=False) hide_window = Func() diff --git a/Lib/idlelib/idle_test/test_calltip.py b/Lib/idlelib/idle_test/test_calltip.py index a76829f3656c80..b23915c5ab7849 100644 --- a/Lib/idlelib/idle_test/test_calltip.py +++ b/Lib/idlelib/idle_test/test_calltip.py @@ -10,7 +10,7 @@ # Test Class TC is used in multiple get_argspec test methods -class TC(): +class TC: 'doc' tip = "(ai=None, *b)" def __init__(self, ai=None, *b): 'doc' @@ -268,7 +268,7 @@ def test_good_entity(self): # open_calltip is about half the code; the others are fairly trivial. # The default mocks are what are needed for open_calltip. -class mock_Shell(): +class mock_Shell: "Return mock sufficient to pass to hyperparser." def __init__(self, text): text.tag_prevrange = Mock(return_value=None) diff --git a/Lib/idlelib/idle_test/test_codecontext.py b/Lib/idlelib/idle_test/test_codecontext.py index 9578cc731a6f93..6969ad73b01a81 100644 --- a/Lib/idlelib/idle_test/test_codecontext.py +++ b/Lib/idlelib/idle_test/test_codecontext.py @@ -20,7 +20,7 @@ } code_sample = """\ -class C1(): +class C1: # Class comment. def __init__(self, a, b): self.a = a @@ -178,29 +178,29 @@ def test_get_context(self): with self.assertRaises(AssertionError): gc(1, stopline=0) - eq(gc(3), ([(2, 0, 'class C1():', 'class')], 0)) + eq(gc(3), ([(2, 0, 'class C1:', 'class')], 0)) # Don't return comment. - eq(gc(4), ([(2, 0, 'class C1():', 'class')], 0)) + eq(gc(4), ([(2, 0, 'class C1:', 'class')], 0)) # Two indentation levels and no comment. - eq(gc(5), ([(2, 0, 'class C1():', 'class'), + eq(gc(5), ([(2, 0, 'class C1:', 'class'), (4, 4, ' def __init__(self, a, b):', 'def')], 0)) # Only one 'def' is returned, not both at the same indent level. - eq(gc(10), ([(2, 0, 'class C1():', 'class'), + eq(gc(10), ([(2, 0, 'class C1:', 'class'), (7, 4, ' def compare(self):', 'def'), (8, 8, ' if a > b:', 'if')], 0)) # With 'elif', also show the 'if' even though it's at the same level. - eq(gc(11), ([(2, 0, 'class C1():', 'class'), + eq(gc(11), ([(2, 0, 'class C1:', 'class'), (7, 4, ' def compare(self):', 'def'), (8, 8, ' if a > b:', 'if'), (10, 8, ' elif a < b:', 'elif')], 0)) # Set stop_line to not go back to first line in source code. # Return includes stop_line. - eq(gc(11, stopline=2), ([(2, 0, 'class C1():', 'class'), + eq(gc(11, stopline=2), ([(2, 0, 'class C1:', 'class'), (7, 4, ' def compare(self):', 'def'), (8, 8, ' if a > b:', 'if'), (10, 8, ' elif a < b:', 'elif')], 0)) @@ -240,37 +240,37 @@ def test_update_code_context(self): # Scroll down to line 2. cc.text.yview(2) cc.update_code_context() - eq(cc.info, [(0, -1, '', False), (2, 0, 'class C1():', 'class')]) + eq(cc.info, [(0, -1, '', False), (2, 0, 'class C1:', 'class')]) eq(cc.topvisible, 3) - eq(cc.context.get('1.0', 'end-1c'), 'class C1():') + eq(cc.context.get('1.0', 'end-1c'), 'class C1:') # Scroll down to line 3. Since it's a comment, nothing changes. cc.text.yview(3) cc.update_code_context() - eq(cc.info, [(0, -1, '', False), (2, 0, 'class C1():', 'class')]) + eq(cc.info, [(0, -1, '', False), (2, 0, 'class C1:', 'class')]) eq(cc.topvisible, 4) - eq(cc.context.get('1.0', 'end-1c'), 'class C1():') + eq(cc.context.get('1.0', 'end-1c'), 'class C1:') # Scroll down to line 4. cc.text.yview(4) cc.update_code_context() eq(cc.info, [(0, -1, '', False), - (2, 0, 'class C1():', 'class'), + (2, 0, 'class C1:', 'class'), (4, 4, ' def __init__(self, a, b):', 'def')]) eq(cc.topvisible, 5) - eq(cc.context.get('1.0', 'end-1c'), 'class C1():\n' + eq(cc.context.get('1.0', 'end-1c'), 'class C1:\n' ' def __init__(self, a, b):') # Scroll down to line 11. Last 'def' is removed. cc.text.yview(11) cc.update_code_context() eq(cc.info, [(0, -1, '', False), - (2, 0, 'class C1():', 'class'), + (2, 0, 'class C1:', 'class'), (7, 4, ' def compare(self):', 'def'), (8, 8, ' if a > b:', 'if'), (10, 8, ' elif a < b:', 'elif')]) eq(cc.topvisible, 12) - eq(cc.context.get('1.0', 'end-1c'), 'class C1():\n' + eq(cc.context.get('1.0', 'end-1c'), 'class C1:\n' ' def compare(self):\n' ' if a > b:\n' ' elif a < b:') @@ -279,12 +279,12 @@ def test_update_code_context(self): cc.update_code_context() cc.context_depth = 1 eq(cc.info, [(0, -1, '', False), - (2, 0, 'class C1():', 'class'), + (2, 0, 'class C1:', 'class'), (7, 4, ' def compare(self):', 'def'), (8, 8, ' if a > b:', 'if'), (10, 8, ' elif a < b:', 'elif')]) eq(cc.topvisible, 12) - eq(cc.context.get('1.0', 'end-1c'), 'class C1():\n' + eq(cc.context.get('1.0', 'end-1c'), 'class C1:\n' ' def compare(self):\n' ' if a > b:\n' ' elif a < b:') @@ -293,7 +293,7 @@ def test_update_code_context(self): cc.text.yview(5) cc.update_code_context() eq(cc.info, [(0, -1, '', False), - (2, 0, 'class C1():', 'class'), + (2, 0, 'class C1:', 'class'), (4, 4, ' def __init__(self, a, b):', 'def')]) eq(cc.topvisible, 6) # context_depth is 1. @@ -440,7 +440,7 @@ def test_get_line_info(self): # Line 1 is not a BLOCKOPENER. eq(gli(lines[0]), (codecontext.INFINITY, '', False)) # Line 2 is a BLOCKOPENER without an indent. - eq(gli(lines[1]), (0, 'class C1():', 'class')) + eq(gli(lines[1]), (0, 'class C1:', 'class')) # Line 3 is not a BLOCKOPENER and does not return the indent level. eq(gli(lines[2]), (codecontext.INFINITY, ' # Class comment.', False)) # Line 4 is a BLOCKOPENER and is indented. diff --git a/Lib/idlelib/idle_test/test_format.py b/Lib/idlelib/idle_test/test_format.py index a79bb515089e7b..e5e903688597aa 100644 --- a/Lib/idlelib/idle_test/test_format.py +++ b/Lib/idlelib/idle_test/test_format.py @@ -418,7 +418,7 @@ def tearDown(self): code_sample = """\ # WS line needed for test. -class C1(): +class C1: # Class comment. def __init__(self, a, b): self.a = a diff --git a/Lib/idlelib/idle_test/test_help_about.py b/Lib/idlelib/idle_test/test_help_about.py index 7c148d23a135b6..b915535acac0cc 100644 --- a/Lib/idlelib/idle_test/test_help_about.py +++ b/Lib/idlelib/idle_test/test_help_about.py @@ -134,7 +134,7 @@ def test_close(self): self.dialog.winfo_class() -class Dummy_about_dialog(): +class Dummy_about_dialog: # Dummy class for testing file display functions. idle_credits = About.show_idle_credits idle_readme = About.show_readme diff --git a/Lib/idlelib/idle_test/test_pyparse.py b/Lib/idlelib/idle_test/test_pyparse.py index f21baf7534420a..fb5726db1d821e 100644 --- a/Lib/idlelib/idle_test/test_pyparse.py +++ b/Lib/idlelib/idle_test/test_pyparse.py @@ -73,11 +73,12 @@ def char_in_string_false(index): return False # Split def across lines. setcode('"""This is a module docstring"""\n' - 'class C():\n' + 'class C:\n' ' def __init__(self, a,\n' ' b=True):\n' ' pass\n' ) + pos0, pos = 33, 42 # Start of 'class...', ' def' lines. # Passing no value or non-callable should fail (issue 32989). with self.assertRaises(TypeError): @@ -91,40 +92,41 @@ def char_in_string_false(index): return False # Make all text look like it's not in a string. This means that it # found a good start position. - eq(start(char_in_string_false), 44) + eq(start(char_in_string_false), pos) # If the beginning of the def line is not in a string, then it # returns that as the index. - eq(start(is_char_in_string=lambda index: index > 44), 44) + eq(start(is_char_in_string=lambda index: index > pos), pos) # If the beginning of the def line is in a string, then it # looks for a previous index. - eq(start(is_char_in_string=lambda index: index >= 44), 33) + eq(start(is_char_in_string=lambda index: index >= pos), pos0) # If everything before the 'def' is in a string, then returns None. # The non-continuation def line returns 44 (see below). - eq(start(is_char_in_string=lambda index: index < 44), None) + eq(start(is_char_in_string=lambda index: index < pos), None) # Code without extra line break in def line - mostly returns the same # values. setcode('"""This is a module docstring"""\n' - 'class C():\n' + 'class C:\n' ' def __init__(self, a, b=True):\n' ' pass\n' - ) - eq(start(char_in_string_false), 44) - eq(start(is_char_in_string=lambda index: index > 44), 44) - eq(start(is_char_in_string=lambda index: index >= 44), 33) + ) # Does not affect class, def positions. + eq(start(char_in_string_false), pos) + eq(start(is_char_in_string=lambda index: index > pos), pos) + eq(start(is_char_in_string=lambda index: index >= pos), pos0) # When the def line isn't split, this returns which doesn't match the # split line test. - eq(start(is_char_in_string=lambda index: index < 44), 44) + eq(start(is_char_in_string=lambda index: index < pos), pos) def test_set_lo(self): code = ( '"""This is a module docstring"""\n' - 'class C():\n' + 'class C:\n' ' def __init__(self, a,\n' ' b=True):\n' ' pass\n' ) + pos = 42 p = self.parser p.set_code(code) @@ -137,8 +139,8 @@ def test_set_lo(self): self.assertEqual(p.code, code) # An index that is preceded by a newline. - p.set_lo(44) - self.assertEqual(p.code, code[44:]) + p.set_lo(pos) + self.assertEqual(p.code, code[pos:]) def test_study1(self): eq = self.assertEqual diff --git a/Lib/idlelib/rpc.py b/Lib/idlelib/rpc.py index aa8cbd36c47926..8efcf048fa3aa2 100644 --- a/Lib/idlelib/rpc.py +++ b/Lib/idlelib/rpc.py @@ -125,7 +125,7 @@ def handle_error(self, request, client_address): response_queue = queue.Queue(0) -class SocketIO(object): +class SocketIO: nextseq = 0 @@ -486,7 +486,7 @@ def EOFhook(self): #----------------- end class SocketIO -------------------- -class RemoteObject(object): +class RemoteObject: # Token mix-in class pass @@ -497,7 +497,7 @@ def remoteref(obj): return RemoteProxy(oid) -class RemoteProxy(object): +class RemoteProxy: def __init__(self, oid): self.oid = oid @@ -547,7 +547,7 @@ def get_remote_proxy(self, oid): return RPCProxy(self, oid) -class RPCProxy(object): +class RPCProxy: __methods = None __attributes = None @@ -596,7 +596,7 @@ def _getattributes(obj, attributes): attributes[name] = 1 -class MethodProxy(object): +class MethodProxy: def __init__(self, sockio, oid, name): self.sockio = sockio diff --git a/Lib/idlelib/run.py b/Lib/idlelib/run.py index 1e84ecc6584ef1..ec575c3d483631 100644 --- a/Lib/idlelib/run.py +++ b/Lib/idlelib/run.py @@ -538,7 +538,7 @@ def decode_interrupthook(self): thread.interrupt_main() -class Executive(object): +class Executive: def __init__(self, rpchandler): self.rpchandler = rpchandler diff --git a/Lib/idlelib/tooltip.py b/Lib/idlelib/tooltip.py index 69658264dbd4a4..d714318dae8ef1 100644 --- a/Lib/idlelib/tooltip.py +++ b/Lib/idlelib/tooltip.py @@ -7,7 +7,7 @@ from tkinter import * -class TooltipBase(object): +class TooltipBase: """abstract base class for tooltips""" def __init__(self, anchor_widget): From 96bcf6f4d4bed1cdf97883eb43e872ff1a92013d Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 25 Jan 2021 02:02:52 -0800 Subject: [PATCH 1952/2163] [3.9] bpo-33289: Return RGB triplet of ints instead of floats from tkinter.colorchooser (GH-6578). (GH-24318) (cherry picked from commit 6713e869c4989c04318158b406c30a147ea52904) Co-authored-by: Cheryl Sabella (cherry picked from commit 3d5434d5cbc945c58be663e3dbd5ef4875677b7a) Co-authored-by: Serhiy Storchaka --- Lib/tkinter/__init__.py | 3 +- Lib/tkinter/colorchooser.py | 60 +++++++++++-------- .../test/test_tkinter/test_colorchooser.py | 43 +++++++++++++ Lib/tkinter/test/test_tkinter/test_misc.py | 20 +++++++ .../2018-04-23-13-44-10.bpo-33289.anBnUr.rst | 2 + 5 files changed, 102 insertions(+), 26 deletions(-) create mode 100644 Lib/tkinter/test/test_tkinter/test_colorchooser.py create mode 100644 Misc/NEWS.d/next/Library/2018-04-23-13-44-10.bpo-33289.anBnUr.rst diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py index 91a6d56480e630..4955c9e002f151 100644 --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -1160,8 +1160,7 @@ def winfo_reqwidth(self): self.tk.call('winfo', 'reqwidth', self._w)) def winfo_rgb(self, color): - """Return tuple of decimal values for red, green, blue for - COLOR in this widget.""" + """Return a tuple of integer RGB values in range(65536) for color in this widget.""" return self._getints( self.tk.call('winfo', 'rgb', self._w, color)) diff --git a/Lib/tkinter/colorchooser.py b/Lib/tkinter/colorchooser.py index 9dc96713364089..5fd4c347f0602a 100644 --- a/Lib/tkinter/colorchooser.py +++ b/Lib/tkinter/colorchooser.py @@ -8,55 +8,67 @@ # fixed initialcolor handling in August 1998 # -# -# options (all have default values): -# -# - initialcolor: color to mark as selected when dialog is displayed -# (given as an RGB triplet or a Tk color string) -# -# - parent: which window to place the dialog on top of -# -# - title: dialog title -# from tkinter.commondialog import Dialog -# -# color chooser class - class Chooser(Dialog): - "Ask for a color" + """Create a dialog for the tk_chooseColor command. + + Args: + master: The master widget for this dialog. If not provided, + defaults to options['parent'] (if defined). + options: Dictionary of options for the tk_chooseColor call. + initialcolor: Specifies the selected color when the + dialog is first displayed. This can be a tk color + string or a 3-tuple of ints in the range (0, 255) + for an RGB triplet. + parent: The parent window of the color dialog. The + color dialog is displayed on top of this. + title: A string for the title of the dialog box. + """ command = "tk_chooseColor" def _fixoptions(self): + """Ensure initialcolor is a tk color string. + + Convert initialcolor from a RGB triplet to a color string. + """ try: - # make sure initialcolor is a tk color string color = self.options["initialcolor"] if isinstance(color, tuple): - # assume an RGB triplet + # Assume an RGB triplet. self.options["initialcolor"] = "#%02x%02x%02x" % color except KeyError: pass def _fixresult(self, widget, result): - # result can be somethings: an empty tuple, an empty string or - # a Tcl_Obj, so this somewhat weird check handles that + """Adjust result returned from call to tk_chooseColor. + + Return both an RGB tuple of ints in the range (0, 255) and the + tk color string in the form #rrggbb. + """ + # Result can be many things: an empty tuple, an empty string, or + # a _tkinter.Tcl_Obj, so this somewhat weird check handles that. if not result or not str(result): - return None, None # canceled + return None, None # canceled - # to simplify application code, the color chooser returns - # an RGB tuple together with the Tk color string + # To simplify application code, the color chooser returns + # an RGB tuple together with the Tk color string. r, g, b = widget.winfo_rgb(result) - return (r/256, g/256, b/256), str(result) + return (r//256, g//256, b//256), str(result) # # convenience stuff -def askcolor(color = None, **options): - "Ask for a color" +def askcolor(color=None, **options): + """Display dialog window for selection of a color. + + Convenience wrapper for the Chooser class. Displays the color + chooser dialog with color as the initial value. + """ if color: options = options.copy() diff --git a/Lib/tkinter/test/test_tkinter/test_colorchooser.py b/Lib/tkinter/test/test_tkinter/test_colorchooser.py new file mode 100644 index 00000000000000..4798bc9c26fc05 --- /dev/null +++ b/Lib/tkinter/test/test_tkinter/test_colorchooser.py @@ -0,0 +1,43 @@ +import unittest +import tkinter +from test.support import requires, run_unittest, swap_attr +from tkinter.test.support import AbstractTkTest +from tkinter import colorchooser + +requires('gui') + + +class ChooserTest(AbstractTkTest, unittest.TestCase): + + @classmethod + def setUpClass(cls): + AbstractTkTest.setUpClass.__func__(cls) + cls.cc = colorchooser.Chooser(initialcolor='dark blue slate') + + def test_fixoptions(self): + cc = self.cc + cc._fixoptions() + self.assertEqual(cc.options['initialcolor'], 'dark blue slate') + + cc.options['initialcolor'] = '#D2D269691E1E' + cc._fixoptions() + self.assertEqual(cc.options['initialcolor'], '#D2D269691E1E') + + cc.options['initialcolor'] = (210, 105, 30) + cc._fixoptions() + self.assertEqual(cc.options['initialcolor'], '#d2691e') + + def test_fixresult(self): + cc = self.cc + self.assertEqual(cc._fixresult(self.root, ()), (None, None)) + self.assertEqual(cc._fixresult(self.root, ''), (None, None)) + self.assertEqual(cc._fixresult(self.root, 'chocolate'), + ((210, 105, 30), 'chocolate')) + self.assertEqual(cc._fixresult(self.root, '#4a3c8c'), + ((74, 60, 140), '#4a3c8c')) + + +tests_gui = (ChooserTest,) + +if __name__ == "__main__": + run_unittest(*tests_gui) diff --git a/Lib/tkinter/test/test_tkinter/test_misc.py b/Lib/tkinter/test/test_tkinter/test_misc.py index 49fb6b16de09eb..2011e7f8ecdd71 100644 --- a/Lib/tkinter/test/test_tkinter/test_misc.py +++ b/Lib/tkinter/test/test_tkinter/test_misc.py @@ -178,6 +178,26 @@ def test_clipboard_astral(self): with self.assertRaises(tkinter.TclError): root.clipboard_get() + def test_winfo_rgb(self): + root = self.root + rgb = root.winfo_rgb + + # Color name. + self.assertEqual(rgb('red'), (65535, 0, 0)) + self.assertEqual(rgb('dark slate blue'), (18504, 15677, 35723)) + # #RGB - extends each 4-bit hex value to be 16-bit. + self.assertEqual(rgb('#F0F'), (0xFFFF, 0x0000, 0xFFFF)) + # #RRGGBB - extends each 8-bit hex value to be 16-bit. + self.assertEqual(rgb('#4a3c8c'), (0x4a4a, 0x3c3c, 0x8c8c)) + # #RRRRGGGGBBBB + self.assertEqual(rgb('#dede14143939'), (0xdede, 0x1414, 0x3939)) + # Invalid string. + with self.assertRaises(tkinter.TclError): + rgb('#123456789a') + # RGB triplet is invalid input. + with self.assertRaises(tkinter.TclError): + rgb((111, 78, 55)) + def test_event_repr_defaults(self): e = tkinter.Event() e.serial = 12345 diff --git a/Misc/NEWS.d/next/Library/2018-04-23-13-44-10.bpo-33289.anBnUr.rst b/Misc/NEWS.d/next/Library/2018-04-23-13-44-10.bpo-33289.anBnUr.rst new file mode 100644 index 00000000000000..52d9ac9dd902cd --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-04-23-13-44-10.bpo-33289.anBnUr.rst @@ -0,0 +1,2 @@ +Correct call to :mod:`tkinter.colorchooser` to return RGB triplet of ints +instead of floats. Patch by Cheryl Sabella. From 7370be30017f81d2f41f1b4b2abf31dd9a3f8fb1 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 25 Jan 2021 03:51:49 -0800 Subject: [PATCH 1953/2163] bpo-43013: Fix old tkinter module names in idlelib (GH-24326) Lowercase 'tkColorChooser', 'tkFileDialog', 'tkSimpleDialog', and 'tkMessageBox' and remove 'tk'. Just lowercase 'tkFont' as 'font' is already used. Adjust import. (cherry picked from commit 879986d8a932c4524cb6ff822afc9537de16e28d) Co-authored-by: Terry Jan Reedy --- Lib/idlelib/configdialog.py | 10 ++++---- Lib/idlelib/editor.py | 16 ++++++------ Lib/idlelib/filelist.py | 6 ++--- Lib/idlelib/idle_test/mock_tk.py | 9 +++---- Lib/idlelib/idle_test/test_configdialog.py | 8 +++--- Lib/idlelib/idle_test/test_replace.py | 6 ++--- Lib/idlelib/idle_test/test_searchengine.py | 6 ++--- Lib/idlelib/idle_test/test_squeezer.py | 4 +-- Lib/idlelib/iomenu.py | 30 +++++++++++----------- Lib/idlelib/pyshell.py | 20 +++++++-------- Lib/idlelib/runscript.py | 8 +++--- Lib/idlelib/searchengine.py | 4 +-- Lib/idlelib/squeezer.py | 6 ++--- 13 files changed, 66 insertions(+), 67 deletions(-) diff --git a/Lib/idlelib/configdialog.py b/Lib/idlelib/configdialog.py index 73e64852c69dfd..c52a04b503adb4 100644 --- a/Lib/idlelib/configdialog.py +++ b/Lib/idlelib/configdialog.py @@ -18,8 +18,8 @@ HORIZONTAL, VERTICAL, ANCHOR, ACTIVE, END) from tkinter.ttk import (Frame, LabelFrame, Button, Checkbutton, Entry, Label, OptionMenu, Notebook, Radiobutton, Scrollbar, Style) -import tkinter.colorchooser as tkColorChooser -import tkinter.font as tkFont +from tkinter import colorchooser +import tkinter.font as tkfont from tkinter import messagebox from idlelib.config import idleConf, ConfigChanges @@ -609,7 +609,7 @@ def load_font_cfg(self): font_bold = configured_font[2]=='bold' # Set sorted no-duplicate editor font selection list and font_name. - fonts = sorted(set(tkFont.families(self))) + fonts = sorted(set(tkfont.families(self))) for font in fonts: self.fontlist.insert(END, font) self.font_name.set(font_name) @@ -663,7 +663,7 @@ def set_samples(self, event=None): Updates font_sample and highlight page highlight_sample. """ font_name = self.font_name.get() - font_weight = tkFont.BOLD if self.font_bold.get() else tkFont.NORMAL + font_weight = tkfont.BOLD if self.font_bold.get() else tkfont.NORMAL new_font = (font_name, self.font_size.get(), font_weight) self.font_sample['font'] = new_font self.highlight_sample['font'] = new_font @@ -1100,7 +1100,7 @@ def get_color(self): target = self.highlight_target.get() prev_color = self.style.lookup(self.frame_color_set['style'], 'background') - rgbTuplet, color_string = tkColorChooser.askcolor( + rgbTuplet, color_string = colorchooser.askcolor( parent=self, title='Pick new color for : '+target, initialcolor=prev_color) if color_string and (color_string != prev_color): diff --git a/Lib/idlelib/editor.py b/Lib/idlelib/editor.py index a4d0c95362fa9b..66e9da5a9dccf9 100644 --- a/Lib/idlelib/editor.py +++ b/Lib/idlelib/editor.py @@ -12,8 +12,8 @@ from tkinter import * from tkinter.font import Font from tkinter.ttk import Scrollbar -import tkinter.simpledialog as tkSimpleDialog -import tkinter.messagebox as tkMessageBox +from tkinter import simpledialog +from tkinter import messagebox from idlelib.config import idleConf from idlelib import configdialog @@ -295,9 +295,9 @@ def __init__(self, flist=None, filename=None, key=None, root=None): window.register_callback(self.postwindowsmenu) # Some abstractions so IDLE extensions are cross-IDE - self.askyesno = tkMessageBox.askyesno - self.askinteger = tkSimpleDialog.askinteger - self.showerror = tkMessageBox.showerror + self.askinteger = simpledialog.askinteger + self.askyesno = messagebox.askyesno + self.showerror = messagebox.showerror # Add pseudoevents for former extension fixed keys. # (This probably needs to be done once in the process.) @@ -596,7 +596,7 @@ def python_docs(self, event=None): try: os.startfile(self.help_url) except OSError as why: - tkMessageBox.showerror(title='Document Start Failure', + messagebox.showerror(title='Document Start Failure', message=str(why), parent=self.text) else: webbrowser.open(self.help_url) @@ -927,7 +927,7 @@ def display_extra_help(helpfile=helpfile): try: os.startfile(helpfile) except OSError as why: - tkMessageBox.showerror(title='Document Start Failure', + messagebox.showerror(title='Document Start Failure', message=str(why), parent=self.text) else: webbrowser.open(helpfile) @@ -963,7 +963,7 @@ def update_recent_files_list(self, new_file=None): except OSError as err: if not getattr(self.root, "recentfiles_message", False): self.root.recentfiles_message = True - tkMessageBox.showwarning(title='IDLE Warning', + messagebox.showwarning(title='IDLE Warning', message="Cannot save Recent Files list to disk.\n" f" {err}\n" "Select OK to continue.", diff --git a/Lib/idlelib/filelist.py b/Lib/idlelib/filelist.py index 0d200854ef0007..254f5caf6b81b0 100644 --- a/Lib/idlelib/filelist.py +++ b/Lib/idlelib/filelist.py @@ -1,7 +1,7 @@ "idlelib.filelist" import os -from tkinter import messagebox as tkMessageBox +from tkinter import messagebox class FileList: @@ -20,7 +20,7 @@ def open(self, filename, action=None): filename = self.canonize(filename) if os.path.isdir(filename): # This can happen when bad filename is passed on command line: - tkMessageBox.showerror( + messagebox.showerror( "File Error", "%r is a directory." % (filename,), master=self.root) @@ -88,7 +88,7 @@ def filename_changed_edit(self, edit): if newkey in self.dict: conflict = self.dict[newkey] self.inversedict[conflict] = None - tkMessageBox.showerror( + messagebox.showerror( "Name Conflict", "You now have multiple edit windows open for %r" % (filename,), master=self.root) diff --git a/Lib/idlelib/idle_test/mock_tk.py b/Lib/idlelib/idle_test/mock_tk.py index b736bd001da87f..db583553838fb3 100644 --- a/Lib/idlelib/idle_test/mock_tk.py +++ b/Lib/idlelib/idle_test/mock_tk.py @@ -59,27 +59,26 @@ def __call__(self, title, message, *args, **kwds): class Mbox: """Mock for tkinter.messagebox with an Mbox_func for each function. - This module was 'tkMessageBox' in 2.x; hence the 'import as' in 3.x. Example usage in test_module.py for testing functions in module.py: --- from idlelib.idle_test.mock_tk import Mbox import module -orig_mbox = module.tkMessageBox +orig_mbox = module.messagebox showerror = Mbox.showerror # example, for attribute access in test methods class Test(unittest.TestCase): @classmethod def setUpClass(cls): - module.tkMessageBox = Mbox + module.messagebox = Mbox @classmethod def tearDownClass(cls): - module.tkMessageBox = orig_mbox + module.messagebox = orig_mbox --- For 'ask' functions, set func.result return value before calling the method - that uses the message function. When tkMessageBox functions are the + that uses the message function. When messagebox functions are the only gui alls in a method, this replacement makes the method gui-free, """ askokcancel = Mbox_func() # True or False diff --git a/Lib/idlelib/idle_test/test_configdialog.py b/Lib/idlelib/idle_test/test_configdialog.py index 1fea6d41df811c..98ddc67afdcc08 100644 --- a/Lib/idlelib/idle_test/test_configdialog.py +++ b/Lib/idlelib/idle_test/test_configdialog.py @@ -423,7 +423,7 @@ def test_custom_name(self): def test_color(self): d = self.page d.on_new_color_set = Func() - # self.color is only set in get_color through ColorChooser. + # self.color is only set in get_color through colorchooser. d.color.set('green') self.assertEqual(d.on_new_color_set.called, 1) del d.on_new_color_set @@ -540,8 +540,8 @@ def test_set_theme_type(self): def test_get_color(self): eq = self.assertEqual d = self.page - orig_chooser = configdialog.tkColorChooser.askcolor - chooser = configdialog.tkColorChooser.askcolor = Func() + orig_chooser = configdialog.colorchooser.askcolor + chooser = configdialog.colorchooser.askcolor = Func() gntn = d.get_new_theme_name = Func() d.highlight_target.set('Editor Breakpoint') @@ -582,7 +582,7 @@ def test_get_color(self): eq(d.color.get(), '#de0000') del d.get_new_theme_name - configdialog.tkColorChooser.askcolor = orig_chooser + configdialog.colorchooser.askcolor = orig_chooser def test_on_new_color_set(self): d = self.page diff --git a/Lib/idlelib/idle_test/test_replace.py b/Lib/idlelib/idle_test/test_replace.py index c3c5d2eeb94998..6c07389b29ad45 100644 --- a/Lib/idlelib/idle_test/test_replace.py +++ b/Lib/idlelib/idle_test/test_replace.py @@ -10,7 +10,7 @@ from idlelib.idle_test.mock_tk import Mbox import idlelib.searchengine as se -orig_mbox = se.tkMessageBox +orig_mbox = se.messagebox showerror = Mbox.showerror @@ -20,7 +20,7 @@ class ReplaceDialogTest(unittest.TestCase): def setUpClass(cls): cls.root = Tk() cls.root.withdraw() - se.tkMessageBox = Mbox + se.messagebox = Mbox cls.engine = se.SearchEngine(cls.root) cls.dialog = ReplaceDialog(cls.root, cls.engine) cls.dialog.bell = lambda: None @@ -32,7 +32,7 @@ def setUpClass(cls): @classmethod def tearDownClass(cls): - se.tkMessageBox = orig_mbox + se.messagebox = orig_mbox del cls.text, cls.dialog, cls.engine cls.root.destroy() del cls.root diff --git a/Lib/idlelib/idle_test/test_searchengine.py b/Lib/idlelib/idle_test/test_searchengine.py index f8401ce9380f25..9d979839419586 100644 --- a/Lib/idlelib/idle_test/test_searchengine.py +++ b/Lib/idlelib/idle_test/test_searchengine.py @@ -4,7 +4,7 @@ import unittest # from test.support import requires from tkinter import BooleanVar, StringVar, TclError # ,Tk, Text -import tkinter.messagebox as tkMessageBox +from tkinter import messagebox from idlelib.idle_test.mock_tk import Var, Mbox from idlelib.idle_test.mock_tk import Text as mockText import re @@ -19,13 +19,13 @@ def setUpModule(): # Replace s-e module tkinter imports other than non-gui TclError. se.BooleanVar = Var se.StringVar = Var - se.tkMessageBox = Mbox + se.messagebox = Mbox def tearDownModule(): # Restore 'just in case', though other tests should also replace. se.BooleanVar = BooleanVar se.StringVar = StringVar - se.tkMessageBox = tkMessageBox + se.messagebox = messagebox class Mock: diff --git a/Lib/idlelib/idle_test/test_squeezer.py b/Lib/idlelib/idle_test/test_squeezer.py index e3912f4bbbec89..ee1bbd76b50562 100644 --- a/Lib/idlelib/idle_test/test_squeezer.py +++ b/Lib/idlelib/idle_test/test_squeezer.py @@ -396,7 +396,7 @@ def test_expand_dangerous_oupput(self): expandingbutton.base_text = expandingbutton.text # Patch the message box module to always return False. - with patch('idlelib.squeezer.tkMessageBox') as mock_msgbox: + with patch('idlelib.squeezer.messagebox') as mock_msgbox: mock_msgbox.askokcancel.return_value = False mock_msgbox.askyesno.return_value = False # Trigger the expand event. @@ -407,7 +407,7 @@ def test_expand_dangerous_oupput(self): self.assertEqual(expandingbutton.text.get('1.0', 'end-1c'), '') # Patch the message box module to always return True. - with patch('idlelib.squeezer.tkMessageBox') as mock_msgbox: + with patch('idlelib.squeezer.messagebox') as mock_msgbox: mock_msgbox.askokcancel.return_value = True mock_msgbox.askyesno.return_value = True # Trigger the expand event. diff --git a/Lib/idlelib/iomenu.py b/Lib/idlelib/iomenu.py index 8bb2fa6a6e7939..5ebf7089fb9abe 100644 --- a/Lib/idlelib/iomenu.py +++ b/Lib/idlelib/iomenu.py @@ -5,8 +5,8 @@ import tempfile import tokenize -import tkinter.filedialog as tkFileDialog -import tkinter.messagebox as tkMessageBox +from tkinter import filedialog +from tkinter import messagebox from tkinter.simpledialog import askstring import idlelib @@ -147,10 +147,10 @@ def loadfile(self, filename): eol_convention = f.newlines converted = True except OSError as err: - tkMessageBox.showerror("I/O Error", str(err), parent=self.text) + messagebox.showerror("I/O Error", str(err), parent=self.text) return False except UnicodeDecodeError: - tkMessageBox.showerror("Decoding Error", + messagebox.showerror("Decoding Error", "File %s\nFailed to Decode" % filename, parent=self.text) return False @@ -159,7 +159,7 @@ def loadfile(self, filename): # If the file does not contain line separators, it is None. # If the file contains mixed line separators, it is a tuple. if eol_convention is not None: - tkMessageBox.showwarning("Mixed Newlines", + messagebox.showwarning("Mixed Newlines", "Mixed newlines detected.\n" "The file will be changed on save.", parent=self.text) @@ -187,10 +187,10 @@ def maybesave(self): return "yes" message = "Do you want to save %s before closing?" % ( self.filename or "this untitled document") - confirm = tkMessageBox.askyesnocancel( + confirm = messagebox.askyesnocancel( title="Save On Close", message=message, - default=tkMessageBox.YES, + default=messagebox.YES, parent=self.text) if confirm: reply = "yes" @@ -249,7 +249,7 @@ def writefile(self, filename): os.fsync(f.fileno()) return True except OSError as msg: - tkMessageBox.showerror("I/O Error", str(msg), + messagebox.showerror("I/O Error", str(msg), parent=self.text) return False @@ -286,7 +286,7 @@ def encode(self, chars): failed = str(err) except UnicodeEncodeError: failed = "Invalid encoding '%s'" % enc - tkMessageBox.showerror( + messagebox.showerror( "I/O Error", "%s.\nSaving as UTF-8" % failed, parent=self.text) @@ -295,10 +295,10 @@ def encode(self, chars): return chars.encode('utf-8-sig') def print_window(self, event): - confirm = tkMessageBox.askokcancel( + confirm = messagebox.askokcancel( title="Print", message="Print to Default Printer", - default=tkMessageBox.OK, + default=messagebox.OK, parent=self.text) if not confirm: self.text.focus_set() @@ -336,10 +336,10 @@ def print_window(self, event): status + output if output: output = "Printing command: %s\n" % repr(command) + output - tkMessageBox.showerror("Print status", output, parent=self.text) + messagebox.showerror("Print status", output, parent=self.text) else: #no printing for this platform message = "Printing is not enabled for this platform: %s" % platform - tkMessageBox.showinfo("Print status", message, parent=self.text) + messagebox.showinfo("Print status", message, parent=self.text) if tempfilename: os.unlink(tempfilename) return "break" @@ -358,7 +358,7 @@ def print_window(self, event): def askopenfile(self): dir, base = self.defaultfilename("open") if not self.opendialog: - self.opendialog = tkFileDialog.Open(parent=self.text, + self.opendialog = filedialog.Open(parent=self.text, filetypes=self.filetypes) filename = self.opendialog.show(initialdir=dir, initialfile=base) return filename @@ -378,7 +378,7 @@ def defaultfilename(self, mode="open"): def asksavefile(self): dir, base = self.defaultfilename("save") if not self.savedialog: - self.savedialog = tkFileDialog.SaveAs( + self.savedialog = filedialog.SaveAs( parent=self.text, filetypes=self.filetypes, defaultextension=self.defaultextension) diff --git a/Lib/idlelib/pyshell.py b/Lib/idlelib/pyshell.py index 6fa138219a24da..d32106c9838744 100755 --- a/Lib/idlelib/pyshell.py +++ b/Lib/idlelib/pyshell.py @@ -21,13 +21,13 @@ except (ImportError, AttributeError, OSError): pass -import tkinter.messagebox as tkMessageBox +from tkinter import messagebox if TkVersion < 8.5: root = Tk() # otherwise create root in main root.withdraw() from idlelib.run import fix_scaling fix_scaling(root) - tkMessageBox.showerror("Idle Cannot Start", + messagebox.showerror("Idle Cannot Start", "Idle requires tcl/tk 8.5+, not %s." % TkVersion, parent=root) raise SystemExit(1) @@ -261,7 +261,7 @@ def store_file_breaks(self): except OSError as err: if not getattr(self.root, "breakpoint_error_displayed", False): self.root.breakpoint_error_displayed = True - tkMessageBox.showerror(title='IDLE Error', + messagebox.showerror(title='IDLE Error', message='Unable to update breakpoint list:\n%s' % str(err), parent=self.text) @@ -771,7 +771,7 @@ def runcode(self, code): exec(code, self.locals) except SystemExit: if not self.tkconsole.closing: - if tkMessageBox.askyesno( + if messagebox.askyesno( "Exit?", "Do you want to exit altogether?", default="yes", @@ -805,7 +805,7 @@ def write(self, s): return self.tkconsole.stderr.write(s) def display_port_binding_error(self): - tkMessageBox.showerror( + messagebox.showerror( "Port Binding Error", "IDLE can't bind to a TCP/IP port, which is necessary to " "communicate with its Python execution server. This might be " @@ -816,7 +816,7 @@ def display_port_binding_error(self): parent=self.tkconsole.text) def display_no_subprocess_error(self): - tkMessageBox.showerror( + messagebox.showerror( "Subprocess Connection Error", "IDLE's subprocess didn't make connection.\n" "See the 'Startup failure' section of the IDLE doc, online at\n" @@ -824,7 +824,7 @@ def display_no_subprocess_error(self): parent=self.tkconsole.text) def display_executing_dialog(self): - tkMessageBox.showerror( + messagebox.showerror( "Already executing", "The Python Shell window is already executing a command; " "please wait until it is finished.", @@ -945,7 +945,7 @@ def get_warning_stream(self): def toggle_debugger(self, event=None): if self.executing: - tkMessageBox.showerror("Don't debug now", + messagebox.showerror("Don't debug now", "You can only toggle the debugger when idle", parent=self.text) self.set_debugger_indicator() @@ -1003,7 +1003,7 @@ def endexecuting(self): def close(self): "Extend EditorWindow.close()" if self.executing: - response = tkMessageBox.askokcancel( + response = messagebox.askokcancel( "Kill?", "Your program is still running!\n Do you want to kill it?", default="ok", @@ -1254,7 +1254,7 @@ def open_stack_viewer(self, event=None): try: sys.last_traceback except: - tkMessageBox.showerror("No stack trace", + messagebox.showerror("No stack trace", "There is no stack trace yet.\n" "(sys.last_traceback is not defined)", parent=self.text) diff --git a/Lib/idlelib/runscript.py b/Lib/idlelib/runscript.py index 028b0dbd21dfe6..55712e904603f8 100644 --- a/Lib/idlelib/runscript.py +++ b/Lib/idlelib/runscript.py @@ -14,7 +14,7 @@ import time import tokenize -import tkinter.messagebox as tkMessageBox +from tkinter import messagebox from idlelib.config import idleConf from idlelib import macosx @@ -195,15 +195,15 @@ def getfilename(self): def ask_save_dialog(self): msg = "Source Must Be Saved\n" + 5*' ' + "OK to Save?" - confirm = tkMessageBox.askokcancel(title="Save Before Run or Check", + confirm = messagebox.askokcancel(title="Save Before Run or Check", message=msg, - default=tkMessageBox.OK, + default=messagebox.OK, parent=self.editwin.text) return confirm def errorbox(self, title, message): # XXX This should really be a function of EditorWindow... - tkMessageBox.showerror(title, message, parent=self.editwin.text) + messagebox.showerror(title, message, parent=self.editwin.text) self.editwin.text.focus_set() self.perf = time.perf_counter() diff --git a/Lib/idlelib/searchengine.py b/Lib/idlelib/searchengine.py index a50038e282ba6c..eddef581ab40a7 100644 --- a/Lib/idlelib/searchengine.py +++ b/Lib/idlelib/searchengine.py @@ -2,7 +2,7 @@ import re from tkinter import StringVar, BooleanVar, TclError -import tkinter.messagebox as tkMessageBox +from tkinter import messagebox def get(root): '''Return the singleton SearchEngine instance for the process. @@ -96,7 +96,7 @@ def report_error(self, pat, msg, col=None): msg = msg + "\nPattern: " + str(pat) if col is not None: msg = msg + "\nOffset: " + str(col) - tkMessageBox.showerror("Regular expression error", + messagebox.showerror("Regular expression error", msg, master=self.root) def search_text(self, text, prog=None, ok=0): diff --git a/Lib/idlelib/squeezer.py b/Lib/idlelib/squeezer.py index be1538a25fdedf..3046d803b74a4e 100644 --- a/Lib/idlelib/squeezer.py +++ b/Lib/idlelib/squeezer.py @@ -17,7 +17,7 @@ import re import tkinter as tk -import tkinter.messagebox as tkMessageBox +from tkinter import messagebox from idlelib.config import idleConf from idlelib.textview import view_text @@ -147,7 +147,7 @@ def expand(self, event=None): if self.is_dangerous is None: self.set_is_dangerous() if self.is_dangerous: - confirm = tkMessageBox.askokcancel( + confirm = messagebox.askokcancel( title="Expand huge output?", message="\n\n".join([ "The squeezed output is very long: %d lines, %d chars.", @@ -155,7 +155,7 @@ def expand(self, event=None): "It is recommended to view or copy the output instead.", "Really expand?" ]) % (self.numoflines, len(self.s)), - default=tkMessageBox.CANCEL, + default=messagebox.CANCEL, parent=self.text) if not confirm: return "break" From c10180ea1458aa0ffd7793cb75629ebffe8a257e Mon Sep 17 00:00:00 2001 From: Andrey Bienkowski Date: Mon, 25 Jan 2021 21:10:40 +0000 Subject: [PATCH 1954/2163] [3.8] bpo-42384: pdb: correctly populate sys.path[0] (GH-23338) (#24320) --- Lib/pdb.py | 3 +- Lib/test/test_pdb.py | 42 +++++++++++++++++++ .../2020-11-17-14-32-39.bpo-42384.1ZnQSn.rst | 1 + 3 files changed, 45 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2020-11-17-14-32-39.bpo-42384.1ZnQSn.rst diff --git a/Lib/pdb.py b/Lib/pdb.py index d7d957159458be..7a5192cbadc3ad 100755 --- a/Lib/pdb.py +++ b/Lib/pdb.py @@ -1686,8 +1686,9 @@ def main(): sys.argv[:] = args # Hide "pdb.py" and pdb options from argument list - # Replace pdb's dir with script's dir in front of module search path. if not run_as_module: + mainpyfile = os.path.realpath(mainpyfile) + # Replace pdb's dir with script's dir in front of module search path. sys.path[0] = os.path.dirname(mainpyfile) # Note on saving/restoring sys.argv: it's a good idea when sys.argv was diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py index 9c9471a8cc8df3..74448747e3e1b5 100644 --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -1658,6 +1658,48 @@ def test_errors_in_command(self): '(Pdb) ', ]) + + def test_issue42384(self): + '''When running `python foo.py` sys.path[0] is an absolute path. `python -m pdb foo.py` should behave the same''' + script = textwrap.dedent(""" + import sys + print('sys.path[0] is', sys.path[0]) + """) + commands = 'c\nq' + + with support.temp_cwd() as cwd: + expected = f'(Pdb) sys.path[0] is {os.path.realpath(cwd)}' + + stdout, stderr = self.run_pdb_script(script, commands) + + self.assertEqual(stdout.split('\n')[2].rstrip('\r'), expected) + + @support.skip_unless_symlink + def test_issue42384_symlink(self): + '''When running `python foo.py` sys.path[0] resolves symlinks. `python -m pdb foo.py` should behave the same''' + script = textwrap.dedent(""" + import sys + print('sys.path[0] is', sys.path[0]) + """) + commands = 'c\nq' + + with support.temp_cwd() as cwd: + cwd = os.path.realpath(cwd) + dir_one = os.path.join(cwd, 'dir_one') + dir_two = os.path.join(cwd, 'dir_two') + expected = f'(Pdb) sys.path[0] is {dir_one}' + + os.mkdir(dir_one) + with open(os.path.join(dir_one, 'foo.py'), 'w') as f: + f.write(script) + os.mkdir(dir_two) + os.symlink(os.path.join(dir_one, 'foo.py'), os.path.join(dir_two, 'foo.py')) + + stdout, stderr = self._run_pdb([os.path.join('dir_two', 'foo.py')], commands) + + self.assertEqual(stdout.split('\n')[2].rstrip('\r'), expected) + + def load_tests(*args): from test import test_pdb suites = [ diff --git a/Misc/NEWS.d/next/Library/2020-11-17-14-32-39.bpo-42384.1ZnQSn.rst b/Misc/NEWS.d/next/Library/2020-11-17-14-32-39.bpo-42384.1ZnQSn.rst new file mode 100644 index 00000000000000..ae990162fbab75 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-11-17-14-32-39.bpo-42384.1ZnQSn.rst @@ -0,0 +1 @@ +Make pdb populate sys.path[0] exactly the same as regular python execution. From d863deeff27f00ac33e9f169d23ca56d69af3f86 Mon Sep 17 00:00:00 2001 From: Andrey Bienkowski Date: Tue, 26 Jan 2021 15:58:33 +0000 Subject: [PATCH 1955/2163] [3.8] bpo-42383: pdb: do not fail to restart the target if the current directory changed (GH-23412) (#24323) --- Lib/test/test_pdb.py | 23 +++++++++++++++++++ .../2020-11-17-14-30-12.bpo-42383.ubl0Y_.rst | 2 ++ 2 files changed, 25 insertions(+) create mode 100644 Misc/NEWS.d/next/Library/2020-11-17-14-30-12.bpo-42383.ubl0Y_.rst diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py index 74448747e3e1b5..f77c355077411b 100644 --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -1699,6 +1699,29 @@ def test_issue42384_symlink(self): self.assertEqual(stdout.split('\n')[2].rstrip('\r'), expected) + def test_issue42383(self): + with support.temp_cwd() as cwd: + with open('foo.py', 'w') as f: + s = textwrap.dedent(""" + print('The correct file was executed') + + import os + os.chdir("subdir") + """) + f.write(s) + + subdir = os.path.join(cwd, 'subdir') + os.mkdir(subdir) + os.mkdir(os.path.join(subdir, 'subdir')) + wrong_file = os.path.join(subdir, 'foo.py') + + with open(wrong_file, 'w') as f: + f.write('print("The wrong file was executed")') + + stdout, stderr = self._run_pdb(['foo.py'], 'c\nc\nq') + expected = '(Pdb) The correct file was executed' + self.assertEqual(stdout.split('\n')[6].rstrip('\r'), expected) + def load_tests(*args): from test import test_pdb diff --git a/Misc/NEWS.d/next/Library/2020-11-17-14-30-12.bpo-42383.ubl0Y_.rst b/Misc/NEWS.d/next/Library/2020-11-17-14-30-12.bpo-42383.ubl0Y_.rst new file mode 100644 index 00000000000000..ccf2106f28a93d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-11-17-14-30-12.bpo-42383.ubl0Y_.rst @@ -0,0 +1,2 @@ +Fix pdb: previously pdb would fail to restart the debugging target if it was +specified using a relative path and the current directory changed. From a178473dd6a3b527a67ab3f9dc38ef8b1a60881a Mon Sep 17 00:00:00 2001 From: Terry Jan Reedy Date: Tue, 26 Jan 2021 13:59:42 -0500 Subject: [PATCH 1956/2163] [3.8] bpo-41841: Prepare IDLE NEWS for 3.8.8 (GH-24344) --- Lib/idlelib/NEWS.txt | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 44782fcb0cecd9..c122e596b5cb16 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -1,5 +1,5 @@ -What's New in IDLE 3.8.7 -Released on 2020-12-?? +What's New in IDLE 3.8.8 +Released on 2021-02-15? ====================================== @@ -12,6 +12,10 @@ bpo-42508: Keep IDLE running on macOS. Remove obsolete workaround that prevented running files with shortcuts when using new universal2 installers built on macOS 11. +What's New in IDLE 3.8.7 +Released on 2020-12-27 +====================================== + bpo-42426: Fix reporting offset of the RE error in searchengine. bpo-42416: Get docstrings for IDLE calltips more often From 6f0346d09f78180886982e070cc92425ada96144 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 26 Jan 2021 16:24:30 -0800 Subject: [PATCH 1957/2163] bpo-43008: Make IDLE respect sys.excepthook (GH-24302) Co-authored-by: Terry Jan Reedy (cherry picked from commit 7a34380ad788886f5ad50d4175ceb2d5715b8cff) Co-authored-by: Ken --- Doc/library/idle.rst | 2 +- Lib/idlelib/NEWS.txt | 3 ++ Lib/idlelib/idle_test/test_run.py | 43 +++++++++++++++++-- Lib/idlelib/run.py | 27 ++++++++---- .../2021-01-26-18-12-17.bpo-43008.mbQUc7.rst | 1 + 5 files changed, 63 insertions(+), 13 deletions(-) create mode 100644 Misc/NEWS.d/next/IDLE/2021-01-26-18-12-17.bpo-43008.mbQUc7.rst diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst index a59a5d3a465703..e7eaabd8bfa25a 100644 --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -250,7 +250,7 @@ View Last Restart Scroll the shell window to the last Shell restart. Restart Shell - Restart the shell to clean the environment. + Restart the shell to clean the environment and reset display and exception handling. Previous History Cycle through earlier commands in history which match the current entry. diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index c122e596b5cb16..0affc55da8f0ad 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,9 @@ Released on 2021-02-15? ====================================== +bpo-43008: Make IDLE invoke :func:`sys.excepthook` in normal, +2-process mode. + bpo-33065: Fix problem debugging user classes with __repr__ method. bpo-32631: Finish zzdummy example extension module: make menu entries diff --git a/Lib/idlelib/idle_test/test_run.py b/Lib/idlelib/idle_test/test_run.py index 37c0d4525e56cd..a31671ee0485fa 100644 --- a/Lib/idlelib/idle_test/test_run.py +++ b/Lib/idlelib/idle_test/test_run.py @@ -1,16 +1,18 @@ "Test run, coverage 49%." from idlelib import run +import io +import sys +from test.support import captured_output, captured_stderr import unittest from unittest import mock +import idlelib from idlelib.idle_test.mock_idle import Func -from test.support import captured_output, captured_stderr -import io -import sys +idlelib.testing = True # Use {} for executing test user code. -class RunTest(unittest.TestCase): +class PrintExceptionTest(unittest.TestCase): def test_print_exception_unhashable(self): class UnhashableException(Exception): @@ -351,5 +353,38 @@ def test_fatal_error(self): self.assertIn('IndexError', msg) eq(func.called, 2) + +class ExecRuncodeTest(unittest.TestCase): + + @classmethod + def setUpClass(cls): + cls.addClassCleanup(setattr,run,'print_exception',run.print_exception) + cls.prt = Func() # Need reference. + run.print_exception = cls.prt + mockrpc = mock.Mock() + mockrpc.console.getvar = Func(result=False) + cls.ex = run.Executive(mockrpc) + + @classmethod + def tearDownClass(cls): + assert sys.excepthook == sys.__excepthook__ + + def test_exceptions(self): + ex = self.ex + ex.runcode('1/0') + self.assertIs(ex.user_exc_info[0], ZeroDivisionError) + + self.addCleanup(setattr, sys, 'excepthook', sys.__excepthook__) + sys.excepthook = lambda t, e, tb: run.print_exception(t) + ex.runcode('1/0') + self.assertIs(self.prt.args[0], ZeroDivisionError) + + sys.excepthook = lambda: None + ex.runcode('1/0') + t, e, tb = ex.user_exc_info + self.assertIs(t, TypeError) + self.assertTrue(isinstance(e.__context__, ZeroDivisionError)) + + if __name__ == '__main__': unittest.main(verbosity=2) diff --git a/Lib/idlelib/run.py b/Lib/idlelib/run.py index ec575c3d483631..07e9a2bf9ceeae 100644 --- a/Lib/idlelib/run.py +++ b/Lib/idlelib/run.py @@ -16,6 +16,7 @@ import threading import warnings +import idlelib # testing from idlelib import autocomplete # AutoComplete, fetch_encodings from idlelib import calltip # Calltip from idlelib import debugger_r # start_debugger @@ -542,14 +543,17 @@ class Executive: def __init__(self, rpchandler): self.rpchandler = rpchandler - self.locals = __main__.__dict__ - self.calltip = calltip.Calltip() - self.autocomplete = autocomplete.AutoComplete() + if idlelib.testing is False: + self.locals = __main__.__dict__ + self.calltip = calltip.Calltip() + self.autocomplete = autocomplete.AutoComplete() + else: + self.locals = {} def runcode(self, code): global interruptable try: - self.usr_exc_info = None + self.user_exc_info = None interruptable = True try: exec(code, self.locals) @@ -562,10 +566,17 @@ def runcode(self, code): print('SystemExit: ' + str(ob), file=sys.stderr) # Return to the interactive prompt. except: - self.usr_exc_info = sys.exc_info() + self.user_exc_info = sys.exc_info() # For testing, hook, viewer. if quitting: exit() - print_exception() + if sys.excepthook is sys.__excepthook__: + print_exception() + else: + try: + sys.excepthook(*self.user_exc_info) + except: + self.user_exc_info = sys.exc_info() # For testing. + print_exception() jit = self.rpchandler.console.getvar("<>") if jit: self.rpchandler.interp.open_remote_stack_viewer() @@ -590,8 +601,8 @@ def get_the_completion_list(self, what, mode): return self.autocomplete.fetch_completions(what, mode) def stackviewer(self, flist_oid=None): - if self.usr_exc_info: - typ, val, tb = self.usr_exc_info + if self.user_exc_info: + typ, val, tb = self.user_exc_info else: return None flist = None diff --git a/Misc/NEWS.d/next/IDLE/2021-01-26-18-12-17.bpo-43008.mbQUc7.rst b/Misc/NEWS.d/next/IDLE/2021-01-26-18-12-17.bpo-43008.mbQUc7.rst new file mode 100644 index 00000000000000..3e0b80a909d728 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2021-01-26-18-12-17.bpo-43008.mbQUc7.rst @@ -0,0 +1 @@ +Make IDLE invoke :func:`sys.excepthook` in normal, 2-process mode. From 81b23a9b148b60f8f541e4b06e1d88c58644e3b8 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 28 Jan 2021 06:54:57 -0800 Subject: [PATCH 1958/2163] bpo-29076: Add fish support to macOS installer (GH-23302) (GH-23938) (cherry picked from commit 7f162e867c674f57c308a87fffcdcca3540c8933) Co-authored-by: Erlend Egeberg Aasland --- .../scripts/postflight.patch-profile | 18 +++++++++++++++++- .../2020-02-28-14-33-15.bpo-29076.Gtixi5.rst | 1 + 2 files changed, 18 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Build/2020-02-28-14-33-15.bpo-29076.Gtixi5.rst diff --git a/Mac/BuildScript/scripts/postflight.patch-profile b/Mac/BuildScript/scripts/postflight.patch-profile index 0a62e327f51b38..68b8e4bb044e10 100755 --- a/Mac/BuildScript/scripts/postflight.patch-profile +++ b/Mac/BuildScript/scripts/postflight.patch-profile @@ -20,7 +20,7 @@ fi # Make sure the directory ${PYTHON_ROOT}/bin is on the users PATH. BSH="`basename "${theShell}"`" case "${BSH}" in -bash|ksh|sh|*csh|zsh) +bash|ksh|sh|*csh|zsh|fish) if [ `id -ur` = 0 ]; then P=`su - ${USER} -c 'echo A-X-4-X@@$PATH@@X-4-X-A' | grep 'A-X-4-X@@.*@@X-4-X-A' | sed -e 's/^A-X-4-X@@//g' -e 's/@@X-4-X-A$//g'` else @@ -76,6 +76,22 @@ bash) PR="${HOME}/.bash_profile" fi ;; +fish) + CONFIG_DIR="${HOME}/.config/fish" + RC="${CONFIG_DIR}/config.fish" + mkdir -p "$CONFIG_DIR" + if [ -f "${RC}" ]; then + cp -fp "${RC}" "${RC}.pysave" + fi + echo "" >> "${RC}" + echo "# Setting PATH for Python ${PYVER}" >> "${RC}" + echo "# The original version is saved in ${RC}.pysave" >> "${RC}" + echo "set -x PATH \"${PYTHON_ROOT}/bin\" \"\$PATH\"" >> "${RC}" + if [ `id -ur` = 0 ]; then + chown "${USER}" "${RC}" + fi + exit 0 + ;; zsh) PR="${HOME}/.zprofile" ;; diff --git a/Misc/NEWS.d/next/Build/2020-02-28-14-33-15.bpo-29076.Gtixi5.rst b/Misc/NEWS.d/next/Build/2020-02-28-14-33-15.bpo-29076.Gtixi5.rst new file mode 100644 index 00000000000000..b38beb0586951c --- /dev/null +++ b/Misc/NEWS.d/next/Build/2020-02-28-14-33-15.bpo-29076.Gtixi5.rst @@ -0,0 +1 @@ +Add fish shell support to macOS installer. From 901a9834420e516c07d6cad356d2be481db6d8d1 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 28 Jan 2021 17:47:41 -0800 Subject: [PATCH 1959/2163] bpo-23544: Disable IDLE Stack Viewer when running user code (GH-17163) (#24366) Starting stack viewer when user code is running, including when Debugger is active, hangs or crashes IDLE. Co-authored-by: Zackery Spytz Co-authored-by: Terry Jan Reedy (cherry picked from commit 23a567c11ca36eedde0e119443c85cc16075deaf) --- Lib/idlelib/NEWS.txt | 3 +++ Lib/idlelib/codecontext.py | 2 +- Lib/idlelib/editor.py | 10 +++++---- Lib/idlelib/idle_test/test_mainmenu.py | 21 +++++++++++++++++++ Lib/idlelib/pyshell.py | 4 ++++ .../2019-11-14-23-41-07.bpo-23544.3etemb.rst | 2 ++ 6 files changed, 37 insertions(+), 5 deletions(-) create mode 100644 Misc/NEWS.d/next/IDLE/2019-11-14-23-41-07.bpo-23544.3etemb.rst diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 0affc55da8f0ad..522ce59f616923 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,9 @@ Released on 2021-02-15? ====================================== +bpo-23544: Disable Debug=>Stack Viewer when user code is running or +Debugger is active, to prevent hang or crash. Patch by Zackery Spytz. + bpo-43008: Make IDLE invoke :func:`sys.excepthook` in normal, 2-process mode. diff --git a/Lib/idlelib/codecontext.py b/Lib/idlelib/codecontext.py index eb19773f56e34b..f2f44f5f8d4e61 100644 --- a/Lib/idlelib/codecontext.py +++ b/Lib/idlelib/codecontext.py @@ -142,7 +142,7 @@ def toggle_code_context_event(self, event=None): self.text.after_cancel(self.t1) self._reset() menu_status = 'Show' - self.editwin.update_menu_label(menu='options', index='* Code Context', + self.editwin.update_menu_label(menu='options', index='*ode*ontext', label=f'{menu_status} Code Context') return "break" diff --git a/Lib/idlelib/editor.py b/Lib/idlelib/editor.py index 66e9da5a9dccf9..b9cb50264ff06f 100644 --- a/Lib/idlelib/editor.py +++ b/Lib/idlelib/editor.py @@ -339,7 +339,7 @@ def __init__(self, flist=None, filename=None, key=None, root=None): text.bind("<>", self.code_context.toggle_code_context_event) else: - self.update_menu_state('options', '*Code Context', 'disabled') + self.update_menu_state('options', '*ode*ontext', 'disabled') if self.allow_line_numbers: self.line_numbers = self.LineNumbers(self) if idleConf.GetOption('main', 'EditorWindow', @@ -347,7 +347,7 @@ def __init__(self, flist=None, filename=None, key=None, root=None): self.toggle_line_numbers_event() text.bind("<>", self.toggle_line_numbers_event) else: - self.update_menu_state('options', '*Line Numbers', 'disabled') + self.update_menu_state('options', '*ine*umbers', 'disabled') def handle_winconfig(self, event=None): self.set_width() @@ -450,7 +450,9 @@ def createmenubar(self): self.menudict = menudict = {} for name, label in self.menu_specs: underline, label = prepstr(label) - menudict[name] = menu = Menu(mbar, name=name, tearoff=0) + postcommand = getattr(self, f'{name}_menu_postcommand', None) + menudict[name] = menu = Menu(mbar, name=name, tearoff=0, + postcommand=postcommand) mbar.add_cascade(label=label, menu=menu, underline=underline) if macosx.isCarbonTk(): # Insert the application menu @@ -1527,7 +1529,7 @@ def toggle_line_numbers_event(self, event=None): else: self.line_numbers.show_sidebar() menu_label = "Hide" - self.update_menu_label(menu='options', index='*Line Numbers', + self.update_menu_label(menu='options', index='*ine*umbers', label=f'{menu_label} Line Numbers') # "line.col" -> line, as an int diff --git a/Lib/idlelib/idle_test/test_mainmenu.py b/Lib/idlelib/idle_test/test_mainmenu.py index 7ec0368371c7df..51d2accfe48a1c 100644 --- a/Lib/idlelib/idle_test/test_mainmenu.py +++ b/Lib/idlelib/idle_test/test_mainmenu.py @@ -2,6 +2,7 @@ # Reported as 88%; mocking turtledemo absence would have no point. from idlelib import mainmenu +import re import unittest @@ -16,6 +17,26 @@ def test_menudefs(self): def test_default_keydefs(self): self.assertGreaterEqual(len(mainmenu.default_keydefs), 50) + def test_tcl_indexes(self): + # Test tcl patterns used to find menuitem to alter. + # On failure, change pattern here and in function(s). + # Patterns here have '.*' for re instead of '*' for tcl. + for menu, pattern in ( + ('debug', '.*tack.*iewer'), # PyShell.debug_menu_postcommand + ('options', '.*ode.*ontext'), # EW.__init__, CodeContext.toggle... + ('options', '.*ine.*umbers'), # EW.__init__, EW.toggle...event. + ): + with self.subTest(menu=menu, pattern=pattern): + for menutup in mainmenu.menudefs: + if menutup[0] == menu: + break + else: + self.assertTrue(0, f"{menu} not in menudefs") + self.assertTrue(any(re.search(pattern, menuitem[0]) + for menuitem in menutup[1] + if menuitem is not None), # Separator. + f"{pattern} not in {menu}") + if __name__ == '__main__': unittest.main(verbosity=2) diff --git a/Lib/idlelib/pyshell.py b/Lib/idlelib/pyshell.py index d32106c9838744..fea3762461e99e 100755 --- a/Lib/idlelib/pyshell.py +++ b/Lib/idlelib/pyshell.py @@ -989,6 +989,10 @@ def open_debugger(self): self.showprompt() self.set_debugger_indicator() + def debug_menu_postcommand(self): + state = 'disabled' if self.executing else 'normal' + self.update_menu_state('debug', '*tack*iewer', state) + def beginexecuting(self): "Helper for ModifiedInterpreter" self.resetoutput() diff --git a/Misc/NEWS.d/next/IDLE/2019-11-14-23-41-07.bpo-23544.3etemb.rst b/Misc/NEWS.d/next/IDLE/2019-11-14-23-41-07.bpo-23544.3etemb.rst new file mode 100644 index 00000000000000..eb4a56bf100b59 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2019-11-14-23-41-07.bpo-23544.3etemb.rst @@ -0,0 +1,2 @@ +Disable Debug=>Stack Viewer when user code is running or Debugger +is active, to prevent hang or crash. Patch by Zackery Spytz. From cf883827496d0fbe2c5fe39e4778cd7525f6f7bc Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 29 Jan 2021 10:22:18 -0800 Subject: [PATCH 1960/2163] bpo-43008: Add 'Patch by Ken Hilton' (GH-24370) (cherry picked from commit 11d75ec807f05eff1148c049e38b808d11c23b8a) Co-authored-by: Terry Jan Reedy --- Lib/idlelib/NEWS.txt | 2 +- Misc/NEWS.d/next/IDLE/2021-01-26-18-12-17.bpo-43008.mbQUc7.rst | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 522ce59f616923..cf80b9efe23890 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -7,7 +7,7 @@ bpo-23544: Disable Debug=>Stack Viewer when user code is running or Debugger is active, to prevent hang or crash. Patch by Zackery Spytz. bpo-43008: Make IDLE invoke :func:`sys.excepthook` in normal, -2-process mode. +2-process mode. Patch by Ken Hilton. bpo-33065: Fix problem debugging user classes with __repr__ method. diff --git a/Misc/NEWS.d/next/IDLE/2021-01-26-18-12-17.bpo-43008.mbQUc7.rst b/Misc/NEWS.d/next/IDLE/2021-01-26-18-12-17.bpo-43008.mbQUc7.rst index 3e0b80a909d728..55ab67ca94959a 100644 --- a/Misc/NEWS.d/next/IDLE/2021-01-26-18-12-17.bpo-43008.mbQUc7.rst +++ b/Misc/NEWS.d/next/IDLE/2021-01-26-18-12-17.bpo-43008.mbQUc7.rst @@ -1 +1,2 @@ Make IDLE invoke :func:`sys.excepthook` in normal, 2-process mode. +Patch by Ken Hilton. From c644db441f82be6d4167021eb0615675d0fc1d15 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 29 Jan 2021 13:39:03 -0800 Subject: [PATCH 1961/2163] Fixed typo in turtle.rst (GH-24371) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found it while translating it to french 🤷 Automerge-Triggered-By: GH:JulienPalard (cherry picked from commit 6372a4ceba126aa0a9f00eee0f8023308f13e77b) Co-authored-by: Jules Lasne --- Doc/library/turtle.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index 2084d75b3a57a0..7bd7e966109124 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -662,7 +662,7 @@ Tell Turtle's state Return the angle between the line from turtle position to position specified by (x,y), the vector or the other turtle. This depends on the turtle's start - orientation which depends on the mode - "standard"/"world" or "logo"). + orientation which depends on the mode - "standard"/"world" or "logo". .. doctest:: :skipif: _tkinter is None From 0b93a558f566a7a9d699a91ad529b3e8969ffa5f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 29 Jan 2021 13:49:57 -0800 Subject: [PATCH 1962/2163] Fixing typos in turtle.rst (GH-24376) Automerge-Triggered-By: GH:JulienPalard (cherry picked from commit 6baaae589d596ed3bb668448f2a22c5f62fc5fdf) Co-authored-by: Jules Lasne --- Doc/library/turtle.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index 7bd7e966109124..6a9d61916ad1a5 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -1105,7 +1105,7 @@ More drawing control :param font: a triple (fontname, fontsize, fonttype) Write text - the string representation of *arg* - at the current turtle - position according to *align* ("left", "center" or right") and with the given + position according to *align* ("left", "center" or "right") and with the given font. If *move* is true, the pen is moved to the bottom-right corner of the text. By default, *move* is ``False``. @@ -1192,7 +1192,7 @@ Appearance :func:`shapesize`. - "noresize": no adaption of the turtle's appearance takes place. - resizemode("user") is called by :func:`shapesize` when used with arguments. + ``resizemode("user")`` is called by :func:`shapesize` when used with arguments. .. doctest:: :skipif: _tkinter is None @@ -1330,7 +1330,7 @@ Appearance matrix as a tuple of 4 elements. Otherwise set the given elements and transform the turtleshape according to the matrix consisting of first row t11, t12 and - second row t21, 22. The determinant t11 * t22 - t12 * t21 must not be + second row t21, t22. The determinant t11 * t22 - t12 * t21 must not be zero, otherwise an error is raised. Modify stretchfactor, shearfactor and tiltangle according to the given matrix. @@ -1513,7 +1513,7 @@ Special Turtle methods :param size: an integer or ``None`` - Set or disable undobuffer. If *size* is an integer an empty undobuffer of + Set or disable undobuffer. If *size* is an integer, an empty undobuffer of given size is installed. *size* gives the maximum number of turtle actions that can be undone by the :func:`undo` method/function. If *size* is ``None``, the undobuffer is disabled. @@ -1821,7 +1821,7 @@ Using screen events existing bindings are removed. Example for a TurtleScreen instance named ``screen`` and a Turtle instance - named turtle: + named ``turtle``: .. doctest:: :skipif: _tkinter is None @@ -2048,7 +2048,7 @@ Methods specific to Screen, not inherited from TurtleScreen .. function:: exitonclick() - Bind bye() method to mouse clicks on the Screen. + Bind ``bye()`` method to mouse clicks on the Screen. If the value "using_IDLE" in the configuration dictionary is ``False`` From de76ce540ed013a31d4c80bca2587a2dd7c23652 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 30 Jan 2021 21:21:54 -0800 Subject: [PATCH 1963/2163] bpo-43059: Remove reference to legacy external sqlite3 repository (GH-24364) (cherry picked from commit e60344364245a23a7a1b25d5ebce6833652a656c) Co-authored-by: L <3177243+LukeLR@users.noreply.github.com> --- Doc/library/sqlite3.rst | 4 ---- 1 file changed, 4 deletions(-) diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index e2e7312c277357..92d5ffaf3971c7 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -99,10 +99,6 @@ This example uses the iterator form:: .. seealso:: - https://github.com/ghaering/pysqlite - The pysqlite web page -- sqlite3 is developed externally under the name - "pysqlite". - https://www.sqlite.org The SQLite web page; the documentation describes the syntax and the available data types for the supported SQL dialect. From 229ef39bcc14ee7838968dfda51d045304e5cf39 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 31 Jan 2021 06:46:17 -0800 Subject: [PATCH 1964/2163] bpo-43030: Fixed a compiler warning in Py_UNICODE_ISSPACE with signed wchar_t (GH-24350) (GH-24397) (cherry picked from commit 42b1806af90b86ec393ca7da14e99ce95ec6c53b) Co-authored-by: Serhiy Storchaka --- Include/cpython/unicodeobject.h | 2 +- Misc/NEWS.d/next/C API/2021-01-27-10-27-47.bpo-43030.loDcD_.rst | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/C API/2021-01-27-10-27-47.bpo-43030.loDcD_.rst diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index 54a13e32ba22b5..87ff31ddbc101c 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -22,7 +22,7 @@ extern "C" { */ #define Py_UNICODE_ISSPACE(ch) \ - ((ch) < 128U ? _Py_ascii_whitespace[(ch)] : _PyUnicode_IsWhitespace(ch)) + ((Py_UCS4)(ch) < 128U ? _Py_ascii_whitespace[(ch)] : _PyUnicode_IsWhitespace(ch)) #define Py_UNICODE_ISLOWER(ch) _PyUnicode_IsLowercase(ch) #define Py_UNICODE_ISUPPER(ch) _PyUnicode_IsUppercase(ch) diff --git a/Misc/NEWS.d/next/C API/2021-01-27-10-27-47.bpo-43030.loDcD_.rst b/Misc/NEWS.d/next/C API/2021-01-27-10-27-47.bpo-43030.loDcD_.rst new file mode 100644 index 00000000000000..7a432522db8a12 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-01-27-10-27-47.bpo-43030.loDcD_.rst @@ -0,0 +1,2 @@ +Fixed a compiler warning in :c:func:`Py_UNICODE_ISSPACE()` on platforms with +signed ``wchar_t``. From 7ca947e93bff2b9d78312dc8ed8e0a2b7d25d3f3 Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Sun, 31 Jan 2021 19:44:12 +0200 Subject: [PATCH 1965/2163] [3.8] bpo-43016: Rewrite tests for curses (GH-24312). (GH-24399) (GH-24401) (cherry picked from commit d64fd4bb5bb4fd2e3277f39d3ad99b5a8d193e1b). (cherry picked from commit e9d4960d15c5282904cf26e469ce7cee39f634f7) --- Lib/test/test_curses.py | 985 +++++++++++++++++++++++++++++++--------- 1 file changed, 771 insertions(+), 214 deletions(-) diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py index 8991134e080635..f47c9876eed88f 100644 --- a/Lib/test/test_curses.py +++ b/Lib/test/test_curses.py @@ -1,18 +1,9 @@ -# -# Test script for the curses module -# -# This script doesn't actually display anything very coherent. but it -# does call (nearly) every method and function. -# -# Functions not tested: {def,reset}_{shell,prog}_mode, getch(), getstr() -# Only called, not tested: getmouse(), ungetmouse() -# - +import functools +import inspect import os import string import sys import tempfile -import functools import unittest from test.support import requires, import_module, verbose, SaveSignals @@ -20,7 +11,6 @@ # Optionally test curses module. This currently requires that the # 'curses' resource be given on the regrtest command line using the -u # option. If not available, nothing after this line will be executed. -import inspect requires('curses') # If either of these don't exist, skip the tests. @@ -36,6 +26,17 @@ def requires_curses_func(name): return unittest.skipUnless(hasattr(curses, name), 'requires curses.%s' % name) +def requires_curses_window_meth(name): + def deco(test): + @functools.wraps(test) + def wrapped(self, *args, **kwargs): + if not hasattr(self.stdscr, name): + raise unittest.SkipTest('requires curses.window.%s' % name) + test(self, *args, **kwargs) + return wrapped + return deco + + def requires_colors(test): @functools.wraps(test) def wrapped(self, *args, **kwargs): @@ -110,208 +111,716 @@ def setUp(self): curses.savetty() self.addCleanup(curses.endwin) self.addCleanup(curses.resetty) + self.stdscr.erase() + + @requires_curses_func('filter') + def test_filter(self): + # TODO: Should be called before initscr() or newterm() are called. + # TODO: nofilter() + curses.filter() + + @requires_curses_func('use_env') + def test_use_env(self): + # TODO: Should be called before initscr() or newterm() are called. + # TODO: use_tioctl() + curses.use_env(False) + curses.use_env(True) + + def test_create_windows(self): + win = curses.newwin(5, 10) + self.assertEqual(win.getbegyx(), (0, 0)) + self.assertEqual(win.getparyx(), (-1, -1)) + self.assertEqual(win.getmaxyx(), (5, 10)) + + win = curses.newwin(10, 15, 2, 5) + self.assertEqual(win.getbegyx(), (2, 5)) + self.assertEqual(win.getparyx(), (-1, -1)) + self.assertEqual(win.getmaxyx(), (10, 15)) + + win2 = win.subwin(3, 7) + self.assertEqual(win2.getbegyx(), (3, 7)) + self.assertEqual(win2.getparyx(), (1, 2)) + self.assertEqual(win2.getmaxyx(), (9, 13)) + + win2 = win.subwin(5, 10, 3, 7) + self.assertEqual(win2.getbegyx(), (3, 7)) + self.assertEqual(win2.getparyx(), (1, 2)) + self.assertEqual(win2.getmaxyx(), (5, 10)) + + win3 = win.derwin(2, 3) + self.assertEqual(win3.getbegyx(), (4, 8)) + self.assertEqual(win3.getparyx(), (2, 3)) + self.assertEqual(win3.getmaxyx(), (8, 12)) + + win3 = win.derwin(6, 11, 2, 3) + self.assertEqual(win3.getbegyx(), (4, 8)) + self.assertEqual(win3.getparyx(), (2, 3)) + self.assertEqual(win3.getmaxyx(), (6, 11)) + + win.mvwin(0, 1) + self.assertEqual(win.getbegyx(), (0, 1)) + self.assertEqual(win.getparyx(), (-1, -1)) + self.assertEqual(win.getmaxyx(), (10, 15)) + self.assertEqual(win2.getbegyx(), (3, 7)) + self.assertEqual(win2.getparyx(), (1, 2)) + self.assertEqual(win2.getmaxyx(), (5, 10)) + self.assertEqual(win3.getbegyx(), (4, 8)) + self.assertEqual(win3.getparyx(), (2, 3)) + self.assertEqual(win3.getmaxyx(), (6, 11)) + + win2.mvderwin(2, 1) + self.assertEqual(win2.getbegyx(), (3, 7)) + self.assertEqual(win2.getparyx(), (2, 1)) + self.assertEqual(win2.getmaxyx(), (5, 10)) + + win3.mvderwin(2, 1) + self.assertEqual(win3.getbegyx(), (4, 8)) + self.assertEqual(win3.getparyx(), (2, 1)) + self.assertEqual(win3.getmaxyx(), (6, 11)) + + def test_move_cursor(self): + stdscr = self.stdscr + win = stdscr.subwin(10, 15, 2, 5) + stdscr.move(1, 2) + win.move(2, 4) + self.assertEqual(stdscr.getyx(), (1, 2)) + self.assertEqual(win.getyx(), (2, 4)) + + win.cursyncup() + self.assertEqual(stdscr.getyx(), (4, 9)) - def test_window_funcs(self): - "Test the methods of windows" + def test_refresh_control(self): stdscr = self.stdscr - win = curses.newwin(10,10) - win = curses.newwin(5,5, 5,5) - win2 = curses.newwin(15,15, 5,5) - - for meth in [stdscr.addch, stdscr.addstr]: - for args in [('a',), ('a', curses.A_BOLD), - (4,4, 'a'), (5,5, 'a', curses.A_BOLD)]: - with self.subTest(meth=meth.__qualname__, args=args): - meth(*args) - - for meth in [stdscr.clear, stdscr.clrtobot, - stdscr.clrtoeol, stdscr.cursyncup, stdscr.delch, - stdscr.deleteln, stdscr.erase, stdscr.getbegyx, - stdscr.getbkgd, stdscr.getmaxyx, - stdscr.getparyx, stdscr.getyx, stdscr.inch, - stdscr.insertln, stdscr.instr, stdscr.is_wintouched, - win.noutrefresh, stdscr.redrawwin, stdscr.refresh, - stdscr.standout, stdscr.standend, stdscr.syncdown, - stdscr.syncup, stdscr.touchwin, stdscr.untouchwin]: - with self.subTest(meth=meth.__qualname__): - meth() - - stdscr.addnstr('1234', 3) - stdscr.addnstr('1234', 3, curses.A_BOLD) - stdscr.addnstr(4,4, '1234', 3) - stdscr.addnstr(5,5, '1234', 3, curses.A_BOLD) - - stdscr.attron(curses.A_BOLD) - stdscr.attroff(curses.A_BOLD) - stdscr.attrset(curses.A_BOLD) - stdscr.bkgd(' ') - stdscr.bkgd(' ', curses.A_REVERSE) - stdscr.bkgdset(' ') - stdscr.bkgdset(' ', curses.A_REVERSE) + # touchwin()/untouchwin()/is_wintouched() + stdscr.refresh() + self.assertIs(stdscr.is_wintouched(), False) + stdscr.touchwin() + self.assertIs(stdscr.is_wintouched(), True) + stdscr.refresh() + self.assertIs(stdscr.is_wintouched(), False) + stdscr.touchwin() + self.assertIs(stdscr.is_wintouched(), True) + stdscr.untouchwin() + self.assertIs(stdscr.is_wintouched(), False) + + # touchline()/untouchline()/is_linetouched() + stdscr.touchline(5, 2) + self.assertIs(stdscr.is_linetouched(5), True) + self.assertIs(stdscr.is_linetouched(6), True) + self.assertIs(stdscr.is_wintouched(), True) + stdscr.touchline(5, 1, False) + self.assertIs(stdscr.is_linetouched(5), False) + + # syncup() + win = stdscr.subwin(10, 15, 2, 5) + win2 = win.subwin(5, 10, 3, 7) + win2.touchwin() + stdscr.untouchwin() + win2.syncup() + self.assertIs(win.is_wintouched(), True) + self.assertIs(stdscr.is_wintouched(), True) + + # syncdown() + stdscr.touchwin() + win.untouchwin() + win2.untouchwin() + win2.syncdown() + self.assertIs(win2.is_wintouched(), True) + + # syncok() + if hasattr(stdscr, 'syncok') and not sys.platform.startswith("sunos"): + win.untouchwin() + stdscr.untouchwin() + for syncok in [False, True]: + win2.syncok(syncok) + win2.addch('a') + self.assertIs(win.is_wintouched(), syncok) + self.assertIs(stdscr.is_wintouched(), syncok) + + def test_output_character(self): + stdscr = self.stdscr + # addch() + stdscr.refresh() + stdscr.move(0, 0) + stdscr.addch('A') + stdscr.addch(b'A') + stdscr.addch(65) + stdscr.addch('\u20ac') + stdscr.addch('A', curses.A_BOLD) + stdscr.addch(1, 2, 'A') + stdscr.addch(2, 3, 'A', curses.A_BOLD) + self.assertIs(stdscr.is_wintouched(), True) + + # echochar() + stdscr.refresh() + stdscr.move(0, 0) + stdscr.echochar('A') + stdscr.echochar(b'A') + stdscr.echochar(65) + self.assertRaises(OverflowError, stdscr.echochar, '\u20ac') + stdscr.echochar('A', curses.A_BOLD) + self.assertIs(stdscr.is_wintouched(), False) + + def test_output_string(self): + stdscr = self.stdscr + # addstr()/insstr() + for func in [stdscr.addstr, stdscr.insstr]: + with self.subTest(func.__qualname__): + stdscr.move(0, 0) + func('abcd') + func(b'abcd') + func('àßçđ') + func('abcd', curses.A_BOLD) + func(1, 2, 'abcd') + func(2, 3, 'abcd', curses.A_BOLD) + + # addnstr()/insnstr() + for func in [stdscr.addnstr, stdscr.insnstr]: + with self.subTest(func.__qualname__): + stdscr.move(0, 0) + func('1234', 3) + func(b'1234', 3) + func('\u0661\u0662\u0663\u0664', 3) + func('1234', 5) + func('1234', 3, curses.A_BOLD) + func(1, 2, '1234', 3) + func(2, 3, '1234', 3, curses.A_BOLD) + + def test_output_string_embedded_null_chars(self): + # reject embedded null bytes and characters + stdscr = self.stdscr + for arg in ['a\0', b'a\0']: + with self.subTest(arg=arg): + self.assertRaises(ValueError, stdscr.addstr, arg) + self.assertRaises(ValueError, stdscr.addnstr, arg, 1) + self.assertRaises(ValueError, stdscr.insstr, arg) + self.assertRaises(ValueError, stdscr.insnstr, arg, 1) - win.border(65, 66, 67, 68, - 69, 70, 71, 72) + def test_read_from_window(self): + stdscr = self.stdscr + stdscr.addstr(0, 1, 'ABCD', curses.A_BOLD) + # inch() + stdscr.move(0, 1) + self.assertEqual(stdscr.inch(), 65 | curses.A_BOLD) + self.assertEqual(stdscr.inch(0, 3), 67 | curses.A_BOLD) + stdscr.move(0, 0) + # instr() + self.assertEqual(stdscr.instr()[:6], b' ABCD ') + self.assertEqual(stdscr.instr(3)[:6], b' AB') + self.assertEqual(stdscr.instr(0, 2)[:4], b'BCD ') + self.assertEqual(stdscr.instr(0, 2, 4), b'BCD ') + self.assertRaises(ValueError, stdscr.instr, -2) + self.assertRaises(ValueError, stdscr.instr, 0, 2, -2) + + def test_getch(self): + win = curses.newwin(5, 12, 5, 2) + + # TODO: Test with real input by writing to master fd. + for c in 'spam\n'[::-1]: + curses.ungetch(c) + self.assertEqual(win.getch(3, 1), b's'[0]) + self.assertEqual(win.getyx(), (3, 1)) + self.assertEqual(win.getch(3, 4), b'p'[0]) + self.assertEqual(win.getyx(), (3, 4)) + self.assertEqual(win.getch(), b'a'[0]) + self.assertEqual(win.getyx(), (3, 4)) + self.assertEqual(win.getch(), b'm'[0]) + self.assertEqual(win.getch(), b'\n'[0]) + + def test_getstr(self): + win = curses.newwin(5, 12, 5, 2) + curses.echo() + self.addCleanup(curses.noecho) + + self.assertRaises(ValueError, win.getstr, -400) + self.assertRaises(ValueError, win.getstr, 2, 3, -400) + + # TODO: Test with real input by writing to master fd. + for c in 'Lorem\nipsum\ndolor\nsit\namet\n'[::-1]: + curses.ungetch(c) + self.assertEqual(win.getstr(3, 1, 2), b'Lo') + self.assertEqual(win.instr(3, 0), b' Lo ') + self.assertEqual(win.getstr(3, 5, 10), b'ipsum') + self.assertEqual(win.instr(3, 0), b' Lo ipsum ') + self.assertEqual(win.getstr(1, 5), b'dolor') + self.assertEqual(win.instr(1, 0), b' dolor ') + self.assertEqual(win.getstr(2), b'si') + self.assertEqual(win.instr(1, 0), b'si dolor ') + self.assertEqual(win.getstr(), b'amet') + self.assertEqual(win.instr(1, 0), b'amet dolor ') + + def test_clear(self): + win = curses.newwin(5, 15, 5, 2) + lorem_ipsum(win) + + win.move(0, 8) + win.clrtoeol() + self.assertEqual(win.instr(0, 0).rstrip(), b'Lorem ip') + self.assertEqual(win.instr(1, 0).rstrip(), b'dolor sit amet,') + + win.move(0, 3) + win.clrtobot() + self.assertEqual(win.instr(0, 0).rstrip(), b'Lor') + self.assertEqual(win.instr(1, 0).rstrip(), b'') + + for func in [win.erase, win.clear]: + lorem_ipsum(win) + func() + self.assertEqual(win.instr(0, 0).rstrip(), b'') + self.assertEqual(win.instr(1, 0).rstrip(), b'') + + def test_insert_delete(self): + win = curses.newwin(5, 15, 5, 2) + lorem_ipsum(win) + + win.move(0, 2) + win.delch() + self.assertEqual(win.instr(0, 0), b'Loem ipsum ') + win.delch(0, 7) + self.assertEqual(win.instr(0, 0), b'Loem ipum ') + + win.move(1, 5) + win.deleteln() + self.assertEqual(win.instr(0, 0), b'Loem ipum ') + self.assertEqual(win.instr(1, 0), b'consectetur ') + self.assertEqual(win.instr(2, 0), b'adipiscing elit') + self.assertEqual(win.instr(3, 0), b'sed do eiusmod ') + self.assertEqual(win.instr(4, 0), b' ') + + win.move(1, 5) + win.insertln() + self.assertEqual(win.instr(0, 0), b'Loem ipum ') + self.assertEqual(win.instr(1, 0), b' ') + self.assertEqual(win.instr(2, 0), b'consectetur ') + + win.clear() + lorem_ipsum(win) + win.move(1, 5) + win.insdelln(2) + self.assertEqual(win.instr(0, 0), b'Lorem ipsum ') + self.assertEqual(win.instr(1, 0), b' ') + self.assertEqual(win.instr(2, 0), b' ') + self.assertEqual(win.instr(3, 0), b'dolor sit amet,') + + win.clear() + lorem_ipsum(win) + win.move(1, 5) + win.insdelln(-2) + self.assertEqual(win.instr(0, 0), b'Lorem ipsum ') + self.assertEqual(win.instr(1, 0), b'adipiscing elit') + self.assertEqual(win.instr(2, 0), b'sed do eiusmod ') + self.assertEqual(win.instr(3, 0), b' ') + + def test_scroll(self): + win = curses.newwin(5, 15, 5, 2) + lorem_ipsum(win) + win.scrollok(True) + win.scroll() + self.assertEqual(win.instr(0, 0), b'dolor sit amet,') + win.scroll(2) + self.assertEqual(win.instr(0, 0), b'adipiscing elit') + win.scroll(-3) + self.assertEqual(win.instr(0, 0), b' ') + self.assertEqual(win.instr(2, 0), b' ') + self.assertEqual(win.instr(3, 0), b'adipiscing elit') + win.scrollok(False) + + def test_attributes(self): + # TODO: attr_get(), attr_set(), ... + win = curses.newwin(5, 15, 5, 2) + win.attron(curses.A_BOLD) + win.attroff(curses.A_BOLD) + win.attrset(curses.A_BOLD) + + win.standout() + win.standend() + + @requires_curses_window_meth('chgat') + def test_chgat(self): + win = curses.newwin(5, 15, 5, 2) + win.addstr(2, 0, 'Lorem ipsum') + win.addstr(3, 0, 'dolor sit amet') + + win.move(2, 8) + win.chgat(curses.A_BLINK) + self.assertEqual(win.inch(2, 7), b'p'[0]) + self.assertEqual(win.inch(2, 8), b's'[0] | curses.A_BLINK) + self.assertEqual(win.inch(2, 14), b' '[0] | curses.A_BLINK) + + win.move(2, 1) + win.chgat(3, curses.A_BOLD) + self.assertEqual(win.inch(2, 0), b'L'[0]) + self.assertEqual(win.inch(2, 1), b'o'[0] | curses.A_BOLD) + self.assertEqual(win.inch(2, 3), b'e'[0] | curses.A_BOLD) + self.assertEqual(win.inch(2, 4), b'm'[0]) + + win.chgat(3, 2, curses.A_UNDERLINE) + self.assertEqual(win.inch(3, 1), b'o'[0]) + self.assertEqual(win.inch(3, 2), b'l'[0] | curses.A_UNDERLINE) + self.assertEqual(win.inch(3, 14), b' '[0] | curses.A_UNDERLINE) + + win.chgat(3, 4, 7, curses.A_BLINK) + self.assertEqual(win.inch(3, 3), b'o'[0] | curses.A_UNDERLINE) + self.assertEqual(win.inch(3, 4), b'r'[0] | curses.A_BLINK) + self.assertEqual(win.inch(3, 10), b'a'[0] | curses.A_BLINK) + self.assertEqual(win.inch(3, 11), b'm'[0] | curses.A_UNDERLINE) + self.assertEqual(win.inch(3, 14), b' '[0] | curses.A_UNDERLINE) + + def test_background(self): + win = curses.newwin(5, 15, 5, 2) + win.addstr(0, 0, 'Lorem ipsum') + + self.assertEqual(win.getbkgd(), 0) + + # bkgdset() + win.bkgdset('_') + self.assertEqual(win.getbkgd(), b'_'[0]) + win.bkgdset(b'#') + self.assertEqual(win.getbkgd(), b'#'[0]) + win.bkgdset(65) + self.assertEqual(win.getbkgd(), 65) + win.bkgdset(0) + self.assertEqual(win.getbkgd(), 32) + + win.bkgdset('#', curses.A_REVERSE) + self.assertEqual(win.getbkgd(), b'#'[0] | curses.A_REVERSE) + self.assertEqual(win.inch(0, 0), b'L'[0]) + self.assertEqual(win.inch(0, 5), b' '[0]) + win.bkgdset(0) + + # bkgd() + win.bkgd('_') + self.assertEqual(win.getbkgd(), b'_'[0]) + self.assertEqual(win.inch(0, 0), b'L'[0]) + self.assertEqual(win.inch(0, 5), b'_'[0]) + + win.bkgd('#', curses.A_REVERSE) + self.assertEqual(win.getbkgd(), b'#'[0] | curses.A_REVERSE) + self.assertEqual(win.inch(0, 0), b'L'[0] | curses.A_REVERSE) + self.assertEqual(win.inch(0, 5), b'#'[0] | curses.A_REVERSE) + + def test_overlay(self): + srcwin = curses.newwin(5, 18, 3, 4) + lorem_ipsum(srcwin) + dstwin = curses.newwin(7, 17, 5, 7) + for i in range(6): + dstwin.addstr(i, 0, '_'*17) + + srcwin.overlay(dstwin) + self.assertEqual(dstwin.instr(0, 0), b'sectetur_________') + self.assertEqual(dstwin.instr(1, 0), b'piscing_elit,____') + self.assertEqual(dstwin.instr(2, 0), b'_do_eiusmod______') + self.assertEqual(dstwin.instr(3, 0), b'_________________') + + srcwin.overwrite(dstwin) + self.assertEqual(dstwin.instr(0, 0), b'sectetur __') + self.assertEqual(dstwin.instr(1, 0), b'piscing elit, __') + self.assertEqual(dstwin.instr(2, 0), b' do eiusmod __') + self.assertEqual(dstwin.instr(3, 0), b'_________________') + + srcwin.overlay(dstwin, 1, 4, 3, 2, 4, 11) + self.assertEqual(dstwin.instr(3, 0), b'__r_sit_amet_____') + self.assertEqual(dstwin.instr(4, 0), b'__ectetur________') + self.assertEqual(dstwin.instr(5, 0), b'_________________') + + srcwin.overwrite(dstwin, 1, 4, 3, 2, 4, 11) + self.assertEqual(dstwin.instr(3, 0), b'__r sit amet_____') + self.assertEqual(dstwin.instr(4, 0), b'__ectetur _____') + self.assertEqual(dstwin.instr(5, 0), b'_________________') + + def test_refresh(self): + win = curses.newwin(5, 15, 2, 5) + win.noutrefresh() + win.redrawln(1, 2) + win.redrawwin() + win.refresh() + curses.doupdate() + + @requires_curses_window_meth('resize') + def test_resize(self): + win = curses.newwin(5, 15, 2, 5) + win.resize(4, 20) + self.assertEqual(win.getmaxyx(), (4, 20)) + win.resize(5, 15) + self.assertEqual(win.getmaxyx(), (5, 15)) + + @requires_curses_window_meth('enclose') + def test_enclose(self): + win = curses.newwin(5, 15, 2, 5) + # TODO: Return bool instead of 1/0 + self.assertTrue(win.enclose(2, 5)) + self.assertFalse(win.enclose(1, 5)) + self.assertFalse(win.enclose(2, 4)) + self.assertTrue(win.enclose(6, 19)) + self.assertFalse(win.enclose(7, 19)) + self.assertFalse(win.enclose(6, 20)) + + def test_putwin(self): + win = curses.newwin(5, 12, 1, 2) + win.addstr(2, 1, 'Lorem ipsum') + with tempfile.TemporaryFile() as f: + win.putwin(f) + del win + f.seek(0) + win = curses.getwin(f) + self.assertEqual(win.getbegyx(), (1, 2)) + self.assertEqual(win.getmaxyx(), (5, 12)) + self.assertEqual(win.instr(2, 0), b' Lorem ipsum') + + def test_borders_and_lines(self): + win = curses.newwin(5, 10, 5, 2) win.border('|', '!', '-', '_', '+', '\\', '#', '/') - with self.assertRaises(TypeError, - msg="Expected win.border() to raise TypeError"): - win.border(65, 66, 67, 68, - 69, [], 71, 72) - - win.box(65, 67) - win.box('!', '_') + self.assertEqual(win.instr(0, 0), b'+--------\\') + self.assertEqual(win.instr(1, 0), b'| !') + self.assertEqual(win.instr(4, 0), b'#________/') + win.border(b'|', b'!', b'-', b'_', + b'+', b'\\', b'#', b'/') + win.border(65, 66, 67, 68, + 69, 70, 71, 72) + self.assertRaises(TypeError, win.border, + 65, 66, 67, 68, 69, [], 71, 72) + self.assertRaises(TypeError, win.border, + 65, 66, 67, 68, 69, 70, 71, 72, 73) + self.assertRaises(TypeError, win.border, + 65, 66, 67, 68, 69, 70, 71, 72, 73) + win.border(65, 66, 67, 68, 69, 70, 71) + win.border(65, 66, 67, 68, 69, 70) + win.border(65, 66, 67, 68, 69) + win.border(65, 66, 67, 68) + win.border(65, 66, 67) + win.border(65, 66) + win.border(65) + win.border() + + win.box(':', '~') + self.assertEqual(win.instr(0, 1, 8), b'~~~~~~~~') + self.assertEqual(win.instr(1, 0), b': :') + self.assertEqual(win.instr(4, 1, 8), b'~~~~~~~~') win.box(b':', b'~') + win.box(65, 67) self.assertRaises(TypeError, win.box, 65, 66, 67) self.assertRaises(TypeError, win.box, 65) win.box() - stdscr.clearok(1) + win.move(1, 2) + win.hline('-', 5) + self.assertEqual(win.instr(1, 1, 7), b' ----- ') + win.hline(b'-', 5) + win.hline(45, 5) + win.hline('-', 5, curses.A_BOLD) + win.hline(1, 1, '-', 5) + win.hline(1, 1, '-', 5, curses.A_BOLD) + + win.move(1, 2) + win.vline('a', 3) + win.vline(b'a', 3) + win.vline(97, 3) + win.vline('a', 3, curses.A_STANDOUT) + win.vline(1, 1, 'a', 3) + win.vline(1, 1, ';', 2, curses.A_STANDOUT) + self.assertEqual(win.inch(1, 1), b';'[0] | curses.A_STANDOUT) + self.assertEqual(win.inch(2, 1), b';'[0] | curses.A_STANDOUT) + self.assertEqual(win.inch(3, 1), b'a'[0]) + + def test_unctrl(self): + # TODO: wunctrl() + self.assertEqual(curses.unctrl(b'A'), b'A') + self.assertEqual(curses.unctrl('A'), b'A') + self.assertEqual(curses.unctrl(65), b'A') + self.assertEqual(curses.unctrl(b'\n'), b'^J') + self.assertEqual(curses.unctrl('\n'), b'^J') + self.assertEqual(curses.unctrl(10), b'^J') + self.assertRaises(TypeError, curses.unctrl, b'') + self.assertRaises(TypeError, curses.unctrl, b'AB') + self.assertRaises(TypeError, curses.unctrl, '') + self.assertRaises(TypeError, curses.unctrl, 'AB') + self.assertRaises(OverflowError, curses.unctrl, 2**64) + + def test_endwin(self): + if not self.isatty: + self.skipTest('requires terminal') + self.assertIs(curses.isendwin(), False) + curses.endwin() + self.assertIs(curses.isendwin(), True) + curses.doupdate() + self.assertIs(curses.isendwin(), False) + + def test_terminfo(self): + self.assertIsInstance(curses.tigetflag('hc'), int) + self.assertEqual(curses.tigetflag('cols'), -1) + self.assertEqual(curses.tigetflag('cr'), -1) + + self.assertIsInstance(curses.tigetnum('cols'), int) + self.assertEqual(curses.tigetnum('hc'), -2) + self.assertEqual(curses.tigetnum('cr'), -2) + + self.assertIsInstance(curses.tigetstr('cr'), (bytes, type(None))) + self.assertIsNone(curses.tigetstr('hc')) + self.assertIsNone(curses.tigetstr('cols')) + + cud = curses.tigetstr('cud') + if cud is not None: + # See issue10570. + self.assertIsInstance(cud, bytes) + curses.tparm(cud, 2) + cud_2 = curses.tparm(cud, 2) + self.assertIsInstance(cud_2, bytes) + curses.putp(cud_2) + + curses.putp(b'abc\n') + + def test_misc_module_funcs(self): + curses.delay_output(1) + curses.flushinp() + + curses.doupdate() + self.assertIs(curses.isendwin(), False) + + curses.napms(100) + + curses.newpad(50, 50) + + def test_env_queries(self): + # TODO: term_attrs(), erasewchar(), killwchar() + self.assertIsInstance(curses.termname(), bytes) + self.assertIsInstance(curses.longname(), bytes) + self.assertIsInstance(curses.baudrate(), int) + self.assertIsInstance(curses.has_ic(), bool) + self.assertIsInstance(curses.has_il(), bool) + self.assertIsInstance(curses.termattrs(), int) + + c = curses.killchar() + self.assertIsInstance(c, bytes) + self.assertEqual(len(c), 1) + c = curses.erasechar() + self.assertIsInstance(c, bytes) + self.assertEqual(len(c), 1) + + def test_output_options(self): + stdscr = self.stdscr + + stdscr.clearok(True) + stdscr.clearok(False) - win4 = stdscr.derwin(2,2) - win4 = stdscr.derwin(1,1, 5,5) - win4.mvderwin(9,9) + stdscr.idcok(True) + stdscr.idcok(False) - stdscr.echochar('a') - stdscr.echochar('a', curses.A_BOLD) - stdscr.hline('-', 5) - stdscr.hline('-', 5, curses.A_BOLD) - stdscr.hline(1,1,'-', 5) - stdscr.hline(1,1,'-', 5, curses.A_BOLD) + stdscr.idlok(False) + stdscr.idlok(True) - stdscr.idcok(1) - stdscr.idlok(1) if hasattr(stdscr, 'immedok'): - stdscr.immedok(1) - stdscr.immedok(0) - stdscr.insch('c') - stdscr.insdelln(1) - stdscr.insnstr('abc', 3) - stdscr.insnstr('abc', 3, curses.A_BOLD) - stdscr.insnstr(5, 5, 'abc', 3) - stdscr.insnstr(5, 5, 'abc', 3, curses.A_BOLD) - - stdscr.insstr('def') - stdscr.insstr('def', curses.A_BOLD) - stdscr.insstr(5, 5, 'def') - stdscr.insstr(5, 5, 'def', curses.A_BOLD) - stdscr.is_linetouched(0) - stdscr.keypad(1) - stdscr.leaveok(1) - stdscr.move(3,3) - win.mvwin(2,2) - stdscr.nodelay(1) - stdscr.notimeout(1) - win2.overlay(win) - win2.overwrite(win) - win2.overlay(win, 1, 2, 2, 1, 3, 3) - win2.overwrite(win, 1, 2, 2, 1, 3, 3) - stdscr.redrawln(1,2) - - stdscr.scrollok(1) - stdscr.scroll() - stdscr.scroll(2) - stdscr.scroll(-3) - - stdscr.move(12, 2) - stdscr.setscrreg(10,15) - win3 = stdscr.subwin(10,10) - win3 = stdscr.subwin(10,10, 5,5) - if hasattr(stdscr, 'syncok') and not sys.platform.startswith("sunos"): - stdscr.syncok(1) - stdscr.timeout(5) - stdscr.touchline(5,5) - stdscr.touchline(5,5,0) - stdscr.vline('a', 3) - stdscr.vline('a', 3, curses.A_STANDOUT) - if hasattr(stdscr, 'chgat'): - stdscr.chgat(5, 2, 3, curses.A_BLINK) - stdscr.chgat(3, curses.A_BOLD) - stdscr.chgat(5, 8, curses.A_UNDERLINE) - stdscr.chgat(curses.A_BLINK) - stdscr.refresh() + stdscr.immedok(True) + stdscr.immedok(False) - stdscr.vline(1,1, 'a', 3) - stdscr.vline(1,1, 'a', 3, curses.A_STANDOUT) + stdscr.leaveok(True) + stdscr.leaveok(False) - if hasattr(stdscr, 'resize'): - stdscr.resize(25, 80) - if hasattr(stdscr, 'enclose'): - stdscr.enclose(10, 10) + stdscr.scrollok(True) + stdscr.scrollok(False) - with tempfile.TemporaryFile() as f: - self.stdscr.putwin(f) - f.seek(0) - curses.getwin(f) + stdscr.setscrreg(5, 10) - self.assertRaises(ValueError, stdscr.getstr, -400) - self.assertRaises(ValueError, stdscr.getstr, 2, 3, -400) - self.assertRaises(ValueError, stdscr.instr, -2) - self.assertRaises(ValueError, stdscr.instr, 2, 3, -2) + curses.nonl() + curses.nl(True) + curses.nl(False) + curses.nl() - def test_embedded_null_chars(self): - # reject embedded null bytes and characters + + def test_input_options(self): stdscr = self.stdscr - for arg in ['a', b'a']: - with self.subTest(arg=arg): - self.assertRaises(ValueError, stdscr.addstr, 'a\0') - self.assertRaises(ValueError, stdscr.addnstr, 'a\0', 1) - self.assertRaises(ValueError, stdscr.insstr, 'a\0') - self.assertRaises(ValueError, stdscr.insnstr, 'a\0', 1) - - def test_module_funcs(self): - "Test module-level functions" - for func in [curses.baudrate, curses.beep, curses.can_change_color, - curses.doupdate, curses.flash, curses.flushinp, - curses.has_colors, curses.has_ic, curses.has_il, - curses.isendwin, curses.killchar, curses.longname, - curses.noecho, curses.nonl, curses.noqiflush, - curses.termattrs, curses.termname, curses.erasechar]: - with self.subTest(func=func.__qualname__): - func() + if self.isatty: - for func in [curses.cbreak, curses.def_prog_mode, - curses.nocbreak, curses.noraw, - curses.reset_prog_mode]: - with self.subTest(func=func.__qualname__): - func() - if hasattr(curses, 'filter'): - curses.filter() - if hasattr(curses, 'getsyx'): - curses.getsyx() - - # Functions that actually need arguments - if curses.tigetstr("cnorm"): - curses.curs_set(1) - curses.delay_output(1) - curses.echo() ; curses.echo(1) + curses.nocbreak() + curses.cbreak() + curses.cbreak(False) + curses.cbreak(True) + curses.intrflush(True) + curses.intrflush(False) + + curses.raw() + curses.raw(False) + curses.raw(True) + curses.noraw() + + curses.noecho() + curses.echo() + curses.echo(False) + curses.echo(True) + + curses.halfdelay(255) curses.halfdelay(1) - if self.isatty: - curses.intrflush(1) - curses.meta(1) - curses.napms(100) - curses.newpad(50,50) - win = curses.newwin(5,5) - win = curses.newwin(5,5, 1,1) - curses.nl() ; curses.nl(1) - curses.putp(b'abc') + + stdscr.keypad(True) + stdscr.keypad(False) + + curses.meta(True) + curses.meta(False) + + stdscr.nodelay(True) + stdscr.nodelay(False) + + curses.noqiflush() + curses.qiflush(True) + curses.qiflush(False) curses.qiflush() - if self.isatty: - curses.raw() ; curses.raw(1) - if hasattr(curses, 'setsyx'): - curses.setsyx(5,5) - curses.tigetflag('hc') - curses.tigetnum('co') - curses.tigetstr('cr') - curses.tparm(b'cr') - if hasattr(curses, 'typeahead'): - curses.typeahead(sys.__stdin__.fileno()) - curses.unctrl('a') - curses.ungetch('a') - if hasattr(curses, 'use_env'): - curses.use_env(1) - - # Functions only available on a few platforms + + stdscr.notimeout(True) + stdscr.notimeout(False) + + stdscr.timeout(-1) + stdscr.timeout(0) + stdscr.timeout(5) + + @requires_curses_func('typeahead') + def test_typeahead(self): + curses.typeahead(sys.__stdin__.fileno()) + curses.typeahead(-1) + + def test_prog_mode(self): + if not self.isatty: + self.skipTest('requires terminal') + curses.def_prog_mode() + curses.reset_prog_mode() + + def test_beep(self): + if (curses.tigetstr("bel") is not None + or curses.tigetstr("flash") is not None): + curses.beep() + else: + try: + curses.beep() + except curses.error: + self.skipTest('beep() failed') + + def test_flash(self): + if (curses.tigetstr("bel") is not None + or curses.tigetstr("flash") is not None): + curses.flash() + else: + try: + curses.flash() + except curses.error: + self.skipTest('flash() failed') + + def test_curs_set(self): + for vis, cap in [(0, 'civis'), (2, 'cvvis'), (1, 'cnorm')]: + if curses.tigetstr(cap) is not None: + curses.curs_set(vis) + else: + try: + curses.curs_set(vis) + except curses.error: + pass + + @requires_curses_func('getsyx') + def test_getsyx(self): + y, x = curses.getsyx() + self.assertIsInstance(y, int) + self.assertIsInstance(x, int) + curses.setsyx(4, 5) + self.assertEqual(curses.getsyx(), (4, 5)) def bad_colors(self): return (-2**31 - 1, 2**31, -2**63 - 1, 2**63, 2**64) @@ -319,6 +828,10 @@ def bad_colors(self): def bad_pairs(self): return (-2**31 - 1, 2**31, -2**63 - 1, 2**63, 2**64) + def test_has_colors(self): + self.assertIsInstance(curses.has_colors(), bool) + self.assertIsInstance(curses.can_change_color(), bool) + def test_start_color(self): if not curses.has_colors(): self.skipTest('requires colors support') @@ -342,7 +855,7 @@ def test_color_content(self): @requires_colors def test_init_color(self): - if not curses.can_change_color: + if not curses.can_change_color(): self.skipTest('cannot change color') old = curses.color_content(0) @@ -430,14 +943,22 @@ def test_color_attrs(self): @requires_curses_func('use_default_colors') @requires_colors def test_use_default_colors(self): - self.assertIn(curses.pair_content(0), - ((curses.COLOR_WHITE, curses.COLOR_BLACK), (-1, -1))) - curses.use_default_colors() + old = curses.pair_content(0) + try: + curses.use_default_colors() + except curses.error: + self.skipTest('cannot change color (use_default_colors() failed)') self.assertEqual(curses.pair_content(0), (-1, -1)) + self.assertIn(old, [(curses.COLOR_WHITE, curses.COLOR_BLACK), (-1, -1), (0, 0)]) - @requires_curses_func('keyname') def test_keyname(self): - curses.keyname(13) + # TODO: key_name() + self.assertEqual(curses.keyname(65), b'A') + self.assertEqual(curses.keyname(13), b'^M') + self.assertEqual(curses.keyname(127), b'^?') + self.assertEqual(curses.keyname(0), b'^@') + self.assertRaises(ValueError, curses.keyname, -1) + self.assertIsInstance(curses.keyname(256), bytes) @requires_curses_func('has_key') def test_has_key(self): @@ -493,22 +1014,46 @@ def test_new_curses_panel(self): @requires_curses_func('is_term_resized') def test_is_term_resized(self): - curses.is_term_resized(*self.stdscr.getmaxyx()) + lines, cols = curses.LINES, curses.COLS + self.assertIs(curses.is_term_resized(lines, cols), False) + self.assertIs(curses.is_term_resized(lines-1, cols-1), True) @requires_curses_func('resize_term') def test_resize_term(self): - curses.resize_term(*self.stdscr.getmaxyx()) + curses.update_lines_cols() + lines, cols = curses.LINES, curses.COLS + new_lines = lines - 1 + new_cols = cols + 1 + curses.resize_term(new_lines, new_cols) + self.assertEqual(curses.LINES, new_lines) + self.assertEqual(curses.COLS, new_cols) + + curses.resize_term(lines, cols) + self.assertEqual(curses.LINES, lines) + self.assertEqual(curses.COLS, cols) @requires_curses_func('resizeterm') def test_resizeterm(self): + curses.update_lines_cols() lines, cols = curses.LINES, curses.COLS new_lines = lines - 1 new_cols = cols + 1 curses.resizeterm(new_lines, new_cols) - self.assertEqual(curses.LINES, new_lines) self.assertEqual(curses.COLS, new_cols) + curses.resizeterm(lines, cols) + self.assertEqual(curses.LINES, lines) + self.assertEqual(curses.COLS, cols) + + def test_ungetch(self): + curses.ungetch(b'A') + self.assertEqual(self.stdscr.getkey(), 'A') + curses.ungetch('B') + self.assertEqual(self.stdscr.getkey(), 'B') + curses.ungetch(67) + self.assertEqual(self.stdscr.getkey(), 'C') + def test_issue6243(self): curses.ungetch(1025) self.stdscr.getkey() @@ -537,10 +1082,6 @@ def test_unget_wch(self): read = stdscr.get_wch() self.assertEqual(read, ch) - def test_issue10570(self): - b = curses.tparm(curses.tigetstr("cup"), 5, 3) - self.assertIs(type(b), bytes) - def test_encoding(self): stdscr = self.stdscr import codecs @@ -580,26 +1121,25 @@ def test_issue21088(self): human_readable_signature = stdscr.addch.__doc__.split("\n")[0] self.assertIn("[y, x,]", human_readable_signature) + @requires_curses_window_meth('resize') def test_issue13051(self): - stdscr = self.stdscr - if not hasattr(stdscr, 'resize'): - raise unittest.SkipTest('requires curses.window.resize') - box = curses.textpad.Textbox(stdscr, insert_mode=True) - lines, cols = stdscr.getmaxyx() - stdscr.resize(lines-2, cols-2) + win = curses.newwin(5, 15, 2, 5) + box = curses.textpad.Textbox(win, insert_mode=True) + lines, cols = win.getmaxyx() + win.resize(lines-2, cols-2) # this may cause infinite recursion, leading to a RuntimeError box._insert_printable_char('a') class MiscTests(unittest.TestCase): - @requires_curses_func('update_lines_cols') def test_update_lines_cols(self): - # this doesn't actually test that LINES and COLS are updated, - # because we can't automate changing them. See Issue #4254 for - # a manual test script. We can only test that the function - # can be called. curses.update_lines_cols() + lines, cols = curses.LINES, curses.COLS + curses.LINES = curses.COLS = 0 + curses.update_lines_cols() + self.assertEqual(curses.LINES, lines) + self.assertEqual(curses.COLS, cols) @requires_curses_func('ncurses_version') def test_ncurses_version(self): @@ -621,6 +1161,7 @@ def test_ncurses_version(self): self.assertGreaterEqual(v.minor, 0) self.assertGreaterEqual(v.patch, 0) + class TestAscii(unittest.TestCase): def test_controlnames(self): @@ -709,5 +1250,21 @@ def test_unctrl(self): self.assertEqual(unctrl(ord('\xc1')), '!A') +def lorem_ipsum(win): + text = [ + 'Lorem ipsum', + 'dolor sit amet,', + 'consectetur', + 'adipiscing elit,', + 'sed do eiusmod', + 'tempor incididunt', + 'ut labore et', + 'dolore magna', + 'aliqua.', + ] + maxy, maxx = win.getmaxyx() + for y, line in enumerate(text[:maxy]): + win.addstr(y, 0, line[:maxx - (y == maxy - 1)]) + if __name__ == '__main__': unittest.main() From 931263baab62b1c3fa7647e45ec6ee6ef4409e7c Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 31 Jan 2021 12:51:23 -0800 Subject: [PATCH 1966/2163] [3.9] bpo-41604: Don't decrement the reference count of the previous user_ptr when set_panel_usertpr fails (GH-21933). (GH-24403) (cherry picked from commit 3243e8a4b4b4cf321f9b28335d565742a34b1976) Co-authored-by: Anonymous Maarten (cherry picked from commit 3c8d6934436e20163be802f5239c5b4e4925eeec) Co-authored-by: Serhiy Storchaka --- .../next/Library/2020-08-21-15-24-14.bpo-41604.rTXleO.rst | 2 ++ Modules/_curses_panel.c | 4 +++- 2 files changed, 5 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2020-08-21-15-24-14.bpo-41604.rTXleO.rst diff --git a/Misc/NEWS.d/next/Library/2020-08-21-15-24-14.bpo-41604.rTXleO.rst b/Misc/NEWS.d/next/Library/2020-08-21-15-24-14.bpo-41604.rTXleO.rst new file mode 100644 index 00000000000000..0f9794cbdb321e --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-21-15-24-14.bpo-41604.rTXleO.rst @@ -0,0 +1,2 @@ +Don't decrement the reference count of the previous user_ptr when +set_panel_userptr fails. diff --git a/Modules/_curses_panel.c b/Modules/_curses_panel.c index 53849e3a29cc02..d22e2adf89c839 100644 --- a/Modules/_curses_panel.c +++ b/Modules/_curses_panel.c @@ -433,7 +433,9 @@ _curses_panel_panel_set_userptr(PyCursesPanelObject *self, PyObject *obj) /* In case of an ncurses error, decref the new object again */ Py_DECREF(obj); } - Py_XDECREF(oldobj); + else { + Py_XDECREF(oldobj); + } return PyCursesCheckERR(rc, "set_panel_userptr"); } From aab84a58063e68cb7ff5f7b8d96c431200d0e340 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 31 Jan 2021 23:37:12 -0800 Subject: [PATCH 1967/2163] bpo-43016: Fix test_curses on platform without cursesw (GH-24405) (GH-24408) (cherry picked from commit a1e9a1e120a11c563e166c15721169184c802f8b) Co-authored-by: Serhiy Storchaka --- Lib/test/test_curses.py | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py index f47c9876eed88f..7dd0695aa72771 100644 --- a/Lib/test/test_curses.py +++ b/Lib/test/test_curses.py @@ -239,13 +239,21 @@ def test_refresh_control(self): def test_output_character(self): stdscr = self.stdscr + encoding = stdscr.encoding # addch() stdscr.refresh() stdscr.move(0, 0) stdscr.addch('A') stdscr.addch(b'A') stdscr.addch(65) - stdscr.addch('\u20ac') + c = '\u20ac' + try: + stdscr.addch(c) + except UnicodeEncodeError: + self.assertRaises(UnicodeEncodeError, c.encode, encoding) + except OverflowError: + encoded = c.encode(encoding) + self.assertNotEqual(len(encoded), 1, repr(encoded)) stdscr.addch('A', curses.A_BOLD) stdscr.addch(1, 2, 'A') stdscr.addch(2, 3, 'A', curses.A_BOLD) @@ -257,19 +265,25 @@ def test_output_character(self): stdscr.echochar('A') stdscr.echochar(b'A') stdscr.echochar(65) - self.assertRaises(OverflowError, stdscr.echochar, '\u20ac') + with self.assertRaises((UnicodeEncodeError, OverflowError)): + stdscr.echochar('\u20ac') stdscr.echochar('A', curses.A_BOLD) self.assertIs(stdscr.is_wintouched(), False) def test_output_string(self): stdscr = self.stdscr + encoding = stdscr.encoding # addstr()/insstr() for func in [stdscr.addstr, stdscr.insstr]: with self.subTest(func.__qualname__): stdscr.move(0, 0) func('abcd') func(b'abcd') - func('àßçđ') + s = 'àßçđ' + try: + func(s) + except UnicodeEncodeError: + self.assertRaises(UnicodeEncodeError, s.encode, encoding) func('abcd', curses.A_BOLD) func(1, 2, 'abcd') func(2, 3, 'abcd', curses.A_BOLD) @@ -280,7 +294,11 @@ def test_output_string(self): stdscr.move(0, 0) func('1234', 3) func(b'1234', 3) - func('\u0661\u0662\u0663\u0664', 3) + s = '\u0661\u0662\u0663\u0664' + try: + func(s, 3) + except UnicodeEncodeError: + self.assertRaises(UnicodeEncodeError, s.encode, encoding) func('1234', 5) func('1234', 3, curses.A_BOLD) func(1, 2, '1234', 3) @@ -470,7 +488,7 @@ def test_background(self): win = curses.newwin(5, 15, 5, 2) win.addstr(0, 0, 'Lorem ipsum') - self.assertEqual(win.getbkgd(), 0) + self.assertIn(win.getbkgd(), (0, 32)) # bkgdset() win.bkgdset('_') From 0869a713f21f4b2fe021d802cf18f1b1af53695f Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 1 Feb 2021 12:52:52 -0800 Subject: [PATCH 1968/2163] bpo-41748: Handles unquoted attributes with commas (GH-24072) * bpo-41748: Adds tests for unquoted attributes with comma * bpo-41748: Handles unquoted attributes with comma * bpo-41748: Addresses review comments * bpo-41748: Addresses review comments * Adds more test cases * Simplifies the regex for handling spaces * bpo-41748: Moves attributes tests under the right class * bpo-41748: Addresses review about duplicate attributes * bpo-41748: Adds NEWS.d entry for this patch (cherry picked from commit 9eb11a139fac5514d8456626806a68b3e3b7eafb) Co-authored-by: Karl Dubost --- Lib/html/parser.py | 2 +- Lib/test/test_htmlparser.py | 92 +++++++++++-------- .../2021-01-05-21-26-29.bpo-41748.KdC0w3.rst | 2 + 3 files changed, 59 insertions(+), 37 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2021-01-05-21-26-29.bpo-41748.KdC0w3.rst diff --git a/Lib/html/parser.py b/Lib/html/parser.py index de81879a631ac7..d19684ed1176df 100644 --- a/Lib/html/parser.py +++ b/Lib/html/parser.py @@ -47,7 +47,7 @@ |"[^"]*" # LIT-enclosed value |(?!['"])[^>\s]* # bare value ) - (?:\s*,)* # possibly followed by a comma + \s* # possibly followed by a space )?(?:\s|/(?!>))* )* )? diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py index 326e34290ff13b..d0dc67d91745a7 100644 --- a/Lib/test/test_htmlparser.py +++ b/Lib/test/test_htmlparser.py @@ -452,42 +452,6 @@ def test_illegal_declarations(self): self._run_check('', [('comment', 'spacer type="block" height="25"')]) - def test_with_unquoted_attributes(self): - # see #12008 - html = ("" - "" - "
    " - "- software-and-i" - "- library
    ") - expected = [ - ('starttag', 'html', []), - ('starttag', 'body', [('bgcolor', 'd0ca90'), ('text', '181008')]), - ('starttag', 'table', - [('cellspacing', '0'), ('cellpadding', '1'), ('width', '100%')]), - ('starttag', 'tr', []), - ('starttag', 'td', [('align', 'left')]), - ('starttag', 'font', [('size', '-1')]), - ('data', '- '), ('starttag', 'a', [('href', '/rabota/')]), - ('starttag', 'span', [('class', 'en')]), ('data', ' software-and-i'), - ('endtag', 'span'), ('endtag', 'a'), - ('data', '- '), ('starttag', 'a', [('href', '/1/')]), - ('starttag', 'span', [('class', 'en')]), ('data', ' library'), - ('endtag', 'span'), ('endtag', 'a'), ('endtag', 'table') - ] - self._run_check(html, expected) - - def test_comma_between_attributes(self): - self._run_check('
    ', [ - ('starttag', 'form', - [('action', '/xxx.php?a=1&b=2&'), - (',', None), ('method', 'post')])]) - - def test_weird_chars_in_unquoted_attribute_values(self): - self._run_check('', [ - ('starttag', 'form', - [('action', 'bogus|&#()value')])]) - def test_invalid_end_tags(self): # A collection of broken end tags.
    is used as separator. # see http://www.w3.org/TR/html5/tokenization.html#end-tag-open-state @@ -773,6 +737,62 @@ def test_end_tag_in_attribute_value(self): [("href", "http://www.example.org/\">;")]), ("data", "spam"), ("endtag", "a")]) + def test_with_unquoted_attributes(self): + # see #12008 + html = ("" + "" + "
    " + "- software-and-i" + "- library
    ") + expected = [ + ('starttag', 'html', []), + ('starttag', 'body', [('bgcolor', 'd0ca90'), ('text', '181008')]), + ('starttag', 'table', + [('cellspacing', '0'), ('cellpadding', '1'), ('width', '100%')]), + ('starttag', 'tr', []), + ('starttag', 'td', [('align', 'left')]), + ('starttag', 'font', [('size', '-1')]), + ('data', '- '), ('starttag', 'a', [('href', '/rabota/')]), + ('starttag', 'span', [('class', 'en')]), ('data', ' software-and-i'), + ('endtag', 'span'), ('endtag', 'a'), + ('data', '- '), ('starttag', 'a', [('href', '/1/')]), + ('starttag', 'span', [('class', 'en')]), ('data', ' library'), + ('endtag', 'span'), ('endtag', 'a'), ('endtag', 'table') + ] + self._run_check(html, expected) + + def test_comma_between_attributes(self): + # see bpo 41478 + # HTMLParser preserves duplicate attributes, leaving the task of + # removing duplicate attributes to a conformant html tree builder + html = ('
    ' # between attrs (unquoted) + '
    ' # between attrs (quoted) + '
    ' # after values (unquoted) + '
    ' # after values (quoted) + '
    ' # one comma values (quoted) + '
    ' # before values (unquoted) + '
    ' # before values (quoted) + '
    ' # before names + '
    ' # after names + ) + expected = [ + ('starttag', 'div', [('class', 'bar,baz=asd'),]), + ('starttag', 'div', [('class', 'bar'), (',baz', 'asd')]), + ('starttag', 'div', [('class', 'bar,'), ('baz', 'asd,')]), + ('starttag', 'div', [('class', 'bar'), (',', None), + ('baz', 'asd'), (',', None)]), + ('starttag', 'div', [('class', 'bar'), (',', None)]), + ('starttag', 'div', [('class', ',bar'), ('baz', ',asd')]), + ('starttag', 'div', [('class', ',"bar"'), ('baz', ',"asd"')]), + ('starttag', 'div', [(',class', 'bar'), (',baz', 'asd')]), + ('starttag', 'div', [('class,', 'bar'), ('baz,', 'asd')]), + ] + self._run_check(html, expected) + + def test_weird_chars_in_unquoted_attribute_values(self): + self._run_check('', [ + ('starttag', 'form', + [('action', 'bogus|&#()value')])]) if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Library/2021-01-05-21-26-29.bpo-41748.KdC0w3.rst b/Misc/NEWS.d/next/Library/2021-01-05-21-26-29.bpo-41748.KdC0w3.rst new file mode 100644 index 00000000000000..52efa3ac3d40eb --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-01-05-21-26-29.bpo-41748.KdC0w3.rst @@ -0,0 +1,2 @@ +Fix HTMLParser parsing rules for element attributes containing +commas with spaces. Patch by Karl Dubost. \ No newline at end of file From 12ec8ce471c5bae2c5ba8ce0d50dd59fc8d233c6 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Tue, 2 Feb 2021 17:23:27 -0800 Subject: [PATCH 1969/2163] Add link to Microsoft docs for limitations in Windows Store package (GH-24422) (cherry picked from commit 1ba08a121a25fcf7c947d8d37e72e46dae59168c) Co-authored-by: Steve Dower --- Doc/using/windows.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index 94d0591ae71692..1d5e9e4b5d9f70 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -339,6 +339,11 @@ full write access to shared locations such as ``TEMP`` and the registry. Instead, it will write to a private copy. If your scripts must modify the shared locations, you will need to install the full installer. +For more detail on the technical basis for these limitations, please consult +Microsoft's documentation on packaged full-trust apps, currently available at +`docs.microsoft.com/en-us/windows/msix/desktop/desktop-to-uwp-behind-the-scenes +`_ + .. _windows-nuget: From 20d43758c317b52683505ab9c575b847dbbff93e Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 3 Feb 2021 13:46:23 -0800 Subject: [PATCH 1970/2163] Fix typo (GH-23019) Fixed possible typo in comment (cherry picked from commit bfe544d2f2c2e7a7c03a764bed3276a1e27a0f5c) Co-authored-by: Harry --- Lib/datetime.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Lib/datetime.py b/Lib/datetime.py index 9777e88df6ca4f..aef8ca6972552a 100644 --- a/Lib/datetime.py +++ b/Lib/datetime.py @@ -2323,7 +2323,7 @@ def _name_from_offset(delta): # This is again a requirement for a sane tzinfo class. # # 4. (x+k).s = x.s -# This follows from #2, and that datimetimetz+timedelta preserves tzinfo. +# This follows from #2, and that datetime.timetz+timedelta preserves tzinfo. # # 5. (x+k).n = x.n + k # Again follows from how arithmetic is defined. From d29dbb122d8b8ee22b2d826af4d234a2c2cf7ba2 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Wed, 3 Feb 2021 15:34:21 -0800 Subject: [PATCH 1971/2163] bpo-42773: fix tests not being run on pushes (GH-24004) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit There was a typo, we were checking if the "GITHUB_BASE_REF" string literal was empty instead of the $GITHUB_BASE_REF value. When $GITHUB_BASE_REF is empty, the action that triggered the run was not a pull request, so we always run the full test suite. Signed-off-by: Filipe LaĂ­ns (cherry picked from commit 4ac923f2756f835f512339ee181348cc535ab07f) Co-authored-by: Filipe LaĂ­ns --- .github/workflows/build.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index e7426ac8c54696..669c541dd4f49b 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -21,7 +21,7 @@ jobs: - name: Check for source changes id: check run: | - if [ -z "GITHUB_BASE_REF" ]; then + if [ -z "$GITHUB_BASE_REF" ]; then echo '::set-output name=run_tests::true' else git fetch origin $GITHUB_BASE_REF --depth=1 From c370596a23167749b8127d1d2e20e039865aa695 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 4 Feb 2021 11:15:26 -0800 Subject: [PATCH 1972/2163] build(deps): bump actions/upload-artifact from v2.2.1 to v2.2.2 (GH-24411) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from v2.2.1 to v2.2.2. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v2.2.1...e448a9b857ee2131e752b06002bf0e093c65e571) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit aa4caf9887944ab280a87712460e2dd49b55fe5e) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/doc.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml index 2a04d668870154..2b3dc878949dfa 100644 --- a/.github/workflows/doc.yml +++ b/.github/workflows/doc.yml @@ -34,7 +34,7 @@ jobs: - name: 'Build documentation' run: xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W -j4" doctest suspicious html - name: 'Upload' - uses: actions/upload-artifact@v2.2.1 + uses: actions/upload-artifact@v2.2.2 with: name: doc-html path: Doc/build/html From b4796875d598b34f5f21cb13a8d3551574532595 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 5 Feb 2021 10:44:45 -0800 Subject: [PATCH 1973/2163] Simple typo fix (GH-24448) (cherry picked from commit 5f18c223391eef8c7d01241b51a7b2429609dd84) Co-authored-by: Andrew Tennikoff --- Doc/howto/urllib2.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst index 046a88af62f0b3..12d525771ddc28 100644 --- a/Doc/howto/urllib2.rst +++ b/Doc/howto/urllib2.rst @@ -97,7 +97,7 @@ schemes. For example, you can make an FTP request like so:: In the case of HTTP, there are two extra things that Request objects allow you to do: First, you can pass data to be sent to the server. Second, you can pass -extra information ("metadata") *about* the data or the about request itself, to +extra information ("metadata") *about* the data or about the request itself, to the server - this information is sent as HTTP "headers". Let's look at each of these in turn. From 920bf6a3a656e329c2bcbb761eb8c13c46c8cd05 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 6 Feb 2021 21:38:54 -0800 Subject: [PATCH 1974/2163] bpo-16781: In 'exec' doc, add 'nonlocal' to 'yield' and 'return' (GH-2446) These 3 statements cannot be used at module scope -- nor in exec with one namespace. (cherry picked from commit 0ec57e25c918b859b9f8d464e34e0ac859c2f8b3) Co-authored-by: Terry Jan Reedy --- Doc/library/functions.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 5a039f7a9476ff..9d67d80789fc98 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -509,7 +509,8 @@ are always available. They are listed here in alphabetical order. occurs). [#]_ If it is a code object, it is simply executed. In all cases, the code that's executed is expected to be valid as file input (see the section "File input" in the Reference Manual). Be aware that the - :keyword:`return` and :keyword:`yield` statements may not be used outside of + :keyword:`nonlocal`, :keyword:`yield`, and :keyword:`return` + statements may not be used outside of function definitions even within the context of code passed to the :func:`exec` function. The return value is ``None``. From c8a48c6d0417cc8f256a40142962825febdc2e20 Mon Sep 17 00:00:00 2001 From: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Date: Wed, 10 Feb 2021 00:55:45 +0800 Subject: [PATCH 1975/2163] [3.8] bpo-41824: Add versionadded for typing.ForwardRef docs (#24224) (#24494) --- Doc/library/typing.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 8c179ff79d15fa..046f5e0f2d9eb8 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -1020,6 +1020,8 @@ The module defines the following classes, functions and decorators: ``List[ForwardRef("SomeClass")]``. This class should not be instantiated by a user, but may be used by introspection tools. + .. versionadded:: 3.7.4 + .. function:: NewType(name, tp) A helper function to indicate a distinct type to a typechecker, From 822f7c266f886677047338e8b167ba39a6abd91e Mon Sep 17 00:00:00 2001 From: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Date: Wed, 10 Feb 2021 01:57:20 +0800 Subject: [PATCH 1976/2163] [3.8] bpo-41824: Fix indentation issue in ForwardRef docs (#24495) --- Doc/library/typing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 046f5e0f2d9eb8..9d80a209060c87 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -1020,7 +1020,7 @@ The module defines the following classes, functions and decorators: ``List[ForwardRef("SomeClass")]``. This class should not be instantiated by a user, but may be used by introspection tools. - .. versionadded:: 3.7.4 + .. versionadded:: 3.7.4 .. function:: NewType(name, tp) From 4230bd52e3f9f289f02e41ab17a95f50ed4db5a6 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 12 Feb 2021 21:06:50 -0800 Subject: [PATCH 1977/2163] bpo-43200: Fix link to shutil.copy() in the shutil doc (GH-24505) Co-authored-by: Terry Jan Reedy (cherry picked from commit 762fe7deed34a1d5294bf82071d318c8427b4893) Co-authored-by: Zackery Spytz --- Doc/library/shutil.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 578fcc6e293d9c..cd925a92a53f96 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -440,8 +440,9 @@ Directory and files operations Platform-dependent efficient copy operations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Starting from Python 3.8 all functions involving a file copy (:func:`copyfile`, -:func:`copy`, :func:`copy2`, :func:`copytree`, and :func:`move`) may use +Starting from Python 3.8, all functions involving a file copy +(:func:`copyfile`, :func:`~shutil.copy`, :func:`copy2`, +:func:`copytree`, and :func:`move`) may use platform-specific "fast-copy" syscalls in order to copy the file more efficiently (see :issue:`33671`). "fast-copy" means that the copying operation occurs within the kernel, avoiding From 7777ae2ff7ba04ad20424db4efcc67246ff27b95 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 14 Feb 2021 06:53:10 -0800 Subject: [PATCH 1978/2163] bpo-43204: Fix LibTomCrypt URL in md5module.c and sha*module.c comments (GH-24507) (GH-24516) Automerge-Triggered-By: GH:tiran (cherry picked from commit 5ec7d535581bc99918e032891167a96abd224ed6) Co-authored-by: Erlend Egeberg Aasland Co-authored-by: Erlend Egeberg Aasland --- Modules/md5module.c | 2 +- Modules/sha1module.c | 2 +- Modules/sha256module.c | 2 +- Modules/sha512module.c | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Modules/md5module.c b/Modules/md5module.c index c2ebaaf61f91c6..64fab8081b5dc1 100644 --- a/Modules/md5module.c +++ b/Modules/md5module.c @@ -74,7 +74,7 @@ typedef struct { * The library is free for all purposes without any express * guarantee it works. * - * Tom St Denis, tomstdenis@gmail.com, http://libtom.org + * Tom St Denis, tomstdenis@gmail.com, https://www.libtom.net */ /* rotate the hard way (platform optimizations could be done) */ diff --git a/Modules/sha1module.c b/Modules/sha1module.c index ce2ad267e775b7..4a8dbd8539b7a1 100644 --- a/Modules/sha1module.c +++ b/Modules/sha1module.c @@ -74,7 +74,7 @@ typedef struct { * The library is free for all purposes without any express * guarantee it works. * - * Tom St Denis, tomstdenis@gmail.com, http://libtom.org + * Tom St Denis, tomstdenis@gmail.com, https://www.libtom.net */ /* rotate the hard way (platform optimizations could be done) */ diff --git a/Modules/sha256module.c b/Modules/sha256module.c index b8d6c4cf8006a1..a1c8b1a3dfb5a1 100644 --- a/Modules/sha256module.c +++ b/Modules/sha256module.c @@ -102,7 +102,7 @@ static void SHAcopy(SHAobject *src, SHAobject *dest) * The library is free for all purposes without any express * guarantee it works. * - * Tom St Denis, tomstdenis@iahu.ca, http://libtom.org + * Tom St Denis, tomstdenis@iahu.ca, https://www.libtom.net */ diff --git a/Modules/sha512module.c b/Modules/sha512module.c index 98b97917f4caf8..4167fd391cc4a3 100644 --- a/Modules/sha512module.c +++ b/Modules/sha512module.c @@ -111,7 +111,7 @@ static void SHAcopy(SHAobject *src, SHAobject *dest) * The library is free for all purposes without any express * guarantee it works. * - * Tom St Denis, tomstdenis@iahu.ca, http://libtom.org + * Tom St Denis, tomstdenis@iahu.ca, https://www.libtom.net */ From e3110c3cfbb7daa690d54d0eff6c264c870a71bf Mon Sep 17 00:00:00 2001 From: Senthil Kumaran Date: Mon, 15 Feb 2021 10:15:02 -0800 Subject: [PATCH 1979/2163] [3.8] bpo-42967: only use '&' as a query string separator (GH-24297) (#24529) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * bpo-42967: only use '&' as a query string separator (#24297) bpo-42967: [security] Address a web cache-poisoning issue reported in urllib.parse.parse_qsl(). urllib.parse will only us "&" as query string separator by default instead of both ";" and "&" as allowed in earlier versions. An optional argument seperator with default value "&" is added to specify the separator. Co-authored-by: Éric Araujo Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Co-authored-by: Éric Araujo (cherry picked from commit fcbe0cb04d35189401c0c880ebfb4311e952d776) * [3.8] bpo-42967: only use '&' as a query string separator (GH-24297) bpo-42967: [security] Address a web cache-poisoning issue reported in urllib.parse.parse_qsl(). urllib.parse will only us "&" as query string separator by default instead of both ";" and "&" as allowed in earlier versions. An optional argument seperator with default value "&" is added to specify the separator. Co-authored-by: Éric Araujo Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Co-authored-by: Éric Araujo . (cherry picked from commit fcbe0cb04d35189401c0c880ebfb4311e952d776) Co-authored-by: Adam Goldschmidt * Update correct version information. * fix docs and make logic clearer Co-authored-by: Adam Goldschmidt Co-authored-by: Fidget-Spinner <28750310+Fidget-Spinner@users.noreply.github.com> --- Doc/library/cgi.rst | 11 ++- Doc/library/urllib.parse.rst | 22 +++++- Doc/whatsnew/3.6.rst | 13 ++++ Doc/whatsnew/3.7.rst | 13 ++++ Doc/whatsnew/3.8.rst | 13 ++++ Lib/cgi.py | 23 ++++--- Lib/test/test_cgi.py | 29 ++++++-- Lib/test/test_urlparse.py | 68 +++++++++++++------ Lib/urllib/parse.py | 19 ++++-- .../2021-02-14-15-59-16.bpo-42967.YApqDS.rst | 1 + 10 files changed, 166 insertions(+), 46 deletions(-) create mode 100644 Misc/NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst index 4048592e7361f7..880074bed60264 100644 --- a/Doc/library/cgi.rst +++ b/Doc/library/cgi.rst @@ -277,14 +277,16 @@ These are useful if you want more control, or if you want to employ some of the algorithms implemented in this module in other circumstances. -.. function:: parse(fp=None, environ=os.environ, keep_blank_values=False, strict_parsing=False) +.. function:: parse(fp=None, environ=os.environ, keep_blank_values=False, strict_parsing=False, separator="&") Parse a query in the environment or from a file (the file defaults to - ``sys.stdin``). The *keep_blank_values* and *strict_parsing* parameters are + ``sys.stdin``). The *keep_blank_values*, *strict_parsing* and *separator* parameters are passed to :func:`urllib.parse.parse_qs` unchanged. + .. versionchanged:: 3.8.8 + Added the *separator* parameter. -.. function:: parse_multipart(fp, pdict, encoding="utf-8", errors="replace") +.. function:: parse_multipart(fp, pdict, encoding="utf-8", errors="replace", separator="&") Parse input of type :mimetype:`multipart/form-data` (for file uploads). Arguments are *fp* for the input file, *pdict* for a dictionary containing @@ -303,6 +305,9 @@ algorithms implemented in this module in other circumstances. Added the *encoding* and *errors* parameters. For non-file fields, the value is now a list of strings, not bytes. + .. versionchanged:: 3.8.8 + Added the *separator* parameter. + .. function:: parse_header(string) diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst index 25e5cc1a6ce0b2..fcad7076e6c77b 100644 --- a/Doc/library/urllib.parse.rst +++ b/Doc/library/urllib.parse.rst @@ -165,7 +165,7 @@ or on combining URL components into a URL string. now raise :exc:`ValueError`. -.. function:: parse_qs(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace', max_num_fields=None) +.. function:: parse_qs(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace', max_num_fields=None, separator='&') Parse a query string given as a string argument (data of type :mimetype:`application/x-www-form-urlencoded`). Data are returned as a @@ -190,6 +190,9 @@ or on combining URL components into a URL string. read. If set, then throws a :exc:`ValueError` if there are more than *max_num_fields* fields read. + The optional argument *separator* is the symbol to use for separating the + query arguments. It defaults to ``&``. + Use the :func:`urllib.parse.urlencode` function (with the ``doseq`` parameter set to ``True``) to convert such dictionaries into query strings. @@ -201,8 +204,14 @@ or on combining URL components into a URL string. .. versionchanged:: 3.8 Added *max_num_fields* parameter. + .. versionchanged:: 3.8.8 + Added *separator* parameter with the default value of ``&``. Python + versions earlier than Python 3.8.8 allowed using both ``;`` and ``&`` as + query parameter separator. This has been changed to allow only a single + separator key, with ``&`` as the default separator. + -.. function:: parse_qsl(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace', max_num_fields=None) +.. function:: parse_qsl(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace', max_num_fields=None, separator='&') Parse a query string given as a string argument (data of type :mimetype:`application/x-www-form-urlencoded`). Data are returned as a list of @@ -226,6 +235,9 @@ or on combining URL components into a URL string. read. If set, then throws a :exc:`ValueError` if there are more than *max_num_fields* fields read. + The optional argument *separator* is the symbol to use for separating the + query arguments. It defaults to ``&``. + Use the :func:`urllib.parse.urlencode` function to convert such lists of pairs into query strings. @@ -235,6 +247,12 @@ or on combining URL components into a URL string. .. versionchanged:: 3.8 Added *max_num_fields* parameter. + .. versionchanged:: 3.8.8 + Added *separator* parameter with the default value of ``&``. Python + versions earlier than Python 3.8.8 allowed using both ``;`` and ``&`` as + query parameter separator. This has been changed to allow only a single + separator key, with ``&`` as the default separator. + .. function:: urlunparse(parts) diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst index 85a6657fdfbdac..03a877a3d91785 100644 --- a/Doc/whatsnew/3.6.rst +++ b/Doc/whatsnew/3.6.rst @@ -2443,3 +2443,16 @@ because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more details, see the documentation for ``loop.create_datagram_endpoint()``. (Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in :issue:`37228`.) + +Notable changes in Python 3.6.13 +================================ + +Earlier Python versions allowed using both ``;`` and ``&`` as +query parameter separators in :func:`urllib.parse.parse_qs` and +:func:`urllib.parse.parse_qsl`. Due to security concerns, and to conform with +newer W3C recommendations, this has been changed to allow only a single +separator key, with ``&`` as the default. This change also affects +:func:`cgi.parse` and :func:`cgi.parse_multipart` as they use the affected +functions internally. For more details, please see their respective +documentation. +(Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in :issue:`42967`.) diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 4933cba3990b1d..824dc13e0c6fd1 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -2556,3 +2556,16 @@ because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more details, see the documentation for ``loop.create_datagram_endpoint()``. (Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in :issue:`37228`.) + +Notable changes in Python 3.7.10 +================================ + +Earlier Python versions allowed using both ``;`` and ``&`` as +query parameter separators in :func:`urllib.parse.parse_qs` and +:func:`urllib.parse.parse_qsl`. Due to security concerns, and to conform with +newer W3C recommendations, this has been changed to allow only a single +separator key, with ``&`` as the default. This change also affects +:func:`cgi.parse` and :func:`cgi.parse_multipart` as they use the affected +functions internally. For more details, please see their respective +documentation. +(Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in :issue:`42967`.) diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index 1a192800b2f025..632ccc1f2c40ad 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -2251,3 +2251,16 @@ The constant values of future flags in the :mod:`__future__` module are updated in order to prevent collision with compiler flags. Previously ``PyCF_ALLOW_TOP_LEVEL_AWAIT`` was clashing with ``CO_FUTURE_DIVISION``. (Contributed by Batuhan Taskaya in :issue:`39562`) + +Notable changes in Python 3.8.8 +=============================== + +Earlier Python versions allowed using both ``;`` and ``&`` as +query parameter separators in :func:`urllib.parse.parse_qs` and +:func:`urllib.parse.parse_qsl`. Due to security concerns, and to conform with +newer W3C recommendations, this has been changed to allow only a single +separator key, with ``&`` as the default. This change also affects +:func:`cgi.parse` and :func:`cgi.parse_multipart` as they use the affected +functions internally. For more details, please see their respective +documentation. +(Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in :issue:`42967`.) diff --git a/Lib/cgi.py b/Lib/cgi.py index 77ab703cc03600..1e880e51848af2 100755 --- a/Lib/cgi.py +++ b/Lib/cgi.py @@ -115,7 +115,8 @@ def closelog(): # 0 ==> unlimited input maxlen = 0 -def parse(fp=None, environ=os.environ, keep_blank_values=0, strict_parsing=0): +def parse(fp=None, environ=os.environ, keep_blank_values=0, + strict_parsing=0, separator='&'): """Parse a query in the environment or from a file (default stdin) Arguments, all optional: @@ -134,6 +135,9 @@ def parse(fp=None, environ=os.environ, keep_blank_values=0, strict_parsing=0): strict_parsing: flag indicating what to do with parsing errors. If false (the default), errors are silently ignored. If true, errors raise a ValueError exception. + + separator: str. The symbol to use for separating the query arguments. + Defaults to &. """ if fp is None: fp = sys.stdin @@ -154,7 +158,7 @@ def parse(fp=None, environ=os.environ, keep_blank_values=0, strict_parsing=0): if environ['REQUEST_METHOD'] == 'POST': ctype, pdict = parse_header(environ['CONTENT_TYPE']) if ctype == 'multipart/form-data': - return parse_multipart(fp, pdict) + return parse_multipart(fp, pdict, separator=separator) elif ctype == 'application/x-www-form-urlencoded': clength = int(environ['CONTENT_LENGTH']) if maxlen and clength > maxlen: @@ -178,10 +182,10 @@ def parse(fp=None, environ=os.environ, keep_blank_values=0, strict_parsing=0): qs = "" environ['QUERY_STRING'] = qs # XXX Shouldn't, really return urllib.parse.parse_qs(qs, keep_blank_values, strict_parsing, - encoding=encoding) + encoding=encoding, separator=separator) -def parse_multipart(fp, pdict, encoding="utf-8", errors="replace"): +def parse_multipart(fp, pdict, encoding="utf-8", errors="replace", separator='&'): """Parse multipart input. Arguments: @@ -205,7 +209,7 @@ def parse_multipart(fp, pdict, encoding="utf-8", errors="replace"): except KeyError: pass fs = FieldStorage(fp, headers=headers, encoding=encoding, errors=errors, - environ={'REQUEST_METHOD': 'POST'}) + environ={'REQUEST_METHOD': 'POST'}, separator=separator) return {k: fs.getlist(k) for k in fs} def _parseparam(s): @@ -315,7 +319,7 @@ class FieldStorage: def __init__(self, fp=None, headers=None, outerboundary=b'', environ=os.environ, keep_blank_values=0, strict_parsing=0, limit=None, encoding='utf-8', errors='replace', - max_num_fields=None): + max_num_fields=None, separator='&'): """Constructor. Read multipart/* until last part. Arguments, all optional: @@ -363,6 +367,7 @@ def __init__(self, fp=None, headers=None, outerboundary=b'', self.keep_blank_values = keep_blank_values self.strict_parsing = strict_parsing self.max_num_fields = max_num_fields + self.separator = separator if 'REQUEST_METHOD' in environ: method = environ['REQUEST_METHOD'].upper() self.qs_on_post = None @@ -589,7 +594,7 @@ def read_urlencoded(self): query = urllib.parse.parse_qsl( qs, self.keep_blank_values, self.strict_parsing, encoding=self.encoding, errors=self.errors, - max_num_fields=self.max_num_fields) + max_num_fields=self.max_num_fields, separator=self.separator) self.list = [MiniFieldStorage(key, value) for key, value in query] self.skip_lines() @@ -605,7 +610,7 @@ def read_multi(self, environ, keep_blank_values, strict_parsing): query = urllib.parse.parse_qsl( self.qs_on_post, self.keep_blank_values, self.strict_parsing, encoding=self.encoding, errors=self.errors, - max_num_fields=self.max_num_fields) + max_num_fields=self.max_num_fields, separator=self.separator) self.list.extend(MiniFieldStorage(key, value) for key, value in query) klass = self.FieldStorageClass or self.__class__ @@ -649,7 +654,7 @@ def read_multi(self, environ, keep_blank_values, strict_parsing): else self.limit - self.bytes_read part = klass(self.fp, headers, ib, environ, keep_blank_values, strict_parsing, limit, - self.encoding, self.errors, max_num_fields) + self.encoding, self.errors, max_num_fields, self.separator) if max_num_fields is not None: max_num_fields -= 1 diff --git a/Lib/test/test_cgi.py b/Lib/test/test_cgi.py index 101942de947fb4..4e1506a6468b93 100644 --- a/Lib/test/test_cgi.py +++ b/Lib/test/test_cgi.py @@ -53,12 +53,9 @@ def do_test(buf, method): ("", ValueError("bad query field: ''")), ("&", ValueError("bad query field: ''")), ("&&", ValueError("bad query field: ''")), - (";", ValueError("bad query field: ''")), - (";&;", ValueError("bad query field: ''")), # Should the next few really be valid? ("=", {}), ("=&=", {}), - ("=;=", {}), # This rest seem to make sense ("=a", {'': ['a']}), ("&=a", ValueError("bad query field: ''")), @@ -73,8 +70,6 @@ def do_test(buf, method): ("a=a+b&b=b+c", {'a': ['a b'], 'b': ['b c']}), ("a=a+b&a=b+a", {'a': ['a b', 'b a']}), ("x=1&y=2.0&z=2-3.%2b0", {'x': ['1'], 'y': ['2.0'], 'z': ['2-3.+0']}), - ("x=1;y=2.0&z=2-3.%2b0", {'x': ['1'], 'y': ['2.0'], 'z': ['2-3.+0']}), - ("x=1;y=2.0;z=2-3.%2b0", {'x': ['1'], 'y': ['2.0'], 'z': ['2-3.+0']}), ("Hbc5161168c542333633315dee1182227:key_store_seqid=400006&cuyer=r&view=bustomer&order_id=0bb2e248638833d48cb7fed300000f1b&expire=964546263&lobale=en-US&kid=130003.300038&ss=env", {'Hbc5161168c542333633315dee1182227:key_store_seqid': ['400006'], 'cuyer': ['r'], @@ -201,6 +196,30 @@ def test_strict(self): else: self.assertEqual(fs.getvalue(key), expect_val[0]) + def test_separator(self): + parse_semicolon = [ + ("x=1;y=2.0", {'x': ['1'], 'y': ['2.0']}), + ("x=1;y=2.0;z=2-3.%2b0", {'x': ['1'], 'y': ['2.0'], 'z': ['2-3.+0']}), + (";", ValueError("bad query field: ''")), + (";;", ValueError("bad query field: ''")), + ("=;a", ValueError("bad query field: 'a'")), + (";b=a", ValueError("bad query field: ''")), + ("b;=a", ValueError("bad query field: 'b'")), + ("a=a+b;b=b+c", {'a': ['a b'], 'b': ['b c']}), + ("a=a+b;a=b+a", {'a': ['a b', 'b a']}), + ] + for orig, expect in parse_semicolon: + env = {'QUERY_STRING': orig} + fs = cgi.FieldStorage(separator=';', environ=env) + if isinstance(expect, dict): + for key in expect.keys(): + expect_val = expect[key] + self.assertIn(key, fs) + if len(expect_val) > 1: + self.assertEqual(fs.getvalue(key), expect_val) + else: + self.assertEqual(fs.getvalue(key), expect_val[0]) + def test_log(self): cgi.log("Testing") diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py index 4ae6ed33858ce2..90c8d6922629e8 100644 --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -32,16 +32,10 @@ (b"&a=b", [(b'a', b'b')]), (b"a=a+b&b=b+c", [(b'a', b'a b'), (b'b', b'b c')]), (b"a=1&a=2", [(b'a', b'1'), (b'a', b'2')]), - (";", []), - (";;", []), - (";a=b", [('a', 'b')]), - ("a=a+b;b=b+c", [('a', 'a b'), ('b', 'b c')]), - ("a=1;a=2", [('a', '1'), ('a', '2')]), - (b";", []), - (b";;", []), - (b";a=b", [(b'a', b'b')]), - (b"a=a+b;b=b+c", [(b'a', b'a b'), (b'b', b'b c')]), - (b"a=1;a=2", [(b'a', b'1'), (b'a', b'2')]), + (";a=b", [(';a', 'b')]), + ("a=a+b;b=b+c", [('a', 'a b;b=b c')]), + (b";a=b", [(b';a', b'b')]), + (b"a=a+b;b=b+c", [(b'a', b'a b;b=b c')]), ] # Each parse_qs testcase is a two-tuple that contains @@ -68,16 +62,10 @@ (b"&a=b", {b'a': [b'b']}), (b"a=a+b&b=b+c", {b'a': [b'a b'], b'b': [b'b c']}), (b"a=1&a=2", {b'a': [b'1', b'2']}), - (";", {}), - (";;", {}), - (";a=b", {'a': ['b']}), - ("a=a+b;b=b+c", {'a': ['a b'], 'b': ['b c']}), - ("a=1;a=2", {'a': ['1', '2']}), - (b";", {}), - (b";;", {}), - (b";a=b", {b'a': [b'b']}), - (b"a=a+b;b=b+c", {b'a': [b'a b'], b'b': [b'b c']}), - (b"a=1;a=2", {b'a': [b'1', b'2']}), + (";a=b", {';a': ['b']}), + ("a=a+b;b=b+c", {'a': ['a b;b=b c']}), + (b";a=b", {b';a': [b'b']}), + (b"a=a+b;b=b+c", {b'a':[ b'a b;b=b c']}), ] class UrlParseTestCase(unittest.TestCase): @@ -884,10 +872,46 @@ def test_parse_qsl_encoding(self): def test_parse_qsl_max_num_fields(self): with self.assertRaises(ValueError): urllib.parse.parse_qs('&'.join(['a=a']*11), max_num_fields=10) - with self.assertRaises(ValueError): - urllib.parse.parse_qs(';'.join(['a=a']*11), max_num_fields=10) urllib.parse.parse_qs('&'.join(['a=a']*10), max_num_fields=10) + def test_parse_qs_separator(self): + parse_qs_semicolon_cases = [ + (";", {}), + (";;", {}), + (";a=b", {'a': ['b']}), + ("a=a+b;b=b+c", {'a': ['a b'], 'b': ['b c']}), + ("a=1;a=2", {'a': ['1', '2']}), + (b";", {}), + (b";;", {}), + (b";a=b", {b'a': [b'b']}), + (b"a=a+b;b=b+c", {b'a': [b'a b'], b'b': [b'b c']}), + (b"a=1;a=2", {b'a': [b'1', b'2']}), + ] + for orig, expect in parse_qs_semicolon_cases: + with self.subTest(f"Original: {orig!r}, Expected: {expect!r}"): + result = urllib.parse.parse_qs(orig, separator=';') + self.assertEqual(result, expect, "Error parsing %r" % orig) + + + def test_parse_qsl_separator(self): + parse_qsl_semicolon_cases = [ + (";", []), + (";;", []), + (";a=b", [('a', 'b')]), + ("a=a+b;b=b+c", [('a', 'a b'), ('b', 'b c')]), + ("a=1;a=2", [('a', '1'), ('a', '2')]), + (b";", []), + (b";;", []), + (b";a=b", [(b'a', b'b')]), + (b"a=a+b;b=b+c", [(b'a', b'a b'), (b'b', b'b c')]), + (b"a=1;a=2", [(b'a', b'1'), (b'a', b'2')]), + ] + for orig, expect in parse_qsl_semicolon_cases: + with self.subTest(f"Original: {orig!r}, Expected: {expect!r}"): + result = urllib.parse.parse_qsl(orig, separator=';') + self.assertEqual(result, expect, "Error parsing %r" % orig) + + def test_urlencode_sequences(self): # Other tests incidentally urlencode things; test non-covered cases: # Sequence and object values. diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py index 95be7181133b4b..0c1c94f5fc986b 100644 --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -650,7 +650,7 @@ def unquote(string, encoding='utf-8', errors='replace'): def parse_qs(qs, keep_blank_values=False, strict_parsing=False, - encoding='utf-8', errors='replace', max_num_fields=None): + encoding='utf-8', errors='replace', max_num_fields=None, separator='&'): """Parse a query given as a string argument. Arguments: @@ -674,12 +674,15 @@ def parse_qs(qs, keep_blank_values=False, strict_parsing=False, max_num_fields: int. If set, then throws a ValueError if there are more than n fields read by parse_qsl(). + separator: str. The symbol to use for separating the query arguments. + Defaults to &. + Returns a dictionary. """ parsed_result = {} pairs = parse_qsl(qs, keep_blank_values, strict_parsing, encoding=encoding, errors=errors, - max_num_fields=max_num_fields) + max_num_fields=max_num_fields, separator=separator) for name, value in pairs: if name in parsed_result: parsed_result[name].append(value) @@ -689,7 +692,7 @@ def parse_qs(qs, keep_blank_values=False, strict_parsing=False, def parse_qsl(qs, keep_blank_values=False, strict_parsing=False, - encoding='utf-8', errors='replace', max_num_fields=None): + encoding='utf-8', errors='replace', max_num_fields=None, separator='&'): """Parse a query given as a string argument. Arguments: @@ -712,19 +715,25 @@ def parse_qsl(qs, keep_blank_values=False, strict_parsing=False, max_num_fields: int. If set, then throws a ValueError if there are more than n fields read by parse_qsl(). + separator: str. The symbol to use for separating the query arguments. + Defaults to &. + Returns a list, as G-d intended. """ qs, _coerce_result = _coerce_args(qs) + if not separator or (not isinstance(separator, (str, bytes))): + raise ValueError("Separator must be of type string or bytes.") + # If max_num_fields is defined then check that the number of fields # is less than max_num_fields. This prevents a memory exhaustion DOS # attack via post bodies with many fields. if max_num_fields is not None: - num_fields = 1 + qs.count('&') + qs.count(';') + num_fields = 1 + qs.count(separator) if max_num_fields < num_fields: raise ValueError('Max number of fields exceeded') - pairs = [s2 for s1 in qs.split('&') for s2 in s1.split(';')] + pairs = [s1 for s1 in qs.split(separator)] r = [] for name_value in pairs: if not name_value and not strict_parsing: diff --git a/Misc/NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst b/Misc/NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst new file mode 100644 index 00000000000000..f08489b41494ea --- /dev/null +++ b/Misc/NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst @@ -0,0 +1 @@ +Fix web cache poisoning vulnerability by defaulting the query args separator to ``&``, and allowing the user to choose a custom separator. From ede1ff226c9ef4efd053109c69b4e33f75b2b17b Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 15 Feb 2021 10:43:52 -0800 Subject: [PATCH 1980/2163] bpo-43108: Fix a reference leak in the curses module (GH-24420) (GH-24429) (cherry picked from commit bb739ec922c6992a2be38f9fd3c544c2cc322dde) Co-authored-by: Pablo Galindo Co-authored-by: Pablo Galindo --- .../next/Library/2021-02-02-20-23-31.bpo-43108.lqcCZ6.rst | 1 + Modules/_cursesmodule.c | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Library/2021-02-02-20-23-31.bpo-43108.lqcCZ6.rst diff --git a/Misc/NEWS.d/next/Library/2021-02-02-20-23-31.bpo-43108.lqcCZ6.rst b/Misc/NEWS.d/next/Library/2021-02-02-20-23-31.bpo-43108.lqcCZ6.rst new file mode 100644 index 00000000000000..8e45640bceae13 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-02-02-20-23-31.bpo-43108.lqcCZ6.rst @@ -0,0 +1 @@ +Fixed a reference leak in the :mod:`curses` module. Patch by Pablo Galindo diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c index 0f35cdd2861217..35070d94e0c56e 100644 --- a/Modules/_cursesmodule.c +++ b/Modules/_cursesmodule.c @@ -365,6 +365,7 @@ PyCurses_ConvertToString(PyCursesWindowObject *win, PyObject *obj, *bytes = obj; /* check for embedded null bytes */ if (PyBytes_AsStringAndSize(*bytes, &str, NULL) < 0) { + Py_DECREF(obj); return 0; } return 1; @@ -679,8 +680,9 @@ _curses_window_addstr_impl(PyCursesWindowObject *self, int group_left_1, #else strtype = PyCurses_ConvertToString(self, str, &bytesobj, NULL); #endif - if (strtype == 0) + if (strtype == 0) { return NULL; + } if (use_attr) { attr_old = getattrs(self->win); (void)wattrset(self->win,attr); From f30aa3c44f807da5db9b053fba96578a8210a3c2 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 15 Feb 2021 15:12:34 -0800 Subject: [PATCH 1981/2163] Add a warning block around the get_referrers() documentation (GH-24511) (cherry picked from commit 813db24f7c2c536d587d1832c3c52b44fa9e242e) Co-authored-by: Pablo Galindo --- Doc/library/gc.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index adaa30295b5e8b..073391d9058bf7 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -135,10 +135,11 @@ The :mod:`gc` module provides the following functions: resulting referrers. To get only currently live objects, call :func:`collect` before calling :func:`get_referrers`. - Care must be taken when using objects returned by :func:`get_referrers` because - some of them could still be under construction and hence in a temporarily - invalid state. Avoid using :func:`get_referrers` for any purpose other than - debugging. + .. warning:: + Care must be taken when using objects returned by :func:`get_referrers` because + some of them could still be under construction and hence in a temporarily + invalid state. Avoid using :func:`get_referrers` for any purpose other than + debugging. .. function:: get_referents(*objs) From f9d7c12b6c7ab978cb6c61a666bc06dd3fec9b3e Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Mon, 15 Feb 2021 15:47:52 -0800 Subject: [PATCH 1982/2163] bpo-42819, readline: Disable bracketed paste (GH-24108) (cherry picked from commit 755f3c1521b422bc2177013d289f5439975fdc4f) Co-authored-by: Dustin Rodrigues --- Misc/ACKS | 1 + .../2021-01-04-23-54-34.bpo-42819.4KO6wU.rst | 8 +++++++ Modules/readline.c | 23 +++++++++++++++++++ 3 files changed, 32 insertions(+) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2021-01-04-23-54-34.bpo-42819.4KO6wU.rst diff --git a/Misc/ACKS b/Misc/ACKS index 8ca1f64c9f5f6a..e181c6171169e8 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1415,6 +1415,7 @@ Mark Roddy Kevin Rodgers Sean Rodman Giampaolo Rodola +Dustin Rodrigues Mauro S. M. Rodrigues Elson Rodriguez Adi Roiban diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-01-04-23-54-34.bpo-42819.4KO6wU.rst b/Misc/NEWS.d/next/Core and Builtins/2021-01-04-23-54-34.bpo-42819.4KO6wU.rst new file mode 100644 index 00000000000000..d067f0bfa76448 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-01-04-23-54-34.bpo-42819.4KO6wU.rst @@ -0,0 +1,8 @@ +:mod:`readline`: Explicitly disable bracketed paste in the interactive +interpreter, even if it's set in the inputrc, is enabled by default (eg GNU +Readline 8.1), or a user calls ``readline.read_init_file()``. The Python REPL +has not implemented bracketed paste support. Also, bracketed mode writes the +``"\x1b[?2004h"`` escape sequence into stdout which causes test failures in +applications that don't support it. It can still be explicitly enabled by +calling ``readline.parse_and_bind("set enable-bracketed-paste on")``. Patch by +Dustin Rodrigues. diff --git a/Modules/readline.c b/Modules/readline.c index 081657fb236991..9b30d597c84818 100644 --- a/Modules/readline.c +++ b/Modules/readline.c @@ -141,6 +141,26 @@ decode(const char *s) } +/* +Explicitly disable bracketed paste in the interactive interpreter, even if it's +set in the inputrc, is enabled by default (eg GNU Readline 8.1), or a user calls +readline.read_init_file(). The Python REPL has not implemented bracketed +paste support. Also, bracketed mode writes the "\x1b[?2004h" escape sequence +into stdout which causes test failures in applications that don't support it. +It can still be explicitly enabled by calling readline.parse_and_bind("set +enable-bracketed-paste on"). See bpo-42819 for more details. + +This should be removed if bracketed paste mode is implemented (bpo-39820). +*/ + +static void +disable_bracketed_paste(void) +{ + if (!using_libedit_emulation) { + rl_variable_bind ("enable-bracketed-paste", "off"); + } +} + /* Exported function to send one line to readline's init file parser */ static PyObject * @@ -187,6 +207,7 @@ read_init_file(PyObject *self, PyObject *args) errno = rl_read_init_file(NULL); if (errno) return PyErr_SetFromErrno(PyExc_OSError); + disable_bracketed_paste(); Py_RETURN_NONE; } @@ -1146,6 +1167,8 @@ setup_readline(readlinestate *mod_state) else rl_initialize(); + disable_bracketed_paste(); + RESTORE_LOCALE(saved_locale) } From dfd7d6893b57d79b7bfd819d28380a0429b5615f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Tue, 16 Feb 2021 19:12:03 +0100 Subject: [PATCH 1983/2163] Python 3.8.8rc1 --- Include/patchlevel.h | 8 +- Lib/pydoc_data/topics.py | 3 +- Misc/NEWS.d/3.8.8rc1.rst | 367 ++++++++++++++++++ .../2020-02-28-14-33-15.bpo-29076.Gtixi5.rst | 1 - .../2020-03-24-09-27-10.bpo-40052.27P2KG.rst | 2 - .../2021-01-27-10-27-47.bpo-43030.loDcD_.rst | 2 - .../2021-01-04-23-54-34.bpo-42819.4KO6wU.rst | 8 - .../2021-01-04-22-14-22.bpo-42811.HY2beA.rst | 2 - .../2021-01-20-23-03-49.bpo-40304.-LK7Ps.rst | 2 - .../2019-06-30-20-31-09.bpo-32631.e7_4BG.rst | 2 - .../2019-11-14-23-41-07.bpo-23544.3etemb.rst | 2 - .../2020-11-30-19-46-05.bpo-42508.fE7w4M.rst | 3 - .../2021-01-10-01-25-43.bpo-33065.zmyHYJ.rst | 1 - .../2021-01-26-18-12-17.bpo-43008.mbQUc7.rst | 2 - .../2018-04-23-13-44-10.bpo-33289.anBnUr.rst | 2 - .../2019-11-16-22-56-51.bpo-36589.0Io76D.rst | 2 - .../2019-12-16-17-55-31.bpo-39068.Ti3f9P.rst | 2 - .../2020-05-30-14-19-47.bpo-26407.MjWLO1.rst | 3 - .../2020-08-21-15-24-14.bpo-41604.rTXleO.rst | 2 - .../2020-10-11-13-48-03.bpo-42005.Jq6Az-.rst | 2 - .../2020-11-14-13-46-27.bpo-42318.wYAcBD.rst | 1 - .../2020-11-17-14-30-12.bpo-42383.ubl0Y_.rst | 2 - .../2020-11-17-14-32-39.bpo-42384.1ZnQSn.rst | 1 - .../2020-11-22-11-22-28.bpo-42388.LMgM6B.rst | 2 - .../2020-12-02-16-28-04.bpo-42531.2sLlFW.rst | 1 - .../2020-12-20-22-50-15.bpo-42681.lDO6jb.rst | 1 - .../2020-12-27-18-47-01.bpo-23328._xqepZ.rst | 1 - .../2020-12-27-22-19-26.bpo-42759.lGi_03.rst | 3 - .../2021-01-05-21-26-29.bpo-41748.KdC0w3.rst | 2 - .../2021-01-08-15-49-20.bpo-42780.rtqi6B.rst | 1 - .../2021-02-02-20-23-31.bpo-43108.lqcCZ6.rst | 1 - .../2021-01-18-09-27-31.bpo-42938.4Zn4Mp.rst | 2 - .../2021-02-14-15-59-16.bpo-42967.YApqDS.rst | 1 - .../2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst | 1 - .../2021-01-01-08-52-36.bpo-42794.-7-XGz.rst | 2 - .../2020-12-23-19-42-11.bpo-42726.a5EkTv.rst | 2 - .../2020-12-07-11-40-52.bpo-42584.AsYnVX.rst | 1 - .../2021-01-05-20-36-40.bpo-41837.bmS7vB.rst | 1 - .../2020-12-07-11-37-35.bpo-42584.LygmqQ.rst | 1 - .../2021-01-04-00-48-08.bpo-41837.dX-unJ.rst | 1 - README.rst | 4 +- 41 files changed, 374 insertions(+), 76 deletions(-) create mode 100644 Misc/NEWS.d/3.8.8rc1.rst delete mode 100644 Misc/NEWS.d/next/Build/2020-02-28-14-33-15.bpo-29076.Gtixi5.rst delete mode 100644 Misc/NEWS.d/next/C API/2020-03-24-09-27-10.bpo-40052.27P2KG.rst delete mode 100644 Misc/NEWS.d/next/C API/2021-01-27-10-27-47.bpo-43030.loDcD_.rst delete mode 100644 Misc/NEWS.d/next/Core and Builtins/2021-01-04-23-54-34.bpo-42819.4KO6wU.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2021-01-04-22-14-22.bpo-42811.HY2beA.rst delete mode 100644 Misc/NEWS.d/next/Documentation/2021-01-20-23-03-49.bpo-40304.-LK7Ps.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2019-06-30-20-31-09.bpo-32631.e7_4BG.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2019-11-14-23-41-07.bpo-23544.3etemb.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2020-11-30-19-46-05.bpo-42508.fE7w4M.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2021-01-10-01-25-43.bpo-33065.zmyHYJ.rst delete mode 100644 Misc/NEWS.d/next/IDLE/2021-01-26-18-12-17.bpo-43008.mbQUc7.rst delete mode 100644 Misc/NEWS.d/next/Library/2018-04-23-13-44-10.bpo-33289.anBnUr.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-11-16-22-56-51.bpo-36589.0Io76D.rst delete mode 100644 Misc/NEWS.d/next/Library/2019-12-16-17-55-31.bpo-39068.Ti3f9P.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-05-30-14-19-47.bpo-26407.MjWLO1.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-08-21-15-24-14.bpo-41604.rTXleO.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-10-11-13-48-03.bpo-42005.Jq6Az-.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-11-14-13-46-27.bpo-42318.wYAcBD.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-11-17-14-30-12.bpo-42383.ubl0Y_.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-11-17-14-32-39.bpo-42384.1ZnQSn.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-11-22-11-22-28.bpo-42388.LMgM6B.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-12-02-16-28-04.bpo-42531.2sLlFW.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-12-20-22-50-15.bpo-42681.lDO6jb.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-12-27-18-47-01.bpo-23328._xqepZ.rst delete mode 100644 Misc/NEWS.d/next/Library/2020-12-27-22-19-26.bpo-42759.lGi_03.rst delete mode 100644 Misc/NEWS.d/next/Library/2021-01-05-21-26-29.bpo-41748.KdC0w3.rst delete mode 100644 Misc/NEWS.d/next/Library/2021-01-08-15-49-20.bpo-42780.rtqi6B.rst delete mode 100644 Misc/NEWS.d/next/Library/2021-02-02-20-23-31.bpo-43108.lqcCZ6.rst delete mode 100644 Misc/NEWS.d/next/Security/2021-01-18-09-27-31.bpo-42938.4Zn4Mp.rst delete mode 100644 Misc/NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst delete mode 100644 Misc/NEWS.d/next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst delete mode 100644 Misc/NEWS.d/next/Tests/2021-01-01-08-52-36.bpo-42794.-7-XGz.rst delete mode 100644 Misc/NEWS.d/next/Tools-Demos/2020-12-23-19-42-11.bpo-42726.a5EkTv.rst delete mode 100644 Misc/NEWS.d/next/Windows/2020-12-07-11-40-52.bpo-42584.AsYnVX.rst delete mode 100644 Misc/NEWS.d/next/Windows/2021-01-05-20-36-40.bpo-41837.bmS7vB.rst delete mode 100644 Misc/NEWS.d/next/macOS/2020-12-07-11-37-35.bpo-42584.LygmqQ.rst delete mode 100644 Misc/NEWS.d/next/macOS/2021-01-04-00-48-08.bpo-41837.dX-unJ.rst diff --git a/Include/patchlevel.h b/Include/patchlevel.h index e73cadbc6fef67..d35bbfb1cc4540 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -18,12 +18,12 @@ /*--start constants--*/ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 8 -#define PY_MICRO_VERSION 7 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL -#define PY_RELEASE_SERIAL 0 +#define PY_MICRO_VERSION 8 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA +#define PY_RELEASE_SERIAL 1 /* Version as a string */ -#define PY_VERSION "3.8.7+" +#define PY_VERSION "3.8.8rc1" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index 723a9b8ac57bc1..ebdab5733b86a8 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Mon Dec 21 17:22:46 2020 +# Autogenerated by Sphinx on Tue Feb 16 19:10:16 2021 topics = {'assert': 'The "assert" statement\n' '**********************\n' '\n' @@ -11387,7 +11387,6 @@ ' There are two types of integers:\n' '\n' ' Integers ("int")\n' - '\n' ' These represent numbers in an unlimited range, subject to\n' ' available (virtual) memory only. For the purpose of ' 'shift\n' diff --git a/Misc/NEWS.d/3.8.8rc1.rst b/Misc/NEWS.d/3.8.8rc1.rst new file mode 100644 index 00000000000000..df1b7da090ec39 --- /dev/null +++ b/Misc/NEWS.d/3.8.8rc1.rst @@ -0,0 +1,367 @@ +.. bpo: 42967 +.. date: 2021-02-14-15-59-16 +.. nonce: YApqDS +.. release date: 2021-02-16 +.. section: Security + +Fix web cache poisoning vulnerability by defaulting the query args separator +to ``&``, and allowing the user to choose a custom separator. + +.. + +.. bpo: 42938 +.. date: 2021-01-18-09-27-31 +.. nonce: 4Zn4Mp +.. section: Security + +Avoid static buffers when computing the repr of :class:`ctypes.c_double` and +:class:`ctypes.c_longdouble` values. + +.. + +.. bpo: 42819 +.. date: 2021-01-04-23-54-34 +.. nonce: 4KO6wU +.. section: Core and Builtins + +:mod:`readline`: Explicitly disable bracketed paste in the interactive +interpreter, even if it's set in the inputrc, is enabled by default (eg GNU +Readline 8.1), or a user calls ``readline.read_init_file()``. The Python +REPL has not implemented bracketed paste support. Also, bracketed mode +writes the ``"\x1b[?2004h"`` escape sequence into stdout which causes test +failures in applications that don't support it. It can still be explicitly +enabled by calling ``readline.parse_and_bind("set enable-bracketed-paste +on")``. Patch by Dustin Rodrigues. + +.. + +.. bpo: 43108 +.. date: 2021-02-02-20-23-31 +.. nonce: lqcCZ6 +.. section: Library + +Fixed a reference leak in the :mod:`curses` module. Patch by Pablo Galindo + +.. + +.. bpo: 42780 +.. date: 2021-01-08-15-49-20 +.. nonce: rtqi6B +.. section: Library + +Fix os.set_inheritable() for O_PATH file descriptors on Linux. + +.. + +.. bpo: 41748 +.. date: 2021-01-05-21-26-29 +.. nonce: KdC0w3 +.. section: Library + +Fix HTMLParser parsing rules for element attributes containing commas with +spaces. Patch by Karl Dubost. + +.. + +.. bpo: 42759 +.. date: 2020-12-27-22-19-26 +.. nonce: lGi_03 +.. section: Library + +Fixed equality comparison of :class:`tkinter.Variable` and +:class:`tkinter.font.Font`. Objects which belong to different Tcl +interpreters are now always different, even if they have the same name. + +.. + +.. bpo: 23328 +.. date: 2020-12-27-18-47-01 +.. nonce: _xqepZ +.. section: Library + +Allow / character in username, password fields on _PROXY envars. + +.. + +.. bpo: 42681 +.. date: 2020-12-20-22-50-15 +.. nonce: lDO6jb +.. section: Library + +Fixed range checks for color and pair numbers in :mod:`curses`. + +.. + +.. bpo: 42531 +.. date: 2020-12-02-16-28-04 +.. nonce: 2sLlFW +.. section: Library + +:func:`importlib.resources.path` now works for :term:`package`\ s missing +the optional :attr:`__file__` attribute (more specifically, packages whose +:attr:`__spec__`\ ``.``\ :attr:`~importlib.machinery.ModuleSpec.origin` +:keyword:`is` :data:`None`). + +.. + +.. bpo: 42388 +.. date: 2020-11-22-11-22-28 +.. nonce: LMgM6B +.. section: Library + +Fix subprocess.check_output(..., input=None) behavior when text=True to be +consistent with that of the documentation and universal_newlines=True. + +.. + +.. bpo: 42384 +.. date: 2020-11-17-14-32-39 +.. nonce: 1ZnQSn +.. section: Library + +Make pdb populate sys.path[0] exactly the same as regular python execution. + +.. + +.. bpo: 42383 +.. date: 2020-11-17-14-30-12 +.. nonce: ubl0Y_ +.. section: Library + +Fix pdb: previously pdb would fail to restart the debugging target if it was +specified using a relative path and the current directory changed. + +.. + +.. bpo: 42318 +.. date: 2020-11-14-13-46-27 +.. nonce: wYAcBD +.. section: Library + +Fixed support of non-BMP characters in :mod:`tkinter` on macOS. + +.. + +.. bpo: 42005 +.. date: 2020-10-11-13-48-03 +.. nonce: Jq6Az- +.. section: Library + +Fix CLI of :mod:`cProfile` and :mod:`profile` to catch +:exc:`BrokenPipeError`. + +.. + +.. bpo: 41604 +.. date: 2020-08-21-15-24-14 +.. nonce: rTXleO +.. section: Library + +Don't decrement the reference count of the previous user_ptr when +set_panel_userptr fails. + +.. + +.. bpo: 26407 +.. date: 2020-05-30-14-19-47 +.. nonce: MjWLO1 +.. section: Library + +Unexpected errors in calling the ``__iter__`` method are no longer masked by +``TypeError`` in :func:`csv.reader`, :func:`csv.writer.writerow` and +:meth:`csv.writer.writerows`. + +.. + +.. bpo: 39068 +.. date: 2019-12-16-17-55-31 +.. nonce: Ti3f9P +.. section: Library + +Fix initialization race condition in :func:`a85encode` and :func:`b85encode` +in :mod:`base64`. Patch by Brandon Stansbury. + +.. + +.. bpo: 36589 +.. date: 2019-11-16-22-56-51 +.. nonce: 0Io76D +.. section: Library + +The :func:`curses.update_lines_cols` function now returns ``None`` instead +of ``1`` on success. + +.. + +.. bpo: 33289 +.. date: 2018-04-23-13-44-10 +.. nonce: anBnUr +.. section: Library + +Correct call to :mod:`tkinter.colorchooser` to return RGB triplet of ints +instead of floats. Patch by Cheryl Sabella. + +.. + +.. bpo: 40304 +.. date: 2021-01-20-23-03-49 +.. nonce: -LK7Ps +.. section: Documentation + +Fix doc for type(name, bases, dict). Patch by Boris Verkhovskiy and Éric +Araujo. + +.. + +.. bpo: 42811 +.. date: 2021-01-04-22-14-22 +.. nonce: HY2beA +.. section: Documentation + +Updated importlib.utils.resolve_name() doc to use __spec__.parent instead of +__package__. (Thanks Yair Frid.) + +.. + +.. bpo: 42794 +.. date: 2021-01-01-08-52-36 +.. nonce: -7-XGz +.. section: Tests + +Update test_nntplib to use offical group name of news.aioe.org for testing. +Patch by Dong-hee Na. + +.. + +.. bpo: 40810 +.. date: 2020-05-30-10-56-38 +.. nonce: LPqDLQ +.. section: Tests + +In :mod:`sqlite3`, fix ``CheckTraceCallbackContent`` for SQLite pre 3.7.15. + +.. + +.. bpo: 29076 +.. date: 2020-02-28-14-33-15 +.. nonce: Gtixi5 +.. section: Build + +Add fish shell support to macOS installer. + +.. + +.. bpo: 41837 +.. date: 2021-01-05-20-36-40 +.. nonce: bmS7vB +.. section: Windows + +Updated Windows installer to include OpenSSL 1.1.1i + +.. + +.. bpo: 42584 +.. date: 2020-12-07-11-40-52 +.. nonce: AsYnVX +.. section: Windows + +Upgrade Windows installer to use SQLite 3.34.0. + +.. + +.. bpo: 41837 +.. date: 2021-01-04-00-48-08 +.. nonce: dX-unJ +.. section: macOS + +Update macOS installer build to use OpenSSL 1.1.1i. + +.. + +.. bpo: 42584 +.. date: 2020-12-07-11-37-35 +.. nonce: LygmqQ +.. section: macOS + +Update macOS installer to use SQLite 3.34.0. + +.. + +.. bpo: 43008 +.. date: 2021-01-26-18-12-17 +.. nonce: mbQUc7 +.. section: IDLE + +Make IDLE invoke :func:`sys.excepthook` in normal, 2-process mode. Patch by +Ken Hilton. + +.. + +.. bpo: 33065 +.. date: 2021-01-10-01-25-43 +.. nonce: zmyHYJ +.. section: IDLE + +Fix problem debugging user classes with __repr__ method. + +.. + +.. bpo: 42508 +.. date: 2020-11-30-19-46-05 +.. nonce: fE7w4M +.. section: IDLE + +Keep IDLE running on macOS. Remove obsolete workaround that prevented +running files with shortcuts when using new universal2 installers built on +macOS 11. + +.. + +.. bpo: 23544 +.. date: 2019-11-14-23-41-07 +.. nonce: 3etemb +.. section: IDLE + +Disable Debug=>Stack Viewer when user code is running or Debugger is active, +to prevent hang or crash. Patch by Zackery Spytz. + +.. + +.. bpo: 32631 +.. date: 2019-06-30-20-31-09 +.. nonce: e7_4BG +.. section: IDLE + +Finish zzdummy example extension module: make menu entries work; add +docstrings and tests with 100% coverage. + +.. + +.. bpo: 42726 +.. date: 2020-12-23-19-42-11 +.. nonce: a5EkTv +.. section: Tools/Demos + +Fixed Python 3 compatibility issue with gdb/libpython.py handling of +attribute dictionaries. + +.. + +.. bpo: 43030 +.. date: 2021-01-27-10-27-47 +.. nonce: loDcD_ +.. section: C API + +Fixed a compiler warning in :c:func:`Py_UNICODE_ISSPACE()` on platforms with +signed ``wchar_t``. + +.. + +.. bpo: 40052 +.. date: 2020-03-24-09-27-10 +.. nonce: 27P2KG +.. section: C API + +Fix an alignment build warning/error in function +``PyVectorcall_Function()``. Patch by Andreas Schneider, Antoine Pitrou and +Petr Viktorin. diff --git a/Misc/NEWS.d/next/Build/2020-02-28-14-33-15.bpo-29076.Gtixi5.rst b/Misc/NEWS.d/next/Build/2020-02-28-14-33-15.bpo-29076.Gtixi5.rst deleted file mode 100644 index b38beb0586951c..00000000000000 --- a/Misc/NEWS.d/next/Build/2020-02-28-14-33-15.bpo-29076.Gtixi5.rst +++ /dev/null @@ -1 +0,0 @@ -Add fish shell support to macOS installer. diff --git a/Misc/NEWS.d/next/C API/2020-03-24-09-27-10.bpo-40052.27P2KG.rst b/Misc/NEWS.d/next/C API/2020-03-24-09-27-10.bpo-40052.27P2KG.rst deleted file mode 100644 index 538488e2fbaccd..00000000000000 --- a/Misc/NEWS.d/next/C API/2020-03-24-09-27-10.bpo-40052.27P2KG.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix an alignment build warning/error in function ``PyVectorcall_Function()``. -Patch by Andreas Schneider, Antoine Pitrou and Petr Viktorin. diff --git a/Misc/NEWS.d/next/C API/2021-01-27-10-27-47.bpo-43030.loDcD_.rst b/Misc/NEWS.d/next/C API/2021-01-27-10-27-47.bpo-43030.loDcD_.rst deleted file mode 100644 index 7a432522db8a12..00000000000000 --- a/Misc/NEWS.d/next/C API/2021-01-27-10-27-47.bpo-43030.loDcD_.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed a compiler warning in :c:func:`Py_UNICODE_ISSPACE()` on platforms with -signed ``wchar_t``. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-01-04-23-54-34.bpo-42819.4KO6wU.rst b/Misc/NEWS.d/next/Core and Builtins/2021-01-04-23-54-34.bpo-42819.4KO6wU.rst deleted file mode 100644 index d067f0bfa76448..00000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-01-04-23-54-34.bpo-42819.4KO6wU.rst +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`readline`: Explicitly disable bracketed paste in the interactive -interpreter, even if it's set in the inputrc, is enabled by default (eg GNU -Readline 8.1), or a user calls ``readline.read_init_file()``. The Python REPL -has not implemented bracketed paste support. Also, bracketed mode writes the -``"\x1b[?2004h"`` escape sequence into stdout which causes test failures in -applications that don't support it. It can still be explicitly enabled by -calling ``readline.parse_and_bind("set enable-bracketed-paste on")``. Patch by -Dustin Rodrigues. diff --git a/Misc/NEWS.d/next/Documentation/2021-01-04-22-14-22.bpo-42811.HY2beA.rst b/Misc/NEWS.d/next/Documentation/2021-01-04-22-14-22.bpo-42811.HY2beA.rst deleted file mode 100644 index 768508e0ce1c19..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-01-04-22-14-22.bpo-42811.HY2beA.rst +++ /dev/null @@ -1,2 +0,0 @@ -Updated importlib.utils.resolve_name() doc to use __spec__.parent -instead of __package__. (Thanks Yair Frid.) diff --git a/Misc/NEWS.d/next/Documentation/2021-01-20-23-03-49.bpo-40304.-LK7Ps.rst b/Misc/NEWS.d/next/Documentation/2021-01-20-23-03-49.bpo-40304.-LK7Ps.rst deleted file mode 100644 index 3f2f14c2d7b893..00000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-01-20-23-03-49.bpo-40304.-LK7Ps.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix doc for type(name, bases, dict). Patch by Boris Verkhovskiy and -Éric Araujo. diff --git a/Misc/NEWS.d/next/IDLE/2019-06-30-20-31-09.bpo-32631.e7_4BG.rst b/Misc/NEWS.d/next/IDLE/2019-06-30-20-31-09.bpo-32631.e7_4BG.rst deleted file mode 100644 index c422f43b6d6dd8..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2019-06-30-20-31-09.bpo-32631.e7_4BG.rst +++ /dev/null @@ -1,2 +0,0 @@ -Finish zzdummy example extension module: make menu entries work; -add docstrings and tests with 100% coverage. diff --git a/Misc/NEWS.d/next/IDLE/2019-11-14-23-41-07.bpo-23544.3etemb.rst b/Misc/NEWS.d/next/IDLE/2019-11-14-23-41-07.bpo-23544.3etemb.rst deleted file mode 100644 index eb4a56bf100b59..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2019-11-14-23-41-07.bpo-23544.3etemb.rst +++ /dev/null @@ -1,2 +0,0 @@ -Disable Debug=>Stack Viewer when user code is running or Debugger -is active, to prevent hang or crash. Patch by Zackery Spytz. diff --git a/Misc/NEWS.d/next/IDLE/2020-11-30-19-46-05.bpo-42508.fE7w4M.rst b/Misc/NEWS.d/next/IDLE/2020-11-30-19-46-05.bpo-42508.fE7w4M.rst deleted file mode 100644 index b449351f7f458e..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2020-11-30-19-46-05.bpo-42508.fE7w4M.rst +++ /dev/null @@ -1,3 +0,0 @@ -Keep IDLE running on macOS. Remove obsolete workaround that prevented -running files with shortcuts when using new universal2 installers built -on macOS 11. diff --git a/Misc/NEWS.d/next/IDLE/2021-01-10-01-25-43.bpo-33065.zmyHYJ.rst b/Misc/NEWS.d/next/IDLE/2021-01-10-01-25-43.bpo-33065.zmyHYJ.rst deleted file mode 100644 index 87948f3cd1baa1..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2021-01-10-01-25-43.bpo-33065.zmyHYJ.rst +++ /dev/null @@ -1 +0,0 @@ -Fix problem debugging user classes with __repr__ method. diff --git a/Misc/NEWS.d/next/IDLE/2021-01-26-18-12-17.bpo-43008.mbQUc7.rst b/Misc/NEWS.d/next/IDLE/2021-01-26-18-12-17.bpo-43008.mbQUc7.rst deleted file mode 100644 index 55ab67ca94959a..00000000000000 --- a/Misc/NEWS.d/next/IDLE/2021-01-26-18-12-17.bpo-43008.mbQUc7.rst +++ /dev/null @@ -1,2 +0,0 @@ -Make IDLE invoke :func:`sys.excepthook` in normal, 2-process mode. -Patch by Ken Hilton. diff --git a/Misc/NEWS.d/next/Library/2018-04-23-13-44-10.bpo-33289.anBnUr.rst b/Misc/NEWS.d/next/Library/2018-04-23-13-44-10.bpo-33289.anBnUr.rst deleted file mode 100644 index 52d9ac9dd902cd..00000000000000 --- a/Misc/NEWS.d/next/Library/2018-04-23-13-44-10.bpo-33289.anBnUr.rst +++ /dev/null @@ -1,2 +0,0 @@ -Correct call to :mod:`tkinter.colorchooser` to return RGB triplet of ints -instead of floats. Patch by Cheryl Sabella. diff --git a/Misc/NEWS.d/next/Library/2019-11-16-22-56-51.bpo-36589.0Io76D.rst b/Misc/NEWS.d/next/Library/2019-11-16-22-56-51.bpo-36589.0Io76D.rst deleted file mode 100644 index 3c1221b2034947..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-16-22-56-51.bpo-36589.0Io76D.rst +++ /dev/null @@ -1,2 +0,0 @@ -The :func:`curses.update_lines_cols` function now returns ``None`` instead -of ``1`` on success. diff --git a/Misc/NEWS.d/next/Library/2019-12-16-17-55-31.bpo-39068.Ti3f9P.rst b/Misc/NEWS.d/next/Library/2019-12-16-17-55-31.bpo-39068.Ti3f9P.rst deleted file mode 100644 index fe6503fdce6b63..00000000000000 --- a/Misc/NEWS.d/next/Library/2019-12-16-17-55-31.bpo-39068.Ti3f9P.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix initialization race condition in :func:`a85encode` and :func:`b85encode` -in :mod:`base64`. Patch by Brandon Stansbury. diff --git a/Misc/NEWS.d/next/Library/2020-05-30-14-19-47.bpo-26407.MjWLO1.rst b/Misc/NEWS.d/next/Library/2020-05-30-14-19-47.bpo-26407.MjWLO1.rst deleted file mode 100644 index d0e45cf1b1f2f4..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-05-30-14-19-47.bpo-26407.MjWLO1.rst +++ /dev/null @@ -1,3 +0,0 @@ -Unexpected errors in calling the ``__iter__`` method are no longer masked -by ``TypeError`` in :func:`csv.reader`, :func:`csv.writer.writerow` and -:meth:`csv.writer.writerows`. diff --git a/Misc/NEWS.d/next/Library/2020-08-21-15-24-14.bpo-41604.rTXleO.rst b/Misc/NEWS.d/next/Library/2020-08-21-15-24-14.bpo-41604.rTXleO.rst deleted file mode 100644 index 0f9794cbdb321e..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-08-21-15-24-14.bpo-41604.rTXleO.rst +++ /dev/null @@ -1,2 +0,0 @@ -Don't decrement the reference count of the previous user_ptr when -set_panel_userptr fails. diff --git a/Misc/NEWS.d/next/Library/2020-10-11-13-48-03.bpo-42005.Jq6Az-.rst b/Misc/NEWS.d/next/Library/2020-10-11-13-48-03.bpo-42005.Jq6Az-.rst deleted file mode 100644 index be4ed7f55ffded..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-10-11-13-48-03.bpo-42005.Jq6Az-.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix CLI of :mod:`cProfile` and :mod:`profile` to catch -:exc:`BrokenPipeError`. diff --git a/Misc/NEWS.d/next/Library/2020-11-14-13-46-27.bpo-42318.wYAcBD.rst b/Misc/NEWS.d/next/Library/2020-11-14-13-46-27.bpo-42318.wYAcBD.rst deleted file mode 100644 index e72daebb2f152a..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-11-14-13-46-27.bpo-42318.wYAcBD.rst +++ /dev/null @@ -1 +0,0 @@ -Fixed support of non-BMP characters in :mod:`tkinter` on macOS. diff --git a/Misc/NEWS.d/next/Library/2020-11-17-14-30-12.bpo-42383.ubl0Y_.rst b/Misc/NEWS.d/next/Library/2020-11-17-14-30-12.bpo-42383.ubl0Y_.rst deleted file mode 100644 index ccf2106f28a93d..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-11-17-14-30-12.bpo-42383.ubl0Y_.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix pdb: previously pdb would fail to restart the debugging target if it was -specified using a relative path and the current directory changed. diff --git a/Misc/NEWS.d/next/Library/2020-11-17-14-32-39.bpo-42384.1ZnQSn.rst b/Misc/NEWS.d/next/Library/2020-11-17-14-32-39.bpo-42384.1ZnQSn.rst deleted file mode 100644 index ae990162fbab75..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-11-17-14-32-39.bpo-42384.1ZnQSn.rst +++ /dev/null @@ -1 +0,0 @@ -Make pdb populate sys.path[0] exactly the same as regular python execution. diff --git a/Misc/NEWS.d/next/Library/2020-11-22-11-22-28.bpo-42388.LMgM6B.rst b/Misc/NEWS.d/next/Library/2020-11-22-11-22-28.bpo-42388.LMgM6B.rst deleted file mode 100644 index 1b19247e841487..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-11-22-11-22-28.bpo-42388.LMgM6B.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix subprocess.check_output(..., input=None) behavior when text=True to be -consistent with that of the documentation and universal_newlines=True. diff --git a/Misc/NEWS.d/next/Library/2020-12-02-16-28-04.bpo-42531.2sLlFW.rst b/Misc/NEWS.d/next/Library/2020-12-02-16-28-04.bpo-42531.2sLlFW.rst deleted file mode 100644 index 7927078acda430..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-12-02-16-28-04.bpo-42531.2sLlFW.rst +++ /dev/null @@ -1 +0,0 @@ -:func:`importlib.resources.path` now works for :term:`package`\ s missing the optional :attr:`__file__` attribute (more specifically, packages whose :attr:`__spec__`\ ``.``\ :attr:`~importlib.machinery.ModuleSpec.origin` :keyword:`is` :data:`None`). \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2020-12-20-22-50-15.bpo-42681.lDO6jb.rst b/Misc/NEWS.d/next/Library/2020-12-20-22-50-15.bpo-42681.lDO6jb.rst deleted file mode 100644 index 34ea74a5a323d8..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-12-20-22-50-15.bpo-42681.lDO6jb.rst +++ /dev/null @@ -1 +0,0 @@ -Fixed range checks for color and pair numbers in :mod:`curses`. diff --git a/Misc/NEWS.d/next/Library/2020-12-27-18-47-01.bpo-23328._xqepZ.rst b/Misc/NEWS.d/next/Library/2020-12-27-18-47-01.bpo-23328._xqepZ.rst deleted file mode 100644 index 07b15d34e8d56b..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-12-27-18-47-01.bpo-23328._xqepZ.rst +++ /dev/null @@ -1 +0,0 @@ -Allow / character in username, password fields on _PROXY envars. diff --git a/Misc/NEWS.d/next/Library/2020-12-27-22-19-26.bpo-42759.lGi_03.rst b/Misc/NEWS.d/next/Library/2020-12-27-22-19-26.bpo-42759.lGi_03.rst deleted file mode 100644 index a5ec7d5820336d..00000000000000 --- a/Misc/NEWS.d/next/Library/2020-12-27-22-19-26.bpo-42759.lGi_03.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fixed equality comparison of :class:`tkinter.Variable` and -:class:`tkinter.font.Font`. Objects which belong to different Tcl -interpreters are now always different, even if they have the same name. diff --git a/Misc/NEWS.d/next/Library/2021-01-05-21-26-29.bpo-41748.KdC0w3.rst b/Misc/NEWS.d/next/Library/2021-01-05-21-26-29.bpo-41748.KdC0w3.rst deleted file mode 100644 index 52efa3ac3d40eb..00000000000000 --- a/Misc/NEWS.d/next/Library/2021-01-05-21-26-29.bpo-41748.KdC0w3.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix HTMLParser parsing rules for element attributes containing -commas with spaces. Patch by Karl Dubost. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-01-08-15-49-20.bpo-42780.rtqi6B.rst b/Misc/NEWS.d/next/Library/2021-01-08-15-49-20.bpo-42780.rtqi6B.rst deleted file mode 100644 index a491690507129e..00000000000000 --- a/Misc/NEWS.d/next/Library/2021-01-08-15-49-20.bpo-42780.rtqi6B.rst +++ /dev/null @@ -1 +0,0 @@ -Fix os.set_inheritable() for O_PATH file descriptors on Linux. diff --git a/Misc/NEWS.d/next/Library/2021-02-02-20-23-31.bpo-43108.lqcCZ6.rst b/Misc/NEWS.d/next/Library/2021-02-02-20-23-31.bpo-43108.lqcCZ6.rst deleted file mode 100644 index 8e45640bceae13..00000000000000 --- a/Misc/NEWS.d/next/Library/2021-02-02-20-23-31.bpo-43108.lqcCZ6.rst +++ /dev/null @@ -1 +0,0 @@ -Fixed a reference leak in the :mod:`curses` module. Patch by Pablo Galindo diff --git a/Misc/NEWS.d/next/Security/2021-01-18-09-27-31.bpo-42938.4Zn4Mp.rst b/Misc/NEWS.d/next/Security/2021-01-18-09-27-31.bpo-42938.4Zn4Mp.rst deleted file mode 100644 index 7df65a156feabd..00000000000000 --- a/Misc/NEWS.d/next/Security/2021-01-18-09-27-31.bpo-42938.4Zn4Mp.rst +++ /dev/null @@ -1,2 +0,0 @@ -Avoid static buffers when computing the repr of :class:`ctypes.c_double` and -:class:`ctypes.c_longdouble` values. diff --git a/Misc/NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst b/Misc/NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst deleted file mode 100644 index f08489b41494ea..00000000000000 --- a/Misc/NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst +++ /dev/null @@ -1 +0,0 @@ -Fix web cache poisoning vulnerability by defaulting the query args separator to ``&``, and allowing the user to choose a custom separator. diff --git a/Misc/NEWS.d/next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst b/Misc/NEWS.d/next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst deleted file mode 100644 index eafd94cabede93..00000000000000 --- a/Misc/NEWS.d/next/Tests/2020-05-30-10-56-38.bpo-40810.LPqDLQ.rst +++ /dev/null @@ -1 +0,0 @@ -In :mod:`sqlite3`, fix ``CheckTraceCallbackContent`` for SQLite pre 3.7.15. diff --git a/Misc/NEWS.d/next/Tests/2021-01-01-08-52-36.bpo-42794.-7-XGz.rst b/Misc/NEWS.d/next/Tests/2021-01-01-08-52-36.bpo-42794.-7-XGz.rst deleted file mode 100644 index 577f2259e1f00c..00000000000000 --- a/Misc/NEWS.d/next/Tests/2021-01-01-08-52-36.bpo-42794.-7-XGz.rst +++ /dev/null @@ -1,2 +0,0 @@ -Update test_nntplib to use offical group name of news.aioe.org for testing. -Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/Tools-Demos/2020-12-23-19-42-11.bpo-42726.a5EkTv.rst b/Misc/NEWS.d/next/Tools-Demos/2020-12-23-19-42-11.bpo-42726.a5EkTv.rst deleted file mode 100644 index 01a6e7fe55f5b3..00000000000000 --- a/Misc/NEWS.d/next/Tools-Demos/2020-12-23-19-42-11.bpo-42726.a5EkTv.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed Python 3 compatibility issue with gdb/libpython.py handling of attribute -dictionaries. diff --git a/Misc/NEWS.d/next/Windows/2020-12-07-11-40-52.bpo-42584.AsYnVX.rst b/Misc/NEWS.d/next/Windows/2020-12-07-11-40-52.bpo-42584.AsYnVX.rst deleted file mode 100644 index afb6530c8f66d7..00000000000000 --- a/Misc/NEWS.d/next/Windows/2020-12-07-11-40-52.bpo-42584.AsYnVX.rst +++ /dev/null @@ -1 +0,0 @@ -Upgrade Windows installer to use SQLite 3.34.0. diff --git a/Misc/NEWS.d/next/Windows/2021-01-05-20-36-40.bpo-41837.bmS7vB.rst b/Misc/NEWS.d/next/Windows/2021-01-05-20-36-40.bpo-41837.bmS7vB.rst deleted file mode 100644 index 8d4bb34ff909c2..00000000000000 --- a/Misc/NEWS.d/next/Windows/2021-01-05-20-36-40.bpo-41837.bmS7vB.rst +++ /dev/null @@ -1 +0,0 @@ -Updated Windows installer to include OpenSSL 1.1.1i diff --git a/Misc/NEWS.d/next/macOS/2020-12-07-11-37-35.bpo-42584.LygmqQ.rst b/Misc/NEWS.d/next/macOS/2020-12-07-11-37-35.bpo-42584.LygmqQ.rst deleted file mode 100644 index 2a625f98e90781..00000000000000 --- a/Misc/NEWS.d/next/macOS/2020-12-07-11-37-35.bpo-42584.LygmqQ.rst +++ /dev/null @@ -1 +0,0 @@ -Update macOS installer to use SQLite 3.34.0. diff --git a/Misc/NEWS.d/next/macOS/2021-01-04-00-48-08.bpo-41837.dX-unJ.rst b/Misc/NEWS.d/next/macOS/2021-01-04-00-48-08.bpo-41837.dX-unJ.rst deleted file mode 100644 index 3f9415f4a36064..00000000000000 --- a/Misc/NEWS.d/next/macOS/2021-01-04-00-48-08.bpo-41837.dX-unJ.rst +++ /dev/null @@ -1 +0,0 @@ -Update macOS installer build to use OpenSSL 1.1.1i. diff --git a/README.rst b/README.rst index 4907629a0f69d6..d6d9c5b2b1bef0 100644 --- a/README.rst +++ b/README.rst @@ -1,5 +1,5 @@ -This is Python version 3.8.7 -============================ +This is Python version 3.8.8rc1 +=============================== .. image:: https://travis-ci.org/python/cpython.svg?branch=3.8 :alt: CPython build status on Travis CI From b0955208da6f5e806de733562fb66506c95093a6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Tue, 16 Feb 2021 20:07:06 +0100 Subject: [PATCH 1984/2163] Post 3.8.8rc1 --- Include/patchlevel.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h index d35bbfb1cc4540..5ed226c9e975fd 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 1 /* Version as a string */ -#define PY_VERSION "3.8.8rc1" +#define PY_VERSION "3.8.8rc1+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. From f03b61aaed28e005305210979261d5040c9df1aa Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 18 Feb 2021 12:07:24 -0800 Subject: [PATCH 1985/2163] Remove all links to mingw.org (GH-24552) This lease on this domain has lapsed. This not only makes these dead links, but a potential attack vector for readers of python.org as the domain can be obtained by an untrustworthy party. I considered redirecting these links to http://mingw-w64.org/ which is a maintained fork of mingw, but beyond my unfamiliarity with the exact level of compatibility, at the time of this PR that site had an expired cert and so is not much of a vulnerability fix. Automerge-Triggered-By: GH:Mariatta (cherry picked from commit 743932d50815edbe4ffd530ae091c53ae47dd34b) Co-authored-by: Jeremy Paige --- Doc/install/index.rst | 3 +-- Doc/using/windows.rst | 2 -- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/Doc/install/index.rst b/Doc/install/index.rst index e6d5a3e6ebde60..ae0d0294ef7773 100644 --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -1064,8 +1064,7 @@ normal libraries do. .. [#] This also means you could replace all existing COFF-libraries with OMF-libraries of the same name. -.. [#] Check https://www.sourceware.org/cygwin/ and http://www.mingw.org/ for more - information +.. [#] Check https://www.sourceware.org/cygwin/ for more information .. [#] Then you have no POSIX emulation available, but you also don't need :file:`cygwin1.dll`. diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index 1d5e9e4b5d9f70..12ec0fc90c0ff7 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -1161,8 +1161,6 @@ For extension modules, consult :ref:`building-on-windows`. MinGW gcc under Windows" or "Installing Python extension with distutils and without Microsoft Visual C++" by SĂ©bastien Sauvage, 2003 - `MingW -- Python extensions `_ - Other Platforms =============== From 8a42eb1492dec2ec0cc79146f0c817acb7328b19 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Thu, 18 Feb 2021 18:03:34 -0800 Subject: [PATCH 1986/2163] closes bpo-43254: Fix *snprintf() man page refs. (GH-24563) (cherry picked from commit e92d67dfbb4790df37aa6a0961fb6dc7e8d2fbbf) Co-authored-by: Erlend Egeberg Aasland --- Doc/c-api/conversion.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/c-api/conversion.rst b/Doc/c-api/conversion.rst index b310fcb5e4f91e..ee76accae62b8d 100644 --- a/Doc/c-api/conversion.rst +++ b/Doc/c-api/conversion.rst @@ -11,14 +11,14 @@ Functions for number conversion and formatted string output. .. c:function:: int PyOS_snprintf(char *str, size_t size, const char *format, ...) Output not more than *size* bytes to *str* according to the format string - *format* and the extra arguments. See the Unix man page :manpage:`snprintf(2)`. + *format* and the extra arguments. See the Unix man page :manpage:`snprintf(3)`. .. c:function:: int PyOS_vsnprintf(char *str, size_t size, const char *format, va_list va) Output not more than *size* bytes to *str* according to the format string *format* and the variable argument list *va*. Unix man page - :manpage:`vsnprintf(2)`. + :manpage:`vsnprintf(3)`. :c:func:`PyOS_snprintf` and :c:func:`PyOS_vsnprintf` wrap the Standard C library functions :c:func:`snprintf` and :c:func:`vsnprintf`. Their purpose is to From 024d8058b0cb2b13dfd65c8e3804a6b194a151ba Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Fri, 19 Feb 2021 11:28:41 +0100 Subject: [PATCH 1987/2163] Python 3.8.8 --- Include/patchlevel.h | 6 +++--- Lib/pydoc_data/topics.py | 2 +- Misc/NEWS.d/3.8.8.rst | 8 ++++++++ README.rst | 4 ++-- 4 files changed, 14 insertions(+), 6 deletions(-) create mode 100644 Misc/NEWS.d/3.8.8.rst diff --git a/Include/patchlevel.h b/Include/patchlevel.h index 5ed226c9e975fd..b4dbb02abaedb2 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -19,11 +19,11 @@ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 8 #define PY_MICRO_VERSION 8 -#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA -#define PY_RELEASE_SERIAL 1 +#define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL +#define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.8.8rc1+" +#define PY_VERSION "3.8.8" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index ebdab5733b86a8..475c256be57a25 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Tue Feb 16 19:10:16 2021 +# Autogenerated by Sphinx on Fri Feb 19 11:26:48 2021 topics = {'assert': 'The "assert" statement\n' '**********************\n' '\n' diff --git a/Misc/NEWS.d/3.8.8.rst b/Misc/NEWS.d/3.8.8.rst new file mode 100644 index 00000000000000..b6fc112ce92be4 --- /dev/null +++ b/Misc/NEWS.d/3.8.8.rst @@ -0,0 +1,8 @@ +.. bpo: 0 +.. date: 2021-02-19 +.. no changes: True +.. nonce: -KzyCx +.. release date: 2021-02-19 +.. section: Library + +There were no new changes in version 3.8.8. diff --git a/README.rst b/README.rst index d6d9c5b2b1bef0..18223f9f4e972c 100644 --- a/README.rst +++ b/README.rst @@ -1,5 +1,5 @@ -This is Python version 3.8.8rc1 -=============================== +This is Python version 3.8.8 +============================ .. image:: https://travis-ci.org/python/cpython.svg?branch=3.8 :alt: CPython build status on Travis CI From 8f4919afb3c7d82031e8d2579c15aea7eb4bb777 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Fri, 19 Feb 2021 13:16:45 +0100 Subject: [PATCH 1988/2163] Post 3.8.8 --- Include/patchlevel.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h index b4dbb02abaedb2..e96ee8266fbcec 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.8.8" +#define PY_VERSION "3.8.8+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. From a072788c57f7a40ecc53cb32f795f4ec844c0aba Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 19 Feb 2021 16:36:07 -0800 Subject: [PATCH 1989/2163] bpo-43042: Augment tutorial sentence (GH-24514) Calling same function also gets new local namespace. (cherry picked from commit b30fcba3a8abaabd1087f2392ae8aec4c1b1f210) Co-authored-by: Terry Jan Reedy --- Doc/tutorial/controlflow.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Doc/tutorial/controlflow.rst b/Doc/tutorial/controlflow.rst index 3af288a17b270d..97b4c6363a239c 100644 --- a/Doc/tutorial/controlflow.rst +++ b/Doc/tutorial/controlflow.rst @@ -294,7 +294,8 @@ referenced. The actual parameters (arguments) to a function call are introduced in the local symbol table of the called function when it is called; thus, arguments are passed using *call by value* (where the *value* is always an object *reference*, -not the value of the object). [#]_ When a function calls another function, a new +not the value of the object). [#]_ When a function calls another function, +or calls itself recursively, a new local symbol table is created for that call. A function definition associates the function name with the function object in From 9a12c213c6f26876c9e00a8cebef3bc9030b6b99 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Fri, 19 Feb 2021 20:31:18 -0800 Subject: [PATCH 1990/2163] Fix typo in dis module doc (GH-24509) (cherry picked from commit 292f23186c6573bc1c3fa4f6e91116e8ee444cf4) Co-authored-by: Irit Katriel --- Doc/library/dis.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 015be2029502a5..f282415cc33d48 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -346,7 +346,7 @@ The Python compiler currently generates the following bytecode instructions. .. opcode:: ROT_FOUR - Lifts second, third and forth stack items one position up, moves top down + Lifts second, third and fourth stack items one position up, moves top down to position four. .. versionadded:: 3.8 From 089a21f7429a3fd3cf78e3779df724757d346d19 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sat, 20 Feb 2021 18:42:24 -0800 Subject: [PATCH 1991/2163] bpo-27646: Say that 'yield from' expression can be any iterable (GH-24595) Previously, the doc at least strongly implied that it had to be an iterator. (cherry picked from commit 2f9ef514fb24b6a95bd3272885f197752810c107) Co-authored-by: Terry Jan Reedy --- Doc/reference/expressions.rst | 4 ++-- .../Documentation/2021-02-20-00-09-13.bpo-27646.HRsmo-.rst | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Documentation/2021-02-20-00-09-13.bpo-27646.HRsmo-.rst diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 81dd6fc860355f..0ac45995aef9c4 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -476,8 +476,8 @@ allowing any pending :keyword:`finally` clauses to execute. .. index:: single: from; yield from expression -When ``yield from `` is used, it treats the supplied expression as -a subiterator. All values produced by that subiterator are passed directly +When ``yield from `` is used, the supplied expression must be an +iterable. The values produced by iterating that iterable are passed directly to the caller of the current generator's methods. Any values passed in with :meth:`~generator.send` and any exceptions passed in with :meth:`~generator.throw` are passed to the underlying iterator if it has the diff --git a/Misc/NEWS.d/next/Documentation/2021-02-20-00-09-13.bpo-27646.HRsmo-.rst b/Misc/NEWS.d/next/Documentation/2021-02-20-00-09-13.bpo-27646.HRsmo-.rst new file mode 100644 index 00000000000000..8ba398adf61826 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-02-20-00-09-13.bpo-27646.HRsmo-.rst @@ -0,0 +1,2 @@ +Clarify that 'yield from ' works with any iterable, not just +iterators. From 6ddb25586524022923d076ddb3b5867214c6ce42 Mon Sep 17 00:00:00 2001 From: "Miss Islington (bot)" <31488909+miss-islington@users.noreply.github.com> Date: Sun, 21 Feb 2021 00:04:11 -0800 Subject: [PATCH 1992/2163] bpo-43283: Rearrange some IDLE doc paragraphs. (GH-24604) In the Running User Code section, gather together paragraphs about two processes and the sys.stdstream replacements, preparing to add another. (cherry picked from commit 4cf7bb8e22bf37e6d65bf4cb5618d09c4a8ad612) Co-authored-by: Terry Jan Reedy --- Doc/library/idle.rst | 29 ++++++++++++++-------------- Lib/idlelib/help.html | 44 +++++++++++++++++++++---------------------- 2 files changed, 37 insertions(+), 36 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst index e7eaabd8bfa25a..fc45e3161bc6d5 100644 --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -726,28 +726,29 @@ with objects that get input from and send output to the Shell window. The original values stored in ``sys.__stdin__``, ``sys.__stdout__``, and ``sys.__stderr__`` are not touched, but may be ``None``. -When Shell has the focus, it controls the keyboard and screen. This is -normally transparent, but functions that directly access the keyboard -and screen will not work. These include system-specific functions that -determine whether a key has been pressed and if so, which. - IDLE's standard stream replacements are not inherited by subprocesses -created in the execution process, whether directly by user code or by modules -such as multiprocessing. If such subprocess use ``input`` from sys.stdin -or ``print`` or ``write`` to sys.stdout or sys.stderr, +created in the execution process, whether directly by user code or by +modules such as multiprocessing. If such subprocess use ``input`` from +sys.stdin or ``print`` or ``write`` to sys.stdout or sys.stderr, IDLE should be started in a command line window. The secondary subprocess will then be attached to that window for input and output. -The IDLE code running in the execution process adds frames to the call stack -that would not be there otherwise. IDLE wraps ``sys.getrecursionlimit`` and -``sys.setrecursionlimit`` to reduce the effect of the additional stack frames. - If ``sys`` is reset by user code, such as with ``importlib.reload(sys)``, IDLE's changes are lost and input from the keyboard and output to the screen will not work correctly. -When user code raises SystemExit either directly or by calling sys.exit, IDLE -returns to a Shell prompt instead of exiting. +When Shell has the focus, it controls the keyboard and screen. This is +normally transparent, but functions that directly access the keyboard +and screen will not work. These include system-specific functions that +determine whether a key has been pressed and if so, which. + +The IDLE code running in the execution process adds frames to the call stack +that would not be there otherwise. IDLE wraps ``sys.getrecursionlimit`` and +``sys.setrecursionlimit`` to reduce the effect of the additional stack +frames. + +When user code raises SystemExit either directly or by calling sys.exit, +IDLE returns to a Shell prompt instead of exiting. User output in Shell ^^^^^^^^^^^^^^^^^^^^ diff --git a/Lib/idlelib/help.html b/Lib/idlelib/help.html index 170999e1280173..1eefa506e2c8d9 100644 --- a/Lib/idlelib/help.html +++ b/Lib/idlelib/help.html @@ -5,7 +5,7 @@ - IDLE — Python 3.10.0a1 documentation + IDLE — Python 3.10.0a5 documentation @@ -18,7 +18,7 @@ @@ -32,7 +32,6 @@ -