Disallow control characters in hostnames in http.client, addressing CVE-2019-18348. Such potentially malicious header injection URLs now cause a InvalidURL to be raised.
Optimize pending calls in multithreaded applications. If a thread different than the main thread schedules a pending call (:cPy_AddPendingCall), the bytecode evaluation loop is no longer interrupted at each bytecode instruction to check for pending calls which cannot be executed. Only the main thread can execute pending calls.
Previously, the bytecode evaluation loop was interrupted at each instruction until the main thread executes pending calls.
Port _weakref extension module to multiphase initialization (489).
Port _collections module to multiphase initialization (489).
Optimize signal handling in multithreaded applications. If a thread different than the main thread gets a signal, the bytecode evaluation loop is no longer interrupted at each bytecode instruction to check for pending signals which cannot be handled. Only the main thread of the main interpreter can handle signals.
Previously, the bytecode evaluation loop was interrupted at each instruction until the main thread handles signals.
If :cPy_AddPendingCall is called in a subinterpreter, the function is now scheduled to be called from the subinterpreter, rather than being called from the main interpreter. Each subinterpreter now has its own list of scheduled calls.
Port _heapq module to multiphase initialization.
Port itertools module to multiphase initialization (489).
Speed up calls to frozenset() by using the 590 vectorcall calling convention. Patch by Dong-hee Na.
subinterpreters: Move _PyRuntimeState.ceval.tracing_possible to PyInterpreterState.ceval.tracing_possible: each interpreter now has its own variable.
Speed up calls to set() by using the 590 vectorcall calling convention. Patch by Dong-hee Na.
Port _statistics module to multiphase initialization (489).
Use inline function to replace extension modules' get_module_state macros.
Correctly raise SyntaxError if await is used inside non-async functions and PyCF_ALLOW_TOP_LEVEL_AWAIT is set (like in the asyncio REPL). Patch by Pablo Galindo.
Allow executing asynchronous comprehensions on the top level when the PyCF_ALLOW_TOP_LEVEL_AWAIT flag is given. Patch by Batuhan Taskaya.
Speed up calls to tuple() by using the 590 vectorcall calling convention. Patch by Dong-hee Na.
Chaged list overallocation strategy. It no longer overallocates if the new size is closer to overallocated size than to the old size and adds padding.
Update Unicode database to Unicode version 13.0.0.
Clear the frames of daemon threads earlier during the Python shutdown to call objects destructors. So "unclosed file" resource warnings are now emitted for daemon threads in a more reliable way.
Fix a bug that was causing incomplete results when calling pathlib.Path.glob in the presence of symlinks that point to files where the user does not have read access. Patch by Pablo Galindo and Matt Wozniski.
Fix :cPyEval_RestoreThread random crash at exit with daemon threads. It now accesses the _PyRuntime variable directly instead of using tstate->interp->runtime, since tstate can be a dangling pointer after :cPy_Finalize has been called. Moreover, the daemon thread now exits before trying to take the GIL.
Fix a possible SystemError in math.{atan2,copysign,remainder}() when the first argument cannot be converted to a float. Patch by Zachary Spytz.
Fix race condition where threads created by PyGILState_Ensure() could get a duplicate id.
This affects consumers of tstate->id like the contextvar caching machinery, which could return invalid cached objects under heavy thread load (observed in embedded scenarios).
Fixed a crash due to incorrect handling of weak references in collections.OrderedDict classes. Patch by Pablo Galindo.
Port audioop extension module to multiphase initialization (489).
Relax decorator grammar restrictions to allow any valid expression (614).
Tweak import deadlock detection code to not deadlock itself.
Port _locale extension module to multiphase initialization (489).
Optimize :cPyUnicode_AsUTF8 and :cPyUnicode_AsUTF8AndSize slightly when they need to create internal UTF-8 cache.
Fix unparsing of ext slices with no items (foo[:,]). Patch by Batuhan Taskaya.
Do not optimize annotations if 'from __future__ import annotations' is used. Patch by Pablo Galindo.
Using NotImplemented in a boolean context has been deprecated. Patch contributed by Josh Rosenberg.
Don't leak environment variable __PYVENV_LAUNCHER__ into the interpreter session on macOS.
Add zipfile.Path to __all__ in the zipfile module.
Improved error messages for validation of ast.Constant nodes. Patch by Batuhan Taskaya.
__module__ of the AST node classes is now set to "ast" instead of "_ast". Added docstrings for dummy AST node classes and deprecated attributes.
uuid.getnode now skips IPv6 addresses with the same string length than a MAC address (17 characters): only use MAC addresses.
Deprecated ast.AugLoad and ast.AugStore node classes because they are no longer used.
Ensure bin/python3.# is always present in virtual environments on POSIX platforms - by Anthony Sottile.
Deprecated ast.Param node class because it's no longer used. Patch by Batuhan Taskaya.
Ensure all workers exit when finalizing a multiprocessing.Pool implicitly via the module finalization handlers of multiprocessing. This fixes a deadlock situation that can be experienced when the Pool is not properly finalized via the context manager or a call to multiprocessing.Pool.terminate. Patch by Batuhan Taskaya and Pablo Galindo.
sys.settrace(), sys.setprofile() and _lsprof.Profiler.enable() now properly report :cPySys_Audit error if "sys.setprofile" or "sys.settrace" audit event is denied.
AIX: Fix _aix_support module when the subprocess is not available, when building Python from scratch. It now uses new private _bootsubprocess module, rather than having two implementations depending if subprocess is available or not. So _aix_support.aix_platform() result is now the same if subprocess is available or not.
collections.OrderedDict now implements | and |= (584).
The column name found in sqlite3.Cursor.description is now truncated on the first '[' only if the PARSE_COLNAMES option is set.
Ensure unittest.mock.AsyncMock.await_args_list has call objects in the order of awaited arguments instead of using unittest.mock.Mock.call_args which has the last value of the call. Patch by Karthikeyan Singaravelan.
Updated os.environ and os.environb to support 584's merge (|) and update (|=) operators.
The ensurepip module now invokes pip via the runpy module. Hence it is no longer tightly coupled with the internal API of the bundled pip version, allowing easier updates to a newer pip version both internally and for distributors.
Fix the random.Random.seed method when a bool is passed as the seed.
More reliable use of os.scandir() in Path.glob(). It no longer emits a ResourceWarning when interrupted.
multiprocessing now supports abstract socket addresses (if abstract sockets are supported in the running platform). When creating arbitrary addresses (like when default-constructing multiprocessing.connection.Listener objects) abstract sockets are preferred to avoid the case when the temporary-file-generated address is too large for an AF_UNIX socket address. Patch by Pablo Galindo.
ast.dump() no longer outputs optional fields and attributes with default values. The default values for optional fields and attributes of AST nodes are now set as class attributes (e.g. Constant.kind is set to None).
Fixed ast.unparse for extended slices containing a single element (e.g. a[i:j,]). Remove redundant tuples when index with a tuple (e.g. a[i, j]).
Fix json.tool to catch BrokenPipeError. Patch by Dong-hee Na.
Avoid a possible "RuntimeError: dictionary changed size during iteration" from inspect.getmodule when it tried to loop through sys.modules.
Revert "bpo-37330: open() no longer accept 'U' in file mode". The "U" mode of open() is kept in Python 3.9 to ease transition from Python 2.7, but will be removed in Python 3.10.
The hosts method on 32-bit prefix length IPv4Networks and 128-bit prefix IPv6Networks now returns a list containing the single Address instead of an empty list.
Add getConnection method to logging HTTPHandler to enable custom connections.
Reimplement distutils.spawn.spawn function with the subprocess module.
Add --without-decimal-contextvar build option. This enables a thread-local rather than a coroutine local context.
collections.defaultdict now implements | (584).
Fix runpy.run_path() when using pathlike objects
Change inspect.Signature.parameters back to collections.OrderedDict. This was changed to dict in Python 3.9.0a4.
Refactor queue_manager in concurrent.futures.ProcessPoolExecutor to make it easier to maintain.
Fix AttributeError when calling get_stack on a PyAsyncGenObject Task
The compileall.compile_dir function's ddir parameter and the compileall command line flag -d no longer write the wrong pathname to the generated pyc file for submodules beneath the root of the directory tree being compiled. This fixes a regression introduced with Python 3.5.
types.MappingProxyType objects now support the merge (|) operator from 584.
The importlib module now ignores the PYTHONCASEOK environment variable when the -E or -I command line options are being used.
Remove tempfile.SpooledTemporaryFile.softspace as files no longer have the softspace attribute in Python 3. Patch by Shantanu.
Improve pathlib.Path compatibility on zipfile.Path and correct performance degradation as found in zipp 3.0.
Keep ASDL signatures in the docstrings for AST nodes. Patch by Batuhan Taskaya
Deprecated ast.Suite node class because it's no longer used. Patch by Batuhan Taskaya.
Add thread_name_prefix to default asyncio executor
Fix handling of header in urllib.request.AbstractDigestAuthHandler when the optional qop parameter is not present.
HTTP status codes 103 EARLY_HINTS and 425 TOO_EARLY are added to http.HTTPStatus. Patch by Dong-hee Na.
Adding HTTP status 418 "I'm a Teapot" to HTTPStatus in http library. Patch by Ross Rhodes.
Remove default value from attrs parameter of xml.etree.ElementTree.TreeBuilder.start for consistency between Python and C implementations.
Open issue in the BPO indicated a desire to make the implementation of codecs.open() at parity with io.open(), which implements a try/except to assure file stream gets closed before an exception is raised.
Added starred expressions support to return and yield statements for lib2to3. Patch by Vlad Emelianov.
When using minidom module to generate XML documents the ability to add Standalone Document Declaration is added. All the changes are made to generate a document in compliance with Extensible Markup Language (XML) 1.0 (Fifth Edition) W3C Recommendation (available here: https://www.w3.org/TR/xml/#sec-prolog-dtd).
Add support for scoped IPv6 addresses to ipaddress. Patch by Oleksandr Pavliuk.
Simplified AST for subscription. Simple indices are now represented by their value, extended slices are represented as tuples. ast classes Index and ExtSlice are considered deprecated and will be removed in future Python versions. In the meantime, Index(value) now returns a value itself, ExtSlice(slices) returns Tuple(slices, Load()).
Updated the Language Reference for 572.
Change 'string' to 'specification' in format doc.
The language reference no longer restricts default class namespaces to dicts only.
Fix misleading documentation about mixed-type numeric comparisons.
Update token documentation to reflect additions in Python 3.8
Changed operand name of MAKE_FUNCTION from argc to flags for module dis
test_gdb now skips tests if it detects that gdb failed to read debug information because the Python binary is optimized.
test_site.test_startup_imports() is now skipped if a path of sys.path contains a .pth file.
Do not fail test_shutil test_chown test when uid or gid of user cannot be resolved to a name.
test_subprocess.test_user() now skips the test on an user name if the user name doesn't exist. For example, skip the test if the user "nobody" doesn't exist on Linux.
Fix build with DTrace but without additional DFLAGS.
setup.py now uses a basic implementation of the subprocess module if the subprocess module is not available: before required C extension modules are built.
Add --with-platlibdir option to the configure script: name of the platform-specific library directory, stored in the new sys.platlibdir attribute. It is used to build the path of platform-specific extension modules and the path of the standard library. It is equal to "lib" on most platforms. On Fedora and SuSE, it is equal to "lib64" on 64-bit platforms. Patch by Jan Matějek, Matěj Cepl, Charalampos Stratakis and Victor Stinner.
Ensures the required vcruntime140.dll is included in install packages.
Avoid hang when computer is hibernated whilst waiting for a mutex (for lock-related objects from threading) around 49-day uptime.
distutils will no longer statically link vcruntime140.dll when a redistributable version is unavailable. All future releases of CPython will include a copy of this DLL to ensure distributed extensions can continue to load.
Update Windows builds to use SQLite 3.31.1
Update Windows release build machines to Visual Studio 2019 (MSVC 14.2).
Package for nuget.org now includes repository reference and bundled icon image.
Update macOS builds to use SQLite 3.31.1
For 'Go to Line', use a Query box subclass with IDLE standard behavior and improved error checking.
Since clicking to get an IDLE context menu moves the cursor, any text selection should be and now is cleared.
Edit "Go to line" now clears any selection, preventing accidental deletion. It also updates Ln and Col on the status bar.
Selecting code context lines no longer causes a jump.
Port python-gdb.py to FreeBSD. python-gdb.py now checks for "take_gil" function name to check if a frame tries to acquire the GIL, instead of checking for "pthread_cond_timedwait" which is specific to Linux and can be a different condition than the GIL.
Added support to fix getproxies in the lib2to3.fixes.fix_urllib module. Patch by José Roberto Meza Cabrera.
Add :cPyModule_AddType helper function: add a type to a module. Patch by Dong-hee Na.
Remove _PyRuntime.getframe hook and remove _PyThreadState_GetFrame macro which was an alias to _PyRuntime.getframe. They were only exposed by the internal C API. Remove also PyThreadFrameGetter type.
Add :cPyThreadState_GetFrame function: get the current frame of a Python thread state.
Add _PyArg_NoKwnames helper function. Patch by Dong-hee Na.
Add :cPyThreadState_GetInterpreter: get the interpreter of a Python thread state.
Add :cPyInterpreterState_Get function to the limited C API.
If :cPySys_Audit fails in :cPyEval_SetProfile or :cPyEval_SetTrace, log the error as an unraisable exception.
Move the static inline function flavor of Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() to the internal C API: they access PyThreadState attributes. The limited C API provides regular functions which hide implementation details.
Py_TRASHCAN_BEGIN_CONDITION and Py_TRASHCAN_END macro no longer access PyThreadState attributes, but call new private _PyTrash_begin() and _PyTrash_end() functions which hide implementation details.
:cPyDescr_NewMethod and :cPyCFunction_NewEx now include the method name in the SystemError "bad call flags" error message to ease debug.
Deprecated :cPyEval_InitThreads and :cPyEval_ThreadsInitialized. Calling :cPyEval_InitThreads now does nothing.
:cPy_UNREACHABLE is now implemented with __builtin_unreachable() and analogs in release mode.
:cPyNumber_ToBase now raises a SystemError instead of crashing when called with invalid base.
The :cPy_FatalError function is replaced with a macro which logs automatically the name of the current function, unless the Py_LIMITED_API macro is defined.
Extension modules: :c~PyModuleDef.m_traverse, :c~PyModuleDef.m_clear and :c~PyModuleDef.m_free functions of :cPyModuleDef are no longer called if the module state was requested but is not allocated yet. This is the case immediately after the module is created and before the module is executed (:cPy_mod_exec function). More precisely, these functions are not called if :c~PyModuleDef.m_size is greater than 0 and the module state (as returned by :cPyModule_GetState) is NULL.
Extension modules without module state (m_size <= 0) are not affected.
Fixed segfault in Py_BuildValue() called with a format containing "#" and undefined PY_SSIZE_T_CLEAN whwn an exception is set.
Add a private API to get and set the frame evaluation function: add :c_PyInterpreterState_GetEvalFrameFunc and :c_PyInterpreterState_SetEvalFrameFunc C functions. The :c_PyFrameEvalFunction function type now takes a tstate parameter.