3.9.0a6.rst

Fixes audit events raised on creating a new socket.

Disallow CR or LF in email.headerregistry.Address arguments to guard against header injection attacks.

CVE-2020-8492: The :class:`~urllib.request.AbstractBasicAuthHandler` class of the :mod:`urllib.request` module uses an inefficient regular expression which can be exploited by an attacker to cause a denial of service. Fix the regex to prevent the catastrophic backtracking. Vulnerability reported by Ben Caller and Matt Schwager.

Improve the performance of bytes.hex().

Switch to a new parser, based on PEG. For more details see PEP 617. To temporarily switch back to the old parser, use -X oldparser or PYTHONOLDPARSER=1. In Python 3.10 we will remove the old parser completely, including the parser module (already deprecated) and anything that depends on it.

Fix the tokenizer to display the correct error message, when there is a SyntaxError on the last input character and no newline follows. It used to be unexpected EOF while parsing, while it should be invalid syntax.

Correctly unparse explicit u prefix for strings when postponed evaluation for annotations activated. Patch by Batuhan Taskaya.

Report a specialized error message, invalid string prefix, when the tokenizer encounters a string with an invalid prefix.

Fix the signal handler: it now always uses the main interpreter, rather than trying to get the current Python thread state.

str.encode() and str.decode() no longer check the encoding and errors in development mode or in debug mode during Python finalization. The codecs machinery can no longer work on very late calls to str.encode() and str.decode().

Fix possible refleaks in :mod:`_json`, memo of PyScannerObject should be traversed.

Speed up calls to dict() by using the PEP 590 vectorcall calling convention.

Add column and line information to ast.keyword nodes. Patch by Pablo Galindo.

Port :mod:`resource` to multiphase initialization (PEP 489).

Port :mod:`math` to multiphase initialization (PEP 489).

Port _uuid module to multiphase initialization (PEP 489).

Convert json module to use :c:func:`PyType_FromSpec`.

Improve the error message for multiple star expressions in an assignment. Patch by Furkan Onder

Port _functools module to multiphase initialization (PEP 489). Patch by Paulo Henrique Silva.

Port operator module to multiphase initialization (PEP 489). Patch by Paulo Henrique Silva.

Fix :c:func:`PyThreadState_Clear()`. PyThreadState.frame is a borrowed reference, not a strong reference: PyThreadState_Clear() must not call Py_CLEAR(tstate->frame).

Port time module to multiphase initialization (PEP 489). Patch by Paulo Henrique Silva.

Port _weakref extension module to multiphase initialization (PEP 489).

Fix a leak and subsequent crash in parsetok.c caused by realloc misuse on a rare codepath.

Added str.removeprefix and str.removesuffix methods and corresponding bytes, bytearray, and collections.UserString methods to remove affixes from a string if present. See PEP 616 for a full description. Patch by Dennis Sweeney.

Implement PEP 585. This supports list[int], tuple[str, ...] etc.

Support unparsing of infinity numbers in postponed annotations. Patch by Batuhan Taşkaya.

Speed up calls to list() by using the PEP 590 vectorcall calling convention. Patch by Mark Shannon.

:func:`typing.get_args` now always returns an empty tuple for special generic aliases.

Functions :func:`typing.get_origin`, :func:`typing.get_args` and :func:`typing.get_type_hints` support now generic aliases like list[int].

Optimize the :mod:`subprocess` module on FreeBSD using closefrom(). A single close(fd) syscall is cheap, but when sysconf(_SC_OPEN_MAX) is high, the loop calling close(fd) on each file descriptor can take several milliseconds.

The workaround on FreeBSD to improve performance was to load and mount the fdescfs kernel module, but this is not enabled by default.

Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans (kevans) and Kubilay Kocak (koobs): https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274

On FreeBSD, os.closerange(fd_low, fd_high) now calls closefrom(fd_low) if fd_high is greater than or equal to sysconf(_SC_OPEN_MAX).

Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans (kevans) and Kubilay Kocak (koobs): https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274

The :mod:`lib2to3` module is pending deprecation due to PEP 617.

Fix the Windows implementation of :func:`os.waitpid` for exit code larger than INT_MAX >> 8. The exit status is now interpreted as an unsigned number.

Set "__main__" as the default module name when "__name__" is missing in :class:`typing.TypeVar`. Patch by Weipeng Hong.

The :mod:`logging` package is now imported lazily in :mod:`unittest` only when the :meth:`~unittest.TestCase.assertLogs` assertion is used.

The :mod:`asyncio` package is now imported lazily in :mod:`unittest` only when the :class:`~unittest.IsolatedAsyncioTestCase` class is used.

In :meth:`ShareableList.__setitem__`, check the size of a new string item after encoding it to utf-8, not before.

Added :meth:`pathlib.Path.with_stem()` to create a new Path with the stem replaced.

Deprecated support for set objects in random.sample().

Improved help for the :mod:`typing` module. Docstrings are now shown for all special forms and special generic aliases (like Union and List). Using help() with generic alias like List[int] will show the help for the correspondent concrete type (list in this case).

func:inspect.getdoc no longer returns docstring inherited from the type of the object or from parent class if it is a class if it is not defined in the object itself. In :mod:`pydoc` the documentation string is now shown not only for class, function, method etc, but for any object that has its own __doc__ attribute.

Fixed SpooledTemporaryFile.seek() to return the position.

Added zscore() to statistics.NormalDist().

Allow random.getrandbits(0) to succeed and to return 0.

Add :func:`random.randbytes` function and :meth:`random.Random.randbytes` method to generate random bytes.

:func:`collections.namedtuple` now provides a human-readable repr for its field accessors.

The included copy of sqlite3 on Windows is now compiled with the json extension. This allows the use of functions such as json_object.

Wait in KqueueSelector.select when no fds are registered

Ensure :mod:`modulefinder` uses :func:`io.open_code` and respects coding comments.

Allow again to spawn daemon threads in subinterpreters (revert change which denied them).

Workers in :class:`~concurrent.futures.ProcessPoolExecutor` are now spawned on demand, only when there are no available idle workers to reuse. This optimizes startup overhead and reduces the amount of lost CPU time to idle workers. Patch by Kyle Stanley.

Fix a hang at fork in the logging module: the new private _at_fork_reinit() method is now used to reinitialize locks at fork in the child process.

Implement traverse and clear slots in _abc._abc_data type.

Remove deprecated :meth:`symtable.SymbolTable.has_exec`.

Fix a bug in the :mod:`symtable` module that was causing incorrectly report global variables as local. Patch by Pablo Galindo.

Add support for _SC_AIX_REALMEM to :func:`posix.sysconf`.

Removed the _field_types attribute of the :class:`typing.NamedTuple` class.

Multiple inheritance with :class:`typing.NamedTuple` now raises an error instead of silently ignoring other types.

Fixed reverting multiple patches in unittest.mock. Patcher's __exit__() is now never called if its __enter__() is failed. Returning true from __exit__() silences now the exception.

CGIHTTPRequestHandler of http.server now logs the CGI script exit code, rather than the CGI script exit status of os.waitpid(). For example, if the script is killed by signal 11, it now logs: "CGI script exit code -11."

Improve the error message when triying to import a module using :mod:`runpy` and incorrently use the ".py" extension at the end of the module name. Patch by Pablo Galindo.

Add :func:`os.waitstatus_to_exitcode` function: convert a wait status to an exit code.

Fix threading._after_fork(): if fork was not called by a thread spawned by threading.Thread, threading._after_fork() now creates a _MainThread instance for _main_thread, instead of a _DummyThread instance.

Add a private _at_fork_reinit() method to :class:`_thread.Lock`, :class:`_thread.RLock`, :class:`threading.RLock` and :class:`threading.Condition` classes: reinitialize the lock at fork in the child process, reset the lock to the unlocked state. Rename also the private _reset_internal_locks() method of :class:`threading.Event` to _at_fork_reinit().

Expose :data:`~socket.CAN_RAW_JOIN_FILTERS` in the :mod:`socket` module.

:class:`~urllib.request.AbstractBasicAuthHandler` of :mod:`urllib.request` now parses all WWW-Authenticate HTTP headers and accepts multiple challenges per header: use the realm of the first Basic challenge.

Removed daemon threads from :mod:`concurrent.futures` by adding an internal threading._register_atexit(), which calls registered functions prior to joining all non-daemon threads. This allows for compatibility with subinterpreters, which don't support daemon threads.

Fix importlib._bootstrap_external: avoid creating a new winreg builtin module if it's already available in :data:`sys.modules`, and remove redundant imports.

Fix os.getgrouplist(): if getgrouplist() function fails because the group list is too small, retry with a larger group list. On failure, the glibc implementation of getgrouplist() sets ngroups to the total number of groups. For other implementations, double the group list size.

Add :data:`time.CLOCK_TAI` constant if the operating system support it.

In re docstring, clarify the relationship between inline and argument compile flags.

Update internal table of OpenSSL error codes in the ssl module.

Added PEP 584 operators to :class:`weakref.WeakValueDictionary`.

Added PEP 584 operators to :class:`weakref.WeakKeyDictionary`.

Fix linear runtime behaviour of the __getitem__ and __setitem__ methods in :class:`multiprocessing.shared_memory.ShareableList`. This avoids quadratic performance when iterating a ShareableList. Patch by Thomas Krennwallner.

Remove undocumented support for closing a pathlib.Path object via its context manager. The context manager magic methods remain, but they are now a no-op, making Path objects immutable.

Added PEP 584 operators (| and |=) to :class:`collections.ChainMap`.

Normalization of line endings in ElementTree attributes was removed, as line endings which were replaced by entity numbers should be preserved in original form.

Properly handle :func:`sys.audit` failures in :func:`sys.set_asyncgen_hooks`.

lib2to3 now recognizes named assignment expressions (the walrus operator, :=)

In platform, delay the invocation of 'uname -p' until the processor attribute is requested.

:meth:`inspect.getsource` now returns correct source code for inner class with same name as module level class. Decorators are also returned as part of source of the class. Patch by Karthikeyan Singaravelan.

Deprecate passing None as an argument for :func:`shlex.split()`'s s parameter. Patch by Zackery Spytz.

Prevent crashes when using an uninitialized _elementtree.XMLParser object. Patch by Oren Milman.

The pickle documentation incorrectly claimed that __new__ isn't called by default when unpickling.

Updated :ref:`datamodel` docs to include :func:`dict` insertion order preservation. Patch by Furkan Onder and Samy Lahfa.

Document :c:macro:`PyDoc_STRVAR` macro in the C-API reference.

Some methods within xml.dom.minidom.Element class are now better documented.

Set expected default encoding in test_c_locale_coercion.py for VxWorks RTOS.

Update Travis CI configuration to OpenSSL 1.1.1f.

Update OpenSSL to 1.1.1f in Azure Pipelines.

Add :func:`test.support.wait_process` function.

test.bisect_cmd now copies Python command line options like -O or -W. Moreover, emit a warning if test.bisect_cmd is used with -w/--verbose2 option.

Add the encoding in :class:`ftplib.FTP` and :class:`ftplib.FTP_TLS` to the constructor as keyword-only and change the default from latin-1 to utf-8 to follow RFC 2640.

Use the same domain when testing make_msgid. Patch by Batuhan Taskaya.

Fix newline handling in doctest.testfile when loading from a package whose loader has a get_data method. Patch by Peter Donis.

Support single-argument form of macOS -isysroot flag.

Fix CPython MSBuild Properties in NuGet Package (build/native/python.props)

Fix configure check on Solaris for "float word ordering": sometimes, the correct "grep" command was not being used. Patch by Arnon Yaari.

Updates Windows to OpenSSL 1.1.1f

Ignore the Windows registry when the -E option is used.

python.org macOS installers now update the Current version symlink of /Library/Frameworks/Python.framework/Versions for 3.9 installs. Previously, Current was only updated for Python 2.x installs. This should make it easier to embed Python 3 into other macOS applications.

Update macOS installer builds to use OpenSSL 1.1.1g.

Add a 256×256 pixel IDLE icon to support more modern environments. Created by Andrew Clover. Delete the unused macOS idle.icns icon file.

IDLE will no longer freeze when inspect.signature fails when fetching a calltip.

Removed the checkpyc.py tool. Please see compileall without force mode as a potential alternative.

Fixed translation of #elif in Argument Clinic.

Fix which.py script exit code: it now uses :func:`os.waitstatus_to_exitcode` to convert :func:`os.system` exit status into an exit code.

Move the :c:type:`PyGC_Head` structure to the internal C API.

Convert :c:func:`PyObject_IS_GC` macro to a function to hide implementation details.

Add the functions :c:func:`PyObject_GC_IsTracked` and :c:func:`PyObject_GC_IsFinalized` to the public API to allow to query if Python objects are being currently tracked or have been already finalized by the garbage collector respectively. Patch by Pablo Galindo.

The :c:func:`PyObject_NEW` macro becomes an alias to the :c:func:`PyObject_New` macro, and the :c:func:`PyObject_NEW_VAR` macro becomes an alias to the :c:func:`PyObject_NewVar` macro, to hide implementation details. They no longer access directly the :c:member:`PyTypeObject.tp_basicsize` member.

:c:func:`PyType_HasFeature` now always calls :c:func:`PyType_GetFlags` to hide implementation details. Previously, it accessed directly the :c:member:`PyTypeObject.tp_flags` member when the limited C API was not used.

Convert the :c:func:`PyObject_GET_WEAKREFS_LISTPTR` macro to a function to hide implementation details: the macro accessed directly to the :c:member:`PyTypeObject.tp_weaklistoffset` member.

Convert :c:func:`PyObject_CheckBuffer` macro to a function to hide implementation details: the macro accessed directly the :c:member:`PyTypeObject.tp_as_buffer` member.

Always declare :c:func:`PyIndex_Check` as an opaque function to hide implementation details: remove PyIndex_Check() macro. The macro accessed directly the :c:member:`PyTypeObject.tp_as_number` member.

Add :c:func:`PyThreadState_GetID` function: get the unique identifier of a Python thread state.