Make the garbage collector thread-safe in `--disable-gil` builds #112529

colesbury · 2023-11-29T16:51:36Z

Feature or enhancement

Python's cyclic garbage collector relies on the global interpreter lock for thread-safety. There are a number of changes needed to make the GC thread-safe in --disable-gil builds.

My intention is to implement these as a series of smaller changes.

Garbage collection is guarded by gcstate->collecting. This need to be made thread-safe.
The --disable-gil builds should stop-the-world when finding garbage, but not when calling finalizers or destructors. (depends on Implement stop-the-world functionality (for --disable-gil builds) #111964)
The --disable-gil builds should find GC-enabled objects by traversing the mimalloc heaps, because the _gc_next/_gc_prev lists are not thread-safe. (Note it's safe to use the lists during stop-the-world pauses; we just can't maintain them safely during normal Python execution).
The --disable-gil builds should probably use only a single generation to avoid frequent stop-the-world pauses
Eventually, we can get rid of the GC pre-header in --disable-gil builds to save memory. (This will require updating the trashcan mechanism.)

Remaining work:

Fix scheduling of free-threaded GC (currently scheduled as if there is a young generation)
Update Python docs
Refactor out common code from gc.c and gc_free_threaded.c

Linked PRs

The text was updated successfully, but these errors were encountered:

The `collecting` field in `GCState` is used to prevent overlapping garbage collections within the same interpreter. This is updated to use atomic operations in order to be thread-safe in `--disable-gil` builds. The GC code is refactored a bit to support this. More of the logic is pushed down to `gc_collect_main()` so that we can safely order the logic setting `collecting`, the selection of the generation, and the invocation of callbacks with respect to the atomic operations and the (future) stop-the-world pauses. The change uses atomic operations for both `--disable-gil` and the default build (with the GIL) to avoid extra `#ifdef` guards and ease the maintenance burden.

* gh-112529: Use atomic operations for `gcstate->collecting` The `collecting` field in `GCState` is used to prevent overlapping garbage collections within the same interpreter. This is updated to use atomic operations in order to be thread-safe in `--disable-gil` builds. The GC code is refactored a bit to support this. More of the logic is pushed down to `gc_collect_main()` so that we can safely order the logic setting `collecting`, the selection of the generation, and the invocation of callbacks with respect to the atomic operations and the (future) stop-the-world pauses. The change uses atomic operations for both `--disable-gil` and the default build (with the GIL) to avoid extra `#ifdef` guards and ease the maintenance burden.

…ator The GC implementation for free-threaded builds will need to accurately detect if the debug allocator is used because it affects the offset of the Python object from the beginning of the memory allocation. The current implementation of `_PyMem_DebugEnabled` only considers if the debug allocator is the outer-most allocator; it doesn't handle the case of "hooks" like tracemalloc being used on top of the debug allocator. This change enables more accurate detection of the debug allocator by tracking when debug hooks are enabled.

…113747) * gh-112529: Track if debug allocator is used as underlying allocator The GC implementation for free-threaded builds will need to accurately detect if the debug allocator is used because it affects the offset of the Python object from the beginning of the memory allocation. The current implementation of `_PyMem_DebugEnabled` only considers if the debug allocator is the outer-most allocator; it doesn't handle the case of "hooks" like tracemalloc being used on top of the debug allocator. This change enables more accurate detection of the debug allocator by tracking when debug hooks are enabled. * Simplify _PyMem_DebugEnabled

The free-threaded build's garbage collector implementation will need to find GC objects by traversing mimalloc heaps. This hooks up the allocation calls with the correct heaps by using a thread-local "current_obj_heap" variable.

…ator The GC implementation for free-threaded builds will need to accurately detect if the debug allocator is used because it affects the offset of the Python object from the beginning of the memory allocation. The current implementation of `_PyMem_DebugEnabled` only considers if the debug allocator is the outer-most allocator; it doesn't handle the case of "hooks" like tracemalloc being used on top of the debug allocator. This change enables more accurate detection of the debug allocator by tracking when debug hooks are enabled.

The free-threaded build's garbage collector implementation will need to find GC objects by traversing mimalloc heaps. This hooks up the allocation calls with the correct heaps by using a thread-local "current_obj_heap" variable.

…h-114157) * gh-112529: Use GC heaps for GC allocations in free-threaded builds The free-threaded build's garbage collector implementation will need to find GC objects by traversing mimalloc heaps. This hooks up the allocation calls with the correct heaps by using a thread-local "current_obj_heap" variable. * Refactor out setting heap based on type

* gh-111926: Set up basic sementics of weakref API for freethreading (gh-113621) --------- Co-authored-by: Sam Gross <colesbury@gmail.com> * gh-113603: Compiler no longer tries to maintain the no-empty-block invariant (#113636) * gh-113258: Write frozen modules to the build tree on Windows (GH-113303) This ensures the source directory is not modified at build time, and different builds (e.g. different versions or GIL vs no-GIL) do not have conflicts. * Document the `co_lines` method on code objects (#113682) Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com> * gh-52161: Enhance Cmd support for docstrings (#110987) In `cmd.Cmd.do_help` call `inspect.cleandoc()`, to clean indentation and remove leading/trailing empty lines from a dosctring before printing. * GH-113689: Fix broken handling of invalid executors (GH-113694) * gh-113696: Docs: Annotate PyObject_CallOneArg and PyObject_CallNoArgs as returning a strong reference (#113697) * gh-113569: Display calls in Mock.assert_has_calls failure when empty (GH-113573) * gh-113538: Don't error in stream reader protocol callback when task is cancelled (#113690) * GH-113225: Speed up `pathlib.Path.glob()` (#113226) Use `os.DirEntry.path` as the string representation of child paths, unless the parent path is empty, in which case we use the entry `name`. * gh-112532: Isolate abandoned segments by interpreter (#113717) * gh-112532: Isolate abandoned segments by interpreter Mimalloc segments are data structures that contain memory allocations along with metadata. Each segment is "owned" by a thread. When a thread exits, it abandons its segments to a global pool to be later reclaimed by other threads. This changes the pool to be per-interpreter instead of process-wide. This will be important for when we use mimalloc to find GC objects in the `--disable-gil` builds. We want heaps to only store Python objects from a single interpreter. Absent this change, the abandoning and reclaiming process could break this isolation. * Add missing '&_mi_abandoned_default' to 'tld_empty' * gh-113320: Reduce the number of dangerous `getattr()` calls when constructing protocol classes (#113401) - Only attempt to figure out whether protocol members are "method members" or not if the class is marked as a runtime protocol. This information is irrelevant for non-runtime protocols; we can safely skip the risky introspection for them. - Only do the risky getattr() calls in one place (the runtime_checkable class decorator), rather than in three places (_ProtocolMeta.__init__, _ProtocolMeta.__instancecheck__ and _ProtocolMeta.__subclasscheck__). This reduces the number of locations in typing.py where the risky introspection could go wrong. - For runtime protocols, if determining whether a protocol member is callable or not fails, give a better error message. I think it's reasonable for us to reject runtime protocols that have members which raise strange exceptions when you try to access them. PEP-544 clearly states that all protocol member must be callable for issubclass() calls against the protocol to be valid -- and if a member raises when we try to access it, there's no way for us to figure out whether it's a callable member or not! * GH-113486: Do not emit spurious PY_UNWIND events for optimized calls to classes. (GH-113680) * gh-113703: Correctly identify incomplete f-strings in the codeop module (#113709) * gh-101100: Fix Sphinx warnings for 2.6 deprecations and removals (#113725) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com> * gh-80532: Do not set ipv6type when cross-compiling (#17956) Co-authored-by: Xavier de Gaye <xdegaye@gmail.com> * gh-101100: Fix Sphinx warnings in `library/pyclbr.rst` (#113739) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> * gh-112532: Tag mimalloc heaps and pages (#113742) * gh-112532: Tag mimalloc heaps and pages Mimalloc pages are data structures that contain contiguous allocations of the same block size. Note that they are distinct from operating system pages. Mimalloc pages are contained in segments. When a thread exits, it abandons any segments and contained pages that have live allocations. These segments and pages may be later reclaimed by another thread. To support GC and certain thread-safety guarantees in free-threaded builds, we want pages to only be reclaimed by the corresponding heap in the claimant thread. For example, we want pages containing GC objects to only be claimed by GC heaps. This allows heaps and pages to be tagged with an integer tag that is used to ensure that abandoned pages are only claimed by heaps with the same tag. Heaps can be initialized with a tag (0-15); any page allocated by that heap copies the corresponding tag. * Fix conversion warning * gh-113688: Split up gcmodule.c (gh-113715) This splits part of Modules/gcmodule.c of into Python/gc.c, which now contains the core garbage collection implementation. The Python module remain in the Modules/gcmodule.c file. * GH-113568: Stop raising auditing events from pathlib ABCs (#113571) Raise auditing events in `pathlib.Path.glob()`, `rglob()` and `walk()`, but not in `pathlib._abc.PathBase` methods. Also move generation of a deprecation warning into `pathlib.Path` so it gets the right stack level. * gh-85567: Fix resouce warnings in pickle and pickletools CLIs (GH-113618) Explicitly open and close files instead of using FileType. * gh-113360: Fix the documentation of module's attribute __test__ (GH-113393) It can only be a dict since Python 2.4. * GH-113568: Stop raising deprecation warnings from pathlib ABCs (#113757) * gh-113750: Fix object resurrection in free-threaded builds (gh-113751) gh-113750: Fix object resurrection on free-threaded builds This avoids the undesired re-initializing of fields like `ob_gc_bits`, `ob_mutex`, and `ob_tid` when an object is resurrected due to its finalizer being called. This change has no effect on the default (with GIL) build. * gh-113729: Fix IDLE's Help -> "IDLE Help" menu bug in 3.12.1 and 3.11.7 (#113731) Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> * gh-113537: support loads str in plistlib.loads (#113582) Add support for loading XML plists from a string value instead of a only bytes value. * gh-111488: Changed error message in case of no 'in' keyword after 'for' in cmp (#113656) * gh-107901: synthetic jumps which are not at end of loop no longer check the eval breaker (#113721) * GH-113528: pathlib ABC tests: add repr to dummy path classes. (#113777) The `DummyPurePath` and `DummyPath` test classes are simple subclasses of `PurePathBase` and `PathBase`. This commit adds `__repr__()` methods to the dummy classes, which makes debugging test failures less painful. * GH-113528: Split up pathlib tests for invalid basenames. (#113776) Split test cases for invalid names into dedicated test methods. This will make it easier to refactor tests for invalid name handling in ABCs later. No change of coverage, just a change of test suite organisation. * GH-113528: Slightly improve `pathlib.Path.glob()` tests for symlink loop handling (#113763) Slightly improve `pathlib.Path.glob()` tests for symlink loop handling When filtering results, ignore paths with more than one `linkD/` segment, rather than all paths below the first `linkD/` segment. This allows us to test that other paths under `linkD/` are correctly returned. * GH-113528: Deoptimise `pathlib._abc.PurePathBase.name` (#113531) Replace usage of `_from_parsed_parts()` with `with_segments()` in `with_name()`, and take a similar approach in `name` for consistency's sake. * GH-113528: Deoptimise `pathlib._abc.PurePathBase.parent` (#113530) Replace use of `_from_parsed_parts()` with `with_segments()`, and move assignments to `_drv`, `_root`, _tail_cached` and `_str` slots into `PurePath`. * GH-113528: Deoptimise `pathlib._abc.PurePathBase.relative_to()` (#113529) Replace use of `_from_parsed_parts()` with `with_segments()` in `PurePathBase.relative_to()`, and move the assignment of `_drv`, `_root` and `_tail_cached` slots into `PurePath.relative_to()`. * gh-89532: Remove LibreSSL workarounds (#28728) Remove LibreSSL specific workaround ifdefs from `_ssl.c` and delete the non-version-specific `_ssl_data.h` file (relevant for OpenSSL < 1.1.1, which we no longer support per PEP 644). Co-authored-by: Christian Heimes <christian@python.org> Co-authored-by: Gregory P. Smith <greg@krypto.org> * gh-112795: Allow `/` folder in a zipfile (#112932) Allow extraction (no-op) of a "/" folder in a zipfile, they are commonly added by some archive creation tools. Co-authored-by: Erlend E. Aasland <erlend@python.org> Co-authored-by: Gregory P. Smith <greg@krypto.org> * gh-73965: New environment variable PYTHON_HISTORY (#13208) It can be used to set the location of a .python_history file --------- Co-authored-by: Levi Sabah <0xl3vi@gmail.com> Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com> * gh-73965: Move PYTHON_HISTORY into the correct usage section (#113798) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> * gh-80109: Fix io.TextIOWrapper dropping the internal buffer during write() (GH-22535) io.TextIOWrapper was dropping the internal decoding buffer during read() and write() calls. * gh-74678: Increase base64 test coverage (GH-21913) Ensure the character y is disallowed within an Ascii85 5-tuple. Co-authored-by: Lee Cannon <leecannon@leecannon.xyz> * gh-110721: Remove unused code from suggestions.c after moving PyErr_Display to use the traceback module (#113712) * gh-113391: fix outdated PyObject_HasAttr docs (#113420) After #53875: PyObject_HasAttr is not an equivalent of hasattr. PyObject_HasAttrWithError is; it already has the note. * gh-113787: Fix refleaks in test_capi (gh-113816) Fix refleaks and a typo. * gh-113755: Fully adapt gcmodule.c to Argument Clinic (#113756) Adapt the following functions to Argument Clinic: - gc.set_threshold - gc.get_referrers - gc.get_referents * Minor algebraic simplification for the totient() recipe (gh-113822) * GH-113528: Move a few misplaced pathlib tests (#113527) `PurePathBase` does not define `__eq__()`, and so we have no business checking path equality in `test_eq_common` and `test_equivalences`. The tests only pass at the moment because we define the test class's `__eq__()` for use elsewhere. Also move `test_parse_path_common` into the main pathlib test suite. It exercises a private `_parse_path()` method that will be moved to `PurePath` soon. Lastly move a couple more tests concerned with optimisations and path normalisation. * gh-113688: fix dtrace build on Solaris (#113814) (the gcmodule -> gc refactoring broke it) * GH-113528: Speed up pathlib ABC tests. (#113788) - Add `__slots__` to dummy path classes. - Return namedtuple rather than `os.stat_result` from `DummyPath.stat()`. - Reduce maximum symlink count in `DummyPathWithSymlinks.resolve()`. * gh-113791: Expose CLOCK_MONOTONIC_RAW_APPROX and CLOCK_UPTIME_RAW_APROX on macOS in the time module (#113792) * GH-111693: Propagate correct asyncio.CancelledError instance out of asyncio.Condition.wait() (#111694) Also fix a race condition in `asyncio.Semaphore.acquire()` when cancelled. * gh-113827: Move Windows frozen modules directory to allow PGO builds (GH-113828) * gh-113027: Fix test_variable_tzname in test_email (#113821) Determine the support of the Kyiv timezone by checking the result of astimezone() which uses the system tz database and not the one populated by zoneinfo. * readme: fix displaying issue of command (#113719) Avoid line break in command as this causes displaying issues on GH. * gh-112806: Remove unused function warnings during mimalloc build on Solaris (#112807) * gh-112808: Fix mimalloc build on Solaris (#112809) * gh-112087: Update list.{pop,clear,reverse,remove} to use CS (gh-113764) * Docs: Link tokens in the format string grammars (#108184) Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com> Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com> * gh-113692: skip a test if multiprocessing isn't available. (GH-113704) * gh-101100: Fix Sphinx warnings for 2.6 port-specific deprecations (#113752) * gh-113842: Add missing error check for PyIter_Next() in Python/symtable.c (GH-113843) * gh-87868: Sort and remove duplicates in getenvironment() (GH-102731) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com> Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com> * gh-103092: Test _ctypes type hierarchy and features (#113727) Test the following features for _ctypes types: - disallow instantiation - inheritance (MRO) - immutability - type name The following _ctypes types are tested: - Array - CField - COMError - PyCArrayType - PyCFuncPtrType - PyCPointerType - PyCSimpleType - PyCStructType - Structure - Union - UnionType - _CFuncPtr - _Pointer - _SimpleCData Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com> * gh-113650: Add workaround option for MSVC ARM64 bug affecting string encoding (GH-113836) * Fix opcode name printing in debug mode (#113870) Fix a few places where the lltrace debug output printed ``(null)`` instead of an opcode name, because it was calling ``_PyUOpName()`` on a Tier-1 opcode. * Simplify binomial approximation example with random.binomialvariate() (gh-113871) * GH-113528: Deoptimise `pathlib._abc.PathBase._make_child_relpath()` (#113532) Call straight through to `joinpath()` in `PathBase._make_child_relpath()`. Move optimised/caching code to `pathlib.Path._make_child_relpath()` * gh-113848: Use PyErr_GivenExceptionMatches() for check for CancelledError (GH-113849) * gh-113848: Handle CancelledError subclasses in asyncio TaskGroup() and timeout() (GH-113850) * gh-113781: Silence AttributeError in warning module during Python finalization (GH-113813) The tracemalloc module can already be cleared. * GH-113661: unittest runner: Don't exit 5 if tests were skipped (#113856) The intention of exiting 5 was to detect issues where the test suite wasn't discovered at all. If we skipped tests, it was correctly discovered. * GH-113528: Deoptimise `pathlib._abc.PathBase.resolve()` (#113782) Replace use of `_from_parsed_parts()` with `with_segments()` in `resolve()`. No effect on `Path.resolve()`, which uses `os.path.realpath()`. * gh-66060: Use actual class name in _io type's __repr__ (#30824) Use the object's actual class name in the following _io type's __repr__: - FileIO - TextIOWrapper - _WindowsConsoleIO * GH-113528: Deoptimise `pathlib._abc.PurePathBase.parts` (#113883) Implement `parts` using `_stack`, which itself calls `pathmod.split()` repeatedly. This avoids use of `_tail`, which will be moved to `PurePath` shortly. * GH-113528: Deoptimise `pathlib._abc.PurePathBase.relative_to()` (again) (#113882) Restore full battle-tested implementations of `PurePath.[is_]relative_to()`. These were recently split up in 3375dfe and a15a773. In `PurePathBase`, add entirely new implementations based on `_stack`, which itself calls `pathmod.split()` repeatedly to disassemble a path. These new implementations preserve features like trailing slashes where possible, while still observing that a `..` segment cannot be added to traverse an empty or `.` segment in *walk_up* mode. They do not rely on `parents` nor `__eq__()`, nor do they spin up temporary path objects. Unfortunately calling `pathmod.relpath()` isn't an option, as it calls `abspath()` and in turn `os.getcwd()`, which is impure. * gh-111968: Introduce _PyFreeListState and _PyFreeListState_GET API (gh-113584) * GH-113528: Deoptimise `pathlib._abc.PurePathBase` (#113559) Apply pathlib's normalization and performance tuning in `pathlib.PurePath`, but not `pathlib._abc.PurePathBase`. With this change, the pathlib ABCs do not normalize away alternate path separators, empty segments, or dot segments. A single string given to the initialiser will round-trip by default, i.e. `str(PurePathBase(my_string)) == my_string`. Implementors can set their own path domain-specific normalization scheme by overriding `__str__()` Eliminating path normalization makes maintaining and caching the path's parts and string representation both optional and not very useful, so this commit moves the `_drv`, `_root`, `_tail_cached` and `_str` slots from `PurePathBase` to `PurePath`. Only `_raw_paths` and `_resolving` slots remain in `PurePathBase`. This frees the ABCs from the burden of some of pathlib's hardest-to-understand code. * pathlib ABCs: Require one or more initialiser arguments (#113885) Refuse to guess what a user means when they initialise a pathlib ABC without any positional arguments. In mainline pathlib it's normalised to `.`, but in the ABCs this guess isn't appropriate; for example, the path type may not represent the current directory as `.`, or may have no concept of a "current directory" at all. * gh-112182: Replace StopIteration with RuntimeError for future (#113220) When an `StopIteration` raises into `asyncio.Future`, this will cause a thread to hang. This commit address this by not raising an exception and silently transforming the `StopIteration` with a `RuntimeError`, which the caller can reconstruct from `fut.exception().__cause__` * GH-113858: GitHub Actions config: Only save ccache on pushes (GH-113859) * gh-113877: Fix Tkinter method winfo_pathname() on 64-bit Windows (GH-113900) winfo_id() converts the result of "winfo id" command to integer, but "winfo pathname" command requires an argument to be a hexadecimal number on Win64. * gh-113879: Fix ResourceWarning in test_asyncio.test_server (GH-113881) * gh-96037: Always insert TimeoutError when exit an expired asyncio.timeout() block (GH-113819) If other exception was raised during exiting an expired asyncio.timeout() block, insert TimeoutError in the exception context just above the CancelledError. * gh-70835: Clarify error message for CSV file opened with wrong newline (GH-113786) Based on patch by SilentGhost. * gh-113594: Fix UnicodeEncodeError in TokenList.fold() (GH-113730) It occurred when try to re-encode an unknown-8bit part combined with non-unknown-8bit part. * gh-113664: Improve style of Big O notation (GH-113695) Use cursive to make it looking like mathematic formulas. * gh-58032: Do not use argparse.FileType in module CLIs and scripts (GH-113649) Open and close files manually. It prevents from leaking files, preliminary creation of output files, and accidental closing of stdin and stdout. * gh-89850: Add default C implementations of persistent_id() and persistent_load() (GH-113579) Previously the C implementation of pickle.Pickler and pickle.Unpickler classes did not have such methods and they could only be used if they were overloaded in subclasses or set as instance attributes. Fixed calling super().persistent_id() and super().persistent_load() in subclasses of the C implementation of pickle.Pickler and pickle.Unpickler classes. It no longer causes an infinite recursion. * gh-66515: Fix locking of an MH mailbox without ".mh_sequences" file (GH-113482) Guarantee that it either open an existing ".mh_sequences" file or create a new ".mh_sequences" file, but do not replace existing ".mh_sequences" file. * gh-111789: Use PyDict_GetItemRef() in Modules/_zoneinfo.c (GH-112078) * gh-109858: Protect zipfile from "quoted-overlap" zipbomb (GH-110016) Raise BadZipFile when try to read an entry that overlaps with other entry or central directory. * gh-111139: Optimize math.gcd(int, int) (#113887) Add a fast-path for the common case. Benchmark: python -m pyperf timeit \ -s 'import math; gcd=math.gcd; x=2*3; y=3*5' \ 'gcd(x,y)' Result: 1.07x faster (-3.4 ns) Mean +- std dev: 52.6 ns +- 4.0 ns -> 49.2 ns +- 0.4 ns: 1.07x faster * GH-113860: All executors are now defined in terms of micro ops. Convert counter executor to use uops. (GH-113864) * gh-111968: Use per-thread freelists for float in free-threading (gh-113886) * Add @requires_zlib() decorator for gh-109858 tests (GH-113918) * gh-113625: Align object addresses in the Descriptor HowTo Guide (#113894) * gh-113753: Clear finalized bit when putting PyAsyncGenASend back into free list (#113754) * gh-112302: Point core developers to SBOM devguide on errors (#113490) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> * gh-77046: os.pipe() sets _O_NOINHERIT flag on fds (#113817) On Windows, set _O_NOINHERIT flag on file descriptors created by os.pipe() and io.WindowsConsoleIO. Add test_pipe_spawnl() to test_os. Co-authored-by: Zackery Spytz <zspytz@gmail.com> * gh-87868: Skip `test_one_environment_variable` in `test_subprocess` when the platform or build cannot do that (#113867) * improve the assert for test_one_environment_variable * skip some test in test_subprocess when python is configured with shared * also skip the test if AddressSanitizer is enabled --------- Co-authored-by: Steve Dower <steve.dower@microsoft.com> * gh-113896: Fix test_builtin.BuiltinTest.test___ne__() (#113897) Fix DeprecationWarning in test___ne__(). Co-authored-by: Nikita Sobolev <mail@sobolevn.me> * gh-111968: Unify naming scheme for freelist (gh-113919) * gh-89811: Check for valid tp_version_tag in specializer (GH-113558) * gh-112640: Add `kwdefaults` parameter to `types.FunctionType.__new__` (#112641) * gh-112419: Document removal of sys.meta_path's 'find_module' fallback (#112421) Co-authored-by: Erlend E. Aasland <erlend@python.org> * gh-113932: assert ``SyntaxWarning`` in test_compile.TestSpecifics.test_… (#113933) * gh-91960: Remove Cirrus CI configuration (#113938) Remove .cirrus.yml which was already disabled by being renamed to .cirrus-DISABLED.yml. In total, Cirrus CI only run for less than one month. * gh-107901: jump leaving an exception handler doesn't need an eval break check (#113943) * GH-113853: Guarantee forward progress in executors (GH-113854) * gh-113845: Fix a compiler warning in Python/suggestions.c (GH-113949) * gh-111968: Use per-thread freelists for tuple in free-threading (gh-113921) * Update KDE recipe to match the standard use of the h parameter (gh-#113958) * gh-81489: Use Unicode APIs for mmap tagname on Windows (GH-14133) Co-authored-by: Erlend E. Aasland <erlend@python.org> * GH-107678: Improve Unicode handling clarity in ``library/re.rst`` (#107679) * Improve kde graph with better caption and number formatting (gh-113967) * gh-111968: Explicit handling for finalized freelist (gh-113929) * gh-113903: Fix an IDLE configdialog test (#113973) test_configdialog.HighPageTest.test_highlight_target_text_mouse fails if a line of the Highlight tab text sample is not visible. If so, bbox() in click_char() returns None and the unpacking iteration fails. This occurred on a Devuan Linux system. Fix by moving the 'see character' call inside click_char, just before the bbox call. Also, reduce the click_char calls to just one per tag name and replace the other nested function with a dict comprehension. * gh-113937 Fix failures in type cache tests due to re-running (GH-113953) * gh-113858: Cut down ccache size (GH-113945) Cut down ccache size - Only save the ccache in the main reusable builds, not on builds that don't use special build options: - Generated files check - OpenSSL tests - Hypothesis tests - Halve the max cache size, to 200M * gh-108364: In sqlite3, disable foreign keys before dumping SQL schema (#113957) sqlite3.Connection.iterdump now ensures that foreign key support is disabled before dumping the database schema, if there is any foreign key violation. Co-authored-by: Erlend E. Aasland <erlend@python.org> * gh-113027: Fix timezone check in test_variable_tzname in test_email (GH-113835) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> * GH-113860: Get rid of `_PyUOpExecutorObject` (GH-113954) * Docs: Amend codeobject.co_lines docs; end number is exclusive (#113970) The end number should be exclusive, not inclusive. * gh-111877: Fixes stat() handling for inaccessible files on Windows (GH-113716) * gh-113980: Fix resource warnings in test_asyncgen (GH-113984) * gh-107901: duplicate blocks with no lineno that have an eval break and multiple predecessors (#113950) * gh-113868: Add a number of MAP_* flags from macOS to module mmap (#113869) The new flags were extracted from the macOS 14.2 SDK. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> * gh-113710: Add types to the interpreter DSL (#113711) Co-authored-by: Jules <57632293+JuliaPoo@users.noreply.github.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> * gh-113971: Make `zipfile.ZipInfo._compresslevel` public as `.compress_level` (#113969) Make zipfile.ZipInfo.compress_level public. A property is used to retain the behavior of the ._compresslevel. People constructing zipfile.ZipInfo instances to pass into existing APIs to control per-file compression levels already treat this as public, there was never a reason for it not to be. I used the more modern name compress_level instead of compresslevel as the keyword argument on other ZipFile APIs is called to be consistent with compress_type and a general long term preference of not runningwordstogether without a separator in names. * GH-111802: set a low recursion limit for `test_bad_getattr()` in `test.pickletester` (GH-113996) * gh-95649: Document that asyncio contains uvloop code (#107536) Some of the asyncio SSL changes in GH-31275 [1] were taken from v0.16.0 of the uvloop project [2]. In order to comply with the MIT license, we need to just need to document the copyright information. [1]: https://github.com/python/cpython/pull/31275 [2]: https://github.com/MagicStack/uvloop/tree/v0.16.0 * gh-101100: Fix Sphinx Lint warnings in `Misc/` (#113946) Fix Sphinx Lint warnings in Misc/ * Fix a grammatical error in `pycore_pymem.h` (#112993) * Tutorial: Clarify 'nonzero exit status' in the appendix (#112039) * Link to the glossary for "magic methods" in ``MagicMock`` (#111292) The MagicMock documentation mentions magic methods several times without actually pointing to the term in the glossary. This can be helpful for people to fully understand what those magic methods are. * datamodel: Fix a typo in ``object.__init_subclass__`` (#111599) * GH-111801: set a lower recursion limit for `test_infintely_many_bases()` in `test_isinstance` (#113997) * gh-89159: Document missing TarInfo members (#91564) * GH-111798: skip `test_super_deep()` from `test_call` under pydebug builds on WASI (GH-114010) * GH-44626, GH-105476: Fix `ntpath.isabs()` handling of part-absolute paths (#113829) On Windows, `os.path.isabs()` now returns `False` when given a path that starts with exactly one (back)slash. This is more compatible with other functions in `os.path`, and with Microsoft's own documentation. Also adjust `pathlib.PureWindowsPath.is_absolute()` to call `ntpath.isabs()`, which corrects its handling of partial UNC/device paths like `//foo`. Co-authored-by: Jon Foster <jon@jon-foster.co.uk> * pathlib ABCs: add `_raw_path` property (#113976) It's wrong for the `PurePathBase` methods to rely so much on `__str__()`. Instead, they should treat the raw path(s) as opaque objects and leave the details to `pathmod`. This commit adds a `PurePathBase._raw_path` property and uses it through many of the other ABC methods. These methods are all redefined in `PurePath` and `Path`, so this has no effect on the public classes. * Add module docstring for `pathlib._abc`. (#113691) * gh-101225: Increase the socket backlog when creating a multiprocessing.connection.Listener (#113567) Increase the backlog for multiprocessing.connection.Listener` objects created by `multiprocessing.manager` and `multiprocessing.resource_sharer` to significantly reduce the risk of getting a connection refused error when creating a `multiprocessing.connection.Connection` to them. * gh-114014: Update `fractions.Fraction()`'s rational parsing regex (#114015) Fix a bug in the regex used for parsing a string input to the `fractions.Fraction` constructor. That bug led to an inconsistent exception message being given for some inputs. --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Mark Dickinson <dickinsm@gmail.com> * gh-111803: Support loading more deeply nested lists in binary plist format (GH-114024) It no longer uses the C stack. The depth of nesting is only limited by Python recursion limit setting. * gh-113317: Move global utility functions into libclinic (#113986) Establish Tools/clinic/libclinic/utils.py and move the following functions over there: - compute_checksum() - create_regex() - write_file() * gh-101100: Fix Sphinx warnings in `howto/urllib2.rst` and `library/http.client.rst` (#114060) * Add `pathlib._abc.PathModuleBase` (#113893) Path modules provide a subset of the `os.path` API, specifically those functions needed to provide `PurePathBase` functionality. Each `PurePathBase` subclass references its path module via a `pathmod` class attribute. This commit adds a new `PathModuleBase` class, which provides abstract methods that unconditionally raise `UnsupportedOperation`. An instance of this class is assigned to `PurePathBase.pathmod`, replacing `posixpath`. As a result, `PurePathBase` is no longer POSIX-y by default, and all its methods raise `UnsupportedOperation` courtesy of `pathmod`. Users who subclass `PurePathBase` or `PathBase` should choose the path syntax by setting `pathmod` to `posixpath`, `ntpath`, `os.path`, or their own subclass of `PathModuleBase`, as circumstances demand. * Replace `pathlib._abc.PathModuleBase.splitroot()` with `splitdrive()` (#114065) This allows users of the `pathlib-abc` PyPI package to use `posixpath` or `ntpath` as a path module in versions of Python lacking `os.path.splitroot()` (3.11 and before). * gh-113317: Move FormatCounterFormatter into libclinic (#114066) * gh-109862: Fix test_create_subprocess_with_pidfd when it was run separately (GH-113991) * gh-114075: Capture `test_compileall` stdout output (#114076) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> * gh-113666: Adding missing UF_ and SF_ flags to module 'stat' (#113667) Add some constants to module 'stat' that are used on macOS. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> * GH-112354: `_GUARD_IS_TRUE_POP` side-exits to target the next instruction, not themselves. (GH-114078) * gh-109598: make PyComplex_RealAsDouble/ImagAsDouble use __complex__ (GH-109647) `PyComplex_RealAsDouble()`/`PyComplex_ImagAsDouble` now try to convert an object to a `complex` instance using its `__complex__()` method before falling back to the ``__float__()`` method. PyComplex_ImagAsDouble() also will not silently return 0.0 for non-complex types anymore. Instead we try to call PyFloat_AsDouble() and return 0.0 only if this call is successful. * gh-112532: Fix memory block count for free-threaded build (gh-113995) This fixes `_PyInterpreterState_GetAllocatedBlocks()` and `_Py_GetGlobalAllocatedBlocks()` in the free-threaded builds. The gh-113263 change that introduced multiple mimalloc heaps per-thread broke the logic for counting the number of allocated blocks. For subtle reasons, this led to reported reference count leaks in the refleaks buildbots. * gh-111968: Use per-thread slice_cache in free-threading (gh-113972) * gh-99437: runpy: decode path-like objects before setting globals * gh-114070: correct the specification of ``digit`` in the float() docs (#114080) * gh-91539: Small performance improvement of urrlib.request.getproxies_environment() (#108771) Small performance improvement of getproxies_environment() when there are many environment variables. In a benchmark with 5k environment variables not related to proxies, and 5 specifying proxies, we get a 10% walltime improvement. * gh-112087: Update list impl to be thread-safe with manual CS (gh-113863) * gh-78502: Add a trackfd parameter to mmap.mmap() (GH-25425) If *trackfd* is False, the file descriptor specified by *fileno* will not be duplicated. Co-authored-by: Erlend E. Aasland <erlend@python.org> Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> * gh-114101: Correct PyErr_Format arguments in _testcapi module (#114102) - use PyErr_SetString() iso. PyErr_Format() in parse_tuple_and_keywords() - fix misspelled format specifier in CHECK_SIGNNESS() macro * GH-113655: Lower the C recursion limit on various platforms (GH-113944) * gh-113358: Fix rendering tracebacks with exceptions with a broken __getattr__ (GH-113359) Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> * gh-113238: add Anchor to importlib.resources (#113801) Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> * gh-114077: Fix OverflowError in socket.sendfile() when pass count >2GiB (GH-114079) * Docs: Align multiprocessing.shared_memory docs with Sphinx recommendations (#114103) - add :class: and :mod: markups where needed - fix incorrect escaping of a star in ShareableList arg spec - mark up parameters with stars: *val* - mark up list of built-in types using list markup - remove unneeded parentheses from :meth: markups * gh-113858: GH Actions: Limit max ccache size for the asan build (GH-114113) * gh-114107: Fix importlib.resources symlink test if symlinks aren't supported (#114108) gh-114107: Fix symlink test if symlinks aren't supported * gh-102468: Document `PyCFunction_New*` and `PyCMethod_New` (GH-112557) Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com> * gh-113626: Add allow_code parameter in marshal functions (GH-113648) Passing allow_code=False prevents serialization and de-serialization of code objects which is incompatible between Python versions. * gh-111968: Use per-thread freelists for PyContext in free-threading (gh-114122) * gh-114107: test.pythoninfo logs Windows Developer Mode (#114121) Also, don't skip the whole collect_windows() if ctypes is missing. Log also ctypes.windll.shell32.IsUserAnAdmin(). * Fix an incorrect comment in iobase_is_closed (GH-102952) This comment appears to have been mistakenly copied from what is now called iobase_check_closed() in commit 4d9aec022063. Also unite the iobase_check_closed() code with the relevant comment. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> * gh-114069: Revise Tutorial Methods paragraph (#114127) Remove excess words in the first and third sentences. * gh-114096: Restore privileges in _winapi.CreateJunction after creating the junction (GH-114089) This avoids impact on later parts of the application which may be able to do things they otherwise shouldn't. * Docs: Improve multiprocessing.SharedMemory reference (#114093) Align the multiprocessing shared memory docs with Diatáxis's recommendations for references. - use a parameter list for the SharedMemory.__init__() argument spec - use the imperative mode - use versionadded, not versionchanged, for added parameters - reflow touched lines according to SemBr * Fix 'expresion' typo in IDLE doc (#114130) The substantive change is on line 577/593. Rest is header/footer stuff ignored when displaying. * gh-113659: Skip hidden .pth files (GH-113660) Skip .pth files with names starting with a dot or hidden file attribute. * Clean up backslash avoiding code in ast, fix typo (#113605) As of #108553, the `_avoid_backslashes` code path is dead `scape_newlines` was introduced in #110271. Happy to drop the typo fix if we don't want it * GH-114013: fix setting `HOSTRUNNER` for `Tools/wasm/wasi.py` (GH-114097) Also fix tests found failing under a pydebug build of WASI thanks to `make test` working due to this change. * Update copyright years to 2024. (GH-113608) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> * gh-112529: Track if debug allocator is used as underlying allocator (#113747) * gh-112529: Track if debug allocator is used as underlying allocator The GC implementation for free-threaded builds will need to accurately detect if the debug allocator is used because it affects the offset of the Python object from the beginning of the memory allocation. The current implementation of `_PyMem_DebugEnabled` only considers if the debug allocator is the outer-most allocator; it doesn't handle the case of "hooks" like tracemalloc being used on top of the debug allocator. This change enables more accurate detection of the debug allocator by tracking when debug hooks are enabled. * Simplify _PyMem_DebugEnabled * gh-113655: Increase default stack size for PGO builds to avoid C stack exhaustion (GH-114148) * GH-78988: Document `pathlib.Path.glob()` exception propagation. (#114036) We propagate the `OSError` from the `is_dir()` call on the top-level directory, and suppress all others. * gh-94220: Align fnmatch docs with the implementation and amend markup (#114152) - Align the argument spec for fnmatch functions with the actual implementation. - Update Sphinx markup to recent recommandations. - Add link to 'iterable' glossary entry. Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> * Fix typo in c_annotations.py comment (#108773) "compatability" => "compatibility" * GH-110109: pathlib docs: bring `from_uri()` and `as_uri()` together. (#110312) This is a very soft deprecation of `PurePath.as_uri()`. We instead document it as a `Path` method, and add a couple of sentences mentioning that it's also available in `PurePath`. Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> * gh-106293: Fix typos in Objects/object_layout.md (#106294) Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> * gh-88531 Fix dataclass __post_init__/__init__ interplay documentation (gh-107404) * Simplify __post_init__ example usage. It applies to all base classes, not just dataclasses. * gh-112043: Align concurrent.futures.Executor.map docs with implementation (#114153) The first parameter is named 'fn', not 'func'. * gh-81479: For Help => IDLE Doc, stop double-spacing some lists. (#114168) This matches Firefox format. Edge double-spaces non-simple list but I think it looks worse. * gh-72284: Revise lists in IDLE doc (#114174) Tkinter is a fact, not necessarily a feature. Reorganize editor key bindings in a logical order and remove those that do not work, at least on Windows. Improve shell bindings list. * gh-86179: Skip test case that fails on POSIX with unversioned binary (GH-114136) * Python 3.13.0a3 * gh-104282: Fix null pointer dereference in `lzma._decode_filter_properties` (GH-104283) * gh-111301: Advertise importlib methods removal in What's new in Python 3.12 (GH-111630) * gh-112343: pdb: Use tokenize to replace convenience variables (#112380) * Post 3.13.0a3 * gh-114178: Fix generate_sbom.py for out-of-tree builds (#114179) * gh-114070: fix token reference warnings in expressions.rst (#114169) * gh-114149: [Enum] fix tuple subclass handling when using custom __new__ (GH-114160) * gh-105102: Fix nested unions in structures when the system byteorder is the opposite (GH-105106) * Fix typo in tkinter.ttk.rst (GH-106157) * gh-38807: Fix race condition in Lib/trace.py (GH-110143) Instead of checking if a directory does not exist and thereafter creating it, directly call os.makedirs() with the exist_ok=True. * gh-112984 Update Windows build and installer for free-threaded builds (GH-113129) * gh-112984: Fix test_ctypes.test_loading.test_load_dll_with_flags when directory name includes a dot (GH-114217) * gh-114149: [Enum] revert #114160 and add more tuple-subclass tests (GH-114215) This reverts commit 05e142b1543eb9662d6cc33722e7e16250c9219f. * gh-104522: Fix OSError raised when run a subprocess (#114195) Only set filename to cwd if it was caused by failed chdir(cwd). _fork_exec() now returns "noexec:chdir" for failed chdir(cwd). Co-authored-by: Robert O'Shea <PurityLake@users.noreply.github.com> * gh-113205: test_multiprocessing.test_terminate: Test the API on threadpools (#114186) gh-113205: test_multiprocessing.test_terminate: Test the API works on threadpools Threads can't be forced to terminate (without potentially corrupting too much state), so the expected behaviour of `ThreadPool.terminate` is to wait for the currently executing tasks to finish. The entire test was skipped in GH-110848 (0e9c364f4ac18a2237bdbac702b96bcf8ef9cb09). Instead of skipping it entirely, we should ensure the API eventually succeeds: use a shorter timeout. For the record: on my machine, when the test is un-skipped, the task manages to start in about 1.5% cases. * gh-114211: Update EmailMessage doc about ordered keys (#114224) Ordered keys are no longer unlike 'real dict's. * gh-96905: In IDLE code, stop redefining built-ins 'dict' and 'object' (#114227) Prefix 'dict' with 'o', 'g', or 'l' for 'object', 'global', or 'local'. Suffix 'object' with '_'. * gh-114231: Fix indentation in enum.rst (#114232) * gh-104522: Fix test_subprocess failure when build Python in the root home directory (GH-114236) * gh-104522: Fix test_subprocess failure when build Python in the root home directory EPERM is raised when setreuid() fails. EACCES is set in execve() when the test user has not access to sys.executable. * gh-114050: Fix crash when more than two arguments are passed to int() (GH-114067) Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru> * gh-103092: Convert some `_ctypes` metatypes to heap types (GH-113620) Co-authored-by: Erlend E. Aasland <erlend@python.org> * gh-110345: show Tcl/Tk patchlevel in `tkinter._test()` (GH-110350) * Delete unused macro (GH-114238) * gh-108303: Move all doctest related files and tests to `Lib/test/test_doctest/` (#112109) Co-authored-by: Brett Cannon <brett@python.org> * gh-114198: Rename dataclass __replace__ argument to 'self' (gh-114251) This change renames the dataclass __replace__ method's first argument name from 'obj' to 'self'. * gh-114087: Speed up dataclasses._asdict_inner (#114088) * gh-111968: Use per-thread freelists for generator in free-threading (gh-114189) * gh-112092: clarify unstable ABI recompilation requirements (#112093) Use different versions in the examples for when extensions do and do not need to be recompiled to make the examples easier to understand. * gh-114123: Migrate docstring from _csv to csv (#114124) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Éric <merwok@netwok.org> * gh-112087: Remove duplicated critical_section (gh-114268) * gh-111968: Fix --without-freelists build (gh-114270) * gh-114286: Fix `maybe-uninitialized` warning in `Modules/_io/fileio.c` (GH-114287) * gh-113884: Refactor `queue.SimpleQueue` to use a ring buffer to store items (#114259) Use a ring buffer instead of a Python list in order to simplify the process of making queue.SimpleQueue thread-safe in free-threaded builds. The ring buffer implementation has no places where critical sections may be released. * gh-114275: Skip doctests that use `asyncio` in `test_pdb` for WASI builds (#114309) * gh-114265: move line number propagation before cfg optimization, remove guarantee_lineno_for_exits (#114267) * Retain shorter tables of contents for Sphinx 5.2.3+ (#114318) Disable toc_object_entries, new in Sphinx 5.2.3 * Add a `clean` subcommand to `Tools/wasm/wasi.py` (GH-114274) * GH-79634: Accept path-like objects as pathlib glob patterns. (#114017) Allow `os.PathLike` objects to be passed as patterns to `pathlib.Path.glob()` and `rglob()`. (It's already possible to use them in `PurePath.match()`) While we're in the area: - Allow empty glob patterns in `PathBase` (but not `Path`) - Speed up globbing in `PathBase` by generating paths with trailing slashes only as a final step, rather than for every intermediate directory. - Simplify and speed up handling of rare patterns involving both `**` and `..` segments. * GH-113225: Speed up `pathlib.Path.walk(top_down=False)` (#113693) Use `_make_child_entry()` rather than `_make_child_relpath()` to retrieve path objects for directories to visit. This saves the allocation of one path object per directory in user subclasses of `PathBase`, and avoids a second loop. This trick does not apply when walking top-down, because users can affect the walk by modifying *dirnames* in-place. A side effect of this change is that, in bottom-up mode, subdirectories of each directory are visited in reverse order, and that this order doesn't match that of the names in *dirnames*. I suspect this is fine as the order is arbitrary anyway. * gh-114332: Fix the flags reference for ``re.compile()`` (#114334) The GH-93000 change set inadvertently caused a sentence in re.compile() documentation to refer to details that no longer followed. Correct this with a link to the Flags sub-subsection. Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com> * GH-99380: Update to Sphinx 7 (#99381) * Docs: structure the ftplib reference (#114317) Introduce the following headings and subheadings: - Reference * FTP objects * FTP_TLS objects * Module variables * gh-112529: Use GC heaps for GC allocations in free-threaded builds (gh-114157) * gh-112529: Use GC heaps for GC allocations in free-threaded builds The free-threaded build's garbage collector implementation will need to find GC objects by traversing mimalloc heaps. This hooks up the allocation calls with the correct heaps by using a thread-local "current_obj_heap" variable. * Refactor out setting heap based on type * gh-114281: Remove incorrect type hints from `asyncio.staggered` (#114282) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> * Docs: Add missing line continuation to FTP_TLS class docs (#114352) Regression introduced by b1ad5a5d4. * Remove the non-test Lib/test/time_hashlib.py. (#114354) I believe I added this while chasing some performance of hash functions when I first created hashlib. It hasn't been used since, is frankly trivial, and not a test. * Remove deleted `time_hashlib.py` from `Lib/test/.ruff.toml` (#114355) * Fix the confusing "User-defined methods" reference in the datamodel (#114276) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com> * Docs: mark up the FTP debug levels as a list (#114360) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> * gh-101100: Fix sphinx warnings in `Doc/c-api/memory.rst` (#114373) * gh-80931: Skip some socket tests while hunting for refleaks on macOS (#114057) Some socket tests related to sending file descriptors cause a file descriptor leak on macOS, all of them tests that send one or more descriptors than cannot be received on the read end. This appears to be a platform bug. This PR skips those tests when doing a refleak test run to avoid hiding other problems. * Docs: mark up FTP() constructor with param list (#114359) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> * gh-114384: Align sys.set_asyncgen_hooks signature in docs to reflect implementation (#114385) * Docs: link to sys.stdout in ftplib docs (#114396) --------- Co-authored-by: Donghee Na <donghee.na@python.org> Co-authored-by: Sam Gross <colesbury@gmail.com> Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Co-authored-by: Itamar Oren <itamarost@gmail.com> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com> Co-authored-by: Filip Łapkiewicz <80906036+fipachu@users.noreply.github.com> Co-authored-by: Brandt Bucher <brandtbucher@microsoft.com> Co-authored-by: Jamie Phan <jamie@ordinarylab.dev> Co-authored-by: wookie184 <wookie1840@gmail.com> Co-authored-by: Guido van Rossum <guido@python.org> Co-authored-by: Barney Gale <barney.gale@gmail.com> Co-authored-by: Mark Shannon <mark@hotpy.org> Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Zackery Spytz <zspytz@gmail.com> Co-authored-by: Xavier de Gaye <xdegaye@gmail.com> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Ronald Oussoren <ronaldoussoren@mac.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: AN Long <aisk@users.noreply.github.com> Co-authored-by: Grigoriev Semyon <33061489+grigoriev-semyon@users.noreply.github.com> Co-authored-by: Rami <72725910+ramikg@users.noreply.github.com> Co-authored-by: Christian Heimes <christian@python.org> Co-authored-by: Gregory P. Smith <greg@krypto.org> Co-authored-by: Erlend E. Aasland <erlend@python.org> Co-authored-by: Levi Sabah <0xl3vi@gmail.com> Co-authored-by: Lee Cannon <leecannon@leecannon.xyz> Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com> Co-authored-by: neonene <53406459+neonene@users.noreply.github.com> Co-authored-by: Raymond Hettinger <rhettinger@users.noreply.github.com> Co-authored-by: Jakub Kulík <Kulikjak@gmail.com> Co-authored-by: Kristján Valur Jónsson <sweskman@gmail.com> Co-authored-by: Steve Dower <steve.dower@python.org> Co-authored-by: mara004 <geisserml@gmail.com> Co-authored-by: William Andrea <william.j.andrea@gmail.com> Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com> Co-authored-by: Vinay Sajip <vinay_sajip@yahoo.co.uk> Co-authored-by: Yan Yanchii <yyanchiy@gmail.com> Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com> Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com> Co-authored-by: Stefano Rivera <stefano@rivera.za.net> Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Seth Michael Larson <sethmichaellarson@gmail.com> Co-authored-by: Steve Dower <steve.dower@microsoft.com> Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru> Co-authored-by: Nikita Sobolev <mail@sobolevn.me> Co-authored-by: Peter Lazorchak <lazorchakp@gmail.com> Co-authored-by: Mariusz Felisiak <felisiak.mariusz@gmail.com> Co-authored-by: Ned Batchelder <ned@nedbatchelder.com> Co-authored-by: Ken Jin <kenjin@python.org> Co-authored-by: Jules <57632293+JuliaPoo@users.noreply.github.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Brett Cannon <brett@python.org> Co-authored-by: Alois Klink <alois@aloisklink.com> Co-authored-by: Joseph Pearson <74079531+JoeyPearson822@users.noreply.github.com> Co-authored-by: Andrew Zipperer <47086307+zipperer@users.noreply.github.com> Co-authored-by: Pierre Equoy <pierre.equoy@canonical.com> Co-authored-by: InSync <122007197+InSyncWithFoo@users.noreply.github.com> Co-authored-by: Stanley <46876382+slateny@users.noreply.github.com> Co-authored-by: Jon Foster <jon@jon-foster.co.uk> Co-authored-by: Crowthebird <78076854+thatbirdguythatuknownot@users.noreply.github.com> Co-authored-by: Mark Dickinson <dickinsm@gmail.com> Co-authored-by: Kamil Turek <kamil.turek@hotmail.com> Co-authored-by: Raphaël Marinier <raphael.marinier@gmail.com> Co-authored-by: Jérome Perrin <perrinjerome@gmail.com> Co-authored-by: Mike Zimin <122507876+mikeziminio@users.noreply.github.com> Co-authored-by: Jonathon Reinhart <JonathonReinhart@users.noreply.github.com> Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com> Co-authored-by: solya0x <41440448+0xSalikh@users.noreply.github.com> Co-authored-by: Kuan-Wei Chiu <visitorckw@gmail.com> Co-authored-by: Mano Sriram <mano.sriram0@gmail.com> Co-authored-by: Steffen Zeile <48187781+Kaniee@users.noreply.github.com> Co-authored-by: Thomas Wouters <thomas@python.org> Co-authored-by: Radislav Chugunov <52372310+chgnrdv@users.noreply.github.com> Co-authored-by: Karolina Surma <33810531+befeleme@users.noreply.github.com> Co-authored-by: Tian Gao <gaogaotiantian@hotmail.com> Co-authored-by: Ethan Furman <ethan@stoneleaf.us> Co-authored-by: Sheidan <37596668+Sh3idan@users.noreply.github.com> Co-authored-by: Christophe Nanteuil <35002064+christopheNan@users.noreply.github.com> Co-authored-by: buermarc <44375277+buermarc@users.noreply.github.com> Co-authored-by: Robert O'Shea <PurityLake@users.noreply.github.com> Co-authored-by: Miyashita Yosuke <44266492+miyashiiii@users.noreply.github.com> Co-authored-by: kcatss <kcats9731@gmail.com> Co-authored-by: Christopher Chavez <chrischavez@gmx.us> Co-authored-by: Phillip Schanely <pschanely@gmail.com> Co-authored-by: keithasaurus <592217+keithasaurus@users.noreply.github.com> Co-authored-by: DerSchinken <53398996+DerSchinken@users.noreply.github.com> Co-authored-by: Skip Montanaro <skip.montanaro@gmail.com> Co-authored-by: Éric <merwok@netwok.org> Co-authored-by: mpage <mpage@meta.com> Co-authored-by: David H. Gutteridge <dhgutteridge@users.noreply.github.com> Co-authored-by: cdzhan <zhancdi@163.com>

…uild (#114564) * gh-112529: Remove PyGC_Head from object pre-header in free-threaded build This avoids allocating space for PyGC_Head in the free-threaded build. The GC implementation for free-threaded CPython does not use the PyGC_Head structure. * The trashcan mechanism uses the `ob_tid` field instead of `_gc_prev` in the free-threaded build. * The GDB libpython.py file now determines the offset of the managed dict field based on whether the running process is a free-threaded build. Those are identified by the `ob_ref_local` field in PyObject. * Fixes `_PySys_GetSizeOf()` which incorrectly incorrectly included the size of `PyGC_Head` in the size of static `PyTypeObject`.

The GC keeps track of the number of allocations (less deallocations) since the last GC. This buffers the count in thread-local state and uses atomic operations to modify the per-interpreter count. The thread-local buffering avoids contention on shared state. A consequence is that the GC scheduling is not as precise, so "test_sneaky_frame_object" is skipped because it requires that the GC be run exactly after allocating a frame object.

We do not want to add locking in `tp_traverse` slot implementations. Instead, stop the world when calling `gc.get_referents`. Note that the the stop the world call is a no-op in the default build. Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>

…hon#112533) * pythongh-112529: Use atomic operations for `gcstate->collecting` The `collecting` field in `GCState` is used to prevent overlapping garbage collections within the same interpreter. This is updated to use atomic operations in order to be thread-safe in `--disable-gil` builds. The GC code is refactored a bit to support this. More of the logic is pushed down to `gc_collect_main()` so that we can safely order the logic setting `collecting`, the selection of the generation, and the invocation of callbacks with respect to the atomic operations and the (future) stop-the-world pauses. The change uses atomic operations for both `--disable-gil` and the default build (with the GIL) to avoid extra `#ifdef` guards and ease the maintenance burden.

…ator (python#113747) * pythongh-112529: Track if debug allocator is used as underlying allocator The GC implementation for free-threaded builds will need to accurately detect if the debug allocator is used because it affects the offset of the Python object from the beginning of the memory allocation. The current implementation of `_PyMem_DebugEnabled` only considers if the debug allocator is the outer-most allocator; it doesn't handle the case of "hooks" like tracemalloc being used on top of the debug allocator. This change enables more accurate detection of the debug allocator by tracking when debug hooks are enabled. * Simplify _PyMem_DebugEnabled

…lds (pythongh-114157) * pythongh-112529: Use GC heaps for GC allocations in free-threaded builds The free-threaded build's garbage collector implementation will need to find GC objects by traversing mimalloc heaps. This hooks up the allocation calls with the correct heaps by using a thread-local "current_obj_heap" variable. * Refactor out setting heap based on type

* pythongh-112529: Implement GC for free-threaded builds This implements a mark and sweep GC for the free-threaded builds of CPython. The implementation relies on mimalloc to find GC tracked objects (i.e., "containers").

…aded build (python#114564) * pythongh-112529: Remove PyGC_Head from object pre-header in free-threaded build This avoids allocating space for PyGC_Head in the free-threaded build. The GC implementation for free-threaded CPython does not use the PyGC_Head structure. * The trashcan mechanism uses the `ob_tid` field instead of `_gc_prev` in the free-threaded build. * The GDB libpython.py file now determines the offset of the managed dict field based on whether the running process is a free-threaded build. Those are identified by the `ob_ref_local` field in PyObject. * Fixes `_PySys_GetSizeOf()` which incorrectly incorrectly included the size of `PyGC_Head` in the size of static `PyTypeObject`.

We do not want to add locking in `tp_traverse` slot implementations. Instead, stop the world when calling `gc.get_referents`. Note that the the stop the world call is a no-op in the default build. Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>

The free-threaded GC uses mimallocs segment thread IDs to restore the overwritten `ob_tid` thread ids in PyObjects. For that reason, it's important that PyObjects and mimalloc use the same identifiers.

…5488) The free-threaded GC uses mimallocs segment thread IDs to restore the overwritten `ob_tid` thread ids in PyObjects. For that reason, it's important that PyObjects and mimalloc use the same identifiers.

…ded build" Reverts: python#115488 Reason: Free-threaded Ubuntu buildbots are timing out

The GC keeps track of the number of allocations (less deallocations) since the last GC. This buffers the count in thread-local state and uses atomic operations to modify the per-interpreter count. The thread-local buffering avoids contention on shared state. A consequence is that the GC scheduling is not as precise, so "test_sneaky_frame_object" is skipped because it requires that the GC be run exactly after allocating a frame object.

…#114732)

The GC keeps track of the number of allocations (less deallocations) since the last GC. This buffers the count in thread-local state and uses atomic operations to modify the per-interpreter count. The thread-local buffering avoids contention on shared state. A consequence is that the GC scheduling is not as precise, so "test_sneaky_frame_object" is skipped because it requires that the GC be run exactly after allocating a frame object.

…alized (python#114732)

The free-threaded GC sometimes sees objects with zero refcount. This can happen due to the delay in merging biased reference counting fields, and, in the future, due to deferred reference counting. We should not untrack these objects or they will never be collected.

The free-threaded GC sometimes sees objects with zero refcount. This can happen due to the delay in merging biased reference counting fields, and, in the future, due to deferred reference counting. We should not untrack these objects or they will never be collected. This fixes the refleaks in the free-threaded build.

commit 9dae05ee59eeba0e67af2a46f2a2907c9f8d7e4a Author: Moshe Kaplan <mosheekaplan@gmail.com> Date: Mon Apr 1 15:53:00 2024 -0400 Docs: specify XML document name in xml.etree.elementtree example (#24223) commit fc2071687b708598264a3403b7f9104667c1092f Author: Matthew Davis <7035647+mdavis-xyz@users.noreply.github.com> Date: Mon Apr 1 21:49:14 2024 +0200 Docs: add more links to PIPE in subprocess docs (#25416) commit fc8007ee3635db6ab73e132ebff987c910b6d538 Author: Barney Gale <barney.gale@gmail.com> Date: Mon Apr 1 20:37:41 2024 +0100 GH-117337: Deprecate `glob.glob0()` and `glob.glob1()`. (#117371) These undocumented functions are no longer used by `msilib`, so there's no reason to keep them around. commit c741ad3537193c63fe697a8f0316aecd45eeb9ba Author: Justin Turner Arthur <justinarthur@gmail.com> Date: Mon Apr 1 12:07:29 2024 -0500 gh-77714: Provide an async iterator version of as_completed (GH-22491) * as_completed returns object that is both iterator and async iterator * Existing tests adjusted to test both the old and new style * New test to ensure iterator can be resumed * New test to ensure async iterator yields any passed-in Futures as-is Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Guido van Rossum <gvanrossum@gmail.com> commit ddf814db744006e0f42328aa15ace97c9d8ad681 Author: Guido van Rossum <guido@python.org> Date: Mon Apr 1 09:13:38 2024 -0700 Silence compiler warnings in gc.c (#117422) commit 179869af922252a0c1cef65fd2923856895e7d1b Author: Petr Viktorin <encukou@gmail.com> Date: Mon Apr 1 17:01:22 2024 +0200 gh-94808: Fix refcounting in PyObject_Print tests (GH-117421) commit dd44ab994b7262f0704d64996e0a1bc37b233407 Author: neonene <53406459+neonene@users.noreply.github.com> Date: Mon Apr 1 22:28:14 2024 +0900 gh-117142: ctypes: Unify meta tp slot functions (GH-117143) Integrates the following ctypes meta tp slot functions: * `CDataType_traverse()` into `CType_Type_traverse()`. * `CDataType_clear()` into `CType_Type_clear()`. * `CDataType_dealloc()` into `CType_Type_dealloc()`. * `CDataType_repeat()` into `CType_Type_repeat()`. commit 3de09cadde788065a4f2d45117e789c9353bbd12 Author: Steve (Gadget) Barnes <gadgetsteve@hotmail.com> Date: Mon Apr 1 14:02:07 2024 +0100 gh-91565: Replace bugs.python.org links with Devguide/GitHub ones (GH-91568) Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com> Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net> Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com> commit 90c3c68a658db6951b77a5be50088ec2f6adc8eb Author: MonadChains <monadchains@gmail.com> Date: Mon Apr 1 13:52:25 2024 +0100 gh-94808:Improve coverage of PyObject_Print (GH-98749) commit 348cf6e0078eae156c503e8f61ef5e27ae28e57b Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Apr 1 11:14:37 2024 +0000 Bump mypy from 1.8.0 to 1.9.0 in /Tools (#117418) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> commit 9b403fb559bfce93a478937e0ef7e539a9a95283 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Apr 1 11:05:14 2024 +0000 build(deps-dev): bump types-psutil from 5.9.5.20240205 to 5.9.5.20240316 in /Tools (#117417) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> commit 93c7d9d17b571b6a181af7c02830d29819535c35 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Apr 1 11:38:38 2024 +0100 build(deps-dev): bump types-setuptools from 69.1.0.20240301 to 69.2.0.20240317 in /Tools (#117419) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 3bb12e407c183946471272f8aee098e54e62a333 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Apr 1 09:54:33 2024 +0000 build(deps): bump actions/add-to-project from 0.6.0 to 1.0.0 (#117415) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 56e99307c49adcc6df355f8070229371c97d654f Author: Adorilson Bezerra <adorilson@gmail.com> Date: Sun Mar 31 23:34:54 2024 +0100 Doc: printf-style library/stdtype improvements (#16741) commit 18e12641a61a88f7d08b2114ebe965892c6661c5 Author: Raymond Hettinger <rhettinger@users.noreply.github.com> Date: Sun Mar 31 16:09:22 2024 -0500 gh-117387 Remove hash mark from introductory text (#117409) commit a32d6939486d7f90ee57e215077f6116e19de24d Author: Deborah <32307299+dlwrnc@users.noreply.github.com> Date: Sun Mar 31 22:11:48 2024 +0200 gh-102190: Add additional zipfile `pwd=` arg docstrings (gh-102195) This just documents the parameter that already exists. --------- Co-authored-by: Gregory P. Smith <greg@krypto.org> Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com> commit 262445358e21c56d7c68e3ee76c13e469d2ea348 Author: Shantanu <12621235+hauntsaninja@users.noreply.github.com> Date: Sun Mar 31 12:02:48 2024 -0700 Link to the Python type system specification (#117400) commit 752e18389ed03087b51b38eac9769ef8dfd167b7 Author: Barney Gale <barney.gale@gmail.com> Date: Sun Mar 31 19:14:48 2024 +0100 GH-114575: Rename `PurePath.pathmod` to `PurePath.parser` (#116513) And rename the private base class from `PathModuleBase` to `ParserBase`. commit bfc57d43d8766120ba0c8f3f6d7b2ac681a81d8a Author: Sam Gross <colesbury@gmail.com> Date: Fri Mar 29 18:58:08 2024 -0400 gh-117303: Don't detach in `PyThreadState_DeleteCurrent()` (#117304) This fixes a crash in `test_threading.test_reinit_tls_after_fork()` when running with the GIL disabled. We already properly handle the case where the thread state is `_Py_THREAD_ATTACHED` in `tstate_delete_common()` -- we just need to remove an assertion. Keeping the thread attached means that a stop-the-world pause, such as for a `fork()`, won't commence until we remove our thread state from the interpreter's linked list. This prevents a crash when the child process tries to clean up the dead thread states. commit 05e0b67a43c5c1778dc2643c8b7c12864e135999 Author: Erlend E. Aasland <erlend@python.org> Date: Fri Mar 29 21:23:28 2024 +0100 gh-116664: In _warnings.c, make filters_version access thread-safe (#117374) - assert that the lock is held in already_warned() - protect 'filters_version' increment in warnings_filters_mutated_impl() commit 019143fecbfc26e69800d28d2a9e3392a051780b Author: Jason R. Coombs <jaraco@jaraco.com> Date: Fri Mar 29 16:06:09 2024 -0400 gh-117348: Refactored RawConfigParser._read for similicity and comprehensibility (#117372) * Extract method for _read_inner, reducing complexity and indentation by 1. * Extract method for _raise_all and yield ParseErrors from _read_inner. Reduces complexity by 1 and reduces touch points for handling errors in _read_inner. * Prefer iterators to splat expansion and literal indexing. * Extract method for _strip_comments. Reduces complexity by 7. * Model the file lines in a class to encapsulate the comment status and cleaned value. * Encapsulate the read state as a dataclass * Extract _handle_continuation_line and _handle_rest methods. Reduces complexity by 8. * Reindent * At least for now, collect errors in the ReadState * Check for missing section header separately. * Extract methods for _handle_header and _handle_option. Reduces complexity by 6. * Remove unreachable code. Reduces complexity by 4. * Remove unreachable branch * Handle error condition early. Reduces complexity by 1. * Add blurb * Move _raise_all to ParsingError, as its behavior is most closely related to the exception class and not the reader. * Split _strip* into separate methods. * Refactor _strip_full to compute the strip just once and use 'not any' to determine the factor. * Replace use of 'sys.maxsize' with direct computation of the stripped value. * Extract has_comments as a dynamic property. * Implement clean as a cached property. * Model comment prefixes in the RawConfigParser within a prefixes namespace. * Use a regular expression to search for the first match. Avoids mutating variables and tricky logic and over-computing all of the starts when only the first is relevant. commit 01bd74eadbc4ff839d39762fae6366f50c1e116e Author: Sam Gross <colesbury@gmail.com> Date: Fri Mar 29 15:33:06 2024 -0400 gh-117300: Use stop the world to make `sys._current_frames` and `sys._current_exceptions` thread-safe. (#117301) This adds a stop the world pause to make the two functions thread-safe when the GIL is disabled in the free-threaded build. Additionally, the main test thread may call `sys._current_exceptions()` as soon as `g_raised.set()` is called. The background thread may not yet reach the `leave_g.wait()` line. commit 94c97423a9c4969f8ddd4a3aa4aacb99c4d5263d Author: Guido van Rossum <guido@python.org> Date: Fri Mar 29 11:31:09 2024 -0700 Fix broken format in error for bad input in summarize_stats.py (#117375) When you pass the script a non-existent input file, you get a TypeError instead of the intended ValueError. commit 5d21d884b6ffa45dac50a5f9a07c41356a8478b4 Author: mpage <mpage@meta.com> Date: Fri Mar 29 10:42:02 2024 -0700 gh-111926: Avoid locking in PyType_IsSubtype (#117275) Read the MRO in a thread-unsafe way in `PyType_IsSubtype` to avoid locking. Fixing this is tracked in #117306. The motivation for this change is in support of making weakrefs thread-safe in free-threaded builds: `WeakValueDictionary` uses a special dictionary function, `_PyDict_DelItemIf` to remove dead weakrefs from the dictionary. `_PyDict_DelItemIf` removes a key if a user supplied predicate evaluates to true for the value associated with the key. Crucially for the `WeakValueDictionary` use case, the predicate evaluation + deletion sequence is atomic, provided that the predicate doesn’t suspend. The predicate used by `WeakValueDictionary` includes a subtype check, which we must ensure doesn't suspend in free-threaded builds. commit 19c1dd60c5b53fb0533610ad139ef591294f26e8 Author: Sam Gross <colesbury@gmail.com> Date: Fri Mar 29 13:35:43 2024 -0400 gh-117323: Make `cell` thread-safe in free-threaded builds (#117330) Use critical sections to lock around accesses to cell contents. The critical sections are no-ops in the default (with GIL) build. commit 397d88db5e9ab2a43de3fdf5f8b973a949edc405 Author: Sam Gross <colesbury@gmail.com> Date: Fri Mar 29 13:34:04 2024 -0400 gh-117344: Skip flaky tests in free-threaded build (#117355) The tests are not reliable with the GIL disabled. In theory, they can fail with the GIL enabled too, but the failures are much more likely with the GIL disabled. commit f05fb2e65c2dffdfae940f2707765c4994925205 Author: Sam Gross <colesbury@gmail.com> Date: Fri Mar 29 13:33:04 2024 -0400 gh-112529: Don't untrack tuples or dicts with zero refcount (#117370) The free-threaded GC sometimes sees objects with zero refcount. This can happen due to the delay in merging biased reference counting fields, and, in the future, due to deferred reference counting. We should not untrack these objects or they will never be collected. This fixes the refleaks in the free-threaded build. commit ddf95b5f16031cdbd0d728e55eb06dff002a8678 Author: Erlend E. Aasland <erlend@python.org> Date: Fri Mar 29 18:26:06 2024 +0100 gh-116664: Fix unused var warnings in _warnings.c in non-free-threaded builds (#117373) The warnings were introduced by commit c1712ef06. commit 0fa571dbcdf19b541276cb00bb929381930467b2 Author: Tian Gao <gaogaotiantian@hotmail.com> Date: Fri Mar 29 09:02:01 2024 -0700 Refactor pdb executable targets (#112570) Co-authored-by: Jason R. Coombs <jaraco@jaraco.com> commit 54f7e14500471d1c46fb553adb3ca24cd1fef084 Author: Pedro Lacerda <pslacerda@users.noreply.github.com> Date: Fri Mar 29 12:05:00 2024 -0300 gh-66449: configparser: Add support for unnamed sections (#117273) Co-authored-by: Jason R. Coombs <jaraco@jaraco.com> commit d9cfe7e565a6e2dc15747a904736264e31a10be4 Author: Nikita Sobolev <mail@sobolevn.me> Date: Fri Mar 29 14:14:25 2024 +0300 gh-117166: Ignore empty and temporary dirs in `test_makefile` (#117190) commit 35b6c4a4da201a947b2ceb96ae4c0d83d4d2df4f Author: Victor Stinner <vstinner@python.org> Date: Fri Mar 29 11:25:17 2024 +0100 gh-117347: Fix test_clinic side effects (#117363) Save/restore converters in ClinicWholeFileTest and ClinicExternalTest. commit 7e2fef865899837c47e91ef0180fa59eb03e840b Author: neonene <53406459+neonene@users.noreply.github.com> Date: Fri Mar 29 18:40:48 2024 +0900 gh-117142: ctypes: Migrate global vars to module state (GH-117189) commit 2e9be80c99f635c2f7761e8356b0260922d6e7a6 Author: Gregory P. Smith <greg@krypto.org> Date: Thu Mar 28 17:58:37 2024 -0700 Fix reversed assertRegex checks in test_ssl. (#117351) commit 8eec7ed714e65d616573b7331780b0aa43c6ed6a Author: 傅立业（Chris Fu） <17433201@qq.com> Date: Fri Mar 29 08:19:20 2024 +0800 gh-117110: Fix subclasses of typing.Any with custom constructors (#117111) commit a17f313e3958e825db9a83594c8471a984316536 Author: Christopher Chianelli <christopher@timefold.ai> Date: Thu Mar 28 18:26:56 2024 -0400 gh-117339: Use NULL instead of None for LOAD_SUPER_ATTR in dis docs (GH-117343) commit 26d328b2ba26374fb8d9ffe8215ecef7c5e3f7a2 Author: Michael Droettboom <mdboom@gmail.com> Date: Thu Mar 28 18:23:08 2024 -0400 GH-117121: Add pystats to JIT builds (GH-117346) commit 14f1ca7d5363386163839b31ce987423daecc3de Author: Nice Zombies <nineteendo19d0@gmail.com> Date: Thu Mar 28 22:20:08 2024 +0100 gh-117335: Handle non-iterables for `ntpath.commonpath` (GH-117336) commit 18cf239e39e25e6cef50ecbb7f197a82f8920ff5 Author: Brandt Bucher <brandtbucher@microsoft.com> Date: Thu Mar 28 14:02:34 2024 -0700 Increase the JIT CI timeouts to 75 minutes (GH-117342) commit 29829b58a8328a7c2ccacaa74c1d7d120a5e5ca5 Author: Malcolm Smith <smith@chaquo.com> Date: Thu Mar 28 19:59:12 2024 +0000 gh-117294: Report DocTestCase as skipped if all examples in the doctest are skipped (GH-117297) commit efcc96844e7c66fcd6c23ac2d557ca141614ce9a Author: Tian Gao <gaogaotiantian@hotmail.com> Date: Thu Mar 28 11:23:29 2024 -0700 gh-69201: Separate stdout and stderr stream in test_pdb (#117308) commit 6702d2bf6edcd5b5415e17837383623b9d76a5b8 Author: Victor Stinner <vstinner@python.org> Date: Thu Mar 28 17:40:58 2024 +0100 gh-114331: Skip decimal test_maxcontext_exact_arith on s390x (#117326) commit c1712ef066321c01bf09cba3f22fc474b5b8dfa7 Author: Erlend E. Aasland <erlend@python.org> Date: Thu Mar 28 16:05:08 2024 +0100 gh-116664: Make module state Py_SETREF's in _warnings thread-safe (#116959) Mark the swap operations as critical sections. Add an internal Py_BEGIN_CRITICAL_SECTION_MUT API that takes a PyMutex pointer instead of a PyObject pointer. commit 9a388b9a64927c372d85f0eaec3de9b7320a6fb5 Author: Joachim Wuttke <j.wuttke@fz-juelich.de> Date: Thu Mar 28 14:43:07 2024 +0100 bpo-43848: explain optional argument mtime in gzip.py. (GH-25410) Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> commit 8dbfdb2957a7baade3a88661517f163ad694c39f Author: Sam Gross <colesbury@gmail.com> Date: Thu Mar 28 09:28:39 2024 -0400 gh-110481: Fix biased reference counting queue initialization. (#117271) The biased reference counting queue must be initialized from the bound (active) thread because it uses `_Py_ThreadId()` as the key in a hash table. commit 9a1e55b8c5723206116f7016921be3937ef2f4e5 Author: Chris Markiewicz <effigies@gmail.com> Date: Thu Mar 28 06:59:31 2024 -0400 gh-117178: Recover lazy loading of self-referential modules (#117179) commit 4c71d51a4b7989fc8754ba512c40e21666f9db0d Author: Jelle Zijlstra <jelle.zijlstra@gmail.com> Date: Thu Mar 28 04:30:31 2024 -0600 gh-117266: Fix crashes on user-created AST subclasses (GH-117276) Fix crashes on user-created AST subclasses commit 8cb7d7ff86a1a2d41195f01ba4f218941dd7308c Author: Gregory P. Smith <greg@krypto.org> Date: Thu Mar 28 03:11:58 2024 -0700 gh-117310: Remove extra DECREF on "no ciphers" error path in `_ssl._SSLContext` constructor (#117309) Remove extra self DECREF on ssl "no ciphers" error path. This doesn't come up in practice because nobody links against a broken OpenSSL library that provides nothing. commit 6c8ac8a32fd6de1960526561c44bc5603fab0f3e Author: Erlend E. Aasland <erlend@python.org> Date: Thu Mar 28 09:40:37 2024 +0100 gh-116303: Handle disabled test modules in test.support helpers (#116482) Make sure test.support helpers skip iso. failing if test extension modules are disabled. Also log TEST_MODULES in test.pythoninfo. commit 0f27672c5002de96c9f1228b12460d5ce3f1d190 Author: Russell Keith-Magee <russell@keith-magee.com> Date: Thu Mar 28 16:13:13 2024 +0800 gh-114099: Add documentation for iOS platform (GH-117057) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Jacob Coffee <jacob@z7x.org> Co-authored-by: Malcolm Smith <smith@chaquo.com> Co-authored-by: Ned Deily <nad@python.org> commit f006338017cfbf846e8f7391b9ee5f69df8dc620 Author: Russell Keith-Magee <russell@keith-magee.com> Date: Thu Mar 28 15:59:33 2024 +0800 gh-114099: Additions to standard library to support iOS (GH-117052) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Malcolm Smith <smith@chaquo.com> Co-authored-by: Ned Deily <nad@python.org> commit b44898299a2ed97045c270f6474785da2ff07ced Author: Tim Hatch <tim@timhatch.com> Date: Wed Mar 27 23:54:51 2024 -0700 gh-89739: gh-77140: Support zip64 in zipimport (GH-94146) * Reads zip64 files as produced by the zipfile module * Include tests (somewhat slow, however, because of the need to create "large" zips) * About the same amount of strictness reading invalid zip files as zipfile has * Still works on files with prepended data (like pex) There are a lot more test cases at https://github.com/thatch/zipimport64/ that give me confidence that this works for real-world files. Fixes #89739 and #77140. --------- Co-authored-by: Itamar Ostricher <itamarost@gmail.com> Reviewed-by: Gregory P. Smith <greg@krypto.org> commit 2cedd25c14d3acfdcb5e8ee55132ce3e334ab8fe Author: Illia Volochii <illia.volochii@gmail.com> Date: Thu Mar 28 08:46:01 2024 +0200 Revert "gh-116886: Temporarily disable CIfuzz (memory) (GH-117018)" (GH-117289) This reverts commit 1ab0d0b1167d78bf19661a3b5e533a2b68a57604. This reverts #117018. I expect the issue to be fixed based on https://github.com/google/oss-fuzz/pull/11708#issuecomment-2006442396 and https://github.com/actions/runner-images/issues/9491. commit eefff682f09394fe4f18b7d7c6ac4c635caadd02 Author: Malcolm Smith <smith@chaquo.com> Date: Wed Mar 27 22:11:44 2024 +0000 gh-108277: Make test_os tolerate 10 ms diff for timerfd on Android emulators (#117223) commit 7aa89bc43e0bcf49eee5a39b5a7ba8f996f20d00 Author: Victor Stinner <vstinner@python.org> Date: Wed Mar 27 23:10:14 2024 +0100 gh-113317: Change how Argument Clinic lists converters (#116853) * Add a new create_parser_namespace() function for PythonParser to pass objects to executed code. * In run_clinic(), list converters using 'converters' and 'return_converters' dictionarties. * test_clinic: add 'object()' return converter. * Use also create_parser_namespace() in eval_ast_expr(). Co-authored-by: Erlend E. Aasland <erlend@python.org> commit 669ef49c7d42f35da6f7ee280102353b9b37f83e Author: Seth Michael Larson <seth@python.org> Date: Wed Mar 27 16:56:14 2024 -0500 gh-99108: Update and check HACL* version information (GH-117295) * Update and check HACL* version information commit 262fb911ab7df8e890ebd0efb0773c3e0b5a757f Author: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Wed Mar 27 17:38:19 2024 +0000 gh-117288: Allocate fewer label IDs in _PyCfg_ToInstructionSequence (#117290) commit 74c8568d07719529b874897598d8b3bc25ff0434 Author: Malcolm Smith <smith@chaquo.com> Date: Wed Mar 27 16:53:27 2024 +0000 gh-71042: Add `platform.android_ver` (#116674) commit ce00de4c8cd39816f992e749c1074487d93abe9d Author: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Date: Wed Mar 27 16:46:35 2024 +0200 gh-117225: doctest: only print "and X failed" when non-zero, don't pluralise "1 items" (#117228) commit 92397d5ead38dde4154e70d00f24973bcf2a925a Author: Raymond Hettinger <rhettinger@users.noreply.github.com> Date: Wed Mar 27 09:04:32 2024 -0500 Add statistics recipe for sampling from an estimated probability density distribution (#117221) commit b3e8c78ed7aa9bbd1084375587b99200c687cec9 Author: Tian Gao <gaogaotiantian@hotmail.com> Date: Tue Mar 26 18:20:12 2024 -0700 gh-113548: Allow CLI arguments to `pdb -m` (#113557) commit 48c0b05cf0dd2db275bd4653f84aa36c22bddcd2 Author: Adorilson Bezerra <adorilson@gmail.com> Date: Tue Mar 26 19:08:08 2024 +0000 Change links on the index page (#117230) commit af1b0e94400d1bf732466d675054df8cf7dfb62d Author: AN Long <aisk@users.noreply.github.com> Date: Wed Mar 27 02:26:48 2024 +0800 gh-104242: Enable test_is_char_device_true in pathlib test on all platform (GH-116983) commit 79be75735c9d77972112cecc8d7e1af28c176ed0 Author: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Tue Mar 26 15:18:17 2024 +0000 gh-115775: Compiler adds __static_attributes__ field to classes (#115913) commit 70969d53a77a8a190c40a30419e772bc874a4f62 Author: Antonio <57417288+amaddio@users.noreply.github.com> Date: Tue Mar 26 15:10:29 2024 +0100 gh-97901 add missing text/rtf to mimetypes (GH-97902) Co-authored-by: Noam Cohen <noam@noam.me> commit 4ec347760f98b156c6a2d42ca397af6b0b6ecc50 Author: AN Long <aisk@users.noreply.github.com> Date: Tue Mar 26 22:09:57 2024 +0800 gh-115538: Use isolate mode when running venv test_multiprocessing_recursion() (#117116) Co-authored-by: Victor Stinner <vstinner@python.org> commit 743f2c68f478279e1e56577fe95a0ed112b9abc5 Author: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Date: Tue Mar 26 16:09:09 2024 +0200 pre-commit: add `check-case-conflict` and `check-merge-conflict` (#117259) commit 4abca7e1e7e2764faf20c7e677ea5c9ea9dbffe2 Author: Paulo Neves <ptsneves@users.noreply.github.com> Date: Tue Mar 26 13:37:50 2024 +0100 gh-98966: Handle stdout=subprocess.STDOUT (GH-98967) Explicitly handle the case where stdout=STDOUT as otherwise the existing error handling gets confused and reports hard to understand errors. Signed-off-by: Paulo Neves <ptsneves@gmail.com> commit 9654daf793b534b44a831c80f43505ab9e380f1f Author: Serhiy Storchaka <storchaka@gmail.com> Date: Tue Mar 26 13:26:45 2024 +0200 gh-66543: Fix mimetype.guess_type() (GH-117217) Fix parsing of the following corner cases: * URLs with only a host name * URLs containing a fragment * URLs containing a query * filenames with only a UNC sharepoint on Windows Co-authored-by: Dong-hee Na <donghee.na92@gmail.com> commit 8bef34f625e21886b1c64544c060e19ee2e229bf Author: Mark Shannon <mark@hotpy.org> Date: Tue Mar 26 11:11:42 2024 +0000 GH-117108: Set the "old space bit" to "visited" for all young objects (#117213) Change old space bit of young objects from 0 to gcstate->visited_space. This ensures that any object created *and* collected during cycle GC has the bit set correctly. commit bf82f77957a31c3731b4ec470c406f5708ca9ba3 Author: Mark Shannon <mark@hotpy.org> Date: Tue Mar 26 09:35:11 2024 +0000 GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits. commit 61599a48f52e951d8813877ee311d2a830ba2cd8 Author: Pablo Galindo Salgado <Pablogsal@gmail.com> Date: Tue Mar 26 09:30:46 2024 +0000 bpo-24612: Improve syntax error for 'not' after an operator (GH-28170) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> commit 771902c257372e6c4df1ead4e8c46308561db7a7 Author: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Date: Tue Mar 26 11:13:32 2024 +0200 gh-83845: Add tests for operator module (#115883) Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> commit ea9a296fce2f786b4cf43c7924e5de01061f27ca Author: yevgeny hong <hongyevgeny@gmail.com> Date: Tue Mar 26 16:45:43 2024 +0900 gh-115627: Fix PySSL_SetError handling SSL_ERROR_SYSCALL (GH-115628) Python 3.10 changed from using SSL_write() and SSL_read() to SSL_write_ex() and SSL_read_ex(), but did not update handling of the return value. Change error handling so that the return value is not examined. OSError (not EOF) is now returned when retval is 0. According to *recent* man pages of all functions for which we call PySSL_SetError, (in OpenSSL 3.0 and 1.1.1), their return value should be used to determine whether an error happened (i.e. if PySSL_SetError should be called), but not what kind of error happened (so, PySSL_SetError shouldn't need retval). To get the error, we need to use SSL_get_error. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Petr Viktorin <encukou@gmail.com> commit d52bdfb19fadd7614a0e5abaf68525fc7300e841 Author: Victor Stinner <vstinner@python.org> Date: Tue Mar 26 08:35:59 2024 +0100 gh-83434: Disable XML in regrtest when -R option is used (#117232) commit 9f74e86c78853c101a23e938f8e32ea838d8f62e Author: Sebastian Pipping <sebastian@pipping.org> Date: Tue Mar 26 02:48:27 2024 +0100 gh-117187: Fix XML tests for vanilla Expat <2.6.0 (GH-117203) This fixes XML unittest fallout from the https://github.com/python/cpython/issues/115398 security fix. When configured using `--with-system-expat` on systems with older pre 2.6.0 versions of libexpat, our unittests were failing. * sax|etree: Simplify Expat version guard where simplifiable Idea by Matěj Cepl * sax|etree: Fix reparse deferral tests for vanilla Expat <2.6.0 This *does not fix* the case of distros with an older version of libexpat with the 2.6.0 feature backported as a security fix. (Ubuntu is a known example of this with its libexpat1 2.5.0-2ubunutu0.1 package) commit 872e212378ef86392069034afd80bb53896fd93d Author: Jonathan Protzenko <protz@microsoft.com> Date: Mon Mar 25 17:35:26 2024 -0700 gh-99108: Refresh HACL*; update modules accordingly; fix namespacing (GH-117237) Pulls in a new update from https://github.com/hacl-star/hacl-star and fixes our C "namespacing" done by `Modules/_hacl/refresh.sh`. commit 8945b7ff55b87d11c747af2dad0e3e4d631e62d6 Author: Eric V. Smith <ericvsmith@users.noreply.github.com> Date: Mon Mar 25 19:59:14 2024 -0400 gh-109870: Dataclasses: batch up exec calls (gh-110851) Instead of calling `exec()` once for each function added to a dataclass, only call `exec()` once per dataclass. This can lead to speed improvements of up to 20%. commit 7ebad77ad65ab4d5d8d0c333256a882262cec189 Author: Raymond Hettinger <rhettinger@users.noreply.github.com> Date: Mon Mar 25 18:49:44 2024 -0500 Sync main docs and docstring for median_grouped(). (gh-117214) commit 0821923aa979a72464c5da8dfa53a719bba5801c Author: Nice Zombies <nineteendo19d0@gmail.com> Date: Mon Mar 25 23:55:11 2024 +0100 gh-117114: Make os.path.isdevdrive available on all platforms (GH-117115) commit c2276176d543a2fc2d57709c2787f99850fbb073 Author: Adorilson Bezerra <adorilson@gmail.com> Date: Mon Mar 25 22:34:20 2024 +0000 Add information about negative indexes to sequence datamodel doc (#110903) Co-authored by Terry Jan Reedy commit 23e4f80ce2a2bac50acd1785e791316d5b578b8d Author: Mark Shannon <mark@hotpy.org> Date: Mon Mar 25 20:43:51 2024 +0000 A few minor tweaks to get stats working and compiling cleanly. (#117219) Fixes a compilation error when configured with `--enable-pystats`, an array size issue, and an unused variable. commit 507896d97dcff2d7999efa264b29d9003c525c49 Author: Victor Stinner <vstinner@python.org> Date: Mon Mar 25 17:32:20 2024 +0100 gh-116936: Add PyType_GetModuleByDef() to the limited C API (#116937) commit 0c1a42cf9c8cd0d4534d5c1d58f118ce7c5c446e Author: Serhiy Storchaka <storchaka@gmail.com> Date: Mon Mar 25 17:32:11 2024 +0200 gh-87193: Support bytes objects with refcount > 1 in _PyBytes_Resize() (GH-117160) Create a new bytes object and destroy the old one if it has refcount > 1. commit 01e7405da400e8997f8964d06cc414045e144681 Author: Tian Gao <gaogaotiantian@hotmail.com> Date: Mon Mar 25 08:18:09 2024 -0700 gh-112948: Make pdb completion similar to repl completion (#112950) commit 9db2a8f914ad59019d448cecc43b6d45f46424a0 Author: Raymond Hettinger <rhettinger@users.noreply.github.com> Date: Mon Mar 25 09:26:42 2024 -0500 Minor markup and grammar fixes in the statistics docs (gh-117216) commit eebea7e515462b503632ada74923ec3246599c9c Author: Kirill Podoprigora <kirill.bast9@mail.ru> Date: Sun Mar 24 20:34:55 2024 +0200 gh-117176: Fix compiler warning in Python/optimizer_bytecodes.c (GH-117199) commit 83485a095363dad6c97b19af2826ca0c34343bfc Author: Totally a booplicate <53382877+Booplicate@users.noreply.github.com> Date: Sun Mar 24 18:48:40 2024 +0300 gh-112571: Move fish venv activation script into the common folder (GH-117169) pythongh-112571: allow using fish venv activation script on windows The fish shell can be used on windows under cygwin or msys2. This change moves the script to the common folder so the venv module will install it on both posix and nt systems (like the bash script). commit 78a651fd7fbe7a3d1702e40f4cbfa72d87241ef0 Author: Terry Jan Reedy <tjreedy@udel.edu> Date: Sun Mar 24 11:38:34 2024 -0400 gh-117194: Properly format 'base64' header in What's New (#117198) It needs 6, not 3, '-'s. commit f267d5bf2a99fbeb26a720d1c87c1f0557424b14 Author: Kerim Kabirov <the.privat33r+gh@pm.me> Date: Sun Mar 24 14:59:14 2024 +0100 GH-115986 Docs: promote pprint.pp usage as a default (#116614) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> commit 39df7732178c8e8f75b12f069a3dbc1715c99995 Author: LilKS <1244886+LilKS@users.noreply.github.com> Date: Sun Mar 24 11:01:07 2024 +0100 gh-101760: Improve the imaplib.IMAP4 example (#101764) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> commit a1e948edba9ec6ba61365429857f7a087c5edf51 Author: Raymond Hettinger <rhettinger@users.noreply.github.com> Date: Sun Mar 24 11:35:58 2024 +0200 Add cumulative option for the new statistics.kde() function. (#117033) commit d610d821fd210dce63a1132c274ffdf8acc510bc Author: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Sat Mar 23 22:32:33 2024 +0000 gh-112383: teach dis how to interpret ENTER_EXECUTOR (#117171) commit 6c83352bfe78a7d567c8d76257df6eb91d5a7245 Author: Ken Jin <kenjin@python.org> Date: Sun Mar 24 06:19:17 2024 +0800 gh-117180: Complete call sequence when trace stack overflow (GH-117184) --------- Co-authored-by: Peter Lazorchak <lazorchakp@gmail.com> Co-authored-by: Guido van Rossum <gvanrossum@users.noreply.github.com> Co-authored-by: Guido van Rossum <gvanrossum@gmail.com> commit f11d0d8be8af28e1368c3c7c116218cf65ddf93e Author: Erik Soma <stillusingirc@gmail.com> Date: Sat Mar 23 11:39:35 2024 -0400 gh-91227: Ignore ERROR_PORT_UNREACHABLE in proactor recvfrom() (#32011) commit 9967b568edd2e35b0415c14c7242f3ca2c0dc03d Author: Victor Stinner <vstinner@python.org> Date: Sat Mar 23 13:01:20 2024 +0100 gh-117008: Fix functools test_recursive_pickle() (#117009) Use support.infinite_recursion() in test_recursive_pickle() of test_functools to prevent a stack overflow on "ARM64 Windows Non-Debug" buildbot. Lower Py_C_RECURSION_LIMIT to 1,000 frames on Windows ARM64. commit 72eea512b88f8fd68b7258242c37da963ad87360 Author: Barney Gale <barney.gale@gmail.com> Date: Fri Mar 22 19:14:09 2024 +0000 GH-106747: Document another difference between `glob` and `pathlib`. (#116518) Document that `path.glob()` might return *path*, whereas `glob.glob(root_dir=path)` will never return an empty string corresponding to *path*. commit e28477f214276db941e715eebc8cdfb96c1207d9 Author: Mark Shannon <mark@hotpy.org> Date: Fri Mar 22 18:43:25 2024 +0000 GH-117108: Change the size of the GC increment to about 1% of the total heap size. (GH-117120) commit e2e0b4b4b92694ba894e02b4a66fd87c166ed10f Author: Serhiy Storchaka <storchaka@gmail.com> Date: Fri Mar 22 20:19:10 2024 +0200 gh-113024: C API: Add PyObject_GenericHash() function (GH-113025) commit 567ab3bd15398c8c7b791f3e376ae3e3c0bbe079 Author: Serhiy Storchaka <storchaka@gmail.com> Date: Fri Mar 22 20:08:00 2024 +0200 gh-117084: Fix ZIP file extraction for directory entry names with backslashes on Windows (GH-117129) commit 5a78f6e798d5c2af1dba2df6c9f1f1e5aac02a86 Author: Serhiy Storchaka <storchaka@gmail.com> Date: Fri Mar 22 20:03:48 2024 +0200 gh-117134: Microoptimize glob() for include_hidden=True (GH-117135) commit 00baaa21de229a6db80ff2b84c2fd6ad1999a24c Author: Vinay Sajip <vinay_sajip@yahoo.co.uk> Date: Fri Mar 22 17:25:51 2024 +0000 [docs] Fix typo in docstring and add example to logging cookbook. (GH-117157) commit 40d75c2b7f5c67e254d0a025e0f2e2c7ada7f69f Author: Jakub Stasiak <jakub@stasiak.at> Date: Fri Mar 22 17:49:56 2024 +0100 GH-113171: Fix "private" (non-global) IP address ranges (GH-113179) * GH-113171: Fix "private" (really non-global) IP address ranges The _private_networks variables, used by various is_private implementations, were missing some ranges and at the same time had overly strict ranges (where there are more specific ranges considered globally reachable by the IANA registries). This patch updates the ranges with what was missing or otherwise incorrect. I left 100.64.0.0/10 alone, for now, as it's been made special in [1] and I'm not sure if we want to undo that as I don't quite understand the motivation behind it. The _address_exclude_many() call returns 8 networks for IPv4, 121 networks for IPv6. [1] https://github.com/python/cpython/issues/61602 commit 3be9b9d8722696b95555937bb211dc4cda714d56 Author: Steve Dower <steve.dower@python.org> Date: Fri Mar 22 15:00:50 2024 +0000 Fix get_packagefamilyname helper function on Windows 32-bit (GH-117153) commit 63d6f2623ef2aa90f51c6a928b96845b9b380d89 Author: NGRsoftlab <78017794+NGRsoftlab@users.noreply.github.com> Date: Fri Mar 22 14:25:38 2024 +0300 gh-117068: Remove useless code in bytesio.c:resize_buffer() (GH-117069) Co-authored-by: i.khabibulin <i.khabibulin@ngrsoftlab.ru> commit 42ae924d278c48a719fb0ab86357f3235a9f7ab9 Author: Petr Viktorin <encukou@gmail.com> Date: Fri Mar 22 10:42:18 2024 +0100 gh-117127: glob tests: Reopen dir_fd to pick up directory changes (GH-117128) commit 8383915031942f441f435a5ae800790116047b80 Author: Tim Peters <tim.peters@gmail.com> Date: Thu Mar 21 22:27:25 2024 -0500 GH-116939: Rewrite binarysort() (#116940) Rewrote binarysort() for clarity. Also changed the signature to be more coherent (it was mixing sortslice with raw pointers). No change in method or functionality. However, I left some experiments in, disabled for now via `#if` tricks. Since this code was first written, some kinds of comparisons have gotten enormously faster (like for lists of floats), which changes the tradeoffs. For example, plain insertion sort's simpler innermost loop and highly predictable branches leave it very competitive (even beating, by a bit) binary insertion when comparisons are very cheap, despite that it can do many more compares. And it wins big on runs that are already sorted (moving the next one in takes only 1 compare then). So I left code for a plain insertion sort, to make future experimenting easier. Also made the maximum value of minrun a `#define` (``MAX_MINRUN`) to make experimenting with that easier too. And another bit of `#if``-disabled code rewrites binary insertion's innermost loop to remove its unpredictable branch. Surprisingly, this doesn't really seem to help overall. I'm unclear on why not. It certainly adds more instructions, but they're very simple, and it's hard to be believe they cost as much as a branch miss. commit 97ba910e47ad298114800587979ce7beb0a705a3 Author: Guido van Rossum <guido@python.org> Date: Thu Mar 21 18:27:48 2024 -0700 gh-108716:: Remove _PyStaticCode_Init/Fini (#117141) More deepfreeze cleanup. commit b3d25df8d38b79310587da54dbd88b06a16d4904 Author: Eric Snow <ericsnowcurrently@gmail.com> Date: Thu Mar 21 18:20:20 2024 -0600 gh-105716: Fix _PyInterpreterState_IsRunningMain() For Embedders (gh-117140) When I added _PyInterpreterState_IsRunningMain() and friends last year, I tried to accommodate applications that embed Python but don't call _PyInterpreterState_SetRunningMain() (not that they're expected to). That mostly worked fine until my recent changes in gh-117049, where the subtleties with the fallback code led to failures; the change ended up breaking test_tools.test_freeze, which exercises a basic embedding situation. The simplest fix is to drop the fallback code I originally added to _PyInterpreterState_IsRunningMain() (and later to _PyThreadState_IsRunningMain()). I've kept the fallback in the _xxsubinterpreters module though. I've also updated Py_FrozenMain() to call _PyInterpreterState_SetRunningMain(). commit c4bf58a14f162557038a1535ca22c52b49d81d7b Author: Thomas A Caswell <tcaswell@gmail.com> Date: Thu Mar 21 19:54:50 2024 -0400 gh-116745: Remove all internal usage of @LIBPYTHON@ (#116746) Replace with MODULE_LDFLAGS. commit 3ec57307e70ee6f42410e844d3399bbd598917ba Author: Malcolm Smith <smith@chaquo.com> Date: Thu Mar 21 23:52:29 2024 +0000 gh-71052: Add Android build script and instructions (#116426) commit 50f9b0b1e0fb181875751cef951351ed007b6397 Author: Victor Stinner <vstinner@python.org> Date: Thu Mar 21 23:17:09 2024 +0100 gh-117061: Fix test_posix.test_sched_setaffinity() on RHEL9 (#117126) On RHEL9, sched_setaffinity(0, []) does not fail. commit 0907871d43bffb613cbd560224e1a9db13d06c06 Author: Ned Batchelder <ned@nedbatchelder.com> Date: Thu Mar 21 15:47:09 2024 -0400 docs: fix over-linking in dataclasses.rst (#117005) commit 570a82d46abfebb9976961113fb0f8bb400ad182 Author: Guido van Rossum <guido@python.org> Date: Thu Mar 21 12:37:41 2024 -0700 gh-117045: Add code object to function version cache (#117028) Changes to the function version cache: - In addition to the function object, also store the code object, and allow the latter to be retrieved even if the function has been evicted. - Stop assigning new function versions after a critical attribute (e.g. `__code__`) has been modified; the version is permanently reset to zero in this case. - Changes to `__annotations__` are no longer considered critical. (This fixes gh-109998.) Changes to the Tier 2 optimization machinery: - If we cannot map a function version to a function, but it is still mapped to a code object, we continue projecting the trace. The operand of the `_PUSH_FRAME` and `_POP_FRAME` opcodes can be either NULL, a function object, or a code object with the lowest bit set. This allows us to trace through code that calls an ephemeral function, i.e., a function that may not be alive when we are constructing the executor, e.g. a generator expression or certain nested functions. We will lose globals removal inside such functions, but we can still do other peephole operations (and even possibly [call inlining](https://github.com/python/cpython/pull/116290), if we decide to do it), which only need the code object. As before, if we cannot retrieve the code object from the cache, we stop projecting. commit c85d84166a84a5cb2d724012726bad34229ad24e Author: Will Childs-Klein <willck93@gmail.com> Date: Thu Mar 21 14:16:36 2024 -0500 gh-116333: Relax error string text expectations in SSL-related tests (GH-116334) * Relax error string text expectations in SSL-related tests As suggested [here][1], this change relaxes the OpenSSL error string text expectations in a number of tests. This was specifically done in support of more easily building CPython [AWS-LC][2], but because AWS-LC is a fork of [BoringSSL][3], it should increase compatibility with that library as well. In addition to the error string relaxations, we also add some guards around the `tls-unique` channel binding being used with TLSv1.3, as that feature (described in [RFC 6929][4]) is [not defined][5] for TLSv1.3. [1]: https://discuss.python.org/t/support-building-ssl-and-hashlib-modules-against-aws-lc/44505/4 [2]: https://github.com/aws/aws-lc [3]: https://github.com/google/boringssl [4]: https://datatracker.ietf.org/doc/html/rfc5929#section-3 [5]: https://datatracker.ietf.org/doc/html/rfc8446#appendix-C.5 commit 1f72fb5447ef3f8892b4a7a6213522579c618e8e Author: Sam Gross <colesbury@gmail.com> Date: Thu Mar 21 14:21:02 2024 -0400 gh-116522: Refactor `_PyThreadState_DeleteExcept` (#117131) Split `_PyThreadState_DeleteExcept` into two functions: - `_PyThreadState_RemoveExcept` removes all thread states other than one passed as an argument. It returns the removed thread states as a linked list. - `_PyThreadState_DeleteList` deletes those dead thread states. It may call destructors, so we want to "start the world" before calling `_PyThreadState_DeleteList` to avoid potential deadlocks. commit 50369e6c34d05222e5a0ec9443a9f7b230e83112 Author: Michael Droettboom <mdboom@gmail.com> Date: Thu Mar 21 13:27:46 2024 -0400 gh-116996: Add pystats about _Py_uop_analyse_and_optimize (GH-116997) commit 617158e07811edfd6fd552a3d84b0beedd8f1d18 Author: Eric Snow <ericsnowcurrently@gmail.com> Date: Thu Mar 21 11:15:02 2024 -0600 gh-76785: Drop PyInterpreterID_Type (gh-117101) I added it quite a while ago as a strategy for managing interpreter lifetimes relative to the PEP 554 (now 734) implementation. Relatively recently I refactored that implementation to no longer rely on InterpreterID objects. Thus now I'm removing it. commit abdd1f938f08e536864532b2071f144515ecc88b Author: Victor Stinner <vstinner@python.org> Date: Thu Mar 21 17:45:43 2024 +0100 gh-85283: Build _testconsole extension with limited C API (#117125) commit 8bea6c411d65cd987616b4ecdb86373e4f21f1c6 Author: Victor Stinner <vstinner@python.org> Date: Thu Mar 21 17:07:00 2024 +0100 gh-115754: Add Py_GetConstant() function (#116883) Add Py_GetConstant() and Py_GetConstantBorrowed() functions. In the limited C API version 3.13, getting Py_None, Py_False, Py_True, Py_Ellipsis and Py_NotImplemented singletons is now implemented as function calls at the stable ABI level to hide implementation details. Getting these constants still return borrowed references. Add _testlimitedcapi/object.c and test_capi/test_object.py to test Py_GetConstant() and Py_GetConstantBorrowed() functions. commit 5a76d1be8ef371b75ca65166726923c249b5f615 Author: Eric Snow <ericsnowcurrently@gmail.com> Date: Thu Mar 21 10:06:35 2024 -0600 gh-105716: Update interp->threads.main After Fork (gh-117049) I missed this in gh-109921. We also update Py_Exit() to call _PyInterpreterState_SetNotRunningMain(), if necessary. commit bbee57fa8c318cb26d6c8651254927a1972c9738 Author: Eric Snow <ericsnowcurrently@gmail.com> Date: Thu Mar 21 09:56:12 2024 -0600 gh-76785: Clean Up Interpreter ID Conversions (gh-117048) Mostly we unify the two different implementations of the conversion code (from PyObject * to int64_t. We also drop the PyArg_ParseTuple()-style converter function, as well as rename and move PyInterpreterID_LookUp(). commit e728303532168efab7694c55c82ea19b18bf8385 Author: Sam Gross <colesbury@gmail.com> Date: Thu Mar 21 10:01:16 2024 -0400 gh-116522: Stop the world before fork() and during shutdown (#116607) This changes the free-threaded build to perform a stop-the-world pause before deleting other thread states when forking and during shutdown. This fixes some crashes when using multiprocessing and during shutdown when running with `PYTHON_GIL=0`. This also changes `PyOS_BeforeFork` to acquire the runtime lock (i.e., `HEAD_LOCK(&_PyRuntime)`) before forking to ensure that data protected by the runtime lock (and not just the GIL or stop-the-world) is in a consistent state before forking. commit 1f8b24ef69896680d6ba6005e75e1cc79a744f9e Author: Malcolm Smith <smith@chaquo.com> Date: Thu Mar 21 13:20:57 2024 +0000 gh-71052: Implement `ctypes.util.find_library` on Android (GH-116379) commit d16c9d1278164f04778861814ebc87ed087511fc Author: Tian Gao <gaogaotiantian@hotmail.com> Date: Thu Mar 21 03:30:10 2024 -0700 gh-116987: Support class code objects in inspect.findsource() (GH-117025) commit 6547330f4e896c6748da23704b617e060e6cc68e Author: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Date: Thu Mar 21 03:49:10 2024 +0000 GH-109653: Defer import of ``importlib.metadata._adapters`` (#109829) * adapters * Add comments for deferred imports with links to rationale. * Add blurb --------- Co-authored-by: Jason R. Coombs <jaraco@jaraco.com> commit 667294d5b2ee812ebe0c9c1efd58e2006b61f827 Author: Jason R. Coombs <jaraco@jaraco.com> Date: Wed Mar 20 23:01:24 2024 -0400 gh-117089: Apply changes from importlib_metadata 7.1.0 (#117094) * Apply changes from importlib_metadata 7.1.0 * Include the data sources in the makefile (even though they're not needed) commit f4cc77d494ee0e10ed84ce369f0910c70a2f6d44 Author: Victor Stinner <vstinner@python.org> Date: Thu Mar 21 00:06:24 2024 +0100 gh-116869: Enable -Werror in test_cext for Free Threading (#117106) Check for warnings, but don't enable the compiler flag -Werror=declaration-after-statement. commit 104602a6078564765b7b8f42888f8eaa37b129b1 Author: Victor Stinner <vstinner@python.org> Date: Wed Mar 20 23:52:23 2024 +0100 gh-105927: Limit PyWeakref_GetRef() to limited C API 3.13 (#117091) commit 8ad88984200b2ccddc0a08229dd2f4c14d1a71fc Author: Jason R. Coombs <jaraco@jaraco.com> Date: Wed Mar 20 17:11:00 2024 -0400 gh-117089: Move importlib.metadata tests to their own package (#117092) * Ensure importlib.metadata tests do not leak references in sys.modules. * Move importlib.metadata tests to their own package for easier syncing with importlib_metadata. * Update owners and makefile for new directories. * Add blurb commit 7d446548ef53f6c3de1097c6d44cada6642ddc85 Author: Carol Willing <carolcode@willingconsulting.com> Date: Wed Mar 20 14:00:59 2024 -0700 Fix sort order for "locale encoding" glossary item (#115794) Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM> commit 63289b9dfbc7d87e81f1517422ee91b6b6d19531 Author: Mark Shannon <mark@hotpy.org> Date: Wed Mar 20 18:24:02 2024 +0000 GH-117066: Tier 2 optimizer: Don't throw away good traces if we can't optimize them perfectly. (GH-117067) commit dcaf33a41d5d220523d71c9b35bc08f5b8405dac Author: Petr Viktorin <encukou@gmail.com> Date: Wed Mar 20 17:33:08 2024 +0100 gh-114314: ctypes: remove stgdict and switch to heap types (GH-116458) Before this change, ctypes classes used a custom dict subclass, `StgDict`, as their `tp_dict`. This acts like a regular dict but also includes extra information about the type. This replaces stgdict by `StgInfo`, a C struct on the type, accessed by `PyObject_GetTypeData()` (PEP-697). All usage of `StgDict` (mainly variables named `stgdict`, `dict`, `edict` etc.) is converted to `StgInfo` (named `stginfo`, `info`, `einfo`, etc.). Where the dict is actually used for class attributes (as a regular PyDict), it's now called `attrdict`. This change -- not overriding `tp_dict` -- is made to make me comfortable with the next part of this PR: moving the initialization logic from `tp_new` to `tp_init`. The `StgInfo` is set up in `__init__` of each class, with a guard that prevents calling `__init__` more than once. Note that abstract classes (like `Array` or `Structure`) are created using `PyType_FromMetaclass` and do not have `__init__` called. Previously, this was done in `__new__`, which also wasn't called for abstract classes. Since `__init__` can be called from Python code or skipped, there is a tested guard to ensure `StgInfo` is initialized exactly once before it's used. Co-authored-by: neonene <53406459+neonene@users.noreply.github.com> Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com> commit 44fbab43d8f3f2df07091d237824cf4fa1f6c57c Author: Russell Keith-Magee <russell@keith-magee.com> Date: Wed Mar 20 23:32:56 2024 +0800 gh-117058: Update GUI and packaging recommendations for macOS. (#117059) commit 9221ef2d8cb7f4cf37592eb650d4c8f972033000 Author: Brett Simmers <swtaarrs@users.noreply.github.com> Date: Wed Mar 20 08:18:26 2024 -0700 gh-116908: Only write to `_pending_calls.calls_to_do` with atomic operations (#117044) These writes to `pending->calls_to_do` need to be atomic, because other threads can read (atomically) from `calls_to_do` without holding `pending->mutex`. commit fc4599800778f9b130d5e336deadbdeb5bd3e5ee Author: jkriegshauser <jkriegshauser@gmail.com> Date: Wed Mar 20 07:33:28 2024 -0700 gh-116773: Ensure overlapped objects on Windows are not deallocated too early by asyncio (GH-116774) commit 519b2ae22b54760475bbf62b9558d453c703f9c6 Author: Serhiy Storchaka <storchaka@gmail.com> Date: Wed Mar 20 15:39:53 2024 +0200 gh-117021: Fix integer overflow in PyLong_AsPid() on non-Windows 64-bit platforms (GH-117064)

The GC keeps track of the number of allocations (less deallocations) since the last GC. This buffers the count in thread-local state and uses atomic operations to modify the per-interpreter count. The thread-local buffering avoids contention on shared state. A consequence is that the GC scheduling is not as precise, so "test_sneaky_frame_object" is skipped because it requires that the GC be run exactly after allocating a frame object.

…alized (python#114732)

…thon#117370) The free-threaded GC sometimes sees objects with zero refcount. This can happen due to the delay in merging biased reference counting fields, and, in the future, due to deferred reference counting. We should not untrack these objects or they will never be collected. This fixes the refleaks in the free-threaded build.

colesbury added type-feature A feature request or enhancement 3.13 new features, bugs and security fixes topic-free-threading labels Nov 29, 2023

colesbury self-assigned this Nov 29, 2023

colesbury mentioned this issue Nov 29, 2023

PEP 703 -- Making the Global Interpreter Lock Optional in CPython #108219

Open

bedevere-app bot mentioned this issue Nov 29, 2023

gh-112529: Use atomic operations for gcstate->collecting #112533

Merged

colesbury added a commit to colesbury/cpython that referenced this issue Jan 5, 2024

Merge branch 'pythongh-113688' into pythongh-112529-gc

68af10c

bedevere-app bot mentioned this issue Jan 5, 2024

gh-112529: Track if debug allocator is used as underlying allocator #113747

Merged

colesbury added a commit to colesbury/cpython that referenced this issue Jan 5, 2024

pythongh-112529: Use GC mimalloc heaps for GC objects

4891c36

colesbury added a commit to colesbury/cpython that referenced this issue Jan 5, 2024

pythongh-112529: Use GC mimalloc heaps for GC objects

4b4ec14

colesbury added a commit to colesbury/cpython that referenced this issue Jan 16, 2024

pythongh-112529: Implement GC for free-threaded builds

8314c7c

bedevere-app bot mentioned this issue Jan 16, 2024

gh-112529: Use GC heaps for GC allocations in free-threaded builds #114157

Merged

colesbury added a commit to colesbury/cpython that referenced this issue Jan 18, 2024

pythongh-112529: Implement GC for free-threaded builds

c93a17c

colesbury added a commit to colesbury/cpython that referenced this issue Jan 18, 2024

pythongh-112529: Implement GC for free-threaded builds

358f2d1

colesbury added a commit to colesbury/cpython that referenced this issue Jan 18, 2024

pythongh-112529: Implement GC for free-threaded builds

7568efc

bedevere-app bot mentioned this issue Jan 18, 2024

gh-112529: Implement GC for free-threaded builds #114262

Merged

colesbury added a commit to colesbury/cpython that referenced this issue Jan 18, 2024

pythongh-112529: Implement GC for free-threaded builds

0a5d5bf

bedevere-app bot mentioned this issue Feb 1, 2024

gh-112529: Make the GC scheduling thread-safe #114880

Merged

colesbury added a commit to colesbury/cpython that referenced this issue Feb 6, 2024

Merge branch 'main' into pythongh-112529-gc-schedule

483b37e

bedevere-app bot mentioned this issue Feb 14, 2024

gh-112529: Use _PyThread_Id() in mimalloc (free-threaded build) #115488

Merged

colesbury added a commit to colesbury/cpython that referenced this issue Feb 14, 2024

Merge branch 'main' into pythongh-112529-gc-schedule

f99d14e

colesbury added a commit to colesbury/cpython that referenced this issue Feb 15, 2024

pythongh-112529: Revert "Use _PyThread_Id() in mimalloc in free-threa…

c780e77

…ded build" Reverts: python#115488 Reason: Free-threaded Ubuntu buildbots are timing out

bedevere-app bot mentioned this issue Feb 15, 2024

gh-112529: Revert "Use _PyThread_Id() in mimalloc in free-threaded build" #115524

Closed

colesbury added a commit that referenced this issue Feb 28, 2024

gh-112529: Simplify PyObject_GC_IsTracked and PyObject_GC_IsFinalized (…

df5212d

…#114732)

woodruffw pushed a commit to woodruffw-forks/cpython that referenced this issue Mar 4, 2024

pythongh-112529: Simplify PyObject_GC_IsTracked and PyObject_GC_IsFin…

d988f63

…alized (python#114732)

adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024

pythongh-112529: Simplify PyObject_GC_IsTracked and PyObject_GC_IsFin…

7c6c093

…alized (python#114732)

bedevere-app bot mentioned this issue Mar 29, 2024

gh-112529: Don't untrack tuples or dicts with zero refcount #117370

Merged

diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024

pythongh-112529: Simplify PyObject_GC_IsTracked and PyObject_GC_IsFin…

15c69a6

…alized (python#114732)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make the garbage collector thread-safe in `--disable-gil` builds #112529

Make the garbage collector thread-safe in `--disable-gil` builds #112529

colesbury commented Nov 29, 2023 •

edited by bedevere-app bot

Make the garbage collector thread-safe in --disable-gil builds #112529

Make the garbage collector thread-safe in --disable-gil builds #112529

Comments

colesbury commented Nov 29, 2023 • edited by bedevere-app bot

Feature or enhancement

Linked PRs

Make the garbage collector thread-safe in `--disable-gil` builds #112529

Make the garbage collector thread-safe in `--disable-gil` builds #112529

colesbury commented Nov 29, 2023 •

edited by bedevere-app bot