Improve C++ test infrastructure: progress reporter, timeouts, and skip hanging Move Subinterpreter test #5942
+184
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit improves the C++ test infrastructure to ensure test output is visible in CI logs, and disables a test that hangs on free-threaded Python 3.14+. Changes: ## CI/test infrastructure improvements - .github/workflows: Added `timeout-minutes: 3` to all C++ test steps to prevent indefinite hangs. - tests/**/CMakeLists.txt: Added `USES_TERMINAL` to C++ test targets (cpptest, test_cross_module_rtti, test_pure_cpp) to ensure output is shown immediately rather than buffered and possibly lost on crash/timeout. - tests/test_with_catch/catch.cpp: Added a custom Catch2 progress reporter with timestamps, Python version info, and a SIGTERM handler to make test execution and failures clearly visible in CI logs. ## Disabled hanging test - The "Move Subinterpreter" test is disabled on free-threaded Python 3.14+ due to a hang in Py_EndInterpreter() when the subinterpreter is destroyed from a different thread than it was created on. Work on fixing the underlying issue will continue under PR pybind#5940. Context: We were in the dark for months (since we started testing with Python 3.14t) because CI logs gave no clue about the root cause of hangs. This led to ignoring intermittent hangs (mostly on macOS). Our hand was forced only with the Python 3.14.1 release, when hangs became predictable on all platforms. For the full development history of these changes, see PR pybind#5933.
Print the total number of test cases and assertions at the end of the test run, making it easy to spot if tests are disabled or added. Example output: [ PASSED ] 20 test cases, 1589 assertions.
Catch2 v2 doesn't have native skip support (v3 does with SKIP()). This macro allows tests to be skipped with a visible message while still appearing in the test list. Use this for the Move Subinterpreter test on free-threaded Python 3.14+ so it shows as skipped rather than being conditionally compiled out. Example output: [ RUN ] Move Subinterpreter [ SKIPPED ] Skipped on free-threaded Python 3.14+ (see PR pybind#5940) [ OK ] Move Subinterpreter
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR improves the C++ test infrastructure to provide better visibility into test execution and prevent CI hangs from going undiagnosed.
Changes
Test timeouts:
timeout-minutes: 3to allcppteststeps in CI workflows (18 total)Unbuffered output:
USES_TERMINALtoadd_custom_targetcommands in CMakeLists.txt files to prevent ninja from buffering test output (critical for diagnosing hangs)Progress reporter (
catch.cpp):ProgressReporterthat prints[ RUN ],[ OK ],[ FAILED ]for each test[ STARTING ]and[ DONE ]timestamps with UTC timeSIGTERMsignal handler to print[ SIGNAL ]when tests are killed by timeoutSkip macro:
PYBIND11_CATCH2_SKIP_IFmacro for runtime test skipping with visible output (Catch2 v2 doesn't have native skip support)Example output:
Background
We were in the dark for months about which test was hanging on free-threaded Python 3.14 because:
This PR ensures that future hangs will be immediately diagnosable from CI logs.
Related
gil_safe_call_once_and_store#5933