gh-108297: Update test_crashers

[vstinner]

• Rename Lib/test/crashers/ to Lib/test/test_crashers/.

• Move Lib/test/test_crashers.py to Lib/test/test_crashers/__init__.py.

• test_crashers is no longer skipped and makes sure that scripts do crash, and no simply fail with a non-zero exit code.

• Update bogus_code_obj.py to use CodeType.replace().

• Scripts crashing Python now uses SuppressCrashReport of test.support to not create coredump files.

• Remove Lib/test/crashers/ scripts which no longer crash:

  • recursive_call.py: fixed by Remove the usage of the C stack in Python to Python calls #89419
  • mutation_inside_cyclegc.py: fixed by Execute the garbage collector only on the eval breaker #97922
  • trace_at_recursion_limit.py: fixed by Python 3.7

• Issue: Update test_crashers #108297

[vstinner]

cc @serhiy-storchaka

[serhiy-storchaka]

I don't understand why they should be moved.

[vstinner]

I created #108303 to move test files to sub-directories: see the rationale there.

[vstinner]

test_crashers was skipped in 2011 by Antoine Pitrou: commit 89ba56d.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit ecb3fd6 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[vstinner]

Buildbot failures.

test_crashers failed on macOS: underlying_dict.py didn't crash.

Checking crasher: underlying_dict.py

AssertionError: Process return code is 0
(...)

stdout:
---
2
---

---

Unrelated: 1 test altered the execution environment: test_ssl. I created #108342 for this issue.

• AMD64 Windows10 PR
• AMD64 Windows11 Bigmem PR
• ARM64 Windows Non-Debug PR
• ARM64 Windows PR

---

Unrelated: aarch64 Fedora Stable Clang Installed PR: 1 test failed: test_venv

======================================================================
ERROR: test_zippath_from_non_installed_posix (test.test_venv.BasicTest.test_zippath_from_non_installed_posix)
Test that when create venv from non-installed python, the zip path
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.13/test/test_venv.py", line 612, in test_zippath_from_non_installed_posix
    subprocess.check_call(cmd, env=child_env)
  (...)
  File "/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.13/subprocess.py", line 1950, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpqzy4slzu/bin/python3'

======================================================================
FAIL: test_upgrade_dependencies (test.test_venv.BasicTest.test_upgrade_dependencies)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.13/test/test_venv.py", line 236, in test_upgrade_dependencies
    builder.upgrade_dependencies(fake_context)
  (...)
  File "/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.13/test/test_venv.py", line 222, in pip_cmd_checker
    self.assertEqual(
AssertionError: Lists differ: ['/tmp/tmpt86asn2c/bin/python3', '-m', 'pip', 'install', '--upgrade', 'pip'] != ['/tmp/tmpt86asn2c/bin/python3.13', '-m', 'pip', 'install', '--upgrade', 'pip']

First differing element 0:
'/tmp/tmpt86asn2c/bin/python3'
'/tmp/tmpt86asn2c/bin/python3.13'

- ['/tmp/tmpt86asn2c/bin/python3', '-m', 'pip', 'install', '--upgrade', 'pip']
+ ['/tmp/tmpt86asn2c/bin/python3.13', '-m', 'pip', 'install', '--upgrade', 'pip']
?                               +++

x86 Gentoo Installed with X PR: same error.

---

Unrelated: AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR

ERROR: test_hmac_sha1 (test.test_socket.LinuxKernelCryptoAPI.test_hmac_sha1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/pull_request.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_socket.py", line 6477, in test_hmac_sha1
    algo.setsockopt(socket.SOL_ALG, socket.ALG_SET_KEY, b"Jefe")
OSError: [Errno 22] Invalid argument
(...)
1 test failed: test_socket

---

Unrelated: PPC64 Fedora PR: https://buildbot.python.org/all/#/builders/256/builds/1206

checking for stdlib extension module _ssl... missing
(...)
Could not build the ssl module!
Python requires a OpenSSL 1.1.1 or newer
(...)
0:38:47 load avg: 2.92 [186/447/2] test_math crashed (Exit code 1) -- running: test_io (1 min 2 sec)
Timeout (0:15:00)!
Thread 0x00003fff7d985450 (most recent call first):
  File "/home/shager/cpython-buildarea/pull_request.edelsohn-fedora-ppc64/build/Lib/test/test_math.py", line 1367 in test_sumprod_stress
(...)
1:13:23 load avg: 1.21 Re-running test_math in verbose mode

Ran 0 tests in 0.000s

NO TESTS RAN
(...)
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
test_build_cpp03 (test.test_cppext.TestCPPExt.test_build_cpp03) ... ERROR
(...)
1 test failed: test_cppext
2 re-run tests: test_cppext test_math
1 test run no tests: test_math

[vstinner]

Buildbot failures.

---

Unrelated. aarch64 Fedora Stable Clang Installed PR: https://buildbot.python.org/all/#/builders/819/builds/750

1 test failed: test_gdb

Same error on aarch64 Fedora Stable Clang PR.

---

Unrelated. buildbot/aarch64 RHEL8 Refleaks PR: https://buildbot.python.org/all/#/builders/802/builds/975

0:02:32 load avg: 27.87 [420/447/1] test_ssl failed (env changed) (1 min 20 sec) -- running: (...)
beginning 6 repetitions
123456
Resource 'ipv6.google.com' is not available
.Resource 'ipv6.google.com' is not available
Warning -- threading_cleanup() failed to cleanup 0 threads (count: 0, dangling: 2)
Warning -- Dangling thread: <SingleConnectionTestServerThread(preauth_data_to_tls_server, stopped 281473271591360)>
Warning -- Dangling thread: <_MainThread(MainThread, started 281473314145680)>
.Resource 'ipv6.google.com' is not available
Warning -- threading_cleanup() failed to cleanup 0 threads (count: 0, dangling: 2)
Warning -- Dangling thread: <SingleConnectionTestServerThread(preauth_data_to_tls_server, stopped 281473271591360)>
Warning -- Dangling thread: <_MainThread(MainThread, started 281473314145680)>
.Resource 'ipv6.google.com' is not available
Warning -- threading_cleanup() failed to cleanup 0 threads (count: 0, dangling: 2)
Warning -- Dangling thread: <SingleConnectionTestServerThread(preauth_data_to_tls_server, stopped 281473271591360)>
Warning -- Dangling thread: <_MainThread(MainThread, started 281473314145680)>
.Resource 'ipv6.google.com' is not available
Warning -- threading_cleanup() failed to cleanup 0 threads (count: 0, dangling: 2)
Warning -- Dangling thread: <SingleConnectionTestServerThread(preauth_data_to_tls_server, stopped 281473271591360)>
Warning -- Dangling thread: <_MainThread(MainThread, started 281473314145680)>
.Resource 'ipv6.google.com' is not available
Warning -- threading_cleanup() failed to cleanup 0 threads (count: 0, dangling: 2)
Warning -- Dangling thread: <SingleConnectionTestServerThread(preauth_data_to_tls_server, stopped 281473271591360)>
Warning -- Dangling thread: <_MainThread(MainThread, started 281473314145680)>
.
(...)
1 test altered the execution environment:
    test_ssl

[vstinner]

Unrelated: s390x RHEL7 LTO + PGO PR: https://buildbot.python.org/all/#/builders/418/builds/1174

Could not build the ssl module!
Python requires a OpenSSL 1.1.1 or newer
(...)
cc1plus: error: command line option ‘-std=gnu11’ is valid for C/ObjC but not for C++ [-Werror]

That's the issue #105776: test_cpp and -std=c11.

• s390x RHEL7 LTO PR: Same error.
• s390x RHEL7 PR: Same error

[vstinner]

test_crashers failed on macOS: underlying_dict.py didn't crash.

That's a major issue :-( The test was skipped in 2011 because it was not reliable. Is it important that crashers scripts crash "in a reliable way"? Many trigger undefined behaviors which have... undefined behaviors :-( Building Python with UBSan would make these undefined behaviors to have a deterministic behavior, but UBSan is rarely used. Today, we only have ASAN GHA job and buildbots.

[serhiy-storchaka]

If the test is crashed in 50% of cases, repeating it twice will make it crashing in 75%, and repeating it 10 times will make it crashing in 99.9%. If attempts are independent.

[vstinner]

I created this PR to reenable test_crashers, but now I dislike this idea since crashes are not deterministic: sometimes scripts crash Python, sometimes not. I would prefer to remove test_crashers and keep Lib/crashers/ instead.

I close this PR.