Open
Description
Bug report
Bug description:
We have had a number of issues reported over the years where a test included in the subset of tests run for PGO profiling fails. #135424 is a recent example of one such failure.
$ ./configure --enable-optimizations
$ make
[...]
# Next, run the profile task to generate the profile information.
LLVM_PROFILE_FILE="/Users/sysadmin/cpython/code-%p.profclangr" ./python.exe -m test --pgo --timeout=
Using random seed: 4213059023
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 1.95 Run 43 tests sequentially in a single process
0:00:00 load avg: 1.95 [ 1/43] test_array
[...]
0:00:23 load avg: 1.69 [40/43] test_tabnanny passed
0:00:23 load avg: 1.69 [41/43] test_time
test test_time failed
0:00:26 load avg: 1.69 [41/43] test_time failed (1 error)
0:00:26 load avg: 1.69 [42/43] test_xml_etree
0:00:26 load avg: 1.69 [42/43] test_xml_etree passed
0:00:26 load avg: 1.69 [43/43] test_xml_etree_c
0:00:26 load avg: 1.69 [43/43] test_xml_etree_c passed
Total duration: 26.6 sec
Total tests: run=9,234 skipped=194
Total test files: run=43/43 failed=1 skipped=1
Result: FAILURE
make: *** [profile-run-stamp] Error 2Unfortunately, this isn't enough information to know why a particular test failed and a downstream builder of Python may not know how to proceed to obtain that information. During a normal run of the test suite, regrtest will helpfully provide the details of why the test failed, i.e.:
$ ./configure
$ make
$ ./python.exe -m test test_time
Using random seed: 3687295663
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 0.74 Run 1 test sequentially in a single process
0:00:00 load avg: 0.74 [1/1] test_time
test test_time failed -- Traceback (most recent call last):
File "/Users/sysadmin/cpython/Lib/test/test_time.py", line 146, in test_clock_settime
time.clock_settime(time.CLOCK_REALTIME, t)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory
0:00:03 load avg: 0.74 [1/1/1] test_time failed (1 error)
== Tests result: FAILURE ==
1 test failed:
test_time
Total duration: 3.1 sec
Total tests: run=63 skipped=2
Total test files: run=1/1 failed=1
Result: FAILURE
The PGO test run step should provide the same error traceback info on failures.
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Linked PRs
- gh-135489: Allow verbose output for PGO runs in _runtest function #135512
- [3.14] gh-135489: Show verbose output for failing tests during PGO profiling step with
--enable-optimizations(GH-135512) #135599 - [3.13] gh-135489: Show verbose output for failing tests during PGO profiling step with
--enable-optimizations(GH-135512) #135600