test_decimal failed in pgo

[penguin-wwy]

Bug report

configure with --enable-optimizations and then make run_profile_task, fail occurred.

./python -m test --pgo --timeout=1200                                                                                                                                                                                                    
0:00:00 load avg: 3.02 Run tests sequentially (timeout: 20 min)
0:00:00 load avg: 3.02 [ 1/44] test_array
0:00:01 load avg: 3.02 [ 2/44] test_base64
0:00:02 load avg: 3.02 [ 3/44] test_binascii
0:00:02 load avg: 3.02 [ 4/44] test_binop
0:00:02 load avg: 3.02 [ 5/44] test_bisect
0:00:02 load avg: 3.02 [ 6/44] test_bytes
0:00:10 load avg: 2.71 [ 7/44] test_bz2
0:00:11 load avg: 2.71 [ 8/44] test_cmath
0:00:12 load avg: 2.71 [ 9/44] test_codecs
0:00:14 load avg: 2.65 [10/44] test_collections
0:00:17 load avg: 2.65 [11/44] test_complex
0:00:17 load avg: 2.65 [12/44] test_dataclasses
0:00:18 load avg: 2.65 [13/44] test_datetime
0:00:24 load avg: 2.44 [14/44] test_decimal
test test_decimal failed
0:00:32 load avg: 2.22 [15/44] test_difflib -- test_decimal failed (1 failure)
0:00:35 load avg: 2.12 [16/44] test_embed
0:00:35 load avg: 2.12 [17/44] test_float
......

Total duration: 2 min 13 sec
Tests result: FAILURE

After then I tried to reproduce the error.

I found make buildbottest TESTOPTS="-j4 -uall,-cpu test_decimal" success! The specific commands it executes are:

./python -u -W default -bb -E -m test -r -w -j 1 -u all -W --slowest --fail-env-changed --timeout=1200 -j4 -uall,-cpu test_decimal
...
======================================================================
FAIL: test_from_tuple (test.test_decimal.CWhitebox.test_from_tuple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/self/code/mine/Lib/test/test_decimal.py", line 5538, in test_from_tuple
    self.assertEqual(str(c.create_decimal(x)), '-0E-1000007')
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '-0E-1000026' != '-0E-1000007'
- -0E-1000026
?          ^^
+ -0E-1000007
?          ^^
......

1 test OK.

10 slowest tests:
- test_decimal: 11.3 sec

1 re-run test:
    test_decimal

Total duration: 11.7 sec
Tests result: FAILURE then SUCCESS

Because pgo mode does not have the -w parameter, it will not re-run after fail.

Finally, I reset commit and locate that gh-91291 causing the problem.

[penguin-wwy]

When I revert #32242, everything was fine.

[dignissimus]

On my own computer I can reproduce this without configuring with --enable-optimizations

sam@samtop ~/Documents/git/cpython (git)-[main] % ./python -m test test_decimal
0:00:00 load avg: 2.15 Run tests sequentially
0:00:00 load avg: 2.15 [1/1] test_decimal
test test_decimal failed -- Traceback (most recent call last):
  File "/home/sam/Documents/git/cpython/Lib/test/test_decimal.py", line 5538, in test_from_tuple
    self.assertEqual(str(c.create_decimal(x)), '-0E-1000007')
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '-0E-1000026' != '-0E-1000007'
- -0E-1000026
?          ^^
+ -0E-1000007
?          ^^


test_decimal failed (1 failure)

== Tests result: FAILURE ==

1 test failed:
    test_decimal

Total duration: 1.8 sec
Tests result: FAILURE
2 sam@samtop ~/Documents/git/cpython (git)-[main] %

[Fidget-Spinner]

I also get the same error on Windows (no PGO, debug build) when running test_decimal.

I'm curious as to why our CI is still passing.

[JelleZijlstra]

I can reproduce the test failure on MacOS, even in debug mode, and even with #32242 reverted. However, it reproduces only with ./python.exe -m unittest test.test_decimal.CWhitebox.test_from_tuple and passes with ./python.exe -m test test.test_decimal -m '*test_from_tuple*'. The difference is in the prec field of the decimal context: it must be 9 for the test to succeed, but in the unittest invocation it's 28. The test module's init() function sets it to 9, but apparently that sometimes gets lost.

[JelleZijlstra]

#77931 (PR #7355) fixed a previous similar problem with this test; cc @serhiy-storchaka.

[emaste]

Fails also on FreeBSD (no PGO)

[JelleZijlstra]

I merged #91825 which should make this test pass consistently.

As a more general solution, I think we should check the decimal global context in libregrtest.save_env, to make sure that it doesn't change between tests.