Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SOURCE_DATE_EPOCH: AssertionError: 'Using random seed ([0-9]+)' not found in 'Using random seed None...' #110932

Closed
hroncok opened this issue Oct 16, 2023 · 5 comments
Labels
tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@hroncok
Copy link
Contributor

hroncok commented Oct 16, 2023

Bug report

Bug description:

After #110168 when the test suite is run with $SOURCE_DATE_EPOCH set, test_regrtest fails.

Traceback:
$ SOURCE_DATE_EPOCH=1649999999 ./python -m test.regrtest  -v test_regrtest
== CPython 3.13.0a1+ (heads/main:a1ac5590e0f, Oct 16 2023, 16:57:59) [GCC 12.3.1 20230508 (Red Hat 12.3.1-1)]
== Linux-6.3.5-100.fc37.x86_64-x86_64-with-glibc2.36 little-endian
== Python build: release
== cwd: .../cpython/build/test_python_worker_3521824æ
== CPU count: 8
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed None
0:00:00 load avg: 6.56 Run 1 test sequentially
0:00:00 load avg: 6.56 [1/1] test_regrtest
test_add_python_opts (test.test_regrtest.ArgsTestCase.test_add_python_opts) ... ok
test_cleanup (test.test_regrtest.ArgsTestCase.test_cleanup) ... ok
test_coverage (test.test_regrtest.ArgsTestCase.test_coverage) ... ok
test_crashed (test.test_regrtest.ArgsTestCase.test_crashed) ... ok
test_doctest (test.test_regrtest.ArgsTestCase.test_doctest) ... ok
test_env_changed (test.test_regrtest.ArgsTestCase.test_env_changed) ... ok
test_failing_test (test.test_regrtest.ArgsTestCase.test_failing_test) ... ok
test_forever (test.test_regrtest.ArgsTestCase.test_forever) ... ok
test_fromfile (test.test_regrtest.ArgsTestCase.test_fromfile) ... ok
test_huntrleaks (test.test_regrtest.ArgsTestCase.test_huntrleaks) ... skipped 'need a debug build'
test_huntrleaks_fd_leak (test.test_regrtest.ArgsTestCase.test_huntrleaks_fd_leak) ... skipped 'need a debug build'
test_huntrleaks_mp (test.test_regrtest.ArgsTestCase.test_huntrleaks_mp) ... skipped 'need a debug build'
test_ignorefile (test.test_regrtest.ArgsTestCase.test_ignorefile) ... ok
test_interrupted (test.test_regrtest.ArgsTestCase.test_interrupted) ... ok
test_leak_tmp_file (test.test_regrtest.ArgsTestCase.test_leak_tmp_file) ... ok
test_list_cases (test.test_regrtest.ArgsTestCase.test_list_cases) ... ok
test_list_tests (test.test_regrtest.ArgsTestCase.test_list_tests) ... ok
test_matchfile (test.test_regrtest.ArgsTestCase.test_matchfile) ... ok
test_multiprocessing_timeout (test.test_regrtest.ArgsTestCase.test_multiprocessing_timeout) ... ok
test_no_test_ran_some_test_exist_some_not (test.test_regrtest.ArgsTestCase.test_no_test_ran_some_test_exist_some_not) ... ok
test_no_tests_ran (test.test_regrtest.ArgsTestCase.test_no_tests_ran) ... ok
test_no_tests_ran_multiple_tests_nonexistent (test.test_regrtest.ArgsTestCase.test_no_tests_ran_multiple_tests_nonexistent) ... ok
test_no_tests_ran_skip (test.test_regrtest.ArgsTestCase.test_no_tests_ran_skip) ... ok
test_print_warning (test.test_regrtest.ArgsTestCase.test_print_warning) ... ok
test_python_command (test.test_regrtest.ArgsTestCase.test_python_command) ... ok
test_random (test.test_regrtest.ArgsTestCase.test_random) ... FAIL
test_random_seed (test.test_regrtest.ArgsTestCase.test_random_seed) ... FAIL
test_random_seed_workers (test.test_regrtest.ArgsTestCase.test_random_seed_workers) ... FAIL
test_rerun_async_setup_hook_failure (test.test_regrtest.ArgsTestCase.test_rerun_async_setup_hook_failure) ... ok
test_rerun_async_teardown_hook_failure (test.test_regrtest.ArgsTestCase.test_rerun_async_teardown_hook_failure) ... ok
test_rerun_fail (test.test_regrtest.ArgsTestCase.test_rerun_fail) ... ok
test_rerun_setup_class_hook_failure (test.test_regrtest.ArgsTestCase.test_rerun_setup_class_hook_failure) ... ok
test_rerun_setup_hook_failure (test.test_regrtest.ArgsTestCase.test_rerun_setup_hook_failure) ... ok
test_rerun_setup_module_hook_failure (test.test_regrtest.ArgsTestCase.test_rerun_setup_module_hook_failure) ... ok
test_rerun_success (test.test_regrtest.ArgsTestCase.test_rerun_success) ... ok
test_rerun_teardown_class_hook_failure (test.test_regrtest.ArgsTestCase.test_rerun_teardown_class_hook_failure) ... ok
test_rerun_teardown_hook_failure (test.test_regrtest.ArgsTestCase.test_rerun_teardown_hook_failure) ... ok
test_rerun_teardown_module_hook_failure (test.test_regrtest.ArgsTestCase.test_rerun_teardown_module_hook_failure) ... ok
test_resources (test.test_regrtest.ArgsTestCase.test_resources) ... ok
test_skip (test.test_regrtest.ArgsTestCase.test_skip) ... ok
test_slowest (test.test_regrtest.ArgsTestCase.test_slowest) ... ok
test_slowest_interrupted (test.test_regrtest.ArgsTestCase.test_slowest_interrupted) ... ok
test_success (test.test_regrtest.ArgsTestCase.test_success) ... ok
test_threading_excepthook (test.test_regrtest.ArgsTestCase.test_threading_excepthook) ... ok
test_uncollectable (test.test_regrtest.ArgsTestCase.test_uncollectable) ... ok
test_unicode_guard_env (test.test_regrtest.ArgsTestCase.test_unicode_guard_env) ... ok
test_unraisable_exc (test.test_regrtest.ArgsTestCase.test_unraisable_exc) ... ok
test_wait (test.test_regrtest.ArgsTestCase.test_wait) ... ok
test_worker_decode_error (test.test_regrtest.ArgsTestCase.test_worker_decode_error) ... ok
test_worker_output_on_failure (test.test_regrtest.ArgsTestCase.test_worker_output_on_failure) ... ok
test_finds_expected_number_of_tests (test.test_regrtest.CheckActualTests.test_finds_expected_number_of_tests)
Check that regrtest appears to find the expected set of tests. ... ok
test_arg (test.test_regrtest.ParseArgsTestCase.test_arg) ... ok
test_arg_option_arg (test.test_regrtest.ParseArgsTestCase.test_arg_option_arg) ... ok
test_coverage (test.test_regrtest.ParseArgsTestCase.test_coverage) ... ok
test_coverdir (test.test_regrtest.ParseArgsTestCase.test_coverdir) ... ok
test_dont_add_python_opts (test.test_regrtest.ParseArgsTestCase.test_dont_add_python_opts) ... ok
test_exclude (test.test_regrtest.ParseArgsTestCase.test_exclude) ... ok
test_failfast (test.test_regrtest.ParseArgsTestCase.test_failfast) ... ok
test_fast_ci (test.test_regrtest.ParseArgsTestCase.test_fast_ci) ... FAIL
test_fast_ci_python_cmd (test.test_regrtest.ParseArgsTestCase.test_fast_ci_python_cmd) ... FAIL
test_fast_ci_resource (test.test_regrtest.ParseArgsTestCase.test_fast_ci_resource) ... FAIL
test_forever (test.test_regrtest.ParseArgsTestCase.test_forever) ... ok
test_fromfile (test.test_regrtest.ParseArgsTestCase.test_fromfile) ... ok
test_header (test.test_regrtest.ParseArgsTestCase.test_header) ... ok
test_help (test.test_regrtest.ParseArgsTestCase.test_help) ... ok
test_huntrleaks (test.test_regrtest.ParseArgsTestCase.test_huntrleaks) ... ok
test_ignore (test.test_regrtest.ParseArgsTestCase.test_ignore) ... ok
test_long_option__partial (test.test_regrtest.ParseArgsTestCase.test_long_option__partial) ... ok
test_match (test.test_regrtest.ParseArgsTestCase.test_match) ... ok
test_memlimit (test.test_regrtest.ParseArgsTestCase.test_memlimit) ... ok
test_multiprocess (test.test_regrtest.ParseArgsTestCase.test_multiprocess) ... ok
test_nocoverdir (test.test_regrtest.ParseArgsTestCase.test_nocoverdir) ... ok
test_nowindows (test.test_regrtest.ParseArgsTestCase.test_nowindows) ... ok
test_option_and_arg (test.test_regrtest.ParseArgsTestCase.test_option_and_arg) ... ok
test_option_with_empty_string_value (test.test_regrtest.ParseArgsTestCase.test_option_with_empty_string_value) ... ok
test_quiet (test.test_regrtest.ParseArgsTestCase.test_quiet) ... ok
test_randomize (test.test_regrtest.ParseArgsTestCase.test_randomize) ... ok
test_randseed (test.test_regrtest.ParseArgsTestCase.test_randseed) ... ok
test_rerun (test.test_regrtest.ParseArgsTestCase.test_rerun) ... ok
test_runleaks (test.test_regrtest.ParseArgsTestCase.test_runleaks) ... ok
test_single (test.test_regrtest.ParseArgsTestCase.test_single) ... ok
test_slow_ci (test.test_regrtest.ParseArgsTestCase.test_slow_ci) ... FAIL
test_slowest (test.test_regrtest.ParseArgsTestCase.test_slowest) ... ok
test_start (test.test_regrtest.ParseArgsTestCase.test_start) ... ok
test_testdir (test.test_regrtest.ParseArgsTestCase.test_testdir) ... ok
test_threshold (test.test_regrtest.ParseArgsTestCase.test_threshold) ... ok
test_timeout (test.test_regrtest.ParseArgsTestCase.test_timeout) ... ok
test_two_options (test.test_regrtest.ParseArgsTestCase.test_two_options) ... ok
test_unknown_option (test.test_regrtest.ParseArgsTestCase.test_unknown_option) ... ok
test_unrecognized_argument (test.test_regrtest.ParseArgsTestCase.test_unrecognized_argument) ... ok
test_use (test.test_regrtest.ParseArgsTestCase.test_use) ... ok
test_verbose (test.test_regrtest.ParseArgsTestCase.test_verbose) ... ok
test_verbose3 (test.test_regrtest.ParseArgsTestCase.test_verbose3) ... ok
test_wait (test.test_regrtest.ParseArgsTestCase.test_wait) ... ok
test_module_autotest (test.test_regrtest.ProgramsTestCase.test_module_autotest) ... FAIL
test_module_from_test_autotest (test.test_regrtest.ProgramsTestCase.test_module_from_test_autotest) ... FAIL
test_module_regrtest (test.test_regrtest.ProgramsTestCase.test_module_regrtest) ... FAIL
test_module_test (test.test_regrtest.ProgramsTestCase.test_module_test) ... FAIL
test_pcbuild_rt (test.test_regrtest.ProgramsTestCase.test_pcbuild_rt) ... skipped 'Windows only'
test_script_autotest (test.test_regrtest.ProgramsTestCase.test_script_autotest) ... FAIL
test_script_regrtest (test.test_regrtest.ProgramsTestCase.test_script_regrtest) ... FAIL
test_tools_buildbot_test (test.test_regrtest.ProgramsTestCase.test_tools_buildbot_test) ... skipped 'Windows only'
test_format_duration (test.test_regrtest.TestUtils.test_format_duration) ... ok
test_format_resources (test.test_regrtest.TestUtils.test_format_resources) ... ok
test_get_signal_name (test.test_regrtest.TestUtils.test_get_signal_name) ... ok
test_normalize_test_name (test.test_regrtest.TestUtils.test_normalize_test_name) ... ok

======================================================================
FAIL: test_random (test.test_regrtest.ArgsTestCase.test_random)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_regrtest.py", line 940, in test_random
    randseed = self.parse_random_seed(output)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cpython/Lib/test/test_regrtest.py", line 665, in parse_random_seed
    match = self.regex_search(r'Using random seed ([0-9]+)', output)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cpython/Lib/test/test_regrtest.py", line 488, in regex_search
    self.fail("%r not found in %r" % (regex, output))
AssertionError: 'Using random seed ([0-9]+)' not found in 'Using random seed None\n0:00:00 load avg: 7.05 Run 1 test sequentially\n0:00:00 load avg: 7.05 [1/1] test_regrtest_random\nTESTRANDOM: 794\ntest_regrtest_random ran no tests\n\n== Tests result: NO TESTS RAN ==\n\n1 test run no tests:\n    test_regrtest_random\n\nTotal duration: 52 ms\nTotal tests: run=0\nTotal test files: run=1/1 run_no_tests=1\nResult: NO TESTS RAN\n'

======================================================================
FAIL: test_random_seed (test.test_regrtest.ArgsTestCase.test_random_seed)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_regrtest.py", line 1954, in test_random_seed
    self._check_random_seed(run_workers=False)
  File ".../cpython/Lib/test/test_regrtest.py", line 1951, in _check_random_seed
    self.assertEqual(matches, [expected] * len(tests))
AssertionError: Lists differ: ['Random numbers: [503, 432, 8, 125, 683, 217, 539, 633, 376[141 chars]76]'] != ['Random numbers: [160, 631, 525, 973, 159, 624, 86, 148, 29[142 chars]36]']

First differing element 0:
'Random numbers: [503, 432, 8, 125, 683, 217, 539, 633, 376, 549]'
'Random numbers: [160, 631, 525, 973, 159, 624, 86, 148, 294, 136]'

- ['Random numbers: [503, 432, 8, 125, 683, 217, 539, 633, 376, 549]',
-  'Random numbers: [155, 758, 193, 409, 656, 829, 891, 649, 708, 353]',
-  'Random numbers: [361, 569, 523, 955, 15, 366, 121, 82, 878, 876]']
+ ['Random numbers: [160, 631, 525, 973, 159, 624, 86, 148, 294, 136]',
+  'Random numbers: [160, 631, 525, 973, 159, 624, 86, 148, 294, 136]',
+  'Random numbers: [160, 631, 525, 973, 159, 624, 86, 148, 294, 136]']

======================================================================
FAIL: test_random_seed_workers (test.test_regrtest.ArgsTestCase.test_random_seed_workers)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_regrtest.py", line 1957, in test_random_seed_workers
    self._check_random_seed(run_workers=True)
  File ".../cpython/Lib/test/test_regrtest.py", line 1951, in _check_random_seed
    self.assertEqual(matches, [expected] * len(tests))
AssertionError: Lists differ: ['Random numbers: [762, 220, 332, 666, 137, 398, 674, 781, 5[142 chars]78]'] != ['Random numbers: [160, 631, 525, 973, 159, 624, 86, 148, 29[142 chars]36]']

First differing element 0:
'Random numbers: [762, 220, 332, 666, 137, 398, 674, 781, 565, 877]'
'Random numbers: [160, 631, 525, 973, 159, 624, 86, 148, 294, 136]'

- ['Random numbers: [762, 220, 332, 666, 137, 398, 674, 781, 565, 877]',
-  'Random numbers: [575, 517, 485, 394, 742, 165, 158, 4, 970, 86]',
-  'Random numbers: [143, 808, 956, 892, 987, 384, 744, 777, 658, 178]']
+ ['Random numbers: [160, 631, 525, 973, 159, 624, 86, 148, 294, 136]',
+  'Random numbers: [160, 631, 525, 973, 159, 624, 86, 148, 294, 136]',
+  'Random numbers: [160, 631, 525, 973, 159, 624, 86, 148, 294, 136]']

======================================================================
FAIL: test_fast_ci (test.test_regrtest.ParseArgsTestCase.test_fast_ci)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_regrtest.py", line 406, in test_fast_ci
    regrtest = self.check_ci_mode(args, use_resources)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cpython/Lib/test/test_regrtest.py", line 394, in check_ci_mode
    self.assertTrue(regrtest.randomize)
AssertionError: False is not true

======================================================================
FAIL: test_fast_ci_python_cmd (test.test_regrtest.ParseArgsTestCase.test_fast_ci_python_cmd)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_regrtest.py", line 413, in test_fast_ci_python_cmd
    regrtest = self.check_ci_mode(args, use_resources, rerun=False)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cpython/Lib/test/test_regrtest.py", line 394, in check_ci_mode
    self.assertTrue(regrtest.randomize)
AssertionError: False is not true

======================================================================
FAIL: test_fast_ci_resource (test.test_regrtest.ParseArgsTestCase.test_fast_ci_resource)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_regrtest.py", line 423, in test_fast_ci_resource
    self.check_ci_mode(args, use_resources)
  File ".../cpython/Lib/test/test_regrtest.py", line 394, in check_ci_mode
    self.assertTrue(regrtest.randomize)
AssertionError: False is not true

======================================================================
FAIL: test_slow_ci (test.test_regrtest.ParseArgsTestCase.test_slow_ci)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_regrtest.py", line 428, in test_slow_ci
    regrtest = self.check_ci_mode(args, use_resources)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cpython/Lib/test/test_regrtest.py", line 394, in check_ci_mode
    self.assertTrue(regrtest.randomize)
AssertionError: False is not true

======================================================================
FAIL: test_module_autotest (test.test_regrtest.ProgramsTestCase.test_module_autotest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_regrtest.py", line 785, in test_module_autotest
    self.run_tests(args)
  File ".../cpython/Lib/test/test_regrtest.py", line 760, in run_tests
    self.check_output(output)
  File ".../cpython/Lib/test/test_regrtest.py", line 754, in check_output
    self.parse_random_seed(output)
  File ".../cpython/Lib/test/test_regrtest.py", line 665, in parse_random_seed
    match = self.regex_search(r'Using random seed ([0-9]+)', output)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cpython/Lib/test/test_regrtest.py", line 488, in regex_search
    self.fail("%r not found in %r" % (regex, output))
AssertionError: 'Using random seed ([0-9]+)' not found in 'Using random seed None\n0:00:00 load avg: 6.86 Run 4 tests in parallel using 4 worker processes (timeout: 1 hour, worker timeout: 1 hour 5 min)\n0:00:00 load avg: 6.86 [1/4] test_regrtest_noop52 passed\n0:00:00 load avg: 6.86 [2/4] test_regrtest_noop53 passed\n0:00:00 load avg: 6.86 [3/4] test_regrtest_noop51 passed\n0:00:00 load avg: 6.86 [4/4] test_regrtest_noop50 passed\n\n== Tests result: SUCCESS ==\n\nAll 4 tests OK.\n\nTotal duration: 177 ms\nTotal tests: run=4\nTotal test files: run=4/4\nResult: SUCCESS\n'

======================================================================
FAIL: test_module_from_test_autotest (test.test_regrtest.ProgramsTestCase.test_module_from_test_autotest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_regrtest.py", line 792, in test_module_from_test_autotest
    self.run_tests(args)
  File ".../cpython/Lib/test/test_regrtest.py", line 760, in run_tests
    self.check_output(output)
  File ".../cpython/Lib/test/test_regrtest.py", line 754, in check_output
    self.parse_random_seed(output)
  File ".../cpython/Lib/test/test_regrtest.py", line 665, in parse_random_seed
    match = self.regex_search(r'Using random seed ([0-9]+)', output)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cpython/Lib/test/test_regrtest.py", line 488, in regex_search
    self.fail("%r not found in %r" % (regex, output))
AssertionError: 'Using random seed ([0-9]+)' not found in 'Using random seed None\n0:00:00 load avg: 6.86 Run 4 tests in parallel using 4 worker processes (timeout: 1 hour, worker timeout: 1 hour 5 min)\n0:00:00 load avg: 6.86 [1/4] test_regrtest_noop56 passed\n0:00:00 load avg: 6.86 [2/4] test_regrtest_noop55 passed\n0:00:00 load avg: 6.86 [3/4] test_regrtest_noop57 passed\n0:00:00 load avg: 6.86 [4/4] test_regrtest_noop54 passed\n\n== Tests result: SUCCESS ==\n\nAll 4 tests OK.\n\nTotal duration: 226 ms\nTotal tests: run=4\nTotal test files: run=4/4\nResult: SUCCESS\n'

======================================================================
FAIL: test_module_regrtest (test.test_regrtest.ProgramsTestCase.test_module_regrtest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_regrtest.py", line 779, in test_module_regrtest
    self.run_tests(args)
  File ".../cpython/Lib/test/test_regrtest.py", line 760, in run_tests
    self.check_output(output)
  File ".../cpython/Lib/test/test_regrtest.py", line 754, in check_output
    self.parse_random_seed(output)
  File ".../cpython/Lib/test/test_regrtest.py", line 665, in parse_random_seed
    match = self.regex_search(r'Using random seed ([0-9]+)', output)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cpython/Lib/test/test_regrtest.py", line 488, in regex_search
    self.fail("%r not found in %r" % (regex, output))
AssertionError: 'Using random seed ([0-9]+)' not found in 'Using random seed None\n0:00:00 load avg: 6.86 Run 4 tests in parallel using 4 worker processes (timeout: 1 hour, worker timeout: 1 hour 5 min)\n0:00:00 load avg: 6.86 [1/4] test_regrtest_noop59 passed\n0:00:00 load avg: 6.86 [2/4] test_regrtest_noop58 passed\n0:00:00 load avg: 6.86 [3/4] test_regrtest_noop61 passed\n0:00:00 load avg: 6.86 [4/4] test_regrtest_noop60 passed\n\n== Tests result: SUCCESS ==\n\nAll 4 tests OK.\n\nTotal duration: 227 ms\nTotal tests: run=4\nTotal test files: run=4/4\nResult: SUCCESS\n'

======================================================================
FAIL: test_module_test (test.test_regrtest.ProgramsTestCase.test_module_test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_regrtest.py", line 773, in test_module_test
    self.run_tests(args)
  File ".../cpython/Lib/test/test_regrtest.py", line 760, in run_tests
    self.check_output(output)
  File ".../cpython/Lib/test/test_regrtest.py", line 754, in check_output
    self.parse_random_seed(output)
  File ".../cpython/Lib/test/test_regrtest.py", line 665, in parse_random_seed
    match = self.regex_search(r'Using random seed ([0-9]+)', output)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cpython/Lib/test/test_regrtest.py", line 488, in regex_search
    self.fail("%r not found in %r" % (regex, output))
AssertionError: 'Using random seed ([0-9]+)' not found in 'Using random seed None\n0:00:00 load avg: 6.86 Run 4 tests in parallel using 4 worker processes (timeout: 1 hour, worker timeout: 1 hour 5 min)\n0:00:00 load avg: 6.86 [1/4] test_regrtest_noop62 passed\n0:00:00 load avg: 6.86 [2/4] test_regrtest_noop64 passed\n0:00:00 load avg: 6.86 [3/4] test_regrtest_noop65 passed\n0:00:00 load avg: 6.86 [4/4] test_regrtest_noop63 passed\n\n== Tests result: SUCCESS ==\n\nAll 4 tests OK.\n\nTotal duration: 175 ms\nTotal tests: run=4\nTotal test files: run=4/4\nResult: SUCCESS\n'

======================================================================
FAIL: test_script_autotest (test.test_regrtest.ProgramsTestCase.test_script_autotest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_regrtest.py", line 798, in test_script_autotest
    self.run_tests(args)
  File ".../cpython/Lib/test/test_regrtest.py", line 760, in run_tests
    self.check_output(output)
  File ".../cpython/Lib/test/test_regrtest.py", line 754, in check_output
    self.parse_random_seed(output)
  File ".../cpython/Lib/test/test_regrtest.py", line 665, in parse_random_seed
    match = self.regex_search(r'Using random seed ([0-9]+)', output)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cpython/Lib/test/test_regrtest.py", line 488, in regex_search
    self.fail("%r not found in %r" % (regex, output))
AssertionError: 'Using random seed ([0-9]+)' not found in 'Using random seed None\n0:00:00 load avg: 6.86 Run 4 tests in parallel using 4 worker processes (timeout: 1 hour, worker timeout: 1 hour 5 min)\n0:00:00 load avg: 6.86 [1/4] test_regrtest_noop69 passed\n0:00:00 load avg: 6.86 [2/4] test_regrtest_noop68 passed\n0:00:00 load avg: 6.86 [3/4] test_regrtest_noop66 passed\n0:00:00 load avg: 6.86 [4/4] test_regrtest_noop67 passed\n\n== Tests result: SUCCESS ==\n\nAll 4 tests OK.\n\nTotal duration: 227 ms\nTotal tests: run=4\nTotal test files: run=4/4\nResult: SUCCESS\n'

======================================================================
FAIL: test_script_regrtest (test.test_regrtest.ProgramsTestCase.test_script_regrtest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../cpython/Lib/test/test_regrtest.py", line 767, in test_script_regrtest
    self.run_tests(args)
  File ".../cpython/Lib/test/test_regrtest.py", line 760, in run_tests
    self.check_output(output)
  File ".../cpython/Lib/test/test_regrtest.py", line 754, in check_output
    self.parse_random_seed(output)
  File ".../cpython/Lib/test/test_regrtest.py", line 665, in parse_random_seed
    match = self.regex_search(r'Using random seed ([0-9]+)', output)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cpython/Lib/test/test_regrtest.py", line 488, in regex_search
    self.fail("%r not found in %r" % (regex, output))
AssertionError: 'Using random seed ([0-9]+)' not found in 'Using random seed None\n0:00:00 load avg: 6.86 Run 4 tests in parallel using 4 worker processes (timeout: 1 hour, worker timeout: 1 hour 5 min)\n0:00:00 load avg: 6.86 [1/4] test_regrtest_noop72 passed\n0:00:00 load avg: 6.86 [2/4] test_regrtest_noop73 passed\n0:00:00 load avg: 6.86 [3/4] test_regrtest_noop70 passed\n0:00:00 load avg: 6.86 [4/4] test_regrtest_noop71 passed\n\n== Tests result: SUCCESS ==\n\nAll 4 tests OK.\n\nTotal duration: 184 ms\nTotal tests: run=4\nTotal test files: run=4/4\nResult: SUCCESS\n'

----------------------------------------------------------------------
Ran 103 tests in 27.042s

FAILED (failures=13, skipped=5)
test test_regrtest failed
test_regrtest failed (13 failures)

== Tests result: FAILURE ==

1 test failed:
    test_regrtest

Total duration: 27.1 sec
Total tests: run=103 failures=13 skipped=5
Total test files: run=1/1 failed=1
Result: FAILURE

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

@hroncok hroncok added the type-bug An unexpected behavior, bug, or error label Oct 16, 2023
@hroncok
Copy link
Contributor Author

hroncok commented Oct 16, 2023

The AssertionError: 'Using random seed ([0-9]+)' not found in 'Using random seed None\n...' failure is pretty much obvious to me -- after #110168 the output has Using random seed None and the assertion expects a number.

The AssertionError: False is not true at self.assertTrue(regrtest.randomize) probably suffers from a similar problem.

The unexpected numbers in test_random_seed* are caused by discarding --randseed value entirely.


I suppose this could all be fixed if the tests invoke subprocess with SOURCE_DATE_EPOCH unset.

def run_tests(self, args, env=None):
output = self.run_python(args, env=env)
self.check_output(output)

@Eclips4 Eclips4 added the tests Tests in the Lib/test dir label Oct 16, 2023
@Eclips4
Copy link
Member

Eclips4 commented Oct 16, 2023

cc @vstinner

@hroncok
Copy link
Contributor Author

hroncok commented Oct 16, 2023

The following naïve patch makes the test pass:

$ git diff
diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py
index e65d9a89ff8..5ffbea5d910 100644
--- a/Lib/test/test_regrtest.py
+++ b/Lib/test/test_regrtest.py
@@ -391,8 +391,12 @@ def check_ci_mode(self, args, use_resources, rerun=True):
         regrtest = main.Regrtest(ns)
         self.assertEqual(regrtest.num_workers, -1)
         self.assertEqual(regrtest.want_rerun, rerun)
-        self.assertTrue(regrtest.randomize)
-        self.assertIsInstance(regrtest.random_seed, int)
+        if 'SOURCE_DATE_EPOCH' in os.environ:
+            self.assertFalse(regrtest.randomize)
+            self.assertIsNone(regrtest.random_seed)
+        else:
+            self.assertTrue(regrtest.randomize)
+            self.assertIsInstance(regrtest.random_seed, int)
         self.assertTrue(regrtest.fail_env_changed)
         self.assertTrue(regrtest.print_slowest)
         self.assertTrue(regrtest.output_on_failure)
@@ -700,6 +704,11 @@ def run_python(self, args, **kw):
         if 'uops' in sys._xoptions:
             # Pass -X uops along
             extraargs.extend(['-X', 'uops'])
+        # discard SOURCE_DATE_EPOCH
+        # https://github.com/python/cpython/issues/110932
+        kw['env'] = kw.get('env') or dict(os.environ)
+        if 'SOURCE_DATE_EPOCH' in kw['env']:
+            del kw['env']['SOURCE_DATE_EPOCH']
         args = [sys.executable, *extraargs, '-X', 'faulthandler', '-I', *args]
         proc = self.run_command(args, **kw)
         return proc.stdout

However, since this is testing regrtest, I suppose a proper fix is to add tests that test the randomness behavior with SOURCE_DATE_EPOCH set and without SOURCE_DATE_EPOCH set, rather than always unsetting it or asserting based on environment.

vstinner added a commit to vstinner/cpython that referenced this issue Oct 20, 2023
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as a string as the random seed.
vstinner added a commit to vstinner/cpython that referenced this issue Oct 20, 2023
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as a string as the random seed.
vstinner added a commit to vstinner/cpython that referenced this issue Oct 20, 2023
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as a string as the random seed.
vstinner added a commit to vstinner/cpython that referenced this issue Oct 20, 2023
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as a string as the random seed.
vstinner added a commit to vstinner/cpython that referenced this issue Oct 20, 2023
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as a string as the random seed.
vstinner added a commit to vstinner/cpython that referenced this issue Oct 20, 2023
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as the random seed.
vstinner added a commit to vstinner/cpython that referenced this issue Oct 20, 2023
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as the random seed.
vstinner added a commit to vstinner/cpython that referenced this issue Oct 20, 2023
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as the random seed.
@vstinner
Copy link
Member

I proposed PR gh-111143 to fix this issue.

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 21, 2023
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as the random seed.
(cherry picked from commit 7237fb5)

Co-authored-by: Victor Stinner <vstinner@python.org>
vstinner added a commit that referenced this issue Oct 21, 2023
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as the random seed.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 21, 2023
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as the random seed.
(cherry picked from commit 7237fb5)

Co-authored-by: Victor Stinner <vstinner@python.org>
vstinner added a commit that referenced this issue Oct 21, 2023
…1153)

gh-110932: Fix regrtest for SOURCE_DATE_EPOCH (GH-111143)

If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as the random seed.
(cherry picked from commit 7237fb5)

Co-authored-by: Victor Stinner <vstinner@python.org>
vstinner added a commit that referenced this issue Oct 21, 2023
…1154)

gh-110932: Fix regrtest for SOURCE_DATE_EPOCH (GH-111143)

If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as the random seed.
(cherry picked from commit 7237fb5)

Co-authored-by: Victor Stinner <vstinner@python.org>
@Eclips4 Eclips4 closed this as completed Oct 21, 2023
@Eclips4
Copy link
Member

Eclips4 commented Oct 21, 2023

Thanks for the fix @vstinner and @hroncok for the report.

aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as the random seed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

3 participants