test_math.test_sumprod_stress: "_PyEval_EvalFrameDefault: Assertion `oparg & 1' failed"

[colesbury]

Crash report

Seen in:

https://buildbot.python.org/#/builders/345/builds/10459/steps/6/logs/stdio

Commit: b6769e9

test_sumprod_stress (test.test_math.MathTests.test_sumprod_stress) ... python: Python/generated_cases.c.h:8149: _PyEval_EvalFrameDefault: Assertion `oparg & 1' failed.
Fatal Python error: Aborted

Current thread 0x00007f44956e7740 (most recent call first):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 1133 in assertTupleEqual
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 916 in assertEqual
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_math.py", line 1462 in test_sumprod_stress
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 660 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 716 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/runner.py", line 259 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 84 in _run_suite
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 42 in run_unittest
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 162 in test_func
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 118 in regrtest_runner
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 165 in _load_run_test
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 210 in _runtest_env_changed_exc
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 319 in _runtest
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 348 in run_single_test
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/worker.py", line 92 in worker_process
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/worker.py", line 127 in main
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/worker.py", line 131 in <module>
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 88 in _run_code
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testinternalcapi (total: 1)

The test passed when retried.

cpython/Python/generated_cases.c.h

Lines 8146 to 8156 in b6769e9

	// _LOAD_ATTR_METHOD_WITH_VALUES
	{
	PyObject *descr = read_obj(&this_instr[6].cache);
	assert(oparg & 1);
	/* Cached method object */
	STAT_INC(LOAD_ATTR, hit);
	assert(descr != NULL);
	assert(_PyType_HasFeature(Py_TYPE(descr), Py_TPFLAGS_METHOD_DESCRIPTOR));
	attr = PyStackRef_FromPyObjectNew(descr);
	self = owner;
	}

[colesbury]

A different interpreter crash: https://buildbot.python.org/all/#/builders/310/builds/1670/steps/6/logs/stdio

So far I've only seen these on angelico-debian-amd64, which makes me wonder if the machine has bad memory. I haven't seen any similar issues on any of the other buildbots.

#130854 was also on angelico-debian-amd64.

@Rosuav - would you be able to run a memory test on that machine?

test_extractall_none_uid (test.test_tarfile.NoneInfoExtractTests_Tar.test_extractall_none_uid) ... python: Python/generated_cases.c.h:12055: _PyEval_EvalFrameDefault: Assertion `STACK_LEVEL() >= level' failed.
Fatal Python error: Aborted

Current thread 0x00007f3b0b39a740 (most recent call first):
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/pathlib/_local.py", line 198 in _str_normcase
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/pathlib/_local.py", line 213 in __eq__
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/_collections_abc.py", line 1023 in __contains__
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/pathlib/_local.py", line 485 in relative_to
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/test_tarfile.py", line 3233 in <genexpr>
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/test_tarfile.py", line 3232 in check_files_present
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/test_tarfile.py", line 3246 in extract_with_none
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/contextlib.py", line 141 in __enter__
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/test_tarfile.py", line 3282 in test_extractall_none_uid
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/unittest/case.py", line 660 in run
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/unittest/case.py", line 716 in __call__
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/unittest/runner.py", line 259 in run
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 84 in _run_suite
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 42 in run_unittest
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 162 in test_func
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 118 in regrtest_runner
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 165 in _load_run_test
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 210 in _runtest_env_changed_exc
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 319 in _runtest
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 348 in run_single_test
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/libregrtest/worker.py", line 92 in worker_process
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/libregrtest/worker.py", line 127 in main
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/libregrtest/worker.py", line 131 in <module>
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/runpy.py", line 88 in _run_code
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testinternalcapi (total: 1)

[Rosuav]

I've actually been doing some thermal stress-testing on that machine recently. I hadn't been aware of buildbot failures, but it is entirely possible that these happened during those tests. (It's an Intel 14700KF and they have proven to be, uhh, one of Intel's more notable releases.) The upshot of the testing is that now I have a sensors config that reduces the temperature cutoff before the system reduces CPU frequency, which seems to have improved things.

Apologies for not noticing that there were builds caught by this, but I was initially under the impression that the failure mode was a full system wedge and consequent restart, and it seems there may possibly have been less critical results. Will be monitoring the buildbot waterfall during future testing. Thanks for pinging me.

[colesbury]

Thanks for the info. I'm going to close this under the assumption that the failures were hardware errors. I'm not 100% sure, but I now think it's more likely than a bug in the interpreter.