-
-
Notifications
You must be signed in to change notification settings - Fork 2.9k
Open
Labels
plugin: subtestsrelated to the subtests builtin pluginrelated to the subtests builtin pluginplugin: unittestrelated to the unittest integration builtin pluginrelated to the unittest integration builtin plugintype: performanceperformance or memory problem/improvementperformance or memory problem/improvementtype: regressionindicates a problem that was introduced in a release which was working previouslyindicates a problem that was introduced in a release which was working previously
Description
Running the following test using pytest 9.0.0 in python 3.10 leads to a quadratic blowup:
from unittest import TestCase
N = 1 # 1, 2, 3, 4, ...
class TestQuadraticBlowup(TestCase):
def test_quadratic_blowup(self) -> None:
for i in range(1000 * N): # multiplying by 1000 to make bigger steps
with self.subTest():
passFor a given N, it takes O(N^2) time to complete. Here are a few samples:
N=10-> 10sN=20-> 35sN=30-> 73sN=40-> 130s
When running this test using python -m unittest test.py, it takes less than 0.5s for N=40.
I traced this test a little:
TestCaseFunction.runtest()is called (unittest.py#360)- it calls
testcase(result=self)(unittest.py#389) which isTestCase.run()(case.py#557) - it constructs
outcome = _Outcome(result)(case.py#582) and assigns toselfa line later - the actual test is executed with
self._callTestMethod(testMethod)(case.py#591) - the test repeatedly calls
with self.subTest()which isTestCase.subTest()(case.py#480) TestCase.subTest()constructs_SubTest(case.py#495)- it enters that subtest with
with self._outcome.testPartExecutor(self._subtest, isTest=True): yield(case.py#497) which is_Outcome.testPartExecutor()(case.py#55) - when the subtest finishes successfully, that
_Subtestinstance is appended to the_Outcome.errorslist withNoneas error (case.py#79) - after all subtests finish,
TestCase.run()callsself._feedErrorsToResult(result, outcome.errors)(case.py#599)- note that
outcome.errorshas1000 * Nelements
- note that
TestCase._feedErrorsToResult()iterates over all those errors (case.py#511)- this gives the first, outer
O(N)
- this gives the first, outer
- for each error, it calls
result.addSubTest(test.test_case, test, exc_info)(case.py#513) which isTestCaseFunction.addSubTest()(unittest.py#404) TestCaseFunction.addSubTest()eventually iterates overself.instance._outcome.errors(unittest.py#455) which is the same list asoutcome.errorsabove (1000 * Nelements)- this gives us the second, nested
O(N), which in total gives usO(N^2)for the entire test
- this gives us the second, nested
There might be similar issues with skipped tests (unittest.py#450). Another potential hot spot is unittest.py#318.
pip list output:
Package Version
----------------- -------
exceptiongroup 1.3.0
iniconfig 2.3.0
packaging 25.0
pip 23.0.1
pluggy 1.6.0
Pygments 2.19.2
pytest 9.0.1
setuptools 65.5.0
tomli 2.3.0
typing_extensions 4.15.0
Python: 3.10.17
Metadata
Metadata
Assignees
Labels
plugin: subtestsrelated to the subtests builtin pluginrelated to the subtests builtin pluginplugin: unittestrelated to the unittest integration builtin pluginrelated to the unittest integration builtin plugintype: performanceperformance or memory problem/improvementperformance or memory problem/improvementtype: regressionindicates a problem that was introduced in a release which was working previouslyindicates a problem that was introduced in a release which was working previously