Clear caches after every test #68027

serhiy-storchaka · 2015-04-01T11:30:42Z

BPO	23839
Nosy	@rhettinger, @pitrou, @vstinner, @rbtcollins, @ezio-melotti, @voidspace, @vadmium, @serhiy-storchaka
PRs	[Do Not Merge] Convert `Misc/NEWS` so that it is managed by towncrier #552
Files	regrtest_clear_caches.patch regrtest_clear_caches_2.patch regrtest_clear_caches_3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-11-11.12:51:26.462>
created_at = <Date 2015-04-01.11:30:41.807>
labels = ['3.7', 'tests', 'performance']
title = 'Clear caches after every test'
updated_at = <Date 2017-03-31.16:36:37.509>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:37.509>
actor = 'dstufft'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-11-11.12:51:26.462>
closer = 'serhiy.storchaka'
components = ['Tests']
creation = <Date 2015-04-01.11:30:41.807>
creator = 'serhiy.storchaka'
dependencies = []
files = ['38804', '45413', '45419']
hgrepos = []
issue_num = 23839
keywords = ['patch']
message_count = 17.0
messages = ['239782', '239791', '239794', '239796', '239799', '239844', '239845', '239898', '249038', '249287', '249288', '280433', '280562', '280571', '280579', '280725', '280768']
nosy_count = 9.0
nosy_names = ['rhettinger', 'pitrou', 'vstinner', 'rbcollins', 'ezio.melotti', 'michael.foord', 'python-dev', 'martin.panter', 'serhiy.storchaka']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue23839'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

The text was updated successfully, but these errors were encountered:

serhiy-storchaka · 2015-04-01T11:30:42Z

Some caches can grow unlimitedly during running the testsuite (in particular linecache). On some machines with limited resources this can cause multiple MemoryErrors. Many of these MemoryErrors can be avoided if clear caches between tests. Lib/test/regrtest.py already contains a code for cleaning caches, but it is used only with the -R option.

vstinner · 2015-04-01T12:05:04Z

Maybe tests known to fill caches should clear theirself caches at cleanup? Always clearing all caches may have an impact on performances and so makes tests longer. Caches are designed to speedup Python :-)

We may add some helper functions in test.support for the most common caches, but not to clear *all* caches.

By the way, forcing calls to gc.collect() may make tests more reliable. It would avoid to get warnings 4 lines after the test leaking a resource finished.

But it's not easy to make this reliable, because unittest stores a lot of things. unittest.TestCase.run() stores exceptions which store a traceback which stores frames which store references to local variables. Instead of storing a whole sys.exc_info(), we can maybe use the new light traceback.TracebackException which only stores info required to format a traceback as text, without storing all these heavily objects creating reference cycles.

(Hum, it becomes a little off-topic, sorry, we may open a new separated issue.)

serhiy-storchaka · 2015-04-01T12:33:17Z

We should try to know how it impacts a performance. I suppose not too much, but may be my intuition lies me.

vstinner · 2015-04-01T12:39:18Z

By the way, a workaround is also to run each test (file) in a new fresh process. It's a common way to work around memory fragmentation ;-)

On some machines with limited resources this can cause multiple MemoryErrors.

Do you have an idea of the memory usage of the Python test suite? You may run it with tracemalloc to get the memory peak from tracemalloc.get_traced_memory()[1].

I mean, memory is cheap nowadays. If you get MemoryError, it's probably more a bug than tests leaking more and more memory, no?

serhiy-storchaka · 2015-04-01T12:57:37Z

max_rss is a little larger 500 MB after ran all testsuite (except bigmem tests) on 32-bit Linux. Only few tests needs more than 200 MB. But may be some 64-bit only tests consume more memory. If a test consume more than 1GiB of memory and is not decorated with bigmemtest, it should be considered as a bug.

rbtcollins · 2015-04-01T20:30:15Z

I'd like to see a definite profile of a bloated stdlib test process before we assume we know the issue - the usual leak I see in test code is used test objects, and I'm not sure we've ported the usual fix for that into unittest yet (we should).

As far as linecache is concerned, we shouldn't be triggering linecache population except when a traceback is actually rendered, and so on a passing test run there should be little hitting it at all outside of tests that actually test rendering of tracebacks.

rbtcollins · 2015-04-01T20:31:47Z

+1 on moving to the summary classes rather than actual tracebacks in unittest. (Or perhaps even just serialised tracebacks like we do in testtools).

serhiy-storchaka · 2015-04-02T10:54:17Z

I ran all tests with and without clearing all caches.

Unpatched (without clearing):

real 31m23.694s
user 16m10.340s
sys 2m25.084s

Patched (with clearing):

real 29m28.859s
user 16m19.048s
sys 1m42.184s

There is no significant difference (may be with the patch tests are even faster).

rbtcollins · 2015-08-24T00:20:52Z

Ok, so this is still in the noise space - it might be useful, it might not. Do we have reports of machines failing to run the test suite (that are also usefully big enough to compile Python and use it)?

serhiy-storchaka · 2015-08-28T18:46:54Z

http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/9963/steps/test/logs/stdio

Passed 395 tests of 398 and failed on test_decimal:

...
[396/398] test_decimal
Traceback (most recent call last):
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 1276, in runtest_inner
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_decimal.py", line 5598, in test_main
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/support/__init__.py", line 1809, in run_unittest
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/support/__init__.py", line 1775, in _run_suite
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/runner.py", line 176, in run
    test(result)
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/suite.py", line 122, in run
    test(result)
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/suite.py", line 122, in run
    test(result)
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/case.py", line 645, in __call__
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/case.py", line 605, in run
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/case.py", line 538, in _feedErrorsToResult
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/runner.py", line 69, in addError
    self.stream.writeln("ERROR")
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/runner.py", line 25, in writeln
    self.write(arg)
MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 1279, in runtest_inner
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 1232, in __exit__
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 1172, in get_sysconfig__CONFIG_VARS
MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 532, in main
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 963, in runtest
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 1300, in runtest_inner
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/traceback.py", line 163, in format_exc
    return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
MemoryError
Traceback (most recent call last):
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/__main__.py", line 3, in <module>
    regrtest.main_in_temp_cwd()
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 1560, in main_in_temp_cwd
    main()
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 738, in main
    raise Exception("Child error on {}: {}".format(test, result[1]))
Exception: Child error on test_decimal:

serhiy-storchaka · 2015-08-28T18:50:16Z

May be related: http://buildbot.python.org/all/builders/x86%20OpenIndiana%203.x/builds/10375/steps/test/logs/stdio

======================================================================
ERROR: testCount (test.test_socket.SendfileUsingSendfileTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 5160, in testCount
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 5063, in recv_data
MemoryError

======================================================================
ERROR: testCount (test.test_socket.SendfileUsingSendfileTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 266, in _tearDown
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 278, in clientRun
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 5153, in _testCount
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/socket.py", line 282, in _sendfile_use_sendfile
    raise _socket.timeout('timed out')
socket.timeout: timed out

======================================================================
ERROR: testWithTimeout (test.test_socket.SendfileUsingSendfileTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 5230, in testWithTimeout
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 5063, in recv_data
MemoryError

======================================================================
ERROR: testWithTimeout (test.test_socket.SendfileUsingSendfileTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 266, in _tearDown
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 278, in clientRun
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 5225, in _testWithTimeout
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/socket.py", line 282, in _sendfile_use_sendfile
    raise _socket.timeout('timed out')
socket.timeout: timed out

serhiy-storchaka · 2016-11-09T18:51:11Z

Synchronized with current code and added clearing typing caches.

rhettinger · 2016-11-11T06:32:46Z

LGTM.

python-dev · 2016-11-11T09:49:20Z

New changeset bc81f2137706 by Serhiy Storchaka in branch '2.7':
Issue bpo-23839: Various caches now are cleared before running every test file.
https://hg.python.org/cpython/rev/bc81f2137706

New changeset 89776a40e0ec by Serhiy Storchaka in branch '3.5':
Issue bpo-23839: Various caches now are cleared before running every test file.
https://hg.python.org/cpython/rev/89776a40e0ec

New changeset c89f213b21e8 by Serhiy Storchaka in branch '3.6':
Issue bpo-23839: Various caches now are cleared before running every test file.
https://hg.python.org/cpython/rev/c89f213b21e8

New changeset 5d1067e89717 by Serhiy Storchaka in branch 'default':
Issue bpo-23839: Various caches now are cleared before running every test file.
https://hg.python.org/cpython/rev/5d1067e89717

serhiy-storchaka · 2016-11-11T12:51:26Z

Thanks for the review Raymond.

vadmium · 2016-11-14T01:05:57Z

When I run the tests with -Werror (or any other -W option), I now get

2 tests altered the execution environment:
test___all__ test_warnings

vstinner · 2016-11-14T12:57:50Z

Martin Panter:
"""
When I run the tests with -Werror (or any other -W option), I now get

2 tests altered the execution environment:
test___all__ test_warnings
"""

I confirm: I opened issue bpo-28688 to track this issue (this issue is now closed).

serhiy-storchaka added tests Tests in the Lib/test dir performance Performance or resource usage labels Apr 1, 2015

serhiy-storchaka added the 3.7 (EOL) end of life label Nov 11, 2016

serhiy-storchaka closed this as completed Nov 11, 2016

serhiy-storchaka self-assigned this Nov 11, 2016

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clear caches after every test #68027

Clear caches after every test #68027

serhiy-storchaka commented Apr 1, 2015

serhiy-storchaka commented Apr 1, 2015

vstinner commented Apr 1, 2015

serhiy-storchaka commented Apr 1, 2015

vstinner commented Apr 1, 2015

serhiy-storchaka commented Apr 1, 2015

rbtcollins commented Apr 1, 2015

rbtcollins commented Apr 1, 2015

serhiy-storchaka commented Apr 2, 2015

rbtcollins commented Aug 24, 2015

serhiy-storchaka commented Aug 28, 2015

serhiy-storchaka commented Aug 28, 2015

serhiy-storchaka commented Nov 9, 2016

rhettinger commented Nov 11, 2016

python-dev mannequin commented Nov 11, 2016

serhiy-storchaka commented Nov 11, 2016

vadmium commented Nov 14, 2016

vstinner commented Nov 14, 2016

Clear caches after every test #68027

Clear caches after every test #68027

Comments

serhiy-storchaka commented Apr 1, 2015

serhiy-storchaka commented Apr 1, 2015

vstinner commented Apr 1, 2015

serhiy-storchaka commented Apr 1, 2015

vstinner commented Apr 1, 2015

serhiy-storchaka commented Apr 1, 2015

rbtcollins commented Apr 1, 2015

rbtcollins commented Apr 1, 2015

serhiy-storchaka commented Apr 2, 2015

rbtcollins commented Aug 24, 2015

serhiy-storchaka commented Aug 28, 2015

serhiy-storchaka commented Aug 28, 2015

serhiy-storchaka commented Nov 9, 2016

rhettinger commented Nov 11, 2016

python-dev mannequin commented Nov 11, 2016

serhiy-storchaka commented Nov 11, 2016

vadmium commented Nov 14, 2016

vstinner commented Nov 14, 2016