Test suite consumes an extraordinary amount of memory #104
Comments
|
Thanks for reporting this - I will check. I observed something similar
when using multiprocessing for concurrency. One of the reason I added
the ability to use gevent.
Best wishes,
Andreas
…On 21/12/2018 18:29, Andreas Tille wrote:
Hi,
there is a bug report <https://bugs.debian.org/841073> for the Debian
packaged version which says that the ruffus test suite consumes an
extraordinary amount of memory on the build machine. The bug was filed
in 2016 for version 2.6.3 of ruffus where a memory consumption of 18GB
was measured. The bug reporter did a new measure for the recent version
2.8.1 he gets 38GB so it became worse. I tried to reproduce the issue
and can confirm that if I do
|while true; do grep Committed_AS /proc/meminfo; sleep 1; done |
in a separate xterm while running the build time tests I get something like
|Committed_AS: 9807008 kB Committed_AS: 22324324 kB Committed_AS:
33082052 kB Committed_AS: 33082052 kB Committed_AS: 17073508 kB
Committed_AS: 41512736 kB Committed_AS: 45553976 kB Committed_AS:
26653296 kB Committed_AS: 5887392 kB |
while the test suite outputs something like
|test_pool_manager.py::TestExecutionEngines::test_pipeline_fails_with_unknown_manager
PASSED [ 25%]
test_pool_manager.py::TestExecutionEngines::test_pipeline_runs_with_gevent_manager
<- ../../../../../../../usr/lib/python2.7/unittest/case.py SKIPPED [
50%]
test_pool_manager.py::TestExecutionEngines::test_pipeline_runs_with_multiprocessing
PASSED [ 75%]
test_pool_manager.py::TestExecutionEngines::test_pipeline_runs_with_multithreading
PASSED [100%] ===================== 3 passed, 1 skipped in 2.50 seconds
====================== pytest -v test_cmdline.py
============================= test session starts
============================== platform linux2 -- Python 2.7.15+,
pytest-3.6.4, py-1.7.0, pluggy-0.8.0 -- /usr/bin/python2 cachedir:
../../../../../.pytest_cache rootdir:
/home/andreas/debian-maintain/salsa/med-team/build-area/python-ruffus-2.8.1,
inifile: collecting ... collected 4 items
test_cmdline.py::Test_cmdline::test_argparse PASSED [ 25%]
test_cmdline.py::Test_cmdline::test_optparse PASSED [ 50%]
test_cmdline.py::Test_cmdline::test_verbose PASSED [ 75%]
test_cmdline.py::Test_cmdline::test_verbose_abbreviated_path PASSED [100%] |
Please note: I'm definitely not sure that this is the affected test
neither that these are the only tests which push the memory limit way
more far than other programs but I confirm that the reporter is right
and that the issue can be found somewhere close here. I'd suggest some
bisecting to find the actual test to find out the reason.
Kind regards, Andreas.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#104>, or mute the
thread
<https://github.com/notifications/unsubscribe-auth/AEOKYO_ZMQXk4A-sIX1GixKxI6zqE9oxks5u7SiegaJpZM4ZekBb>.
|
|
Hi @AndreasHeger, Are there some advances for this issue? Thanks! |
|
Greetings, I kept a close look to the committed memory while running the test suite and I believe I identified a 20GB bump in the memory consumption in ruffus/test/test_with_logger.py. The test runs the pipeline at 500 processes on two occurrences, which seemed a bit unreasonable on low cores count machines, possibly having little memory as well. When I adjusted the process count of the test suite to my PC cores count, the memory consumption delta dropped to 5GB. Would it be possible to dynamically adjust the multiprocess in the pipeline_run calls of the test suite to the machines allowed cores count (e.g. with Have a nice day, :) |
|
The change I had in mind is available under the form of patch at the following location in case you are interested: Please feel free to use it if you think this is useful. |
Hi,
there is a bug report for the Debian packaged version which says that the ruffus test suite consumes an extraordinary amount of memory on the build machine. The bug was filed in 2016 for version 2.6.3 of ruffus where a memory consumption of 18GB was measured. The bug reporter did a new measure for the recent version 2.8.1 he gets 38GB so it became worse. I tried to reproduce the issue and can confirm that if I do
in a separate xterm while running the build time tests I get something like
while the test suite outputs something like
Please note: I'm definitely not sure that this is the affected test neither that these are the only tests which push the memory limit way more far than other programs but I confirm that the reporter is right and that the issue can be found somewhere close here. I'd suggest some bisecting to find the actual test to find out the reason.
Kind regards, Andreas.
The text was updated successfully, but these errors were encountered: