New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running nosetests with multiprocessing tests on Windows leads to infinite explosion of processes #407

Open
jpellerin opened this Issue Dec 14, 2011 · 3 comments

Comments

Projects
None yet
3 participants
@jpellerin
Member

jpellerin commented Dec 14, 2011

Create the following two files in a directory:

== run_nose.py =================
from nose import main
if name == 'main':

main()

== test_me.py ==================
from multiprocessing import Pool
import os, time

def foo(x):
time.sleep(0.1)
return (x, os.getpid())

def test_me():
pool = Pool(processes=4)
x = pool.map(foo, range(10))
a, b = zip(*x)
print a, b
assert list(a) == range(10)

assert 1 < len(set(b)) <= 4

On Windows, if I go to the directory and type

c:\python27\python run_nose.py

...the tests run correctly. If I type

nosetests

...it causes an exponential explosion of processes

(Note, be careful if you try it, in my case I need to have a command window open with "taskkill /F /IM python.exe" ready, otherwise the computer locks up fairly quickly.)

This does not seem to be entirely nose specific since "python setup.py test" seems to do the same thing in my project. But I'm wondering if you have any insight into what's going on here.

Python 2.7, Windows 7, nosetests-script.py version 1.0.0

See http://bugs.python.org/issue11240 for more information.

Google Code Info:
Issue #: 398
Author: matt%who...@gtempaccount.com
Created On: 2011-02-18T21:48:33.000Z
Closed On:

@ghost ghost assigned jpellerin Dec 14, 2011

@jpellerin

This comment has been minimized.

Show comment
Hide comment
@jpellerin

jpellerin Dec 14, 2011

Member

Sorry, forgot to mention: What seems to be happening in both cases ("nosetests" and "python setup.py test") based on the printed output is that each multiprocessing.Process that gets started executes the original command line ("nosetests" or "python setup.py test"), so each process starts running the test suite, causing an infinite fan-out.

Google Code Info:
Author: matt%who...@gtempaccount.com
Created On: 2011-02-18T22:34:05.000Z

Member

jpellerin commented Dec 14, 2011

Sorry, forgot to mention: What seems to be happening in both cases ("nosetests" and "python setup.py test") based on the printed output is that each multiprocessing.Process that gets started executes the original command line ("nosetests" or "python setup.py test"), so each process starts running the test suite, causing an infinite fan-out.

Google Code Info:
Author: matt%who...@gtempaccount.com
Created On: 2011-02-18T22:34:05.000Z

@chrisejones

This comment has been minimized.

Show comment
Hide comment
@chrisejones

chrisejones Jan 17, 2012

You can work around this issue by using:
python.exe -c "import nose; nose.main()"
instead of nosetests

Note that nose.main() with no args parses sys.argv

chrisejones commented Jan 17, 2012

You can work around this issue by using:
python.exe -c "import nose; nose.main()"
instead of nosetests

Note that nose.main() with no args parses sys.argv

@syoung-m4

This comment has been minimized.

Show comment
Hide comment
@syoung-m4

syoung-m4 Jun 6, 2013

@jpellerin I have had the same problem as you. We were seeing this running through buildbot. I have yet to get the workaround to work in buildbot that @chrisejones has provided.

We run the buildbot on both a Windows 7 64-bit environment and a Red Hat 5 64-bit environment both running Python 2.7.3. The Red Hat experiences no problems but the Windows 7 exhibits the same issue you have described. We have also tested with nose 1.1.2 and nose 1.3.0 and both exhibit this behavior.

The workaround will be ok for now as long as I can make it work with buildbot. Hopefully, this will get fixed soon.

syoung-m4 commented Jun 6, 2013

@jpellerin I have had the same problem as you. We were seeing this running through buildbot. I have yet to get the workaround to work in buildbot that @chrisejones has provided.

We run the buildbot on both a Windows 7 64-bit environment and a Red Hat 5 64-bit environment both running Python 2.7.3. The Red Hat experiences no problems but the Windows 7 exhibits the same issue you have described. We have also tested with nose 1.1.2 and nose 1.3.0 and both exhibit this behavior.

The workaround will be ok for now as long as I can make it work with buildbot. Hopefully, this will get fixed soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment