Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows

[mattchaput]

BPO	11240
Nosy	@amauryfa, @pitrou, @applio

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-02-10.16:00:19.019>
created_at = <Date 2011-02-17.23:57:00.178>
labels = ['type-bug', 'OS-windows']
title = 'Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows'
updated_at = <Date 2015-02-10.16:00:19.017>
user = 'https://bugs.python.org/mattchaput'

bugs.python.org fields:

activity = <Date 2015-02-10.16:00:19.017>
actor = 'davin'
assignee = 'none'
closed = True
closed_date = <Date 2015-02-10.16:00:19.019>
closer = 'davin'
components = ['Windows']
creation = <Date 2011-02-17.23:57:00.178>
creator = 'mattchaput'
dependencies = []
files = []
hgrepos = []
issue_num = 11240
keywords = []
message_count = 13.0
messages = ['128768', '128770', '128771', '128772', '128794', '128797', '128799', '128803', '128804', '151479', '212742', '212745', '235692']
nosy_count = 8.0
nosy_names = ['amaury.forgeotdarc', 'pitrou', 'mu_mind', 'mattchaput', 'sbt', 'Chris.Jones', 'mdengler', 'davin']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11240'
versions = ['Python 2.7']

[mattchaput]

If you start unit tests with a command line such as "python setup.py test" or "nosetests", if the tested code starts a multiprocessing.Process on Windows, each new process will act as if it was started as "python setup.py test"/"nosetests", leading to an infinite explosion of processes that eventually locks up the entire machine.

[amauryfa]

Using multiprocessing on Windows can be different; please read http://docs.python.org/library/multiprocessing.html#windows especially the part named "Safe importing of main module".

On Windows, fork() does not exist, so a new interpreter must be started, which will import the current module; this must not start the test suite again!
Adding "if __name__ == '__main__'" somewhere is probably the solution.
If not, you should move the target function to another module.

[mattchaput]

Thank you, I understand all that, but I don't think you understand the issue. My code is not __main__. I am not starting the test suite. It's the distutils/nose code that's doing that.

It seems as if the multiprocessing module is starting new Windows processes by duplicating the command line of the original process. That doesn't seem to work very well, given the example of running test suites, hence the bug.

[amauryfa]

It seems as if the multiprocessing module is starting new Windows
processes by duplicating the command line of the original process.
It does not. The spawned processes use the command::

python.exe -c 'from multiprocessing.forking import main; main()' --multiprocessing-fork [handle#]

And only after, the multiprocessing machinery overrides sys.argv with the same value as the initial process.
There is certainly some code in one of your modules that starts running the tests.

[mattchaput]

I don't know what to tell you... to the best of my knowledge there's absolutely no way for my code to kick off the entire test suite -- I always do that through PyDev (which doesn't cause the bug, by the way). The closest thing is the boilerplate at the bottom of every test file:

if __name__ == "__main__":
    unittest.main()

...but even that would only start the tests in that file, not the entire suite.

Another thing that makes me think multiprocessing is re-running the original command line is that if I use "python setup.py test" to start the tests, when it gets to the MP tests it seems to run that command for each Process that gets started, but if I use "nosetests", it seems to run "nosetests" for each started Process.

[amauryfa]

Nose works correctly for me with multiprocessing. In a directory, I have:

== run_nose.py =================

from nose import main
if __name__ == '__main__':
    main()

================================

== test_me.py ==================

from multiprocessing import Pool
import os, time

def foo(x):
    time.sleep(0.1)
    return (x, os.getpid())

def test_me():
    pool = Pool(processes=4)
    x = pool.map(foo, range(10))
    a, b = zip(*x)
    print a, b
    assert list(a) == range(10)
    assert 1 < len(set(b)) <= 4

================================

Now when I do: "c:\python27\python run_nose.py" the test runs correctly.

Can you try this test in your environment?

[mattchaput]

If I do "c:\python27\python run_nose.py" it works correctly. If I do "nosetests" I get the process explosion. Maybe the bug is in how distutils and nose work from the command line? I'm confused.

[amauryfa]

Ah, is 'nosetests' a .exe file? A frozen executable?
In this case, can you try the solution proposed at http://docs.python.org/library/multiprocessing.html#multiprocessing.freeze_support

[pitrou]

If I do "c:\python27\python run_nose.py" it works correctly. If I do
"nosetests" I get the process explosion. Maybe the bug is in how
distutils and nose work from the command line? I'm confused.

The bug is in nose itself. You should report a bug there.

[ChrisJones]

You can work around this issue by using:
python.exe -c "import nose; nose.main()"
instead of nosetests

Note that nose.main() with no args parses sys.argv

[mattchaput]

IIRC the root issue turned out to be that when you execute any multiprocessing statements at the module/script level on Windows, you need to put it under if __name__ == "__main__", otherwise it will cause infinite spawning.

I think this is mentioned in the multiprocessing docs but should probably be in giant blinking red letters ;)

[mdengler]

the root issue turned out to be that when you execute any multiprocessing statements at the module/script level on Windows, you need to put it under if __name__ == "__main__", otherwise it will cause infinite spawning.

Same for me. The error message and failure mode are completely unhelpful, though.

I think this is mentioned in the multiprocessing docs but should probably be in giant blinking red letters ;)

Indeed. It would be even better if I or someone else had time to contribute a patch to fix the behaviour and, or at least the failure mode / error message. In a large codebase with multiple contributors it might not be so simple to track down the commit that caused the issue, especially if one is just starting out and the tests aren't clean.

[applio]

In some (but not necessarily all) circumstances, the multiprocessing module is now (in 2.7 and 3.x) able to detect the "infinite spawning" behavior described due to unsafe importing of the main module on Windows. The resulting error message looks like this:

RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

      This probably means that you are not using fork to start your
      child processes and you have forgotten to use the proper idiom
      in the main module:

              if __name__ == '__main__':
                  freeze_support()
                  ...

      The "freeze_support()" line can be omitted if the program
      is not going to be frozen to produce an executable.

Hopefully this behavior and the resulting message will both alert developers when such a problem is introduced in a project and communicate to them what the problem is. Giant blinking red letters aside, hopefully this pragmatic solution will do a better job of educating and helping developers notice and deal with such issues.

In the combinatoric explosion of possibilities that combine nosetest-binaries with PyDev with other useful dev tools, if this error message is being suppressed or otherwise not shared with the end user of those tools, that would warrant opening an issue with that tool.