New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Child process running as debug on Windows #56307
Comments
I run this code: def myfunc(x):
assert False
#if __debug__: print 'debug'
return x - 1 if __name__ == '__main__':
pool = Pool(processes=1)
it = pool.imap(myfunc, xrange(5)) # or imap_unordered, map
print it.next() python -O myscript.py The myfunc() always raise AssertionError. But I run script with "-O" (optimization) command. Interpreter is: Thanks! |
I'm unable to reproduce this bug on Linux. $ cat script.py
from multiprocessing import Pool
import os def myfunc(x):
import sys
print("child", os.getpid(), "optimize?", sys.flags.optimize)
assert False, "assert False"
return x
if __name__ == '__main__':
import sys
print("parent optimize?", sys.flags.optimize)
pool = Pool(processes=2)
pool.map(myfunc, xrange(2)) # or imap_unordered, map $ python -O script.py
('parent optimize?', 1)
('child', 30397, 'optimize?', 1)
('child', 30398, 'optimize?', 1) |
In my system (Windows 7 (64) SP1, Python 2.6.6 32-bit) I have:
"""
d:\temp>python -O pool.py
('parent optimize?', 1)
('child', 4712, 'optimize?', 0)
(Traceback (most recent call last):
' File "new.py", line 14, in <module>
chil dpool.map(myfunc, xrange(2)) # or imap_unordered, map'
, File "C:\Python26\lib\multiprocessing\pool.py", line 148, in map
4712, 'optimize ?return self.map_async(func, iterable, chunksize).get()
' File "C:\Python26\lib\multiprocessing\pool.py", line 422, in get
, 0)
raise self._value
AssertionError: assert False
""" |
Under Linux, child processes are created with fork(), so they're run with the exact same environment as the parent process (among which sys.flags.optimize). |
This happens only on Windows, where multiprocessing has to spawn a new intepreter; the -O parameter is certainly omitted. |
I think that the problem is in Popen.get_command_line() of multiprocessing.forking. There are other options changing Python behaviour: -b, -B, -E, -u, etc. |
We already have some logic for that: |
I create patch for Popen.get_command_line() ('2.6' and 'default' branches). I don't know how to write the test. The sys.flags structure are read only. |
Thanks for the patches. |
I updated the patch. |
Ok, thanks for the patches. Some further comments:
|
Failure to build _multiprocessing will mean that multiprocessing cannot be imported. So if the function goes somewhere in multiprocessing then it makes running the test suite with multiple processes dependent on the building of _multiprocessing. Not sure if that is much of a problem since one can just run the test suite normally. Also I don't think "-i" (inspect/interactive) should be passed on: a child process's stdin is os.devnull. BTW, why isn't the use of "-u" (unbuffered stdout, stderr) reflected in sys.flags? |
I don't think that's a problem indeed.
Ah, you're right.
Don't know. sys.flags was introduced in bpo-1816, long after the -u flag itself. The code to reflect "-u" is commented out, perhaps Christian Heimes can shed some light on this. |
New changeset 347735ec92eb by Richard Oudkerk in branch 'default': |
Since multiprocessing also depends on threading, the change has broken the "AMD64 Fedora without threads 3.x" buildbot. I had not realized that the buildbots ran the test suite using multiple processes. I am not sure how best to refactor things -- I don't think multiprocessing should import test.support. Maybe we should just live with the duplication. |
They don't. It's only the import failing, so you should just catch and |
Sorry, I have misread. It actually happens inconditionally in |
New changeset 2034b3de1144 by Antoine Pitrou in branch 'default': |
Reopening this since this it needs backporting to 2.7 |
New changeset 37f52e71f4cd by Kristján Valur Jónsson in branch '2.7': |
Early indications are that this backport broke test running on the buildbots: http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%202.7/builds/1614 |
The buildbots page has been down for a while now, I can't see what's going on. |
New changeset b0b507268d7c by Kristján Valur Jónsson in branch '2.7': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: