Python 2.2.3 crashes using popen2 to spawn lots of children

[frankty]

BPO	886492
Files	bug.zip: zip with test case, drwatson file and hack

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2006-03-23.08:50:17.000>
created_at = <Date 2004-01-28.20:33:45.000>
labels = ['OS-windows']
title = 'Python 2.2.3 crashes using popen2 to spawn lots of children'
updated_at = <Date 2006-03-23.08:50:17.000>
user = 'https://bugs.python.org/frankty'

bugs.python.org fields:

activity = <Date 2006-03-23.08:50:17.000>
actor = 'nnorwitz'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2004-01-28.20:33:45.000>
creator = 'frankty'
dependencies = []
files = ['1191']
hgrepos = []
issue_num = 886492
keywords = []
message_count = 2.0
messages = ['19842', '19843']
nosy_count = 2.0
nosy_names = ['nnorwitz', 'frankty']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue886492'
versions = ['Python 2.2']

[frankty]

WinPython 2.3.3 crashes when using popen2 to spawn
*lots* of children > 100.

Running the test case on either W2k or Wxp causes
Python 2.3.3 to crash. Looking at the drwatson file,
it appears to be a race condition where the child is in
the process of crashing, while the popen2 code in
posixmodule.c is attempting to PyFile_SetBufSize() on
one of the pipes. Evil juju in ntdll.dll ..

I've no idea what system resource is being exhausted
(heap memory?) causing the crash.

If you look at the hacks in posixmodule.c, the Python
crash can be eliminated by simply commenting out the
calls to PyFile_SetBufSize. There is still the problem
of resource exhaustion. To gain more child processes I
included a hack such that with the setting of an
env-var (NOPYCMD) popen2 will NOT use cmd.exe /c to run
a child, but simply run the child command as passed
(yeah, I know that is has to be an absolute path or in
the CWD then ..).

I'm sure there is a better way to eliminate the cmd.exe
/c - like maybe a new method: popen2.popenXabspath ..

In any case, it appears we have an NT kernel bug
combined with an inefficient way of doing popen2 that
is making a mess. I started doing some exploration of
heap_chunk_size to attack to resource issue but haven't
gotten very far.

As for *why* I'm doing this, it's because we're using
Pyton as a system simulator tool to emulate a large
pile of PCs. The actual work *must* be done by
external .exe code that cannot be changed.

The attached zip contains a test case, the posixmodule
hack, drwatson file and a sample program written in
VisC++ which does basically the same as the Python test
case.

cheers,

• Frank

[nnorwitz]

Logged In: YES
user_id=33168

Dupe of 886488