popen3 hangs on windows

[fresh]

BPO	527783
Nosy	@tim-one

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2002-03-20.22:55:43.000>
created_at = <Date 2002-03-09.13:13:46.000>
labels = ['invalid', 'library']
title = 'popen3 hangs on windows'
updated_at = <Date 2002-03-20.22:55:43.000>
user = 'https://bugs.python.org/fresh'

bugs.python.org fields:

activity = <Date 2002-03-20.22:55:43.000>
actor = 'tim.peters'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2002-03-09.13:13:46.000>
creator = 'fresh'
dependencies = []
files = []
hgrepos = []
issue_num = 527783
keywords = []
message_count = 12.0
messages = ['9588', '9589', '9590', '9591', '9592', '9593', '9594', '9595', '9596', '9597', '9598', '9599']
nosy_count = 3.0
nosy_names = ['tim.peters', 'dgallion', 'fresh']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue527783'
versions = []

[fresh]

The following hangs on windows:

(i,o,e)=popen('python test.py')
result=o.read()+e.read()

...where test.py is the test.py of a Zope 3 CVS
checkout. I suspected a Zope 3 problem, but Thomas
Guettler also expereinced this in a different context:

popen3() of the python (2.1.2) which comes with zope
hangs on W2K:

(stdin, stdout, stderr)=popen3('wvWare -x wvware.xml
foo.doc')
text=stdout.read()

Then again, having seen bug bpo-481896, I'm not sure this
is confined to windows.

Any ideas?

[dgallion]

Logged In: YES
user_id=68151

This cuased the problem on Linux.

import os, re
files="""
file 
errXX.py
<<<<<<<<<<<<<
import sys
while 1:
    print >>sys.stderr, 
'x'
<<<<<<<<<<<<<
file runXX.py
<<<<<<<<<<<<<
import 
os
(i,o,e)=os.popen3("python errXX.py")
print 
e.read()
<<<<<<<<<<<<<
cmd python 
runXX.py
<<<<<<<<<<<<<
"""
files=re.split("<<<<<+",files)
for 
x in range(0,len(files), 2):
    cmd=files[x].split()
    
body=files[x+1]
    if cmd[0]=='file':
        
open(cmd[1],'w').write(body)
    elif cmd[0]=='cmd':
        os.system(' 
'.join(cmd[1:])+body)
    else:
        assert(0)

[dgallion]

Logged In: YES
user_id=68151

Don't use e.read()
This will try to read the entire file. Which doesn't
make sense with a stream. Although I think this works with sockets?

I'll look at a select solution.
Not sure how to know how many bytes to
read after the select breaks out.

[fresh]

Logged In: YES
user_id=24723

So what should I use instead of e.read() ad o.read()

In fact, any chance of a replacement line for:

result=o.read()+e.read() 

? ;-)

cheers,

Chris

[tim-one]

Logged In: YES
user_id=31435

Chris, which version of Windows are you talking about? You
identified W2K for Thomas Guettler, but didn't say which
version you were using. The popen implementations are
radically different across Windows flavors, so knowing
which one you're using is important.

Alas, "important" doesn't necessarily imply helpful
<wink>. Historically, popen-on-Windows bugs don't get
fixed, because they hit a brick wall after blame gets
traced to MS internals.

[dgallion]

Logged In: YES
user_id=68151

os.read() doesn't block.
I'd expect you have to
call os.read in a loop.

i,o,e=os.popen3("../../python
errXX.py")
print os.read(o.fileno(),10000),
os.read(e.fileno(),10000)

[tim-one]

Logged In: YES
user_id=31435

If would help a lot if someone produced a small, self-
contained test case that reproduced the problem. The usual
outcome of *trying* to is the discovery that popen() on
Windows hangs "for no identifiable reason at all".
Example: there's a test_popen2.py in the std Python test
suite. It hangs about 1 time in 200 when I run it on
Win98SE. There's no identifiable cause; it just hangs
sometimes, and when it does it's always hung in the bowels
of MS's code. There are no races in the Python code
driving it.

[dgallion]

Logged In: YES
user_id=68151

I don't know about Win98*. But the example
given did hang in Linux also.

The
problem as posted here seems to be caused
by fp.read() blocking until the spawned
program exits. Using os.read() on Linux
avoids this problem. I'll give it a try on
Win2k and XP today.

[dgallion]

Logged In: YES
user_id=68151

http://ezwiki.com/2.54/python/popenTest.py

This test works on
Linux and Win2k.
It makes me think raw_input should flush
stdout.
It's tricky getting this sort of thing to work.

Looks like
test_popen2.py uses "more.exe" on Win98.
That might be a
problem?
The client program must flush stdout after each write.