New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Popen returns pipes that don't support O_NONBLOCK #222

Closed
dekkers opened this Issue Dec 9, 2012 · 6 comments

Comments

Projects
None yet
5 participants
@dekkers

dekkers commented Dec 9, 2012

Popen from the gevent subprocess module returns pipes that don't support non-blocking mode. The following example demonstrates the problem:

import sys, os, fcntl, subprocess, select
from gevent import monkey

monkey.patch_all(subprocess=False)

p = subprocess.Popen([sys.executable, '-c',
                      'import time,sys;print "test";sys.stdout.flush();time.sleep(10)'],
                     stdout=subprocess.PIPE)
fcntl.fcntl(p.stdout, fcntl.F_SETFL, os.O_NONBLOCK)
select.select([p.stdout], [], [])
print p.stdout.read(1024)

Without monkey patching subprocess the code doesn't block, removing subprocess=False causes it to block. The reason is that p.stdout is a FileObjectPosix object instead of a normal File object and calling fcntl on the FileObjectPosix object doesn't really change anything: the underlying file descriptor is already non-blocking.

I don't really know what the right way to fix this is. We could add support for non-blocking I/O to FileObjectPosix, but that doesn't seem easy. The other option would be to change Popen to return normal File objects instead of FileObjectPosix objects.

@vitaly-krugl

This comment has been minimized.

Show comment
Hide comment
@vitaly-krugl

vitaly-krugl Dec 10, 2012

What happens if you use built-in python subprocess and set p.stdout NONBLOCK. Would p.stdout.read(1024) block in that case? The gevent behavior needs to be compatible with the built-in behavior.

vitaly-krugl commented Dec 10, 2012

What happens if you use built-in python subprocess and set p.stdout NONBLOCK. Would p.stdout.read(1024) block in that case? The gevent behavior needs to be compatible with the built-in behavior.

@dekkers

This comment has been minimized.

Show comment
Hide comment
@dekkers

dekkers Dec 10, 2012

p.stdout.read() doesn't block using the built-in subprocess, but it does blocks using gevent's subprocess. So yes, the behavior is different.

dekkers commented Dec 10, 2012

p.stdout.read() doesn't block using the built-in subprocess, but it does blocks using gevent's subprocess. So yes, the behavior is different.

@vladev

This comment has been minimized.

Show comment
Hide comment
@vladev

vladev Feb 5, 2014

The same issue bit me as well.

vladev commented Feb 5, 2014

The same issue bit me as well.

@jamadden

This comment has been minimized.

Show comment
Hide comment
@jamadden

jamadden Oct 16, 2015

Member

The implementation has changed substantially since this was written, but the basic issue remains. I've added clarifying remarks to the documentation, to wit:

Although this object provides a fileno() method and so can itself be passed to fcntl.fcntl(), setting the os.O_NONBLOCK flag will have no effect; likewise, removing that flag will cause this object to no longer be cooperative.

A workaround is to use p.stdout.fileio; that (undocumented!) object by definition is already in non-blocking mode. You'd get out of sync with any buffering performed at higher levels, though, and it only works in bytes mode.

Member

jamadden commented Oct 16, 2015

The implementation has changed substantially since this was written, but the basic issue remains. I've added clarifying remarks to the documentation, to wit:

Although this object provides a fileno() method and so can itself be passed to fcntl.fcntl(), setting the os.O_NONBLOCK flag will have no effect; likewise, removing that flag will cause this object to no longer be cooperative.

A workaround is to use p.stdout.fileio; that (undocumented!) object by definition is already in non-blocking mode. You'd get out of sync with any buffering performed at higher levels, though, and it only works in bytes mode.

@jamadden

This comment has been minimized.

Show comment
Hide comment
@jamadden

jamadden Oct 16, 2015

Member

I can imagine a workaround that involves a monkey-patch on fcntl. It's not especially pretty, but if there's demand for it (and especially if there's a quality PR) it's something we can consider. (It's too late for 1.1.0 though, maybe for 1.2.0.)

Member

jamadden commented Oct 16, 2015

I can imagine a workaround that involves a monkey-patch on fcntl. It's not especially pretty, but if there's demand for it (and especially if there's a quality PR) it's something we can consider. (It's too late for 1.1.0 though, maybe for 1.2.0.)

@koliyo

This comment has been minimized.

Show comment
Hide comment
@koliyo

koliyo May 17, 2016

Same problem here
The p.stdout.fileio hack works for now!

koliyo commented May 17, 2016

Same problem here
The p.stdout.fileio hack works for now!

@jamadden jamadden closed this in 53534f8 Oct 27, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment