setblocking() method on file objects

[dbaarda]

BPO	949667
Nosy	@dbaarda, @pitrou, @vstinner, @giampaolo, @tiran, @devdanzin, @briancurtin
Files	io_blocking.patch test_process.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-10-24.11:27:51.456>
created_at = <Date 2004-05-07.05:26:57.000>
labels = ['invalid', 'type-feature', 'library', 'expert-IO']
title = 'setblocking() method on file objects'
updated_at = <Date 2013-10-24.12:09:32.902>
user = 'https://github.com/dbaarda'

bugs.python.org fields:

activity = <Date 2013-10-24.12:09:32.902>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2013-10-24.11:27:51.456>
closer = 'vstinner'
components = ['Library (Lib)', 'IO']
creation = <Date 2004-05-07.05:26:57.000>
creator = 'abo'
dependencies = []
files = ['19384', '19385']
hgrepos = []
issue_num = 949667
keywords = ['patch']
message_count = 10.0
messages = ['60484', '82049', '82473', '113427', '113431', '119704', '178201', '192580', '201115', '201118']
nosy_count = 8.0
nosy_names = ['abo', 'pitrou', 'vstinner', 'giampaolo.rodola', 'christian.heimes', 'ajaksu2', 'brian.curtin', 'neologix']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue949667'
versions = ['Python 3.4']

[dbaarda]

Currently file object's write() method returns nothing.
It would be very nice if it returned the number of
bytes written. It would also be nice if file objects
had a setblocking() method, similar to socket objects.

Under various situations, such as communicating with
popen'ed processes, write() can block. To avoid this,
you must set the file to non-blocking mode, or do
something like fork a thread to service the process so
that it can "unblock" the write.

Currently, setting a file to unblocking mode requires
using fnctl on the file objects fileno(). It would be
nice if you could directly modify this file behaviour
using a setblocking() method.

When you do set a file to non-blocking mode, the file
object's write() method will raise an exception if not
all the data could be written. This is useless, as
there is no indication how much data was successfuly
written before the blocking happened, so you have no
idea how much data remains to be written when the file
becomes unblocked.

Even using things like select() loops are not reliable,
as write() on an unblocked file can block part way
through even a small data buffer.

The only sure way to avoid a file object's write() from
blocking or raising an unhelpful exception is to write
a single byte at a time. This means you must use
os.write() instead. The os.write() method returns the
number of bytes written, which tells how many bytes of
data remain to be written next time when the file
becomes unblocked.

The existing read() method behaves the same as the
os.read() method and can be used in non-blocking mode.
If the file write() method behaved the same as the
os.write() method, returning the number of bytes
written instead of raising an exception, then it could
also be used in non-blocking mode.

[devdanzin]

Implemented in 3.0, 2.x probably won't get this. Will close if nobody
opposes.

[pitrou]

setblocking() doesn't exist in py3k either, so reopening. I agree it
would be useful to set files as non-blocking in a portableway.

[giampaolo]

Any clue on where to start to do this?
What about Windows where fcntl is not available?

[tjguk]

There are at least two ways to do non-blocking file IO on Windows:

Overlapped I/O
I/O Completion ports

Don't know what's best here, but happy to see what might be achieved
if it was thought worth pursuing.

[vstinner]

Prototype to test nonblocking file objet:

• add getblocking() and setblocking() methods to _io._FileIO and all _pyio classes
• fileio_setblocking() is implemented using fcntl(fd, F_SETFL, flags | O_NONBLOCK) (POSIX only?)
• BufferedReader.read() uses read1() if the file is blocking

Use test_process.py to test it: this script runs a Python interpreter in a subprocess. It uses select() to check if there is data or not. Eg. type '1+1\n' and then 'exit()\n'. Set PYIO_HAVE_BLOCKING constant to False (in test_process.py) to test the script without io_blocking.patch.

I'm not sure that select() is required, but it doesn't work without it (read() blocks).

[neologix]

I'm not sure that a setblocking() method to fileobjects would make much sense, since non-blocking IO doesn't work with regular files (only pipes, sockets...).

[vstinner]

See the PEP-466 which proposes to add a new os.set_blocking() function on UNIX, and blocking parameter to socket constructor.

[vstinner]

There is a real need of non-blocking operation, but I now think that adding a set_blocking() method is not the right solution. First, as said by Charles-Francois: O_NONBLOCK flag has no effect on regular files. Second, there is no portable way to declare a file as "non blocking": as said by Tim, it is complelty different on Windows (I/O Completion ports).

The asyncio module has been merged into Python 3.4: you can now use it to access a file asynchroniously. I don't think that the asyncio is complete for your use case, but if something should done: it is in this module, only in this module, not the API of the io.FileIO class.

asyncio can be used to watch pipes of a subprocess asynchroniously.

[pitrou]

There is a real need of non-blocking operation, but I now think that
adding a set_blocking() method is not the right solution. First, as
said by Charles-Francois: O_NONBLOCK flag has no effect on regular
files. Second, there is no portable way to declare a file as "non
blocking": as said by Tim, it is complelty different on Windows (I/O
Completion ports).

That it's not portable is actually a good reason to add a helper function
or method.