connect_write_pipe doesn't work properly on fds that support bidirectional I/O

[ronf]

The connect_read_pipe() and connect_write_pipe() functions claim to work on character devices, sockets, and pipes. However, _UnixWritePipeTransport assumes that a write pipe has closed if _read_ready() ever triggers on it. This won't work for TTYs that allow both input & output on the same file descriptor. I think the same problem will occur with sockets.

I've been able to reproduce this failure by passing in sys.stdout to connect_write_pipe() when it is not redirected and pointing at /dev/tty. In that case, any input on stdin immediately called the write pipe opened on stdout to close.

Here's code which illustrates this:

import asyncio
import logging
import sys

logging.basicConfig(level='DEBUG')

class Reader(asyncio.Protocol):
    def data_received(self, data):
        print('Got data:', data)

class Writer(asyncio.BaseProtocol):
    pass

async def run(loop):
    stdin_t, _ = await loop.connect_read_pipe(Reader, sys.stdin)
    stdout_t, _ = await loop.connect_write_pipe(Writer, sys.stdout)

loop = asyncio.get_event_loop()
loop.run_until_complete(run(loop))

When run with PYTHONASYNCIODEBUG=1, I see the following output:

DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:asyncio:Read pipe 0 connected: (<_UnixReadPipeTransport fd=0 polling>, <__main__.Reader object at 0x103f62828>)
DEBUG:asyncio:Write pipe 1 connected: (<_UnixWritePipeTransport fd=1 idle bufsize=0>, <__main__.Writer object at 0x1046d9208>)
a    <--- Input provided here
INFO:asyncio:poll took 2221.794 ms: 2 events
Got data: b'a\n'
INFO:asyncio:<_UnixWritePipeTransport fd=1 idle bufsize=0> was closed by peer

[gvanrossum]

Can you suggest a fix, or do we have to update the docs?

--Guido (mobile)
On Jul 3, 2016 7:10 AM, "Ron Frederick" notifications@github.com wrote:

The connect_read_pipe() and connect_write_pipe() functions claim to work
on character devices, sockets, and pipes. However, _UnixWritePipeTransport
assumes that a write pipe has closed if _read_ready() ever triggers on it.
This won't work for TTYs that allow both input & output on the same file
descriptor. I think the same problem will occur with sockets.

I've been able to reproduce this failure by passing in sys.stdout to
connect_write_pipe() when it is not redirected and pointing at /dev/tty. In
that case, any input on stdin immediately called the write pipe opened on
stdout to close.

Here's code which illustrates this:

import asyncioimport loggingimport sys

logging.basicConfig(level='DEBUG')
class Reader(asyncio.Protocol):
def data_received(self, data):
print('Got data:', data)
class Writer(asyncio.BaseProtocol):
pass
async def run(loop):
stdin_t, _ = await loop.connect_read_pipe(Reader, sys.stdin)
stdout_t, _ = await loop.connect_write_pipe(Writer, sys.stdout)

loop = asyncio.get_event_loop()
loop.run_until_complete(run(loop))

When run with PYTHONASYNCIODEBUG=1, I see the following output:

DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:asyncio:Read pipe 0 connected: (<_UnixReadPipeTransport fd=0 polling>, <main.Reader object at 0x103f62828>)
DEBUG:asyncio:Write pipe 1 connected: (<_UnixWritePipeTransport fd=1 idle bufsize=0>, <main.Writer object at 0x1046d9208>)
a <--- Input provided here
INFO:asyncio:poll took 2221.794 ms: 2 events
Got data: b'a\n'
INFO:asyncio:<_UnixWritePipeTransport fd=1 idle bufsize=0> was closed by peer

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#369, or mute the thread
https://github.com/notifications/unsubscribe/ACwrMoWvFjQWMBGqWSNP6SBAd50r_edGks5qR8LUgaJpZM4JD4Fc
.

[ronf]

If we can find a different way to detect when a "write" pipe has been closed that doesn't rely on it being marked readable, it may be possible to continue to support sockets and other character devices. I'll experiment a bit when I get a chance and post my results here, or submit a PR if I can come up with a clean fix.

[ronf]

I took a closer look at this today. It appears that there's already code which disables this read callback on AIX for non-socket objects, relying on the caller to decide when to close the write pipe rather than signalling asynchronously when the other end of the pipe is closed. Here's the code I'm referring to:

        # On AIX, the reader trick (to be notified when the read end of the
        # socket is closed) only works for sockets. On other platforms it
        # works for pipes and sockets. (Exception: OS X 10.4?  Issue #19294.)
        if is_socket or not sys.platform.startswith("aix"):
            # only start reading when connection_made() has been called
            self._loop.call_soon(self._loop.add_reader,
                                 self._fileno, self._read_ready)

My suggestion is either to remove this read callback entirely (always relying on the caller to call write_eof() or close()), or restrict this callback so that it is only applies when dealing with actual pipe objects on non-AIX systems and never on sockets or character devices.

This callback currently applies on AIX for sockets, but I suspect that's a mistake, as it would have the same problem as other platforms where any data written back to the socket associated with the write pipe would falsely trigger a close.

[ronf]

It turns out this is complicated by the fact that _UnixSubprocessTransport uses socketpairs rather than pipes to avoid the issue on AIX that you can't select for read on a write pipe. In that specific use case, the fact that the socketpair is bidirectional isn't necessarily a problem, as nothing should be writing to the read side of that pair. However, it's difficult for connect_write_pipe() to tell the difference.

Since there are other methods already available to do asyncio on sockets, perhaps we should only support sockets which are dedicated socketpairs in connect_write_pipe(), requiring that the asyncio socket functions be used to do I/O on regular sockets. This would allow us to support selecting on read for all FIFOs and sockets, but exclude character devices. As a result, things like sys.stdin and sys.stdout (and other TTY devices) could be used with connect_read_pipe()/connect_write_pipe() without breaking the ability to get async notifications about remote pipe/socketpair closes.

[ronf]

With the merge of #374, this should now be fixed.