asyncio Stream Reader Blocks on read when data fetched is less than limit

[skorpeo]

BPO	33662
Nosy	@asvetlov, @1st1
Files	pty_test.py: demonstraton of blocking behaviour

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-27.23:39:51.306>
created_at = <Date 2018-05-27.22:34:41.713>
labels = ['type-bug', 'invalid', 'expert-asyncio']
title = 'asyncio Stream Reader Blocks on read when data fetched is less than limit'
updated_at = <Date 2018-05-27.23:39:51.305>
user = 'https://bugs.python.org/skorpeo'

bugs.python.org fields:

activity = <Date 2018-05-27.23:39:51.305>
actor = 'yselivanov'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-27.23:39:51.306>
closer = 'yselivanov'
components = ['asyncio']
creation = <Date 2018-05-27.22:34:41.713>
creator = 'skorpeo'
dependencies = []
files = ['47615']
hgrepos = []
issue_num = 33662
keywords = []
message_count = 6.0
messages = ['317810', '317811', '317812', '317814', '317815', '317816']
nosy_count = 3.0
nosy_names = ['asvetlov', 'yselivanov', 'skorpeo']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33662'
versions = ['Python 3.6']

[skorpeo]

I humbly submit what I think may be a bug in the asyncio.StreamReader.read() function. When n=-1 is supplied and the data is less than self._limit the read function creates a future and never wakes it up. I believe the culprit is

cpython/Lib/asyncio/streams.py

Line 632 in 9d3627e

while True:

. To fix the issue a condition is added to break out of the loop if the data read is less than the limit. I can only attach one file so I am providing the fix here for asyncio streams.py:

blocks.append(block)           # existing
if len(block) < self._limit:   # new
    break                      # new

I have also attached a test file that shows the blocking behavior which is alleviated with the above fix. Finally, I am not sure how to handle a situation where the data is exactly equal to the limit and no subsequent data is sent.

[1st1]

"if not block:" means EOF and replacing it with "if len(block) < self._limit:" would break everything.

There might be another bug here (with TTY) or, maybe, there's a bug in pty_test.py. For example, I don't see how you closing reader_pipe or writer_pipe.

[skorpeo]

Yes makes sense, it could be another bug with TTY or just my error and I
did preface that I am submitting this humbly. You are also correct that
there is no clean up for closing the pipes.

On Mon, May 28, 2018 at 1:45 AM, Yury Selivanov <report@bugs.python.org>
wrote:

Yury Selivanov <yselivanov@gmail.com> added the comment:

"if not block:" means EOF and replacing it with "if len(block) <
self._limit:" would break everything.

There might be another bug here (with TTY) or, maybe, there's a bug in
pty_test.py. For example, I don't see how you closing reader_pipe or
writer_pipe.

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue33662\>

---

[1st1]

You are also correct that there is no clean up for closing the pipes.

It's not just about the cleanup. If you don't close the pipes, they will be open forever, so there will be no EOF for which read(-1) will wait forever.

[skorpeo]

yes, in this case they were meant to stay open to write and read multiple
messages. I was hoping to read data when it is available, the other work
around was to specify n, but that also blocked once there was no more data
to be fetched....Either way I will stick to queues and this appears to be
expected behavior so you can disregard the report. I should have known
better that I wasn't going to find a bug in python :)

On Mon, May 28, 2018 at 2:03 AM, Yury Selivanov <report@bugs.python.org>
wrote:

Yury Selivanov <yselivanov@gmail.com> added the comment:

> You are also correct that there is no clean up for closing the pipes.

It's not just about the cleanup. If you don't close the pipes, they will
be open forever, so there will be no EOF for which read(-1) will wait
forever.

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue33662\>

---

[1st1]

I should have known better that I wasn't going to find a bug in python :)

We have a lot of bugs, as any other software :) Don't be afraid to open issues.

I'll close this bug, feel free to re-open.