BufferedReader may issue additional read, may cause hang when backed by blocking socket

[jvmiller]

BPO	9550
Nosy	@pitrou
Files	bufio.py: Test Case / Reproduce Server Hang

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-08-11.13:41:29.956>
created_at = <Date 2010-08-09.22:08:59.150>
labels = ['type-bug', 'expert-IO']
title = 'BufferedReader may issue additional read, may cause hang when backed by blocking socket'
updated_at = <Date 2010-08-11.13:41:29.927>
user = 'https://bugs.python.org/jvmiller'

bugs.python.org fields:

activity = <Date 2010-08-11.13:41:29.927>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2010-08-11.13:41:29.956>
closer = 'pitrou'
components = ['IO']
creation = <Date 2010-08-09.22:08:59.150>
creator = 'jvmiller'
dependencies = []
files = ['18459']
hgrepos = []
issue_num = 9550
keywords = []
message_count = 3.0
messages = ['113489', '113495', '113599']
nosy_count = 2.0
nosy_names = ['pitrou', 'jvmiller']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9550'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']

[jvmiller]

While reading, BufferedReader can cause an additional read() to the underlying I/O object after the entire upper-layer byte count has been satisfied. This is unnecessary and troublesome in some cases.

In the event that the BufferedReader sits on top of a blocking socket (see the included test case,) certain network protocols (HTTP) can cause the server to hang in recv/recvfrom trying to read more data than will be available, causing a hang. I first ran into this issue running bottle on top of the core Python wsgi server.

It looks as though this may be present in 2.x as well, however an associate of mine wasn't able to trigger the issue in 2.x (even after implementing the makefile() code from 3.x)

To run the test case, start the server and then run the client against it.

bufio.py server
bufio.py client

I successfully patched this issue locally with the following:

---
Modules/_io/bufferedio.c:

self->pos = 0;
self->raw_pos = 0;
self->read_end = 0;

+ if ( remaining == 0 )
+ return res;
+

    while (self->read_end < self->buffer_size) {

---

Which aborts execution of the second loop in the event that the last block-sized read satisfied the upper layer call exactly.

[pitrou]

Thanks for caring about such issues. This would also need unit tests and test_io.py (and perhaps a similar change in _pyio.py, although the buffering logic there is different and may not exhibit the issue).

[pitrou]

The original patch wasn't good for all cases. I corrected it, added some tests and committed in r83944 (py3k), r83945 (3.1) and r83946 (2.7). Thank you!