urllib2 maybe blocks too long with small chunks

[AnrsHu]

BPO	14562
Nosy	@orsenthil, @vadmium, @JimJJewett, @ZackerySpytz

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = <Date 2020-05-28.02:27:09.057>
created_at = <Date 2012-04-12.12:23:09.950>
labels = ['type-bug', 'library']
title = 'urllib2 maybe blocks too long with small chunks'
updated_at = <Date 2020-05-28.02:27:09.057>
user = 'https://bugs.python.org/AnrsHu'

bugs.python.org fields:

activity = <Date 2020-05-28.02:27:09.057>
actor = 'benjamin.peterson'
assignee = 'orsenthil'
closed = True
closed_date = <Date 2020-05-28.02:27:09.057>
closer = 'benjamin.peterson'
components = ['Library (Lib)']
creation = <Date 2012-04-12.12:23:09.950>
creator = 'Anrs.Hu'
dependencies = []
files = []
hgrepos = []
issue_num = 14562
keywords = []
message_count = 7.0
messages = ['158124', '158125', '158225', '158246', '165927', '239099', '370091']
nosy_count = 6.0
nosy_names = ['orsenthil', 'hongqn', 'martin.panter', 'Jim.Jewett', 'Anrs.Hu', 'ZackerySpytz']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue14562'
versions = ['Python 2.7']

[AnrsHu]

If HTTP URL response's Transfer-Encoding is 'Chunked', then the urllib2.urlopen(URL).readline() will block until there're enough 8192 bytes, even though the first chunk is just a line.

Every chunks should be processed as soon as posible, so the readline() behavior should read a line and return immediately, rather than read 8K data to buffer and look up a line from the buffer.

[orsenthil]

I am trying to this test this to determine the fault.

[jimjjewett]

It would be helpful to have a testcase, so that it will stay fixed.

[AnrsHu]

Okay, there's a test case of web.py:

Server codes are following:
import web
class index(object):
def GET(self):
yield 'hello\n'
yield 'world\n'
time.sleep(60)

client is Python interpreter
>>> resp = urllib.urlopen(URL)
>>> resp.readline() # will be 'hello'
>>> resp.readline() # will be 'world'
>>> resp.readline() # huh, it's blocked, and we to agree with it.

>> # but to use urllib2 will another behavor.
>> urllib2.urlopen(URL).readline() # huh, it's blocked even if 'hello' and 'world' returned yet. Because urllib2 uses a 8KiB buffer on socket._fileobjece within urllib2.py, it read 8K data to buffer first.

[orsenthil]

I had a discussion with Anrs on this, and it went along these lines -

I confused the buffering issue (encountered with streaming data) of urllib2 with chunked transfer encoding.

The flow will be blocked in the case at the socket level waiting for 8192 bytes. But this buffer size has been kept for buffered reading purposes of normal read scenarios.
However, in case of streaming data, this may be not the best way.

Here it is explained best -

http://stackoverflow.com/questions/1598331/how-to-read-continous-http-streaming-data-in-python

The advise is to make the socket buffer size to 0.

import socket

socket._fileobject.default_bufsize = 0

Now, if we come to chunked transfer encoding, the chunked transfer encoding will behave as it is advertised, like sending one chunk at the time, but still having the readline limit set by MAXLINE in the httplib.py. For the chunked transfer encoding to be recognized the client will have to get a header "transfer-encoding: chunked" from the server and when it receives that header, it will follow the path reading MAXLINE at the time and then returning. For smaller chunks with a blocking behavior of the server ( like you illustrated), we may still need to adopt to turn off default_bufsize to 0 to ensure quick responses to fill the buffer.

At this moment, I think that the above thing could be documented in the urllib2 docs for the issue you had raised. Not sure, if any other approach would be suitable to handle this behavior.

Anrs (The original poster) also responded that they way he had to overcome this for a very small chunks is setting the socket file size to 0 locally.

> resp = opener.open(server, urllib.urlencode(data))
> resp = opener.open(server)
> resp.fp._rbufsize = 0
> for line in iter(resp.readline, ''):
> yield line

I think, this could be documented in a certain fashion (like support for streaming without buffering or transfers for small data sizes without buffering).

[vadmium]

I can reproduce this with Python 2, but not with current Python 3, nor with v3.3.3. Probably doesn’t affect 3.2 either, but I haven’t tried.

[ZackerySpytz]

Python 2 is EOL, so I think this issue should be closed.