httplib gzip support

[mooonz]

BPO	1243678
Nosy	@birkenfeld, @birkenfeld, @devdanzin, @vadmium
Superseder	bpo-1508475: transparent gzip compression in urllib
Files	httplib.patch: diif with cvs rev 1.95

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-03-31.19:35:37.485>
created_at = <Date 2005-07-23.18:51:28.000>
labels = ['type-feature', 'library']
title = 'httplib gzip support'
updated_at = <Date 2015-05-27.22:41:12.022>
user = 'https://bugs.python.org/mooonz'

bugs.python.org fields:

activity = <Date 2015-05-27.22:41:12.022>
actor = 'martin.panter'
assignee = 'none'
closed = True
closed_date = <Date 2009-03-31.19:35:37.485>
closer = 'georg.brandl'
components = ['Library (Lib)']
creation = <Date 2005-07-23.18:51:28.000>
creator = 'mooonz'
dependencies = []
files = ['6739']
hgrepos = []
issue_num = 1243678
keywords = ['patch']
message_count = 12.0
messages = ['48607', '48608', '48609', '48610', '48611', '48612', '48613', '48614', '84694', '84880', '209139', '226398']
nosy_count = 6.0
nosy_names = ['georg.brandl', 'georg.brandl', 'sonderblade', 'ajaksu2', 'mooonz', 'martin.panter', 'Buck.Golemon']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'patch review'
status = 'closed'
superseder = '1508475'
type = 'enhancement'
url = 'https://bugs.python.org/issue1243678'
versions = ['Python 3.1', 'Python 2.7']

[mooonz]

Add gzip support for httplib. It seems to work
correctly - according to the tests I did done, but some
points should be altered (I think to the putrequest
method, where I didn't change anything - except the two
lines of comments who said that the gzip support is not
included)

[mooonz]

Logged In: YES
user_id=826215

There's no uploaded file! You have to check the
checkbox labeled "Check to Upload & Attach File"
when you upload a file. In addition, even if you
*did* check this checkbox, a bug in SourceForge
prevents attaching a file when *creating* an issue.

Please try again.

(This is a SourceForge annoyance that we can do
nothing about. :-( )

[mooonz]

Logged In: YES
user_id=826215

It's better with the patch ;)

[birkenfeld]

Logged In: YES
user_id=1188172

This will need documentation and test suite changes, too.

[sonderblade]

I have applied and tested this patch. The code seems to be
correct. Fetching gzipped data from www.mozilla.org works as it
should.

What the patch does is change Accept-Encoding from identity to
identity,gzip;q=0.9. That hints the HTTP server that we can handle
gzipped data. Some servers take the hint and sends us gzipped data
(www.mozilla.org is such a server). We check if that is so by checking
the Content-Encoding header in the HTTP response headers. If that is
set to gzip, the body of the response is gzipped data.

What then happens is that we create a hacked GzipFile object that
works on a wrapped version of the HTTPResponse itself. It has to be
hacked, because GzipFile works by seeking to the end of the file
object. That ofcourse does not work for us, because the whole file is
not available. But this hacked version employs some kind of StringIO
trick so that instead of seeking to the end of the file, it seeks to
the end of the read data.

So HTTPResponse aquires a reference to GzipFile2 which it reads
from. GzipFile2 in turn, has a reference to GzipedHTTPIO (the wrapper)
which in turn references the HTTPResponse. The read method in
HTTPResponse invokes the read method on GzipedFile2 which invokes the
read of GzipedHTTPIO which invokes the read of HTTPResponse. But
GzipedHTTPIO breaks the potential recursion by specifying raw=True
which means that it want HTTPResponse to feed it uncompressed data. A
very, very clever scheme.

I hope this information is useful. It took me way to long to get this
far. Originally I thought that this patch should be rejected because it
is just to damn complicated, but then I saw that the rest of
httplib.py is equally complicated. :)

[mooonz]

Wow, you digged up something I forgot for my own sake :)
Seriously, did I really made this horrible piece of crap ?
All my apologies for that...

Allthough, if sb is interested in gzip support in httplib, I should have something somewhere on my computer which should do the job in a better way (can't make worst, anyway ;)). It's a complete rewrite of the GzipFile class which doesn't need a random access to the data... I'll upload it tomorrow...

[birkenfeld]

Note that we already have a patch for that, in bpo-1675951.

[mooonz]

I saw it a few seconds after clicking the "submit" button ;)
OK, please forget my last message and this complete thread.

[devdanzin]

Should this be closed in favor of bpo-1675951?

[birkenfeld]

Sounds reasonable.