httplib: header parsing is unlimited

[tiran]

BPO	16037
Nosy	@warsaw, @birkenfeld, @terryjreedy, @pitrou, @larryhastings, @tiran, @benjaminp, @berkerpeksag, @Lukasa
Files	issue16037_py27.patch issue16037_py32.patch issue16037_py26.patch issue16037_py27_v2.patch issue16037_py32_v2.patch issue16037_py32_v3.patch issue16037_py27_v3.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2014-09-30.12:50:04.031>
created_at = <Date 2012-09-25.10:25:22.415>
labels = ['release-blocker', 'library', 'performance']
title = 'httplib: header parsing is unlimited'
updated_at = <Date 2014-09-30.13:17:53.186>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2014-09-30.13:17:53.186>
actor = 'berker.peksag'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2014-09-30.12:50:04.031>
closer = 'georg.brandl'
components = ['Library (Lib)']
creation = <Date 2012-09-25.10:25:22.415>
creator = 'christian.heimes'
dependencies = []
files = ['29201', '29203', '31581', '31582', '31583', '32358', '36214']
hgrepos = []
issue_num = 16037
keywords = ['patch']
message_count = 31.0
messages = ['171240', '171250', '171251', '171258', '182194', '182803', '182805', '185055', '187276', '196862', '196898', '198610', '198618', '198619', '198620', '198621', '200349', '201162', '201255', '201424', '201429', '213240', '222210', '224568', '224802', '224803', '226078', '226079', '226110', '226112', '227890']
nosy_count = 14.0
nosy_names = ['barry', 'georg.brandl', 'terry.reedy', 'pitrou', 'larry', 'christian.heimes', 'benjamin.peterson', 'Arfrever', 'BreamoreBoy', 'nailor', 'python-dev', 'berker.peksag', 'puppet', 'Lukasa']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue16037'
versions = ['Python 3.2']

[tiran]

The httplib module / package can read arbitrary amounts of data from its socket when it's parsing the HTTP header. This may lead to issues when a user connects to a broken HTTP server or something that isn't a HTTP at all. The issue can be broken up into two parts: parsing the HTTP status line parsing and parsing the remaining HTTP headers.

Reading and parsing of the HTTP status line is already limited in Python 3.x. Python 2.7 and lower may read arbitrary amounts of bytes from the socket until it finds a newline char. The small patch below is a backport of the Python 3.x behavior to 2.7:

--- a/Lib/httplib.py
+++ b/Lib/httplib.py
@@ -362,7 +362,9 @@

     def _read_status(self):
         # Initialize with Simple-Response defaults
-        line = self.fp.readline()
+        line = self.fp.readline(_MAXLINE + 1)
+        if len(line) > _MAXLINE:
+            raise LineTooLong("header line")
         if self.debuglevel > 0:
             print "reply:", repr(line)
         if not line:

Both Python 2 and Python 3 accept an unlimited count of HTTP headers with a maximum length of 64k each. As headers are accumulated in an list it may consume lots of memory. I suggest that we limit the maximum amount of HTTP header lines to a sane value. How does 100 sound to you?

[python-dev]

New changeset 8a22a2804a66 by Christian Heimes in branch '2.7':
Issue bpo-16037: Limit httplib's _read_status() function to work around broken
http://hg.python.org/cpython/rev/8a22a2804a66

[tiran]

The readline() limitation in _read_status() was added at some point in the 3.2 line. Python 3.1 has an unlimited readline().

[pitrou]

100 headers sounds more than enough for everybody.

[tiran]

CVE-2013-1752 Unbound readline() DoS vulnerabilities in Python stdlib

[nailor]

Here's a patch that limits the headers to 100. If more than _MAXHEADERS headers are read, this raises exception TooMuchHeaders.

The patch is for 2.7, I'll cook one for 3.2 too.

[nailor]

...and here's the patch for 3.2

[benjaminp]

Not blocking 2.7.4 as discussed on mailing list.

[BreamoreBoy]

Patches LGTM but I suggest TooManyHeaders instead of TooMuchHeaders. I've tried the 3.2 patch against the latest default repo on Windows Vista and it applies cleanly. All tests passed so looks as if this could be committed.

[warsaw]

blocker for 2.6.9