CVE-2019-18348: CRLF injection via the host part of the url passed to urlopen()

[ret2libc]

BPO	38576
Nosy	@gpshead, @vstinner, @larryhastings, @benjaminp, @ned-deily, @mcepl, @koobs, @stratakis, @miss-islington, @tirkarthi, @epicfaace, @ware, @ret2libc, @tapakund, @b1tninja
PRs	bpo-38576: Disallow control characters in hostnames in http.client #18995 [3.8] bpo-38576: Disallow control characters in hostnames in http.client (GH-18995) #19000 [3.7] bpo-38576: Disallow control characters in hostnames in http.client (GH-18995) #19001 [3.6] bpo-38576: Disallow control characters in hostnames in http.client (GH-18995) #19002 [2.7] closes bpo-38576: Disallow control characters in hostnames in http.client. #19052 [3.5] closes bpo-38576: Disallow control characters in hostnames in h… #19231 [3.5] closes bpo-38576: Disallow control characters in hostnames in h… #19300

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/benjaminp'
closed_at = <Date 2020-03-19.01:35:47.815>
created_at = <Date 2019-10-24.07:51:18.864>
labels = ['type-security', 'library', 'release-blocker']
title = 'CVE-2019-18348: CRLF injection via the host part of the url passed to urlopen()'
updated_at = <Date 2022-02-28.20:23:55.129>
user = 'https://github.com/ret2libc'

bugs.python.org fields:

activity = <Date 2022-02-28.20:23:55.129>
actor = 'ned.deily'
assignee = 'benjamin.peterson'
closed = True
closed_date = <Date 2020-03-19.01:35:47.815>
closer = 'benjamin.peterson'
components = ['Library (Lib)']
creation = <Date 2019-10-24.07:51:18.864>
creator = 'rschiron'
dependencies = []
files = []
hgrepos = []
issue_num = 38576
keywords = ['patch']
message_count = 13.0
messages = ['355294', '357073', '357442', '362353', '364190', '364191', '364192', '364193', '364207', '364208', '364499', '364584', '371922']
nosy_count = 18.0
nosy_names = ['gregory.p.smith', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'mcepl', 'python-dev', 'koobs', 'cstratak', 'miss-islington', 'xtreak', 'epicfaace', 'ware', 'rschiron', 'tapakund', 'Anselmo Melo', 'b1tninja', 'kim']
pr_nums = ['18995', '19000', '19001', '19002', '19052', '19231', '19300']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue38576'
versions = ['Python 2.7']

[ret2libc]

Copy-pasted from https://bugs.python.org/issue30458#msg347282

================
The commit b7378d7 is incomplete: it doesn't seem to check for control characters in the "host" part of the URL, only in the "path" part of the URL. Example:
---
try:
from urllib import request as urllib_request
except ImportError:
import urllib2 as urllib_request
import socket
def bug(*args):
raise Exception(args)
# urlopen() must not call create_connection()
socket.create_connection = bug
urllib_request.urlopen('http://127.0.0.1\\r\\n\\x20hihi\\r\\n :11211')
---

The URL comes from the first message of this issue:
https://bugs.python.org/issue30458#msg294360

Development branches 2.7 and master produce a similar output:
---

Traceback (most recent call last):
 ...
Exception: (('127.0.0.1\r\n hihi\r\n ', 11211), ..., None)

---

So urllib2/urllib.request actually does a real network connection (DNS query), whereas it should reject control characters in the "host" part of the URL.

---

A second problem comes into the game. Some C libraries like glibc strip the end of the hostname (strip at the first newline character) and so HTTP Header injection is still possible is this case:
https://bugzilla.redhat.com/show_bug.cgi?id=1673465

---

According to the RFC 3986, the "host" grammar doesn't allow any control character, it looks like:

   host          = IP-literal / IPv4address / reg-name

ALPHA (letters)
DIGIT (decimal digits)
unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"
pct-encoded = "%" HEXDIG HEXDIG
sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
/ "*" / "+" / "," / ";" / "="
reg-name = *( unreserved / pct-encoded / sub-delims )

IP-literal = "[" ( IPv6address / IPvFuture ) "]"
IPvFuture = "v" 1*HEXDIG "." 1*( unreserved / sub-delims / ":" )
IPv6address = 6( h16 ":" ) ls32
/ "::" 5( h16 ":" ) ls32
/ [ h16 ] "::" 4( h16 ":" ) ls32
/ [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32
/ [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32
/ [ *3( h16 ":" ) h16 ] "::" h16 ":" ls32
/ [ *4( h16 ":" ) h16 ] "::" ls32
/ [ *5( h16 ":" ) h16 ] "::" h16
/ [ *6( h16 ":" ) h16 ] "::"
h16 = 1*4HEXDIG
ls32 = ( h16 ":" h16 ) / IPv4address
IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet
================

CVE-2019-18348 was assigned to this flaw, which is similar to CVE-2019-9947 and CVE-2019-9740 but it is about the *host* part of a url.

[b1tninja]

Can't see the specifics of that "restricted" redhat bug, but this was interesting bug and I wanted to ask if perhaps the domain in such cases should be IDN / punycoded ://xn--n28h.ws/ for example is ://💩.la

[ret2libc]

The glibc issue mentioned in the first comment is CVE-2016-10739 .

[mcepl]

Just to say this is reproducible only on rather old enterprise Linux distributions, where CVE-2016-10739 bug in glibc has not been fixed. I believe it means RHEL-6, SUSE SLE-10, 11, 12 (not sure whether it applies to some old Debian as well).

[gpshead]

New changeset 9165add by Ashwin Ramaswami in branch 'master':
bpo-38576: Disallow control characters in hostnames in http.client (GH-18995)
9165add

[gpshead]

Thanks for the PR Ashwin!

[miss-islington]

New changeset 34f85af by Miss Islington (bot) in branch '3.7':
bpo-38576: Disallow control characters in hostnames in http.client (GH-18995)
34f85af

[miss-islington]

New changeset ff69c9d by Miss Islington (bot) in branch '3.8':
bpo-38576: Disallow control characters in hostnames in http.client (GH-18995)
ff69c9d

[ned-deily]

New changeset 83fc701 by Miss Islington (bot) in branch '3.6':
bpo-38576: Disallow control characters in hostnames in http.client (GH-18995) (GH-19002)
83fc701

[gpshead]

If anyone cares about 2.7, the *final* release is coming up in a few weeks. They'll need to figure out what it looks like there and get a 2.7 PR reviewed by the release manager.

[gpshead]

marking as a 2.7 release blocker just to get benjamin's RM attention before the final 2.7.

[benjaminp]

New changeset e176e0c by Matěj Cepl in branch '2.7':
[2.7] closes bpo-38576: Disallow control characters in hostnames in http.client. (GH-19052)
e176e0c

[larryhastings]

New changeset 09d8172 by Tapas Kundu in branch '3.5':
[3.5] closes bpo-38576: Disallow control characters in hostnames in http.client. (bpo-19300)
09d8172