urllib.request.Request accepts but doesn't check bytes headers

[ezio-melotti]

BPO	29891
Nosy	@orsenthil, @ezio-melotti, @vadmium, @soltysh

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-03-23.22:03:56.696>
labels = ['3.7', 'type-bug', 'library']
title = "urllib.request.Request accepts but doesn't check bytes headers"
updated_at = <Date 2017-03-25.01:01:46.654>
user = 'https://github.com/ezio-melotti'

bugs.python.org fields:

activity = <Date 2017-03-25.01:01:46.654>
actor = 'martin.panter'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-03-23.22:03:56.696>
creator = 'ezio.melotti'
dependencies = []
files = []
hgrepos = []
issue_num = 29891
keywords = []
message_count = 2.0
messages = ['290063', '290457']
nosy_count = 4.0
nosy_names = ['orsenthil', 'ezio.melotti', 'martin.panter', 'maciej.szulik']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29891'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[ezio-melotti]

urllib.request.Request allows the user to create a request object like:
req = Request(url, headers={b'Content-Type': b'application/json'})

When calling urlopen(req, data), urllib will check if a 'Content-Type' header is present and fail to recognize b'Content-Type' because it's bytes.
urrlib will therefore add the default Content-Type 'application/x-www-form-urlencoded', and the request will then be sent with both Content-Types. This will result in difficult-to-debug errors because the server will sometimes pick one and sometimes the other, depending on the order.

urllib should either reject bytes headers, or check for both bytes and strings. The docs also don't seem to specify that the headers should be strings.

[vadmium]

If you enable BytesWarning (python -b) you do get an error:

>>> urlopen(req, data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/urllib/request.py", line 162, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 463, in open
    req = meth(req)
  File "/usr/lib/python3.5/urllib/request.py", line 1171, in do_request_
    if not request.has_header('Content-type'):
  File "/usr/lib/python3.5/urllib/request.py", line 356, in has_header
    return (header_name in self.headers or
BytesWarning: Comparison between bytes and string

I believe the “urllib.request” module is only written with text (str) field names in mind, not byte strings. Same for http.client.HTTPConnection.request(headers=...). But the lower-level HTTPConnection.putheader method has special code to handle byte strings: <http://svn.python.org/view?view=revision&revision=58823\>, although this is not documented either.