urllib.parse.parse_qsl does not handle unicode data properly

[maxking]

BPO	30483
Nosy	@maxking, @csabella, @cyrkov

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-05-26.09:49:46.950>
labels = ['3.8', 'type-bug', '3.7']
title = 'urllib.parse.parse_qsl does not handle unicode data properly'
updated_at = <Date 2020-04-16.10:52:01.356>
user = 'https://github.com/maxking'

bugs.python.org fields:

activity = <Date 2020-04-16.10:52:01.356>
actor = 'cyrkov'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2017-05-26.09:49:46.950>
creator = 'maxking'
dependencies = []
files = []
hgrepos = []
issue_num = 30483
keywords = []
message_count = 3.0
messages = ['294541', '318050', '366592']
nosy_count = 3.0
nosy_names = ['maxking', 'cheryl.sabella', 'cyrkov']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30483'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

Linked PRs

• gh-74668: Fix encoded unicode in url byte string #93757
• gh-74668: Fix support of bytes in urllib.parse.parse_qsl() #115771
• [3.12] gh-74668: Fix support of bytes in urllib.parse.parse_qsl() (GH-115771) #116366
• [3.11] gh-74668: Fix support of bytes in urllib.parse.parse_qsl() (GH-115771) #116367

[maxking]

After decoding percentage encoded name and values in the query string, it tries to _coerce_result or encode the result to ascii (which is the value of _implicit_encoding).

  File "/usr/lib/python3.6/urllib/parse.py", line 691, in parse_qsl
    value = _coerce_result(value)
  File "/usr/lib/python3.6/urllib/parse.py", line 95, in _encode_result
    return obj.encode(encoding, errors)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 1: ordinal not in range(128)

As seen in the partial traceback above, it breaks things when trying to parse unicode encode query string values.

[csabella]

Would you be able to include an example for recreating this?

Looking at the code, it uses the ascii encoding for bytes (which can only contain ASCII literal characters) and should not be using that encoding for strings.

Thanks!

[cyrkov]

I have recently stumbled upon this bug, and I can present the example and a solution I've used.
The issue happens when we try to parse x-www-form-urlencoded of type bytes:

>>> from urllib.parse import urlencode, parse_qs
>>> urlencode([('v', 'ö')])
'v=%C3%B6'
>>> parse_qs('v=%C3%B6')
{'v': ['ö']}
>>> parse_qs(b'v=%C3%B6')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.6/urllib/parse.py", line 669, in parse_qs
    encoding=encoding, errors=errors)
  File "/usr/lib64/python3.6/urllib/parse.py", line 722, in parse_qsl
    value = _coerce_result(value)
  File "/usr/lib64/python3.6/urllib/parse.py", line 103, in _encode_result
    return obj.encode(encoding, errors)
UnicodeEncodeError: 'ascii' codec can't encode character '\xf6' in position 0: ordinal not in range(128)

This happens in the parse_qsl function because _coerce_result is a synonym of _encode_result and is called with default parameter encoding='ascii'. As far as I understand, it should be called with the encoding parameter of the parse_qsl function:

742c742
<             name = _coerce_result(name)
---
>             name = _coerce_result(name, encoding=encoding, errors=errors)
745c745
<             value = _coerce_result(value)
---
>             value = _coerce_result(value, encoding=encoding, errors=errors)

I am not sure whether I should commit this to the repo and create a pull request, as described in the devguide.

[frostburn]

Can confirm that this issue exists in Python 3.8.10 and that cyrkov's solution works. (I manually re-implemented parse_qsl as a quick fix.)

[michaelkedar]

I've also run into this issue on Python 3.11.5 in code that calls parse_qsl on a flask.request.query_string that contains %-encoded Unicode characters

[serhiy-storchaka]

Both decoding and encoding can fail or lose information. To avoid this we should either use the lossless encoding or error handler ('latin1' or 'surrogateescape') for both directions, or omit decoding and encoding at all. The latter is usually more efficient, and can even be simpler, like in this case. #115771 supports arbitrary raw and percent-encoded bytes.