HTMLParser cannot handle '&' and non-ascii characters in attribute names

[yanne]

BPO	3932
Nosy	@rhettinger, @ezio-melotti, @merwok, @bitdancer
Files	test.py: Fixed minimal script to produce the error HTMLParser-unescape-fix.diff: HTMLParser.unescape: return str value for str input issue3932-test.diff: Failing test

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ezio-melotti'
closed_at = <Date 2011-12-19.05:20:04.945>
created_at = <Date 2008-09-22.12:32:10.422>
labels = ['type-feature', 'docs']
title = "HTMLParser cannot handle '&' and non-ascii characters in attribute names"
updated_at = <Date 2011-12-19.05:20:04.944>
user = 'https://bugs.python.org/yanne'

bugs.python.org fields:

activity = <Date 2011-12-19.05:20:04.944>
actor = 'ezio.melotti'
assignee = 'ezio.melotti'
closed = True
closed_date = <Date 2011-12-19.05:20:04.945>
closer = 'ezio.melotti'
components = ['Documentation']
creation = <Date 2008-09-22.12:32:10.422>
creator = 'yanne'
dependencies = []
files = ['11690', '14606', '23621']
hgrepos = []
issue_num = 3932
keywords = ['patch']
message_count = 10.0
messages = ['73571', '73908', '74234', '74239', '91084', '96320', '147189', '148123', '148544', '149817']
nosy_count = 10.0
nosy_names = ['rhettinger', 'wiget', 'ezio.melotti', 'hodgestar', 'eric.araujo', 'yanne', 'r.david.murray', 'zchyla', 'sergiomb2', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue3932'
versions = ['Python 2.7']

[yanne]

It seems that HTMLParser.feed throws an exception whenever an attribute
name contains both quotation mark '&' and non-ascii characters.

Running the attached test file with Python 2.5 succeeds, but with Python
2.6, the result is:

C:\Python26>python.exe test.py
Without & in attribute
OK
With & in attribute
Traceback (most recent call last):
  File "test.py", line 18, in <module>
    HP().feed(s)
  File "C:\Python26\lib\HTMLParser.py", line 108, in feed
    self.goahead(0)
  File "C:\Python26\lib\HTMLParser.py", line 148, in goahead
    k = self.parse_starttag(i)
  File "C:\Python26\lib\HTMLParser.py", line 249, in parse_starttag
    attrvalue = self.unescape(attrvalue)
  File "C:\Python26\lib\HTMLParser.py", line 386, in unescape
    return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));",
replaceEntities, s)
  File "C:\Python26\lib\re.py", line 150, in sub
    return _compile(pattern, 0).sub(repl, string, count)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0:
ordinal
not in range(128)

I am running:

Python 2.6rc2 (r26rc2:66507, Sep 18 2008, 14:27:33) [MSC v.1500 32 bit
(Intel)] on win32

[hodgestar]

I can't reproduce this on current trunk (r66633, 27 Sep 2008). I checked
sys.getdefaultencoding() but that returned 'ascii' as expected and I
even tried language Python with "LANG=C ./python" but that didn't fail
either. Perhaps this has been fixed? It looks like it might originally
have been a problem in the re module from the traceback.

[yanne]

It seems that I managed to upload wrong test file the first time.

This attached test should fail, I tested it with Python2.6 final both on
Linux and Windows.

[hodgestar]

I've tracked down the cause to the .unescape(...) method in HTMLParser.
The replaceEntities function passed to re.sub() always returns a unicode
character, even when matching string s is a byte string. Changing line
383 to:

return self.entitydefs[s].encode("utf-8")

makes the test pass. Unfortunately this is obviously not a viable
solution in the general case. The problem is that there is no way to
know what character set to encode in without knowing both the HTTP
headers (which are not available to HTMLParser) and looking at the XML
and HTML headers.

Python 3.0 implicitly rejects non-unicode strings right at the start of
html.parser.HTMLParser.feed(...) by adding '' to the data passed in.

Given Python 3.0's behaviour, the docs should perhaps be updated to say
HTMLParser does not support non-unicode strings? If it should support
byte strings, we'll have to figure out how to handle encoded entity issues.

It's a bit weird that character and entity references outside
tags/attributes result in calls to .entityref(...) and .charref(...)
while those inside get unescape called automatically. Don't really see
what can be done about that though.

[zchyla]

Since HTMLParser.unescape in 2.5 returns str for str input, 2.6
should remain compatible. Therefore I propose the attached patch
(HTMLParser-unescape-fix.diff). With this patch applied the result
will have the same type as the input.

[sergiomb2]

the patch fix parsing in simple tag a with title with <br> ?! and
accents like this:

<a href="8999.html" title="<br>país">

[ezio-melotti]

I'm not sure what is the best solution here.

unescape uses a regex with replaceEntities as callback to replace the entities in attribute values.
The problem is that replaceEntities currently returns unicode, and if unescape receives a str, an automatic coercion to unicode happens and an error is raised whenever the str is non-ascii.

The possible solutions are:

1. Document the status quo (i.e replaceEntities always returns unicode, and an error is raised whenever a string that contains non-ascii chars is passed);
2. Change replaceEntities to return str only for ascii chars (as the patch proposed by Zbigniew does). This works as long as the entity resolves to an ascii character, but keep failing for the other cases.

The first option is cleaner, and means that if you want to parse something you should always use unicode, otherwise it might fail (In case of ambiguity, refuse the temptation to guess).
The second option might allow you to parse a few more documents without converting them to unicode, but only if you are lucky (i.e. you don't get any unicode mixed with non-ascii str). If most of the entities in attributes resolve to ascii (e.g. &quote; & ' > <), it might be more practical to return str and avoid unnecessary errors, while still adding a note in documentation that passing unicode is better.

[merwok]

+1 on refusing the temptation to guess and to be half-working for some cases by accident.

[ezio-melotti]

I'll change this in a doc issue then.

Any suggestions about the wording?
Adding "Passing unicode strings is suggested/advised/preferred." in the .feed() section is a bit vague, and mentioning the problem (with str it might break in some corner cases) while keeping a positive tone is somewhat difficult.

[python-dev]

New changeset 978f45013c34 by Ezio Melotti in branch '2.7':
bpo-3932: suggest passing unicode to HTMLParser.feed().
http://hg.python.org/cpython/rev/978f45013c34