-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
html.entities mapping dicts need updating? #55322
Comments
In Python 3.2b2, html.entities.codepoint2name and name2codepoint only support the 252 HTML entity names defined in the HTML 4 spec from 1997. I'm wondering if there's a reason not to support W3C Recommendation 'XML Entity Definitions for Characters' http://www.w3.org/TR/xml-entity-names/ This standard contains significantly more characters, and it is noted in that spec that the HTML 5 drafts use that spec's entities. You can see the current HTML 5 'Named character references' here: http://www.w3.org/TR/html5/named-character-references.html#named-character-references If this is just a matter of somebody going in to do the grunt work, let me know. If startup costs associated with importing a huge dictionary are a concern, perhaps a more efficient type that enables the same lookup interface can be defined. If other reasons exist to not move in this direction, please do let me know! |
Supporting the ones in HTML 5 would be fine with me. Supporting those of xml-entity-names would be inappropriate - it's not clear (to me, at least) that all of them are really meant for use in HTML. |
Agreed with Martin. I wonder if we should provide a means to use only HTML 4.01 entity references (say with a function parameter html5 defaulting to True) or we should just update the mapping. |
I don't see the need for a parameter to support different sets of entities. Just supporting the ones from HTML 5 seems like the right thing. |
To make my intent explicit: an updated mapping could generate references invalid for 4.01. |
Ah. I hadn't thought of generating them, only parsing them. In that case, then yes, it's an issue for generation. |
I just closed bpo-12329 as a duplicate of this bug. It requested the addition of the apos named entity reference. TTBOMK, the html module (or htmlentitydefs in 2.x) doesn’t claim to support XHTML TTBOMK; an XML parser should be used for XHTML. In HTML 4.01, apos is not defined, but it is in HTML5. |
The reason I raised bpo-12329 was that the v2.7.1 documentation in |
BTW, the HTMLParser module (as well as html.parser in 3.x) does claim to parse both HTML and XHTML, see http://docs.python.org/library/htmlparser.html#module-HTMLParser . |
Ah, this changes the situation. I suppose it’s too late to stop pretending that HTML and XHTML are nearly the same thing (IOW change the doc), so apos needs to be defined for XHTML. IMO, we need a way to have the right entity references for HTML 4.01, XHTML 1.0 and HTML5, not put them all in one mapping. |
Having them in different mappings would be good, but I expect that for most real world application a single mappings that includes them all is the way to go. If I'm parsing a supposedly HTML page that contains an ' I'd rather have it converted even if it's not an HTML entity. |
http://www.w3.org/TR/html5/named-character-references.html lists 2152 HTML 5 entities (see also attached file for a dict generated from that table).
In HTML 5, some of the entities map to a sequence of 2 characters, for example ≂̸ corresponds to [U+2242, U+0338] (i.e. MINUS TILDE + COMBINING LONG SOLIDUS OVERLAY). This means that:
Since 1) is not backward-compatible the HTML5 entities should probably go in a separate dict. Also note that the entities are case-sensitive and some of them include different spellings (e.g. both 'amp' and 'AMP' map to '&'), so the reverse dict won't work too well. Having '&' -> 'amp' seems better than '&' -> 'AMP', but this might not be obvious for all the entities and requires some extra logic in the code to get it right. |
+1 for a separate dict; -1 for a value list. The right value type is As for the reverse mapping: I'd add a dictionary that is reverse to OTOH, it's easy enough to create an inverted dictionary yourself |
Attached another file with a dict that contains the 2231 HTML5 entities listed at http://www.w3.org/TR/html5/named-character-references.html html5namedcharref = {
'Aacute;': '\xc1',
'Aacute': '\xc1',
'aacute;': '\xe1',
'aacute': '\xe1',
'Abreve;': '\u0102',
'abreve;': '\u0103',
...
} A better name could be found for the dict if you have better ideas (maybe html.entities.html5 only?). The dict will be added to html.entities. |
Here is a proper patch, still using the html5namedcharref name. |
How about calling it just "html5", or "HTML5"? That it is about entities already follows from the module name. |
Here's a new patch that uses the "html5" name for the dict, if there aren't other comments I'll commit it. |
New changeset 2b54e25d6ecb by Ezio Melotti in branch 'default': |
The ';' is not part of the entity name but an SGML delimiter, like '&'; the strings in the dict should not include it (like in the other dict they don’t). |
BTW in the doc you may point to collections.ChainMap to explain to people how to make one dict with HTML 4 and HTML 5 entities. (Note that I assume there are two dicts, but I only skimmed the diff.) |
The problem is that the standard allows some charref to end without a ';', but not all of them. So both "Éric" and Éric" will be parsed as "Éric", but only "αcentauri" will result in "αcentauri" -- "&alphacentauri" will be returned unchanged. I'm now working on bpo-15156 to use this dict in HTMLParser, and detecting the ';'-less entities is not easy. A possible solution is to keep the names that are accepted without ',' in a separate (private) dict and expose a function like HTMLParser.unescape that implements all the necessary logic. Regarding ChainMap, the html5 dict should be a superset of the html4 one. |
The explanations make sense, don’t change anything. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: