why not using python builtin html unescape

[remitamine]

• I've verified and I assure that I'm running youtube-dl 2016.04.19
• At least skimmed through README and most notably FAQ and BUGS sections
• Searched the bugtracker for similar - [ ] Bug report (encountered problems with youtube-dl)
• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

moved from 9e6dd23#commitcomment-17012611.
why the html unescape that came with python http://stackoverflow.com/a/2360639 is not used it can handle two cases that the unescapeHTML can't(' and the ones that start with &#X) and it was improved in the last version(handle HTML5 named character references).

[yan12125]

I guess a reason is mentioned in #329:

_unescapeHTML did the same thing as htmlentity_transform but using the undocumented HTMLParser.unescape

[remitamine]

it can handle two cases that the unescapeHTML can't(' and the ones that start with &#X)

if _unescapeHTML did the same thing. it whould give the same result.

[yan12125]

The problem is undocumented. I guess it's OK as CPython is the de facto standard. Other Python implementations, such as PyPy and Jython, are moving towards using the same set of standard libraries. Ideas from other developers needed, though.

[phihag]

This function is really old. As long as the tests pass (we may need to add some to cover calling it with None and b'123'), feel free to replace its implementation. Note as @yan12125 , we should only use public python interfaces.

[yan12125]

If one day youtube-dl supports Python 3.4+ only, we can use it: https://docs.python.org/3/library/html.html#html.unescape. Closing now.

Something not directly related to this topic but also interesting: http://bugs.python.org/issue27032

[yan12125]

Since 55b2f09 and 9631a94 HTML5 entities are recognized. It works fine for Vporn. Are there other websites using HTML5 entites?