Wrong encoding used if charset only specified in content

[jdennis]

HTML pages which declared their charset encoding only in their content
had the wrong encoding applied because the content was never checked.

Currently only the request headers are checked for the charset
encoding, if absent the apparent_encoding heuristic is applied. But
the W3.org doc says one should check the header first for a charset
declaration, then if that's absent check the meta tags in the content
for a charset encoding declaration. It also says if no charset
encoding declaration is found one should assume UTF-8, not ISO-8859-1
(a bad recommendation from the early days of the web).

I have a patch (pull request), more details in the commit comment.

[Lukasa]

All relevant discussion is in #1589, so I'm closing this to centralise the discussion there. =)