Cope with Unicode properly where we try to

[skington]

This is PR 5 rewritten for a cleaner commit history. It fails the author tests, but I believe those fail in master as well.

[nigelm]

Thanks - have merged that and am trying to beat travis CI into shape :-/

Sorry for previously losing your pull request - its been a hectic period.

[nigelm]

Looks like this change breaks handling of non-breaking spaces.

See the test just produced - t/spaces.t

Problem appears to be that   gets mapped to a \xA0 in the character stream from the HTML parser. This then becomings a U+FFFD (Unicode replacement character) when it goes through the decode in _convert_spacelike_characters_to_space() - which is not matched by the character replacement regexp.

@skington - are you able to think of a way to handle this?

[skington]

I'll have a look; my gut instinct is that the HTML parser isn't Unicode-clean.

[skington]

The problem is that HTML::Parser isn't Unicode-clean, but whatever it uses for HTML entities is. (Note that _convert_spacelike_characters_to_space says "supplied with a string in bytes" - i.e. bytes rather than codepoints or graphemes.)

HTML::Parser->parse_file sets binmode on the filehandle and reads the contents in in 512-byte chunks, for instance, so anything we get from that route is going to be in UTF8 if we're lucky. It therefore makes sense to expect bytes, decode them from UTF8 in the if statement, and only if we see an actual \xA0 or \xAD once we've done that, assume that we actually did have UTF8-encoded text, make the appropriate subtitutions, and then turn the Unicode string back into bytes again.

A stop-gap solution would be to use utf8_mode to ensure that the entities matched the guessed-at file encoding. But IMO a proper solution would be to directly address the problem of encodings.