Support Latin 1 <= UTF 8 (AVX)

[lemire]

No description provided.

[aqrit]

When "errors" are expected to be rare:
If we don’t distinguish between bad utf-8 and codepoints above 255 in the fast path…
then validation could use fewer instructions?

Converting from a valid (0xC2 0b10xxxxxx or 0xC3 0b10xxxxxx) sequence to latin1 should be just blend -> and -> or ?
Or does it not pay to do a compress_store?

Here is some half-baked, probably incorrect, pseudo code:
https://gist.github.com/aqrit/ebcbd13a43ac4ee4ef05578074ad3631

I’ve been wondering what to do when an issue is detected in the input stream,
maybe have a flag to:

• stop
• strip
• copy verbatim
• replacement byte

Also have a callback to "transliterate" unmapped codepoints?

[lemire]

Currently we simply do not support fully transcoding bad inputs (i.e. we return an error) but it is something planned for future versions.

The standard approach is to transcode with replacement: bad characters are replaced by a replacement character (possibly one that the user can specify).

[lemire]

Please see #147

[lemire]

“validation could use fewer instruction”

I expect so.

[lemire]

Closed via #301