Fix UTF-8 decode for 4bytes rune

[s-yasu]

Masked a character with a bit at 0x0F0000 position with MAX_4BYTE_RUNE (0x10FFFF) that bit was missing.
Such as 𤰖 (0x024C16).

[skvadrik]

Hi @s-yasu, thanks for finding this nasty bug!

I think your patch can be simplified a bit, in fact all that is needed is to change the value of MAX_4BYTE_RUNE from 0x10FFFFu to 0x1FFFFFu. The algorithm in re2c is an adaptation of re2 decoder with some validity checks thrown away (as re2c validates the input when lexing it), so I'd rather minimize the changes to it.

-const utf8::rune utf8::MAX_4BYTE_RUNE = 0x10FFFFu;
+const utf8::rune utf8::MAX_4BYTE_RUNE = 0x1FFFFFu;

After this change, the test passes with your newly added testcases.

Also, please can you can you return the assert in the test, as it is easy to overlook a non-zero return code, and much harder to overlook an assertion failure. Thanks again for the fix!

[s-yasu]

I think so that if MAX_4BYTE_RUNE change to 0x1FFFFFu, MAX_RUNE will has same value, and maximum value check in rune_to_bytes() may not be expected.

re2c/src/encoding/utf8/utf8.cc

Line 11 in ee125da

const utf8::rune utf8::MAX_4BYTE_RUNE = 0x10FFFFu;

const utf8::rune utf8::MAX_4BYTE_RUNE = 0x1FFFFFu;
const utf8::rune utf8::MAX_RUNE       = utf8::MAX_4BYTE_RUNE;

re2c/src/encoding/utf8/utf8.cc

Line 45 in ee125da

if (c > MAX_RUNE)

uint32_t utf8::rune_to_bytes(uint32_t *str, rune c)
{
       :
    if (c > MAX_RUNE)
        c = ERROR;

I understand to return the test to be the assert().

[skvadrik]

I think so that if MAX_4BYTE_RUNE change to 0x1FFFFFu, MAX_RUNE will has same value, and maximum value check in rune_to_bytes() may not be expected.

This explains the origin of the error, I must have thought that the maximum Unicode value 0x10FFFF and the 4-byte mask is the same value. MAX_RUNE should remain 0x10FFFF, but MAX_4BYTE_RUNE should be changed to 0x1FFFFF, because it is a bit mask that should match all code points up to 0x10FFFF (for example, 0xFFFFF is in this range, but it's not covered by bit mask 0x10FFFF).

[s-yasu]

OK.
I've fixed the branch so check it out.

[s-yasu]

Oops.
It's the same in the end, but a little fix.
(First commit's utf8_literals.c)

[skvadrik]

Great, thank you!