Support Unicode in JSON input #5504
Conversation
| @@ -162,6 +185,57 @@ namespace misc_utils | |||
| val.push_back('\\');break; | |||
| case '/': //Slash character | |||
| val.push_back('/');break; | |||
| case 'u': //Unicode code point | |||
| if (it + 1 == buf_end || it + 2 == buf_end || it + 3 == buf_end || it + 4 == buf_end) | |||
| } | ||
| else | ||
| { | ||
| uint32_t dst = 0; |
There was a problem hiding this comment.
This portions looks like a:
uint16_t dst = 0;
for (unsigned count = 0; count < 4; ++count)
{
dst <<= 4;
const unsigned char tmp = isx[*++it];
CHECK_AND_ASSERT_THROW_MES(tmp != 0xff, "Bad unicode encoding");
dst |= tmp;
}Also note the uint16_t because that is the entire range that can be extracted here.
| val.push_back(0x80 | ((dst >> 6) & 0x3f)); | ||
| val.push_back(0x80 | (dst & 0x3f)); | ||
| } | ||
| else if (dst <= 0x10ffff) |
There was a problem hiding this comment.
This value range is not possible with 4 hex characters. Anything in this range is provided as a UTF-16 surrogate pair. This requires parsing two 16-bit values. If the first 16-bit value is between 0xD800–0xDBFF, then another 16-bit value must follow which represents the entire code point.
There was a problem hiding this comment.
Do you know how to encode code points > 0xffff ? All the examples I've fund use \uxxxx with 4 digits.
There was a problem hiding this comment.
Oh you mean this encoding is actually UTF-16, not raw code points ?
moneromooo-monero
force-pushed
the
euni
branch
2 times, most recently
from
d012094
to
4144dbf
Compare
moneromooo-monero
force-pushed
the
euni
branch
from
4144dbf
to
3e11bb5
Compare
|
Sorry for not re-reviewing this PR earlier, UTF-16 surrogate pairs are still not handled properly. Anything outside of the Basic Multilingual Plane will decode incorrectly (which are less common characters at least). |
|
I will try to update it with tests later this week. |
No description provided.