Unicode Codepoint Escape Syntax

[hikari-no-yume]

https://wiki.php.net/rfc/unicode_escape

[masakielastic]

The range between U+D800 and U+DFFF should be invalid.

RFC 3639

The definition of UTF-8 prohibits encoding character numbers between U+D800 and U+DFFF, which are reserved for use with the UTF-16 encoding form (as surrogate pairs) and do not directly represent characters.

http://tools.ietf.org/html/rfc3629

The Unicode Standard 7.0, Chapter 3

D76 Unicode scalar value: Any Unicode code point except high-surrogate and low-surrogate code points.
• As a result of this definition, the set of Unicode scalar values consists of the ranges 0 to D7FF and E000 to 10FFFF, inclusive.

D92 UTF-8 encoding form: The Unicode encoding form that assigns each Unicode scalar value to an unsigned byte sequence of one to four bytes in length, as specified in Table 3-6 and Table 3-7.

Table 3-7. Well-Formed UTF-8 Byte Sequences

Code Points	First Byte	Second Byte	Third Byte	Fourth Byte
U+0000..U+007F	00..7F
U+0080..U+07FF	C2..DF	80..BF
U+0800..U+0FFF	E0	A0..BF	80..BF
U+1000..U+CFFF	E1..EC	80..BF	80..BF
U+D000..U+D7FF	ED	80..9F	80..BF
U+E000..U+FFFF	EE..EF	80..BF	80..BF
U+10000..U+3FFFF	F0	90..BF	80..BF	80..BF
U+40000..U+FFFFF	F1..F3	80..BF	80..BF	80..BF
U+100000..U+10FFFF	F4	80..8F	80..BF	80..BF

http://www.unicode.org/versions/Unicode7.0.0/ch03.pdf

[mathiasbynens]

@masakielastic It really depends. It’s true that lone surrogates are technically invalid, and UTF-8 doesn’t allow them to be encoded, but some languages (like JavaScript and JSON) still allow lone surrogates in strings. I could imagine a PHP script that outputs JSON that includes strings containing lone surrogates in some testing scenarios (although that may be enough of an edge case to ignore it).

Essentially this boils down to the question: should we use UTF-8 or WTF-8? (I don’t have a strong opinion on this.)

[hikari-no-yume]

I don't think we should disallow producing UTF-8-encoded surrogates, there might be legitimate applications for this.

[hikari-no-yume]

But if we are to allow them (it'd make CESU-8 handling easier, heh), that'd need to have a test and be in the language spec.

[mathiasbynens]

Let’s add a test for leading zeroes? E.g. \u{0000000000001F602}.

[hikari-no-yume]

Done!

[nikic]

Please also add tests to make sure that \u{+123} and \u{-123} are invalid. I suspect that right now the former will be accepted, while the latter will be rejected with a non-obvious error message.

[nikic]

Nevermind, those are rejected as you're validating the string before calling strtol. A test wouldn't hurt though :)

[hikari-no-yume]

Now that you mention it, I ought to use strtoul not strtol.

[hikari-no-yume]

Alright, done :)

[rquadling]

Just a bit of info ... http://graphemica.com/%F0%9F%98%82 ... we are matching Ruby.

[hikari-no-yume]

Ooh, I didn't know Ruby did that. I'll add that to the RFC.

[mathiasbynens]

@rquadling Graphemica isn’t the most reliable source – it displays the escape sequences for JavaScript, JSON, C, C++, Java, Python incorrectly. Better link (IMHO): http://codepoints.net/U+1F602

[mathiasbynens]

Looks like Ruby supports multiple code points between the braces (e.g. \u{20AC A3 A5}) so unlike ES6, it’s not exactly the same syntax.

[hikari-no-yume]

It also supports leading zeroes, as proposed for PHP. Does ES6?

[mathiasbynens]

It also supports leading zeroes, as proposed for PHP. Does ES6?

@TazeTSchnitzel Yes.

[hikari-no-yume]

Aha! :)

[hikari-no-yume]

The spec patch now states:

Implementations MUST allow Unicode codepoints that are not Unicode scalar values, such as high and low surrogates.