Error on lone surrogates

Author: G4Vi

README

@@ -293,6 +293,16 @@ A FEW NOTES ON UNICODE AND PERL
         Unicode RFC 7159 section 8.1. We use now the UTF8_DISALLOW_SUPER
         flag when parsing unicode.
 
+    8. Lone surrogates or illegal surrogate pairs are disallowed.
+        Since RFC 3629, U+D800 through U+DFFF are not legal Unicode values
+        and their UTF-8 encodings must be treated as an invalid byte
+        sequence. RFC 8259 section 8.2 admits the spec allows string values
+        that contain bit sequences that cannot encode Unicode characters and
+        that the behavior of software that recieves such values is
+        unpredictable. To avoid introducing non-Unicode strings into Perl we
+        use the UTF8_DISALLOW_SURROGATE flag when parsing Unicode and verify
+        escaped surrogates form valid pairs.
+
     I hope this helps :)
 
 OBJECT-ORIENTED INTERFACE
@@ -1809,6 +1819,15 @@ ENCODING/CODESET FLAG NOTES
     Unicode RFC 7159 section 8.1. We use now the UTF8_DISALLOW_SUPER flag
     when parsing unicode.
 
+    Since RFC 3629, U+D800 through U+DFFF are not legal Unicode values and
+    their UTF-8 encodings must be treated as an invalid byte sequence. RFC
+    8259 section 8.2 admits the spec allows string values that contain bit
+    sequences that cannot encode Unicode characters and that the behavior of
+    software that recieves such values is unpredictable. To avoid
+    introducing non-Unicode strings into Perl we use the
+    UTF8_DISALLOW_SURROGATE flag when parsing Unicode and verify escaped
+    surrogates form valid pairs.
+
     If you know of other incompatibilities, please let me know.
 
   JSON and YAML

XS.pm

@@ -362,6 +362,17 @@ characters" RFC 7159 section 1 and "JSON text SHALL be encoded in
 Unicode RFC 7159 section 8.1. We use now the UTF8_DISALLOW_SUPER
 flag when parsing unicode.
 
+=item 8. Lone surrogates or illegal surrogate pairs are disallowed.
+
+Since RFC 3629, U+D800 through U+DFFF are not legal Unicode values and
+their UTF-8 encodings must be treated as an invalid byte sequence.
+RFC 8259 section 8.2 admits the spec allows string values that contain
+bit sequences that cannot encode Unicode characters and that the
+behavior of software that recieves such values is unpredictable. To
+avoid introducing non-Unicode strings into Perl we use the
+UTF8_DISALLOW_SURROGATE flag when parsing Unicode and verify escaped
+surrogates form valid pairs.
+
 =back
 
 I hope this helps :)
@@ -1997,6 +2008,15 @@ characters" RFC 7159 section 1 and "JSON text SHALL be encoded in
 Unicode RFC 7159 section 8.1. We use now the UTF8_DISALLOW_SUPER
 flag when parsing unicode.
 
+Since RFC 3629, U+D800 through U+DFFF are not legal Unicode values and
+their UTF-8 encodings must be treated as an invalid byte sequence.
+RFC 8259 section 8.2 admits the spec allows string values that contain
+bit sequences that cannot encode Unicode characters and that the
+behavior of software that recieves such values is unpredictable. To
+avoid introducing non-Unicode strings into Perl we use the
+UTF8_DISALLOW_SURROGATE flag when parsing Unicode and verify escaped
+surrogates form valid pairs.
+
 If you know of other incompatibilities, please let me know.
 
 

XS.xs

@@ -551,17 +551,19 @@ decode_utf8 (pTHX_ unsigned char *s, STRLEN len, int relaxed, STRLEN *clen)
       return ((s[0] & 0x1f) << 6) | (s[1] & 0x3f);
     }
   else {
-/* Since perl 5.14 we can disallow illegal unicode above U+10FFFF.
+/* Since perl 5.14 we can disallow surrogates and illegal unicode above
+   U+10FFFF.
    Before we could only warn with warnings 'utf8'.
-   We accept only valid unicode, unless we are in the relaxed mode,
-   which allows SUPER, above U+10FFFF.
+   Surrogates are never allowed for consistency with unpaired escaped surrogate
+   handling.
+   SUPER, above U+10FFFF is not allowed, unless we are in the relaxed mode.
 */
 #if PERL_VERSION > 36
     UV c = utf8n_to_uvchr (s, len, clen,
-                          UTF8_CHECK_ONLY | (relaxed ? 0 : UTF8_DISALLOW_SUPER));
+                          UTF8_CHECK_ONLY | UTF8_DISALLOW_SURROGATE | (relaxed ? 0 : UTF8_DISALLOW_SUPER));
 #elif PERL_VERSION > 12
     UV c = utf8n_to_uvuni (s, len, clen,
-                           UTF8_CHECK_ONLY | (relaxed ? 0 : UTF8_DISALLOW_SUPER));
+                           UTF8_CHECK_ONLY | UTF8_DISALLOW_SURROGATE | (relaxed ? 0 : UTF8_DISALLOW_SUPER));
 #elif PERL_VERSION >= 8
     UV c = utf8n_to_uvuni (s, len, clen, UTF8_CHECK_ONLY);
 #endif

t/30_jsonspec.t

@@ -9,14 +9,14 @@ my $json    = Cpanel::JSON::XS->new->utf8->allow_nonref;
 my $relaxed = Cpanel::JSON::XS->new->utf8->allow_nonref->relaxed;
 
 # fixme:
-#  n_string_UTF8_surrogate_U+D800     ["EDA080"] <=> [""] unicode
 # done:
 #  i_string_unicode_*_nonchar  ["\uDBFF\uDFFE"] (add warning as in core)
 #  i_string_not_in_unicode_range  Code point 0x13FFFF is not Unicode UTF8_DISALLOW_SUPER
 #  y_string_utf16, y_string_utf16be, y_string_utf32, y_string_utf32be fixed with 3.0222
+#  n_string_UTF8_surrogate_U+D800 Code point 0xD800 is not Unicode UTF8_DISALLOW_SURROGATE
 my %todo;
+$todo{'n_string_UTF8_surrogate_U+D800'}++      if $] < 5.014;
 $todo{'y_string_nonCharacterInUTF-8_U+FFFF'}++ if $] < 5.013;
-$todo{'n_string_UTF8_surrogate_U+D800'}++      if $] >= 5.012;
 if ($] < 5.008) {
   # 5.6 has no multibyte support
   $todo{$_}++ for qw(