Skip to content

Commit 146af14

Browse files
G4Virurban
authored andcommitted
Remove non-character warnings (Fixes GH #227)
Aligns behavior with JSON::PP, no longer warns.
1 parent 2e29000 commit 146af14

5 files changed

Lines changed: 15 additions & 94 deletions

File tree

‎README‎

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -280,10 +280,11 @@ A FEW NOTES ON UNICODE AND PERL
280280

281281
5. A string containing "high" (> 255) character values is *not* a UTF-8
282282
string.
283-
6. Unicode noncharacters only warn, as in core.
283+
6. Raw non-Unicode characters below U+10FFFF are allowed.
284284
The 66 Unicode noncharacters U+FDD0..U+FDEF, and U+*FFFE, U+*FFFF
285-
just warn, see <http://www.unicode.org/versions/corrigendum9.html>.
286-
But illegal surrogate pairs fail to parse.
285+
are allowed without warning, as JSON::PP does, see
286+
<http://www.unicode.org/versions/corrigendum9.html>. But illegal
287+
surrogate pairs fail to parse.
287288

288289
7. Raw non-Unicode characters above U+10FFFF are disallowed.
289290
Raw non-Unicode characters outside the valid unicode range fail to
@@ -1802,11 +1803,6 @@ ENCODING/CODESET FLAG NOTES
18021803
This works because "__proto__" is not valid outside of strings, so every
18031804
occurrence of ""__proto__"\s*:" must be a string used as property name.
18041805

1805-
Unicode non-characters between U+FFFD and U+10FFFF are decoded either to
1806-
the recommended U+FFFD REPLACEMENT CHARACTER (see Unicode PR #121:
1807-
Recommended Practice for Replacement Characters), or in the binary or
1808-
relaxed mode left as is, keeping the illegal non-characters as before.
1809-
18101806
Raw non-Unicode characters outside the valid unicode range fail now to
18111807
parse, because "A string is a sequence of zero or more Unicode
18121808
characters" RFC 7159 section 1 and "JSON text SHALL be encoded in

‎XS.pm‎

Lines changed: 5 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -347,11 +347,12 @@ a Unicode string encoded in UTF-8, giving you a binary string.
347347
=item 5. A string containing "high" (> 255) character values is I<not>
348348
a UTF-8 string.
349349
350-
=item 6. Unicode noncharacters only warn, as in core.
350+
=item 6. Raw non-Unicode characters below U+10FFFF are allowed.
351351
352-
The 66 Unicode noncharacters U+FDD0..U+FDEF, and U+*FFFE, U+*FFFF just
353-
warn, see L<http://www.unicode.org/versions/corrigendum9.html>. But
354-
illegal surrogate pairs fail to parse.
352+
The 66 Unicode noncharacters U+FDD0..U+FDEF, and U+*FFFE, U+*FFFF are
353+
allowed without warning, as JSON::PP does, see
354+
L<http://www.unicode.org/versions/corrigendum9.html>. But illegal
355+
surrogate pairs fail to parse.
355356
356357
=item 7. Raw non-Unicode characters above U+10FFFF are disallowed.
357358
@@ -1990,11 +1991,6 @@ output for these property strings, e.g.:
19901991
This works because C<__proto__> is not valid outside of strings, so every
19911992
occurrence of C<"__proto__"\s*:> must be a string used as property name.
19921993
1993-
Unicode non-characters between U+FFFD and U+10FFFF are decoded either
1994-
to the recommended U+FFFD REPLACEMENT CHARACTER (see Unicode PR #121:
1995-
Recommended Practice for Replacement Characters), or in the binary or
1996-
relaxed mode left as is, keeping the illegal non-characters as before.
1997-
19981994
Raw non-Unicode characters outside the valid unicode range fail now to
19991995
parse, because "A string is a sequence of zero or more Unicode
20001996
characters" RFC 7159 section 1 and "JSON text SHALL be encoded in

‎XS.xs‎

Lines changed: 4 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,7 @@ mingw_modfl(long double x, long double *ip)
242242
#ifndef HvNAMEUTF8
243243
# define HvNAMEUTF8(hv) 0
244244
#endif
245+
#if 0
245246
/* since 5.14 check use warnings 'nonchar' */
246247
#ifdef WARN_NONCHAR
247248
#define WARNER_NONCHAR(hi) \
@@ -259,6 +260,9 @@ mingw_modfl(long double x, long double *ip)
259260
Perl_warner(aTHX_ packWARN(WARN_UTF8), \
260261
"Unicode non-character U+%04lX is illegal", (unsigned long)hi)
261262
#endif
263+
#else
264+
#define WARNER_NONCHAR(hi)
265+
#endif
262266

263267
/* since 5.16 */
264268
#ifndef GV_NO_SVGMAGIC
@@ -3531,15 +3535,6 @@ _decode_str (pTHX_ dec_t *dec, char endstr)
35313535
The WG's consensus was to leave the full range present
35323536
in the ABNF and add the interoperability guidance about
35333537
values outside the Unicode accepted range.
3534-
3535-
http://seriot.ch/parsing_json.html#25 According to the Unicode
3536-
standard, illformed subsequences should be replaced by U+FFFD
3537-
REPLACEMENT CHARACTER. (See Unicode PR #121: Recommended Practice
3538-
for Replacement Characters). Several parsers use replacement
3539-
characters, while other keep the escaped form or produce an
3540-
non-Unicode character (see Section 5 - Parsing Contents). This
3541-
values are not for interchange, only for application internal use.
3542-
They are different from private use. Most parsers accept these.
35433538
*/
35443539
if (UNLIKELY(
35453540
!(dec->json.flags & F_RELAXED)

‎t/01_utf8.t‎

Lines changed: 1 addition & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
use Test::More tests => 162;
1+
use Test::More tests => 152;
22
use utf8;
33
use Cpanel::JSON::XS;
44
use warnings;
@@ -43,56 +43,6 @@ SKIP: {
4343
# TODO: test utf8 hash keys,
4444
# test utf8 strings without any char > 0x80.
4545

46-
# warn on the 66 non-characters as in core
47-
{
48-
BEGIN { 'warnings'->import($] < 5.014 ? 'utf8' : 'nonchar') }
49-
my $w = '';
50-
$SIG{__WARN__} = sub { $w = shift };
51-
my $d = Cpanel::JSON::XS->new->allow_nonref->decode('"\ufdd0"');
52-
my $warn = $w;
53-
{
54-
no warnings 'utf8';
55-
is ($d, "\x{fdd0}", substr($warn,0,31)."...");
56-
}
57-
like ($warn, qr/^Unicode non-character U\+FDD0 is/);
58-
$w = '';
59-
# higher planes
60-
$d = Cpanel::JSON::XS->new->allow_nonref->decode('"\ud83f\udfff"');
61-
$warn = $w;
62-
{
63-
no warnings 'utf8';
64-
is ($d, "\x{1ffff}", substr($warn,0,31)."...");
65-
}
66-
like ($w, qr/^Unicode non-character U\+1FFFF is/);
67-
$w = '';
68-
$d = Cpanel::JSON::XS->new->allow_nonref->decode('"\ud87f\udffe"');
69-
$warn = $w;
70-
{
71-
no warnings 'utf8';
72-
is ($d, "\x{2fffe}", substr($warn,0,31)."...");
73-
}
74-
like ($w, qr/^Unicode non-character U\+2FFFE is/);
75-
76-
$w = '';
77-
$d = Cpanel::JSON::XS->new->allow_nonref->decode('"\ud8a4\uddd1"');
78-
$warn = $w;
79-
is ($d, "\x{391d1}", substr($warn,0,31)."...");
80-
is ($w, '');
81-
}
82-
{
83-
my $w;
84-
BEGIN { 'warnings'->import($] < 5.014 ? 'utf8' : 'nonchar') }
85-
$SIG{__WARN__} = sub { $w = shift };
86-
# no warning with relaxed
87-
my $d = Cpanel::JSON::XS->new->allow_nonref->relaxed->decode('"\ufdd0"');
88-
my $warn = $w;
89-
{
90-
no warnings 'utf8';
91-
is ($d, "\x{fdd0}", "no warning with relaxed");
92-
}
93-
is($w, undef);
94-
}
95-
9646
# security exploits via ill-formed subsequences
9747
# see http://unicode.org/reports/tr36/#UTF-8_Exploit
9848
# testcases from Encode/t/utf8strict.t

‎t/30_jsonspec.t‎

Lines changed: 1 addition & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# regressions and differences from the JSON Specs and JSON::PP
22
# detected by http://seriot.ch/json/parsing.html
3-
use Test::More ($] >= 5.008) ? (tests => 686) : (skip_all => "needs 5.8");
3+
use Test::More ($] >= 5.008) ? (tests => 678) : (skip_all => "needs 5.8");
44
use Cpanel::JSON::XS;
55
BEGIN {
66
require Encode if $] >= 5.008 && $] < 5.020; # Currently required for <5.20
@@ -107,32 +107,16 @@ sub i_undefined {
107107
sub i_pass {
108108
my ($str, $name) = @_;
109109
$@ = '';
110-
my $w;
111-
if ($name =~ /nonchar/) { # check the warning
112-
require warnings;
113-
warnings->import($] < 5.014 ? 'utf8' : 'nonchar');
114-
$SIG{__WARN__} = sub { $w = shift };
115-
}
116110
my $result = $todo{$name} ? eval { $json->decode($str) } : $json->decode($str);
117-
my $warn = $w;
118111
TODO: {
119112
local $TODO = "$name" if exists $todo{$name};
120113
is($@, '', "no parsing error with undefined $name ".substr($@,0,40));
121114
isnt($result, undef, "valid result with undefined $name");
122-
if ($name =~ /nonchar/) {
123-
like ($warn, qr/^Unicode non-character U\+[10DFE]+ is/);
124-
$w = '';
125-
}
126115
$@ = '';
127116
#diag "$name $str";
128117
$result = eval { $relaxed->decode($str) };
129-
$warn = $w;
130118
is($@, '', "no parsing error with undefined $name relaxed ".substr($@,0,40));
131119
isnt($result, undef, "valid result with undefined $name relaxed");
132-
if ($name =~ /nonchar/) {
133-
is($warn, '');
134-
$w = '';
135-
}
136120
}
137121
}
138122
# result undefined, parsing failed

0 commit comments

Comments
 (0)