JSON::Tiny tests output bad text on latest build

[p6rt]

Migrated from rt.perl.org#131881 (status was 'resolved')

Searchable as RT131881$

[p6rt]

From @jdv

I updated to​:

[jdv@​localhost ~]$ perl6 -v
This is Rakudo version 2017.07-144-gec7bc25 built on MoarVM version
2017.07-365-gc0f7a3b
implementing Perl 6.c.
[jdv@​localhost ~]$

from a build from last weekend or so and the json tiny tests failed and
spit out some bad bytes​:

# Failed test 'JSON string «["̥̞ͫͩ̉̎͊ỏ̟͙̞̻̻̹̹r̵̨͇̫̦̺̮̼ͭ,̷̧̬̲͈̤̤̅̉͛ͫ͠
̣͕̝̳̗̲̩̣͗̋͘e̼ͩ̔̿̾̋̾͘a̸̙͙͔̜͙̳̻͊ͯͪͤͬ͊͆͠c̦̃̓̈̾͗́͝͝ḩ͉͓̦̯̞̆
̵͇͍ͪ̀́w̷ͩ͗̃͒͏̲̼̯̕i̞̫̜̮̣̰͎̅̂͋̀ͅń̢̨̪̭̠̼͕͎̐́͆̅ͅd̰̤̝͎̹͕ͩ̃̏̃̍͊̔ȏ̶̼̺̪̾ͨ̈̄ͤ̄̚̕w̙̻̱̙̺͎͚̙ͧͪ̊̒̕͝
̴̼̰̱͈̠ͤ̋̏̇͛&̠̺̟̞͎̳̻̙̽̓́̔̇ͩ͛ͧ̚ ̡̡̫̻͐̌̐̌ͬͦ̂͞"]» parsed'
# at t/01-parse.t line 208
# Looks like you failed 1 test of 93
Input (2 characters) is not a valid JSON string
  in sub from-json at
/Users/ugexe/.zef/store/json.git/957740bb22b122b711363328e224b56935873a61/lib/JSON/Tiny.pm
(JSON​::Tiny) line 59
  in block <unit> at t/03-unicode.t line 15

and then this happens​:

Aug 10 14​:29​:48 localhost.localdomain gnome-terminal-server[3375]​: **
Aug 10 14​:29​:48 localhost.localdomain gnome-terminal-server[3375]​:
Pango​:ERROR​:pango-glyph-item.c​:319​:pango_glyph_item_iter_next_cluster​:
assertion failed​: (iter->start_char < iter->end_char)
Aug 10 14​:29​:48 localhost.localdomain audit[3375]​: ANOM_ABEND auid=1000
uid=1000 gid=1000 ses=5 pid=3375 comm="gnome-terminal-"
exe="/usr/libexec/gnome-terminal-server" sig=6 res=1
Aug 10 14​:29​:48 localhost.localdomain abrt-hook-ccpp[3910]​: Process
3375 (gnome-terminal-server) of user 1000 killed by SIGABRT - dumping
core
Aug 10 14​:29​:48 localhost.localdomain systemd[2268]​:
gnome-terminal-server.service​: Main process exited, code=dumped,
status=6/ABRT
Aug 10 14​:29​:48 localhost.localdomain systemd[2268]​:
gnome-terminal-server.service​: Unit entered failed state.
Aug 10 14​:29​:48 localhost.localdomain systemd[2268]​:
gnome-terminal-server.service​: Failed with result 'core-dump'.

which is the log version of "all my gnome-terminal instances just
vanished...​:(".

[p6rt]

@jdv - Status changed from 'new' to 'deleted'

[p6rt]

@jdv - Status changed from 'deleted' to 'open'

[p6rt]

From @AlexDaniel

It was bisected to rakudo/rakudo@f590863

Example (before and after)​: https://gist.github.com/Whateverable/2a9088ddcff37fd6f748b77ba3339af2

On 2017-08-10 14​:54​:24, jdv79 wrote​:

I updated to​:

[jdv@​localhost ~]$ perl6 -v
This is Rakudo version 2017.07-144-gec7bc25 built on MoarVM version
2017.07-365-gc0f7a3b
implementing Perl 6.c.
[jdv@​localhost ~]$

from a build from last weekend or so and the json tiny tests failed
and
spit out some bad bytes​:

# Failed test 'JSON string «["̥̞ͫͩ̉̎͊ỏ̟͙̞̻̻̹̹r̵̨͇̫̦̺̮̼ͭ,̷̧̬̲͈̤̤̅̉͛ͫ͠
̣͕̝̳̗̲̩̣͗̋͘e̼ͩ̔̿̾̋̾͘a̸̙͙͔̜͙̳̻͊ͯͪͤͬ͊͆͠c̦̃̓̈̾͗́͝͝ḩ͉͓̦̯̞̆
̵͇͍ͪ̀́w̷ͩ͗̃͒͏̲̼̯̕i̞̫̜̮̣̰͎̅̂͋̀ͅń̢̨̪̭̠̼͕͎̐́͆̅ͅd̰̤̝͎̹͕ͩ̃̏̃̍͊̔ȏ̶̼̺̪̾ͨ̈̄ͤ̄̚̕w̙̻̱̙̺͎͚̙ͧͪ̊̒̕͝
̴̼̰̱͈̠ͤ̋̏̇͛&̠̺̟̞͎̳̻̙̽̓́̔̇ͩ͛ͧ̚ ̡̡̫̻͐̌̐̌ͬͦ̂͞"]» parsed'
# at t/01-parse.t line 208
# Looks like you failed 1 test of 93
Input (2 characters) is not a valid JSON string
in sub from-json at
/Users/ugexe/.zef/store/json.git/957740bb22b122b711363328e224b56935873a61/lib/JSON/Tiny.pm
(JSON​::Tiny) line 59
in block <unit> at t/03-unicode.t line 15

and then this happens​:

Aug 10 14​:29​:48 localhost.localdomain gnome-terminal-server[3375]​: **
Aug 10 14​:29​:48 localhost.localdomain gnome-terminal-server[3375]​:
Pango​:ERROR​:pango-glyph-
item.c​:319​:pango_glyph_item_iter_next_cluster​:
assertion failed​: (iter->start_char < iter->end_char)
Aug 10 14​:29​:48 localhost.localdomain audit[3375]​: ANOM_ABEND
auid=1000
uid=1000 gid=1000 ses=5 pid=3375 comm="gnome-terminal-"
exe="/usr/libexec/gnome-terminal-server" sig=6 res=1
Aug 10 14​:29​:48 localhost.localdomain abrt-hook-ccpp[3910]​: Process
3375 (gnome-terminal-server) of user 1000 killed by SIGABRT - dumping
core
Aug 10 14​:29​:48 localhost.localdomain systemd[2268]​:
gnome-terminal-server.service​: Main process exited, code=dumped,
status=6/ABRT
Aug 10 14​:29​:48 localhost.localdomain systemd[2268]​:
gnome-terminal-server.service​: Unit entered failed state.
Aug 10 14​:29​:48 localhost.localdomain systemd[2268]​:
gnome-terminal-server.service​: Failed with result 'core-dump'.

which is the log version of "all my gnome-terminal instances just
vanished...​:(".

[p6rt]

From @moritz

Background of JSON​::Tiny's weird code​:

JSON allows you to have quoted strings, and these strings can start with basically any Unicode codepoint, including combining characters. JSON​::Tiny failed to parse these strings, because Perl 6 combines the opening quote with the following (combining) codepoint to a single grapheme.

Since regexes don't have a codepoint mode (which would be appropriate for parsing JSON), the worakround was to parse the opening quote with :ignoremark. In the old rakudo version, the regex / (​:ignoremark '"') / parsed the quote plus combining character(s) in a single capture, and then transfered the combining characters to the start of the quoted string in the action method.

I would expect the following three lines to all print "True\n"​:

say so qq{"\c[ZERO WIDTH JOINER]a"} ~~ rx​:ignoremark/^ '"' /;
say so qq{"\c[ZERO WIDTH JOINER]a"} ~~ / :ignoremark ^ '"' /;
say so qq{"\c[ZERO WIDTH JOINER]a"} ~~ / (​:ignoremark ^ '"' ) /;

If somebody objects, please then clarify what you think the proper way is to parse JSON with a Perl 6 grammar.

[p6rt]

From @samcv

I have fixed this as of this MoarVM commit​:

MoarVM/MoarVM@712cff3

Full explaination in the commit description.

Thanks a lot for reporting this bug :)

[p6rt]

@samcv - Status changed from 'open' to 'resolved'