Skip to content

Commit

Permalink
Introducing the ultra-fast braindead JSON parser
Browse files Browse the repository at this point in the history
This parser only handles hashes, arrays and strings and will return
Nil if *anything* goes wrong.  It is now plugged into "R:I:JSON.from-json"
to first attempt the very fast parser: of that fails, fall back to the
original, slow, grammar based parser.

How much faster: at least 10x faster!
  • Loading branch information
lizmat committed Sep 19, 2018
1 parent 12cfde2 commit 16ce9c8
Showing 1 changed file with 95 additions and 0 deletions.
95 changes: 95 additions & 0 deletions src/core/Rakudo/Internals/JSON.pm6
Expand Up @@ -140,12 +140,107 @@ my class Rakudo::Internals::JSON {
}

method from-json($text) {
# attempt to use the brain-dead quick parser
.return with self.quick-from-json($text);

# alas, we need to fallback
my $a = JSONPrettyActions.new();
my $o = JSONPrettyGrammar.parse($text, :actions($a));
JSONException.new(:$text).throw unless $o;
$o.ast;
}
method to-json(|c) { to-json(|c) }

#--- Ultra-fast braindead JSON parser good for most META6.json files -----------
sub parse(\item,\iterator) {
item.starts-with('{')
?? parse-hash(iterator)
!! item.starts-with('[')
?? parse-array(iterator)
!! item.starts-with('"')
?? parse-string(item,iterator)
!! die
}

sub parse-hash(\iterator) {
my %hash;
nqp::until(
nqp::eqaddr((my \item := iterator.pull-one),IterationEnd),
nqp::if(
item eq '}' || item eq '},',
(return %hash), # done
nqp::stmts(
nqp::if(
item.starts-with('"'),
(my $key = parse-string(item,iterator)),
die
),
nqp::if(
iterator.pull-one ne ':',
die,
%hash.BIND-KEY($key, parse(iterator.pull-one,iterator))
)
)
)
);
die # should return with valid otherwise
}

sub parse-array(\iterator) {
my @array;
nqp::until(
nqp::eqaddr((my \item = iterator.pull-one),IterationEnd),
nqp::if(
item eq ']' || item eq '],',
(return @array),
@array.push: parse(item,iterator)
)
);
die # should return with valid otherwise
}

sub parse-string(\item, \iterator) {
nqp::if(
item.ends-with('"'),
item.substr(1,*-1),
nqp::if(
item.ends-with('",'),
item.substr(1,*-2),
nqp::stmts(
(my str @parts = item.substr(1)),
nqp::until(
nqp::eqaddr((my \part = iterator.pull-one),IterationEnd),
nqp::if(
part.ends-with('"'),
nqp::stmts(
@parts.push(part.substr(0,*-1)),
return @parts.join(" ")
),
nqp::if(
part.ends-with('",'),
nqp::stmts(
@parts.push(part.substr(0,*-2)),
nqp::if(
(my \string = @parts.join(" ")).contains('\\'),
die,
(return string)
)
),
@parts.push(part)
)
)
),
die # should return with valid otherwise
)
)
)
}

method quick-from-json($text) {
CATCH { return Nil }
my \iterator = $text.words.iterator;
parse(iterator.pull-one, iterator)
}
}

# vim: ft=perl6 expandtab sw=4

5 comments on commit 16ce9c8

@ugexe
Copy link
Member

@ugexe ugexe commented on 16ce9c8 Sep 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is missing the point -- zef has to parse 1MB+ json array of hashes containing meta6 data, not parse 1000+ individual META6.json files. e.g. if a single module in an ecosystem requires the fallback then zef will not see any speed improvement.

@lizmat
Copy link
Contributor Author

@lizmat lizmat commented on 16ce9c8 Sep 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was not clear to me, sorry.

Is there a place where I could get that 1MB JSON file to run tests on?

@lizmat
Copy link
Contributor Author

@lizmat lizmat commented on 16ce9c8 Sep 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps not the right place to ask. But why is it a single JSON file, and not many separate ones? If they were separate ones, we could probably race the creation of the internal representation?

@lizmat
Copy link
Contributor Author

@lizmat lizmat commented on 16ce9c8 Sep 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found the file

@lizmat
Copy link
Contributor Author

@lizmat lizmat commented on 16ce9c8 Sep 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My take on it: ugexe/zef#273

Please sign in to comment.