Should the Ruby YARP implementation consume the binary output?

[HParker]

We could implement the Ruby library by consuming binary output similar to how some runtimes are consuming it. Theoretically this is could be a nice logical separation since now YARP has one API consumed by runtimes and tooling. The downside is that means, unless we allow configuring what data you want from that output we need to provide a syntax tree that is suitable for gems using a parser and runtimes.

Theoretically this could mean that the gem implementation of YARP doesn't need to rely on YARP being present on the system at all as long as something provides it the binary API that YARP provides.

[enebo]

As to your second point I plan on prototyping saving serialized blobs "precompiling" to see how much that changes things. This would mean all core JRuby startup .rb files would then just "test" to make sure the blob is still valid and loading that without the parsing step. It could be extended to any ruby file too. In essence it is what you said but labeled as "precompiling". We already have serialized IR and use it for our AOT compiled classes. I also plan on looking at using serialized data for that instead of our current serialization scheme.

Measuring speed of processing serialized blob and parsing itself is probably a deciding metric for the idea.

[eregon]

Yes, I think this would be best and is also required to support the Ruby API/gem on JRuby AFAIK.
I posted similar thoughts at #756 (comment)
There is already YARP::Serialize.load and YARP.load.

As you are mentioning, YARP.load should have all AST information including tokens, locations, etc, which OTOH we don't want for execution on JRuby and TruffleRuby.
I think that could be handled by e.g. generating two variants of yp_serialize_node in serialize.c, or maybe two serialize_SUFFIX.c files (that's details).

OTOH it feels dangerous to e.g. install the latest version of YARP and use a potentially older libyarp from JRuby or TruffleRuby (there is a high chance the nodes or their fields or serialization format does not match). Hence I think for gem usages, we should use the libyarp of that gem. And then there is no need to have two runtimes variants of serialization, the gem can just always serialize everything.
JRuby can also build libyarp (like the sassc gem does), so that's not a big problem. But it can't run C extensions, so it would need to use FFI to e.g. call yp_parse_serialize.
Adding FFI as a dependency of the gem seems not ideal though. If it's just a few functions like yp_parse_serialize, I think one possibility would be to use FFI on JRuby and TruffleRUby (they both ship with FFI as default so it's always available) and a tiny C extension for CRuby that does the same as FFI.
Or we could use Fiddle, that's available for all 3, so might be better (even though Fiddle's API is not so easy to use, but we need very little here).

We should measure performance-wise how does the C extension compares to using YARP.load.

Then YARP.parse(code) would just use YARP.load YARP.dump(code) of course.

[eregon]

I've measured with:

ruby --yjit -rparser/current -rbenchmark -e '10.times { p Benchmark.realtime { Dir.glob("top-100-gems/activesupport-7.0.4.3/**/*.rb") { |file| code = File.read(file); Parser::CurrentRuby.parse(code) } } }'
ruby --yjit -Ilib -ryarp -rbenchmark -e '10.times { p Benchmark.realtime { Dir.glob("top-100-gems/activesupport-7.0.4.3/**/*.rb") { |file| code = File.read(file); YARP.parse(code) } } }'
ruby --yjit -Ilib -ryarp -rbenchmark -e '10.times { p Benchmark.realtime { Dir.glob("top-100-gems/activesupport-7.0.4.3/**/*.rb") { |file| code = File.read(file); YARP.load code, YARP.dump(code, file) } } }'

on ruby 3.2.1 on my varint branch and with some optimizations we get this:

parser gem: 1.115
YARP.parse: 0.062
YARP.load:  0.089

So it's already >10x faster than the parser gem!
YARP.dump+YARP.load seems a bit slower than YARP.parse but not by much.
More optimizations would probably bring them closer.
Also if the binary format parsing in Ruby is proving the bottleneck we could also generate a C extension for that part.
So far the bottleneck seems mostly allocations.
Reducing the number of Location instances helps a lot, we could go further and object-inline all locations, so we would not create any Ruby Location object while deserializing.

[eregon]

And when compiling librubyparser with -O3 with (be is bundle exec):

be rake clobber && be rake compile_no_debug && be rake compile

we get:

parser gem: 1.115
YARP.parse: 0.052
YARP.load:  0.074

which brings them closer since serialize.c is now compiled with -O3

And on ruby master with YJIT:

YARP.parse: 0.046
YARP.load:  0.065

[eregon]

BTW protobuf varint is a clear win there, because using full 4-bytes uint32_t is much slower, 0.116 instead of 0.074.
Part of that is deserializing a 4-bytes int is not efficient as it involves a String allocation and copying, with io.read(4).unpack1("L".freeze) vs protobuf deserializing being some io.getbyte calls and arithmetic which involve no allocation.
EDIT: Actually IO::Buffer can help there, it gives 0.081, that's still slower than protobuf though and of course using much more serialized size.

[enebo]

I agree that there is no way we can just use libyarp from the gem for execution. Any changes are very likely to effect the Nodes we consume.

Some form of foreign function with a simple call to make serialized blob does seem like what JRuby would need. So @HParker idea of using serialize would make supporting YARP and a gem simpler to support.

[kddnewton]

I think this is now "solved" in the sense that it's working and we don't want to change it at this point.