as-bind Markdown Parser: Quick Benchmark Performance Results

[torch2424]

cc @dcodeIO @MaxGraey

Hello! So I chatted with @dcodeIO about this once over DMs. But I figured I'd open an issue, as another thing we can just be aware of:

So for as-bind, I have a markdown parser demo, and one thing I have in the branch: 0.1.3-wasm-benchmark, is a little benchmark for the as-bind Markdown parser in JS and AssemblyScript and pulldown-cmark wasm.

The results from that can be seen in this as-bind issue. But TL;DR this is not a Rust vs. AS benchmark as they have completely different implementations. But, the JS compiled version starts to outperform the AS compiled version as the input size increases. However, the pulldown-cmark rust implementation stays consistently really fast.

And here's a quick screenshot of like the biggest input I could give both AS and JS without crashing (Note this was a pre-published version being used here):

Again, just another thing for us to be aware of and track, as I remember it was something @dcodeIO said would be fun to go through and see where the performance stuff happens.

And for anyone who happens to come across this: This is a very quickly hacked together markdown parser, and a hacked together benchmark. if you are looking for a better JS vs. AS / Wasm benchmark, please see my article here

Thanks! 😄 👍

[dcodeIO]

I wonder if part of this has something to do with our implementation of MM/GC in general. For instance, a JS engine can do GC in parallel, in another thread, and is most likely mark-and-sweep with lots of pre-allocated memory. In AS, all of MM/GC runs concurrently with the program, in a single thread, and memory must frequently be grown on demand leading to some overhead.

Typically I'd expect that the performance characteristics of managed ARC code in AS are completely different from what one would get in a JS engine or a language like Rust with no GC at all, and thus is hardly comparable. The best sort of comparison possible at this point might be to force a JS engine to use exactly one thread, to preallocate a sufficiently large chunk of memory for the AS module as well, or even to make all scenarios utilize an entire CPU, running multiple AS module instances in parallel to level the field, and compare total CPU consumption vs mean execution time or something like that. Probably highly impractical, and I'd still expect the outcome to be that AS is slower in typical OOP scenarios because TLSF (predictable performance suitable for real-time) has different performance characteristics than lets say dlmalloc (super fast small allocs) or whatever Rust is doing (would assume some sort of region-based alloc and free).

TL;DR I guess some of the fundamental concepts in managed code are so very different that a comparison leading to meaningful clues is nearly impossible, which is sad because any clues would help us optimizing.

[MaxGraey]

@torch2424 wasm-bindgen use TextDecoder / TextEncoder which could be faster for huge strings. We use this instead: https://github.com/AssemblyScript/assemblyscript/blob/master/lib/loader/index.js#L43

You could try modify it and try to use new TextDecoder("utf-16le") object which of course should be once cached globally

[MaxGraey]

Also some string operations in AS not optimal yet like str1.indexOf(str2), string concatenations (for more than 3 strings probably better use [a, b, c, d, ...].join('') instead)

[torch2424]

@dcodeIO Dope! Thanks for the explanation. Yeah I totally agree, JS Engines will definitely have access to multiple threads, and other resources we don't currently have access to in Wasm. Which makes a lot of sense. Which is completely fine! I remember a scenario similar to this that I heard in a talk (Not about Wasm, but just highlighting how fast JS engines have gotten over the years).

@MaxGraey Dope! Yeah once I get the time I'll try those out. Thank you! :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.