Benchmarking, some comments

[alerque]

I've been fiddling around with some benchmarks and reached some surprising conclusions.

$ hyperfine --warmup 1 'sile-rust-luajit-embeded hello.sil' 'sile-rust-openresty-embeded hello.sil' 'sile-lua-openresty hello.sil' 'sile-rust-openresty-dynamic hello.sil' 'typst compile hello.typ'

Benchmark 1: sile-rust-luajit-embeded hello.sil
  Time (mean ± σ):     169.5 ms ±   2.8 ms    [User: 138.3 ms, System: 29.8 ms]
  Range (min … max):   164.6 ms … 175.3 ms    17 runs

Benchmark 2: sile-rust-openresty-embeded hello.sil
  Time (mean ± σ):     164.7 ms ±   2.0 ms    [User: 137.3 ms, System: 26.0 ms]
  Range (min … max):   161.3 ms … 169.2 ms    18 runs

Benchmark 3: sile-lua-openresty hello.sil
  Time (mean ± σ):     132.9 ms ±   2.3 ms    [User: 90.2 ms, System: 40.3 ms]
  Range (min … max):   129.1 ms … 137.2 ms    22 runs

Benchmark 4: sile-rust-openresty-dynamic hello.sil
  Time (mean ± σ):     195.8 ms ±   3.9 ms    [User: 139.6 ms, System: 53.0 ms]
  Range (min … max):   190.3 ms … 202.1 ms    15 runs

Benchmark 5: typst compile hello.typ
  Time (mean ± σ):     401.1 ms ±   4.3 ms    [User: 205.0 ms, System: 189.6 ms]
  Range (min … max):   396.2 ms … 409.3 ms    10 runs

Summary
  sile-lua hello.sil ran
    1.24 ± 0.03 times faster than sile-openresty-embeded hello.sil
    1.27 ± 0.03 times faster than sile-luajit-embeded hello.sil
    1.47 ± 0.04 times faster than sile-openresty-dynamic hello.sil
    3.02 ± 0.06 times faster than typst compile hello.typ

1. LuaJIT is just far and away faster than PUC Lua so I haven't even bothered with PUC much lately.

2. SILE's startup time to render a simple document is 3-4× faster that Typst. I did not expect that. Typst beats the pants off us for longer documents and their incremental compile is brilliant compared to what we are doing, but we can generate 3 PDF files to their one just on startup, render some text, write a PDF, shut down.

3. The plain Lua versions of SILE startup faster than the Rust builds. For hello world documents with no packages this is 40% faster. For longer documents the difference quickly falls to 10% (i.e. 100 pages of lorem text). For complex documents in real life (i.e. the manual) the difference is more like 3%. I'm guessed this was because I was testing a Rust binary that had all the C modules and Lua modules for every core package embeded. This guess turned out to be wrong, because a Rust binary with no embeding and dynamic linking was slower than the static build with all the Lua assets embeded.

4. Both plain Lua and Rust versions use almost exactly the same RAM (70-90MB depending and document length) and 100% of one CPU core. The Rust encapsulated version gives us a lot more possible jumping off points for parallelization, especially since we can do things like clone the VM, restore a previous state, etc.

[brynne8]

I want to know how this project considers performance. If performance is particularly important, it may be possible to speed things up by replacing the Lua C API library with a C library that uses FFI.

Moreover, would you like to share benchmark results on longer documents, for example a research paper of 20 pages length.

[alerque]

Performance is important, but not SILE's main stand-out feature. We'll take performance when we can get it, but not at the expense of flexibility or too much friction. As it stands SILE is several orders of magnitude faster than LaTeX and at least one maybe two orders of magnitude slower than Typst. What we do offer is a mix of relative ease of use and predictability, along with power and flexibility to tinker with the internals at run time via Lua. LaTeX is inscrutable and Typst is locked down.

For a benchmark, typesetting the 125 page manual takes about 18 seconds on my machine.

The Lua C API interface is not our bottle neck. The bottlenecks are:

• being single threaded
• recursing through some large Lua data structures and munging them over and over again
• not having any memoization
• no optimization for secondary passes

As noted in the comments above we're switching to a Rust based wrapper. Low hanging fruit is going to be finding hot loops that are inefficient enough in Lua that it's worth passing the data off to Rust and getting it back. In the process we can look for bits that can be done in parallel (yes starting up more VMs will be an option as is passing data around). Memoization both on the Lua side and in Rust as well as re-render optimizations are good options too.

Honestly there is probably 10× sitting around on the table without even messing with memory copying efficiency and FFIs or anything remotely tricky or high friction. Contributions welcome. What we do want to keep though in the easy access for 3rd parties to hack on anything. For example anything we "move" to Rust needs to stay accessible with an API such that it would be replaced with a Lua segment, and to make that feasible we probably want a Lua reference implementation too.

[Omikhleia]

Small additions to @alerque 's very detailed and clear answer.

• As for test cases: "... for example a research paper of 20 pages length". Folks, 20 pages only? Be more demanding! My own use cases are 100-300p books, and I even experimented with some ~2000p material (God knows why! Joke intended).
• As for raw performances being important...
  • As a user: Sure they are, no question about it.
  • But with my typist's hat, as I use say: writing a book/essay/whatever takes many months (or a year..), typesetting the material takes 1-3 months depending on complexity, then SILE ticks in for generating a PDF, and pre-print and last call proofreading start, taking 1-4 weeks (or more depending, on the number of reviewers..). So whether SILE took 10mn or 45mn to generate 300 to 2000p of content, frankly, the typists deserve a break and a coffee or two. That "small" time is nothing to them :p -- The importance is the output quality. Some solutions might be faster, but is the typographical quality at the same level (Kidding: no, no, LaTeX, not pointing at you; I like you dearly, but you are slow expanding after. Hey, Typst... Er. You don't know your typography yet, right? I understand... Keep trying!)
• More seriously, as for C bindings probably not being the current bottleneck, but as @alerque said "recursing through some large Lua data structures and munging them over and over again": Just to give one example of this fact, our pagebuilder currently goes on recomputing things from the top of the page on an on until full. It does have a "restart" feature but it is disabled by default (and broken). I experimented with a new restart approach there: on long but simple documents (the ~2000p experiments, long but nothing really fancy, mostly text and dropcaps) the difference was not much significant, but on my ~100p "resilient" manual (lots of footnotes, more complex content, lots of short sections, etc.), there was a 5-6% boost...¹ So there's still possible ways on improving the Lua algorithms significantly.

Footnotes

1. There were reasons, however, why the current approach was disabled. It's hard, and my own approach to it did not solve all of them in all possible workflows. This was not a priority to me, so I did not complete the experiment. If I ever do one day, @alerque knows me now (I guess), there will be something on my sile-x project, and if it more-or-less works there, something upstream here afterwards... But it may also remain one (of many) incomplete experiments made for the fun of trying hem, heh :) ↩