Benchmark (with up to 10,000 documents)

[DanielRuf]

I was benchmarking luasmith via Podman on Windows 11 Pro using the podman-machine-default on WSL 2 with my Surface Laptop 6 for Business (24 GiB RAM, 1 TB SSD) with a Intel Core Ultra 165H (16 cores / 22 threads) on highest performance mode connected to the power cord

Steps

• install hyperfine
• create 3 folders: luasmith-100, luasmith-1000, luasmith-10000
• create empty subfolder content in luasmith-100, luasmith-1000, luasmith-10000
• copy luasmith binary into luasmith-100, luasmith-1000, luasmith-10000
• create 2 folders: input_md, output_md
• download and extract 10000-markdown-files.zip
• copy all md files from 10000-markdown-files to input_md
• run python3 generate-frontmatter.py
• sort files in output_md alphabetically
• copy first 100 md files from output_md to luasmith_100/content
• copy first 1000 md files from output_md to luasmith_1000/content
• copy all 10000 md files from output_md to luasmith_10000/content
• run cd luasmith-100 && hyperfine -i "./luasmith blog"
• run cd luasmith-1000 && hyperfine -i "./luasmith blog"
• run cd luasmith-10000 && hyperfine -i "./luasmith blog"

generate-frontmatter.py:

import os
import random
from datetime import datetime, timedelta

src = "input_md"
dst = "output_md"

os.makedirs(dst, exist_ok=True)

for i, filename in enumerate(os.listdir(src)):
    if not filename.endswith(".md"):
        continue

    with open(os.path.join(src, filename), "r", encoding="utf-8") as f:
        content = f.read()

    date = datetime(2020, 1, 1) + timedelta(days=random.randint(0, 1500))

    frontmatter = f"""---
title: "Document {i}"
description: "Benchmark file {i} generated from corpus"
date: "{date.strftime('%Y-%m-%d')}"
---

"""

    with open(os.path.join(dst, filename), "w", encoding="utf-8") as f:
        f.write(frontmatter + content)

Results

first 100 md files:

Benchmark 1: ./luasmith blog
  Time (mean ± σ):     385.3 ms ±  54.4 ms    [User: 71.5 ms, System: 20.1 ms]
  Range (min … max):   349.9 ms … 529.9 ms    10 runs

first 1000 md files:

Benchmark 1: ./luasmith blog
  Time (mean ± σ):      5.152 s ±  0.519 s    [User: 0.928 s, System: 0.289 s]
  Range (min … max):    4.264 s …  5.708 s    10 runs

all 10000 md files:

Benchmark 1: ./luasmith blog
  Time (mean ± σ):     53.186 s ±  1.947 s    [User: 9.304 s, System: 3.006 s]
  Range (min … max):   50.269 s … 55.711 s    10 runs

minimum:

xychart
    title "time in ms per # md documents"
    x-axis "# of documents" [100, 1000, 10000]
    y-axis "time (in ms)" 100 --> 60000
    bar [350, 4264, 50269]

Loading

maximum:

xychart
    title "time in ms per # md documents"
    x-axis "# of documents" [100, 1000, 10000]
    y-axis "time (in ms)" 100 --> 60000
    bar [530, 5708, 55711]

Loading

per document minimum:

xychart
    title "time in ms per md document"
    x-axis "# of documents" [100, 1000, 10000]
    y-axis "time (in ms)" 0 --> 6
    bar [3.50, 4.264, 5.0269]

Loading

per document maximum:

xychart
    title "time in ms per md document"
    x-axis "# of documents" [100, 1000, 10000]
    y-axis "time (in ms)" 0 --> 6
    bar [5.30, 5.708, 5.5711]

Loading

[DanielRuf]

Unfortunately I can not test outside of WSL2 and the Podman machine due to the installed security appliance (regular trick to bypass the security solution as always).

Anyone else might want to directly test on Linux, MacOS and Windows via native shell and binary (not the combined binary).

[jaredkrinke]

Wow, thanks for sharing! If the user+system time is much lower than elapsed time, does that imply it's IO bound? I wouldn't have expected that if you're doing 10 runs in a row (and everything should be in disk/file system caches), so I'm guessing "no"... but maybe WSL2 adds some inefficiencies.

Regardless, luasmith's single-threaded, rebuild-everything-each-time design certainly isn't optimal from the standpoint of a modern, multi-core CPU. One could argue this is a downside, but it might make more sense to just call out that luasmith's intended use case is simple personal sites.

[DanielRuf]

If the user+system time is much lower than elapsed time, does that imply it's IO bound?

maybe WSL2 adds some inefficiencies.

That‘s probably the case.
WSL2 can be and often is slower than WSL1 and the baremetal system, yes (fs and I/O related tasks due to the protocol overhead).

Microsoft also admits that at https://learn.microsoft.com/en-us/windows/wsl/compare-versions

Feature	WSL 1	WSL 2
Performance across OS file systems	✅	❌

I didn't tune any specific WSL configuration or Podman machine details, so this is the current performance ootb.

Regardless, luasmith's single-threaded, rebuild-everything-each-time design certainly isn't optimal from the standpoint of a modern, multi-core CPU.

Right, but at least I see no big peeformance degradation with a huge corpus. And also I did not show the warnings from the link checker, that some of them are invalid.

I could try another run with a custom build of the theme, which does not contain the link check step (https://github.com/jaredkrinke/luasmith/blob/main/themes/blog/theme.lua#L104, https://github.com/jaredkrinke/luasmith/blob/main/main.lua#L835). In fact I would make this optional / configurable. This could have some impact on the performance and is normally not needed by me.

After looking shortly at the code, I see that the link check step and the other code, I can see that the code highlighting step also uses _parseHtml, so probably (did not check in detail) parsing + writing is done twice / in two places instead of doing the opposite logic (parse once, do link check and / or highlighting as step per file as configured and write once). Especially considering when I read "a processing pipeline" as description for a luasmith theme in the description (that in my undersranding is - read / parse once, do the needed steps, write one per file instead of twice).

I jumped into Lua just yesterday, so I might oversee something but the code is very well readable and comprehensible and I might still be a bit sleepy (was 7 AM here when I started writing this reply and I am trying to visualize the whole logic as flowchart in my head).

Another difference I see is, that for checkLinks there is this check:

if string.sub(item.path, -5) == ".html" then

But for highlightSyntax there is none, just some "%.html$") a bit lower (currently on smartphone and can not see to which opening line this belongs due to the space-based indentation of Lua and missing colored indentation visualization on GitHub compared to some IDE plugin).

I will probably dig a bit deeper and try to find some good profiler to find out which function calls take the most time (either as flame graph, icicle graph or call graph). If you have some recommendation for a profiler, please let me know (I'm coming mostly from the PHP and Node.js world and we use specific profilers - partly integrated - for both).

You might want to check the readme, the read over the design links probably does not work or is meant to link to "Under the Hood".

[DanielRuf]

I will create a new issue regarding design proposals.

[DanielRuf]

I think I will have to find a different test system with Windows and without the security solution here (which makes thinks outside of WSL2 in many cases even much slower), the WSL2 performance penalty is too big.

Without linkcheck it is 10 - 20 ms less for the test with 100 documents for user time + system time.

Maybe others can run the benchmarks themselves on baremetal Windows systems connected to the power cord and with highest performance settings in the System options.

[DanielRuf]

Regardless, luasmith's single-threaded, rebuild-everything-each-time design certainly isn't optimal from the standpoint of a modern, multi-core CPU

I will probably experiment a bit further and document my results:

• customized luasmith setup (splitted into file compilation, rss feed generation, index generation, tag index page, link check)
• GNU parallel (or a loop to spawn multiple processes) to speed up builds (as the shortest way to test how far I can get with WSL2) which I had used in the past
• probably a SQLite database for some caching (to speed up some steps using timestamps of the files like last modified) - but that is probably another bottleneck

Update: optimized it in a first rough step to spawn 20 processes after splitting the 10,000 folder with split into 20 subfolders

Benchmark 1: ../compile.sh
  Time (mean ± σ):     13.157 s ±  1.072 s    [User: 17.691 s, System: 8.028 s]
  Range (min … max):   10.961 s … 14.315 s    10 runs

Which results in ~1000 processed md files per second with 20 processes vs the ~200 processed md files per second with only one process.

Without syntax highlighting and without link checking for 10,000 documents:

Benchmark 1: ./luasmith theme.lua
  Time (mean ± σ):     49.369 s ±  2.736 s    [User: 3.007 s, System: 2.974 s]
  Range (min … max):   45.144 s … 53.970 s    10 runs

[DanielRuf]

Closing for now.
The performance is great and currently I do not plan to invest more time in benchmarking, only when I face some real performance problems.

Doing the benchmarks was fun and made me think about some possible improvements for bigger setups.
Since premature optimization is not helpful, I'll keep the information here as reference.

A simple multi-process setup also may have diminishing returns, at least it does not scale that well with I/O parts (SSD, RAM, ...) involved.

[jaredkrinke]

Moving to a discussion so people can easily find it.