bench: cross-version benchmarks for v0.4.0, v1.0.0 and v1.0.2

CHANGELOG.md

@@ -52,11 +52,15 @@ is in the [`1.0.0-rc.0`](#100-rc0---2026-05-26) entry below.
 ## [1.0.2] - 2026-07-28
 
 ### Changed
-- `Lua.new/1` is ~100x faster (roughly 40µs down to 0.4µs) for the default and
-  fully-custom-sandbox configurations. Installing the standard library is pure
-  and deterministic, so the boot-time VM template is now built once per node
-  and memoized in `:persistent_term`; every later `Lua.new/1` starts from the
-  shared template copy-on-write. In `:interactive` mode (dev, IEx, tests) the
+- `Lua.new/1` is ~60x faster for the default configuration — 36.7µs down to
+  0.6µs median, with per-call allocation down from ~92KB to under 1KB — and
+  ~5.5x faster when a custom sandbox is passed (36.0µs down to 6.5µs), as
+  measured by `benchmarks/vm_new.exs` under `mix run`
+  ([full figures](https://github.com/tv-labs/lua/blob/main/bench_results/versions-2026-07-28.md)).
+  Installing the
+  standard library is pure and deterministic, so the boot-time VM template is
+  now built once per node and memoized in `:persistent_term`; every later
+  `Lua.new/1` starts from the shared template copy-on-write. In `:interactive` mode (dev, IEx, tests) the
   cache self-invalidates when the modules that built it are recompiled; hosts
   that hot-load new code in `:embedded` mode (releases) can force a rebuild
   with `Lua.VM.Bootstrap.reset/0` (#398).

Dockerfile

@@ -56,6 +56,10 @@ COPY website/priv priv
 COPY website/lib lib
 COPY website/assets assets
 
+# Recorded benchmark results, read at compile time by lib/website/benchmarks.ex
+# (@external_resource) to render /benchmarks. Compile fails loudly without it.
+COPY bench_results /app/bench_results
+
 # mix compile must run BEFORE assets.deploy because Phoenix's LiveView
 # colocated-hooks compiler generates files under _build/ that esbuild
 # resolves via NODE_PATH (`phoenix-colocated/website`).

bench_results/README.md

@@ -0,0 +1,137 @@
+# `bench_results/`
+
+Recorded benchmark runs and the reports written from them.
+
+The benchmark **scripts** live in [`benchmarks/`](../benchmarks/). This
+directory holds their **output**: raw stdout, a parsed JSON summary, environment
+probes, and the human-readable report for each measurement campaign. Nothing
+here is used by the library at runtime, and none of it ships in the Hex package.
+
+## Contents
+
+| Path | What it is |
+|---|---|
+| [`versions-2026-07-28.md`](./versions-2026-07-28.md) | Cross-version comparison — v0.4.0 vs v1.0.0 vs 1.0.2 (`main` @ `3a0d392`), with Luerl 1.5.1 as a same-run control in every table. Supersedes the 1.0.0-era numbers in [`benchmarks/BASELINE.md`](../benchmarks/BASELINE.md). |
+| [`v0.4.0/`](./v0.4.0/), [`v1.0.0/`](./v1.0.0/), [`v1.0.2/`](./v1.0.2/) | The data behind that report — one directory per released version. This is the ongoing convention: each release gets its own directory here, measured with the full suite of its day. |
+
+### Layout of a version directory
+
+```
+<version>/
+  environment.md      # ref, commit, mode, CPU, OTP/Elixir, timestamp, command form
+  summary.json        # parsed results: workload -> case -> jobs + comparison lines
+  <workload>.txt      # verbatim stdout of one `mix run benchmarks/<workload>.exs`
+  cpu.txt versions.txt timestamp.txt commit.txt
+```
+
+`summary.json` schema, as produced for the 2026-07-28 run:
+
+- Top level is keyed by **workload** (`fibonacci`, `table_ops`, `vm_new`, …).
+- Each workload maps to its **case banners** (`"default"` for single-case
+  workloads, otherwise the banner the script printed, e.g.
+  `"patterns: gsub template substitution (n=200)"`).
+- Each case has `jobs` — a list of `{name, ips, average, deviation, median,
+  p99, memory}` — plus the verbatim Benchee `comparison` lines and, when memory
+  measurement was on, `memory_comparison` or `memory_note`.
+- `table_ops` cases nest one level deeper under `by_input` (or `inputs` on the
+  v0.4.0 run) keyed by input label — `small (n=10)`, `medium (n=100)`,
+  `large (n=1000)`.
+- `vm_new` carries `cold_call` and `second_call` alongside its jobs: the
+  first-ever and second `Lua.new()` on the node, measured before Benchee starts.
+- `encode_decode` is `{"raw": "..."}`. That script uses its own `:timer.tc`
+  harness rather than Benchee and prints a per-element-nanoseconds table, so it
+  is stored unparsed.
+
+Case-name keys are **not** byte-identical across refs: the v0.4.0 run appends
+`" (mode: full)"` to banners and uses `"(single case)"` where later runs use
+`"default"`. Normalise by stripping the mode suffix before joining across refs.
+
+## Reproducing a run
+
+Full per-ref instructions — including the three adaptations v0.4.0 needs and
+the language constraints a cross-version workload must respect — are in the
+[Reproduction section of the report](./versions-2026-07-28.md#reproduction).
+The short version:
+
+```sh
+MIX_ENV=benchmark mix deps.get
+for w in fibonacci closures oop string_ops string_format table_ops \
+         patterns metamethods pcall_varargs vm_new encode_decode; do
+  LUA_BENCH_MODE=full MIX_ENV=benchmark mix run "benchmarks/$w.exs"
+done
+```
+
+Two rules are not optional:
+
+- **Serially, one `mix run` at a time, on a quiet machine.** Concurrent load
+  inflates deviation badly — the table and OOP cases swing enough to flip
+  orderings.
+- **`LUA_BENCH_MODE=full` for anything published.** The default `quick` mode
+  uses short windows, skips memory measurement, and collapses the table
+  workloads to a single input size. It is for "did my change move the needle"
+  iteration, not for numbers anyone reads.
+
+### Older refs
+
+Each ref is measured in a throwaway detached worktree so the main checkout is
+never modified:
+
+```sh
+git worktree add --detach /tmp/lua-<tag> <tag>
+```
+
+- **v1.0.0** — copy in the four workloads that postdate it
+  (`patterns`, `metamethods`, `pcall_varargs`, `vm_new`), then
+  `MIX_ENV=benchmark mix deps.get`.
+- **v0.4.0** — copy in the whole `benchmarks/` directory (the tag has none) and
+  add `{:benchee, "~> 1.3", only: :benchmark}` to `deps/0`. `luerl` is already
+  an unconditional dependency there, so the control rows need nothing.
+
+Remove the worktree when done (`git worktree remove --force /tmp/lua-<tag>`).
+
+## Benchmarking a new release
+
+The convention: **every released version gets a directory here**, so the
+series grows one column per release.
+
+1. After tagging, run the full suite against the tag (serially, full mode,
+   quiet machine — see above) and put the outputs in
+   `bench_results/<version>/` with the same file layout as the existing
+   directories (`environment.md`, `summary.json`, one `.txt` per workload,
+   plus the env probes).
+2. Include the Luerl control rows — they are what make the new column
+   comparable to the old ones despite machine/OTP drift between sittings.
+3. If the suite gained workloads since the last release, note in
+   `environment.md` which workloads are new (older version directories will
+   simply lack those files).
+4. Write or extend a report named `<topic>-<YYYY-MM-DD>.md` quoting
+   **medians**, not averages — several workloads have allocation-driven GC
+   pauses that pull the mean around.
+5. Add a row to the Contents table above.
+6. Do not edit `benchmarks/BASELINE.md`. It is the historical 1.0.0 gate
+   record; a newer report supersedes it by saying so.
+
+### What updates itself
+
+The hosted page at [`/benchmarks`](../website/lib/website_web/controllers/page_html/benchmarks.html.heex)
+reads these directories directly, so steps 1–5 are the whole job:
+
+- **A new `<version>/summary.json` becomes a new column.** Version directories
+  are found by glob and ordered with `Version.compare/2` — no list to extend.
+- **A new `versions-<date>.md` becomes the linked report**, and its date becomes
+  the page's dated eyebrow. The newest report filename wins.
+- **Headline tiles and the "still behind Luerl" figures re-derive** from the new
+  column, including the "N× faster than <previous release>" deltas.
+
+Two things still need a human:
+
+- **A new workload needs a row spec** in `Website.Benchmarks` (`@rows`) before it
+  appears — which cases are worth showing, and under what name, is editorial.
+  A version that lacks a workload another version has renders as `—`.
+- **The prose** — the headline claim and the analysis paragraphs — is written,
+  not generated. Re-read it when the story changes.
+
+`Website.Benchmarks` registers each `summary.json` as an `@external_resource`,
+so editing recorded results recompiles the page in dev. The container build
+copies this directory in (see `Dockerfile`); compilation fails loudly rather
+than shipping an empty page if it is missing.

bench_results/v0.4.0/closures.txt

@@ -0,0 +1,43 @@
+luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks
+Operating System: macOS
+CPU Information: Apple M4
+Number of Available Cores: 10
+Available memory: 32 GB
+Elixir 1.20.0
+Erlang 29.0
+JIT enabled: true
+
+Benchmark suite executing with the following configuration:
+warmup: 2 s
+time: 10 s
+memory time: 1 s
+reduction time: 0 ns
+parallel: 1
+inputs: none specified
+Estimated total run time: 39 s
+Excluding outliers: false
+
+Benchmarking lua (chunk) ...
+Benchmarking lua (eval) ...
+Benchmarking luerl ...
+Calculating statistics...
+Formatting results...
+
+Name                  ips        average  deviation         median         99th %
+lua (chunk)        2.69 K      372.35 μs     ±7.42%      368.92 μs      507.15 μs
+lua (eval)         2.52 K      397.20 μs     ±8.30%      390.13 μs      515.81 μs
+luerl              2.50 K      400.26 μs     ±9.75%      391.42 μs      536.69 μs
+
+Comparison: 
+lua (chunk)        2.69 K
+lua (eval)         2.52 K - 1.07x slower +24.85 μs
+luerl              2.50 K - 1.07x slower +27.91 μs
+
+Memory usage statistics:
+
+Name           Memory usage
+lua (chunk)         1.89 MB
+lua (eval)          1.90 MB - 1.01x memory usage +0.0101 MB
+luerl               1.90 MB - 1.00x memory usage +0.00886 MB
+
+**All measurements for memory usage were the same**

bench_results/v0.4.0/cpu.txt

@@ -0,0 +1 @@
+Apple M4

bench_results/v0.4.0/encode_decode.txt

@@ -0,0 +1,128 @@
+lua 0.4.0 — encode!/decode! decomposition
+(decode+deep_cast column: enabled)
+==================================================================================
+op       shape                     N     total_us   per_elem_ns
+encode   int_list                  8         0.27          33.3
+decode   int_list                  8         0.09          11.2
+dec+cast int_list                  8         0.11          13.7
+encode   int_list                 64         1.61          25.1
+decode   int_list                 64         0.54           8.5
+dec+cast int_list                 64         0.96          15.0
+encode   int_list                512        26.38          51.5
+decode   int_list                512         5.11          10.0
+dec+cast int_list                512         8.60          16.8
+encode   int_list               4096       226.97          55.4
+decode   int_list               4096        43.47          10.6
+dec+cast int_list               4096        88.67          21.6
+----------------------------------------------------------------------------------
+encode   float_list                8         0.22          27.7
+decode   float_list                8         0.09          11.2
+dec+cast float_list                8         0.11          13.2
+encode   float_list               64         1.64          25.7
+decode   float_list               64         0.55           8.5
+dec+cast float_list               64         0.94          14.7
+encode   float_list              512        27.00          52.7
+decode   float_list              512         5.85          11.4
+dec+cast float_list              512         9.00          17.6
+encode   float_list             4096       239.42          58.5
+decode   float_list             4096        64.74          15.8
+dec+cast float_list             4096        98.40          24.0
+----------------------------------------------------------------------------------
+encode   bool_list                 8         0.21          26.4
+decode   bool_list                 8         0.08          10.0
+dec+cast bool_list                 8         0.10          12.6
+encode   bool_list                64         1.60          25.0
+decode   bool_list                64         0.56           8.7
+dec+cast bool_list                64         0.97          15.1
+encode   bool_list               512        25.68          50.1
+decode   bool_list               512         4.80           9.4
+dec+cast bool_list               512         8.38          16.4
+encode   bool_list              4096       224.95          54.9
+decode   bool_list              4096        38.99           9.5
+dec+cast bool_list              4096        84.05          20.5
+----------------------------------------------------------------------------------
+encode   short_string_list         8         0.21          26.7
+decode   short_string_list         8         0.08          10.5
+dec+cast short_string_list         8         0.10          12.9
+encode   short_string_list        64         1.62          25.3
+decode   short_string_list        64         0.55           8.6
+dec+cast short_string_list        64         0.98          15.4
+encode   short_string_list       512        26.45          51.7
+decode   short_string_list       512         4.86           9.5
+dec+cast short_string_list       512         9.05          17.7
+encode   short_string_list      4096       202.93          49.5
+decode   short_string_list      4096        43.46          10.6
+dec+cast short_string_list      4096        73.06          17.8
+----------------------------------------------------------------------------------
+encode   long_string_list          8         0.22          27.2
+decode   long_string_list          8         0.08          10.4
+dec+cast long_string_list          8         0.10          12.7
+encode   long_string_list         64         1.67          26.1
+decode   long_string_list         64         0.59           9.2
+dec+cast long_string_list         64         1.03          16.1
+encode   long_string_list        512        28.42          55.5
+decode   long_string_list        512         5.16          10.1
+dec+cast long_string_list        512         9.09          17.7
+encode   long_string_list       4096       465.96         113.8
+decode   long_string_list       4096        54.39          13.3
+dec+cast long_string_list       4096       108.40          26.5
+----------------------------------------------------------------------------------
+encode   string_map                8         0.52          65.3
+decode   string_map                8         0.07           8.2
+dec+cast string_map                8         0.17          21.1
+encode   string_map               64         6.56         102.5
+decode   string_map               64         0.40           6.2
+dec+cast string_map               64         3.21          50.2
+encode   string_map              512       113.10         220.9
+decode   string_map              512         4.27           8.3
+dec+cast string_map              512        39.39          76.9
+encode   string_map             4096      1295.09         316.2
+decode   string_map             4096        42.02          10.3
+dec+cast string_map             4096       367.76          89.8
+----------------------------------------------------------------------------------
+encode   int_map                   8         0.25          31.2
+decode   int_map                   8         0.09          10.7
+dec+cast int_map                   8         0.10          12.9
+encode   int_map                  64         3.56          55.6
+decode   int_map                  64         0.57           8.9
+dec+cast int_map                  64         0.97          15.1
+encode   int_map                 512        51.12          99.8
+decode   int_map                 512         4.79           9.4
+dec+cast int_map                 512         8.76          17.1
+encode   int_map                4096       443.17         108.2
+decode   int_map                4096        40.30           9.8
+dec+cast int_map                4096        87.24          21.3
+----------------------------------------------------------------------------------
+encode   record_list               8         2.07         258.7
+decode   record_list               8         0.40          49.9
+dec+cast record_list               8         0.80         100.1
+encode   record_list              64        31.54         492.8
+decode   record_list              64         4.54          70.9
+dec+cast record_list              64         7.32         114.4
+encode   record_list             512       287.96         562.4
+decode   record_list             512        52.67         102.9
+dec+cast record_list             512        77.64         151.6
+encode   record_list            4096      2346.38         572.8
+decode   record_list            4096       406.88          99.3
+dec+cast record_list            4096       658.70         160.8
+----------------------------------------------------------------------------------
+nested chain (depth sweep) — isolates recursion/traversal from fan-out
+op       shape                     N     total_us   per_elem_ns
+encode   nested_chain              4         0.51         257.3
+decode   nested_chain              4         0.12          61.7
+dec+cast nested_chain              4         0.24         118.7
+encode   nested_chain             16         2.06        1032.0
+decode   nested_chain             16         0.63         314.9
+dec+cast nested_chain             16         1.10         550.2
+encode   nested_chain             64         8.53        4265.3
+decode   nested_chain             64         4.16        2079.7
+dec+cast nested_chain             64         6.63        3317.0
+encode   nested_chain            256        34.34       17170.4
+decode   nested_chain            256        37.82       18909.7
+dec+cast nested_chain            256        47.43       23716.3
+==================================================================================
+composite anchor — the PR's `original_nested` (matches the 18us/108us figure)
+op       shape                     N     total_us   per_elem_ns
+encode   original_nested          75         3.48         870.0
+decode   original_nested          75         0.58         145.4
+dec+cast original_nested          75         1.32         329.3