Intermittent SIGSEGV (~0.3%/boot) in kernel boot under LuaJIT on aarch64; gone with JIT off

[pyrex41]

Summary

The Shen kernel boot (require("shen").boot{...}) intermittently SIGSEGVs (~0.3% of boots) under LuaJIT on aarch64. It is stochastic and low-rate, so most boots succeed, but any workload that boots the kernel many times (a per-request CLI, a test suite, a CI matrix) will flake. Disabling the JIT (luajit -j off) eliminates it: 0 crashes in 500+ boots.

Environment

• LuaJIT 2.1.0-beta3 (the OpenResty-alpine bundled build), aarch64 (Linux, inside Docker on Apple-silicon/colima).
• shen-lua c58746e (main, 2026-07-14).
• Reproduced across independent runs, both serial and under mild host CPU/memory pressure. Not concurrency-only (it happens with a single serial process), but competition for a small-core VM elevates the rate.

Symptom

• Exit: rc = 139 (= 128 + 11 → SIGSEGV).
• stderr: Segmentation fault (core dumped); stdout empty — so a program that boots the kernel, does one unit of work, and prints a result emits nothing, which downstream consumers read as a malformed/empty reply.
• The fault is in the boot/kernel-load phase, not steady state: a boot-once-then-serve-many process survived 24,000 post-boot calls across 60 processes with zero mid-serve crashes. Essentially all trace compilation happens while loading/first-executing the kernel; once booted it is stable.

Reproduction

From a shen-lua checkout on aarch64 (LuaJIT 2.1 beta3):

# JIT ON (default): intermittent SIGSEGV
crashes=0
for i in $(seq 1 500); do
  luajit -e 'require("shen").boot{quiet=true}' >/dev/null 2>&1
  r=$?; [ "$r" -ge 128 ] && { crashes=$((crashes+1)); echo "iter $i: signal rc=$r"; }
done
echo "crashes=$crashes / 500"     # observed: a handful (~0.3%/boot)

# JIT OFF: no crashes
crashes=0
for i in $(seq 1 500); do
  luajit -j off -e 'require("shen").boot{quiet=true}' >/dev/null 2>&1
  r=$?; [ "$r" -ge 128 ] && crashes=$((crashes+1))
done
echo "crashes=$crashes / 500"     # observed: 0

Measured over ~1000 JIT-on boots aggregate: ≈3 SIGSEGV ≈ 0.3%/boot. Over ~800 JIT-off boots (300 via -j off + 500 in the loop above + a downstream 200/200 confirmation): 0. Because the rate is low, any single 150–200 run may show 0 — the crash is real but needs a few hundred boots to surface reliably.

Root-cause localization

The evidence points to LuaJIT's trace/mcode compiler on arm64 faulting while compiling the large (~19-module) kernel during boot:

• -j off is decisive: 0/800 with JIT off vs ~0.3% with it on.
• Not the KL→Lua compile / kernel cache: cache-load and cold-compile crash at statistically indistinguishable rates — the .shen-kernel-cache.bin skips the KL→Lua step but the loaded bytecode is still JIT-traced on execution, so the crash surface is unchanged.
• Not ordinary GC/stack exhaustion (that would be reproducible/OOM, not a rare SIGSEGV that vanishes with -j off).
• boot.lua (lines ~9–21) already documents this fragility — it raises maxmcode=131072 / maxtrace=8000 "because … on macOS arm64 the suite triggers dozens of full trace-cache flushes per run ('failed to allocate mcode memory')". So arm64 mcode/trace management for this kernel is already known to be at the edge; this looks like the same pressure tipping into a compiler crash rather than just a flush.

Honest limits: I could not capture a backtrace (no gdb/core in the environment) or pin the exact faulting .kl module — the rate is low enough that a phase-instrumented probe did not happen to fault in the runs I could afford. So "LuaJIT arm64 trace compiler during boot" is localized by elimination (-j off clean, mid-serve clean, cache-independent), not by a stack trace.

Note on the existing escape hatch

SHEN_JIT_OPT=off (boot.lua:13–14) only restores LuaJIT's default jit.opt — it does not turn the JIT off, so it does not prevent this crash (the fault is in JIT execution/compilation itself, not the raised limits). There is currently no in-library way to boot with the JIT disabled.

Suggested fixes (for discussion)

1. A SHEN_JIT=off switch (or extend SHEN_JIT_OPT=off to actually jit.off()) so embedders on arm64 can opt into a crash-free boot without wrapping the interpreter flags. This is the minimal, low-risk mitigation.
2. Default the JIT off during kernel boot on arm64, re-enabling it after initialise() if desired — boot is dominated by load time and the compiler is the crash surface, so this trades a little boot speed for reliability where it matters.
3. Upstream to LuaJIT if a minimal non-Shen repro can be distilled — a compiler SIGSEGV is ultimately a LuaJIT bug; shen-lua is the trigger, not necessarily the cause. Worth trying a newer LuaJIT (the 2.1 rolling release) to see if it already fixes it.

Workaround in use downstream

A consumer (the wordd project) currently pins jit.off() at process start in the entrypoints that boot the kernel, plus a bounded retry around the boot as defense-in-depth. This makes the boot reproducibly crash-free (verified 240+/240+ boots), but it is a per-consumer workaround for what looks like a library/toolchain issue — hence this report.

[pyrex41]

Reproduced on aarch64 Linux + root-caused

Reproduced on real aarch64 Linux (OrbStack, linux/arm64) and pinned down the mechanism.

Reproduction

Build	JIT	Result
LuaJIT 2.1.0-beta3 (genuine 2017 v2.1.0-beta3 tag, built on Alpine/musl)	on	50/50 SIGSEGV (rc 139, empty stdout)
LuaJIT 2.1.0-beta3	SHEN_JIT=off	0/30
LuaJIT 2.1.ROLLING (current openresty-alpine)	on	0/60

The genuine 2017 beta3 tag crashes essentially every boot here — a much harsher version of the ~0.3% seen in the original report. That gap is explained: the report's "2.1.0-beta3" was a later rolling build that kept the beta3 version string (OpenResty/Debian did this for years), whereas this is the actual ancient tag. Same bug class, easier to trigger.

Root cause (gdb + -jv)

• The fault PC lands just outside LuaJIT's anonymous executable mcode region (the rwx trace area), with a corrupt/foreign call stack — the signature of a compiled trace branching to a bad target and jumping into unmapped memory. Not a fault in shen-lua's C surface, not the interpreter, not GC.
• -jv shows booting the kernel triggers ~1500 trace compilations up front. That concentrated JIT workload is the crash surface, which is exactly why the fault is boot-time-only while steady state is stable (matching the 24k-calls-clean observation).
• SHEN_JIT=off → 0 crashes confirms it's the JIT and nothing else.

Conclusion

This is a LuaJIT arm64 code-generation bug, and it is already fixed upstream (confirming suggestion #3): the identical 50/50-crashing kernel boots 0/60 under today's 2.1.ROLLING. shen-lua is pure Lua and can't patch LuaJIT's emitter — the kernel isn't doing anything illegal, old LuaJIT is miscompiling correct code.

So the real fix is "run a current LuaJIT." SHEN_JIT=off is the fallback for anyone pinned to an old LuaJIT.

Fix (#44)

• Docs lead with upgrade LuaJIT.
• SHEN_JIT=off → jit.off() at boot (distinct from SHEN_JIT_OPT=off, which only reset jit.opt and left the JIT on — hence never prevented this).
• shen.boot{jit=false} for embedders.
• A one-time stderr warning on the exact risky combo (arm64 + beta-versioned LuaJIT, JIT left on), silenced by SHEN_JIT=on/off, never fires on a rolling build.
• test/jit_spec.lua asserts the switch via jit.status(); full suite green (500/500).