Shen 41.1 port: compiler optimizations and full test suite pass

[pyrex41]

Summary

This PR completes the Shen 41.1 port with critical compiler optimizations that unblock the full kernel boot and test suite. The 41.1 stlib and Prolog-compiled code stress LuaJIT's parser limits far more than the previous 22.4 kernel, requiring several new code-generation strategies to avoid syntax-level overflow and expression-complexity limits.

Test results: 134/134 reports pass (100% pass rate) on the official Shen 41.1 test suite.

Key Changes

Compiler Optimizations (compiler.lua)

1. Strict literal-data hoisting — Only (cons L R) trees are now treated as compile-time literals; side-effecting calls like (set ...), (shen.record-kl ...), and (lambda ...) are always evaluated. The previous heuristic over-hoisted, eliding critical initialization side effects.

2. Flat cons-tree builder (MKTREE) — The stlib's (shen.record-kl ...) calls contain trees up to ~7,000 cons cells / 216 levels deep. Naive nesting blows Lua's expression-complexity limit (~200 levels). The compiler now emits a flat blueprint array consumed by a runtime helper, collapsing deep trees into a single shallow call.

3. Hoisted freeze closures (BIND) — Shen's Prolog compiler chains 60+ (freeze ...) continuations inside argument positions. Each freeze body is now hoisted to a flat KB[N] = function(...) ... end slot inside the enclosing defun; use sites emit BIND(KB[N], cap...) with no nested function literal, sidestepping LuaJIT's 65-syntax-level parser cap.

4. Deep let-floating — (F1 a1 (F2 a2 (let X V B))) is rewritten to (let X V (F1 a1 (F2 a2 B))) when intermediate args are pure. Combined with hoisted freezes, this flattens deeply nested CPS chains into a single block of local declarations.

5. Right-spine call-chain flattening — Long chains like (shen.gc A (shen.gc A (shen.gc A ...))) in tail position are emitted as a sequence of local assignments rather than deeply-nested call expressions (another parser-cap fix).

6. Dynamic arity dispatch — Arity mismatches at compile time now route through APP to use the current runtime arity rather than a stale compile-time value, fixing issues where functions are redefined with different arities between compilation and execution.

Runtime Support (runtime.lua / prims.lua)

• BIND — Wraps per-defun continuation functions with captured values into 0-arity thunks, enabling the hoisted-freeze optimization.
• MKTREE — Consumes flat blueprints to build deep cons trees without expression nesting.
• Self-evaluation in eval — Vectors, streams, and opaque tables passed to eval-kl are returned as-is rather than compiled as KL forms.
• Port-specific primitives — Added shen.char-stoutput? and shen.char-stinput? (both return false for byte streams).
• String primitive bounds checks — pos, tlstr, string->n, n->string now validate inputs and raise errors on out-of-range / invalid arguments.

Test Infrastructure

• run-41.1-tests.lua — Updated to properly set y-or-n? as a non-interactive function and correctly extract pass/fail counts from the test harness's package-scoped globals.
• bench.lua — New self-contained benchmark measuring cold startup, fib(25..32) through the live Shen pipeline, and Einstein's riddle via the Prolog engine.
• 41.1-STATUS.md — Comprehensive documentation of the port state, compiler changes, and test results.

https://claude.ai/code/session_01WkcfnTxzoMzQQtYerRgv1N