Fix recursion-depth gaps vs. OpenSCAD: tail-call optimization + hard abort on detected recursion (#83)

[particlesector]

Summary

Closes #83. Also files (but does not implement) #101 as an intentionally-deferred follow-up.

• Raised kMaxCallDepth from 200 to 300 for genuinely non-tail recursion, backed by an empirical stack-overflow measurement (1 MiB-stack thread, GCC 13 -O3) rather than the previous vague "800-1000" citation.
• Implemented real tail-call optimization (Interpreter::evalFunctionBody()): a named function's body is evaluated via a trampoline that unwraps ternary branches and let bindings in place and, on hitting a call in tail position (self or mutual recursion), loops into the callee directly instead of recursing — no new native stack frame per hop, bounded by a separate 1,000,000-hop cap (kMaxTailHops) matching real OpenSCAD's own cap exactly. This is what actually closes the gap: raising kMaxCallDepth alone can never satisfy OpenSCAD's own tail-recursion-tests.scad, which expects tail-recursive functions to reach depths of 2,000-50,000.
• Hard-abort on detected infinite recursion: extended CsgEvaluator's existing m_aborted mechanism (previously only used for a failed top-level assert()) so a tripped recursion guard now halts the rest of the script with an Error diagnostic, matching real OpenSCAD's fatal "Recursion detected calling function 'X'" behavior instead of silently degrading to undef and continuing.
• Fixed an unrelated, previously-undiscovered bug found while validating this: every real evaluation entry point called loadAssignments() before loadFunctions(), so a top-level assignment calling a user-defined function (e.g. y = f(3);) always evaluated to undef. A test comment had already flagged this as a known quirk but left it unfixed. The fix is a pure reordering — loadFunctions() only registers non-owning pointers, no evaluation — so it's provably safe.

Verification

None of the OpenSCAD test files this issue cites (recursion-test-function.scad, tail-recursion-tests.scad, issue3118-recur-limit.scad) are checked into this repo. Fetched them directly from openscad/openscad and ran them against both a real OpenSCAD 2021.01 binary (apt install openscad) and ChiselCAD's scad_dump tool for a byte-level diff, rather than guessing:

• tail-recursion-tests.scad: every echo() line (minus ftail_mixed, which uses assert()/echo() as expression forms ChiselCAD's grammar doesn't parse at all — a separate, unrelated gap) now matches real OpenSCAD's output exactly, including the 50,000-deep cases.
• recursion-test-function.scad / issue3118-recur-limit.scad: both now match real OpenSCAD's behavior (no output for the triggering statement, error diagnostic, empty geometry, earlier statements' output preserved) — diagnostic wording isn't byte-exact (missing the in file X, line Y suffix and TRACE: lines), tracked separately as [Low] Recursion-detected hard abort doesn't match OpenSCAD's exact diagnostic wording (ERROR/TRACE lines) #101 since it's cosmetic-only, same category the project already deprioritized in [Low] Missing arity-mismatch / file-not-found diagnostic wording parity with OpenSCAD (cosmetic) #85.

Also confirmed the tail-call optimization mirrors real OpenSCAD's actual mechanism, not a guess — read FunctionCall::evaluate's trampoline loop directly in openscad/openscad's src/core/Expression.cc.

Testing

This environment can't do the full Manifold/vcpkg CMake build, so validation used a standalone build of the language/CSG-evaluator subset (self-contained, no Manifold dependency) via apt's openscad/libglm-dev/catch2 packages:

• Full existing test suite (lexer/parser/interpreter/CSG evaluator): 560 test cases / 3173 assertions, all passing (up from 555/3148 before this branch — 5 new regression tests added).
• New tests cover: the raised depth cap, the tail-call trampoline (including 50,000-deep and mutual-recursion cases), non-tail recursion still going through the ordinary guarded path, the recursion-abort flag and its propagation through CsgEvaluator, and the load-order fix.
• Re-verified against the real OpenSCAD oracle and fetched corpus files as described above.

Test plan

• Full lexer/parser/interpreter/CSG-evaluator test suite passes (560/560, standalone build)
• Verified against real OpenSCAD 2021.01 oracle for all three cited corpus files
• New regression tests added for every behavior change (TCO, recursion abort, load-order fix)
• CI (Windows/Linux full builds via vcpkg+Manifold) — not run in this environment; should be checked once this PR's CI executes

---

Generated by Claude Code

[particlesector]

Reviewed. The TCO trampoline, raised depth cap, and load-order fix all look correct and well-tested. One gap noted inline: the recursion-abort check isn't wired into the general primitive/transform/module argument-evaluation paths, only echo()/assert(), so a recursing call as e.g. a cube() argument still leaks a bogus geometry node into the scene before the abort is caught on the next statement.

---

Generated by Claude Code

[particlesector]

Re-reviewed. Commit 6a168c0 addresses the earlier feedback correctly: checkRecursionAbort() now runs right after evalNode()'s std::visit dispatch returns, before the mods (highlight/root/background) handling, so a recursion abort discovered anywhere during a node's own evaluation — not just child statements — discards that node's result instead of letting a degenerate leaf leak into the scene. Verified the placement is before the mods branches (so an aborted node also can't get pushed into m_rootOnlyNodes/backgroundRoots), and the new regression test (cube(crash()); cube(5); → empty roots, one diagnostic) matches the fix.

No further issues found from this pass.

---

Generated by Claude Code