Phase 2 inline lambdas: closure capture#265
Merged
Merged
Conversation
Phase 2 of inline lambdas. Phase 1 (#247) lifted (params>ret;body) literals to synthetic __lit_N decls and rejected free vars with ILO-P017. Phase 2 lifts that restriction: free variables in the body become trailing capture params on the lifted decl, and the call site materialises a closure value that carries them. This commit adds the AST node only. MakeClosure { fn_name, captures } represents the construction of that closure value at the original call site. resolve_aliases_expr walks the capture exprs so symbol aliases still resolve inside them. Codegen, verify, and runtime support follow in subsequent commits.
Keep the non-runtime visitors total over the new AST node. - fmt: render as fn_name[cap1 cap2 ...] for ast-dump/debug output. No surface syntax (MakeClosure is synthetic), so round-trip isn't a goal here, just printer totality. - python: emit (lambda *_a: fn_name(*_a, c1, c2, ...)) so the Python backend reproduces the tree interpreter's call shape (per-item args first, then trailing captures). - graph: record an edge to the lifted fn and walk capture exprs so the dep graph reflects everything the closure touches. - json: closures aren't serialisable, error explicitly like FnRef does. - verify: walk capture exprs so free-var-resolution errors surface at the original call site; report Ty::Unknown for the closure itself so HOF signatures treat it like a fn-ref.
Runtime side of Phase 2 inline-lambda capture.
- New Value::Closure { fn_name, captures } carrying by-value snapshots
of the captured free variables. Constructed by Expr::MakeClosure in
eval_expr; serialises to <closure:fn> in value_to_json.
- resolve_fn_ref accepts Value::Closure so existing HOF dispatch picks
the lifted fn name out of it.
- closure_captures() helper returns the trailing args to append after
per-item args at each call site. Extended every closure-aware HOF —
srt, map, flt, fld, partition, flatmap, uniqby, grp — to call it
once per dispatch and extend call_args before invoking.
- Plain user-call path (Expr::Call) honours closures bound to locals
too: if the resolved callee is Value::Closure, dispatch to its
fn_name and append its captures after the user-supplied args.
Same call shape as #186's single-ctx form generalised to N captures.
By-value semantics: captures are snapshot when the closure is
constructed, not read live at each call.
Drop the Phase 1 ILO-P017 rejection. Phase 2 turns each free
variable in an inline lambda body into a trailing param on the
synthetic __lit_N decl and emits Expr::MakeClosure at the call
site so the runtime can snapshot the captures by value.
- Free-var analysis (already used by Phase 1 to detect captures)
now drives the lift: each free name gets a Param { name, ty: Any }
appended to the lifted decl's param list after the originals.
- If the free-var set is empty, behaviour is unchanged: the call site
is still a bare Expr::Ref(name) like Phase 1.
- collect_free_in_expr handles Expr::MakeClosure so a nested
lambda's captures bubble correctly through an outer lambda — the
inner closure's captures are exprs in the outer scope and must be
scanned as free-var candidates from the outer's perspective.
Captures line up as trailing args on the lifted decl, so the
interpreter's HOF dispatch (`call_args.extend(captures.iter()...)`)
matches the decl signature with no per-call adapter.
VM/Cranelift HOF dispatch is the parked FnRef NaN-tagging effort,
so inline lambdas with captures only run on the tree interpreter
for now. Originally I panicked in compile_expr for Expr::MakeClosure,
but that taints the default runner: vm::compile is called speculatively
in main.rs before the Cranelift JIT runs, and an unwinding panic
crashes the whole process instead of falling back.
Switch to the existing first_error pattern: a new variant
CompileError::UnsupportedClosureCapture { fn_name } gets recorded
on encounter, compile_expr returns a dummy register and continues,
and compile_program propagates the error at the end. main.rs's JIT
path already treats a vm::compile Err as 'skip the JIT, run on the
tree', so the program runs as expected on the default runner.
Wire the new variant through Diagnostic::from with code ILO-R012
so --run-vm and --run-cranelift surface a friendly error instead
of a panic.
Replace the two Phase 1 'capture rejected with ILO-P017' tests with eight Phase 2 capture-works tests, one per scenario: - closure_capture_single_var_filter: flt over a captured threshold - closure_capture_in_sort_key: srt key reading a captured target - closure_capture_in_map: map transform bumping by a captured int - closure_capture_in_fld: fld reducer weighted by a captured int - closure_capture_multiple_vars: flt over both lo and hi captures - closure_capture_text_value: flt with a captured Text needle - closure_capture_by_value_snapshot: srt with bias capture (sanity that the by-value snapshot path runs) run_err stays in the file (now under #[allow(dead_code)]) — it's documentation of how to assert against ilo failures and other regression suites copy it in. All tests run on --run-tree only: VM/JIT HOF dispatch is the parked FnRef NaN-tagging follow-up.
Six small functions, one per scenario: threshold filter, distance sort, bump map, range filter (two captures), substring filter (Text capture), and weighted fold. Each case is asserted by the examples_engines harness with -- run: / -- out:. engine-skip lists vm, jit, cranelift because HOF dispatch on those backends is the parked FnRef NaN-tagging follow-up. Tree-only until that lands.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
5 tasks
danieljohnmorris
added a commit
that referenced
this pull request
May 15, 2026
- slc / take / drop accept negative indices counting from end (bounds clamp), matching at xs i. Closes the quant-trader fencepost and the slc xs -np 1 np ergonomics gap (#266). - Map keys are typed: text or integer. mset m 7 v and mget m 7 work directly, no str conversion. Int(1) and Text("1") are distinct. Float keys floor to i64; jdmp stringifies numeric keys for JSON (#267). - Add map / flt / fld to the builtin reference. All HOFs (map, flt, fld, srt, grp, uniqby, partition, flatmap) now work cross-engine on tree, VM, Cranelift JIT, and AOT (#274 #277 #278 #279 #280 #283). - New Inline lambdas subsection: Phase 1 literals are cross-engine, Phase 2 closure capture is tree-only with automatic fallthrough surfacing ILO-R012 on VM and Cranelift (#265 #284). - AOT-compiled binaries from ilo compile now strip the top-level ~/^ wrapper byte-for-byte the same as in-process runners (#281).
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 2 of inline lambdas. #247 shipped Phase 1:
(params>ret;body)literals lift to synthetic__lit_Ntop-level decls, and free variables in the body got rejected with ILO-P017 pointing at the ctx-arg form. Phase 2 lifts that restriction.Closure capture is the natural next step for the manifesto: writing
srt (x:n>n;abs -x target) xsto sort by distance from a captured target is the obvious form, and forcing the user to either define a helper or threadtargetthrough a ctx-arg costs tokens for no good reason. The ctx-arg form (#186) still works; this just makes the inline form do what an LLM expects.Tree-only. VM and Cranelift HOF dispatch depend on FnRef NaN-tagging, which is the parked follow-up tracked in
fix/vm-cranelift-hof-dispatch. The example carries-- engine-skip: vm jit craneliftandvm::compilereturns a friendlyCompileError::UnsupportedClosureCapture(ILO-R012) so the default runner falls through to the tree interpreter cleanly instead of panicking.Repro
Before this PR (Phase 1):
After:
What's in the diff
Seven small commits, one logical change each:
Expr::MakeClosure { fn_name, captures }, walk captures inresolve_aliases_expr.Value::Closure { fn_name, captures }; thread captures through every closure-aware HOF (srt,map,flt,fld,partition,flatmap,uniqby,grp); Expr::Call path honours closures bound to locals too._:anyparams on the synthetic decl; emitExpr::MakeClosureat the call site.CompileError::UnsupportedClosureCapture(was a panic) sovm::compilereturns Err cleanly and the default runner falls through to tree. New diagnostic code ILO-R012.examples/inline-lambda-capture.ilowith six cases asserted via-- run:/-- out:.Test plan
cargo fmt --checkcleancargo clippy --release --features cranelift --all-targets -- -D warningscleancargo test --release --features cranelift— full suite green (2888 passed, 0 failed, 73 ignored)tests/examples.rs(default JIT-then-tree runner) green: confirmsvm::compileErr falls through cleanlytests/examples_engines.rshonoursengine-skipFollow-ups
fix/vm-cranelift-hof-dispatch). Once that lands,Value::Closureneeds a NanVal encoding and the example can dropengine-skip.Expr::Callcallee resolution is wired but not exhaustively tested; worth adding a regression once a real use case surfaces.