vm + cranelift: native flt + fld + flatmap HOF dispatch (PR 3a of 4) by danieljohnmorris · Pull Request #278 · ilo-lang/ilo

danieljohnmorris · 2026-05-15T09:47:30Z

Summary

Extends the map native HOF dispatch from #277 to the next three pure-bytecode HOFs: flt, fld, flatmap. Each gets a real bytecode loop in the compiler that calls back into the FnRef via OP_CALL_DYN, so VM and Cranelift run them end-to-end with no tree-bridge round-trip.

Builds on #274 (FnRef NaN-tagging) and #277 (map native + OP_CALL_DYN lowering). The Cranelift JIT helper for OP_CALL_DYN is dispatch-uniform across HOFs, so the new arms light up --run-cranelift for free.

This is PR 3a of a four-PR chain that closes out HOF dispatch on VM/Cranelift. The remaining HOFs (grp, uniqby, 2-arg srt, partition), closure-bind ctx-arg forms, and direct FnRef-variable calls land in PR 3b, 3c, 3d, then PR 4 lifts the lambda examples.

Repro before/after

sq x:n>n;*x x
pos x:n>b;>x 0
add a:n b:n>n;+a b
main xs:L n>L n;flt pos xs

ilo --run-vm        # before: error "unsupported HOF". after: [2, 4]
ilo --run-cranelift # before: error "unsupported HOF". after: [2, 4]
ilo --run-tree                                                # [2, 4]

Same story for fld add xs 0 (now 15) and flatmap rep xs (now [1, 2, 2, 3, 3, 3]).

What's in the diff (per-commit)

vm: native flt + fld + flatmap HOF dispatch using OP_CALL_DYN. Three new arms in RegCompiler::compile_expr mirroring the (Builtin::Map, 2) shape from vm + cranelift: native map HOF dispatch (PR 2 of 4) #277. flt adds an OP_ISBOOL typecheck and routes non-bool through OP_WRAPERR + OP_PANIC_UNWRAP. fld allocates a 3-reg contiguous slice for (res, arg0, arg1) and overwrites the accumulator each iter. flatmap nests an inner FOREACHPREP/NEXT over each call result. Drops #[ignore] from vm_flt_positive, vm_flt_predicate_returns_non_bool, vm_flt_wrong_list_arg, vm_fld_sum, vm_fld_wrong_list_arg. Updates the is_tree_bridge_eligible doc comment to reflect what's still parked for PR 3b.
cranelift aot: lower OP_PANIC_UNWRAP through jit_panic_unwrap helper. The JIT already supported this op; AOT (compile_to_binary) was the remaining gap. Without it, flt-using programs would refuse to compile to a binary because the non-bool predicate guard emits OP_PANIC_UNWRAP unconditionally. Pure parity with the JIT lowering, no new behaviour for tree or VM.
test + example: cross-engine coverage for flt + fld + flatmap. 14 cross-engine assertions in tests/regression_hof_flt_fld_flatmap.rs covering happy paths, empty lists, single-element, all-pass / all-fail, mixed inner sizes (flatmap), text accumulator (fld), and composition with map. Adds examples/flt-basics.ilo and examples/fld-sum.ilo with -- run / -- out directives so tests/examples_engines.rs exercises them across every engine.
example: lift vm/cranelift skip from flatmap.ilo. Drops the two engine-skip lines and the explanatory comment.

Test plan

cargo build --release --features cranelift clean
cargo test --release --features cranelift — 4847 passed, 0 failed, 66 ignored (down from 71)
cargo fmt --check clean
cargo clippy --release --features cranelift --all-targets -- -D warnings clean
tests/regression_hof_flt_fld_flatmap.rs passes on tree + VM + Cranelift (14/14)
tests/examples_engines.rs passes (includes new flt-basics.ilo, fld-sum.ilo, and lifted flatmap.ilo)
The 5 previously-ignored HOF tests pass

Follow-ups

PR 3b: grp, uniqby, 2-arg srt, partition. These need either new opcodes (Map ops, Set ops, sort) or a tree-bridge variant that takes the active AST Program for user-fn callback resolution. Approx 17 more Dynamic dispatch (OP_CALL_DYN emission) arrives in PR 2. tests get unignored.
PR 3c: closure-bind 3-arg ctx forms (map fn ctx xs, etc.) across all HOFs. Lifts engine-skip from closure-bind.ilo.
PR 3d: direct FnRef-variable calls (f=dbl; f 10). Unignores vm_fnref_callee_from_scope, vm_fn_ref_via_ref_expr, vm_text_callee_from_scope, vm_user_hof_fn_type.
PR 4: inline lambda Phase 1 cross-engine lift.

Extends the `map` pattern from #277 to the next three pure-bytecode HOFs. Each gets a native bytecode loop in the compiler that calls back into the FnRef via OP_CALL_DYN, so VM and Cranelift both run them end-to-end: - flt fn xs: same FOREACHPREP/NEXT shape as map, but the call result drives a bool typecheck (OP_ISBOOL → OP_JMPT) and the OP_LISTAPPEND only fires on the true arm. Non-bool predicate returns raise via OP_WRAPERR + OP_PANIC_UNWRAP with a message containing "flt" and "bool" (matches the tree-walker semantics pinned by the existing tests). - fld fn xs init: no result list. Accumulator reg carries the running value across iterations; the per-iter call is 2-arg (acc, item), result is the final acc. - flatmap fn xs: nested foreach. The outer loop drives the user fn over xs; the inner FOREACHPREP/NEXT iterates each call result and splices it into acc via OP_LISTAPPEND. A non-list return surfaces as "foreach requires a list" through the inner FOREACHPREP error path, which the test asserts matches. is_tree_bridge_eligible's doc-comment drops flt/fld/flatmap from the not-yet list, parking grp/uniqby/partition/2-arg srt for the next PR in the chain. Also drops the `#[ignore]` from vm_flt_positive, vm_flt_predicate_returns_non_bool, vm_flt_wrong_list_arg, vm_fld_sum, and vm_fld_wrong_list_arg, which pinned the absence of these code paths.

The JIT path already supports OP_PANIC_UNWRAP via the same helper (jit_panic_unwrap in src/vm/mod.rs); the AOT codegen was the remaining gap. Without this, any program using `flt` with the new native dispatch refuses to compile to a binary because the predicate non-bool guard emits OP_PANIC_UNWRAP unconditionally. The lowering mirrors the JIT side: declare the helper FuncId, emit a call to it on the failed value, then `return` the helper's TAG_NIL result so subsequent instructions don't execute against the bad state. The helper itself sets JIT_RUNTIME_ERROR which the post-call check in the caller surfaces as a runtime error, the same contract the VM dispatcher uses with `vm_err!`. No new behaviour for tree or VM — only AOT parity.

Pins the value-level behaviour of the three new native HOF lifts across --run-tree, --run-vm, and --run-cranelift: - flt: positives, empty list, all-pass, all-fail, then map composition (15 assertions per HOF cover empty / single / multi / fully-filtered shapes). - fld: integer sum, empty list returns init, single element, text concat (covers type-changing accumulators), then map composition. - flatmap: rep-n shape from the example file, empty outer, every-empty-inner (pins the inner FOREACHPREP short-circuit on empty results), mixed inner sizes. Builtin-callback coverage stays in regression_hof_map.rs; the Cranelift jit_call_dyn helper is dispatch-uniform across HOFs, so once builtin-callback works for `map` it works for all of them. Adds examples/flt-basics.ilo and examples/fld-sum.ilo as the in-context learning artefacts (per the project convention, also exercised by tests/examples_engines.rs across every engine). The existing examples/flatmap.ilo already covers flatmap.

Native flatmap dispatch landed in the preceding commit, so the example runs on every engine now. Drops the two engine-skip lines and the explanatory comment; the tests/examples_engines.rs harness picks the file up across tree/vm/cranelift automatically.

codecov · 2026-05-15T09:51:03Z

Codecov Report

❌ Patch coverage is 94.32624% with 8 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/vm/compile_cranelift.rs	11.11%	8 Missing ⚠️

📢 Thoughts on this report? Let us know!

- slc / take / drop accept negative indices counting from end (bounds clamp), matching at xs i. Closes the quant-trader fencepost and the slc xs -np 1 np ergonomics gap (#266). - Map keys are typed: text or integer. mset m 7 v and mget m 7 work directly, no str conversion. Int(1) and Text("1") are distinct. Float keys floor to i64; jdmp stringifies numeric keys for JSON (#267). - Add map / flt / fld to the builtin reference. All HOFs (map, flt, fld, srt, grp, uniqby, partition, flatmap) now work cross-engine on tree, VM, Cranelift JIT, and AOT (#274 #277 #278 #279 #280 #283). - New Inline lambdas subsection: Phase 1 literals are cross-engine, Phase 2 closure capture is tree-only with automatic fallthrough surfacing ILO-R012 on VM and Cranelift (#265 #284). - AOT-compiled binaries from ilo compile now strip the top-level ~/^ wrapper byte-for-byte the same as in-process runners (#281).

danieljohnmorris added 4 commits May 15, 2026 10:46

danieljohnmorris merged commit 34a4cf3 into main May 15, 2026
5 checks passed

danieljohnmorris deleted the fix/hof-remaining branch May 15, 2026 09:51

danieljohnmorris mentioned this pull request May 16, 2026

fix: cranelift JIT srt-after-map TLS desync silent miscompile #306

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vm + cranelift: native flt + fld + flatmap HOF dispatch (PR 3a of 4)#278

vm + cranelift: native flt + fld + flatmap HOF dispatch (PR 3a of 4)#278
danieljohnmorris merged 4 commits into
mainfrom
fix/hof-remaining

danieljohnmorris commented May 15, 2026

Uh oh!

codecov Bot commented May 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

danieljohnmorris commented May 15, 2026

Summary

Repro before/after

What's in the diff (per-commit)

Test plan

Follow-ups

Uh oh!

codecov Bot commented May 15, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant