fix: AOT embeds CompiledProgram so HOF / closure dispatch works

[danieljohnmorris]

Summary

Engine audit PR #413 found AOT silently returns nil (or [nil, nil, ...]) for every program that emits OP_CALL_DYN or an OP_*_BY_KEY finalizer whose helper has to re-enter the VM on a user-fn callback. Tree, VM, and Cranelift JIT are all correct on the same shapes. Targeting 0.12.1 since this is a bug fix (silent miscompute -> correct answer), no new API surface, no breaking change.

Root cause: AOT never publishes ACTIVE_PROGRAM / ACTIVE_AST_PROGRAM, so jit_call_dyn and jit_call_builtin_tree hit their null-program guards and silently return TAG_NIL. Fix shape: serialise the full CompiledProgram (chunks + AST + func_names + is_tool + type_registry) with postcard, embed the blob into a .rodata data section in the AOT binary, and add ilo_aot_publish_program(ptr, len) to deserialise and publish the four TLS pointers with_active_registry already sets for the in-process JIT path.

Repro

$ cat /tmp/m.ilo
main>L n;map (x:n>n;*x 2) [1,2,3]

$ ilo --run-tree /tmp/m.ilo main
[2, 4, 6]
$ ilo --run-vm /tmp/m.ilo main
[2, 4, 6]
$ ilo --jit /tmp/m.ilo main
[2, 4, 6]

# Before this PR:
$ ilo compile /tmp/m.ilo -o /tmp/m main && /tmp/m
[nil, nil, nil]

# After this PR:
$ ilo compile /tmp/m.ilo -o /tmp/m main && /tmp/m
[2, 4, 6]

Every shape from the audit's failing rows now matches tree / VM / JIT byte-for-byte under AOT: map over an inline lambda (no capture and with capture), 3-arg closure-bind map fn ctx xs, fld with a user-fn, grp / uniqby with a key fn, and fn-ref return-and-call.

What's in the diff

Five commits:

1. aot: add postcard CompiledProgram wire format - new src/vm/aot_blob.rs module with ProgramBlob / WireChunk / WireConst. Schema_version gated. Chunk constants encode only the variants the compiler emits today (Nil / Number / Text / Bool / List); a future variant trips the From<&Value> guard in the test suite before any binary ships. AST is serialised as JSON inside the postcard envelope because Program::serialize_decls uses serialize_seq(None) which postcard rejects.

2. aot: ilo_aot_publish_program runtime helper - new C ABI helper deserialises the blob, leaks it for the process lifetime (same pattern as ilo_aot_set_registry), and publishes the four TLS pointers. Malformed blob exits 1 with a JSON diagnostic, no silent fallback. ilo_aot_fini now also clears the three new pointers.

3. aot: emit publish_program call in generate_main and the bench harness - declare the helper, serialise the program in compile_to_binary, emit a call from generate_main after the registry setup and before the user-fn call. Same wiring for compile_to_bench_binary's C harness.

4. tests: cross-engine regression for AOT HOF / closure dispatch - eight cases in tests/regression_aot_closures.rs covering each audit failure shape, each compiled to a real AOT binary and compared byte-for-byte against --run-tree, --run-vm, and --jit. Plus examples/aot-closures.ilo as the in-context learning artefact, exercised by the examples_engines harness.

5. docs: AOT now runs Phase 2 closures and HOF / fn-ref dispatch - SPEC.md, ai.txt (regenerated), skills/ilo/SKILL.md, and skills/ilo/ilo-engines.md. Pre-PR these claimed Phase 2 was tree-only with VM/JIT raising ILO-R012, which has been false since feature: VM closure support (Phase 2 PR1) #384 landed; now they describe the actual state including the AOT embed.

Binary size impact

Measured against main on the same machine, same flags:

program	pre-fix	post-fix	delta
main>n;42	9,486,728	9,696,616	+205 KB
main>L n;map (x:n>n;*x 2) [1,2,3]	9,486,744	9,696,648	+205 KB
5-fn program with lambdas + grp + fld	9,486,952	9,696,872	+205 KB

The one-time +205 KB is the libilo.a side: linking in the postcard + serde_json deserialiser code paths reachable from ilo_aot_publish_program. Per-program growth (the actual blob in .rodata) is a few hundred bytes for trivial programs, a few KB for larger ones; in the szhello vs szbig comparison the per-program delta is 256 bytes. Honest trade for fixing the silent-miscompute class.

Test plan

• cargo build --release --features cranelift clean
• cargo test --release --features cranelift clean (192 test files, no failures)
• cargo fmt --check clean
• cargo clippy --release --features cranelift --all-targets clean
• tests/regression_aot_closures.rs exercises the seven audit failure shapes + one regression baseline, all green
• aot_blob module has its own roundtrip + schema-version unit tests
• examples/aot-closures.ilo runs green through tree / VM / JIT via the examples_engines harness
• Manual binary-size measurement before / after

Follow-ups

• Re-run the engine audit harness from PR engine feature-support audit: 49-row corpus + cross-engine harness #413 once that lands to confirm the 9 AOT-fail rows drop to whatever's left after this + the in-flight sum-type-strconst fix.
• AST is currently embedded as JSON inside the postcard envelope. A future PR could ship a native postcard AST by replacing the serialize_seq(None) impl on Program with a Vec-backed one, shrinking the blob a little. Not urgent.

[codecov]

Codecov Report

❌ Patch coverage is 92.25092% with 21 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/vm/mod.rs	83.75%	13 Missing ⚠️
src/vm/aot_blob.rs	95.29%	8 Missing ⚠️

📢 Thoughts on this report? Let us know!