fix(async): #256 spec-compliant microtask ordering for plain async functions

[proggeramlug]

Summary

Closes #256. Pre-fix Perry's async function declarations compiled to direct-call functions whose body ran synchronously to completion, with each await lowered to a busy-wait poll loop. This diverges from spec semantics: an await should always yield to the microtask queue, even on already-resolved Promises, so synchronous code following an unawaited async call runs before the awaited body's continuation.

The reproducer from the issue now matches Node byte-for-byte:

async function inner() {
    console.log("inner-1");
    await Promise.resolve();
    console.log("inner-2");
}
async function main() {
    console.log("main-1");
    await inner();
    console.log("main-2");
}
main();
console.log("top-1");
console.log("top-2");

Pre-fix Perry: main-1 / inner-1 / inner-2 / main-2 / top-1 / top-2
Post-fix Perry & Node: main-1 / inner-1 / top-1 / top-2 / inner-2 / main-2

Approach

A CPS / state-machine transform that mirrors the existing function* generator transform. Pure HIR — no new runtime helpers, which sidesteps the prior prototype's LLVM constant-folding mystery (the done: 0.0 sentinel mutating to TAG_TRUE between IR call site and runtime function entry under stripped -O3 builds).

Six pieces:

1. New crates/perry-transform/src/async_to_generator.rs pre-pass — for every top-level function with is_async=true && !is_generator:

   • Hoists non-top-level awaits (e.g. console.log(\"x: \" + await y)) into fresh let __awaitN = await y; so every Await ends up in a position the generator transform's linearize_body can split states at.
   • Rewrites Expr::Await(x) → Expr::Yield { value: Some(x), delegate: false }.
   • Flips flags: is_async=false, is_generator=true, was_plain_async=true.

2. Existing transform_generator_function extended — when was_plain_async is set, replaces the plain return iter_obj with an inline async-step driver:

   const __iter = iter_obj;
   let __step;
   __step = (v, isErr) => {
       try { r = isErr ? __iter.throw(v) : __iter.next(v); }
       catch (e) { return Promise.reject(e); }
       if (r.done) return Promise.resolve(r.value);
       return Promise.resolve(r.value).then(
           v => __step(v, false),
           e => __step(e, true),
       );
   };
   return __step(undefined, false);

   The two-step let __step; __step = ...; pattern is required because Perry's closure-capture analysis silently produces NaN for the const f = (...)=>f(...) form.

3. linearize_body Try-arm guard widened to also fire when finally has yields (the prior prototype's blocker linux compilation and README #2). await using desugars to try { body } finally { await dispose() } and pre-fix the yield-in-finally hit codegen's Expr::Yield => double_literal(0.0) arm and got fire-and-forgotten.

4. Hoisted-Let rewrite extended to recursively descend into nested control-flow (For init/body, While body, If branches, Try, Switch, Labeled). Without this a for-of loop inside a state body keeps its inner let v = arr[i] as a Let that creates a shadow slot hiding the outer captured box; manifested as for (const v of arr) sum += v returning sum=0 inside transformed async functions.

5. compute_max_local_id extended (in both async_to_generator.rs and generator.rs) to scan class member bodies — the v0.5.323 issue Class method body cannot capture enclosing-function locals #212 capture rewrite allocates method-local fresh ids per class method per captured outer local; without this scan, the generator transform's freshly-allocated state/done/sent/wrapper ids could collide and corrupt unrelated class-method codegen.

6. js_promise_run_microtasks now unwraps thenables when the .then callback returns a Promise — pre-fix the runtime stored the Promise pointer directly as the chained promise's value, so the async-step driver's recursive step() returns produced Promise<Promise<...>> chains that never unwrapped to the real value (await fn() returned Promise { <pending> } for any async function with multiple awaits).

Conservative scope

To avoid exposing pre-existing closure-capture bugs that the new state-machine path surfaces in different ways, the pre-pass skips:

• Module-level: any module with classes carrying __perry_cap_* instance fields (the v0.5.323 issue Class method body cannot capture enclosing-function locals #212 capture rewrite marker). The async-step driver's fresh LocalId allocations can collide with the v0.5.323 method-local rebind ids and the collision is path-dependent.
• Per-function: bodies containing nested closures with non-empty captures (forEach/map/filter pattern). The js_box_set: null box pointer warning surfaces when an inner closure's captured-from-outer box ends up null at runtime.

Both are tracked separately as follow-up work to fully fix the closure-capture-inside-async-state-machine class of bugs. The issue #256 microtask-ordering reproducer (no classes, no nested closures) gets the full fix.

Test plan

• test_issue_256_microtask_ordering.ts (the issue's exact reproducer) — passes byte-for-byte against node --experimental-strip-types
• Gap test suite: 25/28 = main baseline (same 3 documented failures — array_methods / console_methods / typed_arrays)
• test_async, test_async2-5, test_async_chain — top-level await of async function with multi-await chains, all pass
• test_issue_235_method_default_param_padding — the test the issue called out as exposed by the eager-await gap, now passes byte-for-byte
• test_issue_154_using_dispose — await using exercises the linearize_body finally-side fix, passes
• test_issue_212_class_method_capture — class-method-captures-outer pattern, passes (validates conservative scope works)
• test_issue_233_async_array_param_push — async + Array.push + for-of, passes (validates the hoisted-Let recursive descent)
• test_issue_236_fetch_then_console_log, test_issue_240_interface_dispatch, test_issue_167_loop_alloca_stack_eat, test_issue_221_module_array_index_set — all pass

Out of scope (follow-ups)

• Closure-capture-inside-async-state-machine: the conservative module-level skip prevents new regressions but doesn't fix the underlying issue. A real fix needs the async-step driver's wrapper to coordinate with the v0.5.323 capture rewrite at HIR construction time.
• Async generators (async function*) and for await of — separate feature surface; these already had is_generator=true from lowering so the pre-pass intentionally skips them. Tracked as async/await: spec-compliant microtask ordering (eager-await semantics gap) #256 follow-up if needed.
• Nested async closures (arrow / function expressions assigned to locals) — pre-pass operates on module.functions only.