[wasm] JIT should validate that modeled helper-call return arity matches the real helper signature

[lewing]

Description

We have now hit the same bug three times, each found only by a runtime trap in a browser/WASI R2R run:

• [wasm] Model class-init helper as value-returning in the JIT #130813 — CORINFO_HELP_INITCLASS / CORINFO_HELP_INITINSTCLASS modeled TYP_VOID while the managed helpers return void*.
• fa248fe58e8 — the inline-unbox slow-path CORINFO_HELP_UNBOX modeled TYP_VOID while CastHelpers.Unbox returns ref byte.
• [wasm][R2R] objectalloc retargets CORINFO_HELP_UNBOX to CORINFO_HELP_UNBOX_TYPETEST without updating the call's return type #131377 / [wasm] CORINFO_HELP_CHECK_OBJ is modeled TYP_VOID in fgMorphInit but JIT_CheckObj returns Object* #131378 — objectalloc retargeting UNBOX → UNBOX_TYPETEST without retyping, and CORINFO_HELP_CHECK_OBJ modeled TYP_VOID while JIT_CheckObj returns Object*.

The root cause is structural. On wasm, CodeGen::genCallInstruction derives the emitted call_indirect / call signature from the JIT-modeled node type:

const var_types callRetType = call->IsFastTailCall() ? (var_types)call->gtReturnType : genActualType(call);
if (call->ShouldHaveRetBufArg() || (callRetType == TYP_VOID)) { typeStack.Push(CORINFO_WASM_TYPE_VOID); }

but nothing validates that the modeled type matches the helper's real signature. On register machines a "model a value-returning helper as void because the result is unused" shortcut is harmless, so these shortcuts are scattered through the JIT (importer.cpp, morph.cpp, objectalloc.cpp, ...) and are individually invisible until wasm traps at runtime — typically deep in a cold path, with a bare function signature mismatch and no attribution to the JIT site that caused it.

There is also no central table of helper return types in the JIT. HelperCallProperties (src/coreclr/jit/utils.cpp) tracks purity, allocation, no-GC, throw behavior — but not return arity. jithelpers.h knows the backing METHOD__* for DYNAMICJITHELPER entries, and corelib.h knows their signatures, but the JIT never cross-checks.

Proposal

Add a debug-only verification so these mismatches fail at JIT time with a clear message rather than as a wasm engine trap:

1. In a Checked/Debug wasm JIT, when building the call signature in genCallInstruction (or earlier, in gtNewHelperCallNode and after any gtCallMethHnd retarget), resolve the helper's real signature via eeGetMethodSig(eeFindHelper(helper)) and assert that the modeled return arity (void vs. non-void) matches.
2. Emit the helper name and the two conflicting types in the assert message so the failing site is immediately identifiable.

Optionally, a static table of expected return types per CorInfoHelpFunc (analogous to HelperCallProperties) would let this be checked on all targets and would make the intent explicit at each call site.

Given there is currently no wasm-R2R CI leg, an assert like this is the cheapest way to stop this class of bug from being found one production trap at a time.

Related: #129622, #130634, #130813, #131377, #131378

Note

This issue was created with assistance from GitHub Copilot (AI).

[dotnet-policy-service]

Tagging subscribers to 'arch-wasm': @lewing, @pavelsavara
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.