[wasm][R2R] objectalloc retargets CORINFO_HELP_UNBOX to CORINFO_HELP_UNBOX_TYPETEST without updating the call's return type

[lewing]

Description

ObjectAllocator's post-order rewrite in src/coreclr/jit/objectalloc.cpp retargets an existing CORINFO_HELP_UNBOX call node to CORINFO_HELP_UNBOX_TYPETEST when the unboxed object is a stack-allocated box, but it never updates the call node's modeled return type:

// src/coreclr/jit/objectalloc.cpp
call->gtCallMethHnd = m_compiler->eeFindHelper(CORINFO_HELP_UNBOX_TYPETEST);
GenTree* const mt   = m_compiler->gtNewMethodTableLookup(lcl, /* onStack */ true);
call->gtArgs.Remove(secondArg);
call->gtArgs.PushBack(m_compiler, NewCallArg::Primitive(mt));

The two helpers have different return arity:

Helper	Managed implementation	Returns
CORINFO_HELP_UNBOX	CastHelpers.Unbox(MethodTable*, object)	ref byte
CORINFO_HELP_UNBOX_TYPETEST	CastHelpers.Unbox_TypeTest(MethodTable*, MethodTable*)	void

So after the rewrite the node is a TYP_BYREF call to a void-returning helper. gtType/gtReturnType are both stale.

On wasm this is a hard failure, not just sloppy modeling: CodeGen::genCallInstruction builds the call_indirect type from genActualType(call) (see codegenwasm.cpp), so we emit a (...) -> (i32) signature for a helper the runtime dispatches through a ()-returning thunk, and the engine traps with function signature mismatch. This is the same class of defect as #130813 and the inline-unbox slow-path fix.

On other targets it is benign — the result is unused in both the isForEffect and COMMA(call, payloadAddr) shapes — which is presumably why it went unnoticed.

Note this became easier to hit after the inline-unbox slow-path helper started being modeled TYP_BYREF on wasm. Previously that call was TYP_VOID, so isForEffect was true and the void model happened to match Unbox_TypeTest. Now the inline path also reaches objectalloc as TYP_BYREF.

Reproduction Steps

Browser/WASI R2R with object stack allocation enabled (the default) on code where a stack-allocated box flows into an unbox / unbox.any, e.g. a boxed value type that does not escape and is later unboxed.

Expected behavior

After retargeting to CORINFO_HELP_UNBOX_TYPETEST the call node should be modeled TYP_VOID, matching the helper it now calls, on every target.

Actual behavior

The node keeps TYP_BYREF. On wasm the emitted call_indirect signature does not match the callee and the engine traps with function signature mismatch.

Suggested fix

Set the call's type when retargeting (correct on all targets, not wasm-specific):

call->gtCallMethHnd = m_compiler->eeFindHelper(CORINFO_HELP_UNBOX_TYPETEST);
call->gtType        = TYP_VOID;
call->gtReturnType  = TYP_VOID;

and then compute isForEffect before the retype (or from user == nullptr) so the existing COMMA(call, ADD(obj, TARGET_POINTER_SIZE)) shape is preserved.

Also worth auditing the other helper-retarget sites for the same hazard. helperexpansion.cpp (CHKCASTCLASS → CHKCASTCLASS_SPECIAL) and morph.cpp (user call → CORINFO_HELP_ARRADDR_ST) are both arity-preserving and look fine.

Configuration

browser-wasm / wasi-wasm, R2R, main.

Regression?

Latent on all targets; newly reachable on the inline-unbox path on wasm.

Related: #129622, #130813

Note

This issue was created with assistance from GitHub Copilot (AI).

[dotnet-policy-service]

Tagging subscribers to 'arch-wasm': @lewing, @pavelsavara
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.