ptrToIdx() and idxToPtr() macros to unify 2GB, 4GB and Wasm64 programming

[juj]

In the old PR #19289 I proposed three helper functions {{{ ptrToIdx() }}}, {{{ convertPtrToIdx() }}} and {{{ idxToPtr() }}} to provide a unified programming model for converting between Wasm pointers and pre-shifted typed array heap indices.

There was some pushback in the PR, so the feature did not land.

I ended up landing the feature into Unity's fork of Emscripten, relegating to the idea that I would make those a Unity .js library {{{ macro }}}s.

But later on I realize that it is not actually possible for Emscripten users to generate "system level" JS library wide {{{ }}} macros, the parsing order is a PITA to deal with.

So I would like to gauge the temperature of whether we might revisit landing {{{ ptrToIdx() }}}, {{{ convertPtrToIdx() }}} and {{{ idxToPtr() }}} as end-user programmable macros into Emscripten?

Given that Wasm produces pointers as 64-bit BigInts, or int32s that need to be >>> 0d in 4GB mode, it would be ideal to have one programming model to cover all of these cases.

Currently Emscripten has ad hoc #if CAN_ADDRESS_2GB handling in some call sites:

emscripten/src/lib/libstrings.js

Lines 118 to 120 in 6ca3700

	#if CAN_ADDRESS_2GB
	ptr >>>= 0;
	#endif

and ad hoc #if MEMORY64 handling in others:

emscripten/src/lib/libccall.js

Lines 54 to 56 in 6ca3700

	#if MEMORY64
	if (returnType === 'pointer') return Number(ret);
	#endif

along with to64() and from64() macros that perform pointer<->heap index conversion in Wasm64 mode only.

The automatic foo__sig: '....' model helps automate this challenge in some cases, but it has undesirable interactions with JSPI, and it results in pessimization of generated code size.

In the juj/wasm_webgpu library, I ended up needing the following constructs for most convenient handling of 2GB+4GB+Wasm64: https://github.com/juj/wasm_webgpu/blob/54c51d715d68f22a62e1f84eb072737e8a8270c0/lib/lib_webgpu.js#L26-L88

(where the functions are a bit awkwardly named to avoid possible name conflict that I was anticipating with #19289)

Would there be interest in revisiting the idea of adding {{{ ptrToIdx() }}}, {{{ convertPtrToIdx() }}} and {{{ idxToPtr() }}} macros? I would like to be able to provide a single documented API to Unity users so that they can manage 2GB, 4GB and Wasm64 programming models in a unified fashion.

[sbc100]

But later on I realize that it is not actually possible for Emscripten users to generate "system level" JS library wide {{{ }}} macros, the parsing order is a PITA to deal with.

Can you explain more about this limitation? Is it worth fixing this maybe?

[sbc100]

The automatic foo__sig: '....' model helps automate this challenge in some cases, but it has undesirable interactions with JSPI, and it results in pessimization of generated code size.

Are you proposing to replace the automatic __sig stuff manaul macros / conversion in the JS code itself? Or would be more or and either / or?

[juj]

Can you explain more about this limitation? Is it worth fixing this maybe?

When doing this, we parse --js-library on the command line in the order of presence. (I added a test to this some years back). but in build systems, ensuring that certain files need to appear before others, is tricky; especially when multiple sub-libraries are linked together. I tried implementing this into a C#-based build system, but it ended up being a massive headache.

Are you proposing to replace the automatic __sig stuff manaul macros / conversion in the JS code itself? Or would be more or and either / or?

It would be up to the user to use either __sig or the manual macros.

[juj]

Currently we do already have ad hoc generated macros that did not get much design into them:

• {{{ from64('foo') }}} which is like the proposed {{{ ptrToIdx('foo') }}}, but from64() only works in Wasm64 mode, and not in 4GB mode.
• {{{ to64('foo') }}} which is like the proposed {{{ idxToPtr('foo') }}}, but to64() also is Wasm64 only.
• A common recurrence of #if CAN_ADDRESS_2GB ... ptr >>> 0; ... #endif that could be removed/subsumed into a {{{ ptrToIdx('foo') }}}.
• {{{ getHeapOffset('value', 'i32') }}} would be like {{{ ptrToIdx('value', 2) }}}, the benefit of the latter would be that it allows using the same macro for both shifted (u32) and unshifted (u8) indices, and also to perform computation on the shift index, e.g. at main...Unity-Technologies:3.1.38.4-unity#diff-331632581554705597c8ef7b7e699b3925acfb4a68e98fee5e30ecf4e9b0a2c2R161

A robust implementation of those macros would also recognize when integer or BigInt literals or zeroes are passed, to help e.g. perform compile-time arithmetic on, say, idxToPtr(someConstant) or idxToPtr('0') for a null pointer.