Interacting from JavaScript to C functions taking or returning 64 bit integers

[fadams]

I was wondering if there was any documentation on passing values from JavaScript to or from compiled C functions that take 64 bit integers (e.g. longs).

From what I can see the compiled functions have signatures that take the low and high 32 bits as separate parameters and in the case or returning longs it appears that the variable tempRet0 gets populated to contain the most significant (I think...) 32 bits with the least significant 32 bits returned in the normal function return value. Is that correct?

So it's definitely possible to interact with longs, just not especially obvious or convenient and some documentation would be really useful.

Are there any helper functions for converting from JavaScript numbers to low/high long values and back - again that's be really useful.

I did notice that long.js contains the Google Long implementation and

goog.math.Long.fromNumber()

actually yields low/high values that can be supplied to compiled functions, but unfortunately I can't actually use the one from the emscripten runtime because that has been "wrapped" in i64Math and the exported Wrapper doesn't get populated with the methods needed to manipulate the number to low/high.

I didn't notice any other functions to do this are there any?

[kripken]

Basically as you said 64-bit values are returned as 32-bits lower in the return and the higher 32 bits in a global variable, tempRet0.

Runtime.makeBigInt converts one way, the other is fairly trivial.

The long.js stuff adds extra things like proper printing of arbitrary precision values.

[fadams]

One other thing that I've discovered since my initial post is that the tempRet0 thing seems to break on me when I compile (link really) with -O2.
I've got code that looks like this:

    // Getting the tracker is a little tricky as it is a 64 bit number. The way
    // emscripten handles this is to return the low 32 bits directly and pass
    // the high 32 bits via the tempRet0 variable. We use Data.Long to pass the
    // low/high pair around to methods that require a tracker.
    var low = _pn_messenger_incoming_tracker(this._messenger);
    var high = tempRet0;
console.log("get low = " + low);
console.log("get high = " + high);
    return new Data.Long(low, high);

which works fine without optimisations, but with -O2 the high gets reported as undefined :-( I don't suppose that you know of a reliable way to sort that out?

[kripken]

I don't follow. Perhaps make a tiny testcase showing the issue?

[fadams]

I'll try to put a little case together, but basically when I have my LINK_FLAGS with -O2 set the tempRet0 isn't set i.e. it's undefined - I see the following on my console:
get low = 0
get high = undefined

whereas if I have no optimisations then I get the correct value in tempRet0. This seems to be consistent, I've had to return 64 bit numbers in a few places and they all seem to behave in the same way i.e. working with no optimisation and not working with.

bear in mind that I'm talking about trying to call a compiled C function from my native JavaScript binding/wrapper. My money is that with -O2 as far as I'm aware the compiled code is asm.js and (I think) closure compiled (though the wrapper isn't closure compiled unless I also do --closure 1) so I bet that tempRet0 isn't being "exported" to the native JavaScript and I'm not sure how to make it visible - normally one uses Module but it might not be appropriate for a temp return value.

Does that make my question make more sense?

[kripken]

In asm mode, the tempRet0 might be in the asm module. See setTempRet etc.
But it might be a bug too, testcase would be good.

On Mon, Mar 31, 2014 at 11:26 AM, fadams notifications@github.com wrote:

I'll try to put a little case together, but basically when I have my
LINK_FLAGS with -O2 set the tempRet0 isn't set i.e. it's undefined - I see
the following on my console:
get low = 0
get high = undefined

whereas if I have no optimisations then I get the correct value in
tempRet0. This seems to be consistent, I've had to return 64 bit numbers in
a few places and they all seem to behave in the same way i.e. working with
no optimisation and not working with.

bear in mind that I'm talking about trying to call a compiled C function
from my native JavaScript binding/wrapper. My money is that with -O2 as far
as I'm aware the compiled code is asm.js and (I think) closure compiled
(though the wrapper isn't closure compiled unless I also do --closure 1) so
I bet that tempRet0 isn't being "exported" to the native JavaScript and I'm
not sure how to make it visible - normally one uses Module but it might not
be appropriate for a temp return value.

Does that make my question make more sense?

Reply to this email directly or view it on GitHubhttps://github.com//issues/2265#issuecomment-39123437
.

[fadams]

Here is the simplest test case that I could come up with. In my scenario I am writing pure JavaScript bindings to a C library so that it can be called idiomatically from native JavaScript.

test.c

#include <stdio.h>

long long test() {
    long long x = ((long long)1234 << 32) + 5678;
    return x;
}

testbind.js

(function() { // Start of self-calling lambda used to avoid polluting global namespace.

var Module = {
    'noExitRuntime' : true
};

Module['runtest'] = function() {
    var low = _test();
    var high = tempRet0;

    console.log("low = " + low);
    console.log("high = " + high);
};

testbindend.js

})(); // End of self calling lambda used to wrap library.

testrun.js

test = require("./test.js");
test.runtest();

compile with no optimisations using

emcc -s EXPORTED_FUNCTIONS="['_test']" --pre-js testbind.js --post-js testbindend.js test.c -o test.js

when I do
node testrun.js
the result is:

low = 5678
high = 1234

As I'd expect/like, however when compiled with -O2

emcc -s EXPORTED_FUNCTIONS="['_test']" --pre-js testbind.js --post-js testbindend.js -O2 test.c -o test.js

The result is:

low = 5678
high = undefined

As I mentioned previously. In other words the most significant word that gets populated into tempRet0 in the unoptimised case is "hidden"/obfuscated in the asm.js/closure processing of the optimised case i.e. the variable is not being "exported".

I had a look around the asm module but I couldn't see anything that was exposing/exporting tempRet0 though as you say above setTempRet seems to be exported in asm, though that doesn't help here.

Hope I'm starting to make sense now?

[fadams]

@kripken have you managed to give this any more thought? Hopefully the testcase I posted illustrated what I'm seeing.

At a guess some unminified accessor in Module or Runtime is all that would be needed to sort this? I'm guessing Runtime might be more appropriate for this?

As an aside wrt. actually using the Longs made from the two 32 bit values you said previously "Runtime.makeBigInt converts one way, the other is fairly trivial." I'm not so convinced that it's entirely trivial looking at the Google Long.fromNumber. In the simple case of a positive long the low value can be calculated via % and the high value can be calculated via / but for the case of computing the low and high from a negative number the code goes:

else if (value < 0) {
        return Data.Long.fromNumber(-value).negate();
    }

and the negate method itself comprises not() and add() where the add is pretty non-trivial.

I ended up largely copy'n'pasting these things from Long.js because the emscripten wrapper only exposes a small part of the behaviour - maybe that's OK because wrapping calls to 64 bit returning C functions might be a relatively uncommon use case, but it's a slight shame given that the code for what I want to do is actually present in unminified code albeit not accessible.

[kripken]

If it's optimizations preventing this from working, we need to find out which. Does building with -O2 -g still show the issue? How about with vs without closure? (Closure can easily break this kind of code, since tempRet0 is a global variable which it will happily rename.)

[fadams]

It was exactly as in the test case I posted above, with mo optimisation it was fine, but with -O2 and above it was failing, it fails without closure, that is to say using only -O2 without using --closure.

I've not tried the -g and won't be able to for a few days as I've currently got limited IT, but as I say above it looks most likely to be the asm.js and the minification around that which is obfuscating the variable. So I'm not explicitly using closure, but -O2 does minify a good chunk of the compiled code.

[kripken]

Ok, yes, could be the asm.js minifier then. If -g works, that would confirm
it.

On Tue, Apr 15, 2014 at 1:16 AM, fadams notifications@github.com wrote:

It was exactly as in the test case I posted above, with mo optimisation it
was fine, but with -O2 and above it was failing, it fails without closure,
that is to say using only -O2 without using --closure.

I've not tried the -g and won't be able to for a few days as I've
currently got limited IT, but as I say above it looks most likely to be the
asm.js and the minification around that which is obfuscating the variable.
So I'm not explicitly using closure, but -O2 does minify a good chunk of
the compiled code.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2265#issuecomment-40456167
.

[fadams]

@kripken there is definitely something odd going on with this!

When I compiled the test case I described previously (please do try it yourself) with -O2 -g it still fails, with "high = undefined", however it did reveal something interesting when I looked at the compiled test.js

What I saw was that tempRet0 was declared twice, the first comes from preamble.js somewhere around line 252 but the second place that it's declared is within the scope of the asm closure. That seems to get generated via emscripten.py

That is to say when compiling with -O2 (asm enabled) the tempRet0 being used by the compiled C function is the one inside the scope of the asm closure and it isn't being exported anywhere, hence looks undefined to external functions trying to use it.

I did a bit of hacking around the compiled test.js and removed the "var" from the front of tempRet0 in the asm closure so that it used the global tempRet0 and that worked, I then hacked the emscripten.py to prevent the "var" from being generated in the first place and again that worked with -O2, however for good measure I then added --closure 1 to the mix and once again it started to fail.

With the --closure 1 what seems to be happening is that, without the var present in asm, the tempRet0 doesn't get minified there, however it is minified in the testbind.js "runtest" function so it can't see the real tempRet0 - if you see what I mean.

It's interesting that there are a bunch of externalised setTempRet* functions generated, I'm guessing that the right approach may be to create a bunch of getTempRet* functions to be generated by emscripten.py but I don't know then if getTempRet0 would only be present when asm.js was enabled and thus I might need something to try tempRet0 and if that was undefined then try getTempRet0.

TBH this is all a bit frustrating, especially as I'm not all that familiar with how emscripten.py works, it'd be great if you could take a look at this, I don't think that the current behaviour is correct really.

[kripken]

Yes, there is a potential problem with double-declaration of tempRet0. It
is kind of a hack. The global one (first appearance) is the original, then
when we moved to asm.js, we had to add one inside the asm.js function as
asm.js has no upvars (it can't access the outer one). For that reason, code
outside the asm closure must call a function to set it inside the asm
module as well (I think we have asm.setTempRet or such for that).

• Alon

On Tue, Apr 22, 2014 at 9:08 AM, fadams notifications@github.com wrote:

@kripken https://github.com/kripken there is definitely something odd
going on with this!

When I compiled the test case I described previously (please do try it
yourself) with -O2 -g it still fails, with "high = undefined", however it
did reveal something interesting when I looked at the compiled test.js

What I saw was that tempRet0 was declared twice, the first comes from
preamble.js somewhere around line 252 but the second place that it's
declared is within the scope of the asm closure. That seems to get
generated via emscripten.py

That is to say when compiling with -O2 (asm enabled) the tempRet0 being
used by the compiled C function is the one inside the scope of the asm
closure and it isn't being exported anywhere, hence looks undefined to
external functions trying to use it.

I did a bit of hacking around the compiled test.js and removed the "var"
from the front of tempRet0 in the asm closure so that it used the global
tempRet0 and that worked, I then hacked the emscripten.py to prevent the
"var" from being generated in the first place and again that worked with
-O2, however for good measure I then added --closure 1 to the mix and once
again it started to fail.

With the --closure 1 what seems to be happening is that, without the var
present in asm, the tempRet0 doesn't get minified there, however it _is_minified in the testbind.js "runtest" function so it can't see the real
tempRet0 - if you see what I mean.

It's interesting that there are a bunch of externalised setTempRet*
functions generated, I'm guessing that the right approach may be to create
a bunch of getTempRet* functions to be generated by emscripten.py but I
don't know then if getTempRet0 would only be present when asm.js was
enabled and thus I might need something to try tempRet0 and if that was
undefined then try getTempRet0.

TBH this is all a bit frustrating, especially as I'm not all that familiar
with how emscripten.py works, it'd be great if you could take a look at
this, I don't think that the current behaviour is correct really.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2265#issuecomment-41059503
.

[fadams]

Yes there is definitely asm.setTempRet* and a bunch of other exported functions.

TBH I've not seen where setTempRet* functions get called anywhere though.

I'm guessing from your comments that the answer to my issue would be to modify emscripten.py to include (and export) getTempRet* functions?

A few things I'm not clear on though you say "code outside the asm closure must call a function to set it inside the asm module" so this implies that code that needs to access certain variables may behave differently when compiled with or without -O2, which seems undesireable. To be fair this is the first case of this problem occurring for me, so perhaps it's an edge case, certainly for the case of key functions it seems like they are exported and also provided with aliases, for example:

var _test = Module["_test"] = asm["_test"];
var _memcpy = Module["_memcpy"] = asm["_memcpy"];
var _strlen = Module["_strlen"] = asm["_strlen"];
var _memset = Module["_memset"] = asm["_memset"];
var runPostSets = Module["runPostSets"] = asm["runPostSets"];
var dynCall_ii = Module["dynCall_ii"] = asm["dynCall_ii"];
var dynCall_v = Module["dynCall_v"] = asm["dynCall_v"];
var dynCall_iii = Module["dynCall_iii"] = asm["dynCall_iii"];
var dynCall_vi = Module["dynCall_vi"] = asm["dynCall_vi"];

I guess calling compiled functions is more common than accessing "global" variables, but nevertheless as I say it's not so great to have different behaviours when asm.js is enabled.

Another question is that I also have worries about things like HEAP8 - for example I see the following in the asm closure:

  var HEAP8 = new global.Int8Array(buffer);
  var HEAP16 = new global.Int16Array(buffer);
  var HEAP32 = new global.Int32Array(buffer);
  var HEAPU8 = new global.Uint8Array(buffer);
  var HEAPU16 = new global.Uint16Array(buffer);
  var HEAPU32 = new global.Uint32Array(buffer);
  var HEAPF32 = new global.Float32Array(buffer);
  var HEAPF64 = new global.Float64Array(buffer);

Now I think that this isn't an issue because although HEAP8 etc. is "double-declared" it's looking like it points to the same underlying memory block as the original global HEAP8 from preamble.js is that correct?

[fadams]

Hi again @kripken I've submitted #2311 which resolves this issue by providing a Runtime.getTempRet0() accessor that is visible whatever optimisation levelis used.

[kripken]

The heap double declares are ok, as they all alias the singleton
arraybuffer.

On Thu, Apr 24, 2014 at 12:54 AM, fadams notifications@github.com wrote:

Hi again @kripken https://github.com/kripken I've submitted #2311https://github.com/kripken/emscripten/pull/2311which resolves this issue by providing a Runtime.getTempRet0() accessor
that is visible whatever optimisation levelis used.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2265#issuecomment-41252889
.

[stale]

This issue has been automatically marked as stale because there has been no activity in the past 2 years. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.