requiring a module in nodejs

[noamtcohen]

Hi,

I would like to load a module and use it from nodejs.
Lets say I comile a c source file like this:

 emcc in.c -o out.js -s EXPORTED_FUNCTIONS="['_aProcedure']"

What I do now in module.js I use eval to load the script and export the Emscripten Module:

eval(require("fs").readFileSync("./out.js").toString());
module.exports.MyModule = Module;

My main.js file looks like this:

var m = require('./module.js').MyModule;
m._aProcedure();

A better way would be to require the emscripten output file:

var m = require('./out.js');
m._myProcedure();

And that means that I have to give emcc a pre-js that would like this:

module.exports._myProcedure = Module._myProcedure;

And I think the prejs could be generate by emcc since it mirrors the EXPORTED_FUNCTIONS flag, and I'm thinking maybe to add a flag to emcc like --node-module 1 that would tell emcc to emit this pre js.

Now lets say I invoke emcc like this:

 emcc in.c -o out.js --memory-init-file 1 -O3 -s MODULARIZE=1 -s EXPORT_NAME=\"'MyModule'\" -s EXPORTED_FUNCTIONS="['_aProcedure']"

If I want to use a memory init file, emscripten should be able to load this file via NODEFS.
And lastly if I want to MODULARIZE then emcc should generate this pre js:

module.exports.MyModule = MyModule;

So then when I want to use the module I would do this:

var m = require('./out.js');
var myModule = m.MyModule();
myModule._aProcedure();

[kripken]

This does look more natural for node.js usage. Should be simple to add those exports, we can check at runtime if ENVIRONMENT_IS_NODE. The code would probably go in src/shell.js. Pull request would be welcome.

[noamtcohen]

I'm not sure where to start... could you give some direction?
Also what about the memory init file?

[kripken]

Sure. src/shell.js is the basic JS template for all of our output. It has code to set things up, including detecting whether we are node (look for ENVIRONMENT_IS_NODE). If we are, then we can do the things you mentioned above. So hopefully this can be done by adding a little code to that file.

For the memory init file, see src/postamble.js. Looks like in node (and other shells) we call Module['readBinary'] to load the file. In node, that should already use the node FS stuff, so I'm not sure if anything else would be needed - it should already work?

[noamtcohen]

In shell.js how do I know what was specified in EXPORT_NAME and EXPORTED_FUNCTIONS?
Can you also show a line that would emit, lets say console.log('Hi'); from shell.js to the output file?

[kripken]

If you write {{{ EXPORT_NAME }}} then the preprocessor will replace that text with the value of that variable.

It can be useful to look at shell.js, then build a little program and look at that output. Then you can see how shell.js ends up in the output, as well as what modifications are made (like those {{{ }}} templates).

What do you mean "to the output file"? If you add console.log('Hi') to shell.js, then that will be included in the output from emcc. And then it will run when the output is executed, and print that to the console.

[noamtcohen]

Got it. Thanks.
Where do I add tests so that I don't break existing use cases?

[kripken]

In tests/test_other.py.

[noamtcohen]

ok, I'll have to learn a bit of python. I'm up to this.

[noamtcohen]

I got everything setup.
I've added this line at the top of shell.js

console.log({{{ EXPORTED_FUNCTIONS }}});

This is my c test file module_exports.c:

#include<stdio.h>

__attribute__((used))
int api1() {
  return 101;
}
__attribute__((used))
int api2() {
  return 102;
}
__attribute__((used))
int api3() {
  return 103;
}

emscripten/emcc module_exports.c -s EXPORTED_FUNCTIONS="['_api1','_api2','_api3']"

node a.out.js

console.log([object Object]);
                     ^^^^^^
SyntaxError: Unexpected identifier

How do I tell the preprocessor to give me the array value and not it's type (not it's toString() function)?

[kripken]

You can use JSON.stringify to print out the full array, including its contents.

[noamtcohen]

I tried that, didn't work. Anyways the solution is simpler then I thought, all I really have to export is the module name:

module.exports.{{{ EXPORT_NAME }}} = {{{ EXPORT_NAME }}};

Since you would also need all the emscripten APIs like _free and the HEAP views.
So there is not real need to export the specific functions.

[kripken]

Makes sense, and nice it's not a big change.

[noamtcohen]

I've made this changed: https://github.com/noamtcohen/emscripten/blob/master/src/shell.js#L115
and added a small test: https://github.com/noamtcohen/emscripten/tree/master/tests/node-module-export
but I'm not sure how to add it to tests/test_other.py

[kripken]

Just add another test at the end of that file. For example, at the end right now (on incoming) is

  def test_memory_growth_noasm(self):
    check_execute([PYTHON, EMCC, path_from_root('tests', 'hello_world.c'), '-O2', '-s', 'ALLOW_MEMORY_GROWTH=1'])
    src = open('a.out.js').read()
    assert 'use asm' not in src

which will run emcc on "hello_world.c" in tests/, with -O2 and some additional flags. Then it reads the output file and does an assert on that. For your test, perhaps you want to run the code and see that it works, look for run_js examples in that file (you probably want something like seen_output = run_js('a.out.js', engine=NODE_JS)).

[noamtcohen]

Alon I'm sorry, just looked at the code again and found this test: https://github.com/kripken/emscripten/tree/master/tests/Module-exports
This functionality is already supported, I've tested this, it's just implemented in a different way and it's not documents.

[kripken]

I'm sorry for not remembering that...

Why then didn't this just work for you?

Perhaps all we need to do is improve the documentation?

[noamtcohen]

There are a few ways to export an object in node:

module.export.obj = {}
// I would then require in the using file:
var m = require("./the-file.js").obj;

/* or */

module.export = {}
// And then:
var m = require("./the-file.js");

The first one gives you more flexibility as to what you are exporting and this is what I was looking for.
You are using the second method.
Yes, I think the documentation should be improved, this is the only link I found about this:
http://stackoverflow.com/questions/9020050/using-an-emscripten-compiled-c-library-from-node-js

[kripken]

I see, thanks.

Do you want to submit a PR to update the docs, perhaps? Something in the "interacting with code" page might make sense.

[noamtcohen]

Sure, where are the docs? this is what I found but where is the content? https://github.com/kripken/emscripten/blob/42fb486c53d627b203b77af6e5d0c088c0ad03ad/site/source/docs/porting/connecting_cpp_and_javascript/Interacting-with-code.rst#id2
I should commit to incoming and submit a PR, right?

[kripken]

Yes, that link is to the content. It is built using Sphinx into the documentation on the emscripten.org site. A PR with additions to that file is exactly what we need, and yes, to incoming.

[noamtcohen]

What do think of this?
https://github.com/noamtcohen/emscripten/blob/incoming/site/source/docs/porting/connecting_cpp_and_javascript/Interacting-with-code.rst#interacting-with-an-api-written-in-cc-from-nodejs

[kripken]

Looks nice! Do you want to open a pull request to the incoming branch with that?

[noamtcohen]

I'm glad you like it. I have to ask, I committed this to the incoming branch of my fork, is this right?
Seems to me like I didn't understand you correctly a few comments back. Should I commit this to my master brach and open a pull request from my master to your incoming ?

[kripken]

What you committed sounds right: First you commit to your incoming, then you open a pull request to the main incoming here.

[noamtcohen]

I messed-up the repo and couldn't understand how to clean it up so I just re-forked and created a new pull request: #4059

Regarding you question about require('./a.out.js') please see: https://nodejs.org/api/modules.html#modules_all_together
basically if I omit ./ node will try to load a core module or from local/global node_modules directory. If I want to load a file, I must prefix with ./

out of context:
I think this would be a interesting repo for you to see: https://github.com/eranpeer/FakeIt

[kripken]

Ok, pull landed so I think we can close this.

[noamtcohen]

longest 40 lines I ever wrote...

[kripken]

Heh, I hope that wasn't too annoying for you. Sometimes it takes a while to figure out what to do, and the proper thing in the end is small :)

[noamtcohen]

It wasn't annoying at all, I had fun. Thanks!

[sijakret]

is the node exporting (via module.exports) supposed to work in conjunction with MODULARIZE=1 ?
This options wraps everything into a global function and, the assignment to module.exports is done inside that function. However, this function never gets exported so it cannot be called to do the actual export.
I think in the node js +Modularize case the export has to go outside of the function to actually export it. Or am i missing something?

[curiousdannii]

@sijakret Which version of emscripten are you using? That sounds like how it used to be, but it's been fixed for a while.

[sijakret]

@curiousdannii
..
* sdk-1.35.0-64bit INSTALLED
..

[curiousdannii]

@sijakret That's over two years old. There's been like 60 releases since then. You need to update emscripten :)

[sijakret]

omg, this is so stupid..
somehow my emsdk update runs never did anything (i didn't see an error complaining about not being able to write somewhere).´

Got a recent version of the portable SDK and the update + wasm build worked right away.
thank you!

[juj]

omg, this is so stupid..
somehow my emsdk update runs never did anything (i didn't see an error complaining about not being able to write somewhere).´

Apologies for this - in the history there was one update of emsdk that was a lemon, see this: https://groups.google.com/forum/#!topic/emscripten-discuss/bKCQxN6Q76Y