Javascript build take too much memory... #325

Closed
BabyRaptor opened this Issue Mar 28, 2012 · 25 comments

3 participants

@BabyRaptor

I have a game written in C++, which executable file size is about 25MB, and the amount of RAM it use while running is about 40MB.
And then I converted it to Javascript using Emscripten successfully, but the Javascript file size is about 150MB, which is already very big. But the memory Chrome take to run it even bigger, about 1GB. (compare to 40MB of the executable), and increasing as the game run (Chrome will turn Aw-snap when console print out "Enlarging memory array...")

The executable version of the game has no memory leak anywhere.

Is it a bug, or something I haven't think about? How can I reduce the size of the file and the memory it takes?
Thank you.

(In setting.js, I assign only 100MB as heap)
(Firefox refuse to run, throw error: "not enough memory")

@kripken
Owner

Did you compile it with optimizations? -O0 and -O1 are much bigger in code size than -O2. And that can affect how much memory is used at runtime.

If that doesn't fix things, you might have hit a bug in the compiler, because memory usage in the default compilation should be similar to a native app (it uses c-style memory layout).

Can you provide the source code? If not, can you try to make a reduced testcase that you can post?

@BabyRaptor

I'm sorry I cannot provide the code because I was forced to swear not to do so... ^^

But this is the bug it throw when I tried to build at -O2:

JAVASCRIPT COMPILATION FAILED.
emcc: warning: using libcxx turns on CORRECT_* options
Traceback (most recent call last):
File "E:\xxxxxxxxxxxx\tools\CPP2JS/emscripten/emcc", line 637, in
flush_js_optimizer_queue()
File "E:\xxxxxxxxxxxx\tools\CPP2JS/emscripten/emcc", line 618, in flush_js_optimizer_queue
final = shared.Building.js_optimizer(final, js_optimizer_queue)
File "E:\xxxxxxxxxxxx\tools\CPP2JS\emscripten\tools\shared.py", line 695, in js_optimizer
assert len(output) > 0 and not output.startswith('Assertion failed'), 'Error in js optimizer: ' + err + '\n\n' + output
AssertionError: Error in js optimizer:

node.js:201
throw e; // process.nextTick error, or 'error' event on first tick
^

Error: UNKNOWN, unknown error 'E:\src\c:\users\minh~1.hoa\appdata\local\temp\tmptk_o57\gen.bc.ll.o.js'
at Object.openSync (fs.js:230:18)
at Object.readFileSync (fs.js:120:15)
at E:\xxxxxxxxxxxx\tools\CPP2JS\emscripten\tools\js-optimizer.js:96:13
at Object. (E:\xxxxxxxxxxxx\tools\CPP2JS\emscripten\tools\js-optimizer.js:1124:11)
at Module._compile (module.js:441:26)
at Object..js (module.js:459:10)
at Module.load (module.js:348:31)
at Function._load (module.js:308:12)
at Array.0 (module.js:479:10)
at EventEmitter._tickCallback (node.js:192:40)

Did I do something wrong? Some configuration or some other stuff?

EDIT:
It seems that a temp folder was created here:

Error: UNKNOWN, unknown error 'E:\src\c:\users\minh~1.hoa\appdata\local\temp\tmptk_o57\gen.bc.ll.o.js'

But when I go to this folder, instead of the folder "tmptk_o57" like it said, another folder with another random name was created, so the name is different, and it crash obviously...
How can i fix this?

@kripken
Owner

It creates a new random folder each time, that might be confusing.

For the error, I just committed some stuff to the incoming branch that might help. It will merge to master when tests pass. Basically, we have some bugs in Windows that are not yet sorted out, I suspect your error is one of them. To check, try to run the automatic tests,

python tests/runner.py other.test_js_optimizer

That tests what looks like is failing.

@BabyRaptor

This is the test result:

Running Emscripten tests...
(Emscripten: Running sanity checks)
test_js_optimizer (main.other) ...
node.js:201
throw e; // process.nextTick error, or 'error' event on first tick
^
Error: UNKNOWN, unknown error 'E:\src\E:\xxxxxxxxxxx\tools\CPP2JS\emscripten\tools\test-js-optimizer.js'
at Object.openSync (fs.js:230:18)
at Object.readFileSync (fs.js:120:15)
at E:\xxxxxxxxxxx\tools\CPP2JS\emscripten\tools\js-optimizer.js:96:13
at Object. (E:\xxxxxxxxxxx\tools\CPP2JS\emscripten\tools\js-optimizer.js:1124:11)
at Module._compile (module.js:441:26)
at Object..js (module.js:459:10)
at Module.load (module.js:348:31)
at Function._load (module.js:308:12)
at Array.0 (module.js:479:10)
at EventEmitter._tickCallback (node.js:192:40)
ERROR
ERROR

Look at:

Error: UNKNOWN, unknown error 'E:\src\E:\xxxxxxxxxxx\tools\CPP2JS\emscripten\tools\test-js-optimizer.js'

Why it add "E:\src" before the actual path?

(At first time I tried build with -O2, I get an error said that a file "E:\src\utility.js" is not found, although I put the "emscripten" folder somewhere else...
I thought it was hard code so I simple copied the "utility.js" to that folder)

When will you commit the fix for this? I'm looking forward to that... :)

@kripken
Owner

The fix I mentioned before just landed on master.

However, it might not fix all the Windows issues, let me know what problems remain. Not sure if the "E:\src" will be fixed, something Windows-specific is causing that.

@BabyRaptor

Thank you for your kindly support... I've update new emscripten, and try to build with -O2 parameter. Here is the problem I've met:

JAVASCRIPT COMPILATION FAILED.
emcc: warning: using libcxx turns on CORRECT_* options
emcc: warning: using libcxxabi, this may need CORRECT_* options
Traceback (most recent call last):
File "E:\FantasyTown\tools\CPP2JS/emscripten/emcc", line 1070, in
final = shared.Building.eliminator(final)
File "E:\FantasyTown\tools\CPP2JS\emscripten\tools\shared.py", line 725, in eliminator
output, err = Popen([NODE_JS, coffee, eliminator], stdin=PIPE, stdout=PIPE, stderr=PIPE).communicate(input)
File "C:\DevTools\Python26\lib\subprocess.py", line 701, in communicate
return self._communicate(input)
File "C:\DevTools\Python26\lib\subprocess.py", line 911, in _communicate
self.stdin.write(input)
IOError: [Errno 32] Broken pipe

And then I rebuild with no Optimization, it can build Javascript file, but error runtime:

Uncaught TypeError: Cannot read property 'length' of undefined

I traced the code, and found out that the problem was caused by library.js, line 743 and 744, "data" seems to be NULL, so cannot get property "length" of data" (previous build using old tools run normaly)

size = data.length;
blocks = Math.ceil(data.length / 4096);

I guess this was caused by the FS object, so I replace the code of the FS object in the generated JS file, with the old FS object. (I don't remember the revision, but it's maybe 2 months ago... - it's the emscripten I used to build my game successfully) - and the game run normally.

So let me sum it up:

  • What is the change in FS object which cause the error I get, while the old one doesn't?
  • How to build with -O2? Am I missing someting in configuration? (I did install node.js and closure compiler but no V8 and Monkey)
  • When I use closure compiler mannually to compile my game, is it equivalent to -O2? Closure compiler reduce my file size to 43MB, but memory usage still stay the same...

Thank you for your trouble...

@kripken
Owner

The eliminator problem might be fixed on the incoming branch now, I committed more Windows fixes today.

Closure compiler is just one small part of O2 optimizations. You do need -O2 to get it all.

I am not sure what the FS problem you run into is. Can you provide a testcase I can reproduce the problem with and debug?

@BabyRaptor

In coffee eliminator, when I edit:

if os.platform() != 'windows'

to

if os.platform() != 'win32'

And compile O2, the code run forever, and never stop, even on a very small and do nothing project...

About the FS problem:

  • C++ code:

#include "stdio.h"
#include "fstream"
using namespace std;

int main() {
ifstream in("abc");
char str[80];
in >> str;
printf("%s \n", str);

return 0;
}

  • Javascript code:

FS.createLazyFile('/', 'abc', 'abc', true, true); (Add at the end of the generated JAVASCRIPT file)

  • File:
    Create a file name "abc", write something inside it...

  • When run:
    Uncaught TypeError: Cannot read property 'length' of undefined

Well, I build on Windows 32 bit...

Errm... And, what about "printf"? Printf doesn't seems to work anymore after I update Emscripten... What happened?

@kripken
Owner

Can you provide the code that runs forever with -O2 (C++ and generated .js)? Please compile it with --closure 0 so it is more readable.

About FS, that testcase works for me here. Can you provide the generated .js you get for it?

It seems we might have more windows bugs than I thought. I don't have a windows machine so I can't test there myself, sorry.

@BabyRaptor

Any kind of code, even a cpp file with just an empty 'main' function... if I compile with -O2, it'll run forever. I'm sure there's something with my config, not code, but I don't know because it throw no error at all.

About the FS, how can I upload my JS file so you can see it? (I tested on FF, the error is "contents is undefined")

@kripken
Owner

If you get weird errors like that, try to run the test suite. That should at least tell us what works and what doesn't. python tests/runner.py.

You can paste source code in a github gist for example.

@BabyRaptor

It's kind of funny, but I've run the test, and ALL feature ERROR or FAILED...
It's weird because I still can use emscripten to build a runable JS, but just the test. Everything failed, everything error...

@kripken
Owner

Can you run a single test and paste the error?

python tests/runner.py test_hello_world

@BabyRaptor

Running Emscripten tests...
(Emscripten: Running sanity checks)
Checking JS engine ['C:\Users\minh.hoangtuan/Dev/mozilla-central/js/src/js', '-m', '-n', '-e', "gcparam('maxBytes', 102410241024);"] failed. Check~/.emscripten. Details: [Error 2] The system cannot find the file specified
WARNING: Not all the JS engines in JS_ENGINES appears to work, ignoring those.
test_hello_world (main.default) ... ERROR

ERROR: test_hello_world (main.default)

Traceback (most recent call last):
File "tests/runner.py", line 348, in test_hello_world
self.do_run(src, 'hello, world!')
File "tests/runner.py", line 318, in do_run
self.assertContained(expected_output, js_output)
File "tests/runner.py", line 247, in assertContained
additional_info
Exception: Expected to find 'hello, world!' in '', diff:

--- expected
+++ actual
@@ -1,1 +1,1 @@
-hello, world!
+


Ran 1 test in 1.560s

FAILED (errors=1)

@kripken
Owner

Might be a problem with node or llvm.

Can you run the hello world test with EM_SAVE_DIR=1 EMCC_DEBUG=1 python tests/runner.py test_hello_world, and paste the files generated in /tmp/emscripten/temp?

@BabyRaptor

There are 4 files in the temp folder... which one do you want?

In case you want the JS file, it's here: "emcc-3-original.js"
https://gist.github.com/254d44bf7aafa5c35cce

About the FS failed: This is the JS file... It'll load a file name "abc" and print, please create one your self.
https://gist.github.com/4ec5f15338773161b1f7

Return to the memory problem:
I did try to use closure compiler seperately, and end up with a 40MB JS file. Well, a lot better than a 150MB JS file. But the memory the game used is still the same
I split all process and find out:

  • 40MB Javascripts, after loading into Chrome (only load), takes 160MB Ram.
  • The FS object load 80MB Data, result in 160MB Ram in Chrome. (WTF?)
  • The HEAP I assign 100MB, yeah, 100MB to create it. This is obvious.
  • To start running the application, it takes another 120MB RAM to run (although the HEAP was init from the start, and no "enlarging" was called) - Where do these memory come from? isn't Emscripten already use HEAP as memory?

I am very confusing, because as you said, the JS will take memory similar to native apps. Native app only cost 35MB to 40MB to run. Considering the FS load all data, and the HEAP I assign 100MB, it must be around 250MB only for the web version...

@kripken
Owner

The code in the JS file looks like from an old version of emscripten. Can you update to latest master and see if it still fails?

The JS version will take more memory due to preloading files, compared to a native app. Otherwise, it should be similar, except for JS engine's taking memory to optimize the code: JIT code generation, type information, profiling information, etc. That sounds like a lot of memory for those things, so it could be a browser bug or limitation. How does it compare in other browsers?

@BabyRaptor

It can run on Firefox, with the same memory consumption...

I've just update Emscripten again, and now, I can build the game, but it won't run at all. Chrome will crash, Firefox will hang... And not a single error was thrown.

So I replace your new "mmap" code with my old one:

var temp = new Uint8Array(num);
for (i = 0; i < num; i++) {
temp[i] = info.object.contents[offset + i];
}
return allocate(temp, 'i8', ALLOC_NORMAL);

And now it can run, but the memory cost is still the same.

I tried to replace nodeJS with V8Engine. Recreate loop, optimize everything in setting.js, apply closure compiler, apply llvm-opts 3 and llvm-lto 1... The code run with a good speed on Chrome, Firefox a bit slower, Safari have pathetic FPS, IE crash (obviously), but the memory is still a ridiculous problem... :)

@kripken
Owner

So replacing node with v8 as the compiler engine made a difference here?

@BabyRaptor

Yes. It run a bit faster than NodeJS, 2-5 FPS increased on Chrome and FF.
So, can you explain for me how emcc work on each step please?
When emcc compile cpp to bc, it'll use Clang right? What optimization can be applied to this step?
When emcc compile bc to js, it'll use nodeJS or V8 right? Why call --llvm-opts and --llvm-lto on this step?
What will -O2 do (I'll do it step by step, and see which step did it hang forever...)

Thank you.

@crazyjul

When emcc compile bc to js, it converts the binary code to text llvm bitcode. During this process, any argument can be passed to llvm. It's that text that emscripten parse and use to generate javascript. If you compile any file with EMCC_DEBUG=1, you will find a bunch of files in /tmp/emscripten_{random}/, including that text file

@BabyRaptor

Oh. In case of Chrome. After all kind of optimizations, I finally get it run at 350MB RAM.
But...
After I left my computer (still let Chrome run the game) for about 1 hour, when I got back, Chrome only took ~150MB. And the game was still running normally. So, the GC cleaned a lot of "useless" chunk of memory...
Can you tell me if the "useless memory" was cause by my own cpp code or something inside Emscripten? All the memory the apps need will be provide in the HEAP right? And never got cleaned up right?

@kripken
Owner

How optimizations work is documented on

https://github.com/kripken/emscripten/wiki/Optimizing-Code
https://github.com/kripken/emscripten/wiki/Building-Projects

If memory got reclaimed over time, I would assume it was temporary JIT optimization info in the browser. But it's hard to tell, we can't see or control that stuff.

@BabyRaptor

OK. Thank you anyway... I guess this is a limit that I cannot break through.

@kripken
Owner

I recommend you file a bug on Chrome and Firefox if they take too much memory. Please link to the bugs here if you file them.

@kripken kripken closed this Jul 12, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment