New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lumo cannot allocate more than 1.5GB (on 64bit machine) #218

Open
mgerlach-klick opened this Issue Jul 27, 2017 · 4 comments

Comments

Projects
None yet
4 participants
@mgerlach-klick

mgerlach-klick commented Jul 27, 2017

Nodejs has a well-documented heap limit of 1.5GB (on 64 bit machines). The way around this is to specify the --max_old_space_size=NEWLIMIT command line argument for node, but this doesn't work for lumo, nor does the suggested solution of

(def v8 (js/require "v8"))
(.setFlagsFromString v8 "--max_old_space_size=8192")

I am running into this issue when working with large-ish (~60mb) CSV files.

@anmonteiro

This comment has been minimized.

Owner

anmonteiro commented Aug 4, 2017

We can set a better default when compiling Lumo. I'm thinking that 8192 should be enough for most use cases.

Thoughts?

@thosmos

This comment has been minimized.

thosmos commented Nov 20, 2017

I just ran out of heap while attempting to load this huge 189mb json file: https://github.com/zemirco/sf-city-lots-json and then convert it into cljs data with the following line:

(count (js->clj  (js/JSON.parse  (lumo.io/slurp "citylots.json"))))

I'm guessing that a more lazy method would help for something like this.

@mgerlach-klick

This comment has been minimized.

mgerlach-klick commented Nov 20, 2017

FWIW I ended up working around this issue by compiling my lumo script to JS (using shadow-cljs), and then running it "manually" using node and giving it the --max_old_space_size=8192 parameter @thosmos

@arichiardi

This comment has been minimized.

Collaborator

arichiardi commented Nov 23, 2017

I hit this too, did anybody tried the proposed fix? Should we raised the --max-old-space-size ?
Things seem to have been worse with node 9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment