Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of memory or out of time when compiling large JS files #111

Closed
jeaye opened this issue Dec 28, 2017 · 25 comments
Closed

Out of memory or out of time when compiling large JS files #111

jeaye opened this issue Dec 28, 2017 · 25 comments

Comments

@jeaye
Copy link

@jeaye jeaye commented Dec 28, 2017

Do you want to request a feature or report a bug?
Bug.

What is the current behavior?
Metro will either run out of memory or it will give up after 5 minutes of waiting to transform the JS file.

This was first reported here facebook/react-native#8475 and then again here facebook/react-native#12590

If the current behavior is a bug, please provide the steps to reproduce and a minimal repository on GitHub that we can yarn install and yarn test.
I have a repro repo here https://github.com/jeaye/react-native-packager-bug -- it's not in the yarn format, since I don't know what that is.

What is the expected behavior?
The transformation completes in a timely fashion or metro waits long enough for the transformation to finish in however long it needs. In order to achieve the latter, my production build script currently contains the following:

metro=../node_modules/metro-bundler/src/JSTransformer/index.js
sed -i 's/\(TRANSFORM_TIMEOUT_INTERVAL\) = .*;/\1 = 901000;/g' "$metro" || true

Please provide your exact Metro configuration and mention your Metro, node, yarn/npm version and operating system.

React Native: 0.50.4
babel-preset-react-native: 4.0.0
Metro (comes with RN): 0.20.3
Node: 9.3.0
OS: Arch Linux x86_64

@jeaye
Copy link
Author

@jeaye jeaye commented Jan 18, 2018

Happy new year! Any update on this?

@rafeca
Copy link
Contributor

@rafeca rafeca commented Jan 20, 2018

Hey @jeaye ! Happy new year and apologies for the delay.

The timeout on transformations was indeed [removed some time ago]( (d4dcd4c), so from metro v0.21.0 there is no maximum time to transform a file.

Out of curiosity, can you provide more details about the file that you're trying to compile? How big is it? Why is it so big? Metro is heavily optimized to build bundles with a really big amount of small files, but cannot run that fast when trying to compile a small amount of huge files (it's impossible to parallelize, the caching system is not efficient, etc(.

@rafeca rafeca closed this Jan 20, 2018
@jeaye
Copy link
Author

@jeaye jeaye commented Jan 20, 2018

@rafeca In production, my index.android.js is 1.2MB; these out-of-memory and timeout issues happen even before the 1MB mark though. Why is it that big? Well, it's a mid-sized application, about 10K LOC of ClojureScript, compiled with Google Closure.

As linked above, this issue is affect basically everyone using ClojureScript with RN, likely due to the CLJS stdlib and everything else included in the compiled file. Note, this is going through Google Closure, so it's not like there's a lot of waste here which we can trim down; all dead code is removed and all remaining code is optimized and minified.

Closing this issue because the timeout is removed is ok, but, really, waiting over 5 minutes to transform these files is pretty awful. Trying to do this with a non-production build (which doesn't have Google Closure optimizations applied) is a fool's errand, since it'll take more like 30 minutes, if it doesn't run out of memory before that.

@rafeca
Copy link
Contributor

@rafeca rafeca commented Jan 21, 2018

@jeaye : thanks for the explanation! it makes complete sense 😄

In this case, since your index.android.js file has already been transpiled, I suggest you to try to disable babel transformation for that file and check how fast the transformation is (spoiler: it should be way faster). You can do that via the ignore param in your babel config.

Anyways, this flow that you describe is still going to be slow and painful in development mode (as you describe, every time you do a change you have to recompile the whole bundle from clojurescript to javascript, and then metro will see the modified file so it won't be able to leverage its cache and will do the whole bundling).

If possible, a much better flow would be to integrate the clojurescript compiler with metro, so metro would call the compiler on a per-file basis and you will benefit from the speed/parallelization/caching of metro.

I'm not familiar with clojurescript and its compiler, so I don't know if it can even work on a per-file basis, But if it can the integration should not be super-complex:

  1. You need to create you own transformer that transforms a clojurescript file to JS (more specifically, to AST). To get an example, this is our basic transformer
  2. You'll need to configure metro to use the new transformer that you've created. There is the getTransformModulePath option for that (example).

Let me know if this helps

@jeaye
Copy link
Author

@jeaye jeaye commented Jan 21, 2018

@rafeca Thanks for the detailed reply. The ignoring approach was mentioned in the first referenced issue here, but it leads to other issues. When I try it in my production build, rather than my test case repository, I run into issues because my JS actually needs to be transformed. That is, I have some NPM dependencies using newer JS features which need to be transformed for RN to consume them.

The failure looks something like this:

...
:app:bundleReleaseJsAndAssets
Scanning folders for symlinks in my-app/node_modules (4ms)
Scanning folders for symlinks in my-app/node_modules (21ms)
Loading dependency graph, done.
warning: the transform cache was reset.

Export statement may only appear at top level in file "my-app/node_modules/react-native-material-ui/src/RippleFeedback/index.android.js" at line 1:114

:app:bundleReleaseJsAndAssets FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:bundleReleaseJsAndAssets'.
> Process 'command 'node'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED
@rafeca
Copy link
Contributor

@rafeca rafeca commented Jan 21, 2018

@jeaye it seems that there're other files in dependencies called index.android.js, so if you set the ignore param to this, it'll ignore all of them which is not correct.

Have you tried refining the regexp used in the ignore param to only match the huge index.android.js file?

@jeaye
Copy link
Author

@jeaye jeaye commented Jan 21, 2018

@rafeca That's a very good idea; the babel docs don't mention anything of the sort, as far as limiting matches to the top-level directory. I've tried various patterns, from ^index.android.js and ./index.android.js to /index.android.js. Are you aware of how to accomplish limiting the ignoring to only the top-level index.android.js?

My research into how babel looks this up seems like any JS file within node_modules will look for a .babelrc and will continue upward toward / in search of one. Should a catch-all for node_modules be placed at node_modules/.babelrc then, which doesn't have the ignore?

@HikingDev
Copy link

@HikingDev HikingDev commented Feb 10, 2018

Why is this issue closed? I have a json file with 17.4MB.
If i require it the bundler crashes and if comment it out everything works fine.
This seems to be a common problem. Is there a solution for this issue?

"FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory"

@jeaye
Copy link
Author

@jeaye jeaye commented Feb 10, 2018

@roesneb It certainly shouldn't be closed. It was ignored when it was reported, multiple times, in the react native repo, while people suggested it be brought up here. Now that it's brought up here, it's promptly closed and no workarounds are available (for people using ClojureScript).

For your problem, you might try ignoring that JSON file using your babelrc. Alas, if it has a common name, then all JSONs with that name will be ignored, which is the last problem about which I asked @rafeca.

@HikingDev
Copy link

@HikingDev HikingDev commented Feb 10, 2018

@jeaye i've tried the ignoring option already, without success. I could split up my JSON in multiple files but I don't think that should be the way to do it.

@VincentFTS
Copy link

@VincentFTS VincentFTS commented Feb 19, 2018

Any news on this "not solved" issue ?

@jeaye
Copy link
Author

@jeaye jeaye commented Feb 19, 2018

@VincentFTS If this is possible:

Have you tried refining the regexp used in the ignore param to only match the huge index.android.js file?

Then the problem can be resolved for ClojureScript users. No documentation I've found has shown it to be possible though. Instead, renaming your app's index.android.js to myapp.index.android.js or something, may do the trick.

@VincentFTS
Copy link

@VincentFTS VincentFTS commented Feb 20, 2018

The problem is that the "big file" is a javascript file that uses flow, it permits to check the format of a big table.
Hence I cannot add it to the ignore section of the babelrc file.

@ghost
Copy link

@ghost ghost commented Apr 20, 2018

Hello @jeaye

Did you end up solving this issue for CLJS? Our iOS build is functioning fine, but our android build not so well. On a previous version of react native, We did all sorts of monkey patching to make the transformers not run for our ClojureScript files it was horrible, but it worked. After upgrading to the latest RN we were hoping to avoid the money patching if possible.

@jeaye
Copy link
Author

@jeaye jeaye commented Apr 20, 2018

@ChadHarrisVerr Nope, this is still very much an issue. metro doesn't handle large files well and has a non-linear time growth with respect to file size. So far, there's been no official way to speed things up and this ticket, just like many others here and in the react-native repo, has been closed with little help (read the full discussion here to understand more, if you haven't).

I have documented my current workaround here: https://github.com/cljsrn/cljsrn-org/wiki/Avoiding-Metro-timeouts In short, I patch metro to not give up after several minutes of transforming and I just wait a long time for my prod builds. As a result of this, I also can't reasonably use something like jest for testing, since it wants a bundled package as well.

@sundbry
Copy link
Contributor

@sundbry sundbry commented Oct 16, 2018

In order to fix this, you'll need to use a custom Metro transformer to skip loading the entire compiled Clojurescript AST, and also load its dependencies properly. I've documented how to do it here:

https://www.arctype.co/blog/fixing-react-native-compiler-clojurescript-timeouts

cc @bonlemuel @ChadHarrisVerr

@boogie666
Copy link

@boogie666 boogie666 commented Jan 22, 2019

Hi @sundbry,

The article from arctype is great and all :P but i can't get it to work...

i keep getting TypeError: Cannot read property 'transformFile' of undefined
in /node_modules/metro/src/Bundler.js:77:34

Can you help with that?

(btw i'm trying to solve a different problem, where the bundler is eating some functions from clojurescript core causing a wierd error, and no, it's not an externs thing :P)

@sundbry
Copy link
Contributor

@sundbry sundbry commented Jan 22, 2019

@boogie666 Metro changed the interface here in v0.47.0:

f8cfe20

Try reverting your metro to 0.45.4

@boogie666
Copy link

@boogie666 boogie666 commented Jan 30, 2019

@sundbry Lifesaver :* thx a lot :*

@sundbry
Copy link
Contributor

@sundbry sundbry commented Feb 1, 2019

FYI including @boogie666 the RN cljs transformer has been updated to work with the new interfaces in current metro 0.49 : https://gist.github.com/sundbry/55bb902b66a39c0ff83629d9a8015ca4

@familyfriendlymikey
Copy link

@familyfriendlymikey familyfriendlymikey commented Feb 21, 2019

The ignore param doesn't seem to be working. I only need to ignore one file in the app's root dir, yet the transform will still hang when node crashes at 1.43GB of memory usage. If I remove the JSON file, the app builds for release config just fine, so I know even though I'm ignoring the file, babel is still parsing it.

babel.config.js

module.exports = {
  presets: ["module:metro-react-native-babel-preset"],
  ignore: [
    "filename.json"
  ]
}

Any ideas? Can't seem to find a solution anywhere. Nothing has had an impact, including attempting to change node's max_old_space_size.

@leasontou
Copy link

@leasontou leasontou commented Jul 31, 2019

The ignore param doesn't seem to be working. I only need to ignore one file in the app's root dir, yet the transform will still hang when node crashes at 1.43GB of memory usage. If I remove the JSON file, the app builds for release config just fine, so I know even though I'm ignoring the file, babel is still parsing it.

babel.config.js

module.exports = {
  presets: ["module:metro-react-native-babel-preset"],
  ignore: [
    "filename.json"
  ]
}

Any ideas? Can't seem to find a solution anywhere. Nothing has had an impact, including attempting to change node's max_old_space_size.

I have this problem too, how did you solve it?

@familyfriendlymikey
Copy link

@familyfriendlymikey familyfriendlymikey commented Jul 31, 2019

Never really did, I ended up having to just download the whole database from within the app on first startup and show a loading screen.

Either that or depending on your use case you might be able to make the database smaller by changing long keys to something shorter. That got me under the threshold of size that would actually build, pretty big difference. I changed every key to be one character long.

Hopefully this gets fixed some time, wishing you luck.

@leasontou
Copy link

@leasontou leasontou commented Jul 31, 2019

Never really did, I ended up having to just download the whole database from within the app on first startup and show a loading screen.

Either that or depending on your use case you might be able to make the database smaller by changing long keys to something shorter. That got me under the threshold of size that would actually build, pretty big difference. I changed every key to be one character long.

Hopefully this gets fixed some time, wishing you luck.

Thank you, you are a genius, but it doesn't suit my situation.😟

@leasontou
Copy link

@leasontou leasontou commented Jul 31, 2019

I have solved it.
There are some large js files in my project, I increase the memory limit for Node.js, and it works for me.

Add following content in ~/.bash_profile

export NODE_OPTIONS=--max-old-space-size=4096

I'm not sure it can suit you, but maybe it could help someone else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.