Out of memory or out of time when compiling large JS files

[jeaye]

Do you want to request a feature or report a bug?
Bug.

What is the current behavior?
Metro will either run out of memory or it will give up after 5 minutes of waiting to transform the JS file.

This was first reported here react/react-native#8475 and then again here react/react-native#12590

If the current behavior is a bug, please provide the steps to reproduce and a minimal repository on GitHub that we can yarn install and yarn test.
I have a repro repo here https://github.com/jeaye/react-native-packager-bug -- it's not in the yarn format, since I don't know what that is.

What is the expected behavior?
The transformation completes in a timely fashion or metro waits long enough for the transformation to finish in however long it needs. In order to achieve the latter, my production build script currently contains the following:

metro=../node_modules/metro-bundler/src/JSTransformer/index.js
sed -i 's/\(TRANSFORM_TIMEOUT_INTERVAL\) = .*;/\1 = 901000;/g' "$metro" || true

Please provide your exact Metro configuration and mention your Metro, node, yarn/npm version and operating system.

React Native: 0.50.4
babel-preset-react-native: 4.0.0
Metro (comes with RN): 0.20.3
Node: 9.3.0
OS: Arch Linux x86_64

[jeaye]

Happy new year! Any update on this?

[rafeca]

Hey @jeaye ! Happy new year and apologies for the delay.

The timeout on transformations was indeed [removed some time ago]( (d4dcd4c), so from metro v0.21.0 there is no maximum time to transform a file.

Out of curiosity, can you provide more details about the file that you're trying to compile? How big is it? Why is it so big? Metro is heavily optimized to build bundles with a really big amount of small files, but cannot run that fast when trying to compile a small amount of huge files (it's impossible to parallelize, the caching system is not efficient, etc(.

[jeaye]

@rafeca In production, my index.android.js is 1.2MB; these out-of-memory and timeout issues happen even before the 1MB mark though. Why is it that big? Well, it's a mid-sized application, about 10K LOC of ClojureScript, compiled with Google Closure.

As linked above, this issue is affect basically everyone using ClojureScript with RN, likely due to the CLJS stdlib and everything else included in the compiled file. Note, this is going through Google Closure, so it's not like there's a lot of waste here which we can trim down; all dead code is removed and all remaining code is optimized and minified.

Closing this issue because the timeout is removed is ok, but, really, waiting over 5 minutes to transform these files is pretty awful. Trying to do this with a non-production build (which doesn't have Google Closure optimizations applied) is a fool's errand, since it'll take more like 30 minutes, if it doesn't run out of memory before that.

[rafeca]

@jeaye : thanks for the explanation! it makes complete sense 😄

In this case, since your index.android.js file has already been transpiled, I suggest you to try to disable babel transformation for that file and check how fast the transformation is (spoiler: it should be way faster). You can do that via the ignore param in your babel config.

Anyways, this flow that you describe is still going to be slow and painful in development mode (as you describe, every time you do a change you have to recompile the whole bundle from clojurescript to javascript, and then metro will see the modified file so it won't be able to leverage its cache and will do the whole bundling).

If possible, a much better flow would be to integrate the clojurescript compiler with metro, so metro would call the compiler on a per-file basis and you will benefit from the speed/parallelization/caching of metro.

I'm not familiar with clojurescript and its compiler, so I don't know if it can even work on a per-file basis, But if it can the integration should not be super-complex:

1. You need to create you own transformer that transforms a clojurescript file to JS (more specifically, to AST). To get an example, this is our basic transformer
2. You'll need to configure metro to use the new transformer that you've created. There is the getTransformModulePath option for that (example).

Let me know if this helps

[jeaye]

@rafeca Thanks for the detailed reply. The ignoring approach was mentioned in the first referenced issue here, but it leads to other issues. When I try it in my production build, rather than my test case repository, I run into issues because my JS actually needs to be transformed. That is, I have some NPM dependencies using newer JS features which need to be transformed for RN to consume them.

The failure looks something like this:

...
:app:bundleReleaseJsAndAssets
Scanning folders for symlinks in my-app/node_modules (4ms)
Scanning folders for symlinks in my-app/node_modules (21ms)
Loading dependency graph, done.
warning: the transform cache was reset.

Export statement may only appear at top level in file "my-app/node_modules/react-native-material-ui/src/RippleFeedback/index.android.js" at line 1:114

:app:bundleReleaseJsAndAssets FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:bundleReleaseJsAndAssets'.
> Process 'command 'node'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

[rafeca]

@jeaye it seems that there're other files in dependencies called index.android.js, so if you set the ignore param to this, it'll ignore all of them which is not correct.

Have you tried refining the regexp used in the ignore param to only match the huge index.android.js file?

[jeaye]

@rafeca That's a very good idea; the babel docs don't mention anything of the sort, as far as limiting matches to the top-level directory. I've tried various patterns, from ^index.android.js and ./index.android.js to /index.android.js. Are you aware of how to accomplish limiting the ignoring to only the top-level index.android.js?

My research into how babel looks this up seems like any JS file within node_modules will look for a .babelrc and will continue upward toward / in search of one. Should a catch-all for node_modules be placed at node_modules/.babelrc then, which doesn't have the ignore?

[HikingDev]

Why is this issue closed? I have a json file with 17.4MB.
If i require it the bundler crashes and if comment it out everything works fine.
This seems to be a common problem. Is there a solution for this issue?

"FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory"

[jeaye]

@roesneb It certainly shouldn't be closed. It was ignored when it was reported, multiple times, in the react native repo, while people suggested it be brought up here. Now that it's brought up here, it's promptly closed and no workarounds are available (for people using ClojureScript).

For your problem, you might try ignoring that JSON file using your babelrc. Alas, if it has a common name, then all JSONs with that name will be ignored, which is the last problem about which I asked @rafeca.

[HikingDev]

@jeaye i've tried the ignoring option already, without success. I could split up my JSON in multiple files but I don't think that should be the way to do it.

[VincentJousse]

Any news on this "not solved" issue ?

[jeaye]

@VincentFTS If this is possible:

Have you tried refining the regexp used in the ignore param to only match the huge index.android.js file?

Then the problem can be resolved for ClojureScript users. No documentation I've found has shown it to be possible though. Instead, renaming your app's index.android.js to myapp.index.android.js or something, may do the trick.