New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of memory or out of time when compiling large JS files #111

Closed
jeaye opened this Issue Dec 28, 2017 · 16 comments

Comments

Projects
None yet
6 participants
@jeaye

jeaye commented Dec 28, 2017

Do you want to request a feature or report a bug?
Bug.

What is the current behavior?
Metro will either run out of memory or it will give up after 5 minutes of waiting to transform the JS file.

This was first reported here facebook/react-native#8475 and then again here facebook/react-native#12590

If the current behavior is a bug, please provide the steps to reproduce and a minimal repository on GitHub that we can yarn install and yarn test.
I have a repro repo here https://github.com/jeaye/react-native-packager-bug -- it's not in the yarn format, since I don't know what that is.

What is the expected behavior?
The transformation completes in a timely fashion or metro waits long enough for the transformation to finish in however long it needs. In order to achieve the latter, my production build script currently contains the following:

metro=../node_modules/metro-bundler/src/JSTransformer/index.js
sed -i 's/\(TRANSFORM_TIMEOUT_INTERVAL\) = .*;/\1 = 901000;/g' "$metro" || true

Please provide your exact Metro configuration and mention your Metro, node, yarn/npm version and operating system.

React Native: 0.50.4
babel-preset-react-native: 4.0.0
Metro (comes with RN): 0.20.3
Node: 9.3.0
OS: Arch Linux x86_64

@jeaye

This comment has been minimized.

Show comment
Hide comment
@jeaye

jeaye Jan 18, 2018

Happy new year! Any update on this?

jeaye commented Jan 18, 2018

Happy new year! Any update on this?

@rafeca

This comment has been minimized.

Show comment
Hide comment
@rafeca

rafeca Jan 20, 2018

Member

Hey @jeaye ! Happy new year and apologies for the delay.

The timeout on transformations was indeed [removed some time ago]( (d4dcd4c), so from metro v0.21.0 there is no maximum time to transform a file.

Out of curiosity, can you provide more details about the file that you're trying to compile? How big is it? Why is it so big? Metro is heavily optimized to build bundles with a really big amount of small files, but cannot run that fast when trying to compile a small amount of huge files (it's impossible to parallelize, the caching system is not efficient, etc(.

Member

rafeca commented Jan 20, 2018

Hey @jeaye ! Happy new year and apologies for the delay.

The timeout on transformations was indeed [removed some time ago]( (d4dcd4c), so from metro v0.21.0 there is no maximum time to transform a file.

Out of curiosity, can you provide more details about the file that you're trying to compile? How big is it? Why is it so big? Metro is heavily optimized to build bundles with a really big amount of small files, but cannot run that fast when trying to compile a small amount of huge files (it's impossible to parallelize, the caching system is not efficient, etc(.

@rafeca rafeca closed this Jan 20, 2018

@jeaye

This comment has been minimized.

Show comment
Hide comment
@jeaye

jeaye Jan 20, 2018

@rafeca In production, my index.android.js is 1.2MB; these out-of-memory and timeout issues happen even before the 1MB mark though. Why is it that big? Well, it's a mid-sized application, about 10K LOC of ClojureScript, compiled with Google Closure.

As linked above, this issue is affect basically everyone using ClojureScript with RN, likely due to the CLJS stdlib and everything else included in the compiled file. Note, this is going through Google Closure, so it's not like there's a lot of waste here which we can trim down; all dead code is removed and all remaining code is optimized and minified.

Closing this issue because the timeout is removed is ok, but, really, waiting over 5 minutes to transform these files is pretty awful. Trying to do this with a non-production build (which doesn't have Google Closure optimizations applied) is a fool's errand, since it'll take more like 30 minutes, if it doesn't run out of memory before that.

jeaye commented Jan 20, 2018

@rafeca In production, my index.android.js is 1.2MB; these out-of-memory and timeout issues happen even before the 1MB mark though. Why is it that big? Well, it's a mid-sized application, about 10K LOC of ClojureScript, compiled with Google Closure.

As linked above, this issue is affect basically everyone using ClojureScript with RN, likely due to the CLJS stdlib and everything else included in the compiled file. Note, this is going through Google Closure, so it's not like there's a lot of waste here which we can trim down; all dead code is removed and all remaining code is optimized and minified.

Closing this issue because the timeout is removed is ok, but, really, waiting over 5 minutes to transform these files is pretty awful. Trying to do this with a non-production build (which doesn't have Google Closure optimizations applied) is a fool's errand, since it'll take more like 30 minutes, if it doesn't run out of memory before that.

@rafeca

This comment has been minimized.

Show comment
Hide comment
@rafeca

rafeca Jan 21, 2018

Member

@jeaye : thanks for the explanation! it makes complete sense 😄

In this case, since your index.android.js file has already been transpiled, I suggest you to try to disable babel transformation for that file and check how fast the transformation is (spoiler: it should be way faster). You can do that via the ignore param in your babel config.

Anyways, this flow that you describe is still going to be slow and painful in development mode (as you describe, every time you do a change you have to recompile the whole bundle from clojurescript to javascript, and then metro will see the modified file so it won't be able to leverage its cache and will do the whole bundling).

If possible, a much better flow would be to integrate the clojurescript compiler with metro, so metro would call the compiler on a per-file basis and you will benefit from the speed/parallelization/caching of metro.

I'm not familiar with clojurescript and its compiler, so I don't know if it can even work on a per-file basis, But if it can the integration should not be super-complex:

  1. You need to create you own transformer that transforms a clojurescript file to JS (more specifically, to AST). To get an example, this is our basic transformer
  2. You'll need to configure metro to use the new transformer that you've created. There is the getTransformModulePath option for that (example).

Let me know if this helps

Member

rafeca commented Jan 21, 2018

@jeaye : thanks for the explanation! it makes complete sense 😄

In this case, since your index.android.js file has already been transpiled, I suggest you to try to disable babel transformation for that file and check how fast the transformation is (spoiler: it should be way faster). You can do that via the ignore param in your babel config.

Anyways, this flow that you describe is still going to be slow and painful in development mode (as you describe, every time you do a change you have to recompile the whole bundle from clojurescript to javascript, and then metro will see the modified file so it won't be able to leverage its cache and will do the whole bundling).

If possible, a much better flow would be to integrate the clojurescript compiler with metro, so metro would call the compiler on a per-file basis and you will benefit from the speed/parallelization/caching of metro.

I'm not familiar with clojurescript and its compiler, so I don't know if it can even work on a per-file basis, But if it can the integration should not be super-complex:

  1. You need to create you own transformer that transforms a clojurescript file to JS (more specifically, to AST). To get an example, this is our basic transformer
  2. You'll need to configure metro to use the new transformer that you've created. There is the getTransformModulePath option for that (example).

Let me know if this helps

@jeaye

This comment has been minimized.

Show comment
Hide comment
@jeaye

jeaye Jan 21, 2018

@rafeca Thanks for the detailed reply. The ignoring approach was mentioned in the first referenced issue here, but it leads to other issues. When I try it in my production build, rather than my test case repository, I run into issues because my JS actually needs to be transformed. That is, I have some NPM dependencies using newer JS features which need to be transformed for RN to consume them.

The failure looks something like this:

...
:app:bundleReleaseJsAndAssets
Scanning folders for symlinks in my-app/node_modules (4ms)
Scanning folders for symlinks in my-app/node_modules (21ms)
Loading dependency graph, done.
warning: the transform cache was reset.

Export statement may only appear at top level in file "my-app/node_modules/react-native-material-ui/src/RippleFeedback/index.android.js" at line 1:114

:app:bundleReleaseJsAndAssets FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:bundleReleaseJsAndAssets'.
> Process 'command 'node'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

jeaye commented Jan 21, 2018

@rafeca Thanks for the detailed reply. The ignoring approach was mentioned in the first referenced issue here, but it leads to other issues. When I try it in my production build, rather than my test case repository, I run into issues because my JS actually needs to be transformed. That is, I have some NPM dependencies using newer JS features which need to be transformed for RN to consume them.

The failure looks something like this:

...
:app:bundleReleaseJsAndAssets
Scanning folders for symlinks in my-app/node_modules (4ms)
Scanning folders for symlinks in my-app/node_modules (21ms)
Loading dependency graph, done.
warning: the transform cache was reset.

Export statement may only appear at top level in file "my-app/node_modules/react-native-material-ui/src/RippleFeedback/index.android.js" at line 1:114

:app:bundleReleaseJsAndAssets FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:bundleReleaseJsAndAssets'.
> Process 'command 'node'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED
@rafeca

This comment has been minimized.

Show comment
Hide comment
@rafeca

rafeca Jan 21, 2018

Member

@jeaye it seems that there're other files in dependencies called index.android.js, so if you set the ignore param to this, it'll ignore all of them which is not correct.

Have you tried refining the regexp used in the ignore param to only match the huge index.android.js file?

Member

rafeca commented Jan 21, 2018

@jeaye it seems that there're other files in dependencies called index.android.js, so if you set the ignore param to this, it'll ignore all of them which is not correct.

Have you tried refining the regexp used in the ignore param to only match the huge index.android.js file?

@jeaye

This comment has been minimized.

Show comment
Hide comment
@jeaye

jeaye Jan 21, 2018

@rafeca That's a very good idea; the babel docs don't mention anything of the sort, as far as limiting matches to the top-level directory. I've tried various patterns, from ^index.android.js and ./index.android.js to /index.android.js. Are you aware of how to accomplish limiting the ignoring to only the top-level index.android.js?

My research into how babel looks this up seems like any JS file within node_modules will look for a .babelrc and will continue upward toward / in search of one. Should a catch-all for node_modules be placed at node_modules/.babelrc then, which doesn't have the ignore?

jeaye commented Jan 21, 2018

@rafeca That's a very good idea; the babel docs don't mention anything of the sort, as far as limiting matches to the top-level directory. I've tried various patterns, from ^index.android.js and ./index.android.js to /index.android.js. Are you aware of how to accomplish limiting the ignoring to only the top-level index.android.js?

My research into how babel looks this up seems like any JS file within node_modules will look for a .babelrc and will continue upward toward / in search of one. Should a catch-all for node_modules be placed at node_modules/.babelrc then, which doesn't have the ignore?

@roesneb

This comment has been minimized.

Show comment
Hide comment
@roesneb

roesneb Feb 10, 2018

Why is this issue closed? I have a json file with 17.4MB.
If i require it the bundler crashes and if comment it out everything works fine.
This seems to be a common problem. Is there a solution for this issue?

"FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory"

roesneb commented Feb 10, 2018

Why is this issue closed? I have a json file with 17.4MB.
If i require it the bundler crashes and if comment it out everything works fine.
This seems to be a common problem. Is there a solution for this issue?

"FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory"

@jeaye

This comment has been minimized.

Show comment
Hide comment
@jeaye

jeaye Feb 10, 2018

@roesneb It certainly shouldn't be closed. It was ignored when it was reported, multiple times, in the react native repo, while people suggested it be brought up here. Now that it's brought up here, it's promptly closed and no workarounds are available (for people using ClojureScript).

For your problem, you might try ignoring that JSON file using your babelrc. Alas, if it has a common name, then all JSONs with that name will be ignored, which is the last problem about which I asked @rafeca.

jeaye commented Feb 10, 2018

@roesneb It certainly shouldn't be closed. It was ignored when it was reported, multiple times, in the react native repo, while people suggested it be brought up here. Now that it's brought up here, it's promptly closed and no workarounds are available (for people using ClojureScript).

For your problem, you might try ignoring that JSON file using your babelrc. Alas, if it has a common name, then all JSONs with that name will be ignored, which is the last problem about which I asked @rafeca.

@roesneb

This comment has been minimized.

Show comment
Hide comment
@roesneb

roesneb Feb 10, 2018

@jeaye i've tried the ignoring option already, without success. I could split up my JSON in multiple files but I don't think that should be the way to do it.

roesneb commented Feb 10, 2018

@jeaye i've tried the ignoring option already, without success. I could split up my JSON in multiple files but I don't think that should be the way to do it.

@VincentFTS

This comment has been minimized.

Show comment
Hide comment
@VincentFTS

VincentFTS Feb 19, 2018

Any news on this "not solved" issue ?

VincentFTS commented Feb 19, 2018

Any news on this "not solved" issue ?

@jeaye

This comment has been minimized.

Show comment
Hide comment
@jeaye

jeaye Feb 19, 2018

@VincentFTS If this is possible:

Have you tried refining the regexp used in the ignore param to only match the huge index.android.js file?

Then the problem can be resolved for ClojureScript users. No documentation I've found has shown it to be possible though. Instead, renaming your app's index.android.js to myapp.index.android.js or something, may do the trick.

jeaye commented Feb 19, 2018

@VincentFTS If this is possible:

Have you tried refining the regexp used in the ignore param to only match the huge index.android.js file?

Then the problem can be resolved for ClojureScript users. No documentation I've found has shown it to be possible though. Instead, renaming your app's index.android.js to myapp.index.android.js or something, may do the trick.

@VincentFTS

This comment has been minimized.

Show comment
Hide comment
@VincentFTS

VincentFTS Feb 20, 2018

The problem is that the "big file" is a javascript file that uses flow, it permits to check the format of a big table.
Hence I cannot add it to the ignore section of the babelrc file.

VincentFTS commented Feb 20, 2018

The problem is that the "big file" is a javascript file that uses flow, it permits to check the format of a big table.
Hence I cannot add it to the ignore section of the babelrc file.

@ChadHarrisVerr

This comment has been minimized.

Show comment
Hide comment
@ChadHarrisVerr

ChadHarrisVerr Apr 20, 2018

Hello @jeaye

Did you end up solving this issue for CLJS? Our iOS build is functioning fine, but our android build not so well. On a previous version of react native, We did all sorts of monkey patching to make the transformers not run for our ClojureScript files it was horrible, but it worked. After upgrading to the latest RN we were hoping to avoid the money patching if possible.

ChadHarrisVerr commented Apr 20, 2018

Hello @jeaye

Did you end up solving this issue for CLJS? Our iOS build is functioning fine, but our android build not so well. On a previous version of react native, We did all sorts of monkey patching to make the transformers not run for our ClojureScript files it was horrible, but it worked. After upgrading to the latest RN we were hoping to avoid the money patching if possible.

@jeaye

This comment has been minimized.

Show comment
Hide comment
@jeaye

jeaye Apr 20, 2018

@ChadHarrisVerr Nope, this is still very much an issue. metro doesn't handle large files well and has a non-linear time growth with respect to file size. So far, there's been no official way to speed things up and this ticket, just like many others here and in the react-native repo, has been closed with little help (read the full discussion here to understand more, if you haven't).

I have documented my current workaround here: https://github.com/cljsrn/cljsrn-org/wiki/Avoiding-Metro-timeouts In short, I patch metro to not give up after several minutes of transforming and I just wait a long time for my prod builds. As a result of this, I also can't reasonably use something like jest for testing, since it wants a bundled package as well.

jeaye commented Apr 20, 2018

@ChadHarrisVerr Nope, this is still very much an issue. metro doesn't handle large files well and has a non-linear time growth with respect to file size. So far, there's been no official way to speed things up and this ticket, just like many others here and in the react-native repo, has been closed with little help (read the full discussion here to understand more, if you haven't).

I have documented my current workaround here: https://github.com/cljsrn/cljsrn-org/wiki/Avoiding-Metro-timeouts In short, I patch metro to not give up after several minutes of transforming and I just wait a long time for my prod builds. As a result of this, I also can't reasonably use something like jest for testing, since it wants a bundled package as well.

@sundbry

This comment has been minimized.

Show comment
Hide comment
@sundbry

sundbry Oct 16, 2018

In order to fix this, you'll need to use a custom Metro transformer to skip loading the entire compiled Clojurescript AST, and also load its dependencies properly. I've documented how to do it here:

https://www.arctype.co/blog/fixing-react-native-compiler-clojurescript-timeouts

cc @bonlemuel @ChadHarrisVerr

sundbry commented Oct 16, 2018

In order to fix this, you'll need to use a custom Metro transformer to skip loading the entire compiled Clojurescript AST, and also load its dependencies properly. I've documented how to do it here:

https://www.arctype.co/blog/fixing-react-native-compiler-clojurescript-timeouts

cc @bonlemuel @ChadHarrisVerr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment