Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process out of memory - Webpack #1914

Closed
andrewhathaway opened this issue Jan 22, 2016 · 124 comments
Closed

Process out of memory - Webpack #1914

andrewhathaway opened this issue Jan 22, 2016 · 124 comments

Comments

@andrewhathaway
Copy link

@andrewhathaway andrewhathaway commented Jan 22, 2016

After a couple of builds running in a Watch, Webpack crashes due to being out of memory.

<--- Last few GCs --->

 9223585 ms: Scavenge 1390.0 (1454.7) -> 1390.0 (1454.7) MB, 9.0 / 0 ms (+ 0.7 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
 9224727 ms: Mark-sweep 1390.0 (1454.7) -> 1388.3 (1452.7) MB, 1142.7 / 0 ms (+ 9.8 ms in 89 steps since start of marking, biggest step 2.5 ms) [last resort gc].
 9225694 ms: Mark-sweep 1388.3 (1452.7) -> 1388.8 (1454.7) MB, 966.6 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x35903c44a49 <JS Object>
    1: walkFunctionDeclaration [<PROJECT_DIR>/node_modules/webpack/lib/Parser.js:~443] [pc=0xa07a14ec8ee] (this=0x59f67991119 <a Parser with map 0x71f2d115d49>,statement=0x3507a80af661 <a Node with map 0x71f2d1157c9>)
    2: walkStatement [<PROJECT_DIR>/node_modules/webpack/lib/Parser.js:~348] [pc=0xa07a06dfc10] (this=0x59f6799111...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Abort trap: 6
@bebraw bebraw added the bug label Jan 22, 2016
@SpaceK33z
Copy link
Member

@SpaceK33z SpaceK33z commented Jan 22, 2016

I'm having this too with webpack-dev-server on Debian; but I haven't had time to debug this further.
Do you have a minimal webpack config where you can reproduce this? It could also be an external loader or plugin.

@sokra
Copy link
Member

@sokra sokra commented Jan 23, 2016

Don't use hashes when watching. i. e. [chunkhash] causes all assets to be stored.

@SimenB
Copy link
Contributor

@SimenB SimenB commented Jan 23, 2016

@sokra In the documentation for css-loader it says to use hash for easier debugging wrt css modules. Does that have the same problem?

You can configure the generated ident with the localIdentName query parameter (default [hash:base64]). Example: css-loader?localIdentName=[path][name]---[local]---[hash:base64:5] for easier debugging.

We've also started to see this problem in dev (out of memory), but our only hash in dev mode is the css one.

EDIT: Of course it uses hash by default, so I'm guessing it's not a problem.

@sokra
Copy link
Member

@sokra sokra commented Jan 23, 2016

Does that have the same problem?

No. Only filenames.

You can compare it with webpack --watch. If this generates many files in to your output path, the webpack-dev-server generates many files in the memory-fs.

If this is not the issue, you can increase the node.js memory (it defaults to 1.7 GB, which can be too few for big builds).

@SimenB
Copy link
Contributor

@SimenB SimenB commented Jan 23, 2016

I'll test at work on Monday! If the outputted file is of the same name as the old one, then it's overwritten, right?

@sokra
Copy link
Member

@sokra sokra commented Jan 23, 2016

jup.

You can navigate to http://localhost:8080/webpack-dev-server to see a list of all files in memory.

@SimenB
Copy link
Contributor

@SimenB SimenB commented Jan 23, 2016

Ah, didn't know that! Using a small project I have on my own computer, I notice that all of the hot-update files gets added.

image

Could they be a problem? Any way of clearing them out after they've been there for x amount of time? If we develop for a few hours there's probably quite a few of them in the end

@sokra
Copy link
Member

@sokra sokra commented Jan 23, 2016

Could they be a problem?

They could, but they are usually pretty small.

@SpaceK33z
Copy link
Member

@SpaceK33z SpaceK33z commented Jan 23, 2016

I think clearing them out like @SimenB said would be a nice option, or maybe there is some way to delete all in-memory files that are no longer in use?

@sokra
Copy link
Member

@sokra sokra commented Jan 23, 2016

jup. Write a plugin...

@sokra
Copy link
Member

@sokra sokra commented Jan 23, 2016

But deleting them too early can break stuff...

@deepakmani
Copy link

@deepakmani deepakmani commented Apr 26, 2016

We have the exact same memory leak problem after using webpack -w on MacOS X. Can you please suggest what is a the best way to fix / workaround this issue. The ngOfficeUIFabric seems to deal with a different problem.

Thanks,
Deepak

@luchillo17
Copy link

@luchillo17 luchillo17 commented Apr 26, 2016

Have same issue of @deepakmani in OSX El Capitan, however i'm using Ionic 2 which in it's config doesn't use css-loader, instead it just uses ionic-gulp-sass-build, so ikd if the error happens because of webpack or that gulp plugin, also i'm not using HRM:

const path = require('path')
const PrefetchPlugin = require('webpack/lib/PrefetchPlugin')
const ProvidePlugin = require('webpack/lib/ProvidePlugin')
const DedupePlugin = require('webpack/lib/optimize/DedupePlugin')
const UglifyJsPlugin = require('webpack/lib/optimize/UglifyJsPlugin')
const WebpackNotifierPlugin = require('webpack-notifier')


module.exports = {
  entry: {
    polyfills: './app/polyfills.ts',
    vendor: './app/vendor.ts',
    app: path.resolve('app/app')
  },
  output: {
    path: path.resolve('www/build/js'),
    filename: '[name].bundle.js',
    publicPath: "/",
    chunkFilename: "[id].bundle.js",
    pathinfo: false, // show module paths in the bundle, handy for debugging
  },
  devtool: 'eval',
  module: {
    loaders: [
      {
        test: /\.ts$/,
        loader: 'awesome-typescript',
        query: {
          doTypeCheck: true,
          resolveGlobs: false,
          externals: ['typings/main.d.ts']
        },
        include: path.resolve('app'),
        exclude: /node_modules/
      }
      ,{
        test: /\.js$/,
        include: path.resolve('node_modules/angular2'),
        loader: 'strip-sourcemap'
      }
    ],
    noParse: [
      /es6-shim/,
      /reflect-metadata/,
      /zone\.js(\/|\\)dist(\/|\\)zone/
    ]
  },
  plugins: [
    new PrefetchPlugin('rxjs'),
    new ProvidePlugin({
      Chart: 'chart.js/Chart.min.js'
    }),
    new DedupePlugin()
    // ,new UglifyJsPlugin({
    //    // to debug prod builds uncomment //debug lines and comment //prod lines
    //   // beautify: true,//debug
    //   // mangle: false,//debug
    //   // dead_code: false,//debug
    //   // unused: false,//debug
    //   // deadCode: false,//debug
    //   // compress : { screw_ie8 : true, keep_fnames: true, drop_debugger: false, dead_code: false, unused: false, }, // debug
    //   // comments: true,//debug

    //   beautify: false,//prod
    //   // disable mangling because of a bug in angular2 beta.1, beta.2 and beta.3
    //   // TODO(mastertinner): enable mangling as soon as angular2 beta.4 is out
    //   // mangle: { screw_ie8 : true },//prod
    //   mangle: false,
    //   // dead_code : true,
    //   compress : { 
    //     dead_code : true,
    //     screw_ie8 : true
    //   },//prod
    //   comments: false//prod
    // })
    ,new WebpackNotifierPlugin({
      title: 'tao_app_ionic',
      alwaysNotify: true
      // contentImage: path.join(appDir, 'images/notifier.png')
    })
  ],
  resolve: {
    alias: {
      'angular2': path.resolve('node_modules/angular2')
    },
    extensions: ['', '.js', '.ts']
  }
};
@guruprasanth5
Copy link

@guruprasanth5 guruprasanth5 commented May 26, 2016

I am having the same issue, any hack that can fix this for now.

joschi added a commit to Graylog2/graylog2-server that referenced this issue Jun 29, 2016
Before this webpack was using hashes in filenames for the generated
files which are stored in memory. That meant on every change new files
have been generated and stored in memory, quickly filling up the
available memory.

With this change we do not use hashes for filenames in development mode
to avoid filling up memory.

Refs webpack/webpack#1914
joschi pushed a commit to Graylog2/graylog2-server that referenced this issue Jun 29, 2016
Before this webpack was using hashes in filenames for the generated
files which are stored in memory. That meant on every change new files
have been generated and stored in memory, quickly filling up the
available memory.

With this change we do not use hashes for filenames in development mode
to avoid filling up memory.

Refs webpack/webpack#1914
(cherry picked from commit 3ab5918)
@thunderkid
Copy link

@thunderkid thunderkid commented Oct 4, 2016

Having the same problem running webpack via webpack-stream in gulp. I use gulp.watch to rerun on code changes, which then just runs

    return gulp.src(tsStartFile)
              .pipe(webpackStream(webPackConfig))
              .on('error', logError) 
              .pipe(gulp.dest(outputDir))

I'd have thought this would run webpack from scratch each time, but after 5 or so builds I get the out of memory error and have to rerun my gulp file.

@Pines-Cheng
Copy link

@Pines-Cheng Pines-Cheng commented Oct 17, 2016

Did someone resolved this problem?

ivanov added a commit to rgbkrk/nteract that referenced this issue Nov 10, 2016
@tdsmithATabc
Copy link

@tdsmithATabc tdsmithATabc commented Nov 15, 2016

Another data point here; I have filename hashes disabled and my build isn't terribly large (it uses Angular 2, but few other dependencies), but after about an hour of dev work the watch dies with Abort trap: 6.

I've temporarily increased the memory limit, which gives me a lot more time, but I'd rather not be consuming gigabytes just to run a watch command.

FYI if anyone is curious about upping the memory limit, I did it like so in my package.json (obviously the additional flags are dependent on your needs):

{
  "scripts": {
    "start": "node --max_old_space_size=4096 node_modules/webpack-dev-server/bin/webpack-dev-server.js --inline --progress --port 3000 --open"
  }
}
@mikiest
Copy link

@mikiest mikiest commented Nov 17, 2016

Having the same issue as @thunderkid here. Any update?

@rochdev
Copy link

@rochdev rochdev commented Jan 11, 2017

Same issue for a very small project

@aprilmintacpineda
Copy link

@aprilmintacpineda aprilmintacpineda commented Jan 4, 2019

Hmmm.. I don't experienced leak that's directly related to webpack or webpack-dev-server after using that configuration. But I did experienced a leak that's related to one of the webpack-loaders that I use that was misconfigured.

@sibelius
Copy link

@sibelius sibelius commented Jan 4, 2019

Which loader?

@idegree
Copy link

@idegree idegree commented Jan 16, 2019

export NODE_OPTIONS=--max_old_space_size=4096

Works for me.

@Delagen
Copy link

@Delagen Delagen commented Jan 16, 2019

@idegree I used this for a while, but this broke some programs in Windows. I took for about hour to detect that Geforce Experience does not start because of this environment variable.
I hope in Linux all ok )

@aljesusg aljesusg mentioned this issue Jan 23, 2019
7 of 7 tasks complete
fenok added a commit to c7s/react-boilerplate that referenced this issue Feb 1, 2019
@awacode21
Copy link

@awacode21 awacode21 commented Feb 27, 2019

Having the same problem running webpack via webpack-stream in gulp. I use gulp.watch to rerun on code changes, which then just runs

    return gulp.src(tsStartFile)
              .pipe(webpackStream(webPackConfig))
              .on('error', logError) 
              .pipe(gulp.dest(outputDir))

I'd have thought this would run webpack from scratch each time, but after 5 or so builds I get the out of memory error and have to rerun my gulp file.

Same with grunt

@ingava
Copy link

@ingava ingava commented Mar 7, 2019

Not sure if this is going to help anyone but we had a problem after updating to webpack 4 with the out of memory error. The build itself would often get stuck or become very slow. What very unexpectedly helped us was removing the BundleAnalyzerPlugin.

@kevinSuttle
Copy link

@kevinSuttle kevinSuttle commented Mar 11, 2019

I got this in production mode, but not development 🤷‍♂️

@vishalraj82
Copy link

@vishalraj82 vishalraj82 commented Apr 17, 2019

@kevinSuttle +1 I also encounter error in production mode, but not in development

@migueljoserivera
Copy link

@migueljoserivera migueljoserivera commented Jun 14, 2019

I had this problem with the version 10.. of node but, when I upgraded to the v12.4.0 the error gone.

@FuDesign2008
Copy link

@FuDesign2008 FuDesign2008 commented Oct 18, 2019

@buildbreakdo A better option is to do this:

"scripts": {
    "build": "cross-env NODE_OPTIONS=--max_old_space_size=8192 webpack"
}

(It uses the cross-env package.)

The nice thing about using NODE_OPTIONS is that it will affect everything, including any sub-processes that Webpack might spawn.

(It's useful for non-Webpack things as well: we're using it for TypeDoc, which consumes a lot of RAM.)

This solution did resolve the problem in my project.

@kamlesh-cis
Copy link

@kamlesh-cis kamlesh-cis commented Dec 2, 2019

@buildbreakdo A better option is to do this:

"scripts": {
    "build": "cross-env NODE_OPTIONS=--max_old_space_size=8192 webpack"
}

(It uses the cross-env package.)
The nice thing about using NODE_OPTIONS is that it will affect everything, including any sub-processes that Webpack might spawn.
(It's useful for non-Webpack things as well: we're using it for TypeDoc, which consumes a lot of RAM.)

This solution did resolve the problem in my project.

It solved my problem as well I just had to install "cross-env" pkg and add the node option to the webpack command!

Thanks a lot!

@junibrosas
Copy link

@junibrosas junibrosas commented Feb 10, 2020

upping the memory limit will only solve the problem temporarily but the RAM being used up the more you rebuild the application. You will notice your machine gets slower overtime.

@odykyi
Copy link

@odykyi odykyi commented Feb 10, 2020

remove max_old_space_size

just add to ts-loader

           loader: 'ts-loader',
           options: {
             transpileOnly: true // this works for me
           }
@NextTrick
Copy link

@NextTrick NextTrick commented Apr 18, 2020

Update Node versión from v8.16.0 to v12.6.0 and that issue is gone!

@sontd-0882
Copy link

@sontd-0882 sontd-0882 commented Jul 9, 2020

Update Node versión from v8.16.0 to v12.6.0 and that issue is gone!

Me too, thanks! @NextTrick

@alexander-akait
Copy link
Member

@alexander-akait alexander-akait commented Jul 9, 2020

Memory leak found suguru03/neo-async#83

mattcen added a commit to mattcen/riot-web that referenced this issue Jul 16, 2020
When trying to build the Docker image, I kept receiving an out-of-memory
error from Webpack. This issue is apparently explained at
webpack/webpack#1914.
Several comments in the issue suggested using Node 12 or above. Doing so
resolved this issue for me.
mattcen added a commit to mattcen/riot-web that referenced this issue Jul 16, 2020
When trying to build the Docker image, I kept receiving an out-of-memory
error from Webpack. This issue is apparently explained at
webpack/webpack#1914.
Several comments in the issue suggested using Node 12 or above. Doing so
resolved this issue for me.

Signed-off-by: Matt Cengia <mattcen@mattcen.com>
@satheler
Copy link

@satheler satheler commented Aug 11, 2020

remove max_old_space_size

just add to ts-loader

           loader: 'ts-loader',
           options: {
             transpileOnly: true // this works for me
           }

Works for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet