Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Since 2.3.0, webpack builds on CircleCI fail with "Error: Call retries were exceeded" #202

Closed
happypoulp opened this issue Dec 18, 2019 · 15 comments · Fixed by #216
Closed

Comments

@happypoulp
Copy link

happypoulp commented Dec 18, 2019

  • Operating System: ubuntu:18.04 (from CircleCI)
  • Node Version: v10.13.0
  • NPM Version: 6.10.0
  • webpack Version: 4.29.6
  • terser-webpack-plugin Version: >= 2.3.0

Expected Behavior

Building webpack JS bundles on CircleCI should be working the same for version 2.2.3 and versions >= 2.3.0.

Actual Behavior

Builds succeeds for terser-webpack-plugin v2.2.3 and are not working for terser-webpack-plugin >= 2.3.0 (tested with v2.3.0 and v2.3.1).

The error returned when running a webpack command such as: webpack --mode production -p --progress --config webpack.config.js with terser-webpack-plugin v2.3.0 produces this error:

ERROR in javascripts/bundle/feature.js from Terser
Error: Call retries were exceeded
    at ChildProcessWorker.initialize (/root/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/ChildProcessWorker.js:193:21)
    at ChildProcessWorker.onExit (/root/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/ChildProcessWorker.js:263:12)
    at ChildProcess.emit (events.js:182:13)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)

and this happens for multiple of the bundles that webpack tries to build.

Code

// Here is the config I am using for Terser plugin:
  new TerserPlugin({
    cache: true,
    parallel: true,
    sourceMap: true, // Using source-map is intentional
  }),

How Do We Reproduce?

The project I am building has more than 30 JS bundles and few of them are rather heavy (the heaviest is 1.6MB, average size is 600KB). Providing a reproducible example would be complicated to simulate the high number of JS files used and the overall heaviness of the bundles (maybe the issue is triggered in the source-map generation step).
But since the issue was introduced in a specific recent release (2.3.0), I hope that it will be easier to pinpoint the issue.

Please not that this issue is not reproducible on my local machine (a very recent MBP running Mac Mojave 10.14.6)

Thanks!

@happypoulp happypoulp changed the title Since 2.3.0 CircleCI build fails with "Error: Call retries were exceeded" Since 2.3.0, webpack builds on CircleCI fail with "Error: Call retries were exceeded" Dec 18, 2019
@happypoulp
Copy link
Author

Actually I think my analysis was incorrect. It seems the issue also reproduces with version 2.2.3 (it just doesn't happen when debugging a failing build on CircleCI with SSH, only when running a "real" CircleCI build). Will close this issue as it is probably the same as #143 .

@alexander-akait
Copy link
Member

Can you try https://github.com/webpack-contrib/terser-webpack-plugin/releases/tag/v2.3.3? If problem still exists i will reopen the problem, thanks

@hinok
Copy link
Contributor

hinok commented Jan 30, 2020

@evilebottnawi I upgraded in my project to v2.3.3 but still had the same problem. Setting explicitly parallel: 2 resolved the issue.

@alexander-akait
Copy link
Member

alexander-akait commented Jan 30, 2020

Can you provide minimum reproducible test repo and information about environment - real cores and memory, config file for CircleCI

@hinok
Copy link
Contributor

hinok commented Jan 30, 2020

I cannot publish the code 😞The app that I'm talking about is quite large (+/- 4k files, 600+ react components, I don't count any external dependencies here).

While building, we compile server and client bundle with npm-run-all.

Job's resource: 2CPU/4096MB

Zrzut ekranu 2020-01-30 o 21 17 42

circleci config.yml

version: 2

jobs:
  build_dependencies:
    docker:
      - image: circleci/node:10.16.2-stretch
    working_directory: ~/app-frontend
    steps:
      - checkout
      - run: sudo apt-get update && sudo apt-get install -y autoconf libtool pkg-config nasm
      - restore_cache:
          keys:
          - npm-dependencies-{{ checksum "package.json" }}
          # fallback to using the latest cache if no exact match is found
          - npm-dependencies-
      - run: yarn --silent --frozen-lockfile
      - save_cache:
          key: npm-dependencies-{{ checksum "package.json" }}
          paths:
            - node_modules
      - save_cache:
          key: repo-{{ .Environment.CIRCLE_SHA1 }}
          paths:
            - ~/app-frontend

  build:
    docker:
      - image: circleci/node:10.16.2-stretch
    working_directory: ~/app-frontend
    steps:
      - restore_cache:
          key: repo-{{ .Environment.CIRCLE_SHA1 }}
      - run: CONFIG=dev yarn build

workflows:
  version: 2

  build:
    jobs:
      - build_dependencies:
          filters:
            branches:
              ignore:
                - develop
                - master
            tags:
              ignore: /^v.*/
      - build:
          requires:
            - build_dependencies
          filters:
            branches:
              ignore:
                - develop
                - master
            tags:
              ignore: /^v.*/

package.json

{
  "build": "npm-run-all --parallel build:*",
  "build:client": "NODE_ENV=production webpack --config webpack/webpack.client.js --env.prod",
  "build:server": "NODE_ENV=production webpack --config webpack/webpack.server.js --env.prod"
}

webpack terser config

const terserPlugin = new TerserJSPlugin({
    terserOptions: {
      parse: {
        ecma: 8,
      },
      compress: {
        ecma: 5,
        warnings: false,
        comparisons: false,
        inline: 2,
        drop_console: true,
      },
      mangle: {
        safari10: true,
      },
      output: {
        ecma: 5,
        comments: false,
        ascii_only: true,
      },
    },
    parallel: isCircleCi() ? 2 : true,
    cache: true,
    sourceMap: true,
    extractComments: true,
  });

@alexander-akait
Copy link
Member

Thanks for information, what is result of os.cpus() and lscpu?

@hinok
Copy link
Contributor

hinok commented Jan 31, 2020

circleci@<hash>:~$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                36
On-line CPU(s) list:   0-35
Thread(s) per core:    2
Core(s) per socket:    18
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
Stepping:              4
CPU MHz:               3399.925
BogoMIPS:              6000.00
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              25344K
NUMA node0 CPU(s):     0-35
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
> os.cpus()
[
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3397,
        times: {
            user: 26940600,
            nice: 141900,
            sys: 6673300,
            idle: 84844700,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3400,
        times: {
            user: 27778400,
            nice: 153300,
            sys: 7061100,
            idle: 85539900,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3389,
        times: {
            user: 28683800,
            nice: 128700,
            sys: 7030100,
            idle: 85136100,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3440,
        times: {
            user: 27663600,
            nice: 121100,
            sys: 6981800,
            idle: 86086700,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3116,
        times: {
            user: 29731200,
            nice: 133300,
            sys: 7820900,
            idle: 83304900,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3085,
        times: {
            user: 30912800,
            nice: 134600,
            sys: 8423700,
            idle: 80364100,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3355,
        times: {
            user: 30386700,
            nice: 129000,
            sys: 7978900,
            idle: 82519400,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3400,
        times: {
            user: 30117100,
            nice: 140200,
            sys: 8219200,
            idle: 81339400,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3287,
        times: {
            user: 29964800,
            nice: 122800,
            sys: 7932900,
            idle: 82947300,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3204,
        times: {
            user: 30102100,
            nice: 134300,
            sys: 8368700,
            idle: 81286800,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3287,
        times: {
            user: 30583800,
            nice: 139200,
            sys: 8267000,
            idle: 82011400,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3051,
        times: {
            user: 31621100,
            nice: 118100,
            sys: 8382300,
            idle: 79922800,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3399,
        times: {
            user: 30451100,
            nice: 133400,
            sys: 7990200,
            idle: 82395800,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3150,
        times: {
            user: 30496800,
            nice: 152600,
            sys: 7863100,
            idle: 82457000,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3400,
        times: {
            user: 30808600,
            nice: 124900,
            sys: 8456000,
            idle: 81011800,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3361,
        times: {
            user: 30607600,
            nice: 137500,
            sys: 7916100,
            idle: 82274600,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3399,
        times: {
            user: 30067600,
            nice: 140000,
            sys: 8477200,
            idle: 81668900,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3399,
        times: {
            user: 31656800,
            nice: 118100,
            sys: 8421500,
            idle: 80762400,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3395,
        times: {
            user: 29729600,
            nice: 122000,
            sys: 9043100,
            idle: 80276600,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3399,
        times: {
            user: 27666100,
            nice: 125000,
            sys: 8233300,
            idle: 84423800,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3394,
        times: {
            user: 28174800,
            nice: 121000,
            sys: 8073200,
            idle: 84135900,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3400,
        times: {
            user: 27988900,
            nice: 125000,
            sys: 8748100,
            idle: 82338500,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3286,
        times: {
            user: 26778200,
            nice: 133100,
            sys: 7375700,
            idle: 86398500,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3282,
        times: {
            user: 26562400,
            nice: 124500,
            sys: 7311000,
            idle: 86705800,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3282,
        times: {
            user: 27483200,
            nice: 124300,
            sys: 7304700,
            idle: 85339500,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3370,
        times: {
            user: 25486000,
            nice: 123700,
            sys: 7560800,
            idle: 87532900,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3370,
        times: {
            user: 26600400,
            nice: 127400,
            sys: 7662600,
            idle: 85278300,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3395,
        times: {
            user: 26358500,
            nice: 131400,
            sys: 7278300,
            idle: 86987600,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3399,
        times: {
            user: 25715800,
            nice: 131500,
            sys: 7525700,
            idle: 86858100,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3375,
        times: {
            user: 25976400,
            nice: 142500,
            sys: 6956500,
            idle: 87683600,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3379,
        times: {
            user: 25610900,
            nice: 137300,
            sys: 7254800,
            idle: 87717200,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3247,
        times: {
            user: 25417900,
            nice: 136900,
            sys: 7160800,
            idle: 88048800,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3340,
        times: {
            user: 26041700,
            nice: 121100,
            sys: 6681500,
            idle: 87907000,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3400,
        times: {
            user: 25104700,
            nice: 138200,
            sys: 6924000,
            idle: 88608600,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3398,
        times: {
            user: 25153000,
            nice: 130900,
            sys: 6785800,
            idle: 88700200,
            irq: 0
        }
    },
    {
        model: 'Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz',
        speed: 3400,
        times: {
            user: 24982900,
            nice: 151500,
            sys: 6926600,
            idle: 88630400,
            irq: 0
        }
    }
]

@alexander-akait
Copy link
Member

alexander-akait commented Jan 31, 2020

Looks CircleCI doesn't provide real available count of CPUs, here the request for improvement https://ideas.circleci.com/ideas/CCI-I-264

@alexander-akait
Copy link
Member

Unfortunately, we have no way to get the real value of available 😞 Your solution seems to be the only one right at the moment

@hinok
Copy link
Contributor

hinok commented Jan 31, 2020

👍 It'd be good to document that issue in the docs as a short explanation for parallel option.

Basically the trick with parallel came from @cjlarose #143 (comment)

Thanks for confirming that it's the proper way to resolve that.

@alexander-akait
Copy link
Member

@hinok Yes, will be great to documented that, I will do in near future, but you can be a champion 😄

@jsnajdr
Copy link

jsnajdr commented Feb 21, 2020

This bug has been haunting us for some time in the https://github.com/automattic/wp-calypso project, so I did a deep dive trying to find the root cause.

And I figured out that the CircleCI container where the build runs is running out of memory, so the Linux kernel will select a "bad" process and kill it with SIGKILL (read here about how the most "bad" process is chosen).

The jest-worker library won't report why the worker process has exited and will merely try to restart it. It gives up after a few attempts (default maxRetries is 3) and the result is the Call retries were exceeded error.

There are other errors you can get that are caused by the same condition. For example, I've also seen this:

internal/child_process.js:394
    throw errnoException(err, 'spawn');
    ^

Error: spawn ENOMEM
    at ChildProcess.spawn (internal/child_process.js:394:11)
    at spawn (child_process.js:540:9)
    at Object.fork (child_process.js:108:10)
    at ChildProcessWorker.initialize (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:137:44)
    at ChildProcessWorker.onExit (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:263:12)
    at ChildProcess.emit (events.js:223:5)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12) {
  errno: 'ENOMEM',
  code: 'ENOMEM',
  syscall: 'spawn'
}

Here the attempt to restart the process fails: the kernel outright refuses to spawn a new process, because there' not enough memory for it.

The jest-worker library could be improved to report failures better. I don't think that the terser-webpack-plugin itself can do anything better.

@alexander-akait
Copy link
Member

@jsnajdr thanks for feedback, can you open an issue in jest repo?

@zeroarst
Copy link

This bug has been haunting us for some time in the https://github.com/automattic/wp-calypso project, so I did a deep dive trying to find the root cause.

And I figured out that the CircleCI container where the build runs is running out of memory, so the Linux kernel will select a "bad" process and kill it with SIGKILL (read here about how the most "bad" process is chosen).

The jest-worker library won't report why the worker process has exited and will merely try to restart it. It gives up after a few attempts (default maxRetries is 3) and the result is the Call retries were exceeded error.

There are other errors you can get that are caused by the same condition. For example, I've also seen this:

internal/child_process.js:394
    throw errnoException(err, 'spawn');
    ^

Error: spawn ENOMEM
    at ChildProcess.spawn (internal/child_process.js:394:11)
    at spawn (child_process.js:540:9)
    at Object.fork (child_process.js:108:10)
    at ChildProcessWorker.initialize (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:137:44)
    at ChildProcessWorker.onExit (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:263:12)
    at ChildProcess.emit (events.js:223:5)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12) {
  errno: 'ENOMEM',
  code: 'ENOMEM',
  syscall: 'spawn'
}

Here the attempt to restart the process fails: the kernel outright refuses to spawn a new process, because there' not enough memory for it.

The jest-worker library could be improved to report failures better. I don't think that the terser-webpack-plugin itself can do anything better.

We are having exactly the same issue and cannot figure out. Thanks for clarifying what is going on there...

@minnat65
Copy link

minnat65 commented Dec 8, 2022

This bug has been haunting us for some time in the https://github.com/automattic/wp-calypso project, so I did a deep dive trying to find the root cause.

And I figured out that the CircleCI container where the build runs is running out of memory, so the Linux kernel will select a "bad" process and kill it with SIGKILL (read here about how the most "bad" process is chosen).

The jest-worker library won't report why the worker process has exited and will merely try to restart it. It gives up after a few attempts (default maxRetries is 3) and the result is the Call retries were exceeded error.

There are other errors you can get that are caused by the same condition. For example, I've also seen this:

internal/child_process.js:394
    throw errnoException(err, 'spawn');
    ^

Error: spawn ENOMEM
    at ChildProcess.spawn (internal/child_process.js:394:11)
    at spawn (child_process.js:540:9)
    at Object.fork (child_process.js:108:10)
    at ChildProcessWorker.initialize (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:137:44)
    at ChildProcessWorker.onExit (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:263:12)
    at ChildProcess.emit (events.js:223:5)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12) {
  errno: 'ENOMEM',
  code: 'ENOMEM',
  syscall: 'spawn'
}

Here the attempt to restart the process fails: the kernel outright refuses to spawn a new process, because there' not enough memory for it.

The jest-worker library could be improved to report failures better. I don't think that the terser-webpack-plugin itself can do anything better.

I think that this is the best explanation for the problem. I am also getting this error in my current project.
I am trying hard but am not able to solve it. All the details of my problem are in this #12899 issue.
If someone has solved it then please let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants