Freeze of bundler since 1.5.1

[jeanfortheweb]

🤔 Expected Behavior

The bundler should finish on any system it runs on.

😯 Current Behavior

On specific systems, in our case a virtual Windows Server environment, the bundler stops and random files without any errors.

💁 Possible Solution

My investigation showed that it is some wierd scenario of memory issues, causing certain workers to become zombies. Therefore, these workers will never stop working and the bundler stops. It's possible to fix this issue by adjusting the "maxConcurrentCallsPerWorker" settings of the worker-farm. In our case, we reduced this value from 10 to 5 to get a stable result.

To become maximum flexible, I would suggest to make this setting available as env variable, just the like worker count, keeping the default value at 10.

🌍 Your Environment

Software	Version(s)
Parcel	^1.5.1
Node	^8
npm/Yarn	^1.3
Operating System	Virtual Windows Server

[davidnagli]

cc @brandon93s

[embiem]

issue has also been reported in #637

[catamphetamine]

My temporary workaround is to open ./node_modules/parcel-bundler/src/WorkerFarm.js file and add a maxConcurrentCallsPerWorker option to it:

class WorkerFarm extends Farm {
  constructor(options) {
    let opts = {
      maxConcurrentWorkers: getNumWorkers(),
      maxConcurrentCallsPerWorker: 5
    };

(that's assuming you installed Parceljs locally, not globally)

[fathyb]

It'd be cool if someone on Windows could try #1105 to see if it fixes it

[catamphetamine]

@fathyb I can place it to my node_modules, but how do I download it?
custom-workerfarm branch doesn't exist:
https://github.com/DeMoorJasper/parcel/tree/custom-workerfarm

[fathyb]

@catamphetamine Oh, sorry. Jasper has some super-powers to push directly on the main Parcel repo. You can try the PR using (if you are using yarn) :

• yarn add parcel-bundler/parcel#custom-workerfarm
• (cd node_modules/parcel-bundler && yarn)

[catamphetamine]

@fathyb Well, seems that it still hangs:

[catamphetamine]

And it's the latest code in the src folder:

c:\dev\react-website-basic-example - Copy (master -> origin) (react-website-basic-example@1.0.0)
λ cat node_modules\parcel-bundler\src\workerfarm\WorkerFarm.js
const {EventEmitter} = require('events');
const os = require('os');
const fork = require('./fork');
const errorUtils = require('./errorUtils');

const PARCEL_WORKER =
  parseInt(process.versions.node, 10) < 8
    ? require.resolve('../../lib/workerfarm/worker')
    : require.resolve('../../src/workerfarm/worker');

let shared = null;
class WorkerFarm extends EventEmitter {
  constructor(options, farmOptions = {}) {
    super();
    this.options = Object.assign(
      {
        maxConcurrentWorkers: WorkerFarm.getNumWorkers(),
        maxConcurrentCallsPerWorker: 10,
        forcedKillTime: 100,
        warmWorkers: true,
        useLocalWorker: true,
        workerPath: PARCEL_WORKER
      },
      farmOptions
    );

I guess someone else could try it on their Windows (I have Windows 10).

It still works with maxConcurrentCallsPerWorker: 5 though.

[fathyb]

I wanted to be sure because it re-implemented the WorkerFarm. If it doesn't fixes it for you chances are it'll not fix it for anybody. Thanks for testing though, I'll try to find other solutions (but feel free to find alternative solutions).

[DeMoorJasper]

If the cause really is the ipc bug inside node.js there isn't much we can do except perhaps timeout the responses and kill the worker if it takes too long.
I could implement this in the workerfarm but not sure if we can really put a number on how long it might take.
Any ideas?

However if maxConcurrentCalls works it might actually be a ram issue?

Would be nice if anyone would be able to debug it in depth.

My workerfarm PR isn't much different from the node-worker-farm when it comes to handling requests, mainly to keep it sort of the same as before and make sure it's sort of bugfree as node-worker-farm has been tested a lot. (It should in theory be more lightweight though)

[fathyb]

@DeMoorJasper The VSCode team seems to have fixed it, quoting bpasero on nodejs/node#7657

A workaround that seems to prevent this issue for us is to send a message in sequence always from the callback of the process.send message. On Windows at least this basically means that each message gets send after a process.nextTick.

The fix is in microsoft/vscode#13763. We could try implementing it in the new WorkerFarm only on Windows, still better than guessing maxConcurrentCallsPerWorker. If it's this it would totally explain why reducing maxConcurrentCallsPerWorker fixes the issue.

[DeMoorJasper]

@fathyb Intresting, I'll try implement something similar, although I think the calls are already sequential. I'll look into it.
If I understand this correctly it means maxConcurrentCallsPerWorker is always 1 in case of vscode

[DeMoorJasper]

Alright I've implemented a fix for this in the custom-workerfarm branch, it should work exactly the same as the workaround in vscode. (see 64664dd)
Executing every worker call in sequence

[fathyb]

@DeMoorJasper If I understand the WorkerFarm correctly (might be wrong) maxConcurrentCallsPerWorker waits for the method to return. If asset.generate() takes one second, and process.nextTick(callback) takes 1 millisecond to call callback :

• maxConcurrentCallsPerWorker=1 sends one message per second
• the VSCode fix sends 1000 messages per second

[catamphetamine]

@DeMoorJasper I tested again and it still freezes.

[DeMoorJasper]

@catamphetamine After a little chat with fathyb I removed it again as it impacted performance quite a bit and I misunderstood what vscode actually did to fix this.
Currently working on creating a solid fix for this. Already got a test case that consistently fails due to this bug.

[DeMoorJasper]

@catamphetamine I pushed the slightly better fix now, as this one also slightly improves queue handling on other operating systems.
Would really appreciate a test :)

[catamphetamine]

@DeMoorJasper Oh, this one seems to be working.

[supertopoz]

@DeMoorJasper and @catamphetamine where can I get this fix? I'm happy to test it on my windows 10 home.

[DeMoorJasper]

@supertopoz you can try it using PR #1105