DataStream.distribute causes "StreamWorker child timeout"

[peterviergutz]

First of all, thanks for this awesome library! I am very eager to test out hwich perfroamnce gains I can achieve using the distribute method, but somehow I cannot get it to run. Posting as a regular issue, because I am not sure if this is a bug or something wrong on my side.

I derived a small script from the testcase of the distribute method to test out any performance gains, but it keeps exiting on a

Error: StreamWorker child timeout!

const { DataStream } = require('scramjet');

const cpus = require("os").cpus().length * 2;

function* gen() {
    for (let z = 0; z < 1e3; z++)
        yield z;
}

DataStream.fromIterator(gen())
    .distribute(
        i => i % cpus,
        (stream) =>
            stream
            //.each((x) => console.log(process.pid, x))
                .filter(num => {
                    if (num < 2) return false;
                    if (num == 2) return true;
                    for(var i = 2; i < num/2; i++) {
                        if (num % i === 0) return false;
                    }
                    return true;
                })
    )
    .toArray()
    .then((arr) => {
        console.log("arr", arr)
    });

produces:

(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:18749) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 3)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 4)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 5)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 6)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 7)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 8)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 9)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 10)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 11)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 12)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 13)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 14)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 15)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 16)
(node:18749) UnhandledPromiseRejectionWarning: Error: StreamWorker child timeout!
    at Timeout.setTimeout [as _onTimeout] (/Users/peavey/webprojects/airtb/node-jobs/node_modules/scramjet/lib/stream-worker.js:66:79)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
(node:18749) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 17)

System:

• OS: MacOSX 10.14.6
• Node version: v10.16.0
• Scramjet Version: 4.27.0

Thanks for any pointer on how to solve this.

[MichalCz]

Hey, this will be the first under 1 minute reaction time. :D

Looking into - will try to reproduce and come back.

Good issue writing. 👍

[MichalCz]

Ok, strange - I do see an error, but actually a different one:

$ node test.js 
arr [ { error:
     { message:
        'The "id" argument must be of type string. Received type object',
       stack:
        'TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received type object\n    at validateString (internal/validators.js:125:11)\n    at Module.require (internal/modules/cjs/lo
ader.js:687:3)\n    at require (internal/modules/cjs/helpers.js:25:18)\n    at msg.transforms.map (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:24:35)\n    at Array.map (<anonymou
s>)\n    at stream.pipe.JSONParse.shift (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:17:55)\n    at process._tickCallback (internal/process/next_tick.js:68:7)' } },
  { error:
     { message:
        'The "id" argument must be of type string. Received type object',
       stack:
        'TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received type object\n    at validateString (internal/validators.js:125:11)\n    at Module.require (internal/modules/cjs/lo
ader.js:687:3)\n    at require (internal/modules/cjs/helpers.js:25:18)\n    at msg.transforms.map (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:24:35)\n    at Array.map (<anonymou
s>)\n    at stream.pipe.JSONParse.shift (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:17:55)\n    at process._tickCallback (internal/process/next_tick.js:68:7)' } },
  { error:
     { message:
        'The "id" argument must be of type string. Received type object',
       stack:
        'TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received type object\n    at validateString (internal/validators.js:125:11)\n    at Module.require (internal/modules/cjs/lo
ader.js:687:3)\n    at require (internal/modules/cjs/helpers.js:25:18)\n    at msg.transforms.map (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:24:35)\n    at Array.map (<anonymou
s>)\n    at stream.pipe.JSONParse.shift (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:17:55)\n    at process._tickCallback (internal/process/next_tick.js:68:7)' } },
  { error:
     { message:
        'The "id" argument must be of type string. Received type object',
       stack:
        'TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received type object\n    at validateString (internal/validators.js:125:11)\n    at Module.require (internal/modules/cjs/lo
ader.js:687:3)\n    at require (internal/modules/cjs/helpers.js:25:18)\n    at msg.transforms.map (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:24:35)\n    at Array.map (<anonymou
s>)\n    at stream.pipe.JSONParse.shift (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:17:55)\n    at process._tickCallback (internal/process/next_tick.js:68:7)' } },
  { error:
     { message:
        'The "id" argument must be of type string. Received type object',
       stack:
        'TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received type object\n    at validateString (internal/validators.js:125:11)\n    at Module.require (internal/modules/cjs/lo
ader.js:687:3)\n    at require (internal/modules/cjs/helpers.js:25:18)\n    at msg.transforms.map (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:24:35)\n    at Array.map (<anonymou
s>)\n    at stream.pipe.JSONParse.shift (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:17:55)\n    at process._tickCallback (internal/process/next_tick.js:68:7)' } },
  { error:
     { message:
        'The "id" argument must be of type string. Received type object',
       stack:
        'TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received type object\n    at validateString (internal/validators.js:125:11)\n    at Module.require (internal/modules/cjs/lo
ader.js:687:3)\n    at require (internal/modules/cjs/helpers.js:25:18)\n    at msg.transforms.map (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:24:35)\n    at Array.map (<anonymou
s>)\n    at stream.pipe.JSONParse.shift (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:17:55)\n    at process._tickCallback (internal/process/next_tick.js:68:7)' } },
  { error:
     { message:
        'The "id" argument must be of type string. Received type object',
       stack:
        'TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received type object\n    at validateString (internal/validators.js:125:11)\n    at Module.require (internal/modules/cjs/lo
ader.js:687:3)\n    at require (internal/modules/cjs/helpers.js:25:18)\n    at msg.transforms.map (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:24:35)\n    at Array.map (<anonymou
s>)\n    at stream.pipe.JSONParse.shift (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:17:55)\n    at process._tickCallback (internal/process/next_tick.js:68:7)' } },
  { error:
     { message:
        'The "id" argument must be of type string. Received type object',
       stack:
        'TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received type object\n    at validateString (internal/validators.js:125:11)\n    at Module.require (internal/modules/cjs/lo
ader.js:687:3)\n    at require (internal/modules/cjs/helpers.js:25:18)\n    at msg.transforms.map (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:24:35)\n    at Array.map (<anonymou
s>)\n    at stream.pipe.JSONParse.shift (/Users/michal/src/scramjet-test/node_modules/scramjet/lib/stream-child.js:17:55)\n    at process._tickCallback (internal/process/next_tick.js:68:7)' } } ]

Indeed Scramjet is not currently tested on Mac, since I have limited time to do that on my own infrastructure and I'm not aware of any free tools that would support macos tests...

[MichalCz]

Hi, I confirm this is an error on scramjet end. I will investigate further and let you know here as soon as I'll find out what is the reason...

This doesn't work with any kind of child worker anymore.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[seanavery]

Confirm I am also getting The "id" argument must be of type string. Received type object' error on macos.

Update getting the same error on linux.

Realized this is an issue with how I am using the lib.

The api expects string pointing to file or npm package that gets included in a require statement here https://github.com/signicode/scramjet/blob/master/lib/stream-child.js#L24

[MichalCz]

@seanavery would you be able to attempt to fix this? There must be something wrong with the StreamWorker class when it passes the code to be run there.

I sadly don't have the throughput to fix this as I'm working on a major rewrite for v5 and at the same time trying not to suck at my day job. ;)

If you could create a repo that would show the problem then we could try to solve it together and I'd happily accept a fix PR.

[MichalCz]

@peterviergutz @seanavery can you check a one thing for me and just refactor the code from:

someDataStream.distribute(stream => something(stream))

to:

someDataStream
    .separate(({x}) => "____" + x % 12)
    .cluster(stream => something(stream))

This may cause some own issues, but let me know if the result is any different from this.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.