worker: initial implementation #20876

Closed
wants to merge 26 commits into
from

Conversation

@addaleax
Member

addaleax commented May 22, 2018

Hi everyone! 👋

This PR adds threading support for to Node.js. I realize that this is not exactly a small PR and is going to take a while to review, so: I appreciate comments, questions (any kind, as long as it’s somewhat related 😺), partial reviews and all other feedback from anybody, not just Node.js core collaborators.

The super-high-level description of the implementation here is that Workers can share and transfer memory, but not JS objects (they have to be cloned for transferring), and not yet handles like network sockets.

FAQ

See https://gist.github.com/benjamingr/3d5e86e2fb8ae4abe2ab98ffe4758665

Example usage

const { Worker, isMainThread, postMessage, workerData } = require('worker_threads');

if (isMainThread) {
  module.exports = async function parseJSAsync(script) {
    return new Promise((resolve, reject) => {
      const worker = new Worker(__filename, {
        workerData: script
      });
      worker.on('message', resolve);
      worker.on('error', reject);
      worker.on('exit', (code) => {
        if (code !== 0)
          reject(new Error(`Worker stopped with exit code ${code}`));
      });
    });
  };
} else {
  const { parse } = require('some-js-parsing-library');
  const script = workerData;
  postMessage(parse(script));
}

Feature set

The communication between threads largely builds on the MessageChannel Web API. Transferring ArrayBuffers and sharing memory through SharedArrayBuffers is supported.
Almost the entire Node.js core API is require()able or importable.

Some notable differences:

  • stdio streams may be captured by the parent thread.
  • Some functions, e.g. process.chdir() don’t exist in worker threads.
  • Native addons are not loadable from worker threads (yet).
  • No inspector support (yet).

(Keep in mind that PRs can change significantly based on reviews.)

Comparison with child_process and cluster

Workers are conceptually very similar to child_process and cluster.
Some of the key differences are:

  • Communication between Workers is different: Unlike child_process IPC, we don’t use JSON, but rather do the same thing that postMessage() does in browsers.
    • This isn’t necessarily faster, although it can be and there might be more room for optimization. (Keep in mind how long JSON has been around and how much work has therefore been put into making it fast.)
    • The serialized data doesn’t actually need to leave the process, so overall there’s less overhead in communication involved.
    • Memory in the form of typed arrays can be transferred or shared between Workers and/or the main thread, which enables really fast communication for specific use cases.
    • Handles, like network sockets, can not be transferred or shared (yet).
  • There are some limitations on the usable API within workers, since parts of it (e.g. process.chdir()) affect per-process state, loading native addons, etc.
  • Each workers have its own event loop, but some of the resources are shared between workers (e.g. the libuv thread pool for file system work)

Benchmarks

    $ ./node benchmark/cluster/echo.js
    cluster/echo.js n=100000 sendsPerBroadcast=1 payload="string" workers=1: 33,647.30473442063
    cluster/echo.js n=100000 sendsPerBroadcast=10 payload="string" workers=1: 12,927.907405288383
    cluster/echo.js n=100000 sendsPerBroadcast=1 payload="object" workers=1: 28,496.37373941151
    cluster/echo.js n=100000 sendsPerBroadcast=10 payload="object" workers=1: 8,975.53747186485
    $ ./node --experimental-worker benchmark/worker/echo.js
    worker/echo.js n=100000 sendsPerBroadcast=1 payload="string" workers=1: 88,044.32902365089
    worker/echo.js n=100000 sendsPerBroadcast=10 payload="string" workers=1: 39,873.33697018837
    worker/echo.js n=100000 sendsPerBroadcast=1 payload="object" workers=1: 64,451.29132425621
    worker/echo.js n=100000 sendsPerBroadcast=10 payload="object" workers=1: 22,325.635443739284

A caveat here is that startup performance for Workers using this model is still relatively slow (I don’t have exact numbers, but there’s definitely overhead).

Regarding semverness:

The only breaking change here is the introduction of a new top-level module. The name is currently worker, this is not under a scope as suggested in nodejs/TSC#389. It seems like the most natural name for this by far.

I’ve reached out to the owner of the worker module on npm, who declined to provide the name for this purpose – the package has 57 downloads/week, so whether we consider this semver-major because of that is probably a judgement call.

Alternatively, I’d suggest using workers – it’s not quite what we’re used to in core (e.g. child_process), but the corresponding npm package is essentially just a placeholder.

Acknowledgements

People I’d like to thank for their code, comments and reviews for this work in its original form, in no particular order:

… and finally @petkaantonov for a lot of inspiration and the ability to compare with previous work on this topic.

Individual commits

src: cleanup per-isolate state on platform on isolate unregister

Clean up once all references to an Isolate* are gone from the
NodePlatform, rather than waiting for the PerIsolatePlatformData
struct to be deleted since there may be cyclic references between
that struct and the individual tasks.

src: fix MallocedBuffer move assignment operator

src: break out of timers loop if !can_call_into_js()

Otherwise, this turns into an infinite loop.

src: simplify handle closing

Remove one extra closing state and use a smart pointer for
deleting HandleWraps.

worker: implement MessagePort and MessageChannel

Implement MessagePort and MessageChannel along the lines of
the DOM classes of the same names. MessagePorts initially
support transferring only ArrayBuffers.

worker: support MessagePort passing in messages

Support passing MessagePort instances through other MessagePorts,
as expected by the MessagePort spec.

worker: add SharedArrayBuffer sharing

Logic is added to the MessagePort mechanism that
attaches hidden objects to those instances when they are transferred
that track their lifetime and maintain a reference count, to make
sure that memory is freed at the appropriate times.

src: add Env::profiler_idle_notifier_started()

src: move DeleteFnPtr into util.h

This is more generally useful than just in a crypto context.

worker: initial implementation

Implement multi-threading support for most of the API.

test: add test against unsupported worker features

worker: restrict supported extensions

Only allow .js and .mjs extensions to provide future-proofing
for file type detection.

src: enable stdio for workers

Provide stdin, stdout and stderr options for the Worker
constructor, and make these available to the worker thread
under their usual names.

The default for stdin is an empty stream, the default for
stdout and stderr is redirecting to the parent thread’s
corresponding stdio streams.

benchmark: port cluster/echo to worker

worker: improve error (de)serialization

Rather than passing errors using some sort of string representation,
do a best effort for faithful serialization/deserialization of
uncaught exception objects.

test,tools: enable running tests under workers

Enable running tests inside workers by passing --worker
to tools/test.py. A number of tests are marked as skipped,
or have been slightly altered to fit the different environment.

Other work

I know that teams from Microsoft (/cc @fs-eire @helloshuangzi) and Alibaba (/cc @aaronleeatali) have been working on forms of multithreading that have higher degrees of interaction between threads, such as sharing code and JS objects. I’d love if you could take a look at this PR and see how well it aligns with your own work, and what conflicts there might be. (From what I’ve seen of the other code, I’m actually quite optimistic that this PR is just going to help everybody.)

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
@addaleax

This comment was marked as outdated.

Show comment
Hide comment
@addaleax

addaleax May 22, 2018

Member

CI: https://ci.nodejs.org/job/node-test-commit/18659/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1422/

Edit: Will go to sleep now and look at comments tomorrow. Cheers!

Member

addaleax commented May 22, 2018

CI: https://ci.nodejs.org/job/node-test-commit/18659/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1422/

Edit: Will go to sleep now and look at comments tomorrow. Cheers!

@devsnek

This comment has been minimized.

Show comment
Hide comment
@devsnek

devsnek May 22, 2018

Member

i'm so heckin' happy to see this 🎉 🎉 🎉 🎉

regarding the whole name thing, why not use @nodejs/workers? we're pratically being shoved into it due to the owner of workers not wanting to give that name up, and we've been looking for excuses to namespace builtin modules anyway.

Member

devsnek commented May 22, 2018

i'm so heckin' happy to see this 🎉 🎉 🎉 🎉

regarding the whole name thing, why not use @nodejs/workers? we're pratically being shoved into it due to the owner of workers not wanting to give that name up, and we've been looking for excuses to namespace builtin modules anyway.

doc/api/process.md
@@ -1204,6 +1213,8 @@ console.log(process.getgroups()); // [ 27, 30, 46, 1000 ]
This function is only available on POSIX platforms (i.e. not Windows or
Android).
+This feature is not available in [`Worker`][] threads.

This comment has been minimized.

@devsnek

devsnek May 22, 2018

Member

maybe we should set up some sort of doc macro for this so that we can keep them all in sync if we decide to change the formatting of it.

@devsnek

devsnek May 22, 2018

Member

maybe we should set up some sort of doc macro for this so that we can keep them all in sync if we decide to change the formatting of it.

This comment has been minimized.

@addaleax

addaleax May 22, 2018

Member

Do we know how to do that?

@addaleax

addaleax May 22, 2018

Member

Do we know how to do that?

This comment was marked as resolved.

@cedric05

cedric05 May 23, 2018

does node runs on android?
is it experimental?
where can i refer

@cedric05

cedric05 May 23, 2018

does node runs on android?
is it experimental?
where can i refer

This comment was marked as resolved.

@vsemozhetbyt

vsemozhetbyt May 23, 2018

Member

@cedric05 see BUILDING.md#androidandroid-based-devices-eg-firefox-os for building on Android and package-manager/#android for installing precompiled app.

@vsemozhetbyt

vsemozhetbyt May 23, 2018

Member

@cedric05 see BUILDING.md#androidandroid-based-devices-eg-firefox-os for building on Android and package-manager/#android for installing precompiled app.

lib/worker.js
+
+if (!process.binding('config').experimentalWorker) {
+ // TODO(addaleax): Is this the right way to do this?
+ // eslint-disable-next-line no-restricted-syntax

This comment has been minimized.

@devsnek

devsnek May 22, 2018

Member

i would try and gate it in the nativemodule loader instead

@devsnek

devsnek May 22, 2018

Member

i would try and gate it in the nativemodule loader instead

@@ -1375,6 +1375,8 @@ def BuildOptions():
help="Expect test cases to fail", default=False, action="store_true")
result.add_option("--valgrind", help="Run tests through valgrind",
default=False, action="store_true")
+ result.add_option("--worker", help="Run parallel tests inside a worker context",

This comment was marked as resolved.

@devsnek

devsnek May 22, 2018

Member

❤️

@devsnek

devsnek May 22, 2018

Member

❤️

doc/api/errors.md
+<a id="ERR_CLOSED_MESSAGE_PORT"></a>
+### ERR_CLOSED_MESSAGE_PORT
+
+Used when there was an attempt to use a `MessagePort` instance in a closed

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Nit: Remove Used when.

@Trott

Trott May 22, 2018

Member

Nit: Remove Used when.

doc/api/errors.md
+<a id="ERR_MISSING_PLATFORM_FOR_WORKER"></a>
+### ERR_MISSING_PLATFORM_FOR_WORKER
+
+The V8 platform used by this instance of Node does not support creating Workers.

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Nit: Node -> Node.js

@Trott

Trott May 22, 2018

Member

Nit: Node -> Node.js

doc/api/errors.md
+<a id="ERR_WORKER_DOMAIN"></a>
+### ERR_WORKER_DOMAIN
+
+Used when trying to access the `domain` module inside of a worker thread.

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Remove Used when and reword:

The `domain` module cannot be used inside of a worker thread.
@Trott

Trott May 22, 2018

Member

Remove Used when and reword:

The `domain` module cannot be used inside of a worker thread.

This comment has been minimized.

@addaleax

addaleax May 22, 2018

Member

This was one of two outdated error entries anyway. :) It might be cool if our linter could check that entries in this markdown file are also present in lib/internal/errors.h or src/node_errors.h

@addaleax

addaleax May 22, 2018

Member

This was one of two outdated error entries anyway. :) It might be cool if our linter could check that entries in this markdown file are also present in lib/internal/errors.h or src/node_errors.h

doc/api/errors.md
+<a id="ERR_WORKER_NEED_ABSOLUTE_PATH"></a>
+### ERR_WORKER_NEED_ABSOLUTE_PATH
+
+Used when the path for the main script of a worker is not an absolute path.

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Remove Used when

@Trott

Trott May 22, 2018

Member

Remove Used when

doc/api/errors.md
+<a id="ERR_WORKER_OUT_OF_MEMORY"></a>
+### ERR_WORKER_OUT_OF_MEMORY
+
+Used when a worker hits its memory limit.

This comment has been minimized.

@Trott

Trott May 22, 2018

Member
A worker has exceeded its memory limit.
@Trott

Trott May 22, 2018

Member
A worker has exceeded its memory limit.
doc/api/errors.md
+Used when a worker hits its memory limit.
+
+<a id="ERR_WORKER_UNAVAILABLE_FEATURE"></a>
+### ERR_WORKER_UNAVAILABLE_FEATURE

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Missing description?

@Trott

Trott May 22, 2018

Member

Missing description?

doc/api/errors.md
+<a id="ERR_WORKER_UNSERIALIZABLE_ERROR"></a>
+### ERR_WORKER_UNSERIALIZABLE_ERROR
+
+Used when all attempts at serializing an uncaught exception from a worker fail.

This comment has been minimized.

@Trott

Trott May 22, 2018

Member
All attempts to serialize an uncaught exception from a worker failed.

...or perhaps...

All attempts to serialize an uncaught worker exception failed.
@Trott

Trott May 22, 2018

Member
All attempts to serialize an uncaught exception from a worker failed.

...or perhaps...

All attempts to serialize an uncaught worker exception failed.
doc/api/errors.md
+<a id="ERR_WORKER_UNSUPPORTED_EXTENSION"></a>
+### ERR_WORKER_UNSUPPORTED_EXTENSION
+
+Used when the pathname used for the main script of a worker has an

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Remove Used when

@Trott

Trott May 22, 2018

Member

Remove Used when

doc/api/process.md
@@ -918,6 +922,8 @@ console.log(process.env.test);
// => 1
```
+*Note*: `process.env` is read-only in [`Worker`][] threads.

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

My opinion only but I prefer we leave out *Note*: in nearly all cases.

@Trott

Trott May 22, 2018

Member

My opinion only but I prefer we leave out *Note*: in nearly all cases.

This comment has been minimized.

@ChALkeR

ChALkeR May 22, 2018

Member

I believe *Note*: prefixes were eradicated earlier from the docs.

@ChALkeR

ChALkeR May 22, 2018

Member

I believe *Note*: prefixes were eradicated earlier from the docs.

doc/api/process.md
@@ -1030,6 +1036,9 @@ If it is necessary to terminate the Node.js process due to an error condition,
throwing an *uncaught* error and allowing the process to terminate accordingly
is safer than calling `process.exit()`.
+*Note*: in [`Worker`][] threads, this function stops the current thread rather

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

My opinion only but I prefer we leave out *Note*: in nearly all cases.

@Trott

Trott May 22, 2018

Member

My opinion only but I prefer we leave out *Note*: in nearly all cases.

doc/api/worker.md
+added: REPLACEME
+-->
+
+Opposite of `unref`, calling `ref` on a previously `unref`d worker will *not*

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Nit: Add parentheses, so unfref() and ref(). Also elsewhere in this doc.

Nit: Change comma to period on this line.

Nit: We use `unref`ed rather than `unref'd` in existing documentation so use that here and in line 97, 105, and anywhere else for consistency.

@Trott

Trott May 22, 2018

Member

Nit: Add parentheses, so unfref() and ref(). Also elsewhere in this doc.

Nit: Change comma to period on this line.

Nit: We use `unref`ed rather than `unref'd` in existing documentation so use that here and in line 97, 105, and anywhere else for consistency.

doc/api/worker.md
+
+Opposite of `unref`, calling `ref` on a previously `unref`d worker will *not*
+let the program exit if it's the only active handle left (the default behavior).
+If the worker is `ref`d calling `ref` again will have no effect.

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Add comma before "calling".

@Trott

Trott May 22, 2018

Member

Add comma before "calling".

doc/api/worker.md
+* Returns: {undefined}
+
+Disables further sending of messages on either side of the connection.
+This this method can be called once you know that no further communication

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Typo: "This this"

@Trott

Trott May 22, 2018

Member

Typo: "This this"

doc/api/worker.md
+behind this API, see the [serialization API of the `v8` module][v8.serdes].
+
+*Note*: Because the object cloning uses the structured clone algorithm,
+non-enumberable properties, property accessors, and object prototypes are

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Typo: non-enumberable -> non-enumerable

@Trott

Trott May 22, 2018

Member

Typo: non-enumberable -> non-enumerable

doc/api/worker.md
+- [`process.chdir()`][] as well as `process` methods that set group or user ids
+ are not available.
+- [`process.env`][] is a read-only reference to the environment variables.
+- [`process.title`][] can not be modified.

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

can not -> cannot

@Trott

Trott May 22, 2018

Member

can not -> cannot

doc/api/worker.md
+child thread.
+
+To create custom messaging channels (which is encouraged over using the default
+global channel because it facilitates seperation of concerns), users can create

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Typo: seperation -> separation

@Trott

Trott May 22, 2018

Member

Typo: seperation -> separation

doc/api/worker.md
+ * data {any} Any JavaScript value that will be cloned and made
+ available as [`require('worker').workerData`][]. The cloning will occur as
+ described in the [HTML structured clone algorithm][], and an error will be
+ thrown if the object can not be cloned (e.g. because it contains

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

can not -> cannot

@Trott

Trott May 22, 2018

Member

can not -> cannot

doc/api/worker.md
+-->
+
+The `'error'` event is emitted if the worker thread throws an uncaught
+expection. In that case, the worker will be terminated.

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Typo: expection -> exception

@Trott

Trott May 22, 2018

Member

Typo: expection -> exception

doc/api/worker.md
+If the port is `ref`d calling `ref` again will have no effect.
+
+If listeners are attached or removed using `.on('message')`, the port will
+be `ref()`ed and `unref()`ed automatically depending on whether

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Actually, I prefer this style most of all (as opposed to ref'd or ref'ed). Would be great to put this in the net docs and everywhere here.

@Trott

Trott May 22, 2018

Member

Actually, I prefer this style most of all (as opposed to ref'd or ref'ed). Would be great to put this in the net docs and everywhere here.

doc/api/worker.md
+ may be redirected by the parent thread.
+- The [`require('worker').isMainThread`][] property is set to `false`.
+- The [`require('worker').postMessage()`][] method is available and the
+ [`require('worker').on('workerMessage')`][] event is being emitted.

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

is being -> is?

@Trott

Trott May 22, 2018

Member

is being -> is?

doc/api/worker.md
+- Signals will not be delivered through [`process.on('...')`][Signals events].
+- Native addons are not supported.
+- Execution may stop at any point as a result of the [`worker.terminate()`][]
+ method being invoked.

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

Nit/suggestion: Remove the on previous line and method on this line? (It's obviously a method due to the ().)

@Trott

Trott May 22, 2018

Member

Nit/suggestion: Remove the on previous line and method on this line? (It's obviously a method due to the ().)

doc/api/worker.md
+ [`require('worker').on('workerMessage')`][] event is being emitted.
+- [`process.exit()`][] does not stop the whole program, just the single thread,
+ and [`process.abort()`][] is not available.
+- [`process.chdir()`][] as well as `process` methods that set group or user ids

This comment has been minimized.

@Trott

Trott May 22, 2018

Member

as well as -> and

@Trott

Trott May 22, 2018

Member

as well as -> and

@Trott

This comment has been minimized.

Show comment
Hide comment
@Trott

Trott May 22, 2018

Member

@addaleax If you would prefer me to apply the doc nits I suggested myself directly to your branch rather than review and apply them yourself, let me know.

Member

Trott commented May 22, 2018

@addaleax If you would prefer me to apply the doc nits I suggested myself directly to your branch rather than review and apply them yourself, let me know.

doc/api/worker.md
+can be accessed using:
+
+```js
+const worker = require('worker');

This comment has been minimized.

@ChALkeR

ChALkeR May 22, 2018

Member

If this is exposed without a flag, this is a semver-major change as I presume.
/cc @jasnell

$ npm info worker time.created time.modified
time.created = '2017-11-14T00:22:43.069Z'
time.modified = '2018-04-28T07:31:55.508Z'
@ChALkeR

ChALkeR May 22, 2018

Member

If this is exposed without a flag, this is a semver-major change as I presume.
/cc @jasnell

$ npm info worker time.created time.modified
time.created = '2017-11-14T00:22:43.069Z'
time.modified = '2018-04-28T07:31:55.508Z'

This comment has been minimized.

@addaleax

addaleax May 22, 2018

Member

As @benjamingr said below, this module is essentially not in public use, and the creation timestamp is, as I understand it, a bit misleading because the package name was only recently taken over by another author.

@addaleax

addaleax May 22, 2018

Member

As @benjamingr said below, this module is essentially not in public use, and the creation timestamp is, as I understand it, a bit misleading because the package name was only recently taken over by another author.

This comment has been minimized.

@jasnell

jasnell May 22, 2018

Member

That said, yes, the addition of a new top level module automatically makes this semver-major without TSC agreeing otherwise.

@jasnell

jasnell May 22, 2018

Member

That said, yes, the addition of a new top level module automatically makes this semver-major without TSC agreeing otherwise.

@benjamingr

This comment has been minimized.

Show comment
Hide comment
@benjamingr

benjamingr May 22, 2018

Member

@addaleax , I’ve been seeing the activity in your GH feed and waiting ❤️

Awesome work! You really stepped up (again) and tackled a complex topic that has traditionally very hard. Before this PR gets very technical I just want to say thank you.

Also, that if this ever gets “too much” like it did for @petkaantonov - please call it out and speak up.

Member

benjamingr commented May 22, 2018

@addaleax , I’ve been seeing the activity in your GH feed and waiting ❤️

Awesome work! You really stepped up (again) and tackled a complex topic that has traditionally very hard. Before this PR gets very technical I just want to say thank you.

Also, that if this ever gets “too much” like it did for @petkaantonov - please call it out and speak up.

@benjamingr

This comment has been minimized.

Show comment
Hide comment
@benjamingr

benjamingr May 22, 2018

Member

I’ve reached out to the owner of the worker module on npm, who declined to provide the name for this purpose – the package has 57 downloads/week, so whether we consider this semver-major because of that is probably a judgement call.

I also have no issue with just taking the library name from them if they are not willing to concede it. Especially since they are not the original owner of that name and no one is using it yet.

Member

benjamingr commented May 22, 2018

I’ve reached out to the owner of the worker module on npm, who declined to provide the name for this purpose – the package has 57 downloads/week, so whether we consider this semver-major because of that is probably a judgement call.

I also have no issue with just taking the library name from them if they are not willing to concede it. Especially since they are not the original owner of that name and no one is using it yet.

@addaleax

This comment was marked as resolved.

Show comment
Hide comment
@addaleax

addaleax May 22, 2018

Member

@Trott @devsnek I think I’ve addressed most of your comments in 2c8dceb and 3bc8bbb :)

Member

addaleax commented May 22, 2018

@Trott @devsnek I think I’ve addressed most of your comments in 2c8dceb and 3bc8bbb :)

@addaleax addaleax referenced this pull request May 22, 2018

Closed

src: fix MallocedBuffer move assignment operator #20883

2 of 2 tasks complete

shisama added a commit to shisama/node that referenced this pull request Jun 17, 2018

worker: enable stdio
Provide `stdin`, `stdout` and `stderr` options for the `Worker`
constructor, and make these available to the worker thread
under their usual names.

The default for `stdin` is an empty stream, the default for
`stdout` and `stderr` is redirecting to the parent thread’s
corresponding stdio streams.

PR-URL: nodejs#20876
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Shingo Inoue <leko.noor@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: John-David Dalton <john.david.dalton@gmail.com>
Reviewed-By: Gus Caplan <me@gus.host>

shisama added a commit to shisama/node that referenced this pull request Jun 17, 2018

benchmark: port cluster/echo to worker
    $ ./node benchmark/cluster/echo.js
    cluster/echo.js n=100000 sendsPerBroadcast=1 payload="string" workers=1: 33,647.30473442063
    cluster/echo.js n=100000 sendsPerBroadcast=10 payload="string" workers=1: 12,927.907405288383
    cluster/echo.js n=100000 sendsPerBroadcast=1 payload="object" workers=1: 28,496.37373941151
    cluster/echo.js n=100000 sendsPerBroadcast=10 payload="object" workers=1: 8,975.53747186485
    $ ./node --experimental-worker benchmark/worker/echo.js
    worker/echo.js n=100000 sendsPerBroadcast=1 payload="string" workers=1: 88,044.32902365089
    worker/echo.js n=100000 sendsPerBroadcast=10 payload="string" workers=1: 39,873.33697018837
    worker/echo.js n=100000 sendsPerBroadcast=1 payload="object" workers=1: 64,451.29132425621
    worker/echo.js n=100000 sendsPerBroadcast=10 payload="object" workers=1: 22,325.635443739284

Refs: ayojs/ayo#115
Reviewed-By: Anna Henningsen <anna@addaleax.net>

PR-URL: nodejs#20876
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Shingo Inoue <leko.noor@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: John-David Dalton <john.david.dalton@gmail.com>
Reviewed-By: Gus Caplan <me@gus.host>

shisama added a commit to shisama/node that referenced this pull request Jun 17, 2018

worker: improve error (de)serialization
Rather than passing errors using some sort of string representation,
do a best effort for faithful serialization/deserialization of
uncaught exception objects.

PR-URL: nodejs#20876
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Shingo Inoue <leko.noor@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: John-David Dalton <john.david.dalton@gmail.com>
Reviewed-By: Gus Caplan <me@gus.host>

shisama added a commit to shisama/node that referenced this pull request Jun 17, 2018

test,tools: enable running tests under workers
Enable running tests inside workers by passing `--worker`
to `tools/test.py`. A number of tests are marked as skipped,
or have been slightly altered to fit the different environment.

PR-URL: nodejs#20876
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Shingo Inoue <leko.noor@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: John-David Dalton <john.david.dalton@gmail.com>
Reviewed-By: Gus Caplan <me@gus.host>

shisama added a commit to shisama/node that referenced this pull request Jun 17, 2018

doc: explain Worker semantics in async_hooks.md
PR-URL: nodejs#20876
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Shingo Inoue <leko.noor@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: John-David Dalton <john.david.dalton@gmail.com>
Reviewed-By: Gus Caplan <me@gus.host>

shisama added a commit to shisama/node that referenced this pull request Jun 17, 2018

worker: rename to worker_threads
PR-URL: nodejs#20876
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Shingo Inoue <leko.noor@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: John-David Dalton <john.david.dalton@gmail.com>
Reviewed-By: Gus Caplan <me@gus.host>

@targos targos referenced this pull request Jun 19, 2018

Merged

v10.5.0 proposal #21400

targos added a commit that referenced this pull request Jun 20, 2018

2018-16-20, Version 10.5.0 (Current)
Notable changes:

* **crypto**:
  * Support for `crypto.scrypt()` has been added.
    [#20816](#20816)
* **fs**:
  * BigInt support has been added to `fs.stat` and `fs.watchFile`.
    [#20220](#20220)
  * APIs that take `mode` as arguments no longer throw on values larger
    than `0o777`. [#20636](#20636)
    [#20975](#20975)
    (Fixes: [$20498](#20498))
  * Fix crashes in closed event watchers.
    [#20985](#20985)
    (Fixes: [#20297](#20297))
* **Worker Threads**:
  * Support for multi-threading has been added behind the
    `--experimental-worker` flag in the `worker_threads` module.
    This feature is *experimental* and may receive breaking changes at
    any time. [#20876](#20876)

PR-URL: #21400

targos added a commit that referenced this pull request Jun 20, 2018

2018-16-20, Version 10.5.0 (Current)
Notable changes:

* **crypto**:
  * Support for `crypto.scrypt()` has been added.
    [#20816](#20816)
* **fs**:
  * BigInt support has been added to `fs.stat` and `fs.watchFile`.
    [#20220](#20220)
  * APIs that take `mode` as arguments no longer throw on values larger
    than `0o777`. [#20636](#20636)
    [#20975](#20975)
    (Fixes: [#20498](#20498))
  * Fix crashes in closed event watchers.
    [#20985](#20985)
    (Fixes: [#20297](#20297))
* **Worker Threads**:
  * Support for multi-threading has been added behind the
    `--experimental-worker` flag in the `worker_threads` module.
    This feature is *experimental* and may receive breaking changes at
    any time. [#20876](#20876)

PR-URL: #21400

targos added a commit that referenced this pull request Jun 20, 2018

2018-16-20, Version 10.5.0 (Current)
Notable changes:

* **crypto**:
  * Support for `crypto.scrypt()` has been added.
    [#20816](#20816)
* **fs**:
  * BigInt support has been added to `fs.stat` and `fs.watchFile`.
    [#20220](#20220)
  * APIs that take `mode` as arguments no longer throw on values larger
    than `0o777`. [#20636](#20636)
    [#20975](#20975)
    (Fixes: [#20498](#20498))
  * Fix crashes in closed event watchers.
    [#20985](#20985)
    (Fixes: [#20297](#20297))
* **Worker Threads**:
  * Support for multi-threading has been added behind the
    `--experimental-worker` flag in the `worker_threads` module.
    This feature is *experimental* and may receive breaking changes at
    any time. [#20876](#20876)

PR-URL: #21400

calebeby added a commit to Pigmice2733/scouting-frontend that referenced this pull request Jun 20, 2018

Update Node.js to v10.5.0 (#194)
This Pull Request updates dependency [node](https://github.com/nodejs/node) from `v10.4.1` to `v10.5.0`



<details>
<summary>Release Notes</summary>

### [`v10.5.0`](https://github.com/nodejs/node/releases/v10.5.0)
[Compare Source](nodejs/node@v10.4.1...v10.5.0)
##### Notable Changes
 
* **crypto**:
  * Support for `crypto.scrypt()` has been added. [#&#8203;20816](`nodejs/node#20816`)
* **fs**:
  * BigInt support has been added to `fs.stat` and `fs.watchFile`. [#&#8203;20220](`nodejs/node#20220`)
  * APIs that take `mode` as arguments no longer throw on values larger than
    `0o777`. [#&#8203;20636](`nodejs/node#20636`) [#&#8203;20975](`nodejs/node#20975`) (Fixes: [#&#8203;20498](`nodejs/node#20498`))
  * Fix crashes in closed event watchers. [#&#8203;20985](`nodejs/node#20985`) (Fixes: [#&#8203;20297](`nodejs/node#20297`))
* **Worker Threads**:
  * Support for multi-threading has been added behind the
    `--experimental-worker` flag in the `worker_threads` module. This feature
    is *experimental* and may receive breaking changes at any time. [#&#8203;20876](`nodejs/node#20876`)
##### Commits
 
* [[`a6986fe8b6`](nodejs/node@a6986fe] - **async_hooks**: remove deprecated example (Mathias Buus) [#&#8203;20998](`nodejs/node#20998`)
* [[`4b9817bf1e`](nodejs/node@4b9817b] - **benchmark**: disable only the ESLint rule needing it (Rich Trott) [#&#8203;21133](`nodejs/node#21133`)
* [[`ecba1c57b1`](nodejs/node@ecba1c5] - **(SEMVER-MINOR)** **benchmark**: port cluster/echo to worker (Timothy Gu) [#&#8203;20876](`nodejs/node#20876`)
* [[`02adb2d62c`](nodejs/node@02adb2d] - **(SEMVER-MINOR)** **build**: expose openssl scrypt functions to addons (Ben Noordhuis) [#&#8203;20816](`nodejs/node#20816`)
* [[`c3fbac432f`](nodejs/node@c3fbac4] - **build**: install markdown linter for travis (Richard Lau) [#&#8203;21215](`nodejs/node#21215`)
* [[`896017b134`](nodejs/node@896017b] - **build**: build addon tests in parallel (Anna Henningsen) [#&#8203;21155](`nodejs/node#21155`)
* [[`76927fc734`](nodejs/node@76927fc] - **build**: stop distclean from deleting v8 files (Ujjwal Sharma) [#&#8203;21164](`nodejs/node#21164`)
* [[`b044256f2a`](nodejs/node@b044256] - **build**: use LC\_ALL of C for maximum compatibility (Rich Trott) [#&#8203;21222](`nodejs/node#21222`)
* [[`78c7d666fb`](nodejs/node@78c7d66] - **build**: don't change locale on smartos (Refael Ackermann) [#&#8203;21220](`nodejs/node#21220`)
* [[`c688a00a6d`](nodejs/node@c688a00] - **build**: fix 'gas\_version' check on localized environments (Evandro Oliveira) [#&#8203;20394](`nodejs/node#20394`)
* [[`79b3423fb5`](nodejs/node@79b3423] - **build**: initial .travis.yml implementation (Anna Henningsen) [#&#8203;21059](`nodejs/node#21059`)
* [[`ea4be72f22`](nodejs/node@ea4be72] - **child_process**: swallow errors in internal communication (Anatoli Papirovski) [#&#8203;21108](`nodejs/node#21108`)
* [[`9981220e2a`](nodejs/node@9981220] - **crypto**: fix behavior of createCipher in wrap mode (Tobias Nießen) [#&#8203;21287](`nodejs/node#21287`)
* [[`d0cb9cbb35`](nodejs/node@d0cb9cb] - **(SEMVER-MINOR)** **crypto**: drop Math.pow(), use static exponentation (Ben Noordhuis) [#&#8203;20816](`nodejs/node#20816`)
* [[`2d9c3cc89d`](nodejs/node@2d9c3cc] - **(SEMVER-MINOR)** **crypto**: refactor randomBytes() (Ben Noordhuis) [#&#8203;20816](`nodejs/node#20816`)
* [[`6262fa44d6`](nodejs/node@6262fa4] - **(SEMVER-MINOR)** **crypto**: refactor pbkdf2() and pbkdf2Sync() methods (Ben Noordhuis) [#&#8203;20816](`nodejs/node#20816`)
* [[`c9b4592dbf`](nodejs/node@c9b4592] - **(SEMVER-MINOR)** **crypto**: add scrypt() and scryptSync() methods (Ben Noordhuis) [#&#8203;20816](`nodejs/node#20816`)
* [[`495756264a`](nodejs/node@4957562] - **(SEMVER-MINOR)** **crypto**: DRY type checking (Ben Noordhuis) [#&#8203;20816](`nodejs/node#20816`)
* [[`e4a7e0d28b`](nodejs/node@e4a7e0d] - **deps**: float ea7abee from openssl / CVE-2018-0732 (Rod Vagg) [#&#8203;21282](`nodejs/node#21282`)
* [[`0b90b071c4`](nodejs/node@0b90b07] - **deps**: Upgrade node-inspect to 1.11.5 (Jan Krems) [#&#8203;21055](`nodejs/node#21055`)
* [[`ffc29c12da`](nodejs/node@ffc29c1] - **deps**: patch V8 to 6.7.288.46 (Myles Borins) [#&#8203;21260](`nodejs/node#21260`)
* [[`14bb905d18`](nodejs/node@14bb905] - **deps**: V8: cherry-pick a440efb27f from upstream (Yang Guo) [#&#8203;21022](`nodejs/node#21022`)
* [[`65b9c427ac`](nodejs/node@65b9c42] - **dns**: improve setServers() errors and performance (Jamie Davis) [#&#8203;20445](`nodejs/node#20445`)
* [[`bc20ec0c0f`](nodejs/node@bc20ec0] - **doc**: eliminate \_you\_ from N-API doc (Rich Trott) [#&#8203;21382](`nodejs/node#21382`)
* [[`318d6831bf`](nodejs/node@318d683] - **doc**: use imperative in COLLABORATOR\_GUIDE (Rich Trott) [#&#8203;21340](`nodejs/node#21340`)
* [[`177a7c06a8`](nodejs/node@177a7c0] - **doc**: remove obsolete wiki references from BUILDING (Rich Trott) [#&#8203;21369](`nodejs/node#21369`)
* [[`15023df050`](nodejs/node@15023df] - **doc**: add davisjam to collaborators (Jamie Davis) [#&#8203;21273](`nodejs/node#21273`)
* [[`17c21b67ac`](nodejs/node@17c21b6] - **doc**: fix indentation in console.md (Vse Mozhet Byt) [#&#8203;21367](`nodejs/node#21367`)
* [[`ef74368416`](nodejs/node@ef74368] - **doc**: fix heading of optional console method args (Michaël Zasso) [#&#8203;21311](`nodejs/node#21311`)
* [[`4f17841c20`](nodejs/node@4f17841] - **doc**: use Class Method label consistently (Rich Trott) [#&#8203;21357](`nodejs/node#21357`)
* [[`4566ebacf4`](nodejs/node@4566eba] - **doc**: wrap style guide at 80 characters (Rich Trott) [#&#8203;21361](`nodejs/node#21361`)
* [[`6c41f33571`](nodejs/node@6c41f33] - **doc**: wrap pull-requests.md at 80 characters (Rich Trott) [#&#8203;21361](`nodejs/node#21361`)
* [[`b8213f17cc`](nodejs/node@b8213f1] - **doc**: remove linking of url text to url (Rich Trott) [#&#8203;21361](`nodejs/node#21361`)
* [[`3f78220c2b`](nodejs/node@3f78220] - **doc**: correct styling of \_GitHub\_ in onboarding doc (Rich Trott) [#&#8203;21361](`nodejs/node#21361`)
* [[`9e994cb119`](nodejs/node@9e994cb] - **doc**: wrap releases.md at 80 chars (Rich Trott) [#&#8203;21361](`nodejs/node#21361`)
* [[`e00e5e6d5d`](nodejs/node@e00e5e6] - **doc**: switch the order of Writable and Readable (Joseph Gordon) [#&#8203;21333](`nodejs/node#21333`)
* [[`e1b571d6b7`](nodejs/node@e1b571d] - **doc**: make Deprecation cycle explanation more brief (Rich Trott) [#&#8203;21303](`nodejs/node#21303`)
* [[`df0f7a3b4d`](nodejs/node@df0f7a3] - **doc**: clarify async execute callback usage (Michael Dawson) [#&#8203;21217](`nodejs/node#21217`)
* [[`c5a65594ef`](nodejs/node@c5a6559] - **doc**: move 5 collaborators to emeritus status (Rich Trott) [#&#8203;21272](`nodejs/node#21272`)
* [[`c1d53f86f8`](nodejs/node@c1d53f8] - **doc**: update NODE\_OPTIONS section in cli.md (Vse Mozhet Byt) [#&#8203;21229](`nodejs/node#21229`)
* [[`13fd09bfa7`](nodejs/node@13fd09b] - **doc**: add build wg info to releases.md (Jon Moss) [#&#8203;21275](`nodejs/node#21275`)
* [[`0da910f9a5`](nodejs/node@0da910f] - **doc**: move Italo A. Casas to Release Emeritus (Myles Borins) [#&#8203;21315](`nodejs/node#21315`)
* [[`6f7de0b8d9`](nodejs/node@6f7de0b] - **doc**: trim deprecation level definition text (Rich Trott) [#&#8203;21241](`nodejs/node#21241`)
* [[`dd2fc90dcf`](nodejs/node@dd2fc90] - **doc**: fix reference to workerData in worker\_threads (Jeremiah Senkpiel) [#&#8203;21180](`nodejs/node#21180`)
* [[`5e46c16371`](nodejs/node@5e46c16] - **doc**: fix type in stream doc (Aliaksei Tuzik) [#&#8203;21178](`nodejs/node#21178`)
* [[`85dc9ac418`](nodejs/node@85dc9ac] - **doc**: add Michaël Zasso to Release team (Michaël Zasso) [#&#8203;21114](`nodejs/node#21114`)
* [[`5fa5ab6c48`](nodejs/node@5fa5ab6] - **doc**: naming function as suggested in addon docs (Tommaso Allevi) [#&#8203;21067](`nodejs/node#21067`)
* [[`fe5d35123b`](nodejs/node@fe5d351] - **(SEMVER-MINOR)** **doc**: document BigInt support in fs.Stats (Joyee Cheung) [#&#8203;20220](`nodejs/node#20220`)
* [[`2c4f80ffba`](nodejs/node@2c4f80f] - **doc**: remove spaces around slashes (Rich Trott) [#&#8203;21140](`nodejs/node#21140`)
* [[`72e7e1da2d`](nodejs/node@72e7e1d] - **doc**: alphabetize tls options (Rich Trott) [#&#8203;21139](`nodejs/node#21139`)
* [[`06ac81e786`](nodejs/node@06ac81e] - **doc**: streamline errors.md introductory material (Rich Trott) [#&#8203;21138](`nodejs/node#21138`)
* [[`73b8975b41`](nodejs/node@73b8975] - **doc**: simplify deprecation language (Rich Trott) [#&#8203;21136](`nodejs/node#21136`)
* [[`6caa354377`](nodejs/node@6caa354] - **(SEMVER-MINOR)** **doc**: explain Worker semantics in async\_hooks.md (Anna Henningsen) [#&#8203;20876](`nodejs/node#20876`)
* [[`9f9355d6d2`](nodejs/node@9f9355d] - **doc**: fix inconsistent documentation (host vs hostname) (Davis Okoth) [#&#8203;20933](`nodejs/node#20933`)
* [[`a5c571424a`](nodejs/node@a5c5714] - **doc**: document file mode caveats on Windows (Joyee Cheung) [#&#8203;20636](`nodejs/node#20636`)
* [[`a75e44d135`](nodejs/node@a75e44d] - **esm**: ensure require.main for CJS top-level loads (Guy Bedford) [#&#8203;21150](`nodejs/node#21150`)
* [[`04e8f0749e`](nodejs/node@04e8f07] - **(SEMVER-MINOR)** **fs**: support BigInt in fs.\*stat and fs.watchFile (Joyee Cheung) [#&#8203;20220](`nodejs/node#20220`)
* [[`c09bfd81b7`](nodejs/node@c09bfd8] - **fs**: do not crash when using a closed fs event watcher (Joyee Cheung) [#&#8203;20985](`nodejs/node#20985`)
* [[`bacb2cb550`](nodejs/node@bacb2cb] - **fs**: refactor fs module (James M Snell) [#&#8203;20764](`nodejs/node#20764`)
* [[`db0bb5214a`](nodejs/node@db0bb52] - **fs**: improve fchmod{Sync} validation (cjihrig) [#&#8203;20588](`nodejs/node#20588`)
* [[`2ffb9d6b5c`](nodejs/node@2ffb9d6] - **fs**: drop duplicate API in promises mode (Сковорода Никита Андреевич) [#&#8203;20559](`nodejs/node#20559`)
* [[`fc0b3610e2`](nodejs/node@fc0b361] - **fs**: don't limit ftruncate() length to 32 bits (cjihrig) [#&#8203;20851](`nodejs/node#20851`)
* [[`469baa062e`](nodejs/node@469baa0] - **fs**: add length validation to fs.truncate() (cjihrig) [#&#8203;20851](`nodejs/node#20851`)
* [[`6aade4a765`](nodejs/node@6aade4a] - **http**: remove a pair of outdated comments (Mark S. Everitt) [#&#8203;21214](`nodejs/node#21214`)
* [[`bcaf59c739`](nodejs/node@bcaf59c] - **http2**: fix memory leak for uncommon headers (Anna Henningsen) [#&#8203;21336](`nodejs/node#21336`)
* [[`dee250fd77`](nodejs/node@dee250f] - **http2**: safer Http2Session destructor (Anatoli Papirovski) [#&#8203;21194](`nodejs/node#21194`)
* [[`296fd57324`](nodejs/node@296fd57] - **inspector**: stop dragging platform pointer (Eugene Ostroukhov) 
* [[`fb71337bdf`](nodejs/node@fb71337] - **(SEMVER-MINOR)** **lib**: rename checkIsArrayBufferView() (Ben Noordhuis) [#&#8203;20816](`nodejs/node#20816`)
* [[`f3570f201b`](nodejs/node@f3570f2] - **(SEMVER-MINOR)** **lib**: replace checkUint() with validateInt32() (Ben Noordhuis) [#&#8203;20816](`nodejs/node#20816`)
* [[`b4b7d368be`](nodejs/node@b4b7d36] - **lib**: unmask mode\_t values with 0o777 (Joyee Cheung) [#&#8203;20975](`nodejs/node#20975`)
* [[`36e5100a39`](nodejs/node@36e5100] - **lib**: support ranges in validateInt32() (cjihrig) [#&#8203;20588](`nodejs/node#20588`)
* [[`2fe88d2218`](nodejs/node@2fe88d2] - **lib**: mask mode\_t type of arguments with 0o777 (Joyee Cheung) [#&#8203;20636](`nodejs/node#20636`)
* [[`a0cfb0c9d4`](nodejs/node@a0cfb0c] - **lib**: add validateInteger() validator (cjihrig) [#&#8203;20851](`nodejs/node#20851`)
* [[`740d9f1a0e`](nodejs/node@740d9f1] - **lib,src**: make `StatWatcher` a `HandleWrap` (Anna Henningsen) [#&#8203;21244](`nodejs/node#21244`)
* [[`a657984109`](nodejs/node@a657984] - **lib,src**: remove openssl feature conditionals (Ben Noordhuis) [#&#8203;21094](`nodejs/node#21094`)
* [[`653b20b26d`](nodejs/node@653b20b] - **loader**: remove unused error code in module\_job (Gus Caplan) [#&#8203;21354](`nodejs/node#21354`)
* [[`5d3dfedca2`](nodejs/node@5d3dfed] - **meta**: remove CODEOWNERS (Rich Trott) [#&#8203;21161](`nodejs/node#21161`)
* [[`169bff3e9e`](nodejs/node@169bff3] - **n-api**: name CallbackBundle function fields (Anna Henningsen) [#&#8203;21240](`nodejs/node#21240`)
* [[`1dc9330b3a`](nodejs/node@1dc9330] - **n-api**: improve runtime perf of n-api func call (Kenny Yuan) [#&#8203;21072](`nodejs/node#21072`)
* [[`9047c8182c`](nodejs/node@9047c81] - **n-api**: remove unused napi\_env member (Gabriel Schulhof) [#&#8203;21127](`nodejs/node#21127`)
* [[`18c057ab26`](nodejs/node@18c057a] - **net**: emit 'close' when socket ends before connect (Brett Kiefer) [#&#8203;21290](`nodejs/node#21290`)
* [[`a3fd1cd8ea`](nodejs/node@a3fd1cd] - **perf_hooks**: remove less useful bootstrap marks (James M Snell) [#&#8203;21247](`nodejs/node#21247`)
* [[`8fddf591c5`](nodejs/node@8fddf59] - **perf_hooks**: set bootstrap complete in only one place (James M Snell) [#&#8203;21247](`nodejs/node#21247`)
* [[`fc2956d37a`](nodejs/node@fc2956d] - **process**: backport process/methods file (Michaël Zasso) [#&#8203;21172](`nodejs/node#21172`)
* [[`78ad4e9dde`](nodejs/node@78ad4e9] - **src**: remove unused argc var in node\_stat\_watcher (Daniel Bevenius) [#&#8203;21337](`nodejs/node#21337`)
* [[`7fa1344143`](nodejs/node@7fa1344] - **src**: use `%zx` in printf for size\_t (Anna Henningsen) [#&#8203;21323](`nodejs/node#21323`)
* [[`671346ee8f`](nodejs/node@671346e] - **src**: do proper error checking in `AsyncWrap::MakeCallback` (Anna Henningsen) [#&#8203;21189](`nodejs/node#21189`)
* [[`aa468abc4c`](nodejs/node@aa468ab] - **src**: unify native symbol inspection code (Anna Henningsen) [#&#8203;21238](`nodejs/node#21238`)
* [[`e92b89a75d`](nodejs/node@e92b89a] - **src**: fix http2 typos (Anatoli Papirovski) [#&#8203;21194](`nodejs/node#21194`)
* [[`4f01168414`](nodejs/node@4f01168] - **src**: do not persist fs\_poll handle in stat\_watcher (Anatoli Papirovski) [#&#8203;21093](`nodejs/node#21093`)
* [[`685b9b2a6a`](nodejs/node@685b9b2] - **src**: do not persist timer handle in cares\_wrap (Anatoli Papirovski) [#&#8203;21093](`nodejs/node#21093`)
* [[`4757771db3`](nodejs/node@4757771] - **src**: add consistency check to node\_platform.cc (Anna Henningsen) [#&#8203;21156](`nodejs/node#21156`)
* [[`8e2e16721b`](nodejs/node@8e2e167] - **src**: add node\_encoding.cc (James M Snell) [#&#8203;21112](`nodejs/node#21112`)
* [[`39b38754eb`](nodejs/node@39b3875] - **src**: cleanup beforeExit for consistency (James M Snell) [#&#8203;21113](`nodejs/node#21113`)
* [[`314b47d1cf`](nodejs/node@314b47d] - **(SEMVER-MINOR)** **src**: add Env::profiler\_idle\_notifier\_started() (Timothy Gu) [#&#8203;20876](`nodejs/node#20876`)
* [[`5209ff9562`](nodejs/node@5209ff9] - **(SEMVER-MINOR)** **src**: remove unused fields msg\_ and env\_ (Daniel Bevenius) [#&#8203;20876](`nodejs/node#20876`)
* [[`9a734132f9`](nodejs/node@9a73413] - **(SEMVER-MINOR)** **src**: make handle onclose property a Symbol (Anna Henningsen) [#&#8203;20876](`nodejs/node#20876`)
* [[`e6f06807b1`](nodejs/node@e6f0680] - **(SEMVER-MINOR)** **src**: simplify handle closing (Anna Henningsen) [#&#8203;20876](`nodejs/node#20876`)
* [[`65924c70e8`](nodejs/node@65924c7] - **(SEMVER-MINOR)** **src**: remove unused fields isolate\_ (Daniel Bevenius) [#&#8203;20876](`nodejs/node#20876`)
* [[`de7403f813`](nodejs/node@de7403f] - **(SEMVER-MINOR)** **src**: cleanup per-isolate state on platform on isolate unregister (Anna Henningsen) [#&#8203;20876](`nodejs/node#20876`)
* [[`ba17c9e46b`](nodejs/node@ba17c9e] - **src**: refactor bootstrap to use bootstrap object (James M Snell) [#&#8203;20917](`nodejs/node#20917`)
* [[`cbdc1fdf44`](nodejs/node@cbdc1fd] - **src, tools**: add check for left leaning pointers (Daniel Bevenius) [#&#8203;21010](`nodejs/node#21010`)
* [[`935309325b`](nodejs/node@9353093] - **test**: fix deprecation warning due to util.print (Tobias Nießen) [#&#8203;21265](`nodejs/node#21265`)
* [[`d7ba75f8aa`](nodejs/node@d7ba75f] - **test**: add test to check colorMode type of Console (Masashi Hirano) [#&#8203;21248](`nodejs/node#21248`)
* [[`0b00172df8`](nodejs/node@0b00172] - **test**: removing unnecessary parameter from assert call (djmgit) [#&#8203;21307](`nodejs/node#21307`)
* [[`dea3ac7bff`](nodejs/node@dea3ac7] - **test**: improve statwatcher async\_hooks test (Anna Henningsen) [#&#8203;21244](`nodejs/node#21244`)
* [[`792335f712`](nodejs/node@792335f] - **test**: add workerdata-sharedarraybuffer test (Jeremiah Senkpiel) [#&#8203;21180](`nodejs/node#21180`)
* [[`e8d15cb149`](nodejs/node@e8d15cb] - **test**: mark test-inspector-port-zero-cluster flaky (Rich Trott) [#&#8203;21251](`nodejs/node#21251`)
* [[`688bdfef7f`](nodejs/node@688bdfe] - **test**: add crypto check to test-http2-debug (Daniel Bevenius) [#&#8203;21205](`nodejs/node#21205`)
* [[`2270ab2a12`](nodejs/node@2270ab2] - **test**: remove string literals from assert.strictEqual() calls (James Kylstra) [#&#8203;21211](`nodejs/node#21211`)
* [[`187951c0fc`](nodejs/node@187951c] - **test**: move inspector-stress-http to sequential (Rich Trott) [#&#8203;21227](`nodejs/node#21227`)
* [[`bda34ea203`](nodejs/node@bda34ea] - **test**: check gc does not resurrect the loop (Anatoli Papirovski) [#&#8203;21093](`nodejs/node#21093`)
* [[`4d782c4720`](nodejs/node@4d782c4] - **test**: improve assert error messages (Hristijan Gjorgjievski) [#&#8203;21160](`nodejs/node#21160`)
* [[`2655c7b194`](nodejs/node@2655c7b] - **test**: mark fs-readfile-tostring-fail flaky for all (Rich Trott) [#&#8203;21177](`nodejs/node#21177`)
* [[`17954c2b01`](nodejs/node@17954c2] - **test**: improve internal/buffer.js test coverage (Masashi Hirano) [#&#8203;21061](`nodejs/node#21061`)
* [[`2ff4704447`](nodejs/node@2ff4704] - **test**: move test-readuint to test-buffer-readuint (Michaël Zasso) [#&#8203;21170](`nodejs/node#21170`)
* [[`9c3a7bf076`](nodejs/node@9c3a7bf] - **test**: make url-util-format engine agnostic (Rich Trott) [#&#8203;21141](`nodejs/node#21141`)
* [[`3d8ec8f85c`](nodejs/node@3d8ec8f] - **test**: make url-parse-invalid-input engine agnostic (Rich Trott) [#&#8203;21132](`nodejs/node#21132`)
* [[`0b0370f884`](nodejs/node@0b0370f] - **test**: remove unref in http2 test (Anatoli Papirovski) [#&#8203;21145](`nodejs/node#21145`)
* [[`14a017cf8d`](nodejs/node@14a017c] - **test**: apply promises API to fourth appendFile test (Rich Trott) [#&#8203;21131](`nodejs/node#21131`)
* [[`aa9dbf666b`](nodejs/node@aa9dbf6] - **test**: apply promises API to fourth appendFile test (Rich Trott) [#&#8203;21131](`nodejs/node#21131`)
* [[`185b9e45d3`](nodejs/node@185b9e4] - **test**: apply promises API to third appendFile test (Rich Trott) [#&#8203;21131](`nodejs/node#21131`)
* [[`c400448e85`](nodejs/node@c400448] - **test**: improve debug output in trace-events test (Rich Trott) [#&#8203;21120](`nodejs/node#21120`)
* [[`a4ad9891e3`](nodejs/node@a4ad989] - **test**: add test for Linux perf (Matheus Marchini) [#&#8203;20783](`nodejs/node#20783`)
* [[`e16036c462`](nodejs/node@e16036c] - **test**: create new directory v8-updates (Matheus Marchini) [#&#8203;20783](`nodejs/node#20783`)
* [[`93ce63c89f`](nodejs/node@93ce63c] - **(SEMVER-MINOR)** **test**: add test against unsupported worker features (Timothy Gu) [#&#8203;20876](`nodejs/node#20876`)
* [[`94dcdfb898`](nodejs/node@94dcdfb] - **test**: increase coverage for fs.promises.truncate (Masashi Hirano) [#&#8203;20638](`nodejs/node#20638`)
* [[`c9cee63179`](nodejs/node@c9cee63] - **test,tools**: refactor custom ESLint for readability (Rich Trott) [#&#8203;21134](`nodejs/node#21134`)
* [[`ed05d9a821`](nodejs/node@ed05d9a] - **(SEMVER-MINOR)** **test,tools**: enable running tests under workers (Anna Henningsen) [#&#8203;20876](`nodejs/node#20876`)
* [[`6285fe94f6`](nodejs/node@6285fe9] - **tools**: do not disable `quotes` rule in .eslintrc.js (Rich Trott) [#&#8203;21338](`nodejs/node#21338`)
* [[`98346de08c`](nodejs/node@98346de] - **tools**: lint doc/\*.md files (Rich Trott) [#&#8203;21361](`nodejs/node#21361`)
* [[`521f8f1d95`](nodejs/node@521f8f1] - **tools**: add BigInt64Array and BigUint64Array to globals (Joyee Cheung) [#&#8203;21255](`nodejs/node#21255`)
* [[`a5c386d1ba`](nodejs/node@a5c386d] - **tools**: add option to use custom template with js2c.py (Shelley Vohr) [#&#8203;21187](`nodejs/node#21187`)
* [[`7f70fe83ef`](nodejs/node@7f70fe8] - **tools**: add BigInt to globals (Nikolai Vavilov) [#&#8203;21237](`nodejs/node#21237`)
* [[`4e742e379b`](nodejs/node@4e742e3] - **tools**: update tooling to work with new macOS CLI … (Rich Trott) [#&#8203;21173](`nodejs/node#21173`)
* [[`ed2b57bcd5`](nodejs/node@ed2b57b] - **tools**: remove unused global types from type-parser (Rich Trott) [#&#8203;21135](`nodejs/node#21135`)
* [[`d46446afc5`](nodejs/node@d46446a] - **v8**: replace Buffer with FastBuffer in deserialize (Ujjwal Sharma) [#&#8203;21196](`nodejs/node#21196`)
* [[`917960e0a1`](nodejs/node@917960e] - **win, build**: add documentation support to vcbuild (Bartosz Sosnowski) [#&#8203;19663](`nodejs/node#19663`)
* [[`03fbc9e749`](nodejs/node@03fbc9e] - **(SEMVER-MINOR)** **worker**: rename to worker\_threads (Anna Henningsen) [#&#8203;20876](`nodejs/node#20876`)
* [[`9ad42b766e`](nodejs/node@9ad42b7] - **(SEMVER-MINOR)** **worker**: improve error (de)serialization (Anna Henningsen) [#&#8203;20876](`nodejs/node#20876`)
* [[`6b1a887aa2`](nodejs/node@6b1a887] - **(SEMVER-MINOR)** **worker**: enable stdio (Anna Henningsen) [#&#8203;20876](`nodejs/node#20876`)
* [[`c97fb91e55`](nodejs/node@c97fb91] - **(SEMVER-MINOR)** **worker**: restrict supported extensions (Timothy Gu) [#&#8203;20876](`nodejs/node#20876`)
* [[`109c92e8fa`](nodejs/node@109c92e] - **(SEMVER-MINOR)** **worker**: initial implementation (Anna Henningsen) [#&#8203;20876](`nodejs/node#20876`)
* [[`d1f372f052`](nodejs/node@d1f372f] - **(SEMVER-MINOR)** **worker**: add `SharedArrayBuffer` sharing (Anna Henningsen) [#&#8203;20876](`nodejs/node#20876`)
* [[`f447acd87b`](nodejs/node@f447acd] - **(SEMVER-MINOR)** **worker**: support MessagePort passing in messages (Anna Henningsen) [#&#8203;20876](`nodejs/node#20876`)
* [[`337be58ee6`](nodejs/node@337be58] - **(SEMVER-MINOR)** **worker**: implement `MessagePort` and `MessageChannel` (Anna Henningsen) [#&#8203;20876](`nodejs/node#20876`)
* [[`4a54ebc3bd`](nodejs/node@4a54ebc] - **worker,src**: display remaining handles if `uv\_loop\_close` fails (Anna Henningsen) [#&#8203;21238](`nodejs/node#21238`)
* [[`529d24e3e8`](nodejs/node@529d24e] - ***Revert*** "**workers,trace_events**: set thread name for workers" (James M Snell) [#&#8203;21363](`nodejs/node#21363`)
* [[`dfb5cf6963`](nodejs/node@dfb5cf6] - **workers,trace_events**: set thread name for workers (James M Snell) [#&#8203;21246](`nodejs/node#21246`)

---

</details>




---

This PR has been generated by [Renovate Bot](https://renovatebot.com).
@idibidiart

This comment has been minimized.

Show comment
Hide comment
@idibidiart

idibidiart Jun 21, 2018

<<Native addons are not loadable from worker threads (yet).>>

Electron docs:

<<Most existing native modules have been written assuming single-threaded environment, using them in Web Workers will lead to crashes and memory corruptions. Note that even if a native Node.js module is thread-safe it's still not safe to load it in a Web Worker because the process.dlopen function is not thread safe.>>

I assume you'll have a follow-up PR for process.dlopen?

How about require()-ing built-in modules like zlib and fs? mind my lack of knowledge here but are they thread safe?

Thank you for this. It's about time Node has built-in threading support...

EDIT:

Also, how about the File object and JS objects in general? Latest browsers allow us to send JS objects between main thread and worker thread without ceremony, i.e. no longer need to specify as transferrable objects. For File object, I believe the reference to the file on disk is copied, not the entire content of the file. Can we hope for similar functionality in node workers?

idibidiart commented Jun 21, 2018

<<Native addons are not loadable from worker threads (yet).>>

Electron docs:

<<Most existing native modules have been written assuming single-threaded environment, using them in Web Workers will lead to crashes and memory corruptions. Note that even if a native Node.js module is thread-safe it's still not safe to load it in a Web Worker because the process.dlopen function is not thread safe.>>

I assume you'll have a follow-up PR for process.dlopen?

How about require()-ing built-in modules like zlib and fs? mind my lack of knowledge here but are they thread safe?

Thank you for this. It's about time Node has built-in threading support...

EDIT:

Also, how about the File object and JS objects in general? Latest browsers allow us to send JS objects between main thread and worker thread without ceremony, i.e. no longer need to specify as transferrable objects. For File object, I believe the reference to the file on disk is copied, not the entire content of the file. Can we hope for similar functionality in node workers?

@TimothyGu

This comment has been minimized.

Show comment
Hide comment
@TimothyGu

TimothyGu Jun 21, 2018

Member

I assume you'll have a follow-up PR for process.dlopen?

Yes, there will be.

How about require()-ing built-in modules like zlib and fs? mind my lack of knowledge here but are they thread safe?

They have been made to be.

Latest browsers allow us to send JS objects between main thread and worker thread without ceremony, i.e. no longer need to specify as transferrable objects.

The transfer mechanism for general JavaScript objects is implemented identically as browsers do.

For File object, I believe the reference to the file on disk is copied, not the entire content of the file. Can we hope for similar functionality in node workers?

There has been interest to implement File and Blob classes as well as the rest of the File API in Node.js, but we no concrete plan has emerged yet. Stay tuned.


We will be happy to answer any follow-up question if you file an issue at https://github.com/nodejs/help. Thanks for your interest in worker threads!

Member

TimothyGu commented Jun 21, 2018

I assume you'll have a follow-up PR for process.dlopen?

Yes, there will be.

How about require()-ing built-in modules like zlib and fs? mind my lack of knowledge here but are they thread safe?

They have been made to be.

Latest browsers allow us to send JS objects between main thread and worker thread without ceremony, i.e. no longer need to specify as transferrable objects.

The transfer mechanism for general JavaScript objects is implemented identically as browsers do.

For File object, I believe the reference to the file on disk is copied, not the entire content of the file. Can we hope for similar functionality in node workers?

There has been interest to implement File and Blob classes as well as the rest of the File API in Node.js, but we no concrete plan has emerged yet. Stay tuned.


We will be happy to answer any follow-up question if you file an issue at https://github.com/nodejs/help. Thanks for your interest in worker threads!

@idibidiart

This comment has been minimized.

Show comment
Hide comment
@idibidiart

idibidiart Jun 21, 2018

Fantastic! really important work you’re doing! Thank you!!

Fantastic! really important work you’re doing! Thank you!!

@PazzaVlad

This comment has been minimized.

Show comment
Hide comment
@PazzaVlad

PazzaVlad Jun 24, 2018

Thank you, really cool stuff!

Thank you, really cool stuff!

@pavel-sindelka

This comment has been minimized.

Show comment
Hide comment
@pavel-sindelka

pavel-sindelka Jun 25, 2018

Hello guys! Am I able to use multi thread with Puppeteer for parallel testing?
https://github.com/GoogleChrome/puppeteer

Hello guys! Am I able to use multi thread with Puppeteer for parallel testing?
https://github.com/GoogleChrome/puppeteer

@benjamingr

This comment has been minimized.

Show comment
Hide comment
@benjamingr

benjamingr Jun 25, 2018

Member

@pavel-sindelka puppeteer is not CPU bound typically and runs an off-process Chrome - so that use case would not be a good fit for worker-threads.

Member

benjamingr commented Jun 25, 2018

@pavel-sindelka puppeteer is not CPU bound typically and runs an off-process Chrome - so that use case would not be a good fit for worker-threads.

@p3x-robot

This comment has been minimized.

Show comment
Hide comment
@p3x-robot

p3x-robot Jun 26, 2018

@addaleax so is the cluster obsolate? i think cluster is still good right?

@addaleax so is the cluster obsolate? i think cluster is still good right?

@idibidiart

This comment has been minimized.

Show comment
Hide comment
@idibidiart

idibidiart Jun 26, 2018

with a cluster you can use a load balancer but there is no shared memory out of the box and a process is heavier than a thread... each process can then have multiple threads with shared memory... I think the use case for threads is cpu bound work (vertical scaling) while clusters are for horizontal scaling...

would be interested in other opinions

with a cluster you can use a load balancer but there is no shared memory out of the box and a process is heavier than a thread... each process can then have multiple threads with shared memory... I think the use case for threads is cpu bound work (vertical scaling) while clusters are for horizontal scaling...

would be interested in other opinions

@p3x-robot

This comment has been minimized.

Show comment
Hide comment
@p3x-robot

p3x-robot Jun 26, 2018

@idibidiart dont you think that the https, mongodb, redis is better using via cluster and for cpu processing is good for threads? look java? so slow with threads via https connections?

@idibidiart dont you think that the https, mongodb, redis is better using via cluster and for cpu processing is good for threads? look java? so slow with threads via https connections?

@idibidiart

This comment has been minimized.

Show comment
Hide comment
@idibidiart

idibidiart Jun 26, 2018

I would use threads where shared memory is a plus not a minus and where the work should not be done in the event loop (i.e. any cpu bound work), so yes I'd agree with your statement

I would use threads where shared memory is a plus not a minus and where the work should not be done in the event loop (i.e. any cpu bound work), so yes I'd agree with your statement

@p3x-robot

This comment has been minimized.

Show comment
Hide comment
@p3x-robot

p3x-robot Jun 26, 2018

thanks for you response.

thanks for you response.

@p3x-robot

This comment has been minimized.

Show comment
Hide comment
@p3x-robot

p3x-robot Jun 27, 2018

Guys, do you know how can fix this error?

memory leak detected

I use like this:

require('events').EventEmitter.prototype._maxListeners = 100;
require('events').EventEmitter.defaultMaxListeners = 100;
process.setMaxListeners(100)

In both worker and main thread as well.

C:\Users\patrikx3\Projects\patrikx3\play\scripts\worker-thread>node-thread main.js
Spawning thread 1
Spawning thread 2
Spawning thread 3
Spawning thread 4
Spawning thread 5
Spawning thread 6
Spawning thread 7
Spawning thread 8
Spawning thread 9
Spawning thread 10
Spawning thread 11
Spawning thread 12
Spawning thread 13
Spawning thread 14
Spawning thread 15
Spawning thread 16
Spawning thread 17
Spawning thread 18
Spawning thread 19
Spawning thread 20
(node:9816) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added. Use emitter.setMaxListeners() to increase limit
(node:9816) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 21 error listeners added. Use emitter.setMaxListeners() to increase limit
2018-6-27 08:19:11: Instance 1
2018-6-27 08:19:12: Instance 2
2018-6-27 08:19:13: Instance 3
2018-6-27 08:19:13: Instance 4
2018-6-27 08:19:14: Instance 5
2018-6-27 08:19:15: Instance 6
2018-6-27 08:19:15: Instance 7
2018-6-27 08:19:16: Instance 8
2018-6-27 08:19:17: Instance 9
2018-6-27 08:19:17: Instance 10
2018-6-27 08:19:18: Instance 11
2018-6-27 08:19:19: Instance 12
2018-6-27 08:19:20: Instance 13
2018-6-27 08:19:20: Instance 14
2018-6-27 08:19:21: Instance 15
2018-6-27 08:19:22: Instance 16
2018-6-27 08:19:22: Instance 17
2018-6-27 08:19:23: Instance 18
2018-6-27 08:19:23: Instance 19
2018-6-27 08:19:24: Instance 20
Stopped thread instance 1
This is the from main thread instance 1: 102334155
Stopped thread instance 1
Stopped thread instance 2
This is the from main thread instance 2: 102334155
Stopped thread instance 2
Stopped thread instance 3
This is the from main thread instance 3: 102334155
Stopped thread instance 3
Stopped thread instance 4
This is the from main thread instance 4: 102334155
Stopped thread instance 4
Stopped thread instance 5
This is the from main thread instance 5: 102334155
Stopped thread instance 5
2018-6-27 08:19:28: Instance 6
2018-6-27 08:19:28: Instance 7
2018-6-27 08:19:28: Instance 8
2018-6-27 08:19:28: Instance 9
2018-6-27 08:19:28: Instance 10
2018-6-27 08:19:28: Instance 11
2018-6-27 08:19:28: Instance 12
2018-6-27 08:19:28: Instance 13
2018-6-27 08:19:28: Instance 14
2018-6-27 08:19:28: Instance 15
2018-6-27 08:19:28: Instance 16
2018-6-27 08:19:28: Instance 17
2018-6-27 08:19:28: Instance 18
2018-6-27 08:19:28: Instance 19
2018-6-27 08:19:28: Instance 20
Stopped thread instance 6
This is the from main thread instance 6: 102334155
Stopped thread instance 6
Stopped thread instance 7
This is the from main thread instance 7: 102334155
Stopped thread instance 7
Stopped thread instance 8
This is the from main thread instance 8: 102334155
Stopped thread instance 8
Stopped thread instance 9
This is the from main thread instance 9: 102334155
Stopped thread instance 9
Stopped thread instance 10
This is the from main thread instance 10: 102334155
Stopped thread instance 10
Stopped thread instance 11
This is the from main thread instance 11: 102334155
Stopped thread instance 11
Stopped thread instance 12
This is the from main thread instance 12: 102334155
Stopped thread instance 12
Stopped thread instance 13
This is the from main thread instance 13: 102334155
Stopped thread instance 13
Stopped thread instance 14
This is the from main thread instance 14: 102334155
Stopped thread instance 14
Stopped thread instance 15
This is the from main thread instance 15: 102334155
Stopped thread instance 15
Stopped thread instance 16
This is the from main thread instance 16: 102334155
Stopped thread instance 16
Stopped thread instance 17
This is the from main thread instance 17: 102334155
Stopped thread instance 17
Stopped thread instance 18
This is the from main thread instance 18: 102334155
Stopped thread instance 18
Stopped thread instance 19
This is the from main thread instance 19: 102334155
Stopped thread instance 19
Stopped thread instance 20
This is the from main thread instance 20: 102334155
Stopped thread instance 20

#The code

main.js

require('events').EventEmitter.prototype._maxListeners = 100;
require('events').EventEmitter.defaultMaxListeners = 100;
process.setMaxListeners(100)
const {
    Worker, isMainThread, parentPort, workerData
} = require('worker_threads');

intervalCounter = {}

let instance = 0

const clearMainThread = (thisInstance) => {
    clearInterval(intervalCounter[thisInstance])
    console.log(`Stopped thread instance ${thisInstance}`)
}

const threads = () => {
    return new Promise((resolve, reject) => {
        const thisInstance = ++instance

        intervalCounter[thisInstance] = setInterval(() => {
            console.log(`${new Date().toLocaleString()}: Instance ${thisInstance}`)
        }, 1000)


        const worker = new Worker(`${__dirname}/thread.js`, {
            workerData: {
                instance: thisInstance
            },
        });
        worker.on('message', (data) => {
            clearMainThread(thisInstance)
            console.log(data)
            resolve(data)
        });
        worker.on('error', (err) => {
            clearMainThread(thisInstance)
            console.log(err)
            reject(err)
        });
        worker.on('exit', (code) => {
            clearMainThread(thisInstance)
            if (code !== 0)
                reject(new Error(`Worker stopped with exit code ${code}`));
        })
    })
}

const threadCounts = 20
for (let threadCount = 0; threadCount < threadCounts; threadCount++) {
    console.log(`Spawning thread ${instance + 1}`)
    threads()
}

thread.js

require('events').EventEmitter.prototype._maxListeners = 100;
require('events').EventEmitter.defaultMaxListeners = 100;
process.setMaxListeners(100)
const {
    Worker, isMainThread, parentPort, workerData
} = require('worker_threads');

function fib(n) {
    //  console.log(`Count instance ${workerData.instance}: fib(${n}) `)
    if (n > 1) {
        return fib(n - 1) + fib(n - 2)
    } else {
        return n;
    }
}
const fibResult = fib(40)
//console.log(`This is the from thread: ${fibResult}`)
parentPort.postMessage(`This is the from main thread instance ${workerData.instance}: ${fibResult}`);

What is weird is if i set setInterval for pinging from 1000 to 2000 ms it shows now leak

C:\Users\patrikx3\Projects\patrikx3\play\scripts\worker-thread>node-thread main.js
Spawning thread 1
Spawning thread 2
Spawning thread 3
Spawning thread 4
Spawning thread 5
Spawning thread 6
Spawning thread 7
Spawning thread 8
Spawning thread 9
Spawning thread 10
Spawning thread 11
Spawning thread 12
Spawning thread 13
Spawning thread 14
Spawning thread 15
Spawning thread 16
Spawning thread 17
Spawning thread 18
Spawning thread 19
Spawning thread 20
2018-6-27 08:26:26: Instance 1
2018-6-27 08:26:28: Instance 2
2018-6-27 08:26:29: Instance 3
2018-6-27 08:26:30: Instance 4
2018-6-27 08:26:31: Instance 5
2018-6-27 08:26:32: Instance 6
2018-6-27 08:26:33: Instance 7
2018-6-27 08:26:34: Instance 8
2018-6-27 08:26:35: Instance 9
2018-6-27 08:26:36: Instance 10
2018-6-27 08:26:36: Instance 11
2018-6-27 08:26:37: Instance 12
2018-6-27 08:26:37: Instance 13
2018-6-27 08:26:38: Instance 14
2018-6-27 08:26:38: Instance 15
2018-6-27 08:26:39: Instance 16
2018-6-27 08:26:39: Instance 17
2018-6-27 08:26:39: Instance 18
2018-6-27 08:26:40: Instance 19
2018-6-27 08:26:40: Instance 20
2018-6-27 08:26:40: Instance 1
2018-6-27 08:26:40: Instance 2
2018-6-27 08:26:40: Instance 3
2018-6-27 08:26:40: Instance 4
2018-6-27 08:26:40: Instance 5
2018-6-27 08:26:40: Instance 6
2018-6-27 08:26:40: Instance 7
2018-6-27 08:26:40: Instance 8
2018-6-27 08:26:40: Instance 9
2018-6-27 08:26:40: Instance 10
2018-6-27 08:26:40: Instance 11
2018-6-27 08:26:40: Instance 12
2018-6-27 08:26:40: Instance 13
This is the from main thread instance 1: 102334155
This is the from main thread instance 2: 102334155
This is the from main thread instance 3: 102334155
This is the from main thread instance 5: 102334155
This is the from main thread instance 4: 102334155
This is the from main thread instance 6: 102334155
This is the from main thread instance 7: 102334155
This is the from main thread instance 8: 102334155
This is the from main thread instance 9: 102334155
This is the from main thread instance 10: 102334155
This is the from main thread instance 11: 102334155
This is the from main thread instance 12: 102334155
This is the from main thread instance 13: 102334155
This is the from main thread instance 14: 102334155
This is the from main thread instance 15: 102334155
This is the from main thread instance 16: 102334155
This is the from main thread instance 17: 102334155
This is the from main thread instance 18: 102334155
This is the from main thread instance 19: 102334155
This is the from main thread instance 20: 102334155

p3x-robot commented Jun 27, 2018

Guys, do you know how can fix this error?

memory leak detected

I use like this:

require('events').EventEmitter.prototype._maxListeners = 100;
require('events').EventEmitter.defaultMaxListeners = 100;
process.setMaxListeners(100)

In both worker and main thread as well.

C:\Users\patrikx3\Projects\patrikx3\play\scripts\worker-thread>node-thread main.js
Spawning thread 1
Spawning thread 2
Spawning thread 3
Spawning thread 4
Spawning thread 5
Spawning thread 6
Spawning thread 7
Spawning thread 8
Spawning thread 9
Spawning thread 10
Spawning thread 11
Spawning thread 12
Spawning thread 13
Spawning thread 14
Spawning thread 15
Spawning thread 16
Spawning thread 17
Spawning thread 18
Spawning thread 19
Spawning thread 20
(node:9816) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added. Use emitter.setMaxListeners() to increase limit
(node:9816) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 21 error listeners added. Use emitter.setMaxListeners() to increase limit
2018-6-27 08:19:11: Instance 1
2018-6-27 08:19:12: Instance 2
2018-6-27 08:19:13: Instance 3
2018-6-27 08:19:13: Instance 4
2018-6-27 08:19:14: Instance 5
2018-6-27 08:19:15: Instance 6
2018-6-27 08:19:15: Instance 7
2018-6-27 08:19:16: Instance 8
2018-6-27 08:19:17: Instance 9
2018-6-27 08:19:17: Instance 10
2018-6-27 08:19:18: Instance 11
2018-6-27 08:19:19: Instance 12
2018-6-27 08:19:20: Instance 13
2018-6-27 08:19:20: Instance 14
2018-6-27 08:19:21: Instance 15
2018-6-27 08:19:22: Instance 16
2018-6-27 08:19:22: Instance 17
2018-6-27 08:19:23: Instance 18
2018-6-27 08:19:23: Instance 19
2018-6-27 08:19:24: Instance 20
Stopped thread instance 1
This is the from main thread instance 1: 102334155
Stopped thread instance 1
Stopped thread instance 2
This is the from main thread instance 2: 102334155
Stopped thread instance 2
Stopped thread instance 3
This is the from main thread instance 3: 102334155
Stopped thread instance 3
Stopped thread instance 4
This is the from main thread instance 4: 102334155
Stopped thread instance 4
Stopped thread instance 5
This is the from main thread instance 5: 102334155
Stopped thread instance 5
2018-6-27 08:19:28: Instance 6
2018-6-27 08:19:28: Instance 7
2018-6-27 08:19:28: Instance 8
2018-6-27 08:19:28: Instance 9
2018-6-27 08:19:28: Instance 10
2018-6-27 08:19:28: Instance 11
2018-6-27 08:19:28: Instance 12
2018-6-27 08:19:28: Instance 13
2018-6-27 08:19:28: Instance 14
2018-6-27 08:19:28: Instance 15
2018-6-27 08:19:28: Instance 16
2018-6-27 08:19:28: Instance 17
2018-6-27 08:19:28: Instance 18
2018-6-27 08:19:28: Instance 19
2018-6-27 08:19:28: Instance 20
Stopped thread instance 6
This is the from main thread instance 6: 102334155
Stopped thread instance 6
Stopped thread instance 7
This is the from main thread instance 7: 102334155
Stopped thread instance 7
Stopped thread instance 8
This is the from main thread instance 8: 102334155
Stopped thread instance 8
Stopped thread instance 9
This is the from main thread instance 9: 102334155
Stopped thread instance 9
Stopped thread instance 10
This is the from main thread instance 10: 102334155
Stopped thread instance 10
Stopped thread instance 11
This is the from main thread instance 11: 102334155
Stopped thread instance 11
Stopped thread instance 12
This is the from main thread instance 12: 102334155
Stopped thread instance 12
Stopped thread instance 13
This is the from main thread instance 13: 102334155
Stopped thread instance 13
Stopped thread instance 14
This is the from main thread instance 14: 102334155
Stopped thread instance 14
Stopped thread instance 15
This is the from main thread instance 15: 102334155
Stopped thread instance 15
Stopped thread instance 16
This is the from main thread instance 16: 102334155
Stopped thread instance 16
Stopped thread instance 17
This is the from main thread instance 17: 102334155
Stopped thread instance 17
Stopped thread instance 18
This is the from main thread instance 18: 102334155
Stopped thread instance 18
Stopped thread instance 19
This is the from main thread instance 19: 102334155
Stopped thread instance 19
Stopped thread instance 20
This is the from main thread instance 20: 102334155
Stopped thread instance 20

#The code

main.js

require('events').EventEmitter.prototype._maxListeners = 100;
require('events').EventEmitter.defaultMaxListeners = 100;
process.setMaxListeners(100)
const {
    Worker, isMainThread, parentPort, workerData
} = require('worker_threads');

intervalCounter = {}

let instance = 0

const clearMainThread = (thisInstance) => {
    clearInterval(intervalCounter[thisInstance])
    console.log(`Stopped thread instance ${thisInstance}`)
}

const threads = () => {
    return new Promise((resolve, reject) => {
        const thisInstance = ++instance

        intervalCounter[thisInstance] = setInterval(() => {
            console.log(`${new Date().toLocaleString()}: Instance ${thisInstance}`)
        }, 1000)


        const worker = new Worker(`${__dirname}/thread.js`, {
            workerData: {
                instance: thisInstance
            },
        });
        worker.on('message', (data) => {
            clearMainThread(thisInstance)
            console.log(data)
            resolve(data)
        });
        worker.on('error', (err) => {
            clearMainThread(thisInstance)
            console.log(err)
            reject(err)
        });
        worker.on('exit', (code) => {
            clearMainThread(thisInstance)
            if (code !== 0)
                reject(new Error(`Worker stopped with exit code ${code}`));
        })
    })
}

const threadCounts = 20
for (let threadCount = 0; threadCount < threadCounts; threadCount++) {
    console.log(`Spawning thread ${instance + 1}`)
    threads()
}

thread.js

require('events').EventEmitter.prototype._maxListeners = 100;
require('events').EventEmitter.defaultMaxListeners = 100;
process.setMaxListeners(100)
const {
    Worker, isMainThread, parentPort, workerData
} = require('worker_threads');

function fib(n) {
    //  console.log(`Count instance ${workerData.instance}: fib(${n}) `)
    if (n > 1) {
        return fib(n - 1) + fib(n - 2)
    } else {
        return n;
    }
}
const fibResult = fib(40)
//console.log(`This is the from thread: ${fibResult}`)
parentPort.postMessage(`This is the from main thread instance ${workerData.instance}: ${fibResult}`);

What is weird is if i set setInterval for pinging from 1000 to 2000 ms it shows now leak

C:\Users\patrikx3\Projects\patrikx3\play\scripts\worker-thread>node-thread main.js
Spawning thread 1
Spawning thread 2
Spawning thread 3
Spawning thread 4
Spawning thread 5
Spawning thread 6
Spawning thread 7
Spawning thread 8
Spawning thread 9
Spawning thread 10
Spawning thread 11
Spawning thread 12
Spawning thread 13
Spawning thread 14
Spawning thread 15
Spawning thread 16
Spawning thread 17
Spawning thread 18
Spawning thread 19
Spawning thread 20
2018-6-27 08:26:26: Instance 1
2018-6-27 08:26:28: Instance 2
2018-6-27 08:26:29: Instance 3
2018-6-27 08:26:30: Instance 4
2018-6-27 08:26:31: Instance 5
2018-6-27 08:26:32: Instance 6
2018-6-27 08:26:33: Instance 7
2018-6-27 08:26:34: Instance 8
2018-6-27 08:26:35: Instance 9
2018-6-27 08:26:36: Instance 10
2018-6-27 08:26:36: Instance 11
2018-6-27 08:26:37: Instance 12
2018-6-27 08:26:37: Instance 13
2018-6-27 08:26:38: Instance 14
2018-6-27 08:26:38: Instance 15
2018-6-27 08:26:39: Instance 16
2018-6-27 08:26:39: Instance 17
2018-6-27 08:26:39: Instance 18
2018-6-27 08:26:40: Instance 19
2018-6-27 08:26:40: Instance 20
2018-6-27 08:26:40: Instance 1
2018-6-27 08:26:40: Instance 2
2018-6-27 08:26:40: Instance 3
2018-6-27 08:26:40: Instance 4
2018-6-27 08:26:40: Instance 5
2018-6-27 08:26:40: Instance 6
2018-6-27 08:26:40: Instance 7
2018-6-27 08:26:40: Instance 8
2018-6-27 08:26:40: Instance 9
2018-6-27 08:26:40: Instance 10
2018-6-27 08:26:40: Instance 11
2018-6-27 08:26:40: Instance 12
2018-6-27 08:26:40: Instance 13
This is the from main thread instance 1: 102334155
This is the from main thread instance 2: 102334155
This is the from main thread instance 3: 102334155
This is the from main thread instance 5: 102334155
This is the from main thread instance 4: 102334155
This is the from main thread instance 6: 102334155
This is the from main thread instance 7: 102334155
This is the from main thread instance 8: 102334155
This is the from main thread instance 9: 102334155
This is the from main thread instance 10: 102334155
This is the from main thread instance 11: 102334155
This is the from main thread instance 12: 102334155
This is the from main thread instance 13: 102334155
This is the from main thread instance 14: 102334155
This is the from main thread instance 15: 102334155
This is the from main thread instance 16: 102334155
This is the from main thread instance 17: 102334155
This is the from main thread instance 18: 102334155
This is the from main thread instance 19: 102334155
This is the from main thread instance 20: 102334155
@p3x-robot

This comment has been minimized.

Show comment
Hide comment
@p3x-robot

p3x-robot Jun 27, 2018

i test a lot but about 50 threads it freezes the windows.
I think is threads are still bad, I still prefer cluster, processes....
Blocks the whole system, maybe i am on Linux it might be better.
do i was using fibonacci(50) which is huge but yes using all cores in one process.

I think the Java and C# people wil understand threads are kaka...

p3x-robot commented Jun 27, 2018

i test a lot but about 50 threads it freezes the windows.
I think is threads are still bad, I still prefer cluster, processes....
Blocks the whole system, maybe i am on Linux it might be better.
do i was using fibonacci(50) which is huge but yes using all cores in one process.

I think the Java and C# people wil understand threads are kaka...

@p3x-robot

This comment has been minimized.

Show comment
Hide comment
@p3x-robot

p3x-robot Jun 27, 2018

i have tested many benchmarks, but the threads for computing are faster 🥇 . not faster for many threads but the memory usage is smaller, so threads are totally vialable. 💯

i have tested many benchmarks, but the threads for computing are faster 🥇 . not faster for many threads but the memory usage is smaller, so threads are totally vialable. 💯

@larshp larshp referenced this pull request in larshp/abaplint Jun 27, 2018

Open

node, threads vs cluster #216

@idibidiart

This comment has been minimized.

Show comment
Hide comment
@idibidiart

idibidiart Jun 28, 2018

As the author noted, this is work in progress and still in experimental stage ... you can certainly have more threads than much heavier nodejs processes/instances on any given machine... threads give you shared memory with its pros and cons... without threads, nodejs is designed for I/O bound work like fetching something from db... with threads it can do cpu bound work (potentially in parallel if you design it that way) outside the main thread so node can remain responsive to I/O, while processing in background... but even threads are limited by the number of cores and memory available ...

idibidiart commented Jun 28, 2018

As the author noted, this is work in progress and still in experimental stage ... you can certainly have more threads than much heavier nodejs processes/instances on any given machine... threads give you shared memory with its pros and cons... without threads, nodejs is designed for I/O bound work like fetching something from db... with threads it can do cpu bound work (potentially in parallel if you design it that way) outside the main thread so node can remain responsive to I/O, while processing in background... but even threads are limited by the number of cores and memory available ...

@7rulnik 7rulnik referenced this pull request in webpack-contrib/thread-loader Jul 1, 2018

Open

WIP: Native workers #29

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment