Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

child_process,cluster: allow using V8 serialization API #30162

Closed
wants to merge 8 commits into from

Conversation

@addaleax
Copy link
Member

addaleax commented Oct 29, 2019

Add an serialization option that allows child process IPC to
use the (typically more powerful) V8 serialization API.

Fixes: #10965

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
@abenhamdine

This comment has been minimized.

Copy link

abenhamdine commented Oct 29, 2019

Is there any perf gain/loss to expect from this change regarding IPC usage ?

@addaleax

This comment has been minimized.

Copy link
Member Author

addaleax commented Oct 29, 2019

@abenhamdine Here’s an example run of the benchmark:

$ ./node benchmark/cluster/echo.js sendsPerBroadcast=1 n=10000
cluster/echo.js n=10000 advancedSerialization="yes" sendsPerBroadcast=1 payload="string" workers=1: 12,333.296854319007
cluster/echo.js n=10000 advancedSerialization="no" sendsPerBroadcast=1 payload="string" workers=1: 19,558.63935318328
cluster/echo.js n=10000 advancedSerialization="yes" sendsPerBroadcast=1 payload="object" workers=1: 10,631.412279837206
cluster/echo.js n=10000 advancedSerialization="no" sendsPerBroadcast=1 payload="object" workers=1: 12,626.317659152222

It’s not very representative, but I generally think that this mechanism is still a bit slower than JSON when used on simple objects. I think for larger/more complex payloads, there could be an overall speedup, but I can’t really guarantee that without benchmarking more.

@devsnek

This comment has been minimized.

Copy link
Member

devsnek commented Oct 29, 2019

i'd guess on small objects the performance difference is from not having the same callsite optimization as JSON since we have to go through c++.

@abenhamdine

This comment has been minimized.

Copy link

abenhamdine commented Oct 29, 2019

@abenhamdine Here’s an example run of the benchmark:

$ ./node benchmark/cluster/echo.js sendsPerBroadcast=1 n=10000
cluster/echo.js n=10000 advancedSerialization="yes" sendsPerBroadcast=1 payload="string" workers=1: 12,333.296854319007
cluster/echo.js n=10000 advancedSerialization="no" sendsPerBroadcast=1 payload="string" workers=1: 19,558.63935318328
cluster/echo.js n=10000 advancedSerialization="yes" sendsPerBroadcast=1 payload="object" workers=1: 10,631.412279837206
cluster/echo.js n=10000 advancedSerialization="no" sendsPerBroadcast=1 payload="object" workers=1: 12,626.317659152222

It’s not very representative, but I generally think that this mechanism is still a bit slower than JSON when used on simple objects. I think for larger/more complex payloads, there could be an overall speedup, but I can’t really guarantee that without benchmarking more.

Thx you for this precise response.
It's useful for end users to have such data points, notably when considering upgrades.

benchmark/cluster/echo.js Outdated Show resolved Hide resolved
@addaleax addaleax removed the lib / src label Oct 29, 2019
@nodejs-github-bot

This comment has been minimized.

doc/api/child_process.md Outdated Show resolved Hide resolved
doc/api/child_process.md Outdated Show resolved Hide resolved
doc/api/child_process.md Outdated Show resolved Hide resolved
@addaleax addaleax force-pushed the addaleax:child-process-v8-serialize branch from 2869e64 to 08abb25 Nov 1, 2019
Add an `serialization` option that allows child process IPC to
use the (typically more powerful) V8 serialization API.

Fixes: #10965
@addaleax addaleax force-pushed the addaleax:child-process-v8-serialize branch from 08abb25 to 2c88105 Nov 1, 2019
@nodejs-github-bot

This comment has been minimized.

doc/api/process.md Outdated Show resolved Hide resolved
doc/api/process.md Outdated Show resolved Hide resolved
doc/api/cluster.md Outdated Show resolved Hide resolved
doc/api/cluster.md Outdated Show resolved Hide resolved
doc/api/child_process.md Outdated Show resolved Hide resolved
doc/api/child_process.md Outdated Show resolved Hide resolved
addaleax and others added 5 commits Nov 2, 2019
Co-Authored-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Co-Authored-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Co-Authored-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Co-Authored-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
doc/api/child_process.md Outdated Show resolved Hide resolved
@targos
targos approved these changes Nov 2, 2019
doc/api/child_process.md Outdated Show resolved Hide resolved
Co-Authored-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
@nodejs-github-bot

This comment has been minimized.

@nodejs-github-bot

This comment has been minimized.

@addaleax

This comment has been minimized.

Copy link
Member Author

addaleax commented Nov 5, 2019

Landed in 973f324

@addaleax addaleax closed this Nov 5, 2019
addaleax added a commit that referenced this pull request Nov 5, 2019
Add an `serialization` option that allows child process IPC to
use the (typically more powerful) V8 serialization API.

Fixes: #10965

PR-URL: #30162
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Gus Caplan <me@gus.host>
Reviewed-By: David Carlier <devnexen@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
@addaleax addaleax deleted the addaleax:child-process-v8-serialize branch Nov 5, 2019
MylesBorins added a commit that referenced this pull request Nov 17, 2019
Add an `serialization` option that allows child process IPC to
use the (typically more powerful) V8 serialization API.

Fixes: #10965

PR-URL: #30162
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Gus Caplan <me@gus.host>
Reviewed-By: David Carlier <devnexen@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
BridgeAR added a commit to BridgeAR/node that referenced this pull request Nov 19, 2019
Notable changes:

* addons:
  * Deprecate one- and two-argument `AtExit()`. Use the three-argument
    variant of `AtExit()` or `AddEnvironmentCleanupHook()` instead
    (Anna Henningsen) nodejs#30227
* child_process,cluster:
  * The `serialization` option is added that allows child process
    IPC to use the V8 serialization API (to e.g., pass through data
    types like sets or maps) (Anna Henningsen)
    nodejs#30162
* embedder:
  * Exposes the ability to pass cli flags / options through an API
    as embedder (Shelley Vohr)
    nodejs#30466
  * Allow adding linked bindings to Environment (Anna Henningsen)
    nodejs#30274
* stream:
  * Add `writable.writableCorked` property (Robert Nagy)
    nodejs#29012
* worker:
  * Allow specifying resource limits (Anna Henningsen)
    nodejs#26628
BridgeAR added a commit to BridgeAR/node that referenced this pull request Nov 19, 2019
Notable changes:

* addons:
  * Deprecate one- and two-argument `AtExit()`. Use the three-argument
    variant of `AtExit()` or `AddEnvironmentCleanupHook()` instead
    (Anna Henningsen) nodejs#30227
* child_process,cluster:
  * The `serialization` option is added that allows child process
    IPC to use the V8 serialization API (to e.g., pass through data
    types like sets or maps) (Anna Henningsen)
    nodejs#30162
* embedder:
  * Exposes the ability to pass cli flags / options through an API
    as embedder (Shelley Vohr)
    nodejs#30466
  * Allow adding linked bindings to Environment (Anna Henningsen)
    nodejs#30274
* esm:
  * Unflag --experimental-modules (Guy Bedford)
    nodejs#29866
* stream:
  * Add `writable.writableCorked` property (Robert Nagy)
    nodejs#29012
* worker:
  * Allow specifying resource limits (Anna Henningsen)
    nodejs#26628
BridgeAR added a commit to BridgeAR/node that referenced this pull request Nov 19, 2019
Notable changes:

* addons:
  * Deprecate one- and two-argument `AtExit()`. Use the three-argument
    variant of `AtExit()` or `AddEnvironmentCleanupHook()` instead
    (Anna Henningsen) nodejs#30227
* child_process,cluster:
  * The `serialization` option is added that allows child process
    IPC to use the V8 serialization API (to e.g., pass through data
    types like sets or maps) (Anna Henningsen)
    nodejs#30162
* embedder:
  * Exposes the ability to pass cli flags / options through an API
    as embedder (Shelley Vohr)
    nodejs#30466
  * Allow adding linked bindings to Environment (Anna Henningsen)
    nodejs#30274
* esm:
  * Unflag --experimental-modules (Guy Bedford)
    nodejs#29866
* stream:
  * Add `writable.writableCorked` property (Robert Nagy)
    nodejs#29012
* worker:
  * Allow specifying resource limits (Anna Henningsen)
    nodejs#26628
@BridgeAR BridgeAR mentioned this pull request Nov 19, 2019
BridgeAR added a commit to BridgeAR/node that referenced this pull request Nov 19, 2019
Notable changes:

* addons:
  * Deprecate one- and two-argument `AtExit()`. Use the three-argument
    variant of `AtExit()` or `AddEnvironmentCleanupHook()` instead
    (Anna Henningsen) nodejs#30227
* child_process,cluster:
  * The `serialization` option is added that allows child process
    IPC to use the V8 serialization API (to e.g., pass through data
    types like sets or maps) (Anna Henningsen)
    nodejs#30162
* deps:
  * Update `npm` to 6.13.0 (Ruy Adorno)
    nodejs#30271
* embedder:
  * Exposes the ability to pass cli flags / options through an API
    as embedder (Shelley Vohr)
    nodejs#30466
  * Allow adding linked bindings to Environment (Anna Henningsen)
    nodejs#30274
* esm:
  * Unflag --experimental-modules (Guy Bedford)
    nodejs#29866
* stream:
  * Add `writable.writableCorked` property (Robert Nagy)
    nodejs#29012
* worker:
  * Allow specifying resource limits (Anna Henningsen)
    nodejs#26628
* v8:
  * The Serialization API is now stable (Anna Henningsen)
    nodejs#30234
MylesBorins added a commit to BridgeAR/node that referenced this pull request Nov 21, 2019
Notable changes:

* addons:
  * Deprecate one- and two-argument `AtExit()`. Use the three-argument
    variant of `AtExit()` or `AddEnvironmentCleanupHook()` instead
    (Anna Henningsen) nodejs#30227
* child_process,cluster:
  * The `serialization` option is added that allows child process
    IPC to use the V8 serialization API (to e.g., pass through data
    types like sets or maps) (Anna Henningsen)
    nodejs#30162
* deps:
  * Update `npm` to 6.13.0 (Ruy Adorno)
    nodejs#30271
* embedder:
  * Exposes the ability to pass cli flags / options through an API
    as embedder (Shelley Vohr)
    nodejs#30466
  * Allow adding linked bindings to Environment (Anna Henningsen)
    nodejs#30274
* esm:
  * Unflag --experimental-modules (Guy Bedford)
    nodejs#29866
* stream:
  * Add `writable.writableCorked` property (Robert Nagy)
    nodejs#29012
* worker:
  * Allow specifying resource limits (Anna Henningsen)
    nodejs#26628
* v8:
  * The Serialization API is now stable (Anna Henningsen)
    nodejs#30234
MylesBorins added a commit to BridgeAR/node that referenced this pull request Nov 21, 2019
Notable changes:

* addons:
  * Deprecate one- and two-argument `AtExit()`. Use the three-argument
    variant of `AtExit()` or `AddEnvironmentCleanupHook()` instead
    (Anna Henningsen) nodejs#30227
* child_process,cluster:
  * The `serialization` option is added that allows child process
    IPC to use the V8 serialization API (to e.g., pass through data
    types like sets or maps) (Anna Henningsen)
    nodejs#30162
* deps:
  * Update `npm` to 6.13.0 (Ruy Adorno)
    nodejs#30271
* embedder:
  * Exposes the ability to pass cli flags / options through an API
    as embedder (Shelley Vohr)
    nodejs#30466
  * Allow adding linked bindings to Environment (Anna Henningsen)
    nodejs#30274
* esm:
  * Unflag --experimental-modules (Guy Bedford)
    nodejs#29866
* stream:
  * Add `writable.writableCorked` property (Robert Nagy)
    nodejs#29012
* worker:
  * Allow specifying resource limits (Anna Henningsen)
    nodejs#26628
* v8:
  * The Serialization API is now stable (Anna Henningsen)
    nodejs#30234
MylesBorins added a commit to BridgeAR/node that referenced this pull request Nov 21, 2019
Notable changes:

* addons:
  * Deprecate one- and two-argument `AtExit()`. Use the three-argument
    variant of `AtExit()` or `AddEnvironmentCleanupHook()` instead
    (Anna Henningsen) nodejs#30227
* child_process,cluster:
  * The `serialization` option is added that allows child process
    IPC to use the V8 serialization API (to e.g., pass through data
    types like sets or maps) (Anna Henningsen)
    nodejs#30162
* deps:
  * Update V8 to 7.9
  * Update `npm` to 6.13.0 (Ruy Adorno)
    nodejs#30271
* embedder:
  * Exposes the ability to pass cli flags / options through an API
    as embedder (Shelley Vohr)
    nodejs#30466
  * Allow adding linked bindings to Environment (Anna Henningsen)
    nodejs#30274
* esm:
  * Unflag --experimental-modules (Guy Bedford)
    nodejs#29866
* stream:
  * Add `writable.writableCorked` property (Robert Nagy)
    nodejs#29012
* worker:
  * Allow specifying resource limits (Anna Henningsen)
    nodejs#26628
* v8:
  * The Serialization API is now stable (Anna Henningsen)
    nodejs#30234
MylesBorins added a commit to BridgeAR/node that referenced this pull request Nov 21, 2019
Notable changes:

* addons:
  * Deprecate one- and two-argument `AtExit()`. Use the three-argument
    variant of `AtExit()` or `AddEnvironmentCleanupHook()` instead
    (Anna Henningsen) nodejs#30227
* child_process,cluster:
  * The `serialization` option is added that allows child process
    IPC to use the V8 serialization API (to e.g., pass through data
    types like sets or maps) (Anna Henningsen)
    nodejs#30162
* deps:
  * Update V8 to 7.9
  * Update `npm` to 6.13.0 (Ruy Adorno)
    nodejs#30271
* embedder:
  * Exposes the ability to pass cli flags / options through an API
    as embedder (Shelley Vohr)
    nodejs#30466
  * Allow adding linked bindings to Environment (Anna Henningsen)
    nodejs#30274
* esm:
  * Unflag --experimental-modules (Guy Bedford)
    nodejs#29866
* stream:
  * Add `writable.writableCorked` property (Robert Nagy)
    nodejs#29012
* worker:
  * Allow specifying resource limits (Anna Henningsen)
    nodejs#26628
* v8:
  * The Serialization API is now stable (Anna Henningsen)
    nodejs#30234
MylesBorins added a commit to BridgeAR/node that referenced this pull request Nov 21, 2019
Notable changes:

* addons:
  * Deprecate one- and two-argument `AtExit()`. Use the three-argument
    variant of `AtExit()` or `AddEnvironmentCleanupHook()` instead
    (Anna Henningsen) nodejs#30227
* child_process,cluster:
  * The `serialization` option is added that allows child process
    IPC to use the V8 serialization API (to e.g., pass through data
    types like sets or maps) (Anna Henningsen)
    nodejs#30162
* deps:
  * Update V8 to 7.9
  * Update `npm` to 6.13.0 (Ruy Adorno)
    nodejs#30271
* embedder:
  * Exposes the ability to pass cli flags / options through an API
    as embedder (Shelley Vohr)
    nodejs#30466
  * Allow adding linked bindings to Environment (Anna Henningsen)
    nodejs#30274
* esm:
  * Unflag --experimental-modules (Guy Bedford)
    nodejs#29866
* stream:
  * Add `writable.writableCorked` property (Robert Nagy)
    nodejs#29012
* worker:
  * Allow specifying resource limits (Anna Henningsen)
    nodejs#26628
* v8:
  * The Serialization API is now stable (Anna Henningsen)
    nodejs#30234
MylesBorins added a commit to BridgeAR/node that referenced this pull request Nov 21, 2019
Notable changes:

* addons:
  * Deprecate one- and two-argument `AtExit()`. Use the three-argument
    variant of `AtExit()` or `AddEnvironmentCleanupHook()` instead
    (Anna Henningsen) nodejs#30227
* child_process,cluster:
  * The `serialization` option is added that allows child process
    IPC to use the V8 serialization API (to e.g., pass through data
    types like sets or maps) (Anna Henningsen)
    nodejs#30162
* deps:
  * Update V8 to 7.9
  * Update `npm` to 6.13.0 (Ruy Adorno)
    nodejs#30271
* embedder:
  * Exposes the ability to pass cli flags / options through an API
    as embedder (Shelley Vohr)
    nodejs#30466
  * Allow adding linked bindings to Environment (Anna Henningsen)
    nodejs#30274
* esm:
  * Unflag --experimental-modules (Guy Bedford)
    nodejs#29866
* stream:
  * Add `writable.writableCorked` property (Robert Nagy)
    nodejs#29012
* worker:
  * Allow specifying resource limits (Anna Henningsen)
    nodejs#26628
* v8:
  * The Serialization API is now stable (Anna Henningsen)
    nodejs#30234
MylesBorins added a commit to BridgeAR/node that referenced this pull request Nov 21, 2019
Notable changes:

* addons:
  * Deprecate one- and two-argument `AtExit()`. Use the three-argument
    variant of `AtExit()` or `AddEnvironmentCleanupHook()` instead
    (Anna Henningsen) nodejs#30227
* child_process,cluster:
  * The `serialization` option is added that allows child process
    IPC to use the V8 serialization API (to e.g., pass through data
    types like sets or maps) (Anna Henningsen)
    nodejs#30162
* deps:
  * Update V8 to 7.9
  * Update `npm` to 6.13.0 (Ruy Adorno)
    nodejs#30271
* embedder:
  * Exposes the ability to pass cli flags / options through an API
    as embedder (Shelley Vohr)
    nodejs#30466
  * Allow adding linked bindings to Environment (Anna Henningsen)
    nodejs#30274
* esm:
  * Unflag --experimental-modules (Guy Bedford)
    nodejs#29866
* stream:
  * Add `writable.writableCorked` property (Robert Nagy)
    nodejs#29012
* worker:
  * Allow specifying resource limits (Anna Henningsen)
    nodejs#26628
* v8:
  * The Serialization API is now stable (Anna Henningsen)
    nodejs#30234

PR-URL: nodejs#30547
MylesBorins added a commit that referenced this pull request Nov 21, 2019
Notable changes:

* addons:
  * Deprecate one- and two-argument `AtExit()`. Use the three-argument
    variant of `AtExit()` or `AddEnvironmentCleanupHook()` instead
    (Anna Henningsen) #30227
* child_process,cluster:
  * The `serialization` option is added that allows child process
    IPC to use the V8 serialization API (to e.g., pass through data
    types like sets or maps) (Anna Henningsen)
    #30162
* deps:
  * Update V8 to 7.9
  * Update `npm` to 6.13.0 (Ruy Adorno)
    #30271
* embedder:
  * Exposes the ability to pass cli flags / options through an API
    as embedder (Shelley Vohr)
    #30466
  * Allow adding linked bindings to Environment (Anna Henningsen)
    #30274
* esm:
  * Unflag --experimental-modules (Guy Bedford)
    #29866
* stream:
  * Add `writable.writableCorked` property (Robert Nagy)
    #29012
* worker:
  * Allow specifying resource limits (Anna Henningsen)
    #26628
* v8:
  * The Serialization API is now stable (Anna Henningsen)
    #30234

PR-URL: #30547
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.