Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

src: unload native addons when the environment is destroyed #24861

Closed

Conversation

@gabrielschulhof
Copy link
Contributor

commented Dec 6, 2018

This is an alternative to #23319
which attaches the loaded addons to the environment and closes them
when the environment is destroyed.

It includes the test/addons/worker-addon,
test/addons/hello-world/test-worker.js and the documentation update from the original PR written
by @addaleax in a separate commit.

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
src/node_binding.cc Outdated Show resolved Hide resolved
src/node_binding.h Show resolved Hide resolved
src/env-inl.h Outdated Show resolved Hide resolved

@gabrielschulhof gabrielschulhof force-pushed the gabrielschulhof:unload-native-addon2 branch from bc80971 to 5e44c8b Dec 6, 2018

@gabrielschulhof

This comment has been minimized.

Copy link
Contributor Author

commented Dec 6, 2018

@joyeecheung I have addressed your review comments.

@bnoordhuis I have addressed your review comments. I put the body of DLib into a new node_binding-inl.h' which is included from the bottom of node_binding.h`.

@gabrielschulhof gabrielschulhof requested a review from addaleax Dec 6, 2018

src/node_binding-inl.h Outdated Show resolved Hide resolved
src/node_binding.cc Outdated Show resolved Hide resolved
addaleax and others added 3 commits Dec 6, 2018
src: unload addons when environment quits
This is an alternative to #23319
which attaches the loaded addons to the environment and closes them
when the environment is destroyed.

@gabrielschulhof gabrielschulhof force-pushed the gabrielschulhof:unload-native-addon2 branch from 5e44c8b to df7aa7a Dec 7, 2018

@addaleax
Copy link
Member

left a comment

Thank you! 💙

src/env.cc Outdated Show resolved Hide resolved
@addaleax addaleax referenced this pull request Dec 10, 2018
2 of 3 tasks complete

@gabrielschulhof gabrielschulhof force-pushed the gabrielschulhof:unload-native-addon2 branch from bbea5f8 to 584f643 Dec 11, 2018

@gabrielschulhof

This comment has been minimized.

Copy link
Contributor Author

commented Dec 11, 2018

@gabrielschulhof

This comment has been minimized.

Copy link
Contributor Author

commented Dec 11, 2018

Failure looks legit.

@gabrielschulhof

This comment has been minimized.

Copy link
Contributor Author

commented Dec 11, 2018

Resume, just to see if it's 100% reproducible:

https://ci.nodejs.org/job/node-test-pull-request/19427/

@addaleax

This comment has been minimized.

Copy link
Member

commented Dec 15, 2018

@addaleax addaleax added the worker label Dec 15, 2018

@Trott

This comment has been minimized.

Copy link
Member

commented Dec 16, 2018

Fresh CI rather than Resume Build so that it pulls in the fact that a bunch of tests are currently marked flaky: https://ci.nodejs.org/job/node-test-pull-request/19572/

@Trott

This comment has been minimized.

Copy link
Member

commented Dec 16, 2018

Any chance that this AIX failure is related?

16:58:17 not ok 2208 addons/at-exit/test
16:58:17   ---
16:58:17   duration_ms: 0.723
16:58:17   severity: crashed
16:58:17   exitcode: -4
16:58:17   stack: |-
16:58:17   ...

I'm guessing not, but you tell me. ¯\(ツ)

In the meantime, Resume Build CI: https://ci.nodejs.org/job/node-test-pull-request/19573/

@addaleax

This comment has been minimized.

Copy link
Member

commented Dec 16, 2018

@Trott I’m not sure whether we should wait for the outcome of the resume build (run stress tests if it comes back green?), but before/after stress tests with a hundred runs or so might be helpful?

@Trott

This comment has been minimized.

Copy link
Member

commented Dec 16, 2018

before/after stress tests with a hundred runs or so might be helpful?

I guess if we're going to tie up an AIX machine for a few hours doing a stress test, this is the time of day/week to do it, so here we go:

Stress test on master: https://ci.nodejs.org/job/node-stress-single-test/2128/

Stress test on this PR: https://ci.nodejs.org/job/node-stress-single-test/2129/

@Trott

This comment has been minimized.

Copy link
Member

commented Dec 16, 2018

CI Resume Build for AIX showed the same addons failure.

https://ci.nodejs.org/job/node-test-commit-aix/19762/nodes=aix61-ppc64/console

18:59:40 not ok 2208 addons/at-exit/test
18:59:40   ---
18:59:40   duration_ms: 0.918
18:59:40   severity: crashed
18:59:40   exitcode: -4
18:59:40   stack: |-
18:59:40   ...

Still awaiting stress test for this PR to start, but stress test on master is clean so far after almost 900 runs. (It's set to do 1000.)

@Trott
Copy link
Member

left a comment

Stress test is crashing on addons/at-exit/test 100% of the time after 400+ runs on AIX. Impressed/weirded-out that this issue is AIX only. But I guess it needs to be sorted out before landing...

20:22:34 not ok 1 addons/at-exit/test
20:22:34   ---
20:22:34   duration_ms: 0.573
20:22:34   severity: crashed
20:22:34   exitcode: -4
20:22:34   stack: |-
20:22:34   ...
@MylesBorins

This comment has been minimized.

Copy link
Member

commented Dec 26, 2018

This change broke the node-report module

It seems that is was somehow related to nan?

nodejs/node-report#116

Fix landed in nan and awaiting a release nodejs/nan#831

Would someone be willing to open a backport so we can track the progress?

@addaleax

This comment has been minimized.

Copy link
Member

commented Dec 26, 2018

This change broke the node-report module

@MylesBorins This seems very unlikely … can you double-check and maybe share how you tested this?

Fix landed in nan and awaiting a release nodejs/nan#831

That PR was opened before this one landed and the contents seem very much unrelated…

@targos targos added this to Backport requested in v11.x Dec 28, 2018

gabrielschulhof added a commit to gabrielschulhof/node that referenced this pull request Dec 28, 2018
test,doc: add tests and docs for addon unloading
Originally from portions of nodejs#23319.

PR-URL: nodejs#24861
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
gabrielschulhof added a commit to gabrielschulhof/node that referenced this pull request Dec 28, 2018
src: unload addons when environment quits
This is an alternative to nodejs#23319
which attaches the loaded addons to the environment and closes them
when the environment is destroyed.

PR-URL: nodejs#24861
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
@gabrielschulhof gabrielschulhof referenced this pull request Dec 28, 2018
4 of 4 tasks complete
targos added a commit that referenced this pull request Jan 1, 2019
test,doc: add tests and docs for addon unloading
Originally from portions of #23319.

Backport-PR-URL: #25258
PR-URL: #24861
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
targos added a commit that referenced this pull request Jan 1, 2019
src: unload addons when environment quits
This is an alternative to #23319
which attaches the loaded addons to the environment and closes them
when the environment is destroyed.

Backport-PR-URL: #25258
PR-URL: #24861
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

@BridgeAR BridgeAR moved this from Backport requested to Backported in v11.x Jan 10, 2019

refack added a commit to refack/node that referenced this pull request Jan 14, 2019
test,doc: add tests and docs for addon unloading
Originally from portions of nodejs#23319.

PR-URL: nodejs#24861
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
refack added a commit to refack/node that referenced this pull request Jan 14, 2019
src: unload addons when environment quits
This is an alternative to nodejs#23319
which attaches the loaded addons to the environment and closes them
when the environment is destroyed.

PR-URL: nodejs#24861
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
@BridgeAR BridgeAR referenced this pull request Jan 16, 2019
@BridgeAR

This comment has been minimized.

Copy link
Member

commented Jan 18, 2019

This has been backported and released in v11.7.0 but it turned out that this actually breaks node-report. Therefore I am going to remove the backport again and release v11.7.1 without this.
Please have another look at this so it does not cause any issues with node-report.

addaleax added a commit to addaleax/node that referenced this pull request Jan 18, 2019
src: restrict unloading addons to Worker threads
Unloading native addons from the main thread was an (presumably
unintended) significant breaking change, because addons may
rely on their memory being available after an `Environment` exits.

This patch only restricts this to Worker threads, at least for the
time being, and thus matches the behaviour from
nodejs#23319.

Refs: nodejs#24861
Refs: nodejs#23319
@addaleax addaleax referenced this pull request Jan 18, 2019
3 of 3 tasks complete
@addaleax

This comment has been minimized.

Copy link
Member

commented Jan 18, 2019

I’m sorry that I did not add a test case for the issue we’re seeing with node-report when I wrote the original PR at #23319 – I’ve opened #25577 to address it.

(This does not mean that this is not a bug in node-report, btw – it is. But there’s a lot of addons that run into similar issues with this PR, I assume.)

@richardlau

This comment has been minimized.

Copy link
Member

commented Jan 19, 2019

(This does not mean that this is not a bug in node-report, btw – it is. But there’s a lot of addons that run into similar issues with this PR, I assume.)

If it's something we can improve, please report/submit a PR over in https://github.com/nodejs/node-report. Note that one of the issues with addons and master is that nan requires an update to work with the version of V8 in master (AFAIK nodejs/nan#831 has yet to be released) which means addons currently fail in citgm, which is obviously masking stuff like this.

@addaleax

This comment has been minimized.

Copy link
Member

commented Jan 19, 2019

@richardlau I can try to do that, but given the work on integrating it into core in #22712, I don’t think I’d personally want to spend much time on the npm module… the first problem I encountered here is that nodereport_trigger_async is never removed from the event loop (e.g. in an Environment cleanup hook). Generally, in #22712 we had (and I still partly have) a hard time figuring out what lifetime management objects from node-report should have and how much global state is actually desirable and managable.

(Also, it’s absolutely understandable if you’re not interested in going through the nearly one thousand comments in #22712, but there’s a lot in there that applies to the npm module as well, including this issue iirc.)

addaleax added a commit to addaleax/node that referenced this pull request Jan 22, 2019
src: restrict unloading addons to Worker threads
Unloading native addons from the main thread was an (presumably
unintended) significant breaking change, because addons may
rely on their memory being available after an `Environment` exits.

This patch only restricts this to Worker threads, at least for the
time being, and thus matches the behaviour from
nodejs#23319.

Refs: nodejs#24861
Refs: nodejs#23319
danbev added a commit that referenced this pull request Jan 23, 2019
src: restrict unloading addons to Worker threads
Unloading native addons from the main thread was an (presumably
unintended) significant breaking change, because addons may
rely on their memory being available after an `Environment` exits.

This patch only restricts this to Worker threads, at least for the
time being, and thus matches the behaviour from
#23319.

PR-URL: #25577
Refs: #24861
Refs: #23319
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
addaleax added a commit that referenced this pull request Jan 23, 2019
src: restrict unloading addons to Worker threads
Unloading native addons from the main thread was an (presumably
unintended) significant breaking change, because addons may
rely on their memory being available after an `Environment` exits.

This patch only restricts this to Worker threads, at least for the
time being, and thus matches the behaviour from
#23319.

PR-URL: #25577
Refs: #24861
Refs: #23319
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
@MylesBorins MylesBorins referenced this pull request Jan 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
You can’t perform that action at this time.