Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

worker: use copy of process.env #26544

Closed
wants to merge 3 commits into from

Conversation

@addaleax
Copy link
Member

commented Mar 9, 2019

Instead of sharing the OS-backed store for all process.env instances,
create a copy of process.env for every worker that is created.

The copies do not interact. Native-addons do not see modifications to
process.env from Worker threads, but child processes started from
Workers do default to the Worker’s copy of process.env.

This makes Workers behave like child processes as far as process.env
is concerned, and an option corresponding to the child_process
module’s env option is added to the constructor.

Fixes: #24947

/cc @nodejs/workers

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines

@addaleax addaleax added the worker label Mar 9, 2019

Show resolved Hide resolved doc/api/worker_threads.md Outdated
Show resolved Hide resolved doc/api/worker_threads.md Outdated
@BridgeAR
Copy link
Member

left a comment

LG with the comments addressed.

Show resolved Hide resolved lib/internal/main/worker_thread.js Outdated
@addaleax

This comment has been minimized.

Copy link
Member Author

commented Mar 10, 2019

@ZYSzys @vsemozhetbyt @BridgeAR Your comments should be addressed, PTAL

Show resolved Hide resolved doc/api/worker_threads.md Outdated
@vsemozhetbyt
Copy link
Member

left a comment

Docs LGTM.

@BridgeAR

This comment has been minimized.

@ZYSzys

ZYSzys approved these changes Mar 10, 2019

Copy link
Member

left a comment

LGTM

BTW, don't we need to fix the lint ?

@addaleax

This comment has been minimized.

Copy link
Member Author

commented Mar 10, 2019

@ZYSzys

ZYSzys approved these changes Mar 11, 2019

Copy link
Member

left a comment

Still LGTM

@addaleax addaleax force-pushed the addaleax:worker-env branch from b9360a3 to 2176c6e Mar 12, 2019

@addaleax

This comment has been minimized.

Copy link
Member Author

commented Mar 12, 2019

I’ve updated this to address some test failures, but it’s still not complete because we check environment variables before we actually receive the startup message, and the whole pre-execution setup has become kind of icky to deal with recently…

@addaleax

This comment has been minimized.

Copy link
Member Author

commented Mar 12, 2019

Okay, I think I have everything fixed now (including having spent quite a bit of time on debugging until it turned out that object spread doesn’t work with Proxies before V8 7.4 🙁).

CI: https://ci.nodejs.org/job/node-test-pull-request/21485/

@benjamingr @jasnell @BridgeAR (and maybe also @joyeecheung, given the kind of code it touches) The new changes are in 4930221...9a4e61f can you take another look?

@addaleax

This comment has been minimized.

Copy link
Member Author

commented Mar 13, 2019

@Trott I can’t make much of the test failures with --worker … they don’t happen locally for me, but it looks like they might be due to some kind of issue with common.PORT that’s specific to the CI setup?

@Trott

This comment has been minimized.

Copy link
Member

commented Mar 13, 2019

@Trott I can’t make much of the test failures with --worker … they don’t happen locally for me, but it looks like they might be due to some kind of issue with common.PORT that’s specific to the CI setup?

Doesn't fail locally for me either. @nodejs/build Is there something unusual about the custom-suites-freestyle job that might provide a hint here?

Show resolved Hide resolved lib/internal/worker.js
Show resolved Hide resolved src/node_credentials.cc Outdated
@addaleax

This comment has been minimized.

Copy link
Member Author

commented Mar 13, 2019

@joyeecheung I think I’ll try the key-value-store based solution again. That should resolve your concerns, and if you want to review it, great ;)

@addaleax addaleax force-pushed the addaleax:worker-env branch from 9a4e61f to 480e9bf Mar 13, 2019

addaleax added some commits Mar 13, 2019

src: allow per-Environment set of env vars
Abstract the `process.env` backing mechanism in C++ to allow
different kinds of backing stores for `process.env` for different
Environments.
src: implement generic backend for process.env
Allow a generic string-based backing store, with no significance
to the remainder of the process, as a store for `process.env`.
worker: use copy of process.env
Instead of sharing the OS-backed store for all `process.env` instances,
create a copy of `process.env` for every worker that is created.

The copies do not interact. Native-addons do not see modifications to
`process.env` from Worker threads, but child processes started from
Workers do default to the Worker’s copy of `process.env`.

This makes Workers behave like child processes as far as `process.env`
is concerned, and an option corresponding to the `child_process`
module’s `env` option is added to the constructor.

Fixes: #24947

@addaleax addaleax force-pushed the addaleax:worker-env branch from 721b0d8 to 838799b Mar 30, 2019

@nodejs-github-bot

This comment has been minimized.

@addaleax

This comment has been minimized.

Copy link
Member Author

commented Mar 30, 2019

Landed in e4e2b0c...9fbf0c6

@addaleax addaleax closed this Mar 30, 2019

@addaleax addaleax deleted the addaleax:worker-env branch Mar 30, 2019

addaleax added a commit that referenced this pull request Mar 30, 2019

src: allow per-Environment set of env vars
Abstract the `process.env` backing mechanism in C++ to allow
different kinds of backing stores for `process.env` for different
Environments.

PR-URL: #26544
Fixes: #24947
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

addaleax added a commit that referenced this pull request Mar 30, 2019

src: implement generic backend for process.env
Allow a generic string-based backing store, with no significance
to the remainder of the process, as a store for `process.env`.

PR-URL: #26544
Fixes: #24947
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

addaleax added a commit that referenced this pull request Mar 30, 2019

worker: use copy of process.env
Instead of sharing the OS-backed store for all `process.env` instances,
create a copy of `process.env` for every worker that is created.

The copies do not interact. Native-addons do not see modifications to
`process.env` from Worker threads, but child processes started from
Workers do default to the Worker’s copy of `process.env`.

This makes Workers behave like child processes as far as `process.env`
is concerned, and an option corresponding to the `child_process`
module’s `env` option is added to the constructor.

Fixes: #24947

PR-URL: #26544
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

@joyeecheung joyeecheung referenced this pull request Apr 1, 2019

Closed

src: port coverage serialization to C++ #26874

2 of 4 tasks complete

chjj added a commit to chjj/bthreads that referenced this pull request Apr 1, 2019

BethGriggs added a commit that referenced this pull request Apr 5, 2019

src: allow per-Environment set of env vars
Abstract the `process.env` backing mechanism in C++ to allow
different kinds of backing stores for `process.env` for different
Environments.

PR-URL: #26544
Fixes: #24947
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

BethGriggs added a commit that referenced this pull request Apr 5, 2019

src: implement generic backend for process.env
Allow a generic string-based backing store, with no significance
to the remainder of the process, as a store for `process.env`.

PR-URL: #26544
Fixes: #24947
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

BethGriggs added a commit that referenced this pull request Apr 5, 2019

worker: use copy of process.env
Instead of sharing the OS-backed store for all `process.env` instances,
create a copy of `process.env` for every worker that is created.

The copies do not interact. Native-addons do not see modifications to
`process.env` from Worker threads, but child processes started from
Workers do default to the Worker’s copy of `process.env`.

This makes Workers behave like child processes as far as `process.env`
is concerned, and an option corresponding to the `child_process`
module’s `env` option is added to the constructor.

Fixes: #24947

PR-URL: #26544
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

BethGriggs added a commit that referenced this pull request Apr 9, 2019

src: allow per-Environment set of env vars
Abstract the `process.env` backing mechanism in C++ to allow
different kinds of backing stores for `process.env` for different
Environments.

PR-URL: #26544
Fixes: #24947
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Signed-off-by: Beth Griggs <Bethany.Griggs@uk.ibm.com>

BethGriggs added a commit that referenced this pull request Apr 9, 2019

src: implement generic backend for process.env
Allow a generic string-based backing store, with no significance
to the remainder of the process, as a store for `process.env`.

PR-URL: #26544
Fixes: #24947
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Signed-off-by: Beth Griggs <Bethany.Griggs@uk.ibm.com>

BethGriggs added a commit that referenced this pull request Apr 9, 2019

worker: use copy of process.env
Instead of sharing the OS-backed store for all `process.env` instances,
create a copy of `process.env` for every worker that is created.

The copies do not interact. Native-addons do not see modifications to
`process.env` from Worker threads, but child processes started from
Workers do default to the Worker’s copy of `process.env`.

This makes Workers behave like child processes as far as `process.env`
is concerned, and an option corresponding to the `child_process`
module’s `env` option is added to the constructor.

Fixes: #24947

PR-URL: #26544
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Signed-off-by: Beth Griggs <Bethany.Griggs@uk.ibm.com>

BethGriggs added a commit that referenced this pull request Apr 9, 2019

src: allow per-Environment set of env vars
Abstract the `process.env` backing mechanism in C++ to allow
different kinds of backing stores for `process.env` for different
Environments.

PR-URL: #26544
Fixes: #24947
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Signed-off-by: Beth Griggs <Bethany.Griggs@uk.ibm.com>

BethGriggs added a commit that referenced this pull request Apr 9, 2019

src: implement generic backend for process.env
Allow a generic string-based backing store, with no significance
to the remainder of the process, as a store for `process.env`.

PR-URL: #26544
Fixes: #24947
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Signed-off-by: Beth Griggs <Bethany.Griggs@uk.ibm.com>

BethGriggs added a commit that referenced this pull request Apr 9, 2019

worker: use copy of process.env
Instead of sharing the OS-backed store for all `process.env` instances,
create a copy of `process.env` for every worker that is created.

The copies do not interact. Native-addons do not see modifications to
`process.env` from Worker threads, but child processes started from
Workers do default to the Worker’s copy of `process.env`.

This makes Workers behave like child processes as far as `process.env`
is concerned, and an option corresponding to the `child_process`
module’s `env` option is added to the constructor.

Fixes: #24947

PR-URL: #26544
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Signed-off-by: Beth Griggs <Bethany.Griggs@uk.ibm.com>

BethGriggs added a commit that referenced this pull request Apr 9, 2019

2019-04-10, Version 11.14.0 (Current)
Notable changes:

- worker: use copy of process.env (Anna Henningsen)
  [#26544](#26544)
- src:
  - implement generic backend for process.env (Anna Henningsen)
    [#26544](#26544)
  - allow per-Environment set of env vars (Anna Henningsen)
    [#26544](#26544)

BethGriggs added a commit that referenced this pull request Apr 9, 2019

2019-04-10, Version 11.14.0 (Current)
Notable changes:

- worker: use copy of process.env (Anna Henningsen)
  [#26544](#26544)
- src:
  - implement generic backend for process.env (Anna Henningsen)
    [#26544](#26544)
  - allow per-Environment set of env vars (Anna Henningsen)
    [#26544](#26544)

@BethGriggs BethGriggs referenced this pull request Apr 9, 2019

Merged

v11.14.0 proposal #27163

BethGriggs added a commit that referenced this pull request Apr 10, 2019

2019-04-10, Version 11.14.0 (Current)
Notable changes:

- worker: use copy of process.env (Anna Henningsen)
  [#26544](#26544)

PR-URL: #27163

BethGriggs added a commit that referenced this pull request Apr 10, 2019

2019-04-10, Version 11.14.0 (Current)
Notable changes:

- child_process: doc deprecate ChildProcess.\_channel (cjihrig)
  [#26982](#26982)
- deps: update nghttp2 to 1.37.0 (gengjiawen)
  [#26990](#26990)
- dns:
  - make dns.promises enumerable (cjihrig)
    [#26592](#26592)
  - remove dns.promises experimental warning (cjihrig)
    [#26592](#26592)
- stream: make Symbol.asyncIterator support stable (Matteo Collina)
  [#26989](#26989)
- worker: use copy of process.env (Anna Henningsen)
  [#26544](#26544)

PR-URL: #27163

BethGriggs added a commit that referenced this pull request Apr 11, 2019

2019-04-11, Version 11.14.0 (Current)
Notable changes:

- child_process: doc deprecate ChildProcess.\_channel (cjihrig)
  [#26982](#26982)
- deps: update nghttp2 to 1.37.0 (gengjiawen)
  [#26990](#26990)
- dns:
  - make dns.promises enumerable (cjihrig)
    [#26592](#26592)
  - remove dns.promises experimental warning (cjihrig)
    [#26592](#26592)
- fs: remove experimental warning for fs.promises (Anna Henningsen)
  [#26581] (#26581)
- stream: make Symbol.asyncIterator support stable (Matteo Collina)
  [#26989](#26989)
- worker: use copy of process.env (Anna Henningsen)
  [#26544](#26544)

PR-URL: #27163

BethGriggs added a commit that referenced this pull request Apr 11, 2019

2019-04-11, Version 11.14.0 (Current)
Notable changes:

- child_process: doc deprecate ChildProcess.\_channel (cjihrig)
  [#26982](#26982)
- deps: update nghttp2 to 1.37.0 (gengjiawen)
  [#26990](#26990)
- dns:
  - make dns.promises enumerable (cjihrig)
    [#26592](#26592)
  - remove dns.promises experimental warning (cjihrig)
    [#26592](#26592)
- fs: remove experimental warning for fs.promises (Anna Henningsen)
  [#26581] (#26581)
- stream: make Symbol.asyncIterator support stable (Matteo Collina)
  [#26989](#26989)
- worker: use copy of process.env (Anna Henningsen)
  [#26544](#26544)

PR-URL: #27163
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.