process: split execution entry points into main scripts #25667

joyeecheung · 2019-01-23T13:06:38Z

Note: there are some indentation changes in lib/internal/bootstrap/node.js so please use https://github.com/nodejs/node/pull/25667/files?w=1 for reviews

process: split execution into main scripts

This patch splits the execution mode selection from the environment
setup in lib/internal/bootstrap/node.js, and split the entry point
of different execution mode into main scripts under
lib/internal/main:

check_syntax.js: used when -c/--check which only checks the
syntax of the input instead of executing it.
eval_stdin.js: used when -e is passed without value and stdin
is not a TTY (e.g. something is piped).
eval_string: used when -e is passed along with a string argument
inspect.js: for node inspect/node debug
print_bash_completion.js: for --completion-bash
print_help.js: for --help
prof_process.js: for --prof-process
repl.js: for the REPL
run_main_module.js: used when a main module is passed
run_third_party_main.js: for the legacy _third_party_main.js
support
worker_thread.js: for workers

This makes the entry points easier to navigate and paves the way
for customized v8 snapshots (that do not need to deserialize
execution mode setup) and better embedder APIs.

As an example, after this patch, for the most common case where
Node.js executes a user module as an entry point, it essentially
goes through:

lib/internal/per_context.js to setup the v8 Context (which is
also run when setting up contexts for the vm module)
lib/internal/bootstrap/loaders.js to set up internal binding
and builtin module loaders (that are separate from the loaders
accessible in the user land).
lib/internal/bootstrap/node.js: to set up the rest of the
environment, including various globals and the process object
lib/internal/main/run_main_module.js: which is selected from
C++ to prepare execution of the user module.

This patch also removes NativeModuleLoader::CompileAndCall and
exposes NativeModuleLoader::LookupAndCompile directly so that
we can handle syntax errors and runtime errors of bootstrap
scripts differently.

worker: move worker thread setup code into the main script

This patch directly inlines createMessageHandler() and
createWorkerFatalExeception() in the new
lib/internal/main/worker_thread.js since the implementation
of the two methods are related to the execution flow of
workers.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

nodejs-github-bot · 2019-01-23T13:06:40Z

@joyeecheung build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/2359/pipeline

joyeecheung · 2019-01-23T13:09:56Z

CI: https://ci.nodejs.org/job/node-test-pull-request/20285/

joyeecheung · 2019-01-23T13:27:08Z

src/env.cc

@@ -295,6 +295,17 @@ void Environment::Start(const std::vector<std::string>& args,
  HandleScope handle_scope(isolate());
  Context::Scope context_scope(context());

+  if (args.size() > 1) {
+    std::string first_arg = args[1];
+    if (first_arg == "inspect") {


Ideally this should not have been done here - maybe a better place would be the options parser but at that point Environment is not yet available and we do not have a better place to carry this information yet.

src/node_options.cc

src/node.cc

joyeecheung · 2019-01-29T13:03:13Z

I've removed the options changes as those are not really related to the primary purpose of this PR. @addaleax PTAL.

CI: https://ci.nodejs.org/job/node-test-pull-request/20406/

This patch splits the execution mode selection from the environment setup in `lib/internal/bootstrap/node.js`, and split the entry point of different execution mode into main scripts under `lib/internal/main`: - `check_syntax.js`: used when `-c`/`--check` which only checks the syntax of the input instead of executing it. - `eval_stdin.js`: used when `-e` is passed without value and stdin is not a TTY (e.g. something is piped). - `eval_string`: used when `-e` is passed along with a string argument - `inspect.js`: for `node inspect`/`node debug` - `print_bash_completion.js`: for `--completion-bash` - `print_help.js`: for `--help` - `prof_process.js`: for `--prof-process` - `repl.js`: for the REPL - `run_main_module.js`: used when a main module is passed - `run_third_party_main.js`: for the legacy `_third_party_main.js` support - `worker_thread.js`: for workers This makes the entry points easier to navigate and paves the way for customized v8 snapshots (that do not need to deserialize execution mode setup) and better embedder APIs. As an example, after this patch, for the most common case where Node.js executes a user module as an entry point, it essentially goes through: - `lib/internal/per_context.js` to setup the v8 Context (which is also run when setting up contexts for the `vm` module) - `lib/internal/bootstrap/loaders.js` to set up internal binding and builtin module loaders (that are separate from the loaders accessible in the user land). - `lib/internal/bootstrap/node.js`: to set up the rest of the environment, including various globals and the process object - `lib/internal/main/run_main_module.js`: which is selected from C++ to prepare execution of the user module. This patch also removes `NativeModuleLoader::CompileAndCall` and exposes `NativeModuleLoader::LookupAndCompile` directly so that we can handle syntax errors and runtime errors of bootstrap scripts differently.

This patch directly inlines `createMessageHandler()` and `createWorkerFatalExeception()` in the new `lib/internal/main/worker_thread.js` since the implementation of the two methods are related to the execution flow of workers.

joyeecheung · 2019-01-30T12:57:15Z

Rebased. CI: https://ci.nodejs.org/job/node-test-pull-request/20431/

joyeecheung · 2019-01-30T13:08:25Z

Landed in feebdc5...5e1d446

This patch splits the execution mode selection from the environment setup in `lib/internal/bootstrap/node.js`, and split the entry point of different execution mode into main scripts under `lib/internal/main`: - `check_syntax.js`: used when `-c`/`--check` which only checks the syntax of the input instead of executing it. - `eval_stdin.js`: used when `-e` is passed without value and stdin is not a TTY (e.g. something is piped). - `eval_string`: used when `-e` is passed along with a string argument - `inspect.js`: for `node inspect`/`node debug` - `print_bash_completion.js`: for `--completion-bash` - `print_help.js`: for `--help` - `prof_process.js`: for `--prof-process` - `repl.js`: for the REPL - `run_main_module.js`: used when a main module is passed - `run_third_party_main.js`: for the legacy `_third_party_main.js` support - `worker_thread.js`: for workers This makes the entry points easier to navigate and paves the way for customized v8 snapshots (that do not need to deserialize execution mode setup) and better embedder APIs. As an example, after this patch, for the most common case where Node.js executes a user module as an entry point, it essentially goes through: - `lib/internal/per_context.js` to setup the v8 Context (which is also run when setting up contexts for the `vm` module) - `lib/internal/bootstrap/loaders.js` to set up internal binding and builtin module loaders (that are separate from the loaders accessible in the user land). - `lib/internal/bootstrap/node.js`: to set up the rest of the environment, including various globals and the process object - `lib/internal/main/run_main_module.js`: which is selected from C++ to prepare execution of the user module. This patch also removes `NativeModuleLoader::CompileAndCall` and exposes `NativeModuleLoader::LookupAndCompile` directly so that we can handle syntax errors and runtime errors of bootstrap scripts differently. PR-URL: #25667 Reviewed-By: Anna Henningsen <anna@addaleax.net>

This patch directly inlines `createMessageHandler()` and `createWorkerFatalExeception()` in the new `lib/internal/main/worker_thread.js` since the implementation of the two methods are related to the execution flow of workers. PR-URL: #25667 Reviewed-By: Anna Henningsen <anna@addaleax.net>

addaleax · 2019-02-01T15:58:29Z

@joyeecheung This needs a manual backport for v11.x, it seems.

This patch splits the execution mode selection from the environment setup in `lib/internal/bootstrap/node.js`, and split the entry point of different execution mode into main scripts under `lib/internal/main`: - `check_syntax.js`: used when `-c`/`--check` which only checks the syntax of the input instead of executing it. - `eval_stdin.js`: used when `-e` is passed without value and stdin is not a TTY (e.g. something is piped). - `eval_string`: used when `-e` is passed along with a string argument - `inspect.js`: for `node inspect`/`node debug` - `print_bash_completion.js`: for `--completion-bash` - `print_help.js`: for `--help` - `prof_process.js`: for `--prof-process` - `repl.js`: for the REPL - `run_main_module.js`: used when a main module is passed - `run_third_party_main.js`: for the legacy `_third_party_main.js` support - `worker_thread.js`: for workers This makes the entry points easier to navigate and paves the way for customized v8 snapshots (that do not need to deserialize execution mode setup) and better embedder APIs. As an example, after this patch, for the most common case where Node.js executes a user module as an entry point, it essentially goes through: - `lib/internal/per_context.js` to setup the v8 Context (which is also run when setting up contexts for the `vm` module) - `lib/internal/bootstrap/loaders.js` to set up internal binding and builtin module loaders (that are separate from the loaders accessible in the user land). - `lib/internal/bootstrap/node.js`: to set up the rest of the environment, including various globals and the process object - `lib/internal/main/run_main_module.js`: which is selected from C++ to prepare execution of the user module. This patch also removes `NativeModuleLoader::CompileAndCall` and exposes `NativeModuleLoader::LookupAndCompile` directly so that we can handle syntax errors and runtime errors of bootstrap scripts differently. PR-URL: nodejs#25667 Reviewed-By: Anna Henningsen <anna@addaleax.net>

This patch directly inlines `createMessageHandler()` and `createWorkerFatalExeception()` in the new `lib/internal/main/worker_thread.js` since the implementation of the two methods are related to the execution flow of workers. PR-URL: nodejs#25667 Reviewed-By: Anna Henningsen <anna@addaleax.net>

This patch splits the execution mode selection from the environment setup in `lib/internal/bootstrap/node.js`, and split the entry point of different execution mode into main scripts under `lib/internal/main`: - `check_syntax.js`: used when `-c`/`--check` which only checks the syntax of the input instead of executing it. - `eval_stdin.js`: used when `-e` is passed without value and stdin is not a TTY (e.g. something is piped). - `eval_string`: used when `-e` is passed along with a string argument - `inspect.js`: for `node inspect`/`node debug` - `print_bash_completion.js`: for `--completion-bash` - `print_help.js`: for `--help` - `prof_process.js`: for `--prof-process` - `repl.js`: for the REPL - `run_main_module.js`: used when a main module is passed - `run_third_party_main.js`: for the legacy `_third_party_main.js` support - `worker_thread.js`: for workers This makes the entry points easier to navigate and paves the way for customized v8 snapshots (that do not need to deserialize execution mode setup) and better embedder APIs. As an example, after this patch, for the most common case where Node.js executes a user module as an entry point, it essentially goes through: - `lib/internal/per_context.js` to setup the v8 Context (which is also run when setting up contexts for the `vm` module) - `lib/internal/bootstrap/loaders.js` to set up internal binding and builtin module loaders (that are separate from the loaders accessible in the user land). - `lib/internal/bootstrap/node.js`: to set up the rest of the environment, including various globals and the process object - `lib/internal/main/run_main_module.js`: which is selected from C++ to prepare execution of the user module. This patch also removes `NativeModuleLoader::CompileAndCall` and exposes `NativeModuleLoader::LookupAndCompile` directly so that we can handle syntax errors and runtime errors of bootstrap scripts differently. PR-URL: nodejs#25667 Reviewed-By: Anna Henningsen <anna@addaleax.net>

This patch directly inlines `createMessageHandler()` and `createWorkerFatalExeception()` in the new `lib/internal/main/worker_thread.js` since the implementation of the two methods are related to the execution flow of workers. PR-URL: nodejs#25667 Reviewed-By: Anna Henningsen <anna@addaleax.net>

This patch splits the execution mode selection from the environment setup in `lib/internal/bootstrap/node.js`, and split the entry point of different execution mode into main scripts under `lib/internal/main`: - `check_syntax.js`: used when `-c`/`--check` which only checks the syntax of the input instead of executing it. - `eval_stdin.js`: used when `-e` is passed without value and stdin is not a TTY (e.g. something is piped). - `eval_string`: used when `-e` is passed along with a string argument - `inspect.js`: for `node inspect`/`node debug` - `print_bash_completion.js`: for `--completion-bash` - `print_help.js`: for `--help` - `prof_process.js`: for `--prof-process` - `repl.js`: for the REPL - `run_main_module.js`: used when a main module is passed - `run_third_party_main.js`: for the legacy `_third_party_main.js` support - `worker_thread.js`: for workers This makes the entry points easier to navigate and paves the way for customized v8 snapshots (that do not need to deserialize execution mode setup) and better embedder APIs. As an example, after this patch, for the most common case where Node.js executes a user module as an entry point, it essentially goes through: - `lib/internal/per_context.js` to setup the v8 Context (which is also run when setting up contexts for the `vm` module) - `lib/internal/bootstrap/loaders.js` to set up internal binding and builtin module loaders (that are separate from the loaders accessible in the user land). - `lib/internal/bootstrap/node.js`: to set up the rest of the environment, including various globals and the process object - `lib/internal/main/run_main_module.js`: which is selected from C++ to prepare execution of the user module. This patch also removes `NativeModuleLoader::CompileAndCall` and exposes `NativeModuleLoader::LookupAndCompile` directly so that we can handle syntax errors and runtime errors of bootstrap scripts differently. PR-URL: #25667 Reviewed-By: Anna Henningsen <anna@addaleax.net> Backport-PR-URL: #26036

This patch directly inlines `createMessageHandler()` and `createWorkerFatalExeception()` in the new `lib/internal/main/worker_thread.js` since the implementation of the two methods are related to the execution flow of workers. PR-URL: #25667 Reviewed-By: Anna Henningsen <anna@addaleax.net> Backport-PR-URL: #26036

Node removed the NativeModuleLoader CompileAndCall function and exposed the LookupAndCompile function, with the intention of giving the ability to the caller to handle errors. The node PR made the changes nodejs/node#25667. I'm adding this back for now, but this can easily be moved to electron as a helper or can probably be upstreamed somehow.

nodejs-github-bot added the lib / src Issues and PRs related to general changes in the lib or src directory. label Jan 23, 2019

joyeecheung requested a review from addaleax January 23, 2019 13:10

joyeecheung changed the title ~~process: split execution into main scripts~~ process: split execution entry points into main scripts Jan 23, 2019

joyeecheung commented Jan 23, 2019

View reviewed changes

addaleax reviewed Jan 23, 2019

View reviewed changes

src/node_options.cc Outdated Show resolved Hide resolved

addaleax mentioned this pull request Jan 23, 2019

src: pass through Maybes from bootstrapper functions #25620

Closed

2 tasks

addaleax reviewed Jan 23, 2019

View reviewed changes

src/node.cc Outdated Show resolved Hide resolved

joyeecheung mentioned this pull request Jan 24, 2019

Investigate flaky sequential/test-performance #23291

Closed

joyeecheung force-pushed the split-execution branch from 0bab6b0 to e5ddc18 Compare January 29, 2019 13:02

addaleax approved these changes Jan 29, 2019

View reviewed changes

joyeecheung added 3 commits January 30, 2019 07:20

worker: move worker thread setup code into the main script

857d84f

This patch directly inlines `createMessageHandler()` and `createWorkerFatalExeception()` in the new `lib/internal/main/worker_thread.js` since the implementation of the two methods are related to the execution flow of workers.

fixup! process: split execution into main scripts

455c923

joyeecheung force-pushed the split-execution branch from e5ddc18 to 455c923 Compare January 29, 2019 23:25

joyeecheung closed this Jan 30, 2019

addaleax added the backport-requested-v11.x label Feb 1, 2019

targos mentioned this pull request Feb 6, 2019

Help wanted: Backporting to v11.x #25403

Closed

joyeecheung mentioned this pull request Feb 8, 2019

Node 11.4.0+ fails to resolve index.js when combined with -r and --inspect-brk #25967

Closed

ZYSzys mentioned this pull request Feb 10, 2019

[backport 11.x]: src: move public C++ APIs into src/api/*.cc #26028

Closed

4 tasks

targos added the backported-to-v11.x label Feb 10, 2019

targos removed the backport-requested-v11.x label Feb 10, 2019

targos mentioned this pull request Feb 10, 2019

[v11.x] Backport "process: split execution entry points into main scripts" #26036

Closed

targos added this to Backported in v11.x Feb 10, 2019

targos mentioned this pull request Feb 14, 2019

v11.10.0 proposal #26098

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

process: split execution entry points into main scripts #25667

process: split execution entry points into main scripts #25667

joyeecheung commented Jan 23, 2019 •

edited

nodejs-github-bot commented Jan 23, 2019

joyeecheung commented Jan 23, 2019

joyeecheung Jan 23, 2019

joyeecheung commented Jan 29, 2019

joyeecheung commented Jan 30, 2019

joyeecheung commented Jan 30, 2019

addaleax commented Feb 1, 2019

process: split execution entry points into main scripts #25667

process: split execution entry points into main scripts #25667

Conversation

joyeecheung commented Jan 23, 2019 • edited

process: split execution into main scripts

worker: move worker thread setup code into the main script

Checklist

nodejs-github-bot commented Jan 23, 2019

joyeecheung commented Jan 23, 2019

joyeecheung Jan 23, 2019

Choose a reason for hiding this comment

joyeecheung commented Jan 29, 2019

joyeecheung commented Jan 30, 2019

joyeecheung commented Jan 30, 2019

addaleax commented Feb 1, 2019

joyeecheung commented Jan 23, 2019 •

edited