test_runner: add initial CLI runner

[cjihrig]

This commit introduces an initial version of a CLI-based test runner.

[nodejs-github-bot]

Review requested:

• @nodejs/startup

[aduh95]

//cc @nodejs/test_runner

[ljharb]

How does --test affect --require, or loaders? What happens to the REPL with node --test and no other arguments?

If NODE_OPTIONS='--test' is set, and the user doesn't control how node is invoked (via a shebang, for example), does this mean the user can never "undo" test mode?

This really feels to me like it should be an entirely distinct binary, rather than just a "mode" of the main node binary.

[richardlau]

If NODE_OPTIONS='--test' is set, and the user doesn't control how node is invoked (via a shebang, for example), does this mean the user can never "undo" test mode?

As this PR currently is --test hasn't been added to the allow list for NODE_OPTIONS so setting it there would throw an error.

[ljharb]

@richardlau thanks, if that's explicitly going to never be allowed there then that does mitigate that one concern, but all the others remain.

[cjihrig]

How does --test affect --require, or loaders? What happens to the REPL with node --test and no other arguments?

node --test doesn't launch the REPL. I haven't tried a loader, but --require seems to work fine.

If NODE_OPTIONS='--test' is set, and the user doesn't control how node is invoked (via a shebang, for example), does this mean the user can never "undo" test mode?

--test in NODE_OPTIONS explicitly errors out.

[jasnell]

Just a question: why is this? I can definitely see wanting to run the inspector for debugging while running tests.

[cjihrig]

A couple reasons:

1. First, it's trivial to run an individual file with the inspector flags. At this point, the CLI runner is just spawning Node child processes with no special flags.
2. Debugging through the CLI test runner is an awkward experience similar to the cluster module. Each test file will need a different debugger port. We would need to manage that logic as well as relay the correct information to users (I want to debug test X, I need to debug file Y, and connect to debug port Z). It also doesn't make sense to set something like --inspect-brk on a bunch of child processes.
3. I looked at a couple test frameworks, and they didn't seem to have a great story around propagating inspector flags to test files.
4. If someone is passionate about this, it can be added in the future. It just didn't seem worth it to me for an initial version given the previous points. I think it would make sense in the future to have configuration options for execArgv and argv of the child processes.

[ljharb]

Can we use chrome's debugger blackboxing stuff to "hide" the test runner code when inspecting?

[cjihrig]

Possibly - I'm not sure though.

[nodejs-github-bot]

Commit Queue failed

- Loading data for nodejs/node/pull/42658
✔  Done loading data for nodejs/node/pull/42658
----------------------------------- PR info ------------------------------------
Title      test_runner: add initial CLI runner (#42658)
Author     Colin Ihrig  (@cjihrig)
Branch     cjihrig:test-cli -> nodejs:master
Labels     semver-minor, lib / src, author ready, needs-ci, dont-land-on-v12.x, dont-land-on-v14.x, dont-land-on-v16.x, dont-land-on-v17.x
Commits    1
 - test_runner: add initial CLI runner
Committers 1
 - cjihrig 
PR-URL: https://github.com/nodejs/node/pull/42658
Reviewed-By: Antoine du Hamel 
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/42658
Reviewed-By: Antoine du Hamel 
--------------------------------------------------------------------------------
   ⚠  Commits were pushed since the last review:
   ⚠  - test_runner: add initial CLI runner
   ℹ  This PR was created on Fri, 08 Apr 2022 15:34:51 GMT
   ✔  Approvals: 1
   ✔  - Antoine du Hamel (@aduh95) (TSC): https://github.com/nodejs/node/pull/42658#pullrequestreview-941531383
   ✖  This PR needs to wait 1 more hours to land (or 0 hours if there is one more approval)
   ✔  Last GitHub CI successful
   ℹ  Last Full PR CI on 2022-04-15T13:39:53Z: https://ci.nodejs.org/job/node-test-pull-request/43514/
- Querying data for job/node-test-pull-request/43514/
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  Aborted `git node land` session in /home/runner/work/node/node/.ncu

https://github.com/nodejs/node/actions/runs/2172929813

[nodejs-github-bot]

Commit Queue failed

- Loading data for nodejs/node/pull/42658
FetchError: Invalid response body while trying to fetch https://api.github.com/graphql: Premature close
    at consumeBody (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/node_modules/node-fetch/src/body.js:234:60)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Response.text (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/node_modules/node-fetch/src/body.js:158:18)
    at async Request.json (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/request.js:49:18)
    at async Request.query (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/request.js:107:20)
    at async Request.queryAll (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/request.js:134:20)
    at async Request.gql (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/request.js:64:22)
    at async PRData.getComments (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/pr_data.js:97:21)
    at async Promise.all (index 2)
    at async Promise.all (index 1) {
  type: 'system',
  errno: 'ERR_STREAM_PREMATURE_CLOSE',
  code: 'ERR_STREAM_PREMATURE_CLOSE',
  erroredSysCall: undefined
}

https://github.com/nodejs/node/actions/runs/2173428186

[Trott]

@cjihrig I think you'll either need to land this manually or wait for one more approval (or for @aduh95 to re-approve). The commit-queue won't land a PR if a commit has been pushed without a subsequent approval from someone.

[nodejs-github-bot]

Landed in adaf602

[sosoba]

Who will extend the @types/node package with a new node:test module?

[Trott]

Who will extend the @types/node package with a new node:test module?

As always, the maintainers of https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node.