test_runner: add initial CLI runner #42658

cjihrig · 2022-04-08T15:34:51Z

This commit introduces an initial version of a CLI-based test runner.

nodejs-github-bot · 2022-04-08T15:34:55Z

Review requested:

@nodejs/startup

doc/api/test.md

lib/internal/main/test_runner.js

doc/api/test.md

lib/internal/main/test_runner.js

lib/internal/test_runner/tap_stream.js

aduh95 · 2022-04-08T15:49:26Z

//cc @nodejs/test_runner

ljharb

How does --test affect --require, or loaders? What happens to the REPL with node --test and no other arguments?

If NODE_OPTIONS='--test' is set, and the user doesn't control how node is invoked (via a shebang, for example), does this mean the user can never "undo" test mode?

This really feels to me like it should be an entirely distinct binary, rather than just a "mode" of the main node binary.

doc/api/test.md

lib/internal/main/test_runner.js

lib/internal/test_runner/utils.js

lib/internal/test_runner/tap_stream.js

lib/internal/main/test_runner.js

richardlau · 2022-04-08T16:15:21Z

If NODE_OPTIONS='--test' is set, and the user doesn't control how node is invoked (via a shebang, for example), does this mean the user can never "undo" test mode?

As this PR currently is --test hasn't been added to the allow list for NODE_OPTIONS so setting it there would throw an error.

ljharb · 2022-04-08T16:19:48Z

@richardlau thanks, if that's explicitly going to never be allowed there then that does mitigate that one concern, but all the others remain.

cjihrig · 2022-04-08T16:22:52Z

How does --test affect --require, or loaders? What happens to the REPL with node --test and no other arguments?

node --test doesn't launch the REPL. I haven't tried a loader, but --require seems to work fine.

If NODE_OPTIONS='--test' is set, and the user doesn't control how node is invoked (via a shebang, for example), does this mean the user can never "undo" test mode?

--test in NODE_OPTIONS explicitly errors out.

doc/api/test.md

lib/internal/test_runner/utils.js

doc/api/cli.md

doc/api/test.md

jasnell · 2022-04-13T15:22:24Z

src/node_options.cc

+    }
+
+    if (debug_options_.inspector_enabled) {
+      errors->push_back("the inspector cannot be used with --test");


Just a question: why is this? I can definitely see wanting to run the inspector for debugging while running tests.

A couple reasons:

First, it's trivial to run an individual file with the inspector flags. At this point, the CLI runner is just spawning Node child processes with no special flags.

Debugging through the CLI test runner is an awkward experience similar to the cluster module. Each test file will need a different debugger port. We would need to manage that logic as well as relay the correct information to users (I want to debug test X, I need to debug file Y, and connect to debug port Z). It also doesn't make sense to set something like --inspect-brk on a bunch of child processes.

I looked at a couple test frameworks, and they didn't seem to have a great story around propagating inspector flags to test files.

If someone is passionate about this, it can be added in the future. It just didn't seem worth it to me for an initial version given the previous points. I think it would make sense in the future to have configuration options for execArgv and argv of the child processes.

Can we use chrome's debugger blackboxing stuff to "hide" the test runner code when inspecting?

Possibly - I'm not sure though.

nodejs-github-bot · 2022-04-15T16:33:16Z

Commit Queue failed

- Loading data for nodejs/node/pull/42658
FetchError: Invalid response body while trying to fetch https://api.github.com/graphql: Premature close
    at consumeBody (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/node_modules/node-fetch/src/body.js:234:60)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Response.text (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/node_modules/node-fetch/src/body.js:158:18)
    at async Request.json (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/request.js:49:18)
    at async Request.query (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/request.js:107:20)
    at async Request.queryAll (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/request.js:134:20)
    at async Request.gql (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/request.js:64:22)
    at async PRData.getComments (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/pr_data.js:97:21)
    at async Promise.all (index 2)
    at async Promise.all (index 1) {
  type: 'system',
  errno: 'ERR_STREAM_PREMATURE_CLOSE',
  code: 'ERR_STREAM_PREMATURE_CLOSE',
  erroredSysCall: undefined
}

https://github.com/nodejs/node/actions/runs/2173428186

Trott · 2022-04-15T16:36:56Z

@cjihrig I think you'll either need to land this manually or wait for one more approval (or for @aduh95 to re-approve). The commit-queue won't land a PR if a commit has been pushed without a subsequent approval from someone.

nodejs-github-bot · 2022-04-15T17:37:30Z

Landed in adaf602

This commit introduces an initial version of a CLI-based test runner. PR-URL: nodejs#42658 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

This commit introduces an initial version of a CLI-based test runner. PR-URL: #42658 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

@kuriyosh

Notable changes: doc: * add @kuriyosh to collaborators (Yoshiki Kurihara) #42824 lib,src: * (SEMVER-MINOR) implement WebAssembly Web API (Tobias Nießen) #42701 test_runner: * (SEMVER-MINOR) add initial CLI runner (Colin Ihrig) #42658 worker: * (SEMVER-MINOR) add hasRef() to MessagePort (Darshan Sen) #42849 PR-URL: #42943

@kuriyosh

Notable changes: doc: * add @kuriyosh to collaborators (Yoshiki Kurihara) #42824 lib,src: * (SEMVER-MINOR) implement WebAssembly Web API (Tobias Nießen) #42701 test_runner: * (SEMVER-MINOR) add initial CLI runner (Colin Ihrig) #42658 worker: * (SEMVER-MINOR) add hasRef() to MessagePort (Darshan Sen) #42849 PR-URL: #42943

sosoba · 2022-05-04T07:08:55Z

Who will extend the @types/node package with a new node:test module?

Trott · 2022-05-04T14:26:34Z

Who will extend the @types/node package with a new node:test module?

As always, the maintainers of https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node.

This commit introduces an initial version of a CLI-based test runner. PR-URL: nodejs/node#42658 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

@kuriyosh

Notable changes: doc: * add @kuriyosh to collaborators (Yoshiki Kurihara) nodejs/node#42824 lib,src: * (SEMVER-MINOR) implement WebAssembly Web API (Tobias Nießen) nodejs/node#42701 test_runner: * (SEMVER-MINOR) add initial CLI runner (Colin Ihrig) nodejs/node#42658 worker: * (SEMVER-MINOR) add hasRef() to MessagePort (Darshan Sen) nodejs/node#42849 PR-URL: nodejs/node#42943

nodejs-github-bot added lib / src needs-ci labels Apr 8, 2022

aduh95 reviewed Apr 8, 2022

View changes

richardlau added the semver-minor label Apr 8, 2022

ljharb reviewed Apr 8, 2022

View changes

cjihrig added dont-land-on-v12.x dont-land-on-v14.x dont-land-on-v16.x dont-land-on-v17.x labels Apr 8, 2022

bnb reviewed Apr 8, 2022

View changes

doc/api/test.md Outdated Show resolved Hide resolved

targos reviewed Apr 9, 2022

View changes

doc/api/test.md Outdated Show resolved Hide resolved

cjihrig force-pushed the test-cli branch 2 times, most recently from f3263c3 to 805361e Compare Apr 9, 2022

aduh95 reviewed Apr 9, 2022

View changes

doc/api/test.md Outdated Show resolved Hide resolved

cjihrig force-pushed the test-cli branch from 805361e to 907e8ef Compare Apr 9, 2022

aduh95 reviewed Apr 9, 2022

View changes

doc/api/test.md Outdated Show resolved Hide resolved

cjihrig force-pushed the test-cli branch from 907e8ef to f8a34dc Compare Apr 9, 2022

ljharb reviewed Apr 10, 2022

View changes

doc/api/test.md Outdated Show resolved Hide resolved

lib/internal/test_runner/utils.js Show resolved Hide resolved

targos reviewed Apr 10, 2022

View changes

doc/api/cli.md Outdated Show resolved Hide resolved

cjihrig force-pushed the test-cli branch 2 times, most recently from be14d2f to a8c868e Compare Apr 10, 2022

targos mentioned this pull request Apr 12, 2022

2022-04-19, Version 18.0.0 (Current) #42262

Merged

jasnell reviewed Apr 13, 2022

View changes

doc/api/test.md Show resolved Hide resolved

jasnell reviewed Apr 13, 2022

View changes

aduh95 approved these changes Apr 13, 2022

View changes

cjihrig added commit-queue and removed commit-queue-failed labels Apr 15, 2022

nodejs-github-bot added commit-queue-failed and removed commit-queue labels Apr 15, 2022

aduh95 approved these changes Apr 15, 2022

View changes

aduh95 added commit-queue and removed commit-queue-failed labels Apr 15, 2022

nodejs-github-bot removed the commit-queue label Apr 15, 2022

nodejs-github-bot merged commit adaf602 into nodejs:master Apr 15, 2022
57 checks passed

cjihrig deleted the test-cli branch Apr 15, 2022

github-actions bot mentioned this pull request Apr 16, 2022

CI Reliability 2022-04-16 nodejs/reliability#253

Open

16 tasks

github-actions bot mentioned this pull request Apr 17, 2022

CI Reliability 2022-04-17 nodejs/reliability#254

Open

15 tasks

github-actions bot mentioned this pull request Apr 18, 2022

CI Reliability 2022-04-18 nodejs/reliability#255

Open

14 tasks

github-actions bot mentioned this pull request Apr 19, 2022

CI Reliability 2022-04-19 nodejs/reliability#256

Open

16 tasks

github-actions bot mentioned this pull request Apr 20, 2022

CI Reliability 2022-04-20 nodejs/reliability#257

Open

16 tasks

github-actions bot mentioned this pull request Apr 21, 2022

CI Reliability 2022-04-21 nodejs/reliability#258

Open

22 tasks

xtx1130 pushed a commit to xtx1130/node that referenced this issue Apr 25, 2022

test_runner: add initial CLI runner

05d037d

This commit introduces an initial version of a CLI-based test runner. PR-URL: nodejs#42658 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

targos pushed a commit that referenced this issue Apr 28, 2022

test_runner: add initial CLI runner

78a860a

This commit introduces an initial version of a CLI-based test runner. PR-URL: #42658 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

targos mentioned this pull request May 2, 2022

v18.1.0 release proposal #42943

Merged

juanarbol mentioned this pull request May 30, 2022

test: use assert.match() instead of assert(regex.test()) #42803

Merged

tniessen added the test_runner label Jun 8, 2022

LiviaMedeiros pushed a commit to LiviaMedeiros/done that referenced this issue Jun 28, 2022

test_runner: add initial CLI runner

d960e97

This commit introduces an initial version of a CLI-based test runner. PR-URL: nodejs/node#42658 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

test_runner: add initial CLI runner #42658

test_runner: add initial CLI runner #42658

cjihrig commented Apr 8, 2022

nodejs-github-bot commented Apr 8, 2022

aduh95 commented Apr 8, 2022

ljharb left a comment

richardlau commented Apr 8, 2022

ljharb commented Apr 8, 2022

cjihrig commented Apr 8, 2022

jasnell Apr 13, 2022

cjihrig Apr 13, 2022

ljharb Apr 13, 2022

cjihrig Apr 14, 2022

nodejs-github-bot commented Apr 15, 2022

Trott commented Apr 15, 2022

nodejs-github-bot commented Apr 15, 2022

sosoba commented May 4, 2022

Trott commented May 4, 2022

test_runner: add initial CLI runner #42658

test_runner: add initial CLI runner #42658

Conversation

cjihrig commented Apr 8, 2022

nodejs-github-bot commented Apr 8, 2022

aduh95 commented Apr 8, 2022

ljharb left a comment

richardlau commented Apr 8, 2022

ljharb commented Apr 8, 2022

cjihrig commented Apr 8, 2022

jasnell Apr 13, 2022

Choose a reason for hiding this comment

cjihrig Apr 13, 2022

Choose a reason for hiding this comment

ljharb Apr 13, 2022

Choose a reason for hiding this comment

cjihrig Apr 14, 2022

Choose a reason for hiding this comment

nodejs-github-bot commented Apr 15, 2022

Trott commented Apr 15, 2022

nodejs-github-bot commented Apr 15, 2022

sosoba commented May 4, 2022

Trott commented May 4, 2022