Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_runner: add initial CLI runner #42658

Merged
merged 1 commit into from Apr 15, 2022
Merged

Conversation

cjihrig
Copy link
Contributor

@cjihrig cjihrig commented Apr 8, 2022

This commit introduces an initial version of a CLI-based test runner.

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Apr 8, 2022

Review requested:

@nodejs-github-bot nodejs-github-bot added lib / src needs-ci labels Apr 8, 2022
doc/api/test.md Outdated Show resolved Hide resolved
lib/internal/main/test_runner.js Outdated Show resolved Hide resolved
doc/api/test.md Outdated Show resolved Hide resolved
doc/api/test.md Outdated Show resolved Hide resolved
lib/internal/main/test_runner.js Show resolved Hide resolved
lib/internal/main/test_runner.js Outdated Show resolved Hide resolved
lib/internal/test_runner/tap_stream.js Outdated Show resolved Hide resolved
@aduh95
Copy link
Contributor

@aduh95 aduh95 commented Apr 8, 2022

@richardlau richardlau added the semver-minor label Apr 8, 2022
Copy link
Member

@ljharb ljharb left a comment

How does --test affect --require, or loaders? What happens to the REPL with node --test and no other arguments?

If NODE_OPTIONS='--test' is set, and the user doesn't control how node is invoked (via a shebang, for example), does this mean the user can never "undo" test mode?

This really feels to me like it should be an entirely distinct binary, rather than just a "mode" of the main node binary.

doc/api/test.md Outdated Show resolved Hide resolved
doc/api/test.md Show resolved Hide resolved
lib/internal/main/test_runner.js Outdated Show resolved Hide resolved
lib/internal/main/test_runner.js Outdated Show resolved Hide resolved
lib/internal/main/test_runner.js Outdated Show resolved Hide resolved
lib/internal/main/test_runner.js Show resolved Hide resolved
lib/internal/test_runner/utils.js Outdated Show resolved Hide resolved
lib/internal/test_runner/tap_stream.js Outdated Show resolved Hide resolved
lib/internal/main/test_runner.js Show resolved Hide resolved
@cjihrig cjihrig added dont-land-on-v12.x dont-land-on-v14.x dont-land-on-v16.x dont-land-on-v17.x labels Apr 8, 2022
@richardlau
Copy link
Member

@richardlau richardlau commented Apr 8, 2022

If NODE_OPTIONS='--test' is set, and the user doesn't control how node is invoked (via a shebang, for example), does this mean the user can never "undo" test mode?

As this PR currently is --test hasn't been added to the allow list for NODE_OPTIONS so setting it there would throw an error.

@ljharb
Copy link
Member

@ljharb ljharb commented Apr 8, 2022

@richardlau thanks, if that's explicitly going to never be allowed there then that does mitigate that one concern, but all the others remain.

@cjihrig
Copy link
Contributor Author

@cjihrig cjihrig commented Apr 8, 2022

How does --test affect --require, or loaders? What happens to the REPL with node --test and no other arguments?

node --test doesn't launch the REPL. I haven't tried a loader, but --require seems to work fine.

If NODE_OPTIONS='--test' is set, and the user doesn't control how node is invoked (via a shebang, for example), does this mean the user can never "undo" test mode?

--test in NODE_OPTIONS explicitly errors out.

doc/api/test.md Outdated Show resolved Hide resolved
doc/api/test.md Outdated Show resolved Hide resolved
@cjihrig cjihrig force-pushed the test-cli branch 2 times, most recently from f3263c3 to 805361e Compare Apr 9, 2022
doc/api/test.md Outdated Show resolved Hide resolved
doc/api/test.md Outdated Show resolved Hide resolved
doc/api/test.md Outdated Show resolved Hide resolved
lib/internal/test_runner/utils.js Show resolved Hide resolved
doc/api/cli.md Outdated Show resolved Hide resolved
@cjihrig cjihrig force-pushed the test-cli branch 2 times, most recently from be14d2f to a8c868e Compare Apr 10, 2022
doc/api/test.md Show resolved Hide resolved
}

if (debug_options_.inspector_enabled) {
errors->push_back("the inspector cannot be used with --test");
Copy link
Member

@jasnell jasnell Apr 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a question: why is this? I can definitely see wanting to run the inspector for debugging while running tests.

Copy link
Contributor Author

@cjihrig cjihrig Apr 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple reasons:

  1. First, it's trivial to run an individual file with the inspector flags. At this point, the CLI runner is just spawning Node child processes with no special flags.
  2. Debugging through the CLI test runner is an awkward experience similar to the cluster module. Each test file will need a different debugger port. We would need to manage that logic as well as relay the correct information to users (I want to debug test X, I need to debug file Y, and connect to debug port Z). It also doesn't make sense to set something like --inspect-brk on a bunch of child processes.
  3. I looked at a couple test frameworks, and they didn't seem to have a great story around propagating inspector flags to test files.
  4. If someone is passionate about this, it can be added in the future. It just didn't seem worth it to me for an initial version given the previous points. I think it would make sense in the future to have configuration options for execArgv and argv of the child processes.

Copy link
Member

@ljharb ljharb Apr 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use chrome's debugger blackboxing stuff to "hide" the test runner code when inspecting?

Copy link
Contributor Author

@cjihrig cjihrig Apr 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly - I'm not sure though.

aduh95
aduh95 approved these changes Apr 13, 2022
@cjihrig cjihrig added the commit-queue label Apr 15, 2022
@nodejs-github-bot nodejs-github-bot removed the commit-queue label Apr 15, 2022
@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Apr 15, 2022

Commit Queue failed
- Loading data for nodejs/node/pull/42658
✔  Done loading data for nodejs/node/pull/42658
----------------------------------- PR info ------------------------------------
Title      test_runner: add initial CLI runner (#42658)
Author     Colin Ihrig  (@cjihrig)
Branch     cjihrig:test-cli -> nodejs:master
Labels     semver-minor, lib / src, author ready, needs-ci, dont-land-on-v12.x, dont-land-on-v14.x, dont-land-on-v16.x, dont-land-on-v17.x
Commits    1
 - test_runner: add initial CLI runner
Committers 1
 - cjihrig 
PR-URL: https://github.com/nodejs/node/pull/42658
Reviewed-By: Antoine du Hamel 
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/42658
Reviewed-By: Antoine du Hamel 
--------------------------------------------------------------------------------
   ⚠  Commits were pushed since the last review:
   ⚠  - test_runner: add initial CLI runner
   ℹ  This PR was created on Fri, 08 Apr 2022 15:34:51 GMT
   ✔  Approvals: 1
   ✔  - Antoine du Hamel (@aduh95) (TSC): https://github.com/nodejs/node/pull/42658#pullrequestreview-941531383
   ✖  This PR needs to wait 1 more hours to land (or 0 hours if there is one more approval)
   ✔  Last GitHub CI successful
   ℹ  Last Full PR CI on 2022-04-15T13:39:53Z: https://ci.nodejs.org/job/node-test-pull-request/43514/
- Querying data for job/node-test-pull-request/43514/
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  Aborted `git node land` session in /home/runner/work/node/node/.ncu
https://github.com/nodejs/node/actions/runs/2172929813

@nodejs-github-bot nodejs-github-bot added the commit-queue-failed label Apr 15, 2022
@cjihrig cjihrig added commit-queue and removed commit-queue-failed labels Apr 15, 2022
@nodejs-github-bot nodejs-github-bot added commit-queue-failed and removed commit-queue labels Apr 15, 2022
@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Apr 15, 2022

Commit Queue failed
- Loading data for nodejs/node/pull/42658
FetchError: Invalid response body while trying to fetch https://api.github.com/graphql: Premature close
    at consumeBody (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/node_modules/node-fetch/src/body.js:234:60)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Response.text (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/node_modules/node-fetch/src/body.js:158:18)
    at async Request.json (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/request.js:49:18)
    at async Request.query (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/request.js:107:20)
    at async Request.queryAll (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/request.js:134:20)
    at async Request.gql (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/request.js:64:22)
    at async PRData.getComments (file:///opt/hostedtoolcache/node/16.14.2/x64/lib/node_modules/node-core-utils/lib/pr_data.js:97:21)
    at async Promise.all (index 2)
    at async Promise.all (index 1) {
  type: 'system',
  errno: 'ERR_STREAM_PREMATURE_CLOSE',
  code: 'ERR_STREAM_PREMATURE_CLOSE',
  erroredSysCall: undefined
}
https://github.com/nodejs/node/actions/runs/2173428186

@Trott
Copy link
Member

@Trott Trott commented Apr 15, 2022

@cjihrig I think you'll either need to land this manually or wait for one more approval (or for @aduh95 to re-approve). The commit-queue won't land a PR if a commit has been pushed without a subsequent approval from someone.

aduh95
aduh95 approved these changes Apr 15, 2022
@aduh95 aduh95 added commit-queue and removed commit-queue-failed labels Apr 15, 2022
@nodejs-github-bot nodejs-github-bot removed the commit-queue label Apr 15, 2022
@nodejs-github-bot nodejs-github-bot merged commit adaf602 into nodejs:master Apr 15, 2022
57 checks passed
@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Apr 15, 2022

Landed in adaf602

@cjihrig cjihrig deleted the test-cli branch Apr 15, 2022
xtx1130 pushed a commit to xtx1130/node that referenced this issue Apr 25, 2022
This commit introduces an initial version of a CLI-based
test runner.

PR-URL: nodejs#42658
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
targos pushed a commit that referenced this issue Apr 28, 2022
This commit introduces an initial version of a CLI-based
test runner.

PR-URL: #42658
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
targos added a commit that referenced this issue May 2, 2022
Notable changes:

doc:
  * add @kuriyosh to collaborators (Yoshiki Kurihara) #42824
lib,src:
  * (SEMVER-MINOR) implement WebAssembly Web API (Tobias Nießen) #42701
test_runner:
  * (SEMVER-MINOR) add initial CLI runner (Colin Ihrig) #42658
worker:
  * (SEMVER-MINOR) add hasRef() to MessagePort (Darshan Sen) #42849

PR-URL: #42943
@targos targos mentioned this pull request May 2, 2022
targos added a commit that referenced this issue May 3, 2022
Notable changes:

doc:
  * add @kuriyosh to collaborators (Yoshiki Kurihara) #42824
lib,src:
  * (SEMVER-MINOR) implement WebAssembly Web API (Tobias Nießen) #42701
test_runner:
  * (SEMVER-MINOR) add initial CLI runner (Colin Ihrig) #42658
worker:
  * (SEMVER-MINOR) add hasRef() to MessagePort (Darshan Sen) #42849

PR-URL: #42943
@sosoba
Copy link

@sosoba sosoba commented May 4, 2022

Who will extend the @types/node package with a new node:test module?

@Trott
Copy link
Member

@Trott Trott commented May 4, 2022

Who will extend the @types/node package with a new node:test module?

As always, the maintainers of https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author ready dont-land-on-v14.x dont-land-on-v16.x dont-land-on-v17.x lib / src needs-ci semver-minor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet