Discovery/Dry Run?

[JustinGrote]

Is there support for discovering what tests are available but not actually run them? I want to make a vscode extension integrating this into the Testing API but I have no way as far as I can tell to provide a list of tests to selectively run without first running them all.

[aduh95]

I'm not sure this is possible, there will be cases where it's simply not possible:

let someVar;

await test('some test', () => {
  someVar = {};
});

someVar.key = 'value';

await test('other test');

In the above case, I don't think there's a way for us to discover other test without running the first test.

/cc @nodejs/test_runner

[cjihrig]

I think it's impossible to do this with 100% accuracy, and anything we would implement would be a best effort.

• We could set the function for each test to a no-op, but then we would miss out on any subtests.
• We could execute describe() blocks but not it() blocks, with the understanding that describe() is just a container for it()s, but if someone breaks that contract by putting a throw in the describe(), that would break the dry run. This also has the same problem as the previous point if someone is using subtests inside of it().
• We could statically analyze the source code, but due to how dynamic JavaScript is, we could still miss things.

[JustinGrote]

@cjihrig thanks for the feedback, I asked because Mocha/Jest/etc. all have dry run options so I wasn't sure if this would. Without it there would be no way to write a useful test runner extension for running specific tests so I'll abandon that work for now.

[cjihrig]

I asked because Mocha/Jest/etc. all have dry run options

My understanding is that those limitations I listed apply to all test runners. We could implement the same thing for node:test, but it's not implemented at this point in time.

[JustinGrote]

I asked because Mocha/Jest/etc. all have dry run options

My understanding is that those limitations I listed apply to all test runners. We could implement the same thing for node:test, but it's not implemented at this point in time.

Understood, it would at least give them a starting point for basic test discovery, and then running a specific test could then reveal subtests in the UI, but you wouldn't have to run the full test suite just to find out what individual tests you can run.

[cjihrig]

I would support implementing one or both of the first two points in #51 (comment), with the understanding that subtests will not be reported and it's possible to break the dry run. This issue tracker is the wrong place for the feature request though.

[JustinGrote]

Understood, I'll close this issue and you can point me to where a feature request should be made. Thank you for your engagement and your time, it is appreciated.

[cjihrig]

You can create a feature request here: https://github.com/nodejs/node/issues/new?assignees=&labels=feature+request&projects=&template=2-feature-request.yml