TAP: Should test.only() contribute to skipped count summary?

[therealklanni]

I personally think it should, as this would be useful information.

Currently, if you use the test.only() method, it doesn't tally any other test() calls as skipped and thus they go unreported (as if they don't exist).

Side note: maybe even useful in the normal AVA output?

7 passed
2 skipped

[BarryThePenguin]

Personally I use test.only() for debugging. If I use it on a single test, I only care about the result of that test.

In karma-spec-reporterthere are config values to suppress certain results. I turn report of skipped tests off in local development to stop the output being polluted with skipped test lines. Then I rely on the CI server to give me an accurate report of skipped tests.

specReporter: {
    maxLogLines: 5,                // limit number of lines logged per test 
    suppressErrorSummary: true,    // do not print error summary 
    suppressFailed: false,         // do not print information about failed tests 
    suppressPassed: false,         // do not print information about passed tests 
    suppressSkipped: true          // do not print information about skipped tests 
},

In summary I guess it's the job of the individual tap reporter to handle how to interpret the output. And AVA should be giving an accurate output of test results.

[therealklanni]

I agree that it's the job of the reporter to determine how to interpret the output, so yes, I think AVA should give an accurate output (i.e. report all skips).

[sindresorhus]

👍 This also catches the case of forgetting to remove an .only().

@therealklanni Interested in doing a PR?

[jamestalmage]

This also catches the case of forgetting to remove an .only().

I have a really crazy idea. Should the use of .only() cause a non-zero exit code? Or the ability to configure it to exit with a non-zero exit code in that case?

We could probably create a linter rule that handled warning about .only() usage, but I think it would be so nice if AVA just handled it for you.

[sindresorhus]

Should the use of .only() cause a non-zero exit code? Or the ability to configure it to exit with a non-zero exit code in that case?

I like the idea, but need to think about possible cases where you don't want it to fail. I'm also not very interested if it has to be configurable. Trying to keep config to a minimum.

We could probably create a linter rule that handled warning about .only() usage

Already planned regardless: avajs/eslint-plugin-ava#5

[therealklanni]

@sindresorhus I can possibly take a look at this sometime this week.

If anyone else wants to give it a go before I get to it, feel free!

[naptowncode]

Given the discussion around #565, how should the use of .only() affect the rest of the tests? Should they be counted in stats as skipped, just as if .skip() were used? Or something else altogether?

I'm just asking what the API should do when reporting stats, regardless of what the reporter decides to do with those stats.

I would assume that .only() would cause all other tests to be treated as skipped. TAP doesn't seem to have a concept of ONLY, just SKIP and TODO.

[vadimdemedes]

The rest of the tests should not be in the output at all. .skip() has nothing to do with .only().

[naptowncode]

But some information about the rest of the tests should be sent to the reporter, even if the reporter chooses not to output it to the user.

Skipping those tests would feel consistent with the implementation of --match in #477.

[novemberborn]

I can see .only being an inverse .skip, allowing you to home in on a particular failing test. In that case I wouldn't want to see any skipped tests (or todos) in the output, other than for the final summary. Though as @BarryThePenguin mentioned it might make sense to show them on CI servers.

Currently reporters aren't pluggable so we need to figure out what behavior we want from the built-in reporters. Presumably the test events could convey a test is being skipped due to a .skip or .todo or because there's an .only present elsewhere.