Skip to content

Commit

Permalink
Add --fail-skip/todo opts, better skip reporting
Browse files Browse the repository at this point in the history
This reports 'no tests found' as the skip reason for any test that does
not have a parent (so, typically, just the root TAP test) and also emits
no tests with a 1..0 plan, even if it lacks a skipReason in the plan.

Also, if `failSkip` is set in the test options (defaulting to
`env.TAP_FAIL_SKIP` in the root test) then any skipped test points will
be treated as failures. Ditto todo tests with `failTodo`.

This makes it possible to run `tap --fail-skip --fail-todo -s todo.txt`
to easily work through a backlog of skipped tests.

Doesn't exactly play nicely with --grep and --only, but it's hard to
know what to do in that case. You're telling tap that you want to fail
if any tests are skipped, and also telling it to skip tests. It seems
reasonable to assume that you don't want it to fail for tests skipped
with --grep or --only, since it should be as if those tests just didn't
*exist*, so probably going to need to figure out another way to signal
this.

Fix: #892
  • Loading branch information
isaacs committed Sep 19, 2023
1 parent ff9cc67 commit 8b7752d
Show file tree
Hide file tree
Showing 52 changed files with 485 additions and 104 deletions.
12 changes: 12 additions & 0 deletions src/config/src/jack.ts
Original file line number Diff line number Diff line change
Expand Up @@ -410,9 +410,21 @@ export default jack({
passes: {
description: `Include passing tests in assertion summary reports.
Has no effect on TAP output if a reporter is not used, but will
cause passing assertions to be included in the \`Test.lists\` collection
by default.
Note: this usually makes test output QUITE noisy.`,
},

'fail-todo': {
description: `Treat \`# TODO\` tests as failures`,
},

'fail-skip': {
description: `Treat \`# SKIP\` assertions as failures`,
},

color: {
short: 'c',
description: 'Use colors (Default for TTY)',
Expand Down
10 changes: 10 additions & 0 deletions src/config/tap-snapshots/test/jack.ts.test.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,14 @@ Object {
"multiple": true,
"type": "string",
},
"fail-skip": Object {
"description": "Treat \`# SKIP\` assertions as failures",
"type": "boolean",
},
"fail-todo": Object {
"description": "Treat \`# TODO\` tests as failures",
"type": "boolean",
},
"files": Object {
"description": "Alternative way to specify test set rather than using positional arguments. Supported as an option so that test file arguments can be specified in .taprc and package.json files.",
"multiple": true,
Expand Down Expand Up @@ -228,6 +236,8 @@ Object {
"description": String(
Include passing tests in assertion summary reports.
Has no effect on TAP output if a reporter is not used, but will cause passing assertions to be included in the \`Test.lists\` collection by default.
Note: this usually makes test output QUITE noisy.
),
"type": "boolean",
Expand Down
8 changes: 8 additions & 0 deletions src/core/dist/commonjs/base.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,14 @@ export interface BaseOpts extends Extra {
* track passes in the results lists, default false
*/
passes?: boolean;
/**
* treat todo tests as failures, default false
*/
failTodo?: boolean;
/**
* treat skip tests as failures, default false
*/
failSkip?: boolean;
/**
* The amount of time that this test took to complete.
*
Expand Down
2 changes: 1 addition & 1 deletion src/core/dist/commonjs/base.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion src/core/dist/commonjs/base.js.map

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion src/core/dist/commonjs/clean-yaml-object.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions src/core/dist/commonjs/clean-yaml-object.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 8b7752d

Please sign in to comment.