Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add options to treat todo/skip as failures #892

Closed
isaacs opened this issue Sep 17, 2023 · 1 comment · Fixed by #901
Closed

Add options to treat todo/skip as failures #892

isaacs opened this issue Sep 17, 2023 · 1 comment · Fixed by #901

Comments

@isaacs
Copy link
Member

isaacs commented Sep 17, 2023

In most normal cases, the right thing to do is treat todo and skip tests as if they were passing, regardless of status.

And, an empty test (or just a file that doesn't produce any TAP test points) is treated as a skipped test.

This is almost always the right thing to do, but it prevents a workflow that I'm sort of finding to be nice.

  1. Create a new package, with a bunch of files at src/*.ts
  2. Create a coverage map file map.js which does export default t => t.replace(/^test/, 'src')
  3. Create an empty test for each source unit: for i in src/*; do touch ${i/src/test}; done
  4. Treat the skipped tests as my worklist.

It would be nice if I could do this, but avoid running all the tests I wrote as I'm working through the list.

$ tap --save todo.txt --fail-skip --fail-todo

Then the todo and skip tests would log as failures, and get saved to the file, and I could keep whittling it down until everything is done.

Could also be useful as a sort of lint rule in CI, to avoid merging PRs that still have action items.

@isaacs
Copy link
Member Author

isaacs commented Sep 17, 2023

My workaround has been to either put an artificial (await import('tap')).fail('todo') in each of the empty tests, or use find test -size 0 as my worklist, but both feel like hacks.

isaacs added a commit that referenced this issue Sep 18, 2023
This reports 'no tests found' as the skip reason for any test that does
not have a parent (so, typically, just the root TAP test) and also emits
no tests with a 1..0 plan, even if it lacks a skipReason in the plan.

Also, if `failSkip` is set in the test options (defaulting to
`env.TAP_FAIL_SKIP` in the root test) then any skipped test points will
be treated as failures. Ditto todo tests with `failTodo`.

This makes it possible to run `tap --fail-skip --fail-todo -s todo.txt`
to easily work through a backlog of skipped tests.

Doesn't exactly play nicely with --grep and --only, but it's hard to
know what to do in that case. You're telling tap that you want to fail
if any tests are skipped, and also telling it to skip tests. It seems
reasonable to assume that you don't want it to fail for tests skipped
with --grep or --only, since it should be as if those tests just didn't
*exist*, so probably going to need to figure out another way to signal
this.

Fix: #892
isaacs added a commit that referenced this issue Sep 18, 2023
x
Add --fail-skip/todo opts, better skip reporting

This reports 'no tests found' as the skip reason for any test that does
not have a parent (so, typically, just the root TAP test) and also emits
no tests with a 1..0 plan, even if it lacks a skipReason in the plan.

Also, if `failSkip` is set in the test options (defaulting to
`env.TAP_FAIL_SKIP` in the root test) then any skipped test points will
be treated as failures. Ditto todo tests with `failTodo`.

This makes it possible to run `tap --fail-skip --fail-todo -s todo.txt`
to easily work through a backlog of skipped tests.

Doesn't exactly play nicely with --grep and --only, but it's hard to
know what to do in that case. You're telling tap that you want to fail
if any tests are skipped, and also telling it to skip tests. It seems
reasonable to assume that you don't want it to fail for tests skipped
with --grep or --only, since it should be as if those tests just didn't
*exist*, so probably going to need to figure out another way to signal
this.

Fix: #892
isaacs added a commit that referenced this issue Sep 18, 2023
This reports 'no tests found' as the skip reason for any test that does
not have a parent (so, typically, just the root TAP test) and also emits
no tests with a 1..0 plan, even if it lacks a skipReason in the plan.

Also, if `failSkip` is set in the test options (defaulting to
`env.TAP_FAIL_SKIP` in the root test) then any skipped test points will
be treated as failures. Ditto todo tests with `failTodo`.

This makes it possible to run `tap --fail-skip --fail-todo -s todo.txt`
to easily work through a backlog of skipped tests.

Doesn't exactly play nicely with --grep and --only, but it's hard to
know what to do in that case. You're telling tap that you want to fail
if any tests are skipped, and also telling it to skip tests. It seems
reasonable to assume that you don't want it to fail for tests skipped
with --grep or --only, since it should be as if those tests just didn't
*exist*, so probably going to need to figure out another way to signal
this.

Fix: #892
isaacs added a commit that referenced this issue Sep 19, 2023
This reports 'no tests found' as the skip reason for any test that does
not have a parent (so, typically, just the root TAP test) and also emits
no tests with a 1..0 plan, even if it lacks a skipReason in the plan.

Also, if `failSkip` is set in the test options (defaulting to
`env.TAP_FAIL_SKIP` in the root test) then any skipped test points will
be treated as failures. Ditto todo tests with `failTodo`.

This makes it possible to run `tap --fail-skip --fail-todo -s todo.txt`
to easily work through a backlog of skipped tests.

Doesn't exactly play nicely with --grep and --only, but it's hard to
know what to do in that case. You're telling tap that you want to fail
if any tests are skipped, and also telling it to skip tests. It seems
reasonable to assume that you don't want it to fail for tests skipped
with --grep or --only, since it should be as if those tests just didn't
*exist*, so probably going to need to figure out another way to signal
this.

Fix: #892
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant