-
-
Notifications
You must be signed in to change notification settings - Fork 269
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add options to treat todo/skip as failures #892
Comments
My workaround has been to either put an artificial |
isaacs
added a commit
that referenced
this issue
Sep 18, 2023
This reports 'no tests found' as the skip reason for any test that does not have a parent (so, typically, just the root TAP test) and also emits no tests with a 1..0 plan, even if it lacks a skipReason in the plan. Also, if `failSkip` is set in the test options (defaulting to `env.TAP_FAIL_SKIP` in the root test) then any skipped test points will be treated as failures. Ditto todo tests with `failTodo`. This makes it possible to run `tap --fail-skip --fail-todo -s todo.txt` to easily work through a backlog of skipped tests. Doesn't exactly play nicely with --grep and --only, but it's hard to know what to do in that case. You're telling tap that you want to fail if any tests are skipped, and also telling it to skip tests. It seems reasonable to assume that you don't want it to fail for tests skipped with --grep or --only, since it should be as if those tests just didn't *exist*, so probably going to need to figure out another way to signal this. Fix: #892
isaacs
added a commit
that referenced
this issue
Sep 18, 2023
Add --fail-skip/todo opts, better skip reporting This reports 'no tests found' as the skip reason for any test that does not have a parent (so, typically, just the root TAP test) and also emits no tests with a 1..0 plan, even if it lacks a skipReason in the plan. Also, if `failSkip` is set in the test options (defaulting to `env.TAP_FAIL_SKIP` in the root test) then any skipped test points will be treated as failures. Ditto todo tests with `failTodo`. This makes it possible to run `tap --fail-skip --fail-todo -s todo.txt` to easily work through a backlog of skipped tests. Doesn't exactly play nicely with --grep and --only, but it's hard to know what to do in that case. You're telling tap that you want to fail if any tests are skipped, and also telling it to skip tests. It seems reasonable to assume that you don't want it to fail for tests skipped with --grep or --only, since it should be as if those tests just didn't *exist*, so probably going to need to figure out another way to signal this. Fix: #892
isaacs
added a commit
that referenced
this issue
Sep 18, 2023
This reports 'no tests found' as the skip reason for any test that does not have a parent (so, typically, just the root TAP test) and also emits no tests with a 1..0 plan, even if it lacks a skipReason in the plan. Also, if `failSkip` is set in the test options (defaulting to `env.TAP_FAIL_SKIP` in the root test) then any skipped test points will be treated as failures. Ditto todo tests with `failTodo`. This makes it possible to run `tap --fail-skip --fail-todo -s todo.txt` to easily work through a backlog of skipped tests. Doesn't exactly play nicely with --grep and --only, but it's hard to know what to do in that case. You're telling tap that you want to fail if any tests are skipped, and also telling it to skip tests. It seems reasonable to assume that you don't want it to fail for tests skipped with --grep or --only, since it should be as if those tests just didn't *exist*, so probably going to need to figure out another way to signal this. Fix: #892
isaacs
added a commit
that referenced
this issue
Sep 19, 2023
This reports 'no tests found' as the skip reason for any test that does not have a parent (so, typically, just the root TAP test) and also emits no tests with a 1..0 plan, even if it lacks a skipReason in the plan. Also, if `failSkip` is set in the test options (defaulting to `env.TAP_FAIL_SKIP` in the root test) then any skipped test points will be treated as failures. Ditto todo tests with `failTodo`. This makes it possible to run `tap --fail-skip --fail-todo -s todo.txt` to easily work through a backlog of skipped tests. Doesn't exactly play nicely with --grep and --only, but it's hard to know what to do in that case. You're telling tap that you want to fail if any tests are skipped, and also telling it to skip tests. It seems reasonable to assume that you don't want it to fail for tests skipped with --grep or --only, since it should be as if those tests just didn't *exist*, so probably going to need to figure out another way to signal this. Fix: #892
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In most normal cases, the right thing to do is treat
todo
andskip
tests as if they were passing, regardless of status.And, an empty test (or just a file that doesn't produce any TAP test points) is treated as a skipped test.
This is almost always the right thing to do, but it prevents a workflow that I'm sort of finding to be nice.
src/*.ts
map.js
which doesexport default t => t.replace(/^test/, 'src')
for i in src/*; do touch ${i/src/test}; done
It would be nice if I could do this, but avoid running all the tests I wrote as I'm working through the list.
Then the
todo
andskip
tests would log as failures, and get saved to the file, and I could keep whittling it down until everything is done.Could also be useful as a sort of lint rule in CI, to avoid merging PRs that still have action items.
The text was updated successfully, but these errors were encountered: