Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New status: "expected failure" #72

Closed
waywardmonkeys opened this issue Sep 15, 2015 · 1 comment
Closed

New status: "expected failure" #72

waywardmonkeys opened this issue Sep 15, 2015 · 1 comment

Comments

@waywardmonkeys
Copy link
Member

We have a bunch of tests that we know fail today, but we'd like to ignore that and look at keeping the ones that pass passing and look at the current failures later. The way things are now, if a test starts to fail here or there, we won't notice in the sea of existing failures.

I'd like to add a new test result status $expected-failure. This also requires a matching ``$unexpected-success`. The idea is that we run the test, but then look at it to see if we expected it to fail or not and then return 1 of the new statuses.

First, via the test defining macro:

define test foo (expected-failure: #t)
  ...
end test;

There are also cases where we want to expect to fail in some situations, but not all:

define test foo (expected-failure-if: method () $os-name == #"win32" end)
  ...
end test;

However, this doesn't work for things using testworks-specs currently. For that, we need to land my testworks-specs changes that simplify it a lot and let us then modify things so that you can pass keywords to function-test and the other macros.

This will require some changes to how we output result summaries. We will also probably have to modify the reporting output that we use for integrating with Jenkins.
waywardmonkeys added a commit that referenced this issue Sep 15, 2015
@waywardmonkeys
Add expected-failure support.
b31b52e 
It is useful to be able to note that a test is expected to fail
and to note when it unexpectedly succeeds.  This will be useful
for the Dylan test suites where we have some long-standing failures
that can be flagged as expecting to fail so that we will notice
new failures or other changes in the status of tests.

Fixes bug #72.

* components.dylan
  (%expected-failure?): New slot on <runnable>, parent class for
    both <test> and <benchmark>.
  (expected-failure?): New method to see if a test is expected
    to fail. This inspects %expected-failure? and evaluates the
    method if there is one.

* results.dylan
  ($expected-failure, $unexpected-success): New status values.
  (<result-status>): Update to include new status values.
  (status-name): Update to handle new status values.

* reports.dylan
  (count-results): Count $expected-failure as a pass, $unexpected-success
    as a failure.

* coloring.dylan
  ($expected-failure-attributes, $unexpected-success-attributes): New
    text attributes.
  (result-status-to-attributes): Map new status values to new text attributes.

* run.dylan
  (execute-component on <suite>): Count an $expected-failure result as
    a passing result.
  (execute-component on <test>): Check expected-failure? to handle
    $passed and $failed correctly.

* library.dylan
  (module %testworks): Export $expected-failure and $unexpected-success.

* documentation/users-guide/source/reference.rst
  (test-definer, benchmark-definer): Update to discuss expected-failure?.

* documentation/users-guide/source/usage.rst
  (Tests): Update to discuss expected failures.

* tests/testworks-test-suite.dylan: Update to test expected failure and
  unexpected success scenarios.
@waywardmonkeys waywardmonkeys mentioned this issue Sep 15, 2015
Merged
@waywardmonkeys
Copy link
Member Author

This should be fine after #73.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@waywardmonkeys