Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Support Skipped Test Cases #17

Closed
plioi opened this Issue · 2 comments

2 participants

@plioi
Owner

At the time of this writing, test cases either Pass or Fail. The closest a convention author could come to defining skipped tests themselves would be to exclude test methods entirely, via the test discovery filter. This approach misuses the test discovery feature, makes conventions needlessly complex, and hides skipped tests from the overall counts. Hiding skipped tests from the results increases the chance that a developer will forget about their skipped tests.

  • Fixie should support three states for each test case result: Passed, Failed, and Skipped.
  • Test case counts should include counts of all 3 states.
  • Convention authors should have control over how to determine that a test is skipped.
  • Console output should take into account all 3 states. Skipped tests should be highlighted similar to compiler warnings.
  • TestDriven.NET integration should take into account all 3 states.
  • TeamCity integration should take into account all 3 states. TeamCity supports messages of the form: ##teamcity[testIgnored name='testname' message='ignore comment'].

Speculating that this will motivate turning the Case class into an abstract base class with multiple implementations, and convention authors will have a hook allowing them to construct any Case type for each discovered test method.

@plioi plioi was assigned
@mexx

The simplest way would be to check the cases in Convention before they are handed to the execution.
Here we can simply check whether the case should be skipped and inform the listener about the skipped cases.

I would not add extra case result state, currently they are used only to define in which way to inform the listener. But there is a little problem, because the Case holds its execution result, it would be accessible indirect in the listener. Currently there is no such indirect usage. In my opinion execution results should be returned from execution and the case should not know anything about it. Should we refactor it this way?

Do you have some syntax suggestions for the convention configuration?

@plioi
Owner

@mexx: Today, I pushed some changes that finally separate Case from the execution results of running that Case. I think this addresses the concerns you raise in your second paragraph.

A skip convention might look like this:

CaseExecution.Skip(Func<Case, bool>)

I like your idea of not making case execution itself extra complex just to support skips. I had previously thought of Skipped-ness as a funky kind of result, but you're right: it isn't. A skipped test is really just one that is deliberately never executed in the first place, and Convention is probably the right place to make that decision. They don't need to be run, they just need to be set aside, counted, and vigorously reported to the user!

@mexx mexx referenced this issue
Merged

Skip cases #24

@plioi plioi closed this in #24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.