Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Support Skipped Test Cases #17

plioi opened this Issue Oct 30, 2013 · 2 comments


None yet
2 participants

plioi commented Oct 30, 2013

At the time of this writing, test cases either Pass or Fail. The closest a convention author could come to defining skipped tests themselves would be to exclude test methods entirely, via the test discovery filter. This approach misuses the test discovery feature, makes conventions needlessly complex, and hides skipped tests from the overall counts. Hiding skipped tests from the results increases the chance that a developer will forget about their skipped tests.

  • Fixie should support three states for each test case result: Passed, Failed, and Skipped.
  • Test case counts should include counts of all 3 states.
  • Convention authors should have control over how to determine that a test is skipped.
  • Console output should take into account all 3 states. Skipped tests should be highlighted similar to compiler warnings.
  • TestDriven.NET integration should take into account all 3 states.
  • TeamCity integration should take into account all 3 states. TeamCity supports messages of the form: ##teamcity[testIgnored name='testname' message='ignore comment'].

Speculating that this will motivate turning the Case class into an abstract base class with multiple implementations, and convention authors will have a hook allowing them to construct any Case type for each discovered test method.

@ghost ghost assigned plioi Oct 30, 2013


mexx commented Nov 24, 2013

The simplest way would be to check the cases in Convention before they are handed to the execution.
Here we can simply check whether the case should be skipped and inform the listener about the skipped cases.

I would not add extra case result state, currently they are used only to define in which way to inform the listener. But there is a little problem, because the Case holds its execution result, it would be accessible indirect in the listener. Currently there is no such indirect usage. In my opinion execution results should be returned from execution and the case should not know anything about it. Should we refactor it this way?

Do you have some syntax suggestions for the convention configuration?


plioi commented Nov 27, 2013

@mexx: Today, I pushed some changes that finally separate Case from the execution results of running that Case. I think this addresses the concerns you raise in your second paragraph.

A skip convention might look like this:

CaseExecution.Skip(Func<Case, bool>)

I like your idea of not making case execution itself extra complex just to support skips. I had previously thought of Skipped-ness as a funky kind of result, but you're right: it isn't. A skipped test is really just one that is deliberately never executed in the first place, and Convention is probably the right place to make that decision. They don't need to be run, they just need to be set aside, counted, and vigorously reported to the user!

@mexx mexx referenced this issue Nov 28, 2013


Skip cases #24

@plioi plioi closed this in #24 Nov 30, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment