Add runner feature to allow forcing to run skipped tests

[bradwilson]

No description provided.

[niemyjski]

+1

[niemyjski]

Is there any progress on this? I might not want to run a test on my ci server (benchmark) but may want to run it locally.

[clairernovotny]

You could do this today with SkippableFact based off of an environment variable. Just skip if some variable is/isn't present and run when it's not/is.

https://github.com/AArnott/Xunit.SkippableFact
https://www.nuget.org/packages/xunit.skippablefact

[drewnoakes]

@niemyjski I share your use case too. Some tests require MSMQ to be installed, which isn't available on the CI servers (AppVeyor/TravisCI).

I also have some tests that require running with administrative permissions on the machine, and only want to run them when I am running as admin. SkippableFact might help with that.

[bradwilson]

Closed for age.

[niemyjski]

I get that it was closed for age, just really annoying limitation currently.

[bradwilson]

https://twitter.com/bradwilson/status/929046059640225793

[yolofy]

+1

[pbernini]

+1

[i3anaan]

Would be a very nice feature to have

[videokojot]

+1 very annoying when you want to run something, you need to edit code...

For example we have tests which should be run only by hand as they are doing some expensive things (such as paid (expensive) api calls etc)

[MichaelGehling]

I would also appreciate to be able to run my skipped tests manually without changing the source code each time!

[markm77]

JetBrains Rider used to allow manually running skipped Xunit tests but stopped allowing this in Rider 2020.2 which is a big pain. See: https://youtrack.jetbrains.com/issue/RIDER-49097.

It is really important to be able to take tests out of the test suite e.g. temporarily and still be able to run them manually (e.g. to fix or check status).

@bradwilson Please consider this feature for #2133.

[ohroy]

+10085

[samcarswell]

+1

[sentouki]

+1

[markm77]

From my point of view, this functionality will be provided by "explicit" tests (#2518 ), looking forward to it!

[bradwilson]

This brings up a good philosophical question:

What's the difference between [Fact(Skip = "Reason")] and [Fact(Explicit = "Reason")]? Is it only that the former is intended to be temporary and the latter is intended to be permanent?

[markm77]

@bradwilson , I think that's a great question. Personally I think you've hit on what could potentially be a very nice distinction.

For me, there are two reasons I why I pull tests of my automated regression suite.

(1) because a test is broken and I can't immediately fix it (perhaps due to a third-party dependency or due to prioritisation). This removal is temporary in nature (to keep my test suite passing) and perhaps best described by "skip".

(2) because a test is a "special purpose" test, perhaps requiring some manual setup or special conditions, which is for manual use only. This removal is permanent in nature and perhaps best described by "explicit". (There are even some things like database "tasks" I might be tempted to add as "explicit" tests of this nature.... although I realise that's using xUnit for more than just testing....)

However when available I will probably move to "explicit" for both types of test removals due to the lack of the feature requested by this issue. : ) That is because it's a pain to have to make a code change to run "skipped" tests. Personally, if I were in your position, I would try to align "skip" and "explicit" as much as possible to allow code re-use on your side and present a common interface to users. Even if the two became identical, I still think having both with their potentially different meanings adds significant value by clearly communicating the status of a test.

[bradwilson]

I think the biggest difference is how things are presented in the Visual Studio UI (and the fact that you should be able to run an Explicit test by saying "run this", whereas there would be no way to request to run a Skip test without removing Skip). In my mind, this also aligns with your description of "broken" vs. "special purpose".

Thanks!

[markm77]

No problem, sure, makes sense.

(Rider used to allow running of skipped xUnit tests (I use both Rider and Visual Studio) which probably got some of us used to this before it was taken away. 😀)