Runners which skip theory pre-enumeration by default should offer an override switch

[ryanbrandenburg]

In AspNetCore we've created ConditionalTheoryDiscoverer to allow us to skip specific TheoryData, and this has worked fine under vstest, but when we switched to building with xunit.console (and thus with PreEnumerateTheories=false) theory data that previously skipped no longer does.

I've created a pared down repro project to showcase this.
Repro steps:

1. Clone the above project.
2. cd Microsoft.AspNetCore.Testing/test; dotnet restore
3. dotnet xunit. See that a test which was expected to be skipped ran (and failed).
4. dotnet test. See that the appropriate test skipped.

We've found that this behavior can be avoided with this workaround but it seems less than ideal to just override that setting in some scenarios.

Is our implementation of CreateTestCasesForDataRow just wrong, and if so is there an example of a successful implementation of this scenario?

[bradwilson]

That's true. The system was not designed to do this, as the level of skipping is at the Test Case level, and when theory pre-enumeration is turned off, there is a single Test Case for the theory (rather than one per data row).

Renaming this issue to reflect the actual needed feature.

[bradwilson]

I have tagged both Console and MSBuild runners, though it's likely that actual location of the behavior will change for 3.0.

[bradwilson]

In-proc console runner: 4d4964c
Stand-alone console runner: c98c54a
MSBuild runner: 5cf67b2