Invalid platform name passed to PlatformAttribute should mark test NotRunnable

[ChrisMaddock]

From #2200 - if an invalid platform name is passed into PlatformAttribute, the test is currently skipped - the same as if a platform the test is not currently running on was passed in.

An invalid platform name would always be a mistake, so in my opinion should mark the test NotRunnable, and fail the test run.

Thoughts?

[CharliePoole]

Agree on included platforms, what about excluded?

[ChrisMaddock]

I think this should cover all lists. Excluding an invalid platform would currently skip the test on all platforms, which is potentially a hard-to-spot, wrong-side failure.

[CharliePoole]

I thought excluding an invalid platform caused it to run. I'm on my phone so I can't check.

[ChrisMaddock]

I haven't tried, it, but it looks like anything invalid throws an ArgumentException, which is caught by IsPlatformSupported(), which then returns false.

nunit/src/NUnitFramework/framework/Internal/PlatformHelper.cs

Line 135 in 077aa75

catch (Exception ex)

Either way, I still would prefer an invalid test, over the test running on a platform I didn't intend it to. 😄

[jnm2]

My opinion on correctness issues is always that it should fail as soon and as clearly as possible. 😄

[ChrisMaddock]

Relabelling this as an easyfix enhancement.

PlatformHelper currently throws and catches an Exception, for invalid framework names. In my opinion, that exception should instead be caught within PlatformAttribute, which can then mark the test as NotRunnable, with an appropriate reason.