Running tests directly should never surface a NullReferenceException

[jnm2]

We're gathering a few bug reports around this issue. People try to manually run or debug an NUnit test and get a NullReferenceException. This sends the message to the user that this is a bug we do not know about. It would be better if we surfaced an exception with a helpful message getting the user on the right track.

Even better than that, I wish we could fall back gracefully and continue. That is going to be the user's expectation and it allows conveniences during development such as ad-hoc debugging into a test on F5 like what's manually done here:

public static class Program
{
    public static void Main() => new TheFixture().TheTest();
}

[TestFixture]
public sealed class TheFixture
{
    [Test]
    public void TheTest() => Assert.That(...);
}

[CharliePoole]

@jnm2 I had two reasons for the change of title:

1. The method is not actually being called through an API here. Of course, to be precise, Assert is an API, but that's not how we describe it when we tell people there are three APIs for running tests in the docs.

2. It's usually better to put issues in terms of the problem space rather than the solution space - i.e., the internal TestExecutionContext. Of course, our users are also programmers, so that makes it tricky.

[CharliePoole]

@jnm2 In our discussion about this you had raised the issue of figuring out how we were called and actually running successfully, rather than giving an intelligent error message. Have you given up on that?

[jnm2]

@CharliePoole

Have you given up on that?

Not if you think the idea has merit! 😃

[CharliePoole]

@jnm2 If you haven't already noticed 😃 I'm big on JIT. I think this idea has merit but I don't know if it's feasible. If I decided to work on this, that point in time would be just-in-time for figuring out what's possible to do here. I'm just saying that we won't know until somebody takes on the issue.

Here's a bit of agilist propaganda for you...

One of the stated values of XP is Courage, which is taken to mean confidence in our skills and in being able to work things out as needed. Because of Courage, we don't have to plan things out in detail ahead of time, which we know doesn't work anyway. Of course, if your level of Courage doesn't match your actual abilities, you're toast. 😄

Anyway, yes, I think someone taking this on should look beyond the title and see if there is anything that can be done to make it work without bastardizing the code to much.

[jnm2]

@CharliePoole I had given up on convincing the team because of the last few discussions we've had a while back. In that case, since that would be the preferred fix and I think it'll be conceptually on the same order of difficulty, I'll shoot for that first and fall back to giving a message only if it's not feasible.