Resolve differences between nunitlite and console runner

[CharliePoole]

We have a number of issues that relate to this so I'm creating an Epic to bundle them all together.

In this issue itself, I'd like to decide once and for all what our vision is for NUnitLite. Over the years it has varied substantially as some of us have tried to restrict it and others have wanted to add as many features as can be supported. I'll follow up with my own vision in a separate comment. Hopefully others will do the same.

[CharliePoole]

@nunit/framework-team Please let me know if you think any other outstanding issues should be included in this Epic.

[CharliePoole]

Here are my own thoughts on NUnitLite's direction and vision...

HISTORY

This may help explain any very old references you see to NUnitLite.

Originally, NUnitLite was a separate product. It included a version of the NUnit framework with reduced functionality and a built-in runner. With NUnit 3.0, we decided to converge the two products. The framework no longer varies - NUnitLite uses the same NUnit framework - and the name NUnitLite now refers to an alternate runner that is useful in simple cases.

The forked build of NUnitLite used in some Xamarin projects is based on that older NUnitLite concept, not on the newer design of NUnit/NUnitLite 3.

VISION [My personal]

1. NUnitLite should offer a subset of the features of the console runner. New features should not be added unless they are needed specifically in the NUnitLite environment.

2. New framework features should be accessible through NUnitLite as well as the console runner.

3. Command-line options for NUnitLite should be a subset of those for the console runner. The only exception would be if we find some specific option applies because of how NUnitLite works.

4. If a command-line option is valid for both the console and NUnitLite, it should work the same way in both cases.

5. Optionally, it would be desirable to recognize unsupported console runner options under NUnitLite for the sole purpose of giving a special error message. (This can't be done effectively, however, so long as we are using the unmodified Mono.Options code because defining an option would cause the unsupported options to appear in the help!)

6. NUnitLite is intended for use by self-executing tests. Consequently, it should not support listing test assemblies on the command-line. For situations where it isn't possible or practical to use a self-executing test, we currently support nunitlite-runner, which does allow the file to appear on the command-line.

7. NUnitLite currently references the version of the framework for which it is built. It would be desirable to change this so that we make some use of reflection and no longer need to rebuild for each version. In particular, this would assist us in the environment where the tests use a different runtime from that used by the system under test.

[rprouse]

It has been worrying me that the NUnitLite runner has become so close in functionality to the console. Others may disagree, but I see it's purpose as enabling self-executing tests for a single test assembly. For that reason, I don't mind that it is tightly bound to a specific framework.

As I think @CharliePoole states, I think that it should only have command line options to select and run individual tests and to support options in the framework like test parameters. Anything else belongs in the console or other full-featured runners.

Based on that, I could even see it being just a TestExecuter class that is included in the framework for use if you want it.

Others clearly have differnt use cases for it though, so I would like to hear them. I understand @oznetmaster's use cases, but I don't think they are a normal use case. He needs a full featured runner for a specific environment. To me, that should be a port of nunit-console, not an expansion of the responsibilities of NUnitLite.

I think we have allowed too many features to creep into NUnitLite and because of that we are suffering a maintenance issue duplicating code.

I would propose that we start removing and scaling back features. For example, remove the file feature at the command line, the ability to run multiple test fixtures, NUnit 2 output format, etc.

We are a small team and I would prefer to see us concentrating on improving the framework and fixing bugs rather than maintaining parity between the console runner and NUnitLite.

[CharliePoole]

@rprouse It sounds as if we have pretty much the same view of what NUnitLite should be - see my VISION comment above. In case anybody reading this is just going by the title... "Resolve differences..." is not intended to suggest adding features to NUnitLite that are not there. Rather, it's about common features.

Where both have the same feature, that feature should work the same way under both runners. That's not now the case, for example, with with minor usage differences. If we have a feature in NUnitLite and the Console, it should work the same way in both cases. As a (bad) example: the use option files on the command-line is supported in both NUnitLite and the console runner, but works differently in each case both from a user point of view and in how it is implemented.

I can see de-featuring NUnitLite in some cases, but I think we need to discuss each feature separately. Of the ones @rprouse suggests...

I would propose that we start removing and scaling back features. For example, remove the file feature at the command line, the ability to run multiple test fixtures, NUnit 2 output format, etc.

The file feature is discussed in #1877 and nunit/nunit-console#126. I'm already working on the code for both, having laid out a general direction and received some comments. Removing the feature is a different direction, so let's have the discussion sooner rather than later!

@rprouse I'm not sure if you mean running multiple test fixtures or assemblies? What would you want to remove. Maybe we need an issue on this as a child of the epic.

I'd be happy to remove NUnit2 output format from nunitlite but I think it gets some use by TC users. We should discuss - maybe another issue.

The code for reporting was originally part of the nunitlite framework, before we merged it with the nunit framework. Having it in a separate assembly was necessary to support usage where the execution environment didn't match the target environment of the tests. We make some use of this in our own CI builds so we would have to look at it more closely.

[CharliePoole]

I completed work on #1877 and created a PR. We should resolve the future of the arguments file feature in NUnitLite before merging or deleting that PR.

[ChrisMaddock]

We use NUnitLite for all our simple .NET Framework tests, purely as it allows the executables to be self-executing, and doesn't require the installation of any dependencies on downstream machines. (Including CI servers, and test assemblies included in distributions.) We simply don't need any of the additional power of the console - and this gives us projects we can just hit 'run' on in Visual Studio.

We use the NUnitLite NUnit 2 format to integrate with our crusty old CruiseControl.NET CI server. If this was removed, we'd need to switch to using the console for all assemblies instead - that's perfectly possible, but would be a substantial breaking change for our use case.

I agree the current code duplication needs to be sorting, but I'm not sure whether 'de-featuring' is the best solution, versus establishing a policy for future enhancements.

Is there anything we can revisit with regards to sharing code here? (submodules, separate utility assembly etc.)

[rprouse]

I don't think we should make a rush decision on this. @ChrisMaddock your idea of not defeaturing now is probably the best solution, I probably shouldn't have thrown a wrench in the process by stating my feelings so strongly 😄

How about we continue to resolve the differences in the existing features as per this epic and have a policy going forward where we do not allow new features unless they are exposing new functionality in the framework (like the test params) or are bringing the various nunitlite projects inline with each other (adding back functionality to .NET Standard for example).

If we agree with this, then let's move forward with the outstanding PRs.

[ChrisMaddock]

Sounds good to me.

On that basis, I think #344 should be removed from this epic, and #961 closed - that will become a maintenance cost, unless we can share the options somewhere.

[CharliePoole]

@rprouse I agree with that approach. In particular, I don't think it makes sense if we add a feature to the framework without supporting it in all runners. For example, it wouldn't make sense if nunitlite didn't know anything about warnings, since it couldn't show the messages in that case.

@ChrisMaddock I added those two to the epic so we could resolve them - not necessarily by implementing, but make a decision and close them one way or the other. Removing #341 from the epic without closing it doesn't move us forward.

I have some thoughts about sharing between the two runners, which I would like to work on separately, once I have become separate that is. 😄

[CharliePoole]

@rprouse This Epic has one remaining issue in it out of the original eight. I re-assigned it to you only so you could decide what to do about it.

[rprouse]

The last issue in this epic, #341 is closed, so closing the epic.