Update --labels switch with option to show real-time pass/fail results in console runner

[dybs]

When running a large number of test cases, it would be nice to see the pass/fail status of each case in the console as it completes. Using --labels=All, I can see which tests have run, but the results are not available until all cases have completed. Seeing which cases fail during a long scenario would allow me to investigate the failures while the remaining tests continue to run. It also has the advantage of knowing which cases passed/failed if the test scenario happens to crash and doesn't get to write the result file.

Per the discussion in #1735 some ideas are to modify --labels with the following options:

• Results -> same behavior as =All, but includes a Pass/Fail/Error status
• Before -> shows the test case before it executes, and appends the status after the test completes. Similar to On.
• After -> shows the test case and status together after the test case completes.

[CharliePoole]

I like this idea, but I would modify/simplify it by preference:

• Off No change
• On No change
• Before / All shows test label when the test starts executing.
• After Shows all tests after they have completed, including status.

Note that All is described as it is supposed to work already. There's currently a bug in that it runs at the end of a test. We should fix that. We should keep All so as not to break existing usage.

The introduction of After means that we have to treat it as also meaning On. That is, if a test produces output, there should be a label before that output and another one after it, together with the test status..

The change should be made in both the console runner and nunitlite. Once we split the repos, this issue will need to be duplicated.

The fix is relatively easy. Depending on who takes it, I can mentor them as needed.

[constructor-igor]

Hi @CharliePoole,

Probably, I can start to work on the issue.
I checked all existing options of "--labels" parameter. Please, could you help me to understand suggested changes? I added sample with "--lablels=All". As I understand, the issue suggests to add "status" of each test?

thank you,
Igor.

[CharliePoole]

Hi @constructor-igor - I've been meaning to poke you to see what you are up to. :-)

As you probably figured out, all the code is in Console's TestEventHandler and in NUnitLite's TestRunner. There are tests for TestEventHandler but not for TestRunner. :-(

I would first off fix the problem with All, which is supposed to be issued at the start of each test, not at the end. I think I broke this when I was fixing something else.

Once All is correct, you've got Before, since it's intended to work the same way.

After is the new thing, which is supposed to show status of the test. For that, I would use a combination of the result and label attributes to display a set of values like...

• Passed
• Failed
• Error
• Invalid
• Ignored
• Explicit
• Skipped
• Inconclusive
  In NUnitLite, you'll have the ResultState available to use for equivalent logic.

[constructor-igor]

I am going to start from "All".
But, right now, I don't understand expected behavior of "All".
So, when I'll investigate code I'll contact with you.

[CharliePoole]

You might try this sequence - a bit different order from my first suggestion:

• Change "All" in existing code everywhere to "After" because All is now incorrectly running after the test.
• Add status info to After
• Add "Before" to run before the test
• Add back "All" as a synyonym for Before

[constructor-igor]

I try to build nunit by "build.cmd":

VS2013:

[CharliePoole]

Are you using the latest master? Can you build in the IDE?

[constructor-igor]

I cloned today and tried to build via build.cmd and IDE VS2013: same errors.

[CharliePoole]

Using VS 2015?

[rprouse]

I just got latest from master and ran .\build.cmd and it builds fine for me. Maybe do a full clean including deleting your packages folder and rebuild?

[constructor-igor]

I could build nunit solution on other computer.
I tried to build on my computer again (downloaded master to new folder) and see the error.

I investigate the issue and found, probably, same issues

• The predefined type 'System.Runtime.CompilerServices.ExtensionAttribute' is defined in multiple assemblies in the global alias nunit3-vs-adapter#194
• TestContext.TestDirectory throws for .net 2.0 DLL docs#44

I continue my investigation.

[CharliePoole]

Issue moved to nunit/nunit-console #24 via ZenHub