Wrong date format in Ignored TestFixtures

[ChrisMaddock]

@alexamz commented on Mon Jul 31 2017

Hi all,

First of all, many thanks to all the contributors to this great and useful project!

We have problems with the start-time and end-time dates in Ignored TestFixtures.
We are using the last version of the NUnit console, the 3.7.0.

This is an example of a test-case header in the xml generated by NUnit Console with a wrong formatted date:

<test-case id="0-1071" name="AfterStart_AClientMayConnect" fullname="Port.TCP.UnitTests.TcpServerFactoryTests.AfterStart_AClientMayConnect" methodname="AfterStart_AClientMayConnect" classname="Port.TCP.UnitTests.TcpServerFactoryTests" runstate="Runnable" seed="604943940" result="Skipped" label="Ignored" site="Parent" start-time="0001-01-01 00:00:00Z" end-time="0001-01-01 00:00:00Z" duration="0.000000" asserts="0">

On this example we can see that start-time and end-time dates have a wrong date value "0001-01-01 00:00:00Z ". I would expect to have the date of the execution even if they are skipped. Moreover, if you try to publish these dates to TFS API it will fail because of the date value.

This issue is very easy to reproduce , just adding a [Ignore("some ignore reasons")] on a [TextFixture]. And running Nunit Console to generate the xml result output.

Many thanks!

Have a nice day

---

@ChrisMaddock commented on Mon Jul 31 2017

The code for this lives in the NUnit Framework. I'll move this issue over, and comment further over there. 🙂

[ChrisMaddock]

To clarify the issue, @alexamz is referring to the tests inside ignored test-suites.

On these tests, as they are never touched, StartTime is never set, and therefore the default value of DateTime.MinValue is written to the results xml instead. I agree the current behaviour is incorrect, although we need to decide what to do to fix this - cc @nunit/framework-team

The most correct thing to do here would seem to be to not write the start-time/end-time/duration attributes at all, when a test was never started. I imagine that make break a few existing TestResult file readers however, which are expecting these attributes to always be present.

@alexamz - out of interest, would you be able to test if TFS would accept such a file with these attributes removed? e.g. as below. Of course, TFS handling this has no effect on what other applications would handle, but it would be interesting to know.

<test-case id="0-1071" name="AfterStart_AClientMayConnect" fullname="Port.TCP.UnitTests.TcpServerFactoryTests.AfterStart_AClientMayConnect" methodname="AfterStart_AClientMayConnect" classname="Port.TCP.UnitTests.TcpServerFactoryTests" runstate="Runnable" seed="604943940" result="Skipped" label="Ignored" site="Parent" asserts="0">

[alexamz]

Hi @ChrisMaddock

Many thanks for your interest.

I tried to do that, but the TFS does not accept a TRX without a valid date. I mean, I have tried several ways:

• Create a TRX as you mentioned, without the start-time, end-time and duration.
• Create a TRX with these fields, but with an empty value.

In my opinion, the best approach would be to use the current date and time. If the test run is ignored, use the system date in the start-time and the end time. Obviously, the duration would be 0.

Many thanks!

Alex

[CharliePoole]

@alexamz NUnit does not create the TRX file however. The question is really what should we be giving VS through the test explorer API.

This may take experimentation on our part to determine.

However, this is sounding like a VS adapter issue. Before we move it again, can you confirm that?

[alexamz]

Hi @CharliePoole

The example I have pasted is directly the Xml generated by NUnit Console when I call it to test a dll assembly. The TRX that I mentioned is a post step, only parses the XML to another XML (the TRX).
I never used the VS adapter at least for now.

As a workaround, in the XML parser, we validate the dates and if are wrong like now, we use the current date, then we generate the TRX uses an XSL that parses the XML generated by the NUnit Console and we upload to TFS. The best approach, as I mentioned, I think is using the current date or something like that, because TFS needs a well formatted date in order to upload the test results and publish them.

Many thanks!

Alex

[CharliePoole]

OK, then we are in the correct repository here.

I first was of the same mind as @ChrisMaddock that the best thing would be for tests we didn't run to not have start and end dates. However, thinking about it, it's actually possible for a test to run for some time before the code decides it should be ignored and issues an Assert.Ignore. Other tests, are ignored using an attribute and so are never run. So probably the "second-best" solution is to use the same start and end times, set to DateTime.Now at the point where we decide they are not being run. That will give them times that are consistent with any containing test suites. That is, I think we want the common start/end time to be >= the start time of a containing suite and <= its end time.

[alexamz]

Hi @CharliePoole

I think your proposal makes sense so yeah, we are happy with that!

Many thanks!

Alex

[ChrisMaddock]

Relabelling this issue, as I think we've done the design stage.

For clarity - I believe we've decided that tests within an ignored test-suite should have a proper start-time and end-time set, which reflect the time the decision was made to ignore them. (How that's to be implemented - I'm less sure of.) These times would be expected to fall within the start and end time of the containing test-suite.

Marking this as up-for-grabs - Pull Requests welcome. @alexamz? 😄