You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
On this example we can see that start-time and end-time dates have a wrong date value "0001-01-01 00:00:00Z ". I would expect to have the date of the execution even if they are skipped. Moreover, if you try to publish these dates to TFS API it will fail because of the date value.
This issue is very easy to reproduce , just adding a [Ignore("some ignore reasons")] on a [TextFixture]. And running Nunit Console to generate the xml result output.
To clarify the issue, @alexamz is referring to the tests inside ignored test-suites.
On these tests, as they are never touched, StartTime is never set, and therefore the default value of DateTime.MinValue is written to the results xml instead. I agree the current behaviour is incorrect, although we need to decide what to do to fix this - cc @nunit/framework-team
The most correct thing to do here would seem to be to not write the start-time/end-time/duration attributes at all, when a test was never started. I imagine that make break a few existing TestResult file readers however, which are expecting these attributes to always be present.
@alexamz - out of interest, would you be able to test if TFS would accept such a file with these attributes removed? e.g. as below. Of course, TFS handling this has no effect on what other applications would handle, but it would be interesting to know.
I tried to do that, but the TFS does not accept a TRX without a valid date. I mean, I have tried several ways:
Create a TRX as you mentioned, without the start-time, end-time and duration.
Create a TRX with these fields, but with an empty value.
In my opinion, the best approach would be to use the current date and time. If the test run is ignored, use the system date in the start-time and the end time. Obviously, the duration would be 0.
The example I have pasted is directly the Xml generated by NUnit Console when I call it to test a dll assembly. The TRX that I mentioned is a post step, only parses the XML to another XML (the TRX).
I never used the VS adapter at least for now.
As a workaround, in the XML parser, we validate the dates and if are wrong like now, we use the current date, then we generate the TRX uses an XSL that parses the XML generated by the NUnit Console and we upload to TFS. The best approach, as I mentioned, I think is using the current date or something like that, because TFS needs a well formatted date in order to upload the test results and publish them.
I first was of the same mind as @ChrisMaddock that the best thing would be for tests we didn't run to not have start and end dates. However, thinking about it, it's actually possible for a test to run for some time before the code decides it should be ignored and issues an Assert.Ignore. Other tests, are ignored using an attribute and so are never run. So probably the "second-best" solution is to use the same start and end times, set to DateTime.Now at the point where we decide they are not being run. That will give them times that are consistent with any containing test suites. That is, I think we want the common start/end time to be >= the start time of a containing suite and <= its end time.
Relabelling this issue, as I think we've done the design stage.
For clarity - I believe we've decided that tests within an ignored test-suite should have a proper start-time and end-time set, which reflect the time the decision was made to ignore them. (How that's to be implemented - I'm less sure of.) These times would be expected to fall within the start and end time of the containing test-suite.
Marking this as up-for-grabs - Pull Requests welcome. @alexamz? 😄
I recently ran into this problem by Ignoring SpecFlow tests and took me a while to track it back to this work item. @alexamz Are you still working on a resolution for this issue?
In my case, when tests' result state is 'skipped' due to being explicit, would it make sense to omit the (currently-defaulted) start-time and end-time attributes?
For others coming across this problem: I was able to bypass this problem by running a PowerShell script which replaces the wrong date "0001-01-01 00:00:00Z" with a valid date like "1970-01-01 00:00:00Z" before publishing the test results. But obviously this is more of a "hack" than an actual solution.
This has been assigned to @alexamz for three years, without any further contact from him since then. Perhaps it should be freed up for someone else to work on?
Activity
ChrisMaddock commentedon Jul 31, 2017
To clarify the issue, @alexamz is referring to the tests inside ignored test-suites.
On these tests, as they are never touched, StartTime is never set, and therefore the default value of DateTime.MinValue is written to the results xml instead. I agree the current behaviour is incorrect, although we need to decide what to do to fix this - cc @nunit/framework-team
The most correct thing to do here would seem to be to not write the start-time/end-time/duration attributes at all, when a test was never started. I imagine that make break a few existing TestResult file readers however, which are expecting these attributes to always be present.
@alexamz - out of interest, would you be able to test if TFS would accept such a file with these attributes removed? e.g. as below. Of course, TFS handling this has no effect on what other applications would handle, but it would be interesting to know.
alexamz commentedon Jul 31, 2017
Hi @ChrisMaddock
Many thanks for your interest.
I tried to do that, but the TFS does not accept a TRX without a valid date. I mean, I have tried several ways:
In my opinion, the best approach would be to use the current date and time. If the test run is ignored, use the system date in the start-time and the end time. Obviously, the duration would be 0.
Many thanks!
Alex
CharliePoole commentedon Jul 31, 2017
@alexamz NUnit does not create the TRX file however. The question is really what should we be giving VS through the test explorer API.
This may take experimentation on our part to determine.
However, this is sounding like a VS adapter issue. Before we move it again, can you confirm that?
alexamz commentedon Aug 1, 2017
Hi @CharliePoole
The example I have pasted is directly the Xml generated by NUnit Console when I call it to test a dll assembly. The TRX that I mentioned is a post step, only parses the XML to another XML (the TRX).
I never used the VS adapter at least for now.
As a workaround, in the XML parser, we validate the dates and if are wrong like now, we use the current date, then we generate the TRX uses an XSL that parses the XML generated by the NUnit Console and we upload to TFS. The best approach, as I mentioned, I think is using the current date or something like that, because TFS needs a well formatted date in order to upload the test results and publish them.
Many thanks!
Alex
CharliePoole commentedon Aug 1, 2017
OK, then we are in the correct repository here.
I first was of the same mind as @ChrisMaddock that the best thing would be for tests we didn't run to not have start and end dates. However, thinking about it, it's actually possible for a test to run for some time before the code decides it should be ignored and issues an
Assert.Ignore. Other tests, are ignored using an attribute and so are never run. So probably the "second-best" solution is to use the same start and end times, set toDateTime.Nowat the point where we decide they are not being run. That will give them times that are consistent with any containing test suites. That is, I think we want the common start/end time to be >= the start time of a containing suite and <= its end time.alexamz commentedon Aug 1, 2017
Hi @CharliePoole
I think your proposal makes sense so yeah, we are happy with that!
Many thanks!
Alex
ChrisMaddock commentedon Aug 9, 2017
Relabelling this issue, as I think we've done the design stage.
For clarity - I believe we've decided that tests within an ignored test-suite should have a proper start-time and end-time set, which reflect the time the decision was made to ignore them. (How that's to be implemented - I'm less sure of.) These times would be expected to fall within the start and end time of the containing test-suite.
Marking this as up-for-grabs - Pull Requests welcome. @alexamz? 😄
alexamz commentedon Aug 9, 2017
Hi @ChrisMaddock ,
First of all, thanks to all the NUnit Team!
I will fork the project and try to fix this as soon as possible, but you know, it's a bit complicated to find time sometimes! Haha
Thanks again!
Alex
2 remaining items
rprouse commentedon Aug 14, 2017
@alexamz I have invited you to the NUnit contributors team so that one of us can assign this issue to you once you accept.
alexamz commentedon Aug 16, 2017
Hi @rprouse
Many thanks! I have already accepted.
ChrisMaddock commentedon Aug 16, 2017
Have assigned the issue to you @alexamz - look forward to seeing something. 😄
Quintinon commentedon Feb 26, 2018
I recently ran into this problem by Ignoring SpecFlow tests and took me a while to track it back to this work item.
@alexamz Are you still working on a resolution for this issue?
jnm2 commentedon May 25, 2018
In my case, when tests' result state is 'skipped' due to being explicit, would it make sense to omit the (currently-defaulted) start-time and end-time attributes?
HejdasGonnaHate commentedon Sep 24, 2020
For others coming across this problem: I was able to bypass this problem by running a PowerShell script which replaces the wrong date "0001-01-01 00:00:00Z" with a valid date like "1970-01-01 00:00:00Z" before publishing the test results. But obviously this is more of a "hack" than an actual solution.
CharliePoole commentedon Sep 24, 2020
This has been assigned to @alexamz for three years, without any further contact from him since then. Perhaps it should be freed up for someone else to work on?
shepps79 commentedon Dec 1, 2020
@mikkelbu I have some time come available. I'll have a look into this for you if you like.
mikkelbu commentedon Dec 4, 2020
Thanks @poimen. I've assigned the issue to you.