Skip to content

Wrong date format in Ignored TestFixtures #2339

Closed

Description

@ChrisMaddock

@alexamz commented on Mon Jul 31 2017

Hi all,

First of all, many thanks to all the contributors to this great and useful project!

We have problems with the start-time and end-time dates in Ignored TestFixtures.
We are using the last version of the NUnit console, the 3.7.0.

This is an example of a test-case header in the xml generated by NUnit Console with a wrong formatted date:

<test-case id="0-1071" name="AfterStart_AClientMayConnect" fullname="Port.TCP.UnitTests.TcpServerFactoryTests.AfterStart_AClientMayConnect" methodname="AfterStart_AClientMayConnect" classname="Port.TCP.UnitTests.TcpServerFactoryTests" runstate="Runnable" seed="604943940" result="Skipped" label="Ignored" site="Parent" start-time="0001-01-01 00:00:00Z" end-time="0001-01-01 00:00:00Z" duration="0.000000" asserts="0">

On this example we can see that start-time and end-time dates have a wrong date value "0001-01-01 00:00:00Z ". I would expect to have the date of the execution even if they are skipped. Moreover, if you try to publish these dates to TFS API it will fail because of the date value.

This issue is very easy to reproduce , just adding a [Ignore("some ignore reasons")] on a [TextFixture]. And running Nunit Console to generate the xml result output.

Many thanks!

Have a nice day


@ChrisMaddock commented on Mon Jul 31 2017

The code for this lives in the NUnit Framework. I'll move this issue over, and comment further over there. 🙂

Activity

  1. ChrisMaddock commented on Jul 31, 2017

    @ChrisMaddock
    ContributorAuthor

    To clarify the issue, @alexamz is referring to the tests inside ignored test-suites.

    On these tests, as they are never touched, StartTime is never set, and therefore the default value of DateTime.MinValue is written to the results xml instead. I agree the current behaviour is incorrect, although we need to decide what to do to fix this - cc @nunit/framework-team

    The most correct thing to do here would seem to be to not write the start-time/end-time/duration attributes at all, when a test was never started. I imagine that make break a few existing TestResult file readers however, which are expecting these attributes to always be present.

    @alexamz - out of interest, would you be able to test if TFS would accept such a file with these attributes removed? e.g. as below. Of course, TFS handling this has no effect on what other applications would handle, but it would be interesting to know.

    <test-case id="0-1071" name="AfterStart_AClientMayConnect" fullname="Port.TCP.UnitTests.TcpServerFactoryTests.AfterStart_AClientMayConnect" methodname="AfterStart_AClientMayConnect" classname="Port.TCP.UnitTests.TcpServerFactoryTests" runstate="Runnable" seed="604943940" result="Skipped" label="Ignored" site="Parent" asserts="0">
    
  2. alexamz commented on Jul 31, 2017

    @alexamz

    Hi @ChrisMaddock

    Many thanks for your interest.

    I tried to do that, but the TFS does not accept a TRX without a valid date. I mean, I have tried several ways:

    • Create a TRX as you mentioned, without the start-time, end-time and duration.
    • Create a TRX with these fields, but with an empty value.

    In my opinion, the best approach would be to use the current date and time. If the test run is ignored, use the system date in the start-time and the end time. Obviously, the duration would be 0.

    Many thanks!

    Alex

  3. CharliePoole commented on Jul 31, 2017

    @CharliePoole
    Member

    @alexamz NUnit does not create the TRX file however. The question is really what should we be giving VS through the test explorer API.

    This may take experimentation on our part to determine.

    However, this is sounding like a VS adapter issue. Before we move it again, can you confirm that?

  4. alexamz commented on Aug 1, 2017

    @alexamz

    Hi @CharliePoole

    The example I have pasted is directly the Xml generated by NUnit Console when I call it to test a dll assembly. The TRX that I mentioned is a post step, only parses the XML to another XML (the TRX).
    I never used the VS adapter at least for now.

    As a workaround, in the XML parser, we validate the dates and if are wrong like now, we use the current date, then we generate the TRX uses an XSL that parses the XML generated by the NUnit Console and we upload to TFS. The best approach, as I mentioned, I think is using the current date or something like that, because TFS needs a well formatted date in order to upload the test results and publish them.

    Many thanks!

    Alex

  5. CharliePoole commented on Aug 1, 2017

    @CharliePoole
    Member

    OK, then we are in the correct repository here.

    I first was of the same mind as @ChrisMaddock that the best thing would be for tests we didn't run to not have start and end dates. However, thinking about it, it's actually possible for a test to run for some time before the code decides it should be ignored and issues an Assert.Ignore. Other tests, are ignored using an attribute and so are never run. So probably the "second-best" solution is to use the same start and end times, set to DateTime.Now at the point where we decide they are not being run. That will give them times that are consistent with any containing test suites. That is, I think we want the common start/end time to be >= the start time of a containing suite and <= its end time.

  6. alexamz commented on Aug 1, 2017

    @alexamz

    Hi @CharliePoole

    I think your proposal makes sense so yeah, we are happy with that!

    Many thanks!

    Alex

  7. ChrisMaddock commented on Aug 9, 2017

    @ChrisMaddock
    ContributorAuthor

    Relabelling this issue, as I think we've done the design stage.

    For clarity - I believe we've decided that tests within an ignored test-suite should have a proper start-time and end-time set, which reflect the time the decision was made to ignore them. (How that's to be implemented - I'm less sure of.) These times would be expected to fall within the start and end time of the containing test-suite.

    Marking this as up-for-grabs - Pull Requests welcome. @alexamz? 😄

  8. alexamz commented on Aug 9, 2017

    @alexamz

    Hi @ChrisMaddock ,

    First of all, thanks to all the NUnit Team!

    I will fork the project and try to fix this as soon as possible, but you know, it's a bit complicated to find time sometimes! Haha

    Thanks again!

    Alex

  9. 2 remaining items

  10. rprouse commented on Aug 14, 2017

    @rprouse
    Member

    @alexamz I have invited you to the NUnit contributors team so that one of us can assign this issue to you once you accept.

  11. alexamz commented on Aug 16, 2017

    @alexamz

    Hi @rprouse

    Many thanks! I have already accepted.

  12. ChrisMaddock commented on Aug 16, 2017

    @ChrisMaddock
    ContributorAuthor

    Have assigned the issue to you @alexamz - look forward to seeing something. 😄

  13. Quintinon commented on Feb 26, 2018

    @Quintinon

    I recently ran into this problem by Ignoring SpecFlow tests and took me a while to track it back to this work item.
    @alexamz Are you still working on a resolution for this issue?

  14. jnm2 commented on May 25, 2018

    @jnm2
    Contributor

    In my case, when tests' result state is 'skipped' due to being explicit, would it make sense to omit the (currently-defaulted) start-time and end-time attributes?

  15. HejdasGonnaHate commented on Sep 24, 2020

    @HejdasGonnaHate

    For others coming across this problem: I was able to bypass this problem by running a PowerShell script which replaces the wrong date "0001-01-01 00:00:00Z" with a valid date like "1970-01-01 00:00:00Z" before publishing the test results. But obviously this is more of a "hack" than an actual solution.

  16. CharliePoole commented on Sep 24, 2020

    @CharliePoole
    Member

    This has been assigned to @alexamz for three years, without any further contact from him since then. Perhaps it should be freed up for someone else to work on?

  17. shepps79 commented on Dec 1, 2020

    @shepps79
    Contributor

    @mikkelbu I have some time come available. I'll have a look into this for you if you like.

  18. mikkelbu commented on Dec 4, 2020

    @mikkelbu
    Member

    Thanks @poimen. I've assigned the issue to you.

  19. added this to the 3.13.1 milestone on Jan 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions