Attaching files to the test result

[giang-hai-nguyen]

Hi guys,
I am new with NUnit and i am wondering that how can i add attachment file to output in test explorer.
In unit test, there is a method in TestContext class call AddResultFile that can do this.
is there anybody can help me out.

thanks,
giang

[CharliePoole]

The method you mention is part of the TestContext class in Microsoft's test framework. NUnit also has a class of the same name, but it has no relation to the MsTest class and doesn't have such a method. There is no facility in NUnit to attach files to the output.

[giang-hai-nguyen]

Thank you Charlie.
But will AddResultFile method add to NUnit in the future?
This method really helpful.

[CharliePoole]

There is no plan for it at this time. If you would like to change this issue into a request for such a feature, we can do that and re-open it.

If so, you should explain in more detail how it would work. How are files created, where are they stored, etc. Are you asking that NUnit actually create the files or simply retain a reference to them?

Of couse, we would have to accept the idea, prioritize it and schedule it for some release.

[giang-hai-nguyen]

Thank you for your response Charlie.
It would be great if that method is added to NUnit.
it would work just like AddResultFile method of TestContext class in Microsoft's test framework.
something likes link the test result (file html test result or screenshot) to the Test Output in Test Explorer in VS. You can see the screenshot as below.

[CharliePoole]

Thanks for the idea. I'll reopen this.

To take it further, however, we would have to figure out how it works for NUnit itself, not merely the adapter. I'm marking this as an Idea, which is what we call something that is not well enough defined to consider as a future feature. Maybe other folks will chime in with suggestions.

Things we need to consider:

• How would this look in the XML result
• Can we package all this in some way that the result and the attachments are kept together?
• What would the NUnit console do with it, if anything.
• Would the NUnit gui display it in some way

[fredgate]

This feature is a must have.
We are running our automated behavior tests with NUnit and Selenium. If a test fail, we love to attach a screenshot of the browser in the test result : this is a great help to fix the test.
To do this, we had to use Microsoft's test framework, but it is not as powerfull than NUnit (attribute at assembly or class level, generic test class...). So we look forward to this feature in NUnit.

[CharliePoole]

I'm changing this to a feature needing design, which is more solid than an idea. We still need to resolve the questions listed in my earlier comment.

[ChrisMaddock]

I guess it's either serialising the file in the xml, or keeping a referenced directory besides it. One leads to vast xml, general slowness, and the files requiring a specific interpreter, the other leads to the possibility of files being separated from the results, and the references broken.

Which is the lesser of evils? I feel like with the former, the pain will outweigh the benefits of keeping everything in package - and having 'hopefully available' relative links will suffice.

• What would the NUnit console do with it, if anything.
• Would the NUnit gui display it in some way

For the GUI, something like the above interface looks nice enough, just hyperlinks to the attached file. The console, imo, should notify the user in the failed-test message that a file is available, and it's location.

I imagine this will require someone to submit a PR, once the design has been worked out - @fredgate, you interested? 😄

[rprouse]

We should look at how MSTest stores the files. I expect that the Visual Studio extension requires a link to the file on the drive.

[CharliePoole]

Seems like a reference in the XML is the way to go. For NUnit, the logical initial location is the work directory, which is where the XML goes by default. That resolves my first two points above.

Answering my other points:

• I guess the console would do no more than list the attachments with the path, just as it does for results.
• The gui could have some sort of menu or panel to list attachments. It could display them provided a default application has been defined for the particular file type.

Seems like a good feature, which I would lean towards raising to normal priority.

@rprouse Do you recall how Gallio handled attachments?

[CharliePoole]

What does an attachment element need to include in terms of attributes and elements? Something like this?

<attachments>
    <attachment  description="Final screen shot">ScreenShot.jpg</attachment>
    ...
</attachments>

I'm assuming it is a child of the test that produced it, so we have that info. Users may want to include multiple screen shots, hence the description. Anything else?

What is the syntax for attaching something? Should we use TestContext? Should there be a way to associate it with an assert so that the attachment is automatically created on failure?

Does somebody want to write a spec for this?

[rprouse]

Seems like a good feature, which I would lean towards raising to normal priority.

👍

@rprouse Do you recall how Gallio handled attachments?

No, I've never used attachments. I like your XML format. Personally, I think we should start with adding attachments from the TestContext as it is the simpler route. We can then evaluate if we want to expand that out to Asserts.

[CharliePoole]

So TestContext.CurrentContext.AddAttachment ?

With overloads for (string path) and (string path, string description) ?

[CharliePoole]

Maybe we have a spec already!

[skliamis]

Please raise the priority of this feature. It would be very helpful for our test reults summary to attach UI screenshots. Would be these attachments available in the build result inside VSTS (Visual Studio Team Services)?
Thank u very much Charlie

[NikolayPianikov]

I think it is a good feature and I will publish attachments as build artifacts in TeamCity

[asrafiq]

I hope nobody has forgotten about this. The would be a very helpful feature.

[CharliePoole]

I think the prior comments resolved all the design issues, so I'm taking off the design label and adding the implementation issue to Backlog.

It's not so much that we have forgotten this as that we have a large number of things going on. As I mentioned in an earlier comment, normal priority items like this one are not scheduled as part of our release planning. It's up to some team member or outside contributor to get interested enough that they take it on. Hopefully, adding it to the backlog will motivate somebody to step up. Depending on your skills, that could be you! 😄

[ghost]

great feature! would be much appreciated. hopefully someone implements this soon 😄

[ChrisMaddock]

I've just done a PR for the required framework functionality to implement this, in #2152. Given there's a few people tracking this issue, I'm planning to convert this issue to an Epic and track runner support, once we decide what to do there.

Here's my thoughts:

VS Adapter
This is a feature MSTest has, so the VS adapter framework already has functionality to show attached files. Just need to track down how to plug that in. 🙂

Console/NUnitLite
I'm in mixed minds here. With Errored/Warning tests, it makes sense to show an attachment's filepath/description with the result. How about attachments to passed tests? Should we have an 'Information' output to show these? So the console runners would have three types of output - Information/Warnings/Errors?

GUI/Xamarin
Both these should eventually support this, but I don't think it's a high priority. I'll create issues, but not attach them to this Epic - they can be done when there is demand/time allows.

TeamCity
@NikolayPianikov - you said you were interested in this. Maybe worth you tracking #2152. 🙂

Thoughts on all the above? @nunit/framework-team ? I'll make individual issues once we've decided the next steps.

[CharliePoole]

@ChrisMaddock Issues can be design issues as well so you could do one that says "Figure out how to support attachments in the gui" - same for any others where you are unsure.

For the console runner, we do not currently show all tests, although that could be done. Any detail report can be handled by an extension or a custom XSL transform. The latter would be trivial to implement for anyody who is willing to do XSL coding. 😄 It could be a contributed transform.

NUnitLite is tricky, since we want to keep it very light. However, one assumes that the users will name their attachments in an intelligent way and put them all in the same directory, so they should be easy enough to find. In fact, that works for the console runner as wel
I think the natural thing for the gui is to show the name and description of the attachments along with everything else about the test result. We could make it a link if the user has an appropriate application installed that will open the file.

Once we have attachments, I think it's very easy to add them to the results we pass to Visual Studio from the adapter.

[ChrisMaddock]

I've created nunit/docs#197 and nunit/nunit3-vs-adapter#332 to cover the next steps.

Regarding runner support - if the only current calls for this are integration with the VS adapter, I'm inclined just to implement it there, for now. It's a feature that fits better with graphical runners, then cli - and other runner support can always be added in the future, should people have a need.

[kdubau]

@ChrisMaddock I would love to see attachment support from the console runner.

[ChrisMaddock]

@kdubau - perfect timing! 😄 In that case, what would you want it to look like? What would be your use case?

I could imagine adding an 'attachments' block of text to test failures/warnings, listing description and file path. One reason I was debating not doing this, is that I wasn't convinced many users would want to copy/paste a filepath out of their shell, to find the respective file. Would you disagree?

The other issue to think about, is that the console currently only displays failed/warned tests, and not all tests. Should we create a third list of results ('Information'?) which shows all passed-tests-with-attachments?

[ChrisMaddock]

Maybe it would actually be ok the console runner only showing attachments for failed tests, as that's possibly all that's interesting in a console run. I'm imagining relevant log files for failures here, and such.

[CharliePoole]

@kdubau Rather than describing it here, I suggest you create a new issue in the nunit-console repo.

[kdubau]

@ChrisMaddock @CharliePoole I might have misunderstood the context here. I'm only interested in having the attachments referenced in the output XML - no need for actually displaying the attachment information in the console output. My use case is to have my CI system (Visual Studio Team Services) be able to upload the attachments listed in the XML as build artifacts (so it would just need a reference to the file on the local disk).

I was concerned running tests via the console runner would not have the attachments in the XML, but upon reading further I don't think this was the question. Please correct me if I'm wrong :)

[ChrisMaddock]

@kdubau - You're right. 🙂 #2152 has just been merged, which supports attachments in the results xml. If all goes to plan, that will be part of NUnit 3.7, released next week.

[CharliePoole]

Right... if we need "support" in the console, that would mean doing something more than simply saving the XML that comes from the framework. In the case of the nunit3 adapter, I guess @ChrisMaddock is defining what that will mean.

[kdubau]

Hey folks, is this work done or more pending? I couldn't find any open issues or PR's related to it.

Apologies if I'm missing something but my test attachments still are not appearing in the VS Test Explorer as I expect. I am using NUnit3TestAdapter 3.8.0 from NuGet and the API TestContext.AddTestAttachement(file) but I don't see the attachments in the Test Explorer. I am expecting something like this:

Are my expectations wrong?

[ChrisMaddock]

@kdubau - Nope, this should be released in NUnit v3.7 and NUnit3TestAdapter 3.8. See nunit/nunit3-vs-adapter#332 for the implementation. This issue just remains open as I still haven't got round to adding the two sentences of documentation required... 😱

If it's not working, I'd first run your tests in NUnit console - and see if your attachment is recorded correctly in the results xml file. If not, open an issue in this repo, if it is, open an issue in the adapter repo, and we can investigate further. 🙂

[kdubau]

@ChrisMaddock okay, issue created nunit/nunit3-vs-adapter#373

[ChrisMaddock]

Thanks to motivation from @kdubau, I've finally documented this feature, and can close this Epic. 😄