New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate xml file of test results as specified in .runsettings file #215

Closed
PriyankaDhamdhere opened this Issue Aug 5, 2016 · 17 comments

Comments

Projects
None yet
5 participants
@PriyankaDhamdhere
Copy link

PriyankaDhamdhere commented Aug 5, 2016

I am using Nunit adapter nuget package to integrate Nunit and visual studio. It is not generating TestResults.xml when i run nunit tests from test explorer.

I can do the same when i run it from Nunit3-console.exe.

@CharliePoole

This comment has been minimized.

Copy link
Member

CharliePoole commented Aug 5, 2016

You are correct. The adapter does not create an XML result file.

If we want it to do so, it would help to have some kind of spec. What should be the name, location and content of such a file. How will the user tell nunit to save the results.

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Aug 5, 2016

Yet another reason to create a settings UI for the extension 😄

@CharliePoole

This comment has been minimized.

Copy link
Member

CharliePoole commented Aug 5, 2016

Have we talked about such a ui elsewhere? I searched but found nothing. For us to have a ui, we would need to stop being just an adapter and replace at least some of the functionality of the test window.

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Aug 5, 2016

I'm not sure if it would work for the nuget package, but a VSIX can add a settings node to the Visual Studio options.

@CharliePoole

This comment has been minimized.

Copy link
Member

CharliePoole commented Aug 5, 2016

Right. But then we become an extension in our own right and not an adapter. Or some kind of hybrid.

It's not impossible but it is a major escalation. I've thought about doing that as a commercial product.

@PriyankaDhamdhere

This comment has been minimized.

Copy link

PriyankaDhamdhere commented Aug 5, 2016

@CharliePoole

Test results that I am looking for could be same as the one generated by console runner but more specifically want numbers like passed, failed, run, time took to run tests.

ways that I can think of for doing this –

  1.   To import a setting file from VS menu, same like mstest. This could allow us to set if we want to generate XML, its path etc.
    
  2.   To have config by default placed at root level of solution, which adapter can identify (something like NunitSetting.config) which will have filepath to generate results.
    
  3.   To have an UI for settings in Test Explorer
    
@CharliePoole

This comment has been minimized.

Copy link
Member

CharliePoole commented Aug 5, 2016

@PriyankaDhamdhere Thanks for the suggestions...

  1. NUnit already has a runsettings file, which allows you to set a number of things. However it doesn't allow specifying where the XML should go and it doesn't save to TestResult.xml by default. It's definitely possible to add this. It could either be a single setting, formatted like the console command-line argument, or be divided into multiple settings. This is probably the most direct approach and easiest to understand because it is how we handle other options.
  2. I'd rather not proliferate config files - we already have an xml file for lasting settings and the runsettings file for those that pertain to a particular project.
  3. See my earlier answer. This would mean expanding the adapter to be a true VS extension. In addition, I think Settings files are more appropriate for genral, cross-project settings, rather than for a single project.

@CharliePoole CharliePoole changed the title TestResults.xml is not generated when run tests through TestExplorer Generate xml file of test results as specified in .runsettings file Aug 5, 2016

@CharliePoole CharliePoole referenced this issue Oct 7, 2017

Merged

Dump xml #403

@LirazShay

This comment has been minimized.

Copy link

LirazShay commented Nov 20, 2018

Hi, I'm also need this functionality
I'm running SpecFlow tests with NUnit framework
There is a famous extension for BDD, it's called pickles -see http://www.picklesdoc.com/
It has msbuild task that create nice html to show the spec of the system, based of the feature files.
It can combine the nunit test result inside the html if a path was supplied.
It would be great to save each test run from VS in folder called test-results, then that extension will use it on build

@CharliePoole

This comment has been minimized.

Copy link
Member

CharliePoole commented Nov 20, 2018

@LirazShay User can control which path is used to save output under the console by specifying --work with a directory. I'm not sure if this is ported to the runsettings file, however.

@LirazShay

This comment has been minimized.

Copy link

LirazShay commented Nov 20, 2018

It's not required that the exact path will be determined by the settings. Even if it will be saved in default location - it will help me

@OsirisTerje

This comment has been minimized.

Copy link
Member

OsirisTerje commented Nov 20, 2018

@CharliePoole We're setting the WorkDirectory (https://github.com/nunit/docs/wiki/Tips-And-Tricks ) from runsettings to the engine:
package.Settings[PackageSettings.WorkDirectory] = workDir;

Is this the same as the --work parameter to the console? From the notes it says this is just the TestAssembly location, sort of the same as the BinDirectory. Or, does this setting do something else in the engine?

Looking more, we ALWAYS set this parameter to the engine, the runsettings just make it possible to change it. So, there must be another setting to generate the output. I know we discussed this earlier, and you mentioned some way to trigger this in the engine.

(I have never tried this option)

@CharliePoole

This comment has been minimized.

Copy link
Member

CharliePoole commented Nov 20, 2018

@OsirisTerje Yes, that's exactly right. Our WorkDirectory is the place where output files are intended to be saved for the run, whether created by NUnit or by the user, who can access the work directory using TestContext. It's different from TestDirectory, which is the directory containing the test assembly. For a run with multiple assemblies, there could be multiple TestDirectories, but only one WorkDirectory.

The engine has a facility to create the XML output file, but the runner has to call it. It isn't created automatically. You could do something like this...

var resultService = _engine.Services.GetService<IResultService>();
// Following null argument should work for nunit3 format. Empty array is OK as well.
// If you decide to handle other formats in the runsettings, it needs more work.
var resultWriter = resultService.GetResultWriter("nunit3", null); 
resultWriter.WriteResultFile(yourResult, yourOutputPath);
@OsirisTerje

This comment has been minimized.

Copy link
Member

OsirisTerje commented Nov 20, 2018

Thanks @CharliePoole ! Just what I need! I'll add this in, then at least we have the possibility for an output file implemented.

@OsirisTerje OsirisTerje self-assigned this Nov 20, 2018

@OsirisTerje OsirisTerje added this to the 3.12 milestone Nov 20, 2018

@LirazShay

This comment has been minimized.

Copy link

LirazShay commented Nov 21, 2018

@OsirisTerje
Thanks a lot for taking care of this!!!
@CharliePoole
Many thanks for your help!!

@OsirisTerje

This comment has been minimized.

Copy link
Member

OsirisTerje commented Dec 18, 2018

@LirazShay @PriyankaDhamdhere This should now work. There is a pre-release package at https://www.myget.org/feed/nunit/package/nuget/NUnit3TestAdapter/3.12.0-dev-01009 . Would appreciate if you could test it out. I am OOF from Wednesday evening, so if this is going out, it has to be tomorrow. To use it, add the runsettings file, with another NUnit property named TestOutput. That should point to a relative or absolute folder path where you want the output. I'll document the details later. In any case, there is then one result xml file for each test assembly.

@OsirisTerje

This comment has been minimized.

@LirazShay

This comment has been minimized.

Copy link

LirazShay commented Dec 20, 2018

@OsirisTerje
Great work!! thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment