TestResult.xml overwritten with multiple targets

[BrainCrumbz]

Hello there
first of all, thanks for your work with this runner.

It seems there's an issue with TestResult.xml output file.

Scenario:
A solution has a project-under-test, e.g. class library, targeting 2 frameworks: netstandard1.6 and net452. Same solution also has an NUnit test class library, targeting same 2 frameworks, depending on "dotnet-test-nunit": "3.4.0-beta-1" and "NUnit": "3.4.1".

When running tests from console, those are correctly run for both targets in sequence, as visible from console output:

> dotnet test
Project NSpec (.NETStandard,Version=v1.6) was previously compiled. Skipping compilation.
...
Project NSpecSpecs (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
NUnit .NET Core Runner 3.4.0
...
...
    Duration: 6,607 seconds
Results saved as my\project\path\TestResult.xml
Project NSpec (.NETFramework,Version=v4.5.2) was previously compiled. Skipping compilation.
...
Project NSpecSpecs (.NETFramework,Version=v4.5.2) was previously compiled. Skipping compilation.
NUnit .NET Core Runner 3.4.0
...
...
    Duration: 7,474 seconds
Results saved as my\project\path\TestResult.xml
SUMMARY: Total: 2 targets, Passed: 0, Failed: 2.

The issue is that TestResult.xml is reused by both and gets overwritten by last target being run.

Also, although this is even minor: maybe separation between output from two targets could be highlighted more, e.g. with some heading or spacing after each Results saved as ... line.

HTH

[rprouse]

Good catch, I never even thought about that. We could append a date string to the end of the run to differentiate although I always hated how MbUnit did that and filled up the output directory. We might also be able to figure out the target and append that. To the runner, each target is a separate independent run that doesn't know about the other.

[BrainCrumbz]

Re. Datestring:
besides that, from the point of view of tools coming later in a pipeline, datetime is also less predictable and harder somehow to manage.
Better appending target, if there's a way to grab that

[jcansdale]

The simplest way would perhaps be to save them in the codebase of the target test assemblies (e.g. bin\Debug\netcoreapp1.0 and bin\Debug\net451\win7-x64). Alternatively, we could append the TFM/RID like this: TestResult-netcoreapp1.0.xml / TestResult-net451-win7-x64.xml.

I'd be tempted to go with the first option.

[BrainCrumbz]

Not a bad idea to just store it there

[Addisco]

Hi there,
I also stumbled upon this.

My scenario is somehow different but same:
I've a solution with 3 projects. One is the project-under-test, one is for unit-tests and one is for integration-tests. So I don't have multiple targets, I have multiple test-projects and configured a build in VSTS.
I'm using the dotnet core build task (command: test) to run the tests.
Project(s) are set to "**/test/**/project.json".

The tests are all executed but the results of the integrations-tests are overridden by the results of the unit-tests. I might be able to solve this by using the --work parameter like mentioned in Is there a way to specify TestResult.xml location? #100, I'll keep you posted on that.

It would be great to have a better solution for this.

PS: Thanks for your work with this runner!