Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for `dotnet test path/to/my.sln` #411

Open
TheRealPiotrP opened this Issue Jan 31, 2017 · 22 comments

Comments

Projects
None yet
@TheRealPiotrP
Copy link
Contributor

commented Jan 31, 2017

As folks are spending more time with the new command line experience, the desire for a sln-level test command is growing.

Some desirable characteristics:

  • parallelism! allow MSBuild's /m to control the number of projects to test in parallel
  • Unified output. Provide a summary of how the overall test run did, not just individual projects
@ErikSchierboom

This comment has been minimized.

Copy link

commented Mar 6, 2017

I would love to see this implemented! It will makes build scripts just a little bit shorter and easier.

@bernardbr

This comment has been minimized.

Copy link

commented May 11, 2017

In my case, I solved it with a python script. I've tried to do it with a PowerShell script, but I got restriction error.

Python solution:

import subprocess
from argparse import ArgumentParser

parser = ArgumentParser()
parser.add_argument("-f", "--files", nargs="+", dest="files", help="Define a .csproj file list.")

args = parser.parse_args()

for f in args.files:
	subprocess.call(r"C:\Program Files\dotnet\dotnet.exe" +  " test " + f)

PowerShell solution:

param(
    [Parameter(Mandatory = $true, Position = 0)]
    [string[]]$csProjFiles
)    

foreach ($csProjFile in $csProjFiles)
{
    dotnet test $csProjFile
}

My task.json (VsCode):

{
    // See https://go.microsoft.com/fwlink/?LinkId=733558
    // for the documentation about the tasks.json format
    "version": "0.1.0",
    "isShellCommand": true,
    "args": [],
    "tasks": [
        {
            "taskName": "build",
            "command": "dotnet",
            "args": [
                "build",
                "solution.sln"
            ],
            "isBuildCommand": true,
            "showOutput": "silent",
            "problemMatcher": "$msCompile"
        },
        {
            "taskName": "test",
            "command": "py",
            "args": [
                "${cwd}/run_tests.py",
                "-f",
                "${cwd}/test/unit/model/model.csproj",
                "${cwd}/test/unit/repository/repository.csproj"
            ],
            "isTestCommand": true,
            "showOutput": "silent"
        }
    ]
}
@tomasaschan

This comment has been minimized.

Copy link

commented Jun 11, 2017

Unified output would be really nice, but could prove quite complex given the wide range of testing frameworks and how they present their output. However, there's another killer reason for me to want this: reduced test running time. It's quite easy to find the project files and pass each of them to dotnet test, but doing that triggers a new build for each project, meaning that in a larger solution with many test projects, I have to wait for many builds just to run the tests.

@tverboon

This comment has been minimized.

Copy link

commented Jun 12, 2017

It's quite easy to find the project files and pass each of them to dotnet test, but doing that triggers a new build for each project, meaning that in a larger solution with many test projects, I have to wait for many builds just to run the tests.

@tlycken There is already a --no-build option. We build our solution first and then run the tests parallel. It's much faster that way. Only negative, is that you need to find a failing test in the output, so this doesn't scale well.

#!/usr/bin/env bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
CD $DIR/../
dotnet build

find $DIR/../test -not -path '*.TestCommon*' -name '*.csproj' -print0 | xargs -L1 -0 -P 8 dotnet test --no-build

if [ "$?" !=  "0" ] ; then
    echo -e "######### TEST FAILED #########"
fi
@bromanko

This comment has been minimized.

Copy link

commented Jul 10, 2017

This suggestion worked well for me.

@Kritner

This comment has been minimized.

Copy link

commented Jul 20, 2017

I liked the loop solutions to getting it accomplished however, when i was doing it with powershell for my teamcity builds, it seems that team city doesn't really understand "pass vs failure" on a test when run via powershell

i think it only understand when running as a "dotnet test" task, rather than a "powershell" task that just happens to invoke dotnet test

So while I find the loops clever, I can really do the same thing with a GUI test runner - I feel it's more important on the solution level so that my dotnet test task can work. Is there another option in this regard?

@tomasaschan

This comment has been minimized.

Copy link

commented Jul 22, 2017

@Kritner TeamCity (and other build systems) usually infer the success or failure state of script steps from their exit codes; if you amend the PS script to return an exit code that's 0 if all tests passed and something else otherwise, you could probably get it working.

That said, I still want this feature :)

@Kritner

This comment has been minimized.

Copy link

commented Jul 22, 2017

Thanks @tlycken I'll have to give that a go

@ErikSchierboom

This comment has been minimized.

Copy link

commented Aug 8, 2017

Although the scripting solution is a workaround, it is precisely that. To see the problem with this approach, consider doing the same for building a set of projects. Building the projects individually instead of as part of a solution is much, much slower. This is the main reason I'd like to be do dotnet test <name>.sln: improved performance.

@thenathanjones

This comment has been minimized.

Copy link

commented Aug 15, 2017

+1. I'm trying to do an automated build on a CI server, and while the loops "work" they are indeed a workaround. If I'm grouping them in a solution, it's logical that I should be able to run all the projects within that group. I'll work around it, but I feel that some other people may just use it as an excuse to just create a mega project and not deal with the problem.

@tverboon

This comment has been minimized.

Copy link

commented Aug 15, 2017

My script was intended for local use, as it runs all tests in parallel. On the CI server you can call it with MSBuild. Put the following target in a MSBuild project file.

<Target Name="TestAll">
    <ItemGroup>
        <TestProjects Include="$(MSBuildThisFileDirectory)test\**\*.csproj" Exclude="$(MSBuildThisFileDirectory)test\TestCommon\**\*.csproj" />
    </ItemGroup>
    <MSBuild Projects="@(TestProjects)" Targets="vstest" Properties="VSTestNoBuild=true" ContinueOnError="ErrorAndContinue"/>
</Target>

The VSTestNoBuild property will prevent a build.

You can call it like this: dotnet msbuild YourProject.build /t:TestAll to test it.

Only thing is that running all test assemblies in parallel doesn't seem to work.

@hizhubo

This comment has been minimized.

Copy link

commented Aug 17, 2017

@tverboon Error "error MSB4057: The target "vstest" does not exist in the project." occurred after applied your msbuild approach. Is it necessary to manually add target "vstest" in all test projects?

@tverboon

This comment has been minimized.

Copy link

commented Aug 17, 2017

@hizhubo It is part of Microsoft Test platform: https://github.com/Microsoft/vstest/blob/master/src/Microsoft.TestPlatform.Build/Microsoft.TestPlatform.targets.

My test projects target netcoreapp2.0 and have a package reference to Microsoft.NET.Test.Sdk.

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.3.0" />
@hizhubo

This comment has been minimized.

Copy link

commented Aug 17, 2017

@tverboon
My project has the same reference (as shown below) but the issue error MSB4057: The target "vstest" does not exist in the project. still exists. Would you please help me to check if there is any issue in my project, batch script or MSBuild script? Thanks a lot.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.3.0" />
    <PackageReference Include="Microsoft.TestPlatform.Build" Version="15.3.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="1.1.18" />
    <PackageReference Include="MSTest.TestFramework" Version="1.1.18" />
    <PackageReference Include="NETStandard.Library" Version="2.0.0" />
    <PackageReference Include="System.Runtime" Version="4.3.0" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="ProjectToBeTested.csproj" />
  </ItemGroup>
  
</Project>

The command line to invoke MSBuild

"C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe" "%CurrentDir%NugetPackage.sln.Build.proj"

The MSBuild script snippet

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Build">
...
  <Target Name="RunTests">
    <ItemGroup>
      <TestProjects Include="$(MSBuildThisFileDirectory)Test\**\*.Test.csproj" />
    </ItemGroup>
    <MSBuild Projects="@(TestProjects)" Targets="VSTest" Properties="VSTestNoBuild=true" ContinueOnError="ErrorAndContinue"/>
  </Target>
...
</Project>
@tverboon

This comment has been minimized.

Copy link

commented Aug 18, 2017

@hizhubo Your MSBuild files look good to me. Could you try to run it with the dotnet 2.0 cli toolchain?

First build your solution, packages will be restored automatically:
dotnet build NugetPackage.sln

Then run your test target:
dotnet msbuild NugetPackage.sln.Build.proj /t: RunTests

@hizhubo

This comment has been minimized.

Copy link

commented Aug 18, 2017

@tverboon dotnet msbuild NugetPackage.sln.Build.proj /t: RunTests works! Whereas "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe" NugetPackage.sln.Build.proj /t:RunTests doesn't

I guess it's because that MSBuild.exe 15.0 is not dotnet cli toolchain and target VSTest somehow needs to be running under dotnet 2.0 cli.

I changed my command line as shown below and now it works like a charm. Thanks a lot!

dotnet msbuild "%CurrentDir%NugetPackage.sln.Build.proj"
@dasMulli

This comment has been minimized.

Copy link

commented Sep 5, 2017

@Faizan2304 @codito Since the Test SDK NuGet package sets $(IsTestProject) to true, the VSTest target could condition on that:

=>

<Target Name="VSTest" Condition="'$(IsTestProject)' == 'true'">

This works quite well in my manual testing, are there any drawbacks this would introduce?

@brunolm

This comment has been minimized.

Copy link

commented Jul 4, 2018

Looks like specifying a solution is already supported in 2.1.2

> dotnet test .\dotnetcore-how-to.Tests.sln
Build started, please wait...
Build completed.

Test run for D:\BrunoLM\Projects\dotnetcore-how-to\dotnetcore-how-to.Tests\bin\Debug\netcoreapp2.0\dotnetcore-how-to.Tests.dll(.NETCoreApp,Version=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 15.5.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 0.8438 Seconds

> dotnet --version
2.1.2
@tomasaschan

This comment has been minimized.

Copy link

commented Jul 5, 2018

@brunolm does that solution contain any projects that are not test projects? I have 2.1.300, and it does allow dotnet test targeting a solution, but attempts to run tests in all projects, not just the test projects, and then spews out ugly errors for all non-test projects.

@Kritner

This comment has been minimized.

Copy link

commented Jul 5, 2018

@tlycken Yeah, still same issue here. If you add Microsoft.NET.Test.Sdk to all projects, not just test projects, this seems to be enough to get the solution to report an overall "success". It's kinda gross putting a Test.SDK on a non test project, but it's a workaround for the moment.

Be prepared if you go this route to get lots of localization stuff in your build if you didn't already have it (at least I think it's from this nuget package)

@brunolm

This comment has been minimized.

Copy link

commented Jul 5, 2018

@tlycken I have 2 .sln files

dotnetcore-how-to.sln          // all
dotnetcore-how-to.Tests.sln    // tests only

When I run the test sln only my test project is run.

Not as good as having 1 sln and having dotnet automatically figure out what to do, but it works

@toqueteos

This comment has been minimized.

Copy link

commented Feb 15, 2019

@brunolm does that solution contain any projects that are not test projects? I have 2.1.300, and it does allow dotnet test targeting a solution, but attempts to run tests in all projects, not just the test projects, and then spews out ugly errors for all non-test projects.

This has been solved by #1745

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.