New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Arcade testing targets #30330

Merged
merged 5 commits into from Dec 3, 2018

Conversation

Projects
None yet
3 participants
@tmat
Member

tmat commented Oct 4, 2018

Introduce build target Test that can be used to execute tests using xunit runner on desktop, Core or Mono. Use this target to run tests on Core CLR replacing PowerShell and bash scripts.

This change allows for running tests from a test project directory using msbuild/dotnet like so:

C:\R0\src\Compilers\CSharp\Test\Syntax>msbuild /t:Test
Project "C:\R0\src\Compilers\CSharp\Test\Syntax\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.csproj" on node 1 (Test target(s)).
RunTests:
  Creating directory "C:\R0\Binaries\TestResults\Debug\".
  Running tests: C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\net472\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll [net472|x64]
  "C:\Users\tomat\.nuget\packages\xunit.runner.console\2.4.1-pre.build.4059\tools\net452\xunit.console.exe" "C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitT
  ests\net472\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll" -noshadow -xml "C:\R0\Binaries\TestResults\Debug\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests_net472_x64.xml" -htm
  l "C:\R0\Binaries\TestResults\Debug\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests_net472_x64.html"  > "C:\R0\Binaries\Debug\Logs\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests_net
  472_x64.log" 2>&1
  Tests succeeded: C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\net472\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll [net472|x64]
RunTests:
  Running tests: C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\netcoreapp2.1\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll [netcoreapp2.1|x64]
  "dotnet.exe" exec --depsfile "C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\netcoreapp2.1\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.deps.json"
  --runtimeconfig "C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\netcoreapp2.1\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.runtimeconfig.json" "C:\
  Users\tomat\.nuget\packages\xunit.runner.console/2.4.1-pre.build.4059/tools/netcoreapp2.0/xunit.console.dll" "C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.Un
  itTests\netcoreapp2.1\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll" -noautoreporters -xml "C:\R0\Binaries\TestResults\Debug\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests_net
  coreapp2.1_x64.xml" -html "C:\R0\Binaries\TestResults\Debug\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests_netcoreapp2.1_x64.html"  > "C:\R0\Binaries\Debug\Logs\Microsoft.CodeAnaly
  sis.CSharp.Syntax.UnitTests_netcoreapp2.1_x64.log" 2>&1
  Tests succeeded: C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\netcoreapp2.1\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll [netcoreapp2.1|x64]
Done Building Project "C:\R0\src\Compilers\CSharp\Test\Syntax\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.csproj" (Test target(s)).

or

C:\R0\src\Compilers\CSharp\Test\Syntax>dotnet msbuild /t:Test
  Running tests: C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\net472\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll [net472|x64]
  Tests succeeded: C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\net472\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll [net472|x64]
  Running tests: C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\netcoreapp2.1\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll [netcoreapp2.1|x64]
  Tests succeeded: C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\netcoreapp2.1\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll [netcoreapp2.1|x64]

If a test project targets multiple platforms you can choose to run against a subset of these frameworks using /p:TestTargetFrameworks:

C:\R0\src\Compilers\CSharp\Test\Syntax>dotnet msbuild /t:Test /p:TestTargetFrameworks=net472
  Running tests: C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\net472\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll [net472|x64]
  Tests succeeded: C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\net472\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll [net472|x64]

Switch test architecture to 'x86' (for desktop TFMs only right now, since 32-bit Core CLR is not installed by restore right now):

C:\R0\src\Compilers\CSharp\Test\Syntax>dotnet msbuild /t:Test /p:TestArchitectures=x86 /p:TestTargetFrameworks=net472

  Running tests: C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\net472\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll [net472|x86]
  Tests succeeded: C:\R0\Binaries\Debug\UnitTests\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests\net472\Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests.dll [net472|x86]

If you have Mono installed on the machine you can also choose to run desktop tests on Mono by specifying /p:TestRuntime=Mono.

Fixes #25049

@tmat tmat requested review from dotnet/roslyn-compiler as code owners Oct 4, 2018

echo "build/scripts/tests.sh (Debug|Release) (dotnet|mono|mono-debug) [test assembly name] [xunit args]"
echo "If specified, the test assembly name must be a substring match for one or more test assemblies."
echo "Note that it's a substring match so '.dll' would match all unit test DLLs and run them all."
echo "Any xunit args specified after the assembly name will be passed directly to the test runner so you can run individual tests, i.e. -method \"*.Query_01\""

This comment has been minimized.

@jaredpar

jaredpar Oct 4, 2018

Member

This script had two features that aren't immediately obvious how to do on the new system:

  1. Run a given test project under Mono
  2. Run an individual test project vs. the entire set

This comment has been minimized.

@tmat

tmat Oct 4, 2018

Member
  1. msbuild /t:Test /p:TestRuntime=Mono MyProj.csproj
  2. msbuild /t:Test MyProj.csproj

This comment has been minimized.

@tmat

tmat Oct 4, 2018

Member

Also:
msbuild /t:Test /p:TestRuntime=Mono /p:MonoTool="mono --debug" MyProj.csproj to pass parameters or choose where mono is executed from.

<XUnitPath>$(NuGetPackageRoot)\xunit.runner.console\$(xunitrunnerconsoleVersion)\tools\net452\xunit.console.x86.exe</XUnitPath>
<XUnitTestResultsDirectory>$(OutDir)\xUnitResults</XUnitTestResultsDirectory>
<XUnitArguments>"$(OutDir)\$(AssemblyName).dll" -html "$(XUnitTestResultsDirectory)\$(AssemblyName).html" -noshadow</XUnitArguments>
<RuntimeIdentifier Condition="'$(TargetFramework)' == 'net46' and '$(RuntimeIdentifier)' == '' and '$(OS)' == 'Windows_NT'">$(RoslynDesktopRuntimeIdentifier)</RuntimeIdentifier>

This comment has been minimized.

@jaredpar

jaredpar Oct 4, 2018

Member

I wonder if we should add $(RoslynDesktopTargetFramework) instead of hard coding net46 everywhere.

This comment has been minimized.

@tmat

tmat Oct 4, 2018

Member

Maybe. It won't change that often and it's easy to search for net46, so don't think it makes much of a difference.

build.sh Outdated
fi
"${root_path}"/build/scripts/tests.sh "${build_configuration}" "${test_runtime}"
dotnet build "${root_path}/Compilers.sln" "/t:test" "/p:Configuration=${build_configuration}" "/p:TestRuntime=${test_runtime}" "/bl:${binaries_path}/Test.binlog"

This comment has been minimized.

@jaredpar

jaredpar Oct 4, 2018

Member

Okay, so that's how to specify the runtime

@tmat tmat force-pushed the tmat:ArcadeTest branch 2 times, most recently from fb62cdf to 03d14e3 Oct 9, 2018

@tmat

This comment has been minimized.

Member

tmat commented Oct 10, 2018

test this please

@tmat tmat force-pushed the tmat:ArcadeTest branch 3 times, most recently from 028153d to 7c1dd9a Oct 10, 2018

@tmat tmat force-pushed the tmat:ArcadeTest branch 4 times, most recently from 05a056b to 7e5a3ee Oct 30, 2018

@tmat tmat changed the title from WIP: Use Arcade testing targets to Use Arcade testing targets Nov 1, 2018

@tmat

This comment has been minimized.

Member

tmat commented Nov 1, 2018

@jaredpar PTAL

@tmat

This comment has been minimized.

Member

tmat commented Nov 1, 2018

test windows_debug_vs-integration_prtest please

@tmat tmat force-pushed the tmat:ArcadeTest branch 3 times, most recently from 1de9f0d to ea775f5 Nov 24, 2018

@tmat

This comment has been minimized.

Member

tmat commented Nov 28, 2018

test windows_debug_vs-integration_prtest please

@tmat

This comment has been minimized.

Member

tmat commented Nov 28, 2018

test windows_release_vs-integration_prtest please

@tmat tmat force-pushed the tmat:ArcadeTest branch 2 times, most recently from 3fa6715 to 2c581ce Nov 28, 2018

@tmat tmat closed this Nov 28, 2018

@tmat tmat reopened this Nov 28, 2018

@tmat tmat closed this Nov 29, 2018

@tmat tmat reopened this Nov 29, 2018

@tmat

This comment has been minimized.

Member

tmat commented Nov 29, 2018

@tmat tmat force-pushed the tmat:ArcadeTest branch 2 times, most recently from 7e5ad98 to 5e31275 Nov 29, 2018

@tmat tmat closed this Nov 29, 2018

@tmat tmat reopened this Nov 29, 2018

@tmat tmat closed this Nov 30, 2018

@tmat tmat reopened this Nov 30, 2018

@jaredpar

This comment has been minimized.

Member

jaredpar commented Nov 30, 2018

@kg as this impacts the workflow for Mono testing. Mostly just an FYI here.

@@ -29,7 +29,7 @@ phases:
ArtifactName: 'Windows Visual Studio Integration $(_configuration)'
publishLocation: Container
continueOnError: true
condition: or(failed(), canceled())
condition: not(succeeded())

This comment has been minimized.

@jaredpar

jaredpar Nov 30, 2018

Member

Yeah that's a much better pattern.

@tmat tmat force-pushed the tmat:ArcadeTest branch from 5e31275 to 4d1c2c0 Nov 30, 2018

@@ -10,6 +10,7 @@
<TargetFrameworks>$(RoslynPortableTargetFrameworks)</TargetFrameworks>
<RoslynProjectType>UnitTestPortable</RoslynProjectType>
<RuntimeIdentifiers>$(RoslynPortableRuntimeIdentifiers)</RuntimeIdentifiers>
<SkipTests Condition="'$(TestRuntime)' == 'Mono'">true</SkipTests>

This comment has been minimized.

@jaredpar

jaredpar Nov 30, 2018

Member

Should have a bug tracking link here.

This comment has been minimized.

@tmat

tmat Nov 30, 2018

Member

I linked an issue where we have issue listed in tests.sh, but for this one we do not:
https://github.com/dotnet/roslyn/blob/master/build/scripts/tests.sh#L54-L55

@tmat

This comment has been minimized.

Member

tmat commented Nov 30, 2018

@jaredpar OK to merge? I can follow up, file bugs on the skip tests and link them later. I'd like to get this change merged ASAP as it is needed to get other Arcade stuff working.

@tmat tmat force-pushed the tmat:ArcadeTest branch from 4d1c2c0 to 4ff74fe Nov 30, 2018

@tmat tmat force-pushed the tmat:ArcadeTest branch from 4ff74fe to 61f2d2b Dec 3, 2018

@tmat tmat merged commit 2f345d0 into dotnet:master Dec 3, 2018

2 of 3 checks passed

roslyn-integration-CI in progress
Details
license/cla All CLA requirements met.
Details
roslyn-CI #20181203.4 succeeded
Details

@tmat tmat deleted the tmat:ArcadeTest branch Dec 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment