Consume Microsoft.DotNet.CoreFxTesting package

[ViktorHofer]

Depends on dotnet/arcade#1318

Changes:

• Consume Microsoft.DotNet.CoreFXTesting arcade package
• *.Performance.Tests.csproj renamed to *.PerformanceTests.csproj
• "Test support assemblies", assemblies which are not invoked by the test runner directly don't reference the whole testing framework and only get what they need.
• Code coverage with coverlet

cc @weshaggard @ericstj @safern @danmosemsft @wtgodbe

[ViktorHofer]

@safern @ericstj PTAL

[ViktorHofer]

test code coverage

[ViktorHofer]

@mmitche I'm changing the location of the coverage report dir in this PR. How can we change the htmlpublisher to now point to corefx/artifacts/coverage instead of corefx/artifacts/bin/tests/coverage? Thanks

[mmitche]

Change this line: https://github.com/dotnet/corefx/blob/master/netci.groovy#L74

[ViktorHofer]

@mmitche I changed that with commit 84fd227 yesterday but I believe the path wasn't picked up. Let me try it again.

[ViktorHofer]

test code coverage

[ViktorHofer]

@mmitche the path isn't being picked up correctly: 12:05:31 ERROR: Specified HTML directory 'D:\j\workspace\code_coverage---d5838fef\artifacts\bin\tests\coverage' does not exist. (https://ci.dot.net/job/dotnet_corefx/job/master/job/code_coverage_windows_prtest/79/console)

[ViktorHofer]

@dotnet-bot test Windows x64 Debug Build

[safern]

@mmitche the path isn't being picked up correctly: 12:05:31 ERROR: Specified HTML directory 'D:\j\workspace\code_coverage---d5838fef\artifacts\bin\tests\coverage' does not exist.

From what I know, anything under netci.groovy will not take effect until it is merged, so the builds coming from netci.groovy in this PR come from the file in master branch rather than your PR branch.

[ViktorHofer]

Awesome, that explains the error. Thanks

[mmitche]

Yes that is correct.

[ViktorHofer]

test Packaging All Configurations x64 Debug Build

[ViktorHofer]

test UWP NETNative x86 Release Build
test Windows x64 Debug Build

[ViktorHofer]

test UWP NETNative x86 Release Build

[safern]

I'm seeing the following warning in the logs for OSX and Linux builds:

19:22:00 EXEC : warning NU1101: Unable to find package coverlet.console. No packages exist with this id in source(s): /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/.dotnet/sdk/NuGetFallbackFolder, dotnet-core [/Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/.packages/microsoft.dotnet.arcade.sdk/1.0.0-beta.18603.8/tools/Tools.proj]
19:22:00   The tool package could not be restored.
19:22:00   Tool 'coverlet.console' failed to install. This failure may have been caused by:
19:22:00   
19:22:00   * You are attempting to install a preview release and did not use the --version option to specify the version.
19:22:00   * A package by this name was found, but it was not a .NET Core tool.
19:22:00   * The required NuGet feed cannot be accessed, perhaps because of an Internet connection problem.
19:22:00   * You mistyped the name of the tool.
19:22:00 /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/eng/Tools.props(90,5): warning MSB3073: The command "/Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/.dotnet/dotnet tool install --tool-path /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/artifacts/tools/ coverlet.console --version 1.3.0" exited with code 1. [/Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/.packages/microsoft.dotnet.arcade.sdk/1.0.0-beta.18603.8/tools/Tools.proj]
19:22:00 EXEC : warning NU1101: Unable to find package dotnet-reportgenerator-globaltool. No packages exist with this id in source(s): /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/.dotnet/sdk/NuGetFallbackFolder, dotnet-core [/Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/.packages/microsoft.dotnet.arcade.sdk/1.0.0-beta.18603.8/tools/Tools.proj]
19:22:00   The tool package could not be restored.
19:22:00   Tool 'dotnet-reportgenerator-globaltool' failed to install. This failure may have been caused by:
19:22:00   
19:22:00   * You are attempting to install a preview release and did not use the --version option to specify the version.
19:22:00   * A package by this name was found, but it was not a .NET Core tool.
19:22:00   * The required NuGet feed cannot be accessed, perhaps because of an Internet connection problem.
19:22:00   * You mistyped the name of the tool.

Is this expected? Should we condition the command to windows for now?

[ViktorHofer]

I'll investigate now. That's unexpected

[ericstj]

Remember that this won't be using our nuget.config from the root of the repo since the project being restored is in the arcade sdk package. OSX and Linux likely won't have machine-wide nuget.config, so you may need to add sources into tools.props.