Make VSTest runner work and enable html log #33915

ViktorHofer · 2020-03-21T12:57:08Z

The VSTest runner is now supposed to work with the /p:TestRunner=vstest
switch. Currently this is an opt-in but the desire is to get rid of
xunit.console completely and just use VSTest.

An html and trx log example for System.Text.RegularExpressions.Tests: testResults.zip

cc @stephentoub @danmosemsft

The VSTest runner is now supposed to work with the /p:TestRunner=vstest switch. Currently this is an opt-in but the desire is to get rid of xunit.console completely and just use VSTest.

stephentoub · 2020-03-21T18:37:18Z

Thanks. What does an error like this mean?

Non-serializable data ('System.Object[]') found for 'System.Text.RegularExpressions.Tests.RegexConstructorTests.Ctor'; falling back to single test case.

Did the test run and it just can't report on it, or did it skip it?

ViktorHofer · 2020-03-21T18:44:55Z

#30324

Did the test run and it just can't report on it, or did it skip it?

It ran completely but it can't fetch individual data rows, means you won't be able to debug individual data rows in VS Test Explorer. I can only hide those diagnostic messages by disabling all, which might be possible as we only used them for the long running test support for which VSTest might have integrated support.

jkotas · 2020-03-21T20:22:45Z

Are we going to have a workflow in place to make changes in the VSTest runner that are required for not-yet-released .NET Core versions?

For example, let's say that that we add a new platform and this starts failing. How we are going to make progress?

I am asking about this because of I am stuck on a similar problem with BenchmarkDotNet in #33694.

ViktorHofer · 2020-03-21T20:42:27Z

We already consume vstest via dependency flow which means getting live updates which aren't bound to VS releases. Additionally to that, we are actively collaborating with the vstest team (cc @nohwnd) and benefiting from each other.

safern · 2020-03-24T20:21:10Z

eng/testing/xunit/vstest.props

@@ -2,19 +2,15 @@
  <PropertyGroup>
    <!-- Microsoft.Net.Test.Sdk brings a lot of satellite assemblies in. -->
    <SatelliteResourceLanguages>en</SatelliteResourceLanguages>
+    <TrxTestResultsName>testResults.trx</TrxTestResultsName>


Do we know if the helix sdk supports .trx files to show the test results in AzDo, submitting to kusto and failing the build? If not we should request a feature for when we want to move to use this as our default test runner. cc: @alexperovich

Helix supports trx files for uploading to azure devops. Kusto is only supported for xunit.

I think a lot of people use Kusto for metrics and what not as the AzDo APIs aren't that great to get test results and search through them. Would it be possible to add Kusto from a trx file?

The support for processing xunit results into kusto is on the server side for helix and extending that for other formats would be a very large an potentially disruptive change. The azure devops uploading code is in arcade and very easy to change so it supports many formats. My understanding is that we want to consolidate everything to azure devops and make that experience better, rather than continue investing in our custom reporting. @Chrisboh may have more context.

Yeah I knew that trx is supported but I didn't know about Kusto only working with the xunit xml format.

safern · 2020-03-24T20:22:21Z

eng/testing/xunit/vstest.targets

@@ -1,10 +1,13 @@
 <Project>
  <PropertyGroup>
-    <RunCommand>"$(RunScriptHost)"</RunCommand>
+    <RunCommand>"$(DotNetTool)"</RunCommand>
+    <RunCommand Condition="'$(ContinuousIntegrationBuild)' == 'true'">dotnet</RunCommand>


Just to understand, could you explain what's the difference in between DotNetTool and dotnet and why does it has to be different in CI?

DotNetTool is a property set by Arcade, mean it's available during the test invocation on your machine. On a different machine you don't want to hardcode the path into the RunTemplate.cmd/sh. There is more work required here anyway to make this work on CI.

My goal in a subsequent PR is to just use dotnet here instead and then for local local development, modify PATH to point to the RunScriptHost if not on CI. As said, I don't want to do this now as this just enables VSTest optionally and not on CI.

Another blocker for enabling VSTest in CI is #1980.

safern · 2020-03-24T20:23:06Z

eng/testing/xunit/vstest.xunit.json

@@ -0,0 +1,5 @@
+{
+    "diagnosticMessages": false,
+    "longRunningTestSeconds": 120,


So this support is now built-in in VSTest?

Yeah but I still need to play around with it. There's a blame mode which creates a sequence.xml which tells you which tests have run and which haven't but I don't know if that mode slows down the test invocation. cc @nohwnd

safern · 2020-03-24T20:25:06Z

eng/Versions.props

@@ -118,7 +118,6 @@
    <XUnitVersion>2.4.1</XUnitVersion>
    <TraceEventVersion>2.0.5</TraceEventVersion>
    <NewtonsoftJsonVersion>12.0.3</NewtonsoftJsonVersion>
-    <XUnitXmlTestLoggerVersion>2.1.26</XUnitXmlTestLoggerVersion>


Why is this removed? Just left-over?

Yeah this was introduced only for vstest in a past PR but actually we don't need it.

safern

Thanks for solving all my questions.

ViktorHofer · 2020-03-26T13:36:34Z

Issue was #32377.

Dotnet-GitSync-Bot added the area-Infrastructure-libraries label Mar 21, 2020

ViktorHofer force-pushed the DefaultVSTest branch from 08007de to 6d13cb3 Compare March 21, 2020 13:00

ViktorHofer mentioned this pull request Mar 21, 2020

Remove duplicates and fix namespaces in S.T.RegularExpressions.Tests #33917

Merged

ViktorHofer force-pushed the DefaultVSTest branch from 6d13cb3 to 0ac4ab5 Compare March 21, 2020 13:54

Make VSTest runner work and enable html log

df5c90a

The VSTest runner is now supposed to work with the /p:TestRunner=vstest switch. Currently this is an opt-in but the desire is to get rid of xunit.console completely and just use VSTest.

ViktorHofer force-pushed the DefaultVSTest branch from 0ac4ab5 to df5c90a Compare March 21, 2020 14:37

ericstj mentioned this pull request Mar 21, 2020

Several tests fail on mono with mono_threads_pthread_kill: pthread_kill failed with error 11 #32377

Closed

ViktorHofer requested a review from safern March 23, 2020 15:55

ViktorHofer self-assigned this Mar 23, 2020

safern reviewed Mar 24, 2020

View reviewed changes

safern approved these changes Mar 25, 2020

View reviewed changes

ViktorHofer merged commit 4a55b46 into master Mar 26, 2020

ViktorHofer deleted the DefaultVSTest branch March 26, 2020 13:36

dotnet locked as resolved and limited conversation to collaborators Dec 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make VSTest runner work and enable html log #33915

Make VSTest runner work and enable html log #33915

ViktorHofer commented Mar 21, 2020 •

edited

stephentoub commented Mar 21, 2020

ViktorHofer commented Mar 21, 2020 •

edited

jkotas commented Mar 21, 2020 •

edited

ViktorHofer commented Mar 21, 2020

safern Mar 24, 2020

alexperovich Mar 24, 2020

safern Mar 24, 2020

alexperovich Mar 24, 2020

ViktorHofer Mar 25, 2020

safern Mar 24, 2020

ViktorHofer Mar 25, 2020

ViktorHofer Mar 25, 2020

safern Mar 24, 2020

ViktorHofer Mar 25, 2020

safern Mar 24, 2020

ViktorHofer Mar 25, 2020

safern left a comment

ViktorHofer commented Mar 26, 2020

Make VSTest runner work and enable html log #33915

Make VSTest runner work and enable html log #33915

Conversation

ViktorHofer commented Mar 21, 2020 • edited

stephentoub commented Mar 21, 2020

ViktorHofer commented Mar 21, 2020 • edited

jkotas commented Mar 21, 2020 • edited

ViktorHofer commented Mar 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

safern left a comment

Choose a reason for hiding this comment

ViktorHofer commented Mar 26, 2020

ViktorHofer commented Mar 21, 2020 •

edited

ViktorHofer commented Mar 21, 2020 •

edited

jkotas commented Mar 21, 2020 •

edited