Upgrade xunit to 2.4.0, switch to dotnet test #1445

thomaslevesque · 2018-09-11T12:38:39Z

thomaslevesque · 2018-09-11T12:40:10Z

tests/Directory.Build.props

-    <PackageReference Include="xunit.core" Version="2.3.0-beta2-build3683" />
-    <PackageReference Include="xunit.runner.visualstudio" Version="2.3.0-beta2-build1317" />
-    <PackageReference Include="Microsoft.TestPlatform.TestHost" Version="15.0.0" />
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.8.0" />


I think we should have been using this instead of Microsoft.TestPlatform.TestHost. Anyway, xUnit's "getting started" section uses this package.

thomaslevesque · 2018-09-11T12:46:26Z

tests/Directory.Build.props

+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.8.0" />
+    <PackageReference Include="xunit.core" Version="2.4.0" />
+    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.0" />
+    <PackageReference Include="XunitXml.TestLogger" Version="2.0.0" />


There's no built-in XML logger. This is the one recommended on this page. However I see 2 issues with it:

it's not an official implementation (https://github.com/spekt/xunit.testlogger)

it doesn't take the framework into account, so there's only one result file, which seems to contain results only for the last framework run.

Alternative options:

Do we really need those files? Me, I never looked at them. Maybe we could just disable reporting.

Do we really need them to be in the xUnit format? We could use the built-in trx logger

it doesn't take the framework into account, so there's only one result file, which seems to contain results only for the last framework run.

The trx logger has the same problem. So if we want per-framework result files, we have to run the tests per framework, as @adamralph did in #1442...

It's true. On the other hand, I'm not against submitting an issue/PR to spekt/xunit.testlogger.

What about dropping the result files completely? Do you see any value in them?

FWIW, I don't have them in any of my projects, and I don't miss them.

I'm not against submitting an issue/PR to spekt/xunit.testlogger.

Sure, but there are 2 PRs open since July that didn't even receive a comment... not a very good sign.

It's true. And the project is an absolute mess.

I mostly enjoy the results files when the console output is too hard to read. Seems like we're making that easier, so I'm game to try without them.

thomaslevesque · 2018-09-11T12:46:57Z

tools/FakeItEasy.Build/Program.cs

@@ -8,7 +8,7 @@

    public class Program
    {
-        private const string TestsDirectory = "artifacts/tests";


I didn't really like that name, it was misleading

thomaslevesque · 2018-09-11T12:54:06Z

tools/FakeItEasy.Build/Program.cs

-            var xml = Path.GetFullPath(Path.Combine(TestsDirectory, Path.GetFileName(testDirectory) + ".TestResults.xml"));
-            Run("dotnet", $"xunit -configuration Release -nologo -nobuild -noautoreporters -notrait \"explicit=yes\" -xml {xml}", testDirectory);
+            var xml = Path.GetFullPath(Path.Combine(TestResultsDirectory, Path.GetFileName(testDirectory) + ".TestResults.xml"));
+            Run("dotnet", $"test -c Release --no-build --logger:\"xunit;LogFilePath={xml}\"", testDirectory);


There are many dotnet xunit options that don't have equivalent in dotnet test

-nologo: I spent a bit of time trying to find it, but apparently there's no way to suppress the tool output header. I also didn't succeed in reducing the verbosity, even with --verbosity quiet (the default seems to be minimal, which produces the same output quiet)

-noautoreporters: no equivalent, as far as I can tell. The tests are reported to AppVeyor again, and I don't know how to disable it.

-notrait: there is a --filter option, but --filter "explicit!=yes doesn't work, even though the docs mention it's possible. We're not using this at the moment anyway.

For the first problem, I'd found microsoft/vstest#1701.
Ah, I forgot to check the auto reporters. Good find.

For the first problem, I'd found microsoft/vstest#1701.

Yes, but it doesn't provide a solution...

No, no. But it's nice to know that someone's aware of it.

blairconrad · 2018-09-11T14:03:39Z

@thomaslevesque, I generally like the way this is going (except for -configuration → -c, but I'm picking nits now).

I agree we could drop the logger.

For me the only remaining issue of concern would be the inability to disable autoreporters. That was a real pain point for the AppVeyor builds.

adamralph · 2018-09-11T14:13:18Z

For me the only remaining issue of concern would be the inability to disable autoreporters. That was a real pain point for the AppVeyor builds.

Out of interest, what is this about? I don't do anything special around "autoreporters" in other projects and the Appveyor builds are fine.

thomaslevesque · 2018-09-11T14:21:09Z

(except for -configuration → -c, but I'm picking nits now).

Well, I tend to prefer the short form, but I can change it to --configuration if you prefer.

Out of interest, what is this about? I don't do anything special around "autoreporters" in other projects and the Appveyor builds are fine.

Tests are reported to AppVeyor and appear in the "Tests" tab of the build. This looks like a good idea, but it has several issues:

Test result collection by AppVeyor takes a very significant time; disabling this made the build much faster (I don't remember the exact numbers, but I think it was about twice as fast)
We have a total of almost 20000 reported tests. Just loading them in the page takes a long time and tends to freeze the browser (at least when using Firefox)
It shows very little useful information: test name, status (failed/passed), and duration.
The tested are listed in seemingly random order

blairconrad · 2018-09-11T14:27:56Z

Well, I tend to prefer the short form, but I can change it to --configuration if you prefer.

I usually like the explicit version in scripts, but I'm not too bent out of shape. Vote with your ❤️.

blairconrad · 2018-09-11T15:00:19Z

@adamralph, @thomaslevesque, for interest, I checked the latest build for this PR, and the autotest reporting added at least 2:12 to the build time (measured from time of last line of the console output to the reported build time near the top of the page).

blairconrad

I love it, @thomaslevesque!

blairconrad · 2018-09-11T15:24:20Z

tools/FakeItEasy.Build/Program.cs

-            Run("dotnet", $"test -c Release --no-build --logger:\"xunit;LogFilePath={xml}\"", testDirectory);
-        }
+        private static void RunTests(string testDirectory) =>
+            Run("dotnet", $"test --configuration Release --no-build -- RunConfiguration.NoAutoReporters=true", testDirectory);


The $ is bonus now.
Oh! You put the --configuration in. How kind. I guess after you saved all those "testresult" characters, why not?

The $ is bonus now.

Oops. I'll remove it. Usually R# tells me about these things, but I did everything in VS Code...

blairconrad · 2018-09-11T15:48:13Z

@thomaslevesque, looks great. Is there a reason we're still [WIP]?

thomaslevesque · 2018-09-11T17:01:08Z

No, I just forgot to remove it. I'm on my phone now, so I can't do it easily. Feel free to rename! Le mar. 11 sept. 2018 à 17:48, Blair Conrad <notifications@github.com> a écrit :

…

@thomaslevesque <https://github.com/thomaslevesque>, looks great. Is there a reason we're still [WIP]? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1445 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABxyWQweO3eOJB_4OWQDXJ3xD2RhRumQks5uZ9tDgaJpZM4WjORY> .

adamralph · 2018-09-11T17:12:18Z

Test result collection by AppVeyor takes a very significant time; disabling this made the build much faster (I don't remember the exact numbers, but I think it was about twice as fast)

We have a total of almost 20000 reported tests. Just loading them in the page takes a long time and tends to freeze the browser (at least when using Firefox)

@adamralph, @thomaslevesque, for interest, I checked the latest build for this PR, and the autotest reporting added at least 2:12 to the build time (measured from time of last line of the console output to the reported build time near the top of the page).

Ouch! I can understand the reason for suppressing it!

The tested are listed in seemingly random order

Yes, this is very frustrating. I've been chiming in on appveyor/ci#239 about that.

blairconrad · 2018-09-11T17:18:55Z

Thanks, @thomaslevesque!

thomaslevesque · 2018-09-11T22:04:41Z

Thanks for the merge! Le mar. 11 sept. 2018 à 19:18, Blair Conrad <notifications@github.com> a écrit :

…

Thanks, @thomaslevesque <https://github.com/thomaslevesque>! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1445 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABxyWYSb6JHtLIbFt3aaMPHd-XMNjI2dks5uZ_CAgaJpZM4WjORY> .

thomaslevesque changed the title ~~Upgrade xunit to 2.4.0, switch to dotnet test~~ [WIP] Upgrade xunit to 2.4.0, switch to dotnet test Sep 11, 2018

thomaslevesque commented Sep 11, 2018

View reviewed changes

blairconrad approved these changes Sep 11, 2018

View reviewed changes

Upgrade xunit to 2.4.0, switch to dotnet test

d178354

thomaslevesque force-pushed the switch-to-dotnet-test branch from c61d5ed to d178354 Compare September 11, 2018 15:30

blairconrad changed the title ~~[WIP] Upgrade xunit to 2.4.0, switch to dotnet test~~ Upgrade xunit to 2.4.0, switch to dotnet test Sep 11, 2018

blairconrad merged commit 9b7952b into FakeItEasy:develop Sep 11, 2018

thomaslevesque deleted the switch-to-dotnet-test branch September 12, 2018 07:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade xunit to 2.4.0, switch to dotnet test #1445

Upgrade xunit to 2.4.0, switch to dotnet test #1445

thomaslevesque commented Sep 11, 2018

thomaslevesque Sep 11, 2018

thomaslevesque Sep 11, 2018

thomaslevesque Sep 11, 2018

blairconrad Sep 11, 2018

thomaslevesque Sep 11, 2018

adamralph Sep 11, 2018

thomaslevesque Sep 11, 2018

blairconrad Sep 11, 2018

blairconrad Sep 11, 2018

thomaslevesque Sep 11, 2018

thomaslevesque Sep 11, 2018

blairconrad Sep 11, 2018

thomaslevesque Sep 11, 2018

blairconrad Sep 11, 2018

blairconrad commented Sep 11, 2018

adamralph commented Sep 11, 2018

thomaslevesque commented Sep 11, 2018

blairconrad commented Sep 11, 2018

blairconrad commented Sep 11, 2018

blairconrad left a comment

blairconrad Sep 11, 2018

thomaslevesque Sep 11, 2018

blairconrad commented Sep 11, 2018

thomaslevesque commented Sep 11, 2018 via email

adamralph commented Sep 11, 2018

blairconrad commented Sep 11, 2018

thomaslevesque commented Sep 11, 2018 via email

Upgrade xunit to 2.4.0, switch to dotnet test #1445

Upgrade xunit to 2.4.0, switch to dotnet test #1445

Conversation

thomaslevesque commented Sep 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blairconrad commented Sep 11, 2018

adamralph commented Sep 11, 2018

thomaslevesque commented Sep 11, 2018

blairconrad commented Sep 11, 2018

blairconrad commented Sep 11, 2018

blairconrad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blairconrad commented Sep 11, 2018

thomaslevesque commented Sep 11, 2018 via email

adamralph commented Sep 11, 2018

blairconrad commented Sep 11, 2018

thomaslevesque commented Sep 11, 2018 via email