Huge delay before actual test start

[ekalchev]

Description

Running NUnit tests takes a lot of time (over 20 sec for my solution) to actually start the tests. I believe this is related to vstest not NUnit test adapter (not sure though)

Steps to reproduce

1. Create Wpf application project using .net framework 4.8
2. Create new class TestClass with method TestMethod that uses code from Newtonsoft.Json nuget package
3. Create Nunit test assembly that runs TestClass.TestMethod
4. Select the test in test explorer and run it.

Diagnostic logs

Using Tests diagnostic log I can see the delay caused by many of those lines (over 80) for every nuget package we use.

[5/5/2021 5:50:14.408 PM] Failed to determine installation location for NuGet package 'System.Reactive.Core' referenced in project 'xxxxx'. Check to make sure that a NuGet restore operation has been performed for this project and that no errors were reported during this operation.
[5/5/2021 5:50:14.408 PM] Failed to determine installation location for NuGet package 'System.Reactive.Interfaces' referenced in project 'xxxxx'. Check to make sure that a NuGet restore operation has been performed for this project and that no errors were reported during this operation.
[5/5/2021 5:50:14.408 PM] Failed to determine installation location for NuGet package 'System.Reactive.Linq' referenced in project 'xxxxxxx'. Check to make sure that a NuGet restore operation has been performed for this project and that no errors were reported during this operation.
[5/5/2021 5:50:14.408 PM] Failed to determine installation location for NuGet package 'System.Reactive.PlatformServices' referenced in project 'xxxxxxx'. Check to make sure that a NuGet restore operation has been performed for this project and that no errors were reported during this operation.

Environment

We recently upgraded to latest version of VS2019. VS2017 doesn't have this problem.
After I spent some time troubleshooting this, I found that this is related to custom nuget.config file we have. nuget.config is located in the solution directory

This is nuget.config content

<?xml version="1.0" encoding="utf-8"?>
<configuration>
 <config>
   <add key="globalPackagesFolder" value="./packages" /> <!-- used from the new csproj format with PackageReference -->
   <add key="repositoryPath" value="./packages" /> <!-- used from the old csproj format with packages.config -->
 </config>
 <packageSources>
   <clear />
   <add key="LocalNugetRepo" value="./packages" />
   <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
   <add key="Microsoft Visual Studio Offline Packages" value="C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\" />
 </packageSources>
 <packageRestore>
   <add key="enabled" value="True" />
   <add key="automatic" value="False" />
 </packageRestore>
</configuration>

Here what I found out so far:

1. VS2019 - if I delete nuget.config the packages are restored in their default location %USER_DIR%\.nuget\packages - no issues with test runs and no nuget warnings in the test log
2. VS2019 if I keep the nuget.config but copy the content from ./packages directory to %USER_DIR%\.nuget\packages - no issues
3. VS2017 WITH nuget.config, there is no nuget warnings in the test log and I am not experiencing any delays.

[Sanan07]

Why it should be related to vstest? It looks like nuget restore has some issues. Try to run dotnet restore in your solution folder before running tests.

[ekalchev]

Why it should be related to vstest? It looks like nuget restore has some issues. Try to run dotnet restore in your solution folder before running tests.

I think it is related to vstest, because the errors are in Output->Tests and before any nunit log entry.
dotnet restore is restoring the packages in %USERDIR%\.nuget\packages and these warnings are gone, but that is not the packages directory that visual studio uses, because I have different package directory configured in nuget.config file.

Its seems there is an inconsistency between vstest and vs2019 build where to look for packages.

[Evangelink]

@ekalchev Are you still experiencing the issue with latest version of Test Platform? If so, could you please provide some simple repro?

[ekalchev]

Yes, the problem is present with latest update of VS2019. No, I can't provide simple repro.

We are using VS2022 now and starting the test is much faster (but still noticeable slower than VS2017)

[nohwnd]

There is no clear repro to this problem, if you have one please share it so I can investigate this.

[egraff]

I'm seeing this as well, with enormous delays occurring deterministically every time a test is run from the Test Explorer. The problem is so significant that the Test Explorer in Visual Studio is becoming practically useless, and I'm more or less forced to run tests from the console instead to get a timely feedback cycle. It's therefore especially problematic when having to debug tests.

@nohwnd I created a repro here: https://github.com/egraff/vstest-2889-repro 😊 I suspect the problem might be in Visual Studio, and not in vstest as such. But I'm hoping that you will be able to forward the issue to the right place, and escalate it there 😇