New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NUnit VSIX test adapters deprecation #518

Closed
Shyam-Gupta opened this Issue Jun 16, 2018 · 6 comments

Comments

Projects
None yet
3 participants
@Shyam-Gupta

Shyam-Gupta commented Jun 16, 2018

At present, test execution from Visual Studio causes Test Platform to load test adapters from multiple locations. For details refer RFC: https://github.com/Microsoft/vstest-docs/blob/master/RFCs/0022-User-Specified-TestAdapter-Lookup.md

For managed test projects, test adapters are generally available in the test dll output folder itself. Hence, if the adapter lookup is limited to the test dll folder only, then it can provide significant perf improvement. Our analysis shows 20% perf improvement when user executes <= 10 tests in a 10k NUnit test project. Similar improvement is seen for MSTest and xUnit test projects as well.

However, in case when NUnit VSIX adapter is installed, then the above assumption of adapter present in test dll output folder, does not hold true. In this scenario Test Explorer has to fallback to the old behavior of loading adapters from Extensions folder as well, thus incurring a perf hit.

Since in Dev16 we plan to deprecate VSIX test adapters, it will be good to put a message on NUnit VSIX adapters installation webpage recommending users to move to nuget based test adapter so that less users get impacted with perf degradation.

@rprouse

This comment has been minimized.

Member

rprouse commented Jun 16, 2018

Thanks for the update. Definitely something we should consider and think about. Can you match Dev16 to a VS release like 15.8 so that we have a better idea of timing?

@OsirisTerje

This comment has been minimized.

Member

OsirisTerje commented Jun 18, 2018

Dev 16 should be VS 2019. No timeline yet, but should not be too far away, see https://blogs.msdn.microsoft.com/visualstudio/2018/06/06/whats-next-for-visual-studio/

@OsirisTerje

This comment has been minimized.

Member

OsirisTerje commented Jun 18, 2018

@Shyam-Gupta Is the fact that the adapter is present in the output folder of a project dependent upon the adapter being referenced by the same project? Or are you doing "magic" stuff in the background to ensure this?
I ask because with the old project system the adapter is normally only referenced from one of the projects. With the new project system it is more likely to be present in all csprojs, since one often use a template (or copy/paste the basics) for those.

@OsirisTerje OsirisTerje self-assigned this Jun 18, 2018

@OsirisTerje OsirisTerje added the is:docs label Jun 18, 2018

@OsirisTerje OsirisTerje added this to the 3.11 milestone Jun 18, 2018

@Shyam-Gupta

This comment has been minimized.

Shyam-Gupta commented Jun 19, 2018

@OsirisTerje Yes adapter present in output folder of a project is dependent upon the adapter being referenced by the same project.

Though the scenario - 'adapter being referenced by a single project in a solution supports discovery\execution for other test projects having only framework reference' was working earlier, its not the supported one. With this change, discovery\execution will not work by default for the projects which do not have adapter reference and those projects will need to add adapter reference to take advantage of perf improvements.

Following is a snapshot from an internal build of VS, which shows the new options we are adding in Tools -> Options -> Test dialog:

image

As mentioned in the options, for improved performance, Test Platform will load test adapters from test assembly folder or as specified in runsettings file. Only for NUnit VSIX and MSTest v1 adapters, we will fallback to current behavior of loading adapters from all locations.

Also we found that NUnit nuget package: NunitTestAdapter.WithFramework adds both framework and adapter to the test project but only framework dll gets copied to the test dll output folder. With this change, Discovery\Execution will not work for projects referencing this nuget package. Since in the documentation of this package, its mentioned that its supported only for VS versions <= 2015, do you think we need to update this package so that the test adapter dll starts getting copied to project output folder ?

@OsirisTerje

This comment has been minimized.

Member

OsirisTerje commented Jun 24, 2018

The adapter with framework package only exist for NUnit2, not for NUnit3, and I don't think we should move that forward.

@OsirisTerje

This comment has been minimized.

Member

OsirisTerje commented Jun 24, 2018

Updated information in marketplace for :
NUnit3 adapter
NUnit2 adapter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment