Support for running without VSIX

[sirotnikov]

Nunit Test Adapter can be called by vstest.console.exe either via the /TestAdapterPath switch or by pointing to their dll's in the .runsetttings file :

<RunConfiguration>
    <!-- Path to Test Adapters -->
    <TestAdaptersPaths>c:\path\adapter</TestAdaptersPaths>
</RunConfiguration>

This allows running NUnit tests like regular MSTests without customizing the TFS server configuration and uploading the test adapter to the TFS controller.

GoogleTestAdapter is for some reason not correctly recognized as an adapter when supplied like that, and thus can't be called from TFS without writing custom scripts, to call it correctly, and then publish the test results (as well as coverage info).

I'd be willing to help research and patch it, if you're willing to spare some time for guidance.

[csoltenborn]

This might be related to issue #5....

Anyway, thanks for offering help! What kind of guidance do you need?

[sirotnikov]

I'm a bit shy when it comes to C#, so I'm not sure how to capture the point where vtest.console is querying supported interfaces, to catch what is is missing. I'll find someone proficient in my team, and then maybe I'll have some questions about specific VSTEST interfaces.

I will try to see if the Nunit Guys can point at what they do to achieve VSIX independence, since it's promoted as a feature.

[csoltenborn]

Sounds good to me! @jgefele has already briefly looked into #5 - he will add his knowledge to this bug later today.

As a side note (and in case you haven't seen): The project Wiki contains a couple of links to sources of information - there's no official VS test framework documentation, but a couple of blog entries, the most important of which we have linked there.

[jgefele]

Hi,

this has been on my list for the next release, but it seems that issue #5 has magically been fixed :-)

This does work now:

vstest.console.exe /UseVsixExtensions:true \Path\To\MyTests.exe

And I just found that there seems to be just a tiny bit missing. The following works...

vstest.console.exe /TestAdapterPath:"%LocalAppData%\Microsoft\VisualStudio\14.0\Extensions" \Path\To\MyTests.exe

... if and only if I rename GoogleTestAdapter.VS.dll to GoogleTestAdapter.TestAdapter.dll (file name magic FTW!).

@sirotnikov: Could you please try if this file rename makes your .runsettings approach work, too?

If that's the case, all that is left would be a nice nuget package to simplify deployment.

Fingers crossed,
Jonas

[sirotnikov]

Hi!
Thanks!
This almost works but not quite. The adapter is now properly recognized, however it can't find my tests.

It seems that the issue is a different bug which makes GoogleTestAdapter not find any tests to run when run under a /Platform:x64 configuration.

When I run it from VisualStudio, it somehow works with DefaultProcessorArchitecture:X86 despite the test executable being x64. However, when running from vstest.console.exe it will complain when x86 architecture is supplied (or assumed).

So technically bug closed, but should investigate a new bug.

Also, could you enlighten me as to why file name magic works? I assume it has to do with assumptions dotnet does about filenames and assemblies and such?

[jgefele]

Hi @sirotnikov,

I reorganized the project structure so that the relevant .dll is named *.TestAdapter.dll
Somehow this also must have straightened something else (I did some other changes on the go...). So I'm now in the situation to say: "It seems to work now with /Platform:x64, but I don't know why...". :-O

Please give this build a try and let me know if it works for you.

I suggest to use this issue for the whole use-case: "Use GoogleTestAdapter from vstest.console.exe" instead of opening and closing new issues all the time.

Regarding "file name magic": I don't think this is dotnet-related but rather specific to vstest.console.exe. Visual Studio knows in which DLL to look from the UnitTestExtension Asset in the source.extension.vsixmanifest. vstest.console.exe on the other hand is just given a path and needs to scan it for DLLs implementing the test discovery/execution interface. I guess that they just scan *.TestAdapter.dll for performance reasons.

Unfortunately, all this is not documented at all. Luckily, I noticed that other test adapters have their DLLs named in that format and just gave it a try...

Cheers,
Jonas

[sirotnikov]

Hi,
I appreciate the support and the time you take to solve this one specific issue.

Sadly, still no dice with the x64 flag. No tests are found, in VS or vstest.console.exe

It does work with GoogleTestRunner though, after I renamed their dll as you suggested.
I'd use it, but I don't want the TFS admin to require F# runtime, and I also don't think it will be actively maintained.

[jgefele]

Hmmm. Maybe it was just some strange coincidence that it worked for me yesterday. I didn't test it in a clean environment. And as I said, I didn't even intend to fix it (yet). It just seemed to work suddenly.

But let's make sure we're trying the same things: can you try it with a 64-debug build of the "SampleTests" project? Could you also send me your exact command line, the .runsettings, and if you're using VS 2013 or 2015?

Thanks,
Jonas

[sirotnikov]

I'm using VS 2015 sp1, latest SDK and WDK.

I don't get any tests when running against a 64 debug build of *=SampleTests either.

I decided stepping through the code of GoogleTestAdapter, as it is configured in the debug settings of the TestAdapter project.

vstest.console.exe /TestAdapterPath:GoogleTestAdapter\GoogleTestAdapter\TestAdapter\bin\Debug\ GoogleTestAdapter\SampleTests\Debug\Tests.exe

Apparently it's actually ran by a child process, so after getting "debug child processes" to work, I successfully stepped through when it was /Platform:x86 mode against the x86 executable.

However when ran with /Platform:x64 against a x64 compiled executable, the application quit before reaching your extension code. It seems there is some uncaught exception thrown by TE.ProcessHost.Managed.EXE which shuts it down before your classes are called.

This does not happen to GoogleTestRunner

[sirotnikov]

Could you try it yourself on the x64 Tests build?

vstest.console.exe /TestAdapterPath:GoogleTestAdapter\GoogleTestAdapter\TestAdapter\bin\Debug\ GoogleTestAdapter\SampleTests\x64\Debug\Tests.exe /Platform:x64

Maybe it's an issue with my machine?

[jgefele]

I can confirm your observation. Probably it worked for me only once due to cosmic rays.

I'm in exactly the same situation for /Platform:x64. No line of code in GoogleTestAdapter is run. If there's no better idea, I will try to decompile and debug vstest.console.exe and TE.ProcessHost.Managed.EXE to get an idea of what's going wrong.

I don't know what's different with GoogleTestRunner. They don't seem to do anything special for 64-bit.

[sirotnikov]

I tried running vstest.console.exe under debug, but didn't learn much since I lack symbols.

I did manage to see messages in the debug window showing that GoogleTestAdapter was acccessed several times by CLR to load and construct its classes. Then somewhere an exception in TE.ProcessHost.Managed.EXE is unhandled causing vstest.console.exe to fail. This seems to happen exactly where the x86 platform would jump into TestExecutor::RunTests(IEnumerable<string> executables, ..).

Sadly, I don't know enough about CLR implementation to properly capture more information, and don't have symbols for vstest

Is there anyone from the VS team who could help? It's conceivable that the issue is not with GTA but rather we're exposing a specific bug in TE.ProcessHost.Managed.EXE or something. I had something similar when a VisualStudio component chocked and crashed on a unicode text file. It turned out it had a single byte truncated, and VS incorrectly computed memory size and then got a page fault when read the text contents to a buffer one byte short.

[csoltenborn]

In fact, Jonas might have an idea on what's going on... Stay tuned!

[jgefele]

It's not what I thought it was yesterday. It's assembly version hell again. Similar to the DIA issues we had before.

We reference the Microsoft.VisualStudio.Test* assemblies of Visual Studio 2013 to build one binary for Visual Studio 2013 and 2015. We thought they are the common ground for later versions, but it seems that no upward compatibility is guaranteed. This has worked well for Visual Studio 2015 without SP (so no cosmic rays - I just had a non-updated VS at home and an updated VS at work). For some reasons this also works for VS 2015 SP1 with 32-bit tests, but not for 64-bit.

Long story short: in branch vs2015_sp1 I've updated the referenced assemblies (try the build on AppVeyor) and for me it works with vstest.console.exe for VS2015 noSP/SP1 in 32-bit and 64-bit. I guess it will stop working for VS2013, but didn't test it. I had some issues in VS itself which I don't know yet if they are related.

This intensifies the already existing need for a solution to dynamically reference the correct assemblies depending on the VS version we're running in (or have separate builds for each version - a solution I would like to avoid).

[sirotnikov]

Oh wow, that's crappy.

This has made me realize I should probably save a couple of versions of the plugin to see which one works on the build controller.

It'll take me a couple of days to test. Thanks for the build.

[sirotnikov]

Forgot to update that the alternate binaries worked fine with VS2015 sp1,

Current known ways to run without installing VSIX:

1. via /TestAdapterPath switch
2. via .runsettings TestAdaptersPaths element

It should also work when the binaries are placed in a subfolder in the packages directory in the solution directory. However I couldn't get this to work when I tested on TFS - I'm guessing either because I didn't have a meaningful name ("GoogleTestAdapter") or perhaps this is only searched if the project has a pacakage reference.

I'm on leave from work, for exams, so I won't be able to thoroughly test

[sirotnikov]

So is there any solution to the issue of referenced assemblies?

[sirotnikov]

As far as I'm concerned this can be closed as solved, and the assemblies issue should have its own feature.

[csoltenborn]

Version 0.9.0 should (finally) work fine with x64 executables. Note that since V0.8.0 we also provide a NuGet package.