Testing: Need to step-in-to and debug the test adapter discovery process

[clairernovotny]

While working on getting xUnit working better on the new test platform, I've been struggling with code that executes at discovery time.

I'm sure that the issues themselves would be easy to spot/fix once in a debugger, but that's the real issue -- there doesn't seem to be any way of getting a debugger to attach to the correct dotnet instance that's loading the adapter in discovery mode.

If you run dotnet vstest path\to\test.dll, it appears so spawn off another dotnet instance.

Things that don't help -- the Child Process Debugging tool requires that win32 debugging be enabled. It also does not seem to currently support CoreCLR as one of the debug engines you can start a child on.

What I really need: A way to debug the test adapter code in discovery mode. Debugging test execution mode is a little easier because you can "debug selected test" and VS does attach to the right process and we can set breakpoints in our adapter code before it gets to the test.

[TheRealPiotrP]

@dotnet/vstest can you please take a look?

[codito]

This instruction works for vstest.console.exe, dotnet test and dotnet vstest.

Debug test platform components

The runner and test host processes support waiting for debugger attach. You can
enable following environment variable to keep vstest.console.exe waiting for
debugger attach:

> set VSTEST_RUNNER_DEBUG=1
> vstest.console testApp.dll
Waiting for debugger attach...
Process Id: 51928, Name: vstest.console

For test host process (testhost.exe) use the following environment variable.
Execution will halt until debugger is attached.

> set VSTEST_HOST_DEBUG=1
> vstest.console testApp.dll
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Host debugging is enabled. Please attach debugger to testhost process to continue.

[codito]

Note that adapters load in the testhost, so just set VSTEST_HOST_DEBUG=1 should be sufficient.

[clairernovotny]

When trying to debug the test host, I don't see it in the debugger window. Having it output a PID would be useful:

[clairernovotny]

I found it in ProcExcplorer, it's another dotnet instance, which makes it really hard to find.

As I navigate between process explorer, to find the process, identify the PID and then try to find it in the "attach to process" dialog in VS, I then get the following timeout:

Error: Failed to initialize client proxy: could not connect to test process.

Is this configurable to be longer? It's just not enough time to get started.

[codito]

Timeout is not configurable at the moment. We should be able to print out process id, will track this for upcoming release.

If you're attaching from VS, observation is that the top most dotnet.exe (in Attach to process dialog) is the newer process. In the Task Explorer, it is possible to view the command line of dotnet.exe (Go to process details view: Right click on headers -> Select Columns -> Command line). It may help to identify the one with testhost.

[clairernovotny]

Putting out a PID would be good next step. You can see how bad the instructions are:
https://github.com/xunit/visualstudio.xunit/blob/master/README.md

We have to rely on Process Explorer.

That said, for me, opening the Attach to Process dialog takes 5-10 sec for it to load.... it's super slow to keep doing that in a tight loop. It would be much better if VS could auto-attach to child dotnet processes and/or do something like the "Debug selected test" function does where it attaches to the correct instance. We just need that for discovery mode somehow.

[ChristopherHaws]

@onovotny Its not what I would like, but this by far the best solution that I have found:

Microsoft Child Process Debugging Power Tool

Install the Microsoft Child Process Debugging Power Tool which was created by an employee at Microsoft. This allows you to configure the Visual Studio debugger to attach to child processes (which is how vstest.console.exe executes tests)

Once installed, open your solution and enable child process debugging:

1. Go to the child process debug settings at the following Visual Studio Menu location: Debug -> Other Debug Targets -> Child Process Debugging Settings...
2. Enable child process debugging: true and Save
3. Optionally persist the settings using the dropdown so that this setting can be checked into source control

If you choose to persist the settings, your settings file might look something like this:

<?xml version="1.0" encoding="utf-8"?>
<!-- EngineFilter Guid was found here: https://blogs.msdn.microsoft.com/martintracy/2006/05/16/debug-engine-guids/ -->
<ChildProcessDebuggingSettings IsEnabled="true" xmlns="http://schemas.microsoft.com/vstudio/ChildProcessDebuggingSettings/2014">
    <DefaultRule Attach="false" />
    <Rule IsEnabled="true" ProcessName="TE.ProcessHost.Managed.exe" EngineFilter="{92ef0900-2251-11d2-b72e-0000f87572ef}" />
    <Rule IsEnabled="true" ProcessName="vstest.discoveryengine.exe" EngineFilter="{92ef0900-2251-11d2-b72e-0000f87572ef}" />
    <Rule IsEnabled="true" ProcessName="vstest.discoveryengine.x86.exe" EngineFilter="{92ef0900-2251-11d2-b72e-0000f87572ef}" />
    <Rule IsEnabled="true" ProcessName="vstest.executionengine.exe" EngineFilter="{92ef0900-2251-11d2-b72e-0000f87572ef}" />
    <Rule IsEnabled="true" ProcessName="vstest.executionengine.x86.exe" EngineFilter="{92ef0900-2251-11d2-b72e-0000f87572ef}" />
</ChildProcessDebuggingSettings>

Once this has been setup, you then just need to make sure your project is setup to debug with vstest.console.exe. The key point here is making sure that you enable native/unmanaged debugging, otherwise the child process debugging tool wont work.

New csproj System

Edit or create a launchSettings.json file to look similar to this:

{
    "profiles": {
        "DebugTestAdapter": {
            "commandName": "Executable",
            "executablePath": "C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Enterprise\\Common7\\IDE\\Extensions\\TestPlatform\\vstest.console.exe",
            "commandLineArgs": "Tests.dll --ListTests --TestAdapterPath:.",
            "workingDirectory": "C:\\Projects\\TestAdapter\\Tests\\bin\\Debug\\net46"
        }
    }
}

Modify your csproj file to contain the following property, which enabled native debugging:

<PropertyGroup>
    <EnableUnmanagedDebugging>true</EnableUnmanagedDebugging>
</PropertyGroup>

Old csproj System

In the debug properties page of your project, set the following settings:

Start external program:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe

Command line arguments:

Tests.dll --ListTests --TestAdapterPath:.

Working directory:

C:\Projects\TestAdapter\Tests\bin\Debug

Enable native code debugging:
Set this value to true

[sandeepiiit]

@codito Do VSTEST_RUNNER_DEBUG and VSTEST_HOST_DEBUG also work for version 14.0 of vstest.console.exe? I am trying it out, but it doesn't wait for the debugger to be attached.

[codito]

@sandeepiiit no, it is only supported in newer versions of vstest.console. For older versions, try Debugger.Launch if you have the source code or dnspy if source code is not available.

…

On Thu 22 Mar, 2018, 5:29 AM sandeepiiit, ***@***.***> wrote: @codito <https://github.com/codito> Do VSTEST_RUNNER_DEBUG and VSTEST_HOST_DEBUG also work for version 14.0 of vstest.console.exe? I am trying it out, but it doesn't wait for the debugger to be attached. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/dotnet/cli/issues/5136#issuecomment-375134559>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABwXnttyrotARE-TA-FdYQIFl0xEByAks5tgulxgaJpZM4LUjax> .

[github-actions]

Due to lack of recent activity, this issue has been labeled as 'stale'. It will be closed if no further activity occurs within 30 more days. Any new comment will remove the label.

[github-actions]

This issue will now be closed since it has been labeled 'stale' without activity for 30 days.