Skip to content
This repository has been archived by the owner on Oct 12, 2022. It is now read-only.

Latest commit

 

History

History
248 lines (189 loc) · 11.6 KB

DeveloperGuide.md

File metadata and controls

248 lines (189 loc) · 11.6 KB
Raw

Implementation Details

"How This Whole Crazy Thing Works: An extended apology"
by Steve Dower

Intent

We are building an extension for Visual Studio that integrates into many (most) of its features. We would like to automatically test this extension.

In general, automated tests for extensions are most efficient when the platform is simulated ("mocked out"). Over time, we aim to mock out as much of VS as we can, but integration testing can only reasonably take place within a running VS instance.

This package (VSTestHost) allows automated tests to be launched from VS or MSTest that run in the context of a Visual Studio instance. This enables developers to validate the product in its running context.

Making all of this work is messy, and this document is meant to help whoever has to touch it next.

Requirements

  • Write and run tests using the normal VS unit testing framework
  • Execute tests in a selectable VS instance (SKU, version, hive, etc.)
  • Debug tests by selecting "Debug selected tests" within VS
  • Resilient to VS crashes during test runs

Architecture

There are three executable processes involved when running these tests. Within VSTestHost, they are referred to as the TESTER, the TESTEE, and the EXECUTION ENGINE. Another concept is the TEST ADAPTER, which is a .NET class.

The TESTER is the instance of Visual Studio that the developer is using to launch tests. If the developer is debugging tests, the TESTER is the VS instance that will attach to the other instance.

When running tests from the command line, there is no TESTER.

The TESTEE is the instance of Visual Studio where the test will actually run. The testsettings file selected by the TESTER will determine which version, SKU, and hive of Visual Studio will be started as the TESTEE.

The TESTEE is launched and terminated by the TEST ADAPTER from the EXECUTION ENGINE

The EXECUTION ENGINE is a separate process that allows unit tests to run with different process-wide settings to what the TESTER was launched with. When tests are launched from the TESTER, it will start the EXECUTION ENGINE with a request to start running the selected tests.

  • The EXECUTION ENGINE is not aware of the TESTER (which may not exist)

The TEST ADAPTER is a .NET class that is loaded in the EXECUTION ENGINE and controls the test execution sequence. The TEST ADAPTER is initialized for a run and is notified when the run is paused, resumed, stopped, or aborted. The TEST ADAPTER is passed each test in turn to be executed and the result passed back via a result sink.

VSTestHost includes two TEST ADAPTERs to support the IPC required to run tests in the TESTEE. The TEST ADAPTER loaded in the EXECUTION ENGINE is responsible for marshalling calls (including error handling) to the TESTEE's TEST ADAPTER, which is responsible for executing the test.

Neither TEST ADAPTER is the "real" unit test adapter, so executing a test looks more like instantiating another ITestAdapter and invoking its Run().

Test Settings

VSTestHost depends on the use of a testsettings file or TestProperty attributes to specify the SKU and hive of VS to use as the TESTEE. Setting the version is possible but not recommended - by default, the same version will be used for the TESTEE as the TESTER, which is most stable. The available settings are as follows:

Setting Description
VSApplication The registry key name, like "VisualStudio" or "WDExpress"
VSExecutable The executable name, like "devenv" or "wdexpress"
VSVersion The version number, like "12.0" or "14.0"
VSHive The hive name, like "Exp" or "Default"
VSLaunchTimeoutInSeconds [opt] The number of seconds to wait for launch
VSDebugMixedMode True to use mixed-mode debugging for tests
VSReuseInstance [opt] False to disable the reuse of the same VS instance for distinct tests
ScreenCapture [opt] Relative path to capture screenshots to
ScreenCaptureInterval [opt] Milliseconds between screenshots

Note that identical screenshots are not saved.

Run Test Sequence

Because a TESTEE instance has no way of knowing whether it is actually a TESTEE or just an instance of VS with VSTestHost installed, all instances will open an IPC server and wait a short period of time for incoming connections. This server channel is uniquely named for the process and will not collide with other VS instances. The TESTEE's TEST ADAPTER is made available over this IPC channel.

The typical execution sequence looks like this:

  1. Developer clicks "Run selected test" in the TESTER
  2. TESTER launches EXECUTION ENGINE
  3. EXECUTION ENGINE loads the TEST ADAPTER
  4. EXECUTION ENGINE calls Initialize() on the TEST ADAPTER
  5. EXECUTION ENGINE's TEST ADAPTER reads the configuration and launches the TESTEE
  6. TESTEE opens an IPC channel that is unique to the process and publishes its own TEST ADAPTER
  7. EXECUTION ENGINE's TEST ADAPTER connects to the TESTEE's TEST ADAPTER and returns from Initialize().
  8. EXECUTION ENGINE calls its TEST ADAPTER's Run() method for each test.
  9. EXECUTION ENGINE's TEST ADAPTER calls TESTEE's TEST ADAPTER's Run() method
  10. TESTEE's TEST ADAPTER instantiates/caches the real unit test adapter
  11. TESTEE's TEST ADAPTER invokes the unit test adapter's Run() method
  12. Repeat 8-11 for each test.
  13. EXECUTION ENGINE calls its TEST ADAPTER's PreTestRunFinished() and Cleanup() methods
  14. EXECUTION ENGINE's TEST ADAPTER exits the TESTEE.
  15. Test run is complete.

Debug Test Sequence

Debugging is more complicated, because the TESTER needs to perform debug/attach but only the EXECUTION ENGINE knows the TESTEE's process ID. However, the EXECUTION ENGINE does not know the TESTER's process ID, and so cannot send a message directly to the TESTER. Instead, we use a global IPC server that is only active for a period of time after the TESTER begins debugging.

Reliably detecting when a debugging session is for a unit test requires extra dependencies for the package loaded in the TESTER, which may prevent the assembly from loading or being useful in as many VS configurations as we would like. With our current architecture, we have a single DLL that is installed into the GAC and is loaded in all instances of VS, whether they will be the TESTER, the TESTEE, or are not taking part in unit tests at all. An alternate implementation would require three assemblies with two VSPackages and at least one vsixmanifest for the TESTER's package, as well as multiple dependencies between these assemblies and other dependencies that lead to a complicated and error-prone deployment process.

Rather than dealing with this, we will open the debug IPC channel each time any VS instance starts debugging. If the channel is already open, the new instance will connect to it and signal it to terminate, so that the most recent debugging session is the one listening. This results in a potential race if the user starts multiple debugging sessions where at least one is supposed to attach to a TESTEE, but since this race is with a human, it is considered unlikely. If the EXECUTION ENGINE needs to attach but is unable to find an IPC server, it will abort the run and no tests are executed. The workaround is to restart debugging. There will also be more IPC channels opened than necessary, since we will open one for every debugging session unless we know the instance is a TESTEE. These are not much more expensive than creating a named pipe, so it is considered a worthwhile cost.

In short, debugging works fine unless you try to break it (for example, by starting to debug a test then quickly starting a separate debugging session in another VS instance).

The typical debug execution sequence looks like this: (Steps prefixed with an asterisk are different from the Run Test sequence.)

  1. *Developer clicks "Debug selected test" in the TESTER
  2. *TESTER launches EXECUTION ENGINE under its managed debugger.
  3. *TESTER opens the debug IPC server. If another process has opened the server already, then TESTER tells it to close.
  4. EXECUTION ENGINE loads the TEST ADAPTER
  5. EXECUTION ENGINE calls Initialize() on the TEST ADAPTER
  6. EXECUTION ENGINE's TEST ADAPTER reads the configuration and launches the TESTEE
  7. TESTEE opens an IPC channel that is unique to the process and publishes its own TEST ADAPTER
  8. *EXECUTION ENGINE's TEST ADAPTER connects to the TESTEE's TEST ADAPTER
  9. *EXECUTION ENGINE's TEST ADAPTER connects to the debug IPC server and tells it to attach to TESTEE.
  10. *EXECUTION ENGINE's TEST ADAPTER returns from Initialize().
  11. *TESTER attaches its debugger to TESTEE.
  12. EXECUTION ENGINE calls its TEST ADAPTER's Run() method for each test.
  13. EXECUTION ENGINE's TEST ADAPTER calls TESTEE's TEST ADAPTER's Run() method
  14. TESTEE's TEST ADAPTER instantiates/caches the real unit test adapter
  15. TESTEE's TEST ADAPTER invokes the unit test adapter's Run() method
  16. Repeat 12-15 for each test.
  17. EXECUTION ENGINE calls its TEST ADAPTER's PreTestRunFinished() and Cleanup() methods
  18. EXECUTION ENGINE's TEST ADAPTER exits the TESTEE.
  19. Test run is complete.

VSTestHost Classes

This is an overview of the classes within VSTestHost and how they are used. Some classes are used in multiple places with slightly different purposes, and because not every VS SKU or version can support being a TESTER, some parts are #if'd out to allow separate builds for those. Because of the reference to Microsoft.VisualStudio.Shell.##.0, each VS version requires its own build of VSTestHost.

VSTestHostPackage

This package is auto-loaded in every instance of VS.

For TESTEEs (bearing in mind that we always initially assume this), package initialization creates the IPC channel and publishes its test adapter. This channel is not closed until the package is disposed at process exit.

For TESTERs (again, this is always assumed initially), the debugging state is hooked so the debug IPC channel can be activated/deactivated as necessary. Channel creation is handled by the static members of TesterDebugAttacher. When the timeout expires, another TESTER manually aborts the wait, or an EXECUTION ENGINE signals to attach, the channel is closed.

TesterDebugAttacher

This class is published over the debug IPC channel to allow the EXECUTION ENGINE to instruct the debugger in the TESTER to attach to the TESTEE. Other TESTERs may also use it to abort listeners so they can take over the channel.

The class has static methods for use by the TESTER to simplify listening for commands.

TesterTestAdapter

This class is loaded in the EXECUTION ENGINE to run tests that have a [HostType("VSTestHost")] attribute. It is largely responsible for connecting to an instance of TesteeTestAdapter and marshalling calls to the TESTEE.

TesteeTestAdapter

This class is published from the TESTEE on the unique IPC channel created by VSTestHostPackage. It implements the logic for running arbitrary tests within VS by loading the intended test adapter.

VisualStudio

This class encapsulates the logic for starting VS and getting its DTE object from the EXECUTION ENGINE. Currently, DTE is only used by the EXECUTION ENGINE to safely exit VS at the end of a test run, but in the future could be used for tests that need to drive VS from outside the TESTEE process.

VSTestContext

This class is used by tests running within the TESTEE to easily access the VS instance's DTE and global ServiceProvider objects. Unit test projects should include the following reference for VSTestHost:

<Reference Include="Microsoft.VisualStudioTools.VSTestHost.$(VSTarget)" />

Test projects that do not refer to VSTestContext do not require this reference.

VSTestContext is the only public and supported class in the Microsoft.VisualStudioTools.VSTestHost assembly. All other public classes are infrastructure.