Skip to content

Latest commit

 

History

History
207 lines (140 loc) · 7.65 KB

report.md

File metadata and controls

207 lines (140 loc) · 7.65 KB
Raw

Reporting test results

Test discovery, execution results in a test run can be controlled with test loggers. This document will cover details on installation, usage and authoring of test loggers.

Test loggers

A test logger is a test platform extension to control reporting of test results. It can perform tasks when a test run message, individual test results or the test run completion events are reported by the test platform.

You can author a test logger to print messages on the console, generate result files of a specific reporting format, or even report results to various CI/CD services. Default inputs to a test logger can be provided in the command line.

Please refer to this section for instructions on creating a test logger and todo if you're interested in the architecture of a test logger.

Available test loggers

Scenario Nuget Package Source Repository
Local, CI, CD Inbuilt Trx Logger
Local, CI, CD Inbuilt Console Logger
Local, CI, CD Inbuilt Html Logger
Local, CI, CD XunitXml.TestLogger Xunit Logger
Local, CI, CD NunitXml.TestLogger Nunit Logger
Local, CI, CD JunitXml.TestLogger Junit Logger
AppVeyor AppVeyor.TestLogger AppVeyor Logger
Azure Pipelines AzurePipelines.TestLogger Azure Pipelines Logger
GitHub Actions GitHubActionsTestLogger GitHub Actions Test Logger
TeamCity TeamCity.VSTest.TestAdapter Teamcity Logger

Want to add your logger? Please send a PR with changes in this doc.

Acquisition

A test logger should be made available as a NuGet package (preferred), or as a zip file (for e.g. loggers for C++ etc.).

If it's a NuGet package, the test logger assemblies should get copied to the build output directory. When looking for a test logger, vstest will look for them in the same directory as the test assemblies. In most cases this means that test projects reference either the project (same codebase) or NuGet package with the logger.

If the test logger is made available as a zip file, it should be extracted to one of the following locations:

  1. the Extensions folder along side vstest.console.exe. E.g. in case of dotnet-cli, the path could be /sdk/<version>/Extensions directory.
  2. any well known location on the filesystem

Version Note: new in 15.1 In case of #2, user can specify the full path to the location using /TestAdapterPath:<path> command line switch. Test platform will locate extensions from the provided directory.

Naming

Test platform will look for assemblies named *.testlogger.dll when it's trying to load test loggers.

Version Note: < 15.1 For 15.0 version, the test loggers are also discovered from *.testadapter.dll

Create a test logger

Go through the following steps to create your own logger

  1. Add a nuget reference of package Microsoft.TestPlatform.ObjectModel.
  2. Implement ITestLoggerWithParameters (or ITestLogger, if your logger is not expecting any parameter). Logger Example
  3. Name your logger assembly *.testlogger.dll. Detailed

Enable a test logger

A test logger must be explicitly enabled using the command line. E.g.

 vstest.console test_project.dll /logger:mylogger

Where mylogger is the LoggerUri or FriendlyName of the logger.

Configure reporting

Additional arguments to a logger can also be passed in the command line. E.g.

 vstest.console test_project.dll /logger:mylogger;Setting=Value

Where mylogger is the LoggerUri or FriendlyName of the logger. Setting is the name of the additional argument and Valueis its value.

It is upto the logger implementation to support additional arguments.

Syntax of default loggers

1) Console logger

Console logger is the default logger and it is used to output the test results into console window.

Syntax

For dotnet test or dotnet vstest :
--logger:console[;verbosity=<Defaults to "minimal">]

For vstest.console.exe :
/logger:console[;verbosity=<Defaults to "normal">]
 
Argument "verbosity" define the verbosity level of console logger. Allowed values for verbosity are "quiet", "minimal", "normal" and "detailed".

Example

vstest.console.exe Tests.dll /logger:"console;verbosity=normal"

If you are using "dotnet test", then use the following command

dotnet test Tests.csproj --logger:"console;verbosity=normal"

or you can also use argument "-v | --verbosity" of "dotnet test"

dotnet test Tests.csproj -v normal

2) Trx logger

Trx logger is used to log test results into a Visual Studio Test Results File (TRX).

Syntax

/logger:trx [;LogFileName=<Defaults to unique file name>]

Where "LogFileName" can be absolute or relative path. If path is relative, it will be relative to "TestResults" directory, created under current working directory.

Examples

Suppose the current working directory is "c:\tempDirecory".

1) vstest.console.exe Tests.dll /logger:trx
trx file will get generated in location "c:\tempDirecory\TestResults"

2) vstest.console.exe Tests.dll /logger:"trx;LogFileName=relativeDir\logFile.txt"
trx file will be "c:\tempDirecory\TestResults\relativeDir\logFile.txt"

3) vstest.console.exe Tests.dll /logger:"trx;LogFileName=c:\temp\logFile.txt"
trx file will be "c:\temp\logFile.txt"

3) Html logger

Html logger is used to log test results into a html file.

Syntax

/logger:html [;LogFileName=<Defaults to unique file name>]

Where "LogFileName" can be absolute or relative path. If path is relative, it will be relative to "TestResults" directory, created under current working directory.

Examples

Suppose the current working directory is "c:\tempDirecory".

1) vstest.console.exe Tests.dll /logger:html
Html file will get generated in location "c:\tempDirecory\TestResults"

2) vstest.console.exe Tests.dll /logger:"html;LogFileName=relativeDir\logFile.html"
Html file will be "c:\tempDirecory\TestResults\relativeDir\logFile.html"

3) vstest.console.exe Tests.dll /logger:"html;LogFileName=c:\temp\logFile.html"
Html file will be "c:\temp\logFile.html"

Related links

TODO: link to author a test logger