Reporting

Stephen McCafferty edited this page Oct 1, 2018 · 14 revisions

SpecFlow provides various options to generate reports related to the acceptance tests.

Note: The specflow.exe command line tool that is used to generate reports can be found in the packages\Specflow.{version number}\tools directory, when you installed SpecFlow through NuGet. Start the tool with no parameters or use the --help option to display an overview of the available options.

Test Execution Report

This report provides a formatted HTML report of a test execution. The report contains a summary about the executed tests and the result and also a detailed report for the individual scenario executions.

See the following sub-sections for how to generate the test execution report depending on the unit test provider used.

NUnit Test Execution Report

NUnit 2

In order to generate this report you have to execute the acceptance tests with the nunit-console runner. This tool generates an XML summary about the test executions. To have the detailed scenario execution traces visible, you also need to capture the test output using the /out and the /labels options as it can be seen in the following example.

nunit-console.exe /labels /out=TestResult.txt /xml=TestResult.xml bin\Debug\BookShop.AcceptanceTests.dll

NUnit 3

In order to generate this report you have to execute the acceptance tests with the nunit3-console runner set to output the results as nunit2 format. To have the detailed scenario execution traces visible, you also need to capture the test output using the --out and the --labels=All options as it can be seen in the example below. Important: you'll need the NUnit.Extension.NUnitV2ResultWriter package in your project, otherwise you'll receive the message: "Unknown result format: nunit2"

nunit3-console.exe --labels=All --out=TestResult.txt "--result=TestResult.xml;format=nunit2" bin\Debug\BookShop.AcceptanceTests.dll

SpecFlow Report Generator

(examples and parameters are for version 2.4.* and above, for docs old version take a look in the history of this help)

The report generation step is the same for both versions of nunit. The two generated files can be used to invoke the SpecFlow report generation. If you use the default output file names shown above it enough to specify info about C# test project containing *.feature files, then specflow takes default files TestResult.xml and TestResult.txt to produce default named TestResult.html

specflow.exe nunitexecutionreport --ProjectFile BookShop.AcceptanceTests.csproj
specflow.exe nunitexecutionreport --ProjectFile BookShop.AcceptanceTests.csproj --xmlTestResult CustomNunitTestResult.xml --testOutput CustomNunitTestOutput.txt --OutputFile CustomSpecflowTestReport.html

The following table contains the possible arguments for this command.

Attribute Value Description
-p, --ProjectFile A path of the project file containing the *.feature files. Required.
-t, --xmlTestResult NUnit XML test result file The XML test result file generated by nunit-console. Optional.
Default: TestResult.xml
--testOutput NUnit test output file The labeled test output file generated by nunit-console. Optional.
Default: TestResult.txt
--OutputFile HTML file Generated Output File. Optional.
Default: TestResult.html
-x, --XsltFile XSLT file Custom XSLT file to use, defaults to built-in stylesheet if not provided. Optional.
Default: not specified
-n, --ProjectName Project name Project name which can be passed explicitly instead of implicitly getting it from --ProjectFile. Optional.
Default: not specified
-l, --FeatureLanguage Feature language Default: en-US

MsTest Test Execution Report

specflow.exe mstestexecutionreport BookShop.AcceptanceTests.csproj /testResult:result.trx /out:MyResult.html

The following table contains the possible arguments for this command.

Attribute Value Description
-p, --ProjectFile A path of the project file containing the feature files. This is a mandatory argument.
-t, --testResult MsTest test result (trx) file The TRX test result file generated by MsTest. Optional.
Default: TestResult.trx
-n, --ProjectName Project name Project name which can be passed explicitly instead of implicitly getting it from --ProjectFile. Optional.
Default: not specified
-o, --OutputFile HTML file Generated Output File. Optional.
Default: TestResult.html
-l, --FeatureLanguage Feature language Default: en-US
-x, --XsltFile XSLT file Custom XSLT file to use, defaults to built-in stylesheet if not provided. Optional.
Default: not specified

Step Definition Report

This report shows the usage and binding status of the steps for the entire project. You can use this report to find both unused code in the automation layer and scenario steps that have no definition yet.

  • Steps with a red background are steps that exist in the automation layer but are not used in any feature files.
  • Steps with a yellow background are steps that exist in a feature file but do not have a corresponding definition.
  • Steps without a special backgrounds are steps that exist both in feature files and the automation layer. Ideally, all your steps are like this.
specflow.exe stepdefinitionreport BookShop.AcceptanceTests.csproj /BinFolder:bin/debug

The following table contains the possible arguments for this command.

Attribute Value Description
projectFile A path of the project file containing the feature files. This is a mandatory argument.
/binFolder A folder path (absolute or relative to the project folder) A path for the compiled SpecFlow project. Optional.
Default: bin\Debug
/out HTML file Generated Output File. Optional.
Default: StepDefinitionReport.html
/xsltFile XSLT file Custom XSLT file to use, defaults to built-in stylesheet if not provided. Optional.
Default: not specified
Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.