Skip to content

Gherkin: bloated NUnit test report #1179

Closed
Closed
Gherkin: bloated NUnit test report#1179
@gnuechtel

Description

@gnuechtel
gnuechtel
opened on Dec 20, 2018

1. General summary of the issue

The created NUnit-Test report after Invoke-Gherkin has unnecessary nested elements.
Additionally the order is wrong when steps are used repeatedly.

The reason is the following code in Gherkin.ps1:

$Pester.AddTestResult($DisplayText, ${Pester Result}.Result, $Elapsed, $PesterErrorRecord.Exception.Message, ${Pester Result}.StackTrace, $Source, $NamedArguments, $PesterErrorRecord)

$Source is used as ParameterizedSuiteName to group the test results.

This is problematic for two reasons:

  • It introduces a new nested test-suite/results elements and bloats the result XML
  • It creates test-case elements which order is different to the execution order

2. Describe Your Environment

Pester version     : 4.5.0 D:\Documents\WindowsPowerShell\Modules\Pester\Pester.psm1
PowerShell version : 5.1.15063.1446
OS version         : Microsoft Windows NT 10.0.15063.0

3. Expected Behavior

Given the following feature:

Feature: A test feature for reporting 1

  Scenario: Scenario 1

    Given step_001
    When step_002
    Then step_003

  Scenario Outline: Scenario 2

    Given step_<given>
      And and_<given>
    When step_<when>
      And and_<when>
    Then step_<then>
      And and_<then>

    Examples: Examples 1
      | given | when | then |
      | 101   | 102  | 103  |

    Examples: Examples 2
      | given | when | then |
      | 201   | 202  | 203  |

  Scenario: Scenario 3

    Given step_301
    When step_302
    Then step_303
    When step_302
    Then step_304

The expected NUnit test report would look like this:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-results xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="nunit_schema_2.5.xsd" name="Pester" total="20" errors="0" failures="2" not-run="0" inconclusive="0" ignored="0" skipped="0" invalid="0" date="2018-12-20" time="20:12:14">
  <environment user="cgnuechtel" machine-name="myhost" cwd="D:\Development\workspaces\git\pester-temp" user-domain="XYZ" platform="Microsoft Windows 10 Enterprise|C:\windows|\Device\Harddisk1\Partition3" nunit-version="2.5.8.0" os-version="10.0.15063" clr-version="4.0.30319.42000" />
  <culture-info current-culture="en-US" current-uiculture="en-US" />
  <test-suite type="TestFixture" name="Pester" executed="True" result="Failure" success="False" time="0.0103" asserts="0" description="Pester">
    <results>
      <test-suite type="TestFixture" name="D:\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting1.feature" executed="True" result="Success" success="True" time="0" asserts="0" description="D:\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting1.feature">
        <results>
          <test-suite type="TestFixture" name="Scenario 1" executed="True" result="Success" success="True" time="0" asserts="0" description="Scenario 1">
            <results>
              <test-case description="Given step_001" name="Scenario 1.Given step_001" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="When step_002" name="Scenario 1.When step_002" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="Then step_003" name="Scenario 1.Then step_003" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
            </results>
          </test-suite>
          <test-suite type="TestFixture" name="Scenario 2&#xA;  Examples: Examples 1" executed="True" result="Success" success="True" time="0" asserts="0" description="Scenario 2&#xA;  Examples: Examples 1">
            <results>
              <test-case description="Given step_101" name="Scenario 2&#xA;  Examples: Examples 1.Given step_101" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="And and_101" name="Scenario 2&#xA;  Examples: Examples 1.And and_101" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="When step_102" name="Scenario 2&#xA;  Examples: Examples 1.When step_102" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="And and_102" name="Scenario 2&#xA;  Examples: Examples 1.And and_102" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="Then step_103" name="Scenario 2&#xA;  Examples: Examples 1.Then step_103" time="0.0014" asserts="0" success="False" result="Failure" executed="True">
                <failure>
                  <message>An example error</message>
                  <stack-trace>at &lt;ScriptBlock&gt;, D:\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 23
23:     throw "An example error"

From D:\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting1.feature: line 15</stack-trace>
                </failure>
              </test-case>
              <test-case description="And and_103" name="Scenario 2&#xA;  Examples: Examples 1.And and_103" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
            </results>
          </test-suite>
          <test-suite type="TestFixture" name="Scenario 2&#xA;  Examples: Examples 2" executed="True" result="Success" success="True" time="0" asserts="0" description="Scenario 2&#xA;  Examples: Examples 2">
            <results>
              <test-case description="Given step_201" name="Scenario 2&#xA;  Examples: Examples 2.Given step_201" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="And and_201" name="Scenario 2&#xA;  Examples: Examples 2.And and_201" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="When step_202" name="Scenario 2&#xA;  Examples: Examples 2.When step_202" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="And and_202" name="Scenario 2&#xA;  Examples: Examples 2.And and_202" time="0.0005" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="Then step_203" name="Scenario 2&#xA;  Examples: Examples 2.Then step_203" time="0.0005" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="And and_203" name="Scenario 2&#xA;  Examples: Examples 2.And and_203" time="0.0006" asserts="0" success="True" result="Success" executed="True" />
            </results>
          </test-suite>
          <test-suite type="TestFixture" name="Scenario 3" executed="True" result="Success" success="True" time="0" asserts="0" description="Scenario 3">
            <results>
              <test-case description="Given step_301" name="Scenario 3.Given step_301" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="When step_302" name="Scenario 3.When step_302" time="0.0005" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="Then step_303" name="Scenario 3.Then step_303" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="When step_302" name="Scenario 3.When step_302" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
              <test-case description="Then step_304" name="Scenario 3.Then step_304" time="0.0017" asserts="0" success="False" result="Failure" executed="True">
                <failure>
                  <message>Another example error</message>
                  <stack-trace>at &lt;ScriptBlock&gt;, D:\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 57
57:     throw "Another example error"

From D:\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting1.feature: line 32</stack-trace>
                </failure>
              </test-case>
            </results>
          </test-suite>
        </results>
      </test-suite>
    </results>
  </test-suite>
</test-results>

4.Current Behavior

The current implementation produces the following XML:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-results xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="nunit_schema_2.5.xsd" name="Pester" total="20" errors="0" failures="2" not-run="0" inconclusive="0" ignored="0" skipped="0" invalid="0" date="2018-12-20" time="20:17:23">
  <environment user="cgnuechtel" machine-name="myhost" cwd="D:\Development\workspaces\git\pester-temp" user-domain="XYZ" platform="Microsoft Windows 10 Enterprise|C:\windows|\Device\Harddisk1\Partition3" nunit-version="2.5.8.0" os-version="10.0.15063" clr-version="4.0.30319.42000" />
  <culture-info current-culture="en-US" current-uiculture="en-US" />
  <test-suite type="TestFixture" name="Pester" executed="True" result="Failure" success="False" time="0.0111" asserts="0" description="Pester">
    <results>
      <test-suite type="TestFixture" name="D:\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting1.feature" executed="True" result="Success" success="True" time="0" asserts="0" description="D:\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting1.feature">
        <results>
          <test-suite type="TestFixture" name="Scenario 1" executed="True" result="Success" success="True" time="0" asserts="0" description="Scenario 1">
            <results>
              <test-suite type="ParameterizedTest" name="Scenario 1...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 1" executed="True" result="Success" success="True" time="0.0035" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 1">
                <results>
                  <test-case description="Given step_001" name="Scenario 1.Given step_001" time="0.0035" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 1...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 4" executed="True" result="Success" success="True" time="0.0005" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 4">
                <results>
                  <test-case description="When step_002" name="Scenario 1.When step_002" time="0.0005" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 1...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 7" executed="True" result="Success" success="True" time="0.0003" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 7">
                <results>
                  <test-case description="Then step_003" name="Scenario 1.Then step_003" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
            </results>
          </test-suite>
          <test-suite type="TestFixture" name="Scenario 2&#xA;  Examples: Examples 1" executed="True" result="Success" success="True" time="0" asserts="0" description="Scenario 2&#xA;  Examples: Examples 1">
            <results>
              <test-suite type="ParameterizedTest" name="Scenario 2&#xA;  Examples: Examples 1...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 10" executed="True" result="Success" success="True" time="0.0003" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 10">
                <results>
                  <test-case description="Given step_101" name="Scenario 2&#xA;  Examples: Examples 1.Given step_101" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 2&#xA;  Examples: Examples 1...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 13" executed="True" result="Success" success="True" time="0.0003" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 13">
                <results>
                  <test-case description="And and_101" name="Scenario 2&#xA;  Examples: Examples 1.And and_101" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 2&#xA;  Examples: Examples 1...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 16" executed="True" result="Success" success="True" time="0.0003" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 16">
                <results>
                  <test-case description="When step_102" name="Scenario 2&#xA;  Examples: Examples 1.When step_102" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 2&#xA;  Examples: Examples 1...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 19" executed="True" result="Success" success="True" time="0.0003" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 19">
                <results>
                  <test-case description="And and_102" name="Scenario 2&#xA;  Examples: Examples 1.And and_102" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 2&#xA;  Examples: Examples 1...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 22" executed="True" result="Failure" success="False" time="0.0012" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 22">
                <results>
                  <test-case description="Then step_103" name="Scenario 2&#xA;  Examples: Examples 1.Then step_103" time="0.0012" asserts="0" success="False" result="Failure" executed="True">
                    <failure>
                      <message>An example error</message>
                      <stack-trace>at &lt;ScriptBlock&gt;, D:\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 23
23:     throw "An example error"

From D:\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting1.feature: line 15</stack-trace>
                    </failure>
                  </test-case>
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 2&#xA;  Examples: Examples 1...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 26" executed="True" result="Success" success="True" time="0.0003" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 26">
                <results>
                  <test-case description="And and_103" name="Scenario 2&#xA;  Examples: Examples 1.And and_103" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
            </results>
          </test-suite>
          <test-suite type="TestFixture" name="Scenario 2&#xA;  Examples: Examples 2" executed="True" result="Success" success="True" time="0" asserts="0" description="Scenario 2&#xA;  Examples: Examples 2">
            <results>
              <test-suite type="ParameterizedTest" name="Scenario 2&#xA;  Examples: Examples 2...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 29" executed="True" result="Success" success="True" time="0.0003" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 29">
                <results>
                  <test-case description="Given step_201" name="Scenario 2&#xA;  Examples: Examples 2.Given step_201" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 2&#xA;  Examples: Examples 2...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 32" executed="True" result="Success" success="True" time="0.0003" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 32">
                <results>
                  <test-case description="And and_201" name="Scenario 2&#xA;  Examples: Examples 2.And and_201" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 2&#xA;  Examples: Examples 2...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 35" executed="True" result="Success" success="True" time="0.0004" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 35">
                <results>
                  <test-case description="When step_202" name="Scenario 2&#xA;  Examples: Examples 2.When step_202" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 2&#xA;  Examples: Examples 2...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 38" executed="True" result="Success" success="True" time="0.0003" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 38">
                <results>
                  <test-case description="And and_202" name="Scenario 2&#xA;  Examples: Examples 2.And and_202" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 2&#xA;  Examples: Examples 2...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 41" executed="True" result="Success" success="True" time="0.0003" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 41">
                <results>
                  <test-case description="Then step_203" name="Scenario 2&#xA;  Examples: Examples 2.Then step_203" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 2&#xA;  Examples: Examples 2...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 44" executed="True" result="Success" success="True" time="0.0004" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 44">
                <results>
                  <test-case description="And and_203" name="Scenario 2&#xA;  Examples: Examples 2.And and_203" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
            </results>
          </test-suite>
          <test-suite type="TestFixture" name="Scenario 3" executed="True" result="Success" success="True" time="0" asserts="0" description="Scenario 3">
            <results>
              <test-suite type="ParameterizedTest" name="Scenario 3...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 47" executed="True" result="Success" success="True" time="0.0003" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 47">
                <results>
                  <test-case description="Given step_301" name="Scenario 3.Given step_301" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 3...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 50" executed="True" result="Success" success="True" time="0.0006" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 50">
                <results>
                  <test-case description="When step_302" name="Scenario 3.When step_302" time="0.0004" asserts="0" success="True" result="Success" executed="True" />
                  <test-case description="When step_302" name="Scenario 3.When step_302" time="0.0002" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 3...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 53" executed="True" result="Success" success="True" time="0.0003" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 53">
                <results>
                  <test-case description="Then step_303" name="Scenario 3.Then step_303" time="0.0003" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="ParameterizedTest" name="Scenario 3...\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 56" executed="True" result="Failure" success="False" time="0.0013" asserts="0" description="..\..\..\..\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 56">
                <results>
                  <test-case description="Then step_304" name="Scenario 3.Then step_304" time="0.0013" asserts="0" success="False" result="Failure" executed="True">
                    <failure>
                      <message>Another example error</message>
                      <stack-trace>at &lt;ScriptBlock&gt;, D:\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting.Steps.ps1: line 57
57:     throw "Another example error"

From D:\Documents\WindowsPowerShell\Modules\Pester\Examples\Gherkin\JustForReporting1.feature: line 32</stack-trace>
                    </failure>
                  </test-case>
                </results>
              </test-suite>
            </results>
          </test-suite>
        </results>
      </test-suite>
    </results>
  </test-suite>
</test-results>

The nested elements are just ugly.
I think, they are not necessary.

The grouping for repeated steps are a greater problem, because the order is changed.
This is problematic for further tools like XSLT which produces HTML reports for example.

5. Possible Solution

A pull request will be provided very soon, which will contain tests.

The fix is very simple. Just replace $Source by $null.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions