xunit report with scenario outline has inconsistent tests count / testcases #432

Open
fcostin opened this Issue Feb 24, 2014 · 5 comments

2 participants

@fcostin

To replicate:

  1. set up the minimal scenario outline example as described here: http://lettuce.it/tutorial/scenario-outlines.html

  2. run lettuce --with-xunit --xunit-file=report.xml

  3. inspect report.xml. note that the <testsuite /> element has tests="15" but there are only 5 <testcase /> elements in the report. alternatively, if xunitparser is installed, observe that this command fails during an assertion sanity-checking the number of test cases parsed from the report: python -c "import xunitparser; xunitparser.parse(open('report.xml'))"

@danni
Collaborator
@fcostin

I'm not familiar with subunit - is there any documentation describing how to use it with lettuce? Perhaps i could compare against lettuce -- subunit report -> convert to xunit report.

@danni
Collaborator
@fcostin

thank you for following this up so quickly!

i've had a try with subunit output, using revision a483847ed8de4485199e7edc587b5fa6d783da15 of lettuce.

when I try the following:

(lettuce --with-subunit --subunit-file=report.subunit.txt || true) && (cat report.subunit.txt | subunit2junitxml > report.junit.txt)

I see the following in report.junit.txt:

<testsuite errors="0" failures="1" name="" tests="1" time="0.047">
<testcase classname="" name="Compute factorial: Factorials [0-4]" time="0.047">
<failure type="testtools.testresult.real._StringException">_StringException: Empty attachments:
  stderr
  stdout

steps: {{{
✔ Given I have the number 0
✔ When I compute its factorial
❌ Then I see the number 1
✔ Given I have the number 1
✔ When I compute its factorial
❌ Then I see the number 1
✔ Given I have the number 2
✔ When I compute its factorial
❌ Then I see the number 2
✔ Given I have the number 3
✔ When I compute its factorial
❌ Then I see the number 6
✔ Given I have the number 4
✔ When I compute its factorial
❌ Then I see the number 24
}}}

Traceback (most recent call last):
  File "/mnt/hgfs/c_projects/lettuce/lettuce/core.py", line 144, in __call__
    ret = self.function(self.step, *args, **kw)
  File "/mnt/hgfs/c_projects/lettuce_bug/features/steps.py", line 15, in check_number
    "Got %d" % world.number
AssertionError: Got -1
Traceback (most recent call last):
  File "/mnt/hgfs/c_projects/lettuce/lettuce/core.py", line 144, in __call__
    ret = self.function(self.step, *args, **kw)
  File "/mnt/hgfs/c_projects/lettuce_bug/features/steps.py", line 15, in check_number
    "Got %d" % world.number
AssertionError: Got -1
Traceback (most recent call last):
  File "/mnt/hgfs/c_projects/lettuce/lettuce/core.py", line 144, in __call__
    ret = self.function(self.step, *args, **kw)
  File "/mnt/hgfs/c_projects/lettuce_bug/features/steps.py", line 15, in check_number
    "Got %d" % world.number
AssertionError: Got -1
Traceback (most recent call last):
  File "/mnt/hgfs/c_projects/lettuce/lettuce/core.py", line 144, in __call__
    ret = self.function(self.step, *args, **kw)
  File "/mnt/hgfs/c_projects/lettuce_bug/features/steps.py", line 15, in check_number
    "Got %d" % world.number
AssertionError: Got -1
Traceback (most recent call last):
  File "/mnt/hgfs/c_projects/lettuce/lettuce/core.py", line 144, in __call__
    ret = self.function(self.step, *args, **kw)
  File "/mnt/hgfs/c_projects/lettuce_bug/features/steps.py", line 15, in check_number
    "Got %d" % world.number
AssertionError: Got -1

</failure>
</testcase>
</testsuite>

this fixes the exact issue i mentioned above (difference between number of <testsuite> elements and the tests value in the root <testsuite /> element, but the fact that the output is all inside a single <testcase> element seems undesirable. it would be preferable to have each instance of the scenario parameters generate its own <testcase> as in the current behaviour.

subunit2pyunit and subunit2gtk similarly report 1 failure out of 1 test case.

@fcostin

when i fix the obvious bug in the factorial definition in the example lettuce features steps.py and re-run the above, lettuce reports all 15 tests pass, but it still shows up as 1 failure in junit'd subunit output:

<testsuite errors="0" failures="1" name="" tests="1" time="0.032">
<testcase classname="" name="Compute factorial: Factorials [0-4]" time="0.032">
<failure type="testtools.testresult.real._StringException">_StringException: Empty attachments:
  stderr
  stdout

steps: {{{
✔ Given I have the number 0
✔ When I compute its factorial
✔ Then I see the number 1
✔ Given I have the number 1
✔ When I compute its factorial
✔ Then I see the number 1
✔ Given I have the number 2
✔ When I compute its factorial
✔ Then I see the number 2
✔ Given I have the number 3
✔ When I compute its factorial
✔ Then I see the number 6
✔ Given I have the number 4
✔ When I compute its factorial
✔ Then I see the number 24
}}}

</failure>
</testcase>
</testsuite>

apologies if i am doing something wildly wrong here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment