Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

JUnit formater doesn't report Scenario outline example with -x argument #124

Closed
floreal opened this Issue · 10 comments

9 participants

@floreal

Hello,
I noticed a little annoying bug. consider my Rakefile:

require 'rubygems'
require 'rake'
require 'cucumber/rake/task'

# some other tasks

namespace :cucumber do
  Cucumber::Rake::Task.new(:report) do |t|
    t.cucumber_opts = "-x -p #{browser} -f junit --out report -f pretty --no-color"
  end

  Cucumber::Rake::Task.new(:pretty) do |t|
    t.cucumber_opts = "-x -p #{browser}"
  end
end

The cucumber proffiles only define environment variable to set selenium url and desired browser capabilities

When i execure rake cucumber:report, only Scenarios are reported to the JUnit result report, Scenario Outlines examples aren't.
When I remove the -x flag they are reported.

@floreal

I managed to see what happen actually: when -x argument is passed, whenever an error (even error on asseertions) occurs in a scennario outline case, the other cases of the outline wont execute (and will be considerated as failed)

@tobhahn

Here is a scenario to reproduce the error on today's master.

Steps to reproduce:

  • Save the following feature into legacy_features/junit_formatter_expand.feature:
Feature: JUnit output formatter
  In order for developers to create test reports with ant
  Cucumber should be able to output JUnit xml files

  Background:
    Given I am in junit
    And the tmp directory is empty


  Scenario: Expand scenario outline into one passing and one failing scenario
    When I run cucumber --format junit --expand --out tmp/ features/scenario_outline.feature
    Then it should fail with
      """

      """
    And "fixtures/junit/tmp/TEST-features-scenario_outline.xml" with junit duration "0.005" should contain
      """
      <?xml version="1.0" encoding="UTF-8"?>
      <testsuite errors="0" failures="1" name="Scenario outlines" skipped="0" tests="2" time="0.005">
      <testcase classname="Scenario outlines.Using scenario outlines" name="Using scenario outlines (outline example : | passing |)" time="0.005">
      </testcase>
      <testcase classname="Scenario outlines.Using scenario outlines" name="Using scenario outlines (outline example : | failing |)" time="0.005">
        <failure message="failed Using scenario outlines (outline example : | failing |)" type="failed">
          <![CDATA[Scenario Outline: Using scenario outlines

      Scenario: |failing|

      Given a failing scenario

      Message:
      ]]>
          <![CDATA[ (RuntimeError)
      features/scenario_outline.feature:4:in `Given a <type> scenario']]>
        </failure>
      </testcase>
      </testsuite>

      """
  • Run bin/cucumber legacy_features/junit_formatter_expand.feature.

Expected results:
No error.

Actual results:
The scenario fails with

Using the default profile...
......F

(::) failed steps (::)

expected: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuite errors=\"0\" failures=\"1\" name=\"Scenario outlines\" skipped=\"0\" tests=\"2\" time=\"0.005\">\n<testcase classname=\"Scenario outlines.Using scenario outlines\" name=\"Using scenario outlines (outline example : | passing |)\" time=\"0.005\">\n</testcase>\n<testcase classname=\"Scenario outlines.Using scenario outlines\" name=\"Using scenario outlines (outline example : | failing |)\" time=\"0.005\">\n  <failure message=\"failed Using scenario outlines (outline example : | failing |)\" type=\"failed\">\n    <![CDATA[Scenario Outline: Using scenario outlines\n\nScenario: |failing|\n\nGiven a failing scenario\n\nMessage:\n]]>\n    <![CDATA[ (RuntimeError)\nfeatures/scenario_outline.feature:4:in `Given a <type> scenario']]>\n  </failure>\n</testcase>\n</testsuite>\n"
     got: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuite errors=\"0\" failures=\"0\" name=\"Scenario outlines\" skipped=\"0\" tests=\"0\" time=\"0.005\">\n</testsuite>\n" (using ==)
Diff:
@@ -1,20 +1,4 @@
 <?xml version="1.0" encoding="UTF-8"?>
-<testsuite errors="0" failures="1" name="Scenario outlines" skipped="0" tests="2" time="0.005">
-<testcase classname="Scenario outlines.Using scenario outlines" name="Using scenario outlines (outline example : | passing |)" time="0.005">
-</testcase>
-<testcase classname="Scenario outlines.Using scenario outlines" name="Using scenario outlines (outline example : | failing |)" time="0.005">
-  <failure message="failed Using scenario outlines (outline example : | failing |)" type="failed">
-    <![CDATA[Scenario Outline: Using scenario outlines
-
-Scenario: |failing|
-
-Given a failing scenario
-
-Message:
-]]>
-    <![CDATA[ (RuntimeError)
-features/scenario_outline.feature:4:in `Given a <type> scenario']]>
-  </failure>
-</testcase>
+<testsuite errors="0" failures="0" name="Scenario outlines" skipped="0" tests="0" time="0.005">
 </testsuite>
 (RSpec::Expectations::ExpectationNotMetError)
./legacy_features/step_definitions/cucumber_steps.rb:110:in `/^"([^"]*)" with junit duration "([^"]*)" should contain$/'
legacy_features/junit_formatter_expand.feature:16:in `And "fixtures/junit/tmp/TEST-features-scenario_outline.xml" with junit duration "0.005" should contain'

Failing Scenarios:
cucumber legacy_features/junit_formatter_expand.feature:10 # Scenario: one feature, expand one scenario outline, two examples: one passing, one failing

1 scenario (1 failed)
5 steps (1 failed, 4 passed)
@shyam-prabhakar

Any plans on fixing this, this seems like a fairly big issue for folks who want to use the 'expand' option with the junit formatter for scenario outlines.

@cjapes

This is big issue for the people that are using cucumber with Jenkins, that depend on that junit output.
Any plans on fixing this?

@rtyler

This is affecting me as well, since I was probably one of the last people to touch the JUnit formatter, I'll look into fixing this up.

@mattwynne
Owner

@rtyler do you think you're still likely to fix this?

@trcjr trcjr referenced this issue from a commit in trcjr/cucumber
@trcjr trcjr Added a unit test for #124 4982894
@chrismdp chrismdp modified the milestone: 2.0
@mattwynne
Owner

Hands up if you still care about this issue.

@mattwynne
Owner

@brasmusson @richarda either of you fancy a crack at this one?

@richarda
@richarda richarda referenced this issue from a commit in richarda/cucumber
@richarda richarda add failing tests for issue #124 8ab10de
@richarda richarda referenced this issue from a commit in richarda/cucumber
@richarda richarda add failing tests for issue #124 0ba0a0c
@tooky
Collaborator

Closed in favour of #808

@tooky tooky closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.