rework summary formatter output

[charlierudolph]

@aslakhellesoy @jbpros @mattwynne

This is a rework of the progress formatter output that I am thinking about for cucumber-js and wanted to bring up to the whole cucumber community. Please tag anyone else you think should see this

Given a feature:

Feature: A feature

  Scenario: 1
    Given a passing step

  Scenario: 2
    Given a passing step
    And a failing step
    And a skipped step

  Scenario: 3
    Given a passing step
    And a ambiguous step
    And a skipped step

  Scenario: 4
    Given a passing step
    And a undefined step
    And a skipped step

  Scenario: 5
    Given a passing step
    And a pending step
    And a skipped step

The current progress formatter output is

..F-.A-.U-.P-

(::) failed steps (::)

Error: error
    at World.<anonymous> (/.../features/step_definitions/steps.js:5:11)

Failing scenarios:
features/a.feature:6 # Scenario: 2

5 scenarios (1 failed, 1 undefined, 1 ambiguous, 1 pending, 1 passed)
13 steps (1 failed, 1 undefined, 1 ambiguous, 1 pending, 4 skipped, 5 passed)
0m00.003s

You can implement step definitions for undefined steps with these snippets:

this.Given(/^a undefined step$/, function (callback) {
  // Write code here that turns the phrase above into concrete actions
  callback.pending();
});


The following steps have multiple matching definitions:

"a ambiguous step" matches:
/^an? ambiguous step$/ # features/step_definitions/steps.js:8
/^a ambiguous step$/   # features/step_definitions/steps.js:9

The fact that failed steps only lists the errors without the context (the scenario and step) has always bugged me and almost always made me rerun with pretty formatter the failing test in order to understand the context. I also dislike the that the different issues are shown in different places.

I propose the following new output

..F-.A-.U-.P-

Issues:

1) Scenario: 2 # features/a.feature:6
  Given a failing step # features/step_definitions/steps.js:4
    Error: error
      at World.<anonymous> (/.../features/step_definitions/steps.js:5:11)


2) Scenario: 3 # features/a.feature:11
  Given a ambiguous step
    Multiple step definitions match:
      /^an? ambiguous step$/ # features/step_definitions/steps.js:8
      /^a ambiguous step$/   # features/step_definitions/steps.js:9

3) Scenario: 4 # features/a.feature:16
  Given a undefined step
    Use the following snippet to implement this step
      this.Given(/a undefined step/, function(){
        // Write code to here to turn the step into concrete actions
        callback.pending();
      })

4) Scenario: 5 # features/a.feature:21
  Given a pending step # features/step_definitions/steps.js:11
    Pending

5 scenarios (1 failed, 1 undefined, 1 ambiguous, 1 pending, 1 passed)
13 steps (1 failed, 1 undefined, 1 ambiguous, 1 pending, 4 skipped, 5 passed)
0m00.003s

Thoughts?

[mattwynne]

I like this Charlie, it's something that irritates me too, but I've sort of grown used to it and stopped noticing!

When you propose is very similar to RSpec's output. In RSpec, they categorise the different kinds of issues, so you'd probably see something more like this:

..F-.A-.U-.P-

Failures:

  1) Scenario: 2 # features/a.feature:6
     Given a failing step # features/step_definitions/steps.js:4
       Error: error
         at World.<anonymous> (/.../features/step_definitions/steps.js:5:11)

  2) Scenario: 3 # features/a.feature:11
      Given a ambiguous step
      Multiple step definitions match:
        /^an? ambiguous step$/ # features/step_definitions/steps.js:8
        /^a ambiguous step$/   # features/step_definitions/steps.js:9

Undefined: (Failures listed here are expected and do not affect your suite's status)

  1) Scenario: 4 # features/a.feature:16
      Given a undefined step
        Use the following snippet to implement this step
          this.Given(/a undefined step/, function(){
            // Write code to here to turn the step into concrete actions
            callback.pending();
          })

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) Scenario: 5 # features/a.feature:21
      Given a pending step # features/step_definitions/steps.js:11
        Pending

5 scenarios (1 failed, 1 undefined, 1 ambiguous, 1 pending, 1 passed)
13 steps (1 failed, 1 undefined, 1 ambiguous, 1 pending, 4 skipped, 5 passed)
0m00.003s

Personally I like this sorting as it helps you to focus on the big problems first (failures) before dealing with the less urgent undefined / pending issues. The (Failures listed here are expected and do not affect your suite's status) message is ripped off frominspired by RSpec and would not be shown if you're running in --strict mode I suppose.

Just a suggestion to consider. I like trying to make these tools converge. I wonder what @everzet does in Behat? cc @tooky

[charlierudolph]

Here is what cucumber-js@1.3.0 currently has as the output for the progress formatter (which prints the status along the way and then prints the same as the summary formatter). For reference on cucumber/cucumber-ruby#1030 and if anyone else is interested.

..F-.A-.U-.P-

Failures:

1) Scenario: 2 - features/test.feature:6
   Step: And a failing step - features/test.feature:8
   Step Definition: features/step_definitions/browser_steps.js:22
   Message:
     Error: error
         at CustomWorld.<anonymous> (/Users/charlesrudolph/github/test/features/step_definitions/browser_steps.js:23:11)
         at _combinedTickCallback (internal/process/next_tick.js:67:7)
         at process._tickCallback (internal/process/next_tick.js:98:9)

2) Scenario: 3 - features/test.feature:11
   Step: And a ambiguous step - features/test.feature:13
   Message:
     Multiple step definitions match:
       /^a ambiguous step$/   - features/step_definitions/browser_steps.js:28
       /^an? ambiguous step$/ - features/step_definitions/browser_steps.js:29

Warnings:

1) Scenario: 4 - features/test.feature:16
   Step: And a undefined step - features/test.feature:18
   Message:
     Undefined. Implement with the following snippet:

       this.Given(/^a undefined step$/, function (callback) {
         // Write code here that turns the phrase above into concrete actions
         callback(null, 'pending');
       });

2) Scenario: 5 - features/test.feature:21
   Step: And a pending step - features/test.feature:23
   Step Definition: features/step_definitions/browser_steps.js:25
   Message:
     Pending

5 scenarios (1 failed, 1 ambiguous, 1 undefined, 1 pending, 1 passed)
13 steps (1 failed, 1 ambiguous, 1 undefined, 1 pending, 4 skipped, 5 passed)
0m03.254s

For colors, the scenario name (Example: "1") and step name (Example: "And a failing step") are in bold. The relative paths (Example: features/test.feature:21) are all in gray. The failure messages are in red and the warning messages are in yellow.

If you would like to reproduce, use the feature at the top, and the following step definitions

// features/step_definitions/test_steps.js
module.exports = function () {
  this.Given(/^a passing step$/, function() {});
  this.Given(/^a failing step$/, function() {
    throw new Error('error');
  });
  this.Given(/^a pending step$/, function() {
    return 'pending';
  });
  this.Given(/^a ambiguous step$/, function() {});
  this.Given(/^an? ambiguous step$/, function() {});
  this.Given(/^a skipped step$/, function() {});
};

[mattwynne]

Ref cucumber/cucumber-ruby#1030

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[charlierudolph]

For reference, cucumber-js removed the pretty formatter but now the summary output prints the failed scenarios in the previous pretty format.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.