New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Scenario Outlines #62

Closed
alisterscott opened this Issue Oct 29, 2011 · 4 comments

Comments

Projects
None yet
5 participants
@alisterscott

alisterscott commented Oct 29, 2011

Scenario Outlines are not currently supported.

For example


  Scenario Outline: Search for Watir
    Given I am on the <search engine> Home Page
    When I search for "Watir"
    Then I should see at least <expected number of> results
    Scenarios:
      | search engine | expected number of  |
      | Google        | 100,000             |
      | Bing          |  30,000             |

This gives:


Scenario: Search for Watir
    ?  Given I am on the <search engine> Home Page                  
    ~  When I search for "Watir"                                    
    ~  Then I should see at least <expected number of> results      

        Step 'I am on the <search engine> Home Page' not found

        You can define it with: 

          Given  'I am on the <search engine> Home Page' do
            raise 'step not implemented'
          end

even though I created the valid step, it doesn't do the substitution of the value.

@josepjaume

This comment has been minimized.

Show comment
Hide comment
@josepjaume

josepjaume Oct 30, 2011

Member

Hi Alister,

We haven't (still) implemented scenario outlines for a reason: They usually are a "test" smell. We've found out that if you try to be as abstract as possible in your feature definitions, there's barely no situation where scenario outlines are needed. If you're using them, you're probably testing something that should be on a unit test.

For example, I've found myself testing the same scenario against different user roles using scenario outlines and that's wrong. Unit tests should guarantee that, and features should always run on the happy path.

Also, scenario outlines trend to be less readable than standard feature definitions. In addition, there's a principle we wanted to encourage with spinach: "When you write a feature, you shouldn't be thinking of any semantics more than the language in which you're writing them". Having to think about <blah> and putting them just below your scenario feels kinda unnatural.

For example, in your case I would better write two scenarios for that instead of an outline.

  Scenario: Search for Watir on Google
    Given I search for "Watir" on google
    Then I should see at least 100,000 results

  Scenario: Search for Watir on Bing
    Given I search for "Watir" on Bing
    Then I should see at least 30,000 results

So here's why we haven't implemented scenario outlines to the moment. If you can prove us wrong, please feel free to troll us :D

What do you think?

Member

josepjaume commented Oct 30, 2011

Hi Alister,

We haven't (still) implemented scenario outlines for a reason: They usually are a "test" smell. We've found out that if you try to be as abstract as possible in your feature definitions, there's barely no situation where scenario outlines are needed. If you're using them, you're probably testing something that should be on a unit test.

For example, I've found myself testing the same scenario against different user roles using scenario outlines and that's wrong. Unit tests should guarantee that, and features should always run on the happy path.

Also, scenario outlines trend to be less readable than standard feature definitions. In addition, there's a principle we wanted to encourage with spinach: "When you write a feature, you shouldn't be thinking of any semantics more than the language in which you're writing them". Having to think about <blah> and putting them just below your scenario feels kinda unnatural.

For example, in your case I would better write two scenarios for that instead of an outline.

  Scenario: Search for Watir on Google
    Given I search for "Watir" on google
    Then I should see at least 100,000 results

  Scenario: Search for Watir on Bing
    Given I search for "Watir" on Bing
    Then I should see at least 30,000 results

So here's why we haven't implemented scenario outlines to the moment. If you can prove us wrong, please feel free to troll us :D

What do you think?

@mrcasals

This comment has been minimized.

Show comment
Hide comment
@mrcasals

mrcasals Oct 30, 2011

Member

IMHO and experience, scenario outlines are a pain in the ass to debug. Imagine you have something like:

Scenario: User adds a post
  Given I am logged in as a <user_type>
  And I am at the post creation page
  When I add a post
  Then I should see the post in my post list

Examples:
  | user_type |
  | admin |
  | editor |

So at first you have them both working, but you change something on your code and now the editor, for some reason you don't know, can't add a post. So you try to debug: add a binding.pry or a save_and_open_page in the last step... The step will be executed twice, and there's no way to help it (unless your remove the admin user_type in the examples table, but then the scenario outline is useless.

Even so, when you run cucumber features and one of the outlines fails, you can't know in which step... So a real pain in the ass :)

Member

mrcasals commented Oct 30, 2011

IMHO and experience, scenario outlines are a pain in the ass to debug. Imagine you have something like:

Scenario: User adds a post
  Given I am logged in as a <user_type>
  And I am at the post creation page
  When I add a post
  Then I should see the post in my post list

Examples:
  | user_type |
  | admin |
  | editor |

So at first you have them both working, but you change something on your code and now the editor, for some reason you don't know, can't add a post. So you try to debug: add a binding.pry or a save_and_open_page in the last step... The step will be executed twice, and there's no way to help it (unless your remove the admin user_type in the examples table, but then the scenario outline is useless.

Even so, when you run cucumber features and one of the outlines fails, you can't know in which step... So a real pain in the ass :)

@divins

This comment has been minimized.

Show comment
Hide comment
@divins
Member

divins commented Nov 13, 2011

+1 to @mrcasals

@txus

This comment has been minimized.

Show comment
Hide comment
@txus

txus Jan 18, 2012

Member

I think we won't support scenario outlines. I consider them a code smell, since they usually indicate you should be pushing some tests down to lower test layers (functional, unit). Integration tests should be mainly happy path tests.

Member

txus commented Jan 18, 2012

I think we won't support scenario outlines. I consider them a code smell, since they usually indicate you should be pushing some tests down to lower test layers (functional, unit). Integration tests should be mainly happy path tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment