HOWTO dev Cucumber_multiple_feature_runs

HOWTO: Cucumber: same feature, multiple runs

What?

You can't run multiple times a single Feature file or Scenario by adding multiple hooks to it, even if those are customized by exotic Around code.

Why would I need that?

For instance, trying to change the current_driver used by Capybara just to test the same Scenario in 2 different configurations.

Like with @javascript turned off and on. Or when testing the same scenario on different registered Capybara drivers (as in @headless_chrome_iphone5, @headless_chrome_iphonex, @headless_chrome_ipad, ...)

Why it doesn't work?

Although it's seems somehow possible to invoke multiple times the block yield from a Scenario or a whole Feature by a customized Around hook in Cucumber, the problem lies in how Scenarios are created, configured and handled by the current version of Cucumber.

You cannot easily reset the resulting state of a Scenario block.

For example, the following won't work as expected...

Around('@run_with_and_without_javascript') do |_scenario, block|
  Capybara.current_driver = Capybara.javascript_driver
  block.call
  Capybara.use_default_driver
  block.call
end

...Because if the first block.call succeeds, it will also the second one.

You can call multiple times the block being executed: the output formatter gets the whole list of repeated steps but, as soon as the first called block passes, the whole list of blocks is passing.

You will get a (very) long list of green steps without actually having tested them all.

Config. consideration and nice takeaway #1

When registering a Capybara driver, the symbol used for registering it can (and should) be used as a tag to auto-set the current_driver to it. (Capybara will generate a tag for that registered driver.)

For example, this...

Capybara.register_driver(:headless_chrome_ipad) do |app|
  capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
    'goog:chromeOptions': {
      'args': %w[headless disable-gpu disable-extensions],
      'mobileEmulation': { 'deviceName': 'iPad' }
    }
  )
  Capybara::Selenium::Driver.new(app, browser: :chrome, desired_capabilities: capabilities)
end

...Will make a shiny new @headless_chrome_ipad tag available at your disposal. (Which could be used to test a Scenario on that configuration/driver/browser combination)

So how can I test multiple times the same Scenario?

Due to the fact that a Feature/Scenario is executed only once and all tags are applied to it in sequence as configuration before running it - and for what we've said above -, you can't test the same Scenario multiple times by enlisting all the needed driver tags on its first line.

That, again, is because only the last tag will prevail, overriding all the previous changes to Capybara.current_driver.

The only-known, current, best-practice solution is to use multiple copies of the same file, renaming it with a different prefix for each driver used.

As in: my_super_feature.iphone5.feature, my_super_feature.iphone8.feature, my_super_feature.ipad.feature and so on.

It sure is a lot of duplicated feature files (differing for a tag or a few steps, at most), but it's the only currently verified, fast and fool-proof way to do it. (If you think you've found yourself a better or easier way to do it, please do edit this page.)

The <DEVICE_NAME>.feature suffix can be anything, it just serves the purpose to distinguish between different version of the same file, in which, allegedly, only the header tag should change.

Having accepted the "duplication" of test code under different filenames & tags, should you run into a possible flaky test and found yourself in need of ruling out any random failure on it, use a simple for loop on a Bash console, like this one:

$> for i in {1..10}; do \
     cucumber --format pretty \
              features/my_feature/my_test.galaxys5.feature:9 ; \
   done