Suites

[mattwynne]

See https://github.com/Behat/Behat/blob/master/features/suite.feature

Cucumber.configure do |config|
  config.suite(:domain) do |suite_config|
    suite_config.world_modules << DriveDomain
    suite_config.filters << TagFilter.new("~@web")
    suite_config.filters << LocationFilter.new("features/core")
  end

  config.suite(:web) do |suite_config|
    suite_config.world_modules << DriveUI
    suite_config.filters << TagFilter.new("@web")
  end
end

This would run the features twice, applying the specified filters and mixing in the specified world modules. The overall result would aggregate results from both suites.

[aslakhellesoy]

Want want!

[richarda]

@mattwynne I'm pretty interested in this. In an effort to minimize UI testing, I've been trying to carefully word steps so that they imply either a domain model, like a controller with no view attached or a fully connected app with a clickable view and controller, but I feel like I'm dancing around something more fundamental and leaving a confusing trail of nuanced wording.

For example, a step:
When I choose to "Cancel"
could either mean using an automation framework to click a button called Cancel or just calling cancel() on a controller directly. At this point, to distinguish, I might put something like When I select "Cancel" vs When I choose to "Cancel" so that I could resolve to different step definitions. That doesn't feel very clean though.

Is that similar to your example?

[mattwynne]

Yes similar @richarda. This is getting off-topic a bit, but what are you trying to cancel? I'd try and express the intent behind the step - the action the user is trying to perform, and try to stop thinking about the interaction. I find that helps me name the methods on my lower level APIs.

We're going to explore this some more in the last Cucumber School, so stay posted for that!

[richarda]

Thanks @mattwynne . You raise a very good point. Taking a closer look, there were 2 slightly different intents.

• One intent is to make the application go away or hide from view when you didn't want to use it any more. This scenario was implemented with UI steps to make sure that the app got hidden away.
• Another intent is to clear a list of selected items when you don't want to follow through with the series of tasks. This scenario uses steps that just create a controller (view model), fill a list then cancel, but no UI elements are needed to verify this behavior.

Maybe there is a better scenario that covers all of that together, but it seemed valuable to try and minimize the UI interactions and be a little more granular with the behaviors.

However, based on reviewing this right now, in this case I might actually just delete the non-UI scenario. I can't remember which scenario went in first, but at this point, the UI is still necessary to keep around for regressing, and the domain model version is probably better as a unit test.

BTW, I'm using a C# app with SpecFlow as the example here, but I prefer exploring Cucumber with Ruby :).

So back to suites,
Would a suite be a collection of scenarios that are most appropriate for one type of world rather than a way to run the same scenario in multiple worlds? Do suites contain mutually exclusive scenarios?

Thanks!

[mattwynne]

Hi @richarda - sorry for the lag.

Here's an example I can imagine. You have a set of scenarios that you want to run on both an iOS and an Android app. A few of the scenarios don't apply to one platform or the other and have tags to mark them as such. You have a mechanism for choosing the appropriate world to drive the the iOS or Android app.

Right now, the build runs like this:

$ WORLD=IOS cucumber --tags ~@android-only $ WORLD=ANDROID cucumber --tags ~@ios-only

The disadvantage here is that you get two sets of results. With suites, you'd be able to configure them like this:

Cucumber.configure do |config|
  config.suite(:ios) do |suite_config|
    suite_config.world_modules << IosAutomation
    suite_config.filters << TagFilter.new("~@android-only")
  end

  config.suite(:android) do |suite_config|
    suite_config.world_modules << AndroidAutomation
    suite_config.filters << TagFilter.new("~@ios-only")
  end
end

Then when you run cucumber you'd run both suites, and see the aggregate results.

Make sense?

[richarda]

@mattwynne Yes. That makes sense. I will look around and see where the World gets setup.

[richarda]

@mattwynne sorry for the delay/slow progress. I started a PR, #917, with a suites.feature based on the description above.

[jfinkhaeuser]

There's a related use case for this that I keep running into, that is to test the same functionality in multiple locales.

Here's something more complicated still: test translation software, and you have a matrix of locales and translation language to test.

There's an argument to be made that functionally speaking, one combination of locale and translation language should have enough coverage, but practically speaking, that ignores unicode weirdness.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[stale]

This issue has been automatically closed because of inactivity. You can support the Cucumber core team on opencollective.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.