New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rules + Examples #250

Closed
ryanwmarsh opened this Issue Aug 21, 2017 · 31 comments

Comments

Projects
None yet
@ryanwmarsh

ryanwmarsh commented Aug 21, 2017

Summary

There is no useful mapping between Gherkin and "rules and examples".

Expected Behavior

I can express Rules and Examples naturally (aside from the fact that Gherkin is context free grammar and will never be as fully expressive as English).

Current Behavior

Features, Scenarios and Steps are the high level concepts. Rules and Examples must be shoe-horned in by the user.

Possible Solution

Feature: is replaced with Spec:

Rule: is a new section in-between what is currently Scenario: and Feature:

Scenario: is replaced with Example:

Scenario Outline: is replaced with Example Outline:

Spec: User sees the progress of processing for a new file

  Rule: Uploading accounts for 50% of processing 

    Example Outline: user sees upload at various states
       Given the user is uploading a new file
       When the upload is at <a_percent_uploaded>
       Then the progress bar should show <a_percent_complete>

       Examples:
       | a_percent_uploaded | a_percent_complete |
       | 10%                | 5%                 |
       | 25%                | 12.5%              |
       | 51%                | 25.5%              |
       | 75%                | 37.5%              |
       | 90%                | 45%                |
       | 99%                | 49.5%              |

  Rule: Preparing transcode job waits at 55%

    Example: A file is waiting on transcode job creation
      Given a file is waiting on transcode job creation
      When the progress bar updates
      Then we should see it is at 55%

  Rule: Transcoding happens from 60 - 75% and waits at 75%

    Example: File is transcoding
      Given a file is transcoding
      And it has been transcoding for <time> seconds
      When the progress bar updates
      Then it should show <percent> progress
      | time | percent |
      | 1    | 60      |
      | 2    | 65      |
      | 3    | 70      |
      | 4    | 75      |
      | 8    | 75      |

Context & Motivation

We teach that:

  • The outcome of deliberate discovery should be rules and examples
  • These rules and examples should become part of the team's living documentation
  • We get living documentation by connecting examples to tests

The three most important concepts in Gherkin + Cucumber are features, scenarios, and steps.

There is no useful mapping between Gherkin and "rules and examples".

@aslakhellesoy

This comment has been minimized.

Show comment
Hide comment
@aslakhellesoy

aslakhellesoy Aug 27, 2017

Contributor

I really like this. I'd love to be able to:

a) Generate and display an example map (in an example mapping tool) from existing Gherkin documents (that follow this new structure).
b) Generate skeleton Gherkin documents from an example mapping tool.

If we change the Gherkin grammar, it's essential that old documents still work:

  • Example just becomes a new synonym for Scenario, which will still work
  • Rule becomes a new keyword, which is optional

The AST would have to change so that every Gherkin Document has a list of Rule nodes, each of which has a list of Background/Example/Example Outline nodes. This would allow a Background per rule.

For legacy documents (no rules), we'd have to build the AST with a "dummy" rule.

Automatic rule validation

We've been wanting to add property based testing support to Gherkin and Cucumber for a while. Recently @ELetier introduced to a tool called Alloy, which can be used to validate rules using a different approach.

The overall idea is that Cucumber users would be able write Gherkin documents with rules without examples. A property based testing tool or a tool like Alloy (which would have to be integrated with Cucumber somehow) would generate examples at runtime, based on the properties associated with the rules.

I'm bringing this up because I want to make sure that a change to the Gherkin grammar to allow for rules also makes it possible to hook in property based testing tools and/or tools like Alloy.

For example:

Rule: An infant must sit in an adult's lap
Rule: An adult is 18 or older
Rule: An infant is 2 or younger

The text of the rules above would actually be interpreted (with a rule definition?), and there wouldn't (necessarily) be any explicit scenarios below.

Let's discuss!

Contributor

aslakhellesoy commented Aug 27, 2017

I really like this. I'd love to be able to:

a) Generate and display an example map (in an example mapping tool) from existing Gherkin documents (that follow this new structure).
b) Generate skeleton Gherkin documents from an example mapping tool.

If we change the Gherkin grammar, it's essential that old documents still work:

  • Example just becomes a new synonym for Scenario, which will still work
  • Rule becomes a new keyword, which is optional

The AST would have to change so that every Gherkin Document has a list of Rule nodes, each of which has a list of Background/Example/Example Outline nodes. This would allow a Background per rule.

For legacy documents (no rules), we'd have to build the AST with a "dummy" rule.

Automatic rule validation

We've been wanting to add property based testing support to Gherkin and Cucumber for a while. Recently @ELetier introduced to a tool called Alloy, which can be used to validate rules using a different approach.

The overall idea is that Cucumber users would be able write Gherkin documents with rules without examples. A property based testing tool or a tool like Alloy (which would have to be integrated with Cucumber somehow) would generate examples at runtime, based on the properties associated with the rules.

I'm bringing this up because I want to make sure that a change to the Gherkin grammar to allow for rules also makes it possible to hook in property based testing tools and/or tools like Alloy.

For example:

Rule: An infant must sit in an adult's lap
Rule: An adult is 18 or older
Rule: An infant is 2 or younger

The text of the rules above would actually be interpreted (with a rule definition?), and there wouldn't (necessarily) be any explicit scenarios below.

Let's discuss!

@enkessler

This comment has been minimized.

Show comment
Hide comment
@enkessler

enkessler Aug 30, 2017

Those Rules don't look like they have a mechanical effect. Are they just a descriptive/organizational thing? Similar to RSpec's describe blocks? Could Rules have a mechanical effect such as being able to 'run' a rule (which would run all examples within the rule)?

enkessler commented Aug 30, 2017

Those Rules don't look like they have a mechanical effect. Are they just a descriptive/organizational thing? Similar to RSpec's describe blocks? Could Rules have a mechanical effect such as being able to 'run' a rule (which would run all examples within the rule)?

@aslakhellesoy

This comment has been minimized.

Show comment
Hide comment
@aslakhellesoy

aslakhellesoy Aug 30, 2017

Contributor

The way @ryanwmarsh suggested it initially, the only effect of Rule would be its presence in the AST, which other tools could use go generate an example map.

However, Cucumber (or a Cucumber plugin) could use the Rule to run a "rule definition", which could intergate with tools like property based testing tools or Alloy, to do validation of the rule itself.

Contributor

aslakhellesoy commented Aug 30, 2017

The way @ryanwmarsh suggested it initially, the only effect of Rule would be its presence in the AST, which other tools could use go generate an example map.

However, Cucumber (or a Cucumber plugin) could use the Rule to run a "rule definition", which could intergate with tools like property based testing tools or Alloy, to do validation of the rule itself.

@ryanwmarsh

This comment has been minimized.

Show comment
Hide comment
@ryanwmarsh

ryanwmarsh Aug 30, 2017

@enkessler I intended it to simply be present in the AST as @aslakhellesoy stated however, I recognize that some folks would like this to actually be a "Rule definition" as @aslakhellesoy also mentioned

ryanwmarsh commented Aug 30, 2017

@enkessler I intended it to simply be present in the AST as @aslakhellesoy stated however, I recognize that some folks would like this to actually be a "Rule definition" as @aslakhellesoy also mentioned

@ryanwmarsh

This comment has been minimized.

Show comment
Hide comment
@ryanwmarsh

ryanwmarsh Aug 30, 2017

text of the rules above would actually be interpreted (with a rule definition?), and there wouldn't (necessarily) be any explicit scenarios below.
@aslakhellesoy

So are we thinking the step def side would look something like:

// Given/When/Then
StepDef("An adult is traveling with an infant", function() {
    // implement this step
}); 

Rule("An adult is 18 or older", function() {
    // implement this rule
});

Or does the rule need some added context?

ryanwmarsh commented Aug 30, 2017

text of the rules above would actually be interpreted (with a rule definition?), and there wouldn't (necessarily) be any explicit scenarios below.
@aslakhellesoy

So are we thinking the step def side would look something like:

// Given/When/Then
StepDef("An adult is traveling with an infant", function() {
    // implement this step
}); 

Rule("An adult is 18 or older", function() {
    // implement this rule
});

Or does the rule need some added context?

@aslakhellesoy

This comment has been minimized.

Show comment
Hide comment
@aslakhellesoy

aslakhellesoy Aug 31, 2017

Contributor

Something like that. Not sure if the ruledef needs additional context, I'm not yet familiar enough with the other tools I mentioned.

Contributor

aslakhellesoy commented Aug 31, 2017

Something like that. Not sure if the ruledef needs additional context, I'm not yet familiar enough with the other tools I mentioned.

@mattwynne

This comment has been minimized.

Show comment
Hide comment
@mattwynne

mattwynne Aug 31, 2017

Member

Didn't @sebrose suggest something similar, maybe in the Gherkin repo? On phone now so can't easily search.

Member

mattwynne commented Aug 31, 2017

Didn't @sebrose suggest something similar, maybe in the Gherkin repo? On phone now so can't easily search.

@aslakhellesoy

This comment has been minimized.

Show comment
Hide comment
@aslakhellesoy

aslakhellesoy Oct 4, 2017

Contributor

Hello @cucumber/committers - I've written up some thoughts about how we could add property based testing to Cucumber over at cucumber-ltd/shouty.rb#2.

Do you see any problems with this?

Contributor

aslakhellesoy commented Oct 4, 2017

Hello @cucumber/committers - I've written up some thoughts about how we could add property based testing to Cucumber over at cucumber-ltd/shouty.rb#2.

Do you see any problems with this?

@gasparnagy

This comment has been minimized.

Show comment
Hide comment
@gasparnagy

gasparnagy Oct 4, 2017

Member

I have no problem with the structure of the feature file, although I would like to see also what properties do you check of the rule in the Gherkin file itself.

In Cukenhagen we discussed about this and based on that discussion I made this example:

https://github.com/gasparnagy/SpecFlow.FsCheck/blob/master/sample/AdditionSample/MyCalculator.Tests/Addition.feature

But of couse this can be an optional part. If there would be a "Rule" level in the feature file and some way to hook any kind of automation for it that's a good start...

Member

gasparnagy commented Oct 4, 2017

I have no problem with the structure of the feature file, although I would like to see also what properties do you check of the rule in the Gherkin file itself.

In Cukenhagen we discussed about this and based on that discussion I made this example:

https://github.com/gasparnagy/SpecFlow.FsCheck/blob/master/sample/AdditionSample/MyCalculator.Tests/Addition.feature

But of couse this can be an optional part. If there would be a "Rule" level in the feature file and some way to hook any kind of automation for it that's a good start...

@mgundala

This comment has been minimized.

Show comment
Hide comment
@mgundala

mgundala Oct 4, 2017

Contributor

I would like to know the difference between "Rule" and "Scenario", aren't we conveying the same in both the keywords?

According to my knowledge "Rule" is a condition that needs to verify through Gherkin Steps.
The scenario is a test Case that also needs to verify through Gherkin Steps.

Is Rule is a kind of Condition check for Scenario?

Contributor

mgundala commented Oct 4, 2017

I would like to know the difference between "Rule" and "Scenario", aren't we conveying the same in both the keywords?

According to my knowledge "Rule" is a condition that needs to verify through Gherkin Steps.
The scenario is a test Case that also needs to verify through Gherkin Steps.

Is Rule is a kind of Condition check for Scenario?

@aslakhellesoy

This comment has been minimized.

Show comment
Hide comment
@aslakhellesoy

aslakhellesoy Oct 4, 2017

Contributor

@mgundala see @mattwynne's article about Example Mapping for an explanation of rules vs scenarios (aka examples). This is the technique we recommend for designing scenarios in three amigos sessions.

Contributor

aslakhellesoy commented Oct 4, 2017

@mgundala see @mattwynne's article about Example Mapping for an explanation of rules vs scenarios (aka examples). This is the technique we recommend for designing scenarios in three amigos sessions.

@aslakhellesoy

This comment has been minimized.

Show comment
Hide comment
@aslakhellesoy

aslakhellesoy Oct 5, 2017

Contributor

@ryanwmarsh what's your motivation for suggesting to rename Feature to Spec?

Contributor

aslakhellesoy commented Oct 5, 2017

@ryanwmarsh what's your motivation for suggesting to rename Feature to Spec?

@mattwynne

This comment has been minimized.

Show comment
Hide comment
@mattwynne

mattwynne Oct 5, 2017

Member

IMO we don't need Example Outline and I don't like how it reads. I've always thought of a scenario outline as a rule, with the examples in the table below being illustrations of that rule.

So ideally for me we'd have:

Spec: User sees the progress of processing for a new file

  Rule: Uploading accounts for 50% of processing 
    Given the user is uploading a new file
    When the upload is at <a_percent_uploaded>
    Then the progress bar should show <a_percent_complete>

    Examples:
    | a_percent_uploaded | a_percent_complete |
    | 10%                | 5%                 |
    | 25%                | 12.5%              |
    | 51%                | 25.5%              |
    | 75%                | 37.5%              |
    | 90%                | 45%                |
    | 99%                | 49.5%              |

  Rule: Preparing transcode job waits at 55%

    Example: A file is waiting on transcode job creation
      Given a file is waiting on transcode job creation
      When the progress bar updates
      Then we should see it is at 55%

i.e. you can use Rule either as just a container of regular Scenarios, or as a scenario outline with table-based examples.

Member

mattwynne commented Oct 5, 2017

IMO we don't need Example Outline and I don't like how it reads. I've always thought of a scenario outline as a rule, with the examples in the table below being illustrations of that rule.

So ideally for me we'd have:

Spec: User sees the progress of processing for a new file

  Rule: Uploading accounts for 50% of processing 
    Given the user is uploading a new file
    When the upload is at <a_percent_uploaded>
    Then the progress bar should show <a_percent_complete>

    Examples:
    | a_percent_uploaded | a_percent_complete |
    | 10%                | 5%                 |
    | 25%                | 12.5%              |
    | 51%                | 25.5%              |
    | 75%                | 37.5%              |
    | 90%                | 45%                |
    | 99%                | 49.5%              |

  Rule: Preparing transcode job waits at 55%

    Example: A file is waiting on transcode job creation
      Given a file is waiting on transcode job creation
      When the progress bar updates
      Then we should see it is at 55%

i.e. you can use Rule either as just a container of regular Scenarios, or as a scenario outline with table-based examples.

@mattwynne

This comment has been minimized.

Show comment
Hide comment
@mattwynne

mattwynne Oct 5, 2017

Member

@lunivore I'd love to know what you think about this

Member

mattwynne commented Oct 5, 2017

@lunivore I'd love to know what you think about this

@SabotageAndi

This comment has been minimized.

Show comment
Hide comment
@SabotageAndi

SabotageAndi Oct 6, 2017

Contributor

I was thinking the last days about this, but I am still not sure about this suggestion.
How fix are the new keywords?
I have the fear, that the reuse and combination of some keywords (example, outline) will irritate people, especially people that start using Gherkin.
Also I am currently not a fan of the shortening of Specification to Spec. In English this works to a degree, but the short form of "Spezifikation" in German would be Spez or Spek. This doesn't look good in a professional environment.

Contributor

SabotageAndi commented Oct 6, 2017

I was thinking the last days about this, but I am still not sure about this suggestion.
How fix are the new keywords?
I have the fear, that the reuse and combination of some keywords (example, outline) will irritate people, especially people that start using Gherkin.
Also I am currently not a fan of the shortening of Specification to Spec. In English this works to a degree, but the short form of "Spezifikation" in German would be Spez or Spek. This doesn't look good in a professional environment.

@aslakhellesoy

This comment has been minimized.

Show comment
Hide comment
@aslakhellesoy

aslakhellesoy Oct 6, 2017

Contributor

As there are several suggestions in this issue I suggest we focus on adding Rule and ignore the following suggestions:

  • Rename Feature to Spec
  • Rename Scenario to Example
  • Rename Scenario Outline to Example Outline

Feel free to create separate issues for those. This issue is exclusively about adding a Rule keyword from here on.

Contributor

aslakhellesoy commented Oct 6, 2017

As there are several suggestions in this issue I suggest we focus on adding Rule and ignore the following suggestions:

  • Rename Feature to Spec
  • Rename Scenario to Example
  • Rename Scenario Outline to Example Outline

Feel free to create separate issues for those. This issue is exclusively about adding a Rule keyword from here on.

@enkessler

This comment has been minimized.

Show comment
Hide comment
@enkessler

enkessler Oct 7, 2017

Maybe it's just because I'm not familiar either rules and property based testing, but every time I see Rule in a feature file, my mind tries to interpret it as a setup step that only applies to scenarios within its scope.

Rule('Shouts have a range of {int}m') do |range|
  Shouty.set_range(range)
end

But it looks like a rule definition is no different from a step definition so I suppose that different people could use it in different ways.

enkessler commented Oct 7, 2017

Maybe it's just because I'm not familiar either rules and property based testing, but every time I see Rule in a feature file, my mind tries to interpret it as a setup step that only applies to scenarios within its scope.

Rule('Shouts have a range of {int}m') do |range|
  Shouty.set_range(range)
end

But it looks like a rule definition is no different from a step definition so I suppose that different people could use it in different ways.

@brasmusson

This comment has been minimized.

Show comment
Hide comment
@brasmusson

brasmusson Oct 8, 2017

Contributor

@enkessler Apart from the Example Mapping article, I think the @lunivore blog post Acceptance Criteria vs. Scenarios is a good description of the meaning of "Rule" (defined by the Acceptance Criteria) and "Example" (defined by the Scenarios) as used in the discussion.

Contributor

brasmusson commented Oct 8, 2017

@enkessler Apart from the Example Mapping article, I think the @lunivore blog post Acceptance Criteria vs. Scenarios is a good description of the meaning of "Rule" (defined by the Acceptance Criteria) and "Example" (defined by the Scenarios) as used in the discussion.

@lunivore

This comment has been minimized.

Show comment
Hide comment
@lunivore

lunivore Oct 8, 2017

lunivore commented Oct 8, 2017

@brasmusson

This comment has been minimized.

Show comment
Hide comment
@brasmusson

brasmusson Oct 9, 2017

Contributor

@mattwynne I do not think a Scenario Outline is a Rule, but a Scenario Outline can be used to define the set of Examples illustrating a Rule. I think a statement like "the Steps of a Rule" is nonsense, therefore I think it would be a mistake to place the Outline Steps directly below the Rule line, there need to be something in between to mark the line between the Rule itself and the Examples illustrating the Rule.

Contributor

brasmusson commented Oct 9, 2017

@mattwynne I do not think a Scenario Outline is a Rule, but a Scenario Outline can be used to define the set of Examples illustrating a Rule. I think a statement like "the Steps of a Rule" is nonsense, therefore I think it would be a mistake to place the Outline Steps directly below the Rule line, there need to be something in between to mark the line between the Rule itself and the Examples illustrating the Rule.

@dotnetprofessional

This comment has been minimized.

Show comment
Hide comment
@dotnetprofessional

dotnetprofessional Oct 10, 2017

I'm still a little confused over the addition. Is the idea to provide an intermediate level of description for the feature? That is I find feature is at a level down from what I might want, in so far as an Epic would describe a larger piece of functionality and the features would break that down, with the scenarios being more concrete user interactions. This 'seems' to be similar in that a rule is a level down or a way to group a set of scenarios. In our Features we often put as part of the description a bullet point list of Rules which are trying to give a high level description of the scenarios below. So if we have the additional Rule keyword it would wrap the scenarios/examples for that rule. I can see value in that.

I'm still not clear why you'd add validation to the rule however, if the scenarios/examples are filling that role. If you want to make it more terse then having separate keywords for Specification that then have multiple When, Then, When thens etc. would seem clearer.

dotnetprofessional commented Oct 10, 2017

I'm still a little confused over the addition. Is the idea to provide an intermediate level of description for the feature? That is I find feature is at a level down from what I might want, in so far as an Epic would describe a larger piece of functionality and the features would break that down, with the scenarios being more concrete user interactions. This 'seems' to be similar in that a rule is a level down or a way to group a set of scenarios. In our Features we often put as part of the description a bullet point list of Rules which are trying to give a high level description of the scenarios below. So if we have the additional Rule keyword it would wrap the scenarios/examples for that rule. I can see value in that.

I'm still not clear why you'd add validation to the rule however, if the scenarios/examples are filling that role. If you want to make it more terse then having separate keywords for Specification that then have multiple When, Then, When thens etc. would seem clearer.

@mlvandijk

This comment has been minimized.

Show comment
Hide comment
@mlvandijk

mlvandijk Oct 10, 2017

Member

A 'Rule' would specify a particular business rule (i.e. the range in Shouty, or the age at which Fluffy bunny can be sold); traditionally called 'acceptance criterion' in software development. Imho it would be helpful to explicitly specify these rules, as they would be tested by the scenarios/examples but might not be explicitly mentioned here.
To me a Scenario or Scenario Outline specifies a collection of steps to be taken (Given/When/Then); the examples that go with a scenario outline are different sets of input -> expected output.

I do not understand if/how a Rule would then also be implemented to actually 'do' something (as in the example PR by @aslakhellesoy ); to me it would be additional background info to state which Rule (or Rules?) are being tested in certain Scenarios/Examples. The Scenarios/Examples would specify how the 'Rule" is applied in different situations and test that the 'Rule' is correctly implemented in the system.

However, it could also be used as a 'setting' (as the range in the Shouty example), so you could test for different versions of a Rule? (i.e. different ranges). If that is the case, then I would wonder how this is different from a Given or a Background?

Member

mlvandijk commented Oct 10, 2017

A 'Rule' would specify a particular business rule (i.e. the range in Shouty, or the age at which Fluffy bunny can be sold); traditionally called 'acceptance criterion' in software development. Imho it would be helpful to explicitly specify these rules, as they would be tested by the scenarios/examples but might not be explicitly mentioned here.
To me a Scenario or Scenario Outline specifies a collection of steps to be taken (Given/When/Then); the examples that go with a scenario outline are different sets of input -> expected output.

I do not understand if/how a Rule would then also be implemented to actually 'do' something (as in the example PR by @aslakhellesoy ); to me it would be additional background info to state which Rule (or Rules?) are being tested in certain Scenarios/Examples. The Scenarios/Examples would specify how the 'Rule" is applied in different situations and test that the 'Rule' is correctly implemented in the system.

However, it could also be used as a 'setting' (as the range in the Shouty example), so you could test for different versions of a Rule? (i.e. different ranges). If that is the case, then I would wonder how this is different from a Given or a Background?

@lunivore

This comment has been minimized.

Show comment
Hide comment
@lunivore

lunivore Oct 11, 2017

Criteri on, not criteri um (it's Latinized Greek -> κριτήριον ), plural criteria.

Rules might be about context (Given / Background), but they might also be triggered by actions or only affect outcome. So for instance:

  • Given a trade contract has been generated, when it takes more than 2 days to audit, then the system should send a reminder <- triggered by action of time passing.

  • Given a trade for chocolate has been scheduled over Christmas, when we request the contract for the trade, then a note that it's a Christmas trade should appear at the top of the contract <- we already generate contracts; this is just another rule that only affects the outcome.

lunivore commented Oct 11, 2017

Criteri on, not criteri um (it's Latinized Greek -> κριτήριον ), plural criteria.

Rules might be about context (Given / Background), but they might also be triggered by actions or only affect outcome. So for instance:

  • Given a trade contract has been generated, when it takes more than 2 days to audit, then the system should send a reminder <- triggered by action of time passing.

  • Given a trade for chocolate has been scheduled over Christmas, when we request the contract for the trade, then a note that it's a Christmas trade should appear at the top of the contract <- we already generate contracts; this is just another rule that only affects the outcome.

@mlvandijk

This comment has been minimized.

Show comment
Hide comment
@mlvandijk

mlvandijk Oct 12, 2017

Member

Now I'm possibly more confused about the use of the 'Rule' keyword.
To try and clarify, I'd rewrite:

Rule: Trade contract should be audited within 2 days.
Scenario:
Given a trade contract has been generated
When it takes more than 2 days to audit
Then the system should send a reminder

So, there's a 'Rule'; it is exemplified(?; I mean that an example is provided) / tested by the Scenario, the specific rule may come into play at any step in the scenario though?

Second example: what does 'over Christmas' mean?

Member

mlvandijk commented Oct 12, 2017

Now I'm possibly more confused about the use of the 'Rule' keyword.
To try and clarify, I'd rewrite:

Rule: Trade contract should be audited within 2 days.
Scenario:
Given a trade contract has been generated
When it takes more than 2 days to audit
Then the system should send a reminder

So, there's a 'Rule'; it is exemplified(?; I mean that an example is provided) / tested by the Scenario, the specific rule may come into play at any step in the scenario though?

Second example: what does 'over Christmas' mean?

@lunivore

This comment has been minimized.

Show comment
Hide comment
@lunivore

lunivore Oct 12, 2017

Some rules require n+1 examples to illustrate. So in this case, you'd have two: the one where it took more than one day to audit, and the one where it didn't.

I only illustrated one because that wasn't a facet I was focused on, but there it is.

If you have a lot of different rules, you often have the "happy path" (your +1) and then all the edge cases ("n"). If you're just adding a new outcome, you're introducing a new rule without introducing any new examples, but you'll be extending an existing one. I'm sure there's some kind of mathematical construct, but my main point is that rules can have multiple examples.

Remember though that the main purpose of BDD is to capture conversations, which explore the space and only produce specifications and tests as a nice by-product.

We just had an exercise today for which we had a post-it on our forehead and could only ask yes/no questions in the room, then had to find our matching post-it pair. My question was, "Can you see my pair from here? Is my pair in that direction?" Perfectly within the rules, but I'm pretty sure it wasn't what they intended. That's the kind of thing that examples help to clarify.

lunivore commented Oct 12, 2017

Some rules require n+1 examples to illustrate. So in this case, you'd have two: the one where it took more than one day to audit, and the one where it didn't.

I only illustrated one because that wasn't a facet I was focused on, but there it is.

If you have a lot of different rules, you often have the "happy path" (your +1) and then all the edge cases ("n"). If you're just adding a new outcome, you're introducing a new rule without introducing any new examples, but you'll be extending an existing one. I'm sure there's some kind of mathematical construct, but my main point is that rules can have multiple examples.

Remember though that the main purpose of BDD is to capture conversations, which explore the space and only produce specifications and tests as a nice by-product.

We just had an exercise today for which we had a post-it on our forehead and could only ask yes/no questions in the room, then had to find our matching post-it pair. My question was, "Can you see my pair from here? Is my pair in that direction?" Perfectly within the rules, but I'm pretty sure it wasn't what they intended. That's the kind of thing that examples help to clarify.

@gasparnagy

This comment has been minimized.

Show comment
Hide comment
@gasparnagy

gasparnagy Oct 24, 2017

Member

Do we have a conclusion here?

Considering the idea of adding an additional, but optional "Rule" grouping around scenarios and scenario outlines...

  • As it seems we can all be fine with this (even though some of us are not fully sure about the benefits).
  • Since this would be an optional grouping, this would still allow situations where this concept does not fit well or when the team needs more time to adopt.
  • I think we all came to the conclusion that neither Scenario Outlines nor the Feature file itself is a good replacement for rules (even with a different aliases) because we 1) sometimes want to collect examples of different text structure under a rule that would not be possible with one SO 2) we want to support simple rules with 1-2 scenarios, where a new file for each rule would be inconvenient.
  • It is a little bit unclear how we can attach direct test logic to rules (e.g. with property-based testing), but this is anyway more related to the actual tool/platform and not so much on Gherkin anyway, so maybe we could still keep this part experimental.
  • Renaming the other artifacts (Scenario->Example, Scenario Outline->Example Outline, etc. can be discussed separately, as @aslakhellesoy suggested)

I think the introduction of a Rule level grouping is quite an safe change. Even if the change will not be backward compatible it is easy to make an extension that provides the "flat" scenario structure for a feature file by visiting the rule/scenario tree.

I run into this problem (lack of structure to express rules) more and more often and I would really like to have better support for it.

Member

gasparnagy commented Oct 24, 2017

Do we have a conclusion here?

Considering the idea of adding an additional, but optional "Rule" grouping around scenarios and scenario outlines...

  • As it seems we can all be fine with this (even though some of us are not fully sure about the benefits).
  • Since this would be an optional grouping, this would still allow situations where this concept does not fit well or when the team needs more time to adopt.
  • I think we all came to the conclusion that neither Scenario Outlines nor the Feature file itself is a good replacement for rules (even with a different aliases) because we 1) sometimes want to collect examples of different text structure under a rule that would not be possible with one SO 2) we want to support simple rules with 1-2 scenarios, where a new file for each rule would be inconvenient.
  • It is a little bit unclear how we can attach direct test logic to rules (e.g. with property-based testing), but this is anyway more related to the actual tool/platform and not so much on Gherkin anyway, so maybe we could still keep this part experimental.
  • Renaming the other artifacts (Scenario->Example, Scenario Outline->Example Outline, etc. can be discussed separately, as @aslakhellesoy suggested)

I think the introduction of a Rule level grouping is quite an safe change. Even if the change will not be backward compatible it is easy to make an extension that provides the "flat" scenario structure for a feature file by visiting the rule/scenario tree.

I run into this problem (lack of structure to express rules) more and more often and I would really like to have better support for it.

@stale

This comment has been minimized.

Show comment
Hide comment
@stale

stale bot Feb 17, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

stale bot commented Feb 17, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

@stale stale bot added the Stale label Feb 17, 2018

@stale stale bot removed the Stale label Feb 17, 2018

@ryanwmarsh

This comment has been minimized.

Show comment
Hide comment
@ryanwmarsh

ryanwmarsh Feb 17, 2018

Wow great discussion here. Somehow I didn't get the notifications. So many great points. I apologize for letting this go stale. 🙏

  • I think @mattwynne's example is spot on. 👍

  • I acknowledge @lunivore's concern about people avoiding exploration by example because specification is a strong word. The problem there would be culture. A given team can collaborate on a specification or it can follow anti-patterns. Spec is a well known term (RSpec, SpecFlow, etc...). IMHO if we have a word that accurately describes a thing then we should call it by that word. Fixing culture on a team is a separate problem. However, I am always open to a better term. 🤔

Now allow me to get crazy for a moment. I'm boiling the 🐸 slowly here.

What if rules and examples didn't have to be in the same file?

Spec: User sees the progress of processing for a new file

  Rule: Uploading accounts for 50% of processing 

  Rule: Preparing transcode job waits at 55%

  Rule: ... 
     ... free text description 

  Rule: ... 
     ... 

  Rule: ... 
     ... 

Spec: Examples for uploading

  @rule:uploading_accounts_for_50_of_processing
  Example: 
    Given the user is uploading a new file
    When the upload is at <a_percent_uploaded>
    Then the progress bar should show <a_percent_complete>

  Examples:
    | a_percent_uploaded | a_percent_complete |
    | 10%                | 5%                 |
    | 25%                | 12.5%              |
    | 51%                | 25.5%              |
    | 75%                | 37.5%              |
    | 90%                | 45%                |
    | 99%                | 49.5%              |

  @rule:preparing_transcode job_waits_at_55
  Example: A file is waiting on transcode job creation
    Given a file is waiting on transcode job creation
    When the progress bar updates
    Then we should see it is at 55%

  @rule1 @rule2 
  Example: 
    Given ..
    When ..
    Then .. 

Here's why. In some organizations rules are a meaningful artifact. In some organizations rules must be given an identifier so they may be referenced in various forms of documentation and discussion. Alternatively rules must correlate with an industry regulation. What I have proposed is an incomplete solution for organizing and referring to rules. How do we reason about the rules in our system when we have hundreds?

My suggestion:

  • Make rules and examples first class citizens.
  • Allow examples to refer back to rules, lets call this a rule-ref
  • rule-refs should be human readable

Boiling the 🐸 further.

We still have the problem of terms. In many regulated businesses the terms used in rules matter greatly. Where are these terms to be defined? My gut says terms need to be first class citizens as well. I feel I might be considered disruptive with this but that is not my intention. I care that we say Cucumber is how we connect tests to rules and examples to produce "living documentation".

¯\_(ツ)_/¯

ryanwmarsh commented Feb 17, 2018

Wow great discussion here. Somehow I didn't get the notifications. So many great points. I apologize for letting this go stale. 🙏

  • I think @mattwynne's example is spot on. 👍

  • I acknowledge @lunivore's concern about people avoiding exploration by example because specification is a strong word. The problem there would be culture. A given team can collaborate on a specification or it can follow anti-patterns. Spec is a well known term (RSpec, SpecFlow, etc...). IMHO if we have a word that accurately describes a thing then we should call it by that word. Fixing culture on a team is a separate problem. However, I am always open to a better term. 🤔

Now allow me to get crazy for a moment. I'm boiling the 🐸 slowly here.

What if rules and examples didn't have to be in the same file?

Spec: User sees the progress of processing for a new file

  Rule: Uploading accounts for 50% of processing 

  Rule: Preparing transcode job waits at 55%

  Rule: ... 
     ... free text description 

  Rule: ... 
     ... 

  Rule: ... 
     ... 

Spec: Examples for uploading

  @rule:uploading_accounts_for_50_of_processing
  Example: 
    Given the user is uploading a new file
    When the upload is at <a_percent_uploaded>
    Then the progress bar should show <a_percent_complete>

  Examples:
    | a_percent_uploaded | a_percent_complete |
    | 10%                | 5%                 |
    | 25%                | 12.5%              |
    | 51%                | 25.5%              |
    | 75%                | 37.5%              |
    | 90%                | 45%                |
    | 99%                | 49.5%              |

  @rule:preparing_transcode job_waits_at_55
  Example: A file is waiting on transcode job creation
    Given a file is waiting on transcode job creation
    When the progress bar updates
    Then we should see it is at 55%

  @rule1 @rule2 
  Example: 
    Given ..
    When ..
    Then .. 

Here's why. In some organizations rules are a meaningful artifact. In some organizations rules must be given an identifier so they may be referenced in various forms of documentation and discussion. Alternatively rules must correlate with an industry regulation. What I have proposed is an incomplete solution for organizing and referring to rules. How do we reason about the rules in our system when we have hundreds?

My suggestion:

  • Make rules and examples first class citizens.
  • Allow examples to refer back to rules, lets call this a rule-ref
  • rule-refs should be human readable

Boiling the 🐸 further.

We still have the problem of terms. In many regulated businesses the terms used in rules matter greatly. Where are these terms to be defined? My gut says terms need to be first class citizens as well. I feel I might be considered disruptive with this but that is not my intention. I care that we say Cucumber is how we connect tests to rules and examples to produce "living documentation".

¯\_(ツ)_/¯

@mattwynne

This comment has been minimized.

Show comment
Hide comment
@mattwynne

mattwynne Feb 18, 2018

Member

I agree about being able to relate Rules and Examples across Gherkin docs. I don’t see the need for a rule-ref, or the additional syntax to go with it. I think it would be in the spirit of Gherkin’s simplicity to just use the name of the Rule as it’s unique ID. So by duplicating the name of the rule in several places you could associate more scenarios to the Rule.

I think it’s time for someone here to make a concrete proposal and we can go ahead and implement something.

Member

mattwynne commented Feb 18, 2018

I agree about being able to relate Rules and Examples across Gherkin docs. I don’t see the need for a rule-ref, or the additional syntax to go with it. I think it would be in the spirit of Gherkin’s simplicity to just use the name of the Rule as it’s unique ID. So by duplicating the name of the rule in several places you could associate more scenarios to the Rule.

I think it’s time for someone here to make a concrete proposal and we can go ahead and implement something.

@jon-acker

This comment has been minimized.

Show comment
Hide comment
@jon-acker

jon-acker Mar 17, 2018

@ryanwmarsh I'm very much in favour of incorporating the Rules keyword, I almost always include it anyway. I'd be wary of separating them into two files though. Without capability to manage the relationship between them (would Cucumber do this?) we would quickly end up with unrelated rules and examples or rules without examples, etc.
I agree with @lunivore on keeping the Scenario keyword (I might go for example as I use it interchangeably), and I'd leave Feature as it's the most familiar way of grouping system functionality.

jon-acker commented Mar 17, 2018

@ryanwmarsh I'm very much in favour of incorporating the Rules keyword, I almost always include it anyway. I'd be wary of separating them into two files though. Without capability to manage the relationship between them (would Cucumber do this?) we would quickly end up with unrelated rules and examples or rules without examples, etc.
I agree with @lunivore on keeping the Scenario keyword (I might go for example as I use it interchangeably), and I'd leave Feature as it's the most familiar way of grouping system functionality.

@aslakhellesoy

This comment has been minimized.

Show comment
Hide comment
@aslakhellesoy

aslakhellesoy Jul 5, 2018

Contributor

This has been done on master and will be released in Gherkin 6. Then we need to wait for Cucumbers to upgrade to Gherkin 6, which is quite a bit of work.

But it's happening at least!

Contributor

aslakhellesoy commented Jul 5, 2018

This has been done on master and will be released in Gherkin 6. Then we need to wait for Cucumbers to upgrade to Gherkin 6, which is quite a bit of work.

But it's happening at least!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment