Skip to content
This repository has been archived by the owner on May 28, 2019. It is now read-only.

Support for Macros #178

Open
mattwynne opened this issue Apr 18, 2012 · 32 comments
Open

Support for Macros #178

mattwynne opened this issue Apr 18, 2012 · 32 comments

Comments

@mattwynne
Copy link
Contributor

3 different people have asked me about this in the last 24 hours, so I'm going to write it up and we can discuss how it might work

The goal is to allow non-technical people to compose high-level, declarative steps from low-level steps without leaving the familiar, safe, Gherkin layer.

Here's my suggestion. We add a new Macro: keyword to Gherkin which allows you to define a step that maps to other steps.

Example

Scenario: Greet user
  Given a user named matt
  When I sign in as matt
  Then I should see a greeting "hello matt"

Macro: I sign in as <username>
  Given I have visited the homepage
  When I click "sign in"
  And I fill in "Username" with "<username>"
  And I fill in "Password" with "password"
  And I click "Submit"

The idea is that by doing this declaratively at the Gherkin level, we can do away with the gnarly code in Cucumber for supporting #steps. In fact, for a first iteration of this feature, Cucumber would not even need to know about the macros - Gherkin can unpack them and send the low-level steps to Cucumber for invocation.

Some details:

  • macros will be global, so a macro defined in any .feature file will be applicable to any scenario.
  • basic variable substitution will be supported using <angle brackets>. No regexps.

I'm agnostic about whether this is a good idea. I know it will make Cucumber more usable to semi-technical people, but that will come at the cost of building and maintaining this extra complexity in Gherkin. I certainly am not motivated enough to build it myself but I am happy to discuss how it would be done and help someone else built it.

@josephwilk
Copy link
Contributor

Unless I see some really compelling examples I lean against this proposal. -1

My initial thoughts:

  1. I cannot tell in a feature if a plaintext line is a macro or a step definition. Where do I search to find it? Now it could be in the steps or the features. (You could reduce this factor with something like a usage formatter but that requires execution.)
  2. We removed GivenScenario. While this is a more refined GivenScenario I feel a lot of the points we discussed back then still apply (http://blog.josephwilk.net/ruby/cucumber-waves-goodbye-to-givenscenario.html)
  3. While the code around #steps is gnarly as a consumer of Cucumbers api it is not gnarly. Are you suggesting we remove #steps?

My main worry may be unwarranted but I'm afraid this will open Cucumber up to creating a mess of documentation. Though given that is still possible now I feel this would tip the boat.

@jbpros
Copy link
Contributor

jbpros commented Apr 18, 2012

TL;DR

-1, Gherkin is not a programming/scripting language and Macro: would make it too complex.

The whole thing

I've been thinking about this for quite a while and I'm not in favor of this feature.

I second Joseph's thoughts on the potential maintainability issues and the lurking problems that disappeared with the removal of GivenScenario.

I believe Gherkin's (the syntax, not the parser) strength lies in its simplicity and the fact it is not a programming language. Every keyword is a simple documentation label telling what is being described in the system under test. They have very little programmatic meaning, if not none at all. In other words, you can (and should, says Liz Keogh ;)) write scenarios which are not to be automated and yet be immensely useful as a way to communicate and document features and drive development.

The Macro: keyword would be the first "meta" keyword. Unlike all the others, it would not only describe the system under test but the testing system itself too.

That shift of focus could foster writers to express implementation details too soon and not where it belongs. Instead of pushing them down to step definitions or - even better - to some domain layer/DSL, they could easily fit them into macros.

This is quite a big concern in the Cucumber world today: many people are still unsure what level of abstraction is right or wrong; many are even not aware of such concepts as an "implementation detail". We are slowly recovering from the web_steps era in Cucumber and I think the Macro: construct could bring some similar harm, in its own way.

In Gherkin, a step is the atom: it is the smallest piece of information. Steps represent a context, an action or an outcome but their meaning is defined somewhere else: in a step definition (it looks like that name was chosen for a reason ;)). In other words, the definition of a step is not expressed at the Gherkin level and this is a good thing: it leaves room for discussion as it decouples the behaviour from the implementation. Macro: could dim that fragile separation.

I think the Gherkin syntax is good at what it was made for because it is not a programming language. It is an efficient tool for filling the gap between "the business", "the devs" and the code, when it comes to communicating specifications and their validations. I feel that adding the Macro: keyword might be one step too much towards the "dev" side of things, with the risk of fraying that essential communication thread between the stakeholders.

@josephwilk: for my part, yes I'd like to see #steps go away. But that's another story. :)

@relaxdiego
Copy link

Was +1 a couple weeks ago. Now -1.

I'm for the idea of letting step definition writers (QA folks in our company) use english to write stepdefs. As for the proposed solution, I realized that it's actually going to add confusion for Cucumber users. I like to keep separation of concerns wherein feature files contain business requirements whereas the stepdefs contain the test scripts. The solution described above seems to blur the line of separation.

@zdennis
Copy link

zdennis commented Apr 18, 2012

I'm not for or against this at this point, but I'd like to balance out the comments from @josephwilk and @jbpros.

The request seems to be to find a way for non-technical people to combine re-usable sets of steps in a way that doesn't require developers to have to be involved. Before implementation is considered, I would want to know if there is value in this? And not from a developer perspective, but from the perspective of the primary audience of this feature.

There appears to be some perceived value since it has been requested. Are there only two alternatives right now: copy and pasting steps or getting with a programmer? Copy and pasting has its own annoyances. Getting with a programmer also can be problematic and costly.

While Gherkin is not a programming language, I don't think finding a way to make it more useful for one of its intended audiences is a bad thing. This is assuming one of Cucumber's core audiences is non-technical business analysts and QA folks.

Macro may not be the right way to solve this problem, but first we should decide if non-technical people would benefit enough from a feature that allowed them to reduce unnecessary duplication or incur unnecessary cost (or dependency on the programmer). If that is real and valuable then we should explore different approaches to solve that problem.

For empathy...

If you couldn't write language-specific step definitions and you maintained the feature suite for my project and you wanted to write new scenarios based on similar usages of steps from other scenarios, how would you want to go about that? What are your options and how would you prioritize them?

If anyone here is working with non-technical analysts it would be awesome to get some real feedback.

@relaxdiego
Copy link

I'm currently working with non-technical testers and analysts. The analysts work on the feature files and they work great as they are now because they focus on writing business requirements, not test scripts.

    Given I have a role of <Role> in the project
     Then I <Can or Cannot Create> an instance in the project

      Examples: Authorized Roles
        | Role            | Can or Cannot Create |
        | Project Owner   | Can Create           |
        | Developer       | Can Create           |

      Examples: Unauthorized Roles
        | Role            | Can or Cannot Create |
        | Network Admin   | Cannot Create        |
        | Non-Member      | Cannot Create        |

The testers, on the other hand, work on the step definitions and that's where the test scripts are written. Currently, we've implemented the Page Object pattern to help make the syntax easier to write:

When /^I log in with the following credentials: (.*), (.*)$/ do |username, password|
  page = LoginPage.new
  page.visit
  page.should_be_valid
  page.fill_in_username_field username
  page.fill_in_password_field password
  page.submit
end

On the other hand, it would be easier for or non-technical testers to write in a language that's close to english.

When I think of it now, the solution should probably be a DSL that can work alongside (not part of) Gherkin. Something that is meant to be used by stepdef writers who don't have a strong programming background. Something that allows them to write the above as:

When /^I log in with the following credentials: (.*), (.*)$/ do |username, password|
  Visit the Login Page
  Current page should be the Login Page
  Fill in the username field with #{ username }
  Fill in the password field with #{ password }
  Submit the page
end

@mattwynne
Copy link
Contributor Author

To clarify for @josephwilk and @jbpros yes, my primary reason for suggesting this so that we can remove #steps from Cucumber 2.0. As @zdennis has pointed out, there is definitely a need for something like this from a proportion of our user base, and IMO #steps is the wrong solution: it forces people who are not comfortable doing programming to work in Ruby files, and it is horrible to implement from our side.

Rather than just yanking out #steps, I'd like to look for a better way to give this kind of user what they want.

If we're going to support this feature at all, I think we need something declarative that will allow IDEs and other tools to let users click through to macro definitions (or whatever we call them) without having to execute anything to understand the mapping. At the moment, you don't know whether you're looking at a nested step or a regular step, but if these were defined in the Gherkin it would be easier for a tool to analyse the Gherkin statically and see what was happening.

Some suggestions:

  1. Remember just because they're stored in .feature files, they don't have to clutter up your specs. They can live in `features/macros', for example.
  2. Perhaps another name would make them feel less technical? The term 'macro' comes from Fitnesse I think, but perhaps we can come up with something better, something more like a footnote you'd have in a traditional document[1].

How about Expand:

Expand: When I log in with the following credentials: <username>, <password>
  * Visit the Login Page
  * Current page should be the Login Page
  * Fill in the username field with <username>
  * Fill in the password field with <password>
  * Submit the page

Let's keep brainstorming on the idea and listening to users like @relaxdiego who obviously have a need for something like this, and see if we can figure out a solution that works better than #steps.

[1] like this?

@josephwilk
Copy link
Contributor

To clarify @relaxdiego are in favour or against this feature? I thought your where against it from your previous post.

@relaxdiego
Copy link

@josephwilk I'm for the idea of giving stepdef writers the ability to write stepdefs in English. I was against the proposed solution but @mattwynne's clarification makes it sound like it's headed in the right direction after all. Can I be conditionally in favor of it? My concerns:

  1. It shouldn't cause confusion re separation of concern. I.e. it should be clear that feature files are the home of business requirements whereas "macros" are the home of test scripts.
  2. These items within these "macros" should not be directly usable in the feature files. They should only be used within "macros." This is to avoid the problems caused by web_steps.rb
  3. How will variables be handled between each item in the macro? In the Page Object based example I shared above "page" is re-used until the end of the stepdef. Will this be possible with the macro?

@josephwilk
Copy link
Contributor

Thanks @relaxdiego thats helpful.

@zdennis I think one problem we have is that this is not the right medium to get feedback and examples from non-technical analysts. If we want to learn more from those people I feel we need to move this discussion somewhere more suitable. Having said that I'm not sure what that medium is 😔

@jbpros
Copy link
Contributor

jbpros commented Apr 19, 2012

@josephwilk I'll answer your question tonight when I get some more time.

In the meantime, I've started a little poll about the removal of #steps: http://checkthis.com/3hrs (the last option was added a few seconds ago).

@zspencer
Copy link

I am torn.

On the one hand, I greatly appreciate the desire for non technical people writing feature files to group and reuse sets of step definitions.

On the other hand, every time I have seen this played out it has caused difficulties with maintainability of feature files.

In the past, I have extracted each steps implementation and mixed it into world; then I have written a step that explains the permutation (i.e "Given a user who may access the flim flam machine") and in that step definition called the extracted methods.

Remember, Gherkin is not something that is handed from a business owner to the developer. It's the start of a conversation between the people who know what should be built, and the people who are building it.

@relaxdiego
Copy link

@zspencer maybe we should stop thinking of this feature as a way to group and reuse sets of step definitions. Rather, we should think of it as something that allows step definition writers (not feature file writers) to use an English-like language for creating step definitions. In this case, it's really an alternative to Ruby when writing stepdefs. It's a language that is not part of Gherkin, but something totally different and only used for stepdefs.

@mattwynne
Copy link
Contributor Author

@relaxdiego right. This is a way for non-technical people to create good declarative scenarios from a small number of basic building block steps, without having to work in Ruby files.

Crucially, it also gives those people visibility about what goes on inside that step, thereby building more trust in what the scenario is doing. Ideally, they should have a good trusting relationship with the programmer who implements the step definition code (that calls nicely extracted World methods) but this isn't the context that everyone operates in.

@gdinwiddie
Copy link

The word "macro" sounds a bit techy to me. It may be clear enough to a broad range of casual computer users from experience with spreadsheets, though.

The basic problem I see is this: I think it's beneficial for testers or testers paired with business people to be able to create new scenario variants without having to go get a programmer to help them. When they do so, however, I see lots of duplication in the scenarios. They become more imperative and less declarative.

I would like to help them learn the rudiments of programming (avoiding duplication, symbolic representation of larger concepts) while remaining in gherkin. I would like to give them more of a sense of being masters of their own fate, and less beholden to the programmers, without having to retreat to record-and-playback tools.

@mattwynne
Copy link
Contributor Author

So @gdinwiddie do I take it that you like the idea, just not the name?

Can you suggest any alternative names? I was thinking Abstraction: might be a good one. Does what it says on the tin.

@gdinwiddie
Copy link

Naming is hard. Maybe Fragment: would be clear? Perhaps we could try something and adjust it (with aliases) over time.

For those who think this feature is a bad idea, it's not intended to replace coded steps. It's an overlap. There's not a distinct point where you should make a switch from gherkin to a programming language.

@gdinwiddie
Copy link

Or Scenario Fragment: which is similar to Scenario Outline:

@relaxdiego
Copy link

I don't understand why we have to invent a keyword for this. I'm also getting mixed messages. At one point I thought this was turning into an alternative to using Ruby when creating stepdefs (which i think is the right solution) but then it's beginning to sound like it's an extension to Gherkin (which I'm not so sure is a good idea)

If this feature is be meant for testers who don't have a programming background, then the solution should really just be a DSL that they can use for writing step definition files. The file that they write should be saved in the step_definitions directory so that there will be no confusion about where the tester will save what. In this solution, there's no need to invent new keywords such as Macro or Step Fragment because Cucmber already knows to look i the step_definitions directory for executing each step.

@mattwynne
Copy link
Contributor Author

@relaxdiego actually, having a Ruby DSL is precisely how I personally use, and recommend other teams try to use Cucumber. You can already do that.

However, I'm raising this ticket because I meet lots of people who would like to be able to refactor their features, raising the level of abstraction in their scenarios by effectively extracting a high-level step, without going into a Ruby file.

@aslakhellesoy
Copy link
Contributor

No matter how you slice or dice it, this feature request is about adding functions or subroutines [1] to the gherkin language. The reason for doing so is to make it easier for non-programmers to create more declarative scenarios on their own.

By making it easier for non-programmers to work on their own we are also undermining one of Cucumber's essential values - Collaboration.

The people who want this feature don't know how to program. If they did they would create their subroutines in their Cucumber implementation's programming language (Ruby, Java, JavaScript etc).

Guess what a person without programming skill and without a programmer buddy will do with subroutines. They will create a mess.

Down the road people will blame this mess on Cucumber, which I'm not really interested in. But more importantly, the poor people using this feature will create a mess for themselves. And they won't realise this until they have a big stinking pile of poo.

I understand why people want this feature, but I don't see anything good coming out of it.

There is no shortcut to creating high level, declarative scenarios. You either have to learn to program or work closely with someone who does.

[1] http://en.wikipedia.org/wiki/Function_(computer_science)

@relaxdiego
Copy link

No matter how you slice or dice it, this feature request is about adding functions or subroutines [1] to the gherkin language. The reason for doing so is to make it easier for non-programmers to create more declarative scenarios on their own.

By making it easier for non-programmers to work on their own we are also undermining one of Cucumber's essential values - Collaboration.

This hasn't been the case in our experience. Right now, we're using nested 'web' and 'low level' steps. The process thus far has been:

  • When non-programmer wants to write a stepdef: Non-programmer writes the stepdef using nested low-level steps, then shows it to a programmer for review.
  • When non-programmer wants to review a stepdef: Non-programmer looks at the stepdef and reviews each step. If he notices something that doesn't jive, he contacts a programmer and they discuss.

Here's a sample stepdef from our project:

When /^I create a project with attributes (.*), (.*)$/ do |name, desc|
  steps %{
    * Click the Logout button if currently logged in

    * Visit the login page
    * Fill in the Username field with #{ current_user.name }
    * Fill in the Password field with #{ current_user.password }
    * Click the Login button

    * Visit the Projects page

    * Click the New Project button
    * Fill in the Project Name field with #{ name }
    * Fill in the Project Description field with #{ description }
    * Click the Save Project button
  }
end

So far, this has actually enhanced collaboration in our team because now the BAs and the testers can join in the discussion regarding how a step should be executed. Compare that to our previous process where only the programmers know how to read the stepdef, while BAs and testers are limited to the feature files because they can't understand the stepdefs.

Note the last phrase in the above paragraph. That's been one of the reasons why Cucumber has been misused (i.e. tendency for a team to write imperative steps). Because stepdefs are written in a language that is alien to many team members, the tendency of the team is to write imperative steps so that the BAs and the testers know what the heck is going on. With nesting/macros/whatever, this doesn't have to be the case.

Granted the proposal as it stands right now may need some more improvement, that doesn't invalidate the whole thing. This is useful and, I believe, will improve collaboration rather than undermine it.

@diabolo
Copy link

diabolo commented May 22, 2012

-1

The idea of 'non-technical' people combining a re-usable set of steps to create automated tests has been around for quite some time. It is and has always been fundamentally flawed. Cucumber by its design makes it feasible for non-technical people to specify what a test should do. But as soon as this slips into how a test should do something then some degree of technicality is required.

At the very least every feature should fail before it passes. This requires technical knowledge - you have to know its failing for the correct reason, and you have to know that additional functionality that is added is now responsible for making it pass.

One of the biggest counter arguments to using Cucumber is that it makes everything so difficult. This feature will exacerbate this problem by vastly increasing the complexity of feature files, and encouraging the creation of features that are highly coupled and prone to breaking.

The problem of combining a number of operations in a sensible and repeatable way has been well addressed by programming languages. They have the tools (and we all know these extend far beyond just having subroutines) for dealing with this problem with grace and elegance. Cucumber currently encourages people to avoid using the appropriate tools for this composition problem by allowing nested steps. Allowing nested features will make things even worse. The place for nesting is in the programming language that implements the step definition , because only a programming language has sufficient power to implement this composition effectively.

Finally the re-use of step definitions is highly counterproductive and overrated. Generally all it does is restrict the language business users can use to express themselves and encourage a kind of programming by proxy. The essential beauty of cucumber is the clear separation between features and step definitions. This was horribly muddied by web steps and early implementations of feature sets, and unfortunately cucumber still hasn't recovered from this.

@gdinwiddie
Copy link

I've been thinking about an alternative implementation. Perhaps plain-text steps should just be in a .step file that could be handled similarly to steps written in any other language. Something like:

Step: I sign in as <username>
  I have visited the homepage
  I click "sign in"
  I fill in "Username" with "<username>"
  I fill in "Password" with "password"
   I click "Submit"

Some of the comments on this issue, here and elsewhere, seem quite militant that implementation of steps should be left to real programmers using a real programming language. I find this attitude anti-collaborative. It smacks of "don't you worry your pretty little head about this, we'll take care of the complicated stuff." Sometimes we should remind ourselves that testers and analysts are capable adults who can also do things that we cannot.

Technical knowledge and programming ability isn't a boolean value. We should not strive for a sharp distinction between the realm of a tester and that of a programmer.

I agree that web-steps was a mistake, but not because it gave low-level imperative step definitions. It was a mistake because it took those definitions out of the control of the team using them. It's beneficial to have an overlap between what can be done in "plain language" and what can be done in a programming language. It's important, though, that it be easy for teams to make their own decisions about what gets done where, based on the needs of the project and the capabilities of the participants.

The ability to refactor imperative scenarios into declarative ones by using "extract step definition" is a huge win, in my opinion. Having to simultaneously reimplement these steps in a different language is a bigger, more error prone, step than I would prefer, even as a programmer.

@relaxdiego
Copy link

+1 to @gdinwiddie's comments. Also, I created a sample app for a fictitious framework in the hope of providing a more concrete example of what I'd like to see happen. See https://github.com/relaxdiego/norm_app

@diabolo
Copy link

diabolo commented May 22, 2012

Just to be clear I believe that that testing and programming are heavily overlapping activities and that both features and steps should be written collaboratively. However I'm also absolutely clear that writing features is a distinctly different activity to implementing steps.

If you make writing features a programming activity then you undermine its core function which is to capture business knowledge. It should be possible for anyone with domain knowledge to write features.

If you don't apply programming disciplines to the activity of writing step definitions you end up with a set of step definitions that hampers the writing of new features. This can take many forms including

  • forcing/coercing new features to use a particular language
  • creating a fragile feature set, where changes to one part of the implementation break other features
  • encouraging the idea that features can just be written and work without the need for a separate implementation activity

Generally I believe that the level of programming discipline required to implement a really useful set of step definitions is quite minimal, and can be learnt by all sorts of people, programmers, business analysts, testers even business owners with the right guidance. Cucumber has all the tools (and more) to do this. However there is a general trend to make this far more difficult than it needs to be, and then claim that we need additional functionality like 'support for macros', because writing step definitions is too difficult.

Cucumber already has a reputation for being difficult and unnecessary. Encouraging the effective use of what the tool already gives us, might be more productive than adding additional features that the tool doesn't need.

@relaxdiego
Copy link

Cucumber already has a reputation for being difficult and unnecessary. Encouraging the effective use of what the tool already gives us, might be more productive than adding additional features that the tool doesn't need.

Encouraging the effective use of Cucumber might not help if how the suggested use is in conflict with how the tool is currently designed. Cucumber is touted as a tool that allows for collaboration between POs, BAs, testers, and developers. However, when you look at its current design, and if you take away Gherkin, you are really still left with a tool for developers. Gherkin right now isn't enough. While it gives POs, BAs, and testers a glimpse of what the team is trying to design, going beyond that is not possible. How will they know that the devs are testing the right thing?

We can't expect POs, BAs, and testers to learn Ruby considering that they already have their own concerns re the team's work. This then puts a lot of burden on the devs to maintain Cucumber and probably even explain to the non-programmers the ins and outs every now and then.

However, if we allow the understanding of the ins and outs of Cucumber to spread out to the rest of the team (i.e. allow stepdefs to read like natural language bulletpoints) then there will be better collaboration within it.

@mattwynne
Copy link
Contributor Author

I'm not actually interested in arguing about the rights and wrongs of this feature. Cucumber already supports #steps calls from Ruby code, and we know from feedback that it's popular enough we need to support that kind of functionality somehow. It might not be for everyone, but that doesn't make it wrong. The question I'm raising here is how we best provide this feature.

From what I can tell, this feature is most used by teams where there is a lot of ownership and interest in the Cucumber features by people whose strength isn't writing and maintaining Ruby code. My suggestion is that we recognise that, and move this feature into a medium that doesn't involve them having to open up Ruby files.

Having them expressed declaratively in Gherkin would have the additional benefit of being able to use tools to reflect on the relationships between these steps without having to execute them. Right now, the only way we can find out about nested steps is by running the scenarios, and I think this is a disadvantage.

@diabolo
Copy link

diabolo commented May 24, 2012

Matt,

If you are going to move Cucumber along this path could I ask that you consider some sort of extension architecture. Is the idea perhaps that there is a simple core gherkin and then extensions for this sort of functionality e.g.

macros
looping
... whatever comes next

In addition could there be refactorings that move things currently in gherkin into extensions? e.g. example groups, tables.

I think this approach might allow you to meet the needs of both audiences.

All best

Andrew

@diabolo
Copy link

diabolo commented May 24, 2012

I've been thinking a lot more about this feature and my attempt at a compromise above. I've come to the conclusion that any change to the gherkin language is absolutely fundamental. My extension proposal would result in the fragmentation of gherkin - minimal gherkin, gherkin with loops etc.. Gherkin is now pretty much a language specification.

So I've discarded any thoughts of a minimal cucumber. The disruption of the gherkin language caused by removing things is a very high a price to pay for any improvements in the purity of the language.

We are in a situation where removing functionality from Gherkin is very disruptive and we have to be extra careful about adding functionality to gherkin - it will be very hard to go back if we get things wrong.

My opposition to this change is stronger than ever. The cure is far worse than the disease. However it also means that I remove all of my objections to any additional syntax in step definitions or even some intermediate .steps file that help people write steps in the way that George or Mark are proposing. Better to leave in nested steps or even enhance them than to move the functionality into gherkin.

All best

Andrew

@relaxdiego
Copy link

NOTE: I originally posted this to the Cucumber mailing list but realized later that I should've posted it here

Here's a proof of concept framework that I wrote the past couple of days to illustrate the benefits of natural language in test scripts. I call this framework 'Norm'. Note that this is an executable application, not just a bunch of text files. It also has unit tests to keep out the bugs that I know of.

We start with listing the requirements which looks something like the feature files but not exactly:

directives/requirements/create_a_project.requirements

Create a Project
================

  Requirement:
    A user with a role of <Role> in the system <Can or Cannot Create> a project

    Examples:
      | Role            | Can or Cannot Create |
      | System Admin    | Can Create           |
      | User            | Cannot Create        |


  Requirement:
    A new project with attributes <Name>, <Description> is <Valid or Not>

    Examples:
      ----------------------------------------------------------------
      | Name    | Description     | Valid or Not | Remarks           |
      |---------|-----------------|--------------|-------------------|
      | Awesome | Another project | Valid        |                   |
      | (None)  | Another project | Invalid      | Empty Name        |
      | Awesome | (None)          | Invalid      | Empty Description |
      ----------------------------------------------------------------

The difference with this and feature files apart from the filename extension is that it doesn't require Given/When/Then keywords (although you're welcome to start your sentences with those words if you really want to). The requirement declaration can be written in free form, using multiple lines if desired. Here's a multi-line example:

directives/requirements/log_in.requirements

Log In
======

  Requirement:
    Given my username is rstark and my password is 
    w1nt3rf3ll, I <Can or Cannot Log In> with the 
    following credentials: <Username>, <Password>

    Examples:
      | Username  | Password   | Can or Cannot Log In |
      |-----------|------------|----------------------|
      | rstark    | w1nt3rf3ll | Can Log In           |
      | RSTARK    | w1nt3rf3ll | Cannot Log In        |

Note that you can optionally add borders and header separators for the example tables. Also note that you only need to use the 'Requirement:' keyword and have no need for a 'Requirement Outline:' keyword. That is, you can use the 'Requirement' keyword for formats that are similar to 'Scenario' and 'Scenario Outline' in the latest version of Cucumber.

That's for the requirement level. Now for the test script level:

Each requirement can be matched by any of the defined test scripts. For create_a_project.requirements, its corresponding test scripts may be found at:

directives/test_cases/create_a_project.test_cases

Test Case:
  A user with a role of (.+) in the system can create a project

  Variables:
    * Role Name    = $1
    * Project Name = 'test'
    * My Username  = my_username
    * My Password  = my_password

  Preconditions:
    * I have a role of <Role Name> in the system

  Script:
    * Click the Logout button if I'm currently logged in
    * Fill in the Username field with <My Username>
    * Fill in the Password field with <My Password>
    * Click the Login button

    * Click the Projects link
    * Click the New Project button
    * Fill in the Project Name field with <Project Name>
    * Fill in the Project Description field with 'Test project'
    * Click the Create Project button

    * A project named <Project Name> should be visible in the page
    * The project should exist in the backend cloud

  Cleanup:
    * Delete the project named <Project Name> if it exists
    * Delete my username at exit

Test Case files have a different syntax from Requirements files. For instance:

  • Test cases contain three 4 sections: Variables (optional), Setup (optional), Script, and Cleanup (optional)
  • The Cleanup section will execute no matter what happens in the preceding sections
  • Lines cannot break
  • Lines may optionally begin with an asterisk
  • Variables can be assigned a value through regex match variables ($1, $2, etc), literals, and functions
  • Variables can be used in the Setup, Script, and Cleanup section

Finally, the steps level

Each line in the Setup, Script, and Cleanup section of a test script can be matched by a step defined in the utilities folder. For example:

directives/utilities/steps/user_steps.rb

Steps.add /^I have a role of (.+) in the system$/ do |role_name|
  # Add ruby code here
end

This part is pretty much the same as how steps are defined in Cucumber. The difference is that steps are accessible from within test cases only. Requirements cannot use steps directly.

Please note that I have no plans for maintaining this proof of concept. It's just meant to illustrate how it might 'feel' if Cucumber implemented this feature. For more information on how to play with Norm, please see the readme at https://github.com/relaxdiego/norm

@aslakhellesoy
Copy link
Contributor

@famished-tiger
Copy link

To put some oil on the fire: I started a little project in Github called Macros4Cuke. It results in a gem macros4cuke that adds a macro-facility to Cucumber.

With it, it is possible to rewrite the original example from Matt like this:

Scenario: The macro is defined here...
    Given I define the step "When I [sign in as <username>]" to mean:  
    """
  Given I have visited the homepage
  When I click "sign in"
  And I fill in "Username" with "<username>"
  And I fill in "Password" with "password"
  And I click "Submit"
  """

Scenario: Greet user
  Given a user named matt
  When I [sign in as "matt"]
  Then I should see a greeting "hello matt"

The nice thing about all this, is that no change in Cucumber is required.
The syntax remains familiar, except for a small detail: use of the square brackets [...].

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants