Does Behat generate Code Coverage?

[fabiofabbrucci]

Hi
i asking sorry in case I didnt see the options that enable this feature.
I case its not my mistake can behat show codecoverage, in clover format?
Is there a way to integrate behat with phpunit?

Thank you. And have a nice day.

[stof]

Behat does not know about the way the logic is implemented to achieve the behavior so it would be rather hard. When using Mink for instance, it does an HTTP request to a website and tests the output. It has no way to know what has been executed. It could even run on a separate machine (see the demo testing wikipedia for instance)

[everzet]

*.feature files are not tests. It's your project features descriptions. And Behat just gives you a way to describe (with step definitions) how to automatically check whether or not those features are implemented/finished/works.

Code coverage is nonesense for StoryBDD and Behat especially, cuz we're not talking bout code at all. And feature coverage is impossible till someone invents human-mind-(reading|parsing) software for machines :-)

[fabiofabbrucci]

Thank you. Your anserws were very precious to me, to understand better the meaning of BDD versus a TDD.
Thank you very much. Bye bye :)

[Daniel15]

If the feature context is running code, can't you get the coverage of the code it's executing?

[stof]

@Daniel15 the issue is that you could get the code coverage of your step definition, not of your application. Mink does HTTP requests to your applications, so it does not have access at all to the logic in it to get code coverage (it does not even know that the application itself is written in PHP)

[Daniel15]

@stof In some situations, that would be okay. Using Mink is just one way to do testing with Behat. In my case, I'm directly interacting with PHP objects in the step definitions, so getting the code coverage of the step definitions would work fine. Specflow's (similar to Behat or Cucumber but for .NET) approach is to wrap a unit test around each scenario, so tests are just run with a regular unit test runner instead of a specific runner (like Behat), and standard code coverage tools can be used.

[everzet]

Seriously, you're trying to achieve wrong thing here. Behat tests your *.feature files (that your features requirements are achieved), not your code. That's where its name coming from, btw (BEHAvior Tester).

Code coverage is controversial idea and code coverage for StoryBDD framework is just nonsense. If you're doing code testing with StoryBDD - you're doing it wrong.

[Daniel15]

I don't quite follow what you're saying. What's the difference between a test like this:

public function testBasicAddition()
{
    // Arrange / Given
    $calculator = new Calculator();

    // Act / When
    $result = $calculator->add(1, 1);

    // Assert / Then
    Assert::AreEqual(2, $result);
}

And a specification like this:

Scenario: Basic addition
    Given a new calculator
    When I add 1 and 1
    Then the result should be 2

They're both doing the exact same test, except one is written as a scenario instead of a "traditional" unit test. The top one is probably what you'd do when using TDD, whereas the bottom one could be what you do when following BDD> In the end, they're testing the exact same thing.

I don't care about code coverage as a number, I just think getting a coverage report is useful to determine whether my scenarios are comprehensive, and which scenarios I'm missing. From what I know, BDD essentially extends TDD by adding test cases written in a natural language. And guess what TDD involves? Unit testing. And guess what people often do with unit tests? Generate code coverage reports. I really don't think it's nonsense.

Are you saying that Specflow is "doing it wrong" as well? They create unit tests based on the feature files (which could be used for unit tests, or functional tests, or whatever you want). And I'm sure that Cucumber promotes the same thing - They show writing of code that directly interacts with Ruby objects.

[everzet]

You're completely loosing point here. There's two complementary but fairly different ways of doing BDD - SpecBDD and StoryBDD.

SpecBDD is all about developing specifications for your low-level implementation units and communication between those. It's evolution of usual unit tests. It's something you're trying to achieve with your example above, but doing it with wrong tool. Good examples of SpecBDD frameworks are http://rspec.info, http://phpspec.net or in case of your .Net - http://nspec.org.

StoryBDD is about developing human-readable business-targeted behavior descriptions about your application functionality. It's evolution of DDD and functional testing into something mixed. With StoryBDD you're writing scenarios for stakeholders, but those scenarios aren't tests and have no aim to target all application functionality, except the one interesting for business (stakeholder). Although scenarios aren't tests, their acceptance phase could be automated with StoryBDD framework like http://cukes.info, http://www.specflow.org or http://behat.org. With StoryBDD you could start with low-level descriptions like you provided above, but the final idea then is still needs to be "to describe domain area of business", not "to test the code".

In best case, you should use both techniques with or without specific tools for them. But using StoryBDD tool for SpecBDD or vise versa is worst idea you could ever make for your development flow.

Behat isn't a SpecBDD tool, it wasn't been created and highly inefficient as such. Behat is a StoryBDD tool. Although you could use same testing code or assertions behind both, approaches are completely different. And if in one case code coverage might have any sense (SpecBDD) for another (StoryBDD) it's completely nonsense.

[Daniel15]

In best case, you should use both techniques with or without specific tools for them. But using StoryBDD tool for SpecBDD or vise versa is worst idea you could ever make for your development flow.

Why? What disadvantages does using Behat, Specflow or Cucumber for SpecBDD have? I've seen lots of people do it before.

[Daniel15]

In best case, you should use both techniques with or without specific tools for them. But using StoryBDD tool for SpecBDD or vise versa is worst idea you could ever make for your development flow.

Why? What disadvantages does using Behat, Specflow or Cucumber for SpecBDD have? I've seen lots of people do it before.

[MarcelloDuarte]

StoryBDD is specifying how your feature works, SpecBDD is specifying how you classes will achieve that. SpecBDD frameworks are designed to achieve just that so, in principle, they should do that part better. A SpecBDD framework is much more like a unit test framework, mirroring you project class files, so it communicates better which part of the code you are testing. With a StoryBDD you want to know which features you are testing, and they are great for that.

So apart from improving the "test as documentation" part, using the right tool at each level also gives you a better design perspective. A StoryBDD framework implements the focus on the user part of BDD. That focus is carried down to your classes using A SpecBDD framework. The language is their to help you focus on what matters most. Unit Tests frameworks language put the focus on verification rather than specification, which leads to bad tests in the end, as in tests focused on structure, not behaviour. In Cucumber you can still use BDD expectations to keep the focus on behaviour, and I use them in Behat myself, even when I am writing feature focused step definitions - I have a SpecContext I use for that, so I can say e.g. $output->should->be('Chuck Norris') as opposed to $this->assertEquals($output, 'Chuck Norris'). But that's just really because I am a bit too crazy about BDD.

BDD was created to teach developers putting their focus on what matters most while writing tests. In 1999 a ATDD framework was created (which is like a StoryBDD framework) by Ward Cunningham. Ward is a big part of the XP community, do you can see how important it is to have that. That didn't stop him using sUnit for his Smalltalk code. We are running the StoryBDD tool as well and that is telling us what is the next class to Spec.

[everzet]

@Daniel15 main disadvantage about using StoryBDD for writing code specs exists because SpecBDD and StoryBDD operate on absolutely different levels of abstractions.

In StoryBDD, you're forming (creating) domain dictionary by exploring business requirements and added values of the features. This dictionary (you could think - step definitions) is undefined at the beginning of the project and different from area to area. It means, that for every project (and event for most roles inside this project), you should create your own set of language (step definitions), that you will use to describe project domain. And StoryBDD gives you ability to do exactly that - create new dictionaries, explore domain area and expose business values of the features.

In SpecBDD there's only one possible domain language - language of objects and their relations. Language on which you're already writing your code. "This object should be created and this object should be provided as argument". That's why all SpecBDD frameworks are using syntax of language on which they were written - because that's the way your objects (domain) talk in the system.

Now, let's see how using wrong tool for the job could ruin your flow:

1. Using SpecBDD tool for StoryBDD. You'll pretend, that you're talking with business people, but doing it using language which is irrational and not readable to any person outside technical world. Most likely, you'll finish with scenarios or specifications, that will be both unreadable by business people and unmaintainable by developers:

   <?php
   // this code is fictional, but could easily become reality
   // if you're using wrong tools for the job
   $msg = $this->getConfiguration()->get('test_email');
   $story->Given()->I->sendEmail('tester', $msg);
   $story->Then()->getUser('tester')->should()->recieve($msg);

2. Using StoryBDD tool for SpecBDD. You'll try to describe purely implementational (code) details in the tool, that aims on easing exploration of new domains. But in your case (code specs), domain is already known - it's programming language you're using. So, most likely, you'll finish with bloated feature suite and step definition dictionary, that will tend to solve only one thing - recreating your programming language syntax and logic with regular expressions:

   # this step is fictional, but could easily become reality
   # if you're using wrong tools for the job
   Given I instantiate "Daniel15\Entity\TopicMessage" with:
     | author  | hi      |

Both those types of tests share same big problem - after month of active development, they'll become umaintainable, unusable and, finally, abandoned. Just because they will not be able to solve problems they tended to solve - easing development and communications.