Does Behat generate Code Coverage?

[fabiofabbrucci]

Hi
i asking sorry in case I didnt see the options that enable this feature.
I case its not my mistake can behat show codecoverage, in clover format?
Is there a way to integrate behat with phpunit?

Thank you. And have a nice day.

[stof]

Behat does not know about the way the logic is implemented to achieve the behavior so it would be rather hard. When using Mink for instance, it does an HTTP request to a website and tests the output. It has no way to know what has been executed. It could even run on a separate machine (see the demo testing wikipedia for instance)

[everzet]

*.feature files are not tests. It's your project features descriptions. And Behat just gives you a way to describe (with step definitions) how to automatically check whether or not those features are implemented/finished/works.

Code coverage is nonesense for StoryBDD and Behat especially, cuz we're not talking bout code at all. And feature coverage is impossible till someone invents human-mind-(reading|parsing) software for machines :-)

[fabiofabbrucci]

Thank you. Your anserws were very precious to me, to understand better the meaning of BDD versus a TDD.
Thank you very much. Bye bye :)

[Daniel15]

If the feature context is running code, can't you get the coverage of the code it's executing?

[stof]

@Daniel15 the issue is that you could get the code coverage of your step definition, not of your application. Mink does HTTP requests to your applications, so it does not have access at all to the logic in it to get code coverage (it does not even know that the application itself is written in PHP)

[Daniel15]

@stof In some situations, that would be okay. Using Mink is just one way to do testing with Behat. In my case, I'm directly interacting with PHP objects in the step definitions, so getting the code coverage of the step definitions would work fine. Specflow's (similar to Behat or Cucumber but for .NET) approach is to wrap a unit test around each scenario, so tests are just run with a regular unit test runner instead of a specific runner (like Behat), and standard code coverage tools can be used.

[everzet]

Seriously, you're trying to achieve wrong thing here. Behat tests your *.feature files (that your features requirements are achieved), not your code. That's where its name coming from, btw (BEHAvior Tester).

Code coverage is controversial idea and code coverage for StoryBDD framework is just nonsense. If you're doing code testing with StoryBDD - you're doing it wrong.

[Daniel15]

I don't quite follow what you're saying. What's the difference between a test like this:

public function testBasicAddition()
{
    // Arrange / Given
    $calculator = new Calculator();

    // Act / When
    $result = $calculator->add(1, 1);

    // Assert / Then
    Assert::AreEqual(2, $result);
}

And a specification like this:

Scenario: Basic addition
    Given a new calculator
    When I add 1 and 1
    Then the result should be 2

They're both doing the exact same test, except one is written as a scenario instead of a "traditional" unit test. The top one is probably what you'd do when using TDD, whereas the bottom one could be what you do when following BDD> In the end, they're testing the exact same thing.

I don't care about code coverage as a number, I just think getting a coverage report is useful to determine whether my scenarios are comprehensive, and which scenarios I'm missing. From what I know, BDD essentially extends TDD by adding test cases written in a natural language. And guess what TDD involves? Unit testing. And guess what people often do with unit tests? Generate code coverage reports. I really don't think it's nonsense.

Are you saying that Specflow is "doing it wrong" as well? They create unit tests based on the feature files (which could be used for unit tests, or functional tests, or whatever you want). And I'm sure that Cucumber promotes the same thing - They show writing of code that directly interacts with Ruby objects.

[everzet]

You're completely loosing point here. There's two complementary but fairly different ways of doing BDD - SpecBDD and StoryBDD.

SpecBDD is all about developing specifications for your low-level implementation units and communication between those. It's evolution of usual unit tests. It's something you're trying to achieve with your example above, but doing it with wrong tool. Good examples of SpecBDD frameworks are http://rspec.info, http://phpspec.net or in case of your .Net - http://nspec.org.

StoryBDD is about developing human-readable business-targeted behavior descriptions about your application functionality. It's evolution of DDD and functional testing into something mixed. With StoryBDD you're writing scenarios for stakeholders, but those scenarios aren't tests and have no aim to target all application functionality, except the one interesting for business (stakeholder). Although scenarios aren't tests, their acceptance phase could be automated with StoryBDD framework like http://cukes.info, http://www.specflow.org or http://behat.org. With StoryBDD you could start with low-level descriptions like you provided above, but the final idea then is still needs to be "to describe domain area of business", not "to test the code".

In best case, you should use both techniques with or without specific tools for them. But using StoryBDD tool for SpecBDD or vise versa is worst idea you could ever make for your development flow.

Behat isn't a SpecBDD tool, it wasn't been created and highly inefficient as such. Behat is a StoryBDD tool. Although you could use same testing code or assertions behind both, approaches are completely different. And if in one case code coverage might have any sense (SpecBDD) for another (StoryBDD) it's completely nonsense.

[Daniel15]

In best case, you should use both techniques with or without specific tools for them. But using StoryBDD tool for SpecBDD or vise versa is worst idea you could ever make for your development flow.

Why? What disadvantages does using Behat, Specflow or Cucumber for SpecBDD have? I've seen lots of people do it before.

[Daniel15]

In best case, you should use both techniques with or without specific tools for them. But using StoryBDD tool for SpecBDD or vise versa is worst idea you could ever make for your development flow.

Why? What disadvantages does using Behat, Specflow or Cucumber for SpecBDD have? I've seen lots of people do it before.

[MarcelloDuarte]

StoryBDD is specifying how your feature works, SpecBDD is specifying how you classes will achieve that. SpecBDD frameworks are designed to achieve just that so, in principle, they should do that part better. A SpecBDD framework is much more like a unit test framework, mirroring you project class files, so it communicates better which part of the code you are testing. With a StoryBDD you want to know which features you are testing, and they are great for that.

So apart from improving the "test as documentation" part, using the right tool at each level also gives you a better design perspective. A StoryBDD framework implements the focus on the user part of BDD. That focus is carried down to your classes using A SpecBDD framework. The language is their to help you focus on what matters most. Unit Tests frameworks language put the focus on verification rather than specification, which leads to bad tests in the end, as in tests focused on structure, not behaviour. In Cucumber you can still use BDD expectations to keep the focus on behaviour, and I use them in Behat myself, even when I am writing feature focused step definitions - I have a SpecContext I use for that, so I can say e.g. $output->should->be('Chuck Norris') as opposed to $this->assertEquals($output, 'Chuck Norris'). But that's just really because I am a bit too crazy about BDD.

BDD was created to teach developers putting their focus on what matters most while writing tests. In 1999 a ATDD framework was created (which is like a StoryBDD framework) by Ward Cunningham. Ward is a big part of the XP community, do you can see how important it is to have that. That didn't stop him using sUnit for his Smalltalk code. We are running the StoryBDD tool as well and that is telling us what is the next class to Spec.

[everzet]

@Daniel15 main disadvantage about using StoryBDD for writing code specs exists because SpecBDD and StoryBDD operate on absolutely different levels of abstractions.

In StoryBDD, you're forming (creating) domain dictionary by exploring business requirements and added values of the features. This dictionary (you could think - step definitions) is undefined at the beginning of the project and different from area to area. It means, that for every project (and event for most roles inside this project), you should create your own set of language (step definitions), that you will use to describe project domain. And StoryBDD gives you ability to do exactly that - create new dictionaries, explore domain area and expose business values of the features.

In SpecBDD there's only one possible domain language - language of objects and their relations. Language on which you're already writing your code. "This object should be created and this object should be provided as argument". That's why all SpecBDD frameworks are using syntax of language on which they were written - because that's the way your objects (domain) talk in the system.

Now, let's see how using wrong tool for the job could ruin your flow:

1. Using SpecBDD tool for StoryBDD. You'll pretend, that you're talking with business people, but doing it using language which is irrational and not readable to any person outside technical world. Most likely, you'll finish with scenarios or specifications, that will be both unreadable by business people and unmaintainable by developers:

   <?php
   // this code is fictional, but could easily become reality
   // if you're using wrong tools for the job
   $msg = $this->getConfiguration()->get('test_email');
   $story->Given()->I->sendEmail('tester', $msg);
   $story->Then()->getUser('tester')->should()->recieve($msg);

2. Using StoryBDD tool for SpecBDD. You'll try to describe purely implementational (code) details in the tool, that aims on easing exploration of new domains. But in your case (code specs), domain is already known - it's programming language you're using. So, most likely, you'll finish with bloated feature suite and step definition dictionary, that will tend to solve only one thing - recreating your programming language syntax and logic with regular expressions:

   # this step is fictional, but could easily become reality
   # if you're using wrong tools for the job
   Given I instantiate "Daniel15\Entity\TopicMessage" with:
     | author  | hi      |

Both those types of tests share same big problem - after month of active development, they'll become umaintainable, unusable and, finally, abandoned. Just because they will not be able to solve problems they tended to solve - easing development and communications.

[everzet]

Reopened issue and moved it into FAQ section to prevent such questions in the future and provide detailed explanation on why not.

[Daniel15]

Thanks for all the information, @everzet and @MarcelloDuarte. I've learned some great things here, I appreciate it . I'll look into PHPSpec and NSpec. :)

With StoryBDD, isn't code coverage information about your business logic classes still somewhat useful, to know if you've missed a certain business rule in your tests (if you're writing tests for an existing codebase)?

[everzet]

@Daniel15 classes in most cases is a bad representation of business features. It's great representation of their implementation :-)

As of pure business features coverage report, indeed, it could be extremely useful. The problem here is that computers at the current level of technology doesn't have the way to predict or create new business rules for concrete application. Even developers in most cases can't. The only person, who is sufficient with this kind of business predictions is product owner (stakeholder). Coverage of the non-described business features for the product highly relies on ability to predict/calculate those business features first.

I think, technology that will be able to automatically tell you what features you need based only on initial idea of the project would be a next big thing after iPad mini. But we don't know yet whether any of those would be actually released :-)

As about already defined business requirements (written *.feature files), Behat already have informal coverage for them in form of undefined, pending and failed scenarios. Building some sort of chart from this data is as easy as creating new Behat formatter.

[AndyAtTogether]

I think code coverage on .feature files is very relevant. My reasoning is that i don't want code that i don't need. If i've got code that is uncovered then either it is redundant and should be removed or someone has had to write code that is required but is not driven by a .feature and that should be an indicator that its time to pause and have a re-think.

Anyway thats my 2 pence worth on the subject, however it appears to be moot as using specflow i can't get coverage information on the code under test, only on the step definitions which other than for identifying redundant step definitions (which is suppose there may be some merit in) renders the coverage information pretty useless

[everzet]

@AndyAtTogether again. *.feature files are not functional tests - they are business-oriented domain descriptions, that could be tested (domain descriptions, not application).

If something isn't described in *.feature - it doesn't mean, that this part of application is redundant or should be removed - it just means, that this part is not important for business. Though some application code is not important to business, it still could be important part of it's implementation (support/extension points f.e.).

And yes, StoryBDD or Behat particularly doesn't replace functional testing entirely, though intersect it in general cases.

[AndyAtTogether]

I agree with you, and i'd like to know what code has been written that is "not important for the business", I think its a really useful sanity check to see what code has been driven by features and what code is not.

Happily i can now do this to my hearts content using specflow and dotcover as now I'm able to get the coverage information i was looking for :-)

[nicopaez]

I agree with AndyAtTogether.
*.feature files are executable, that is the value they add, if there weren't executable, then we would continue writing Word-based specifications. So, when I run my .features the application code is invoked and it is important to know if any fragment of your code is not been invoked because I could be that the code is not required.

In Ruby I do this with Cucumber and SimpleCov.

[everzet]

@nicopaez you misunderstood purposes of Behat (and probably cucumber). Behat doesn't run your features, it tests them. Your features are not runnable - they are just plain text descriptions of your project business features. Context describes how to test them, Behat automates this testing. Nobody runs features, they're just testable. You're writing not tests, but textual descriptions of application business values, that could be tested itself later with Behat. And same goes for Cucumber. *.feature files are not tests. If you're writing them as tests, you're creating big problems for yourself in the future - read this tread to understand why.

[nicopaez]

@everzet how is Behat supposed to test them? It executes your code!!!

And if that is your opinion, then you should update Behat documentation:

Behat is a tool that makes behavior driven development (BDD) possible. With BDD, you write human-readable stories that describe the behavior of your application. These stories can then be run as actual tests against your application.

[everzet]

And if that is your opinion, then you should update Behat documentation...

@nicopaez thanks for pointing. Fixed: https://github.com/Behat/en-docs.behat.org/commit/6de9edaaa43305b6ff5892fdc2be7b17d209b151

[everzet]

@everzet how is Behat supposed to test them? It executes your code!!!

Absolutely right. FeatureContext is a test, *.feature file is not. Don't confuse those two as they even have different aims. *.feature and Gherkin describes your application business, Cucumber & Behat tests your application AND those business rules from *.feature files, not contrary.

And i don't want to repeat myself, but:

If something isn't described in *.feature - it doesn't mean, that this part of application is redundant or should be removed - it just means, that this part is not important for business. Though some application code is not important to business, it still could be important part of it's implementation (support/extension points f.e.).

[nicopaez]

Thanks!

[xczimi]

I understand why you are trying to enforce this, but.
BDD stands for Behavior Driven Development not Business Driven Development. There are behaviors simple or complex on many different levels/tiers of all applications.
When it comes to applications, business is the consumer, but all other tiers of software have consumers (other software, developers, etc) and they all depend on behavior of the underlying "thing" .
Gherkin allows us to describe these behaviors in English in a computer or solution agnostic way. This helps understanding expectations on many levels.
So accepting that not for the primary reason behat was constructed, I'd argue that code coverage could be a relevant property of a behat execution of tests described by gherkin.

[everzet]

@xczimi usage of Behat doesn't cancel requirement to spec or unit test your code. Of course, you can write websites in assembler and even have some benefits out of the process, but there are much better tools for this problem on the market and developer's job to choose and use them. Behat wasn't been created to operate on code level, it's StoryBDD tool (yes, StoryBDD, not WholeBDD) - there's also SpecBDD, which has different goals and provides different tools for implementation level.

[yuriteixeira]

Well, I respect and completely understand the points from you guys, but in my humble opinion, in some cases, like an API developed with Silex, that was previously covered with automated functional tests, Behat comes as a great alternative and code coverage feature would be a great addition. Let me try to explain myself:

Look at this code: https://github.com/yuriteixeira/behat-silex. Instead of define how the API should behave with bloated functional tests, I just described how it should work with a natural language.

So, to test it, I just instantiate the Application Kernel and through \Symfony\BrowserKit\Client, I run the API without do a real http request. Because of this, PHPUnit can generate a code coverage report and so Behat, since I do the same in my FeatureContext.php.

Don't you think that at least on this case, when I want to test how an API behaves, code coverage couldn't be an interesting addition?

[sooqini]

@everzet I feel your argument is too purist, sure code coverage and StoryBDD doesn't make much sense compared to SpecBDD or TDD and usage of Behat doesn't cancel the requirement to spec or unit test your code.

But in the real world of constrained budgets and time, we tend to end up cutting some corners and knowing the coverage of your feature tests is valuable information without incurring a big overhead.

That being said, I don't think its particularly the job of Behat to manage/generate the coverage specially as its not the coverage of the story that we are interested in here.

[everzet]

@sooqini I'm not against doing it. I'm against doing it in the core.

[alwinmark]

I think the Point is, that using Behat for the codecoverage is only useful if you don't do BDD or TDD. Because otherwise you create the tests before implementing the classes and therefore you should have a very high code coverage measuring the qualitiy of your behaviour or unittests. Behat would falsify that quality measurement.

But if you aren't using BDD or TDD you can reduce your effort trying to bring your Codecoverage to an high value, because you know, you are already testing certain lines.

But I also think Behat would make sense for testing Bugs and Edgecases as well. Especially for Bugs it would be nice creating the Gherkin description, implementing the Tests so the Bugreporter can see on the CI how far the bugfix allready is done.
And the other thing are the Edgecases. When a developer sees an edgecase and extends the Featurestory with this Scenario the PO can review this decision easily and at least recognize them, which he wouldn't if they would be written in phpunit.

[SvenBrunk]

For BDD you could actually use a code coverage report the other way round.
When you are finished with your behaviour tests and every requirement is covered... if you then find lines uncovered, they can most probably be removed. Else you have hidden requirements.

[everzet]

Which would mean you didn't do BDD in the first place. But I see your point. Coverage probably could be used for such sanity checks. That said, I think risk of abusing this functionality is still too high to include it in the core!

Sent from my iPhone

On 24 Oct 2013, at 07:45, SvenBrunk notifications@github.com wrote:

For BDD you could actually use a code coverage report the other way round.
When you are finished with your behaviour tests and every requirement is covered... if you then find lines uncovered, they can most probably be removed. Else you have hidden requirements.

—
Reply to this email directly or view it on GitHub.

[SvenBrunk]

I don't even know if it would work as something else but an additional tool. But it would need some interface to the core I guess. And of course one to the system beeing tested. (Which isn't necessarily the same as the one running the tests after all.)

[srcspider]

@everzet

That said, I think risk of abusing this functionality is still too high to include it in the core!

Doing nothing is still by far the worst option.

As long as you take control over the functionality you can clearly label it as "just a sanity check tool" so there's very little risk of it getting out of hand or causing misinformation. People that would abuse it (for whatever reason), would either abuse something similar anyway or not do anything at all.

[stof]

@srcspider but this is not easy to implement either. When specifying a web app with Behat, the code of the web app will often run separately (it is the case for all Mink drivers doing real HTTP requests to a webserver, i.e. all except the symfony2 driver). So collecting the code coverage in the Behat thread won't help (except for cleaning your FeatureContext from unused code).
And collecting the code coverage in your app requires modifying its source code, which cannot be done by Behat itself.

[srcspider]

I know, I've tried for the purpose of using it as a metric. =P

I'm just saying if the implementation were to come along it would be a
mistake to not add it and moderate it.

For what it's worth code coverage with unit tests is just as brittle, even
if you have everything green it doesn't mean you're actually testing every
possible case, just passing though the code with cases that don't cause
problems; at least a code converge with behat would more easily be
understood as just a "sanity check" since you don't make direct contact
with the implementation.

On Thu, Oct 24, 2013 at 3:24 PM, Christophe Coevoet <
notifications@github.com> wrote:

@srcspider https://github.com/srcspider but this is not easy to
implement either. When specifying a web app with Behat, the code of the web
app will often run separately (it is the case for all Mink drivers doing
real HTTP requests to a webserver, i.e. all except the symfony2 driver). So
collecting the code coverage in the Behat thread won't help (except for
cleaning your FeatureContext from unused code).
And collecting the code coverage in your app requires modifying its source
code, which cannot be done by Behat itself.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/92#issuecomment-26987533
.

[everzet]

Doing nothing is still by far the worst option.

You’re missing the point here. It is not Behat's responsibility to do something in this case. Behat has very specific and narrow use-case - to support BDD delivery by providing scenario automation tool. Making sure that you’re not writing useless code outside of the cycle is not a test automator job.

Konstantin Kudryashov
--- PHP developer, BDD evangelist, Scrum master

a: http://about.me/everzet
b: http://everzet.com
c: https://github.com/everzet

On 24 Oct 2013, at 12:50, srcspider notifications@github.com wrote:

That said, I think risk of abusing this functionality is still too high to include it in the core!

Doing nothing is still by far the worst option.

As long as you take control over the functionality you can clearly label it as "just a sanity check tool" so there's very little risk of it getting out of hand or causing misinformation. People that would abuse it (for whatever reason), would either abuse something similar anyway or not do anything at all.

—
Reply to this email directly or view it on GitHub.

[everzet]

I'm just saying if the implementation were to come along it would be a
mistake to not add it and moderate it.

I don’t see how not adding and, especially, not moderating a functionality that most users don’t need is a mistake :/

Feature you talk about is very narrow and has very specific use-case - you need it only if you’re using BDD tool without doing BDD. And I prefer tools to help people in fixing problems, not symptoms.

Konstantin Kudryashov
--- PHP developer, BDD evangelist, Scrum master

a: http://about.me/everzet
b: http://everzet.com
c: https://github.com/everzet

On 24 Oct 2013, at 13:31, srcspider notifications@github.com wrote:

I know, I've tried for the purpose of using it as a metric. =P

I'm just saying if the implementation were to come along it would be a
mistake to not add it and moderate it.

For what it's worth code coverage with unit tests is just as brittle, even
if you have everything green it doesn't mean you're actually testing every
possible case, just passing though the code with cases that don't cause
problems; at least a code converge with behat would more easily be
understood as just a "sanity check" since you don't make direct contact
with the implementation.

On Thu, Oct 24, 2013 at 3:24 PM, Christophe Coevoet <
notifications@github.com> wrote:

@srcspider https://github.com/srcspider but this is not easy to
implement either. When specifying a web app with Behat, the code of the web
app will often run separately (it is the case for all Mink drivers doing
real HTTP requests to a webserver, i.e. all except the symfony2 driver). So
collecting the code coverage in the Behat thread won't help (except for
cleaning your FeatureContext from unused code).
And collecting the code coverage in your app requires modifying its source
code, which cannot be done by Behat itself.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/92#issuecomment-26987533
.

—
Reply to this email directly or view it on GitHub.

[srcspider]

I don’t see how not adding and, especially, not moderating a functionality that most users don’t need is a mistake :/

There's obviously some interest in the feature, and "most users don't need" is really subject to the problem that "you don't really know you don't need something until you tried it" especially with checking/debugging/etc tools. If everything in the world was so perfect we shouldn't need profilers and debuggers, but we do. There's also the issue of "circumstances," if this "feature" helps even a little to produce better more reliable software I believe it's worth it.

If you don't want it that's that, my point with moderating was that a feature like this is likely to cause splits in the community, and while forks are cool, forks that are purely for the purpose of minor things like this that would obviously bother nobody that "doesn't need them" if they were in are just unnecessary splits in the community. Why do we need jslint vs jshint for example; we shouldn't.

Git, as an example, has a ton of tools not all of which might be considered "bare minimum," (in my opinion) and some of which are hidden away. Nobody seems to be bothered by their existence (assuming they are even aware of their existence), but I'm sure anyone who really needs them is thankful they are there.

[MarcelloDuarte]

@srcspider I too struggle to understand the value of code coverage (in general, but more so for Behat). Just why are you using Behat to hit code in the first place? Why not a unit test framework? Even if we accept that the metric is useful, it doesn't make any sense here, unless you are using this tool for something it was not designed to do in the first place.

Considering you are not using Behat to describe code, but to describe features (as it is designed to do), what value do you see having all the code lines used by the feature ticked green for "code" coverage. It is not even "code" coverage, is it? The code works within a context, not in ever context it is used. Even if you get 100% green lines you will not have any assurance of "code" or "feature" coverage with this feature request.

[srcspider]

@MarcelloDuarte I don't disagree with that view; I've pretty much expressed it several messages earlier.

[MarcelloDuarte]

@srcspider yes, I noticed that. Let's just leave it then. BDD was created as a training tool, anyway. Should the guys start coming up with the code coverage discussion again, we can always instruct them with a better perspective.

[cordoval]

for the case on phpspec i also thought this held, however someone respectable came up with an extension to do code coverage in phpspec, at first I thought it was a bad idea, but then i realized some use cases or in some flows it made sense. Not sure it is the case of behat, but who knows. Sometimes tools find different use cases, even if it is not the former one. So with care, let's say that is not the best idea for behat right now.

[srcspider]

@MarcelloDuarte it's hard to force a perspective on someone that is working under different circumstances to you. The loudest of us almost always point out the most idealistic circumstances, but that doesn't necessarily apply to all users. Some people have higher time constraints, work in smaller teams etc. Just like with choosing hardware there are different metrics and views. Personally even when I use code coverage I almost never look at it as more then a rough statistic (essentially no better then what you all are describing a behat coverage would be) that I use (among other things) to base a rough judgement on how stable/usable/etc my piece of code is. (I know that's now how everyone else does it; unless I'm mistaken a lot of people use it as a "I'm done" indicator.) Since for me it's just a statistic it's not like I have any "forgone conclusions about it," if it ends up being useless at times that's perfectly fine, because it's really cheap to do (or at least would be with a no-extra-knobs implementation).

[MarcelloDuarte]

@srcspider Yes! It is really cheap to implement this. There is nothing preventing a feature like that to be coded as an extension, like @everzet suggested. Plus Sebastian has nicely separated the code coverage component from PHPUnit, so it can be reused in situations like this. It's a really easy task to get an extension up and running for this. If someone has a use case for this, we can support with pointers on how to implement – probably better on IRC. In core we keep the tool a bit more focused, because it is also a goal of the BDD tools to educate users in the right way to do TDD.

[stof]

@MarcelloDuarte as I said above, implementing code coverage in Behat is not cheap. PhpSpec runs the code, so it is easy to write an extension for it collecting code coverage (and it has been done). Behat generally runs your app, not just your code. As the code ends up being run in a different PHP process, the code triggering the code coverage collection cannot be added in Behat. It needs to be added in the app (and then some code added in Behat to fetch the collected data and build a report for it).

[MarcelloDuarte]

@stof it would not be PhpSpec or PHPUnit collecting the code coverage data. xdebug does that. To use xdebug_start_code_coverage and xdebug_get_code_coverage in the same process that runs the application, the extension could interfere with the bootstrap of the application and make sure these methods decorate the run

[stof]

@MarcelloDuarte well, it is the PhpSpec extension which calls xdebug to trigger the coverage.

Decorating the bootstrap of the application to collect code coverage for Behat is not easy when testing a web app with Mink: unless you are faking the HTTP calls with the symfony2 driver (which works only for Symfony2 projects when you don't need JS), Mink will not call your app itself. It will call your HTTP server through Goutte or Selenium. So adding the xdebug coverage requires modifying the source code of the app itself to interfere with it (which is dangerous as it requires modifying versionned files, and requires to be implemented in an app-specific way). It becomes even worse when you consider that the HTTP server could run on a different machine (see the example of the doc running a behat feature against the wikipedia website).
This is why I'm saying it is not a cheap implementation (and probably also why nobody has implemented it yet as an extension)

[MarcelloDuarte]

@stof HTTP fake calls via framework drivers or a maybe just a separated cc collector component (not necessarily a Behat extension) configured to whitelist/blacklist directories that would be listening to the calls for the application – much like in the way of xdebug profiling with cache grind. Developers wouldn't have to touch the versioned files. This could be achieved this via configuration.

I am not saying this is natural, just not that complex.

[alwinmark]

+1 I like that... especially this could be a good sanity check for the feature scenarios. If they only cover a small code, they are maybe not precise enough...

Von Samsung Mobile gesendet

-------- Ursprüngliche Nachricht --------
Von: Marcello Duarte notifications@github.com
Datum: 24.10.2013 17:51 (GMT+01:00)
An: Behat/Behat Behat@noreply.github.com
Cc: CansaSCityShuffle alwin@atk-solutions.de
Betreff: Re: [Behat] Does Behat generate Code Coverage? (#92)

@stof HTTP fake calls via framework drivers or a maybe just a separated cc collector component (not necessarily a Behat extension) configured to whitelist/blacklist directories that would be listening to the calls for the application – much like in the way of xdebug profiling with cache grind. Developers wouldn't have to touch the versioned files. This could be achieved this via configuration.

I am not saying this is natural, just not that complex.

—
Reply to this email directly or view it on GitHub.

[robocoder]

https://github.com/vipsoft/code-coverage-extension is a prototype. (I'll add filtering and xcache coverager support later this week.)

Feedback and PRs appreciated.

[gggeek]

I will try to refrain from the test-purist vs. test-pragmatist approach discussion, but I definitely see some value in being able to get code coverage out of behat tests:

• using only 1 testing framework instead of 2 or 3 has a lot of value, esp. when you have to teach all team members how to write tests, or when the set of configs to run your tests against is complex enough that you have to write your test-executor-framework on top of the chosen test-framework
• even if believing strongly in code coverage is bad, looking at the reports as ballpark-figures can be enlightening: finding scenarios you forgot to write, changes between spec time and implementation time leading to huge chunks of dead code etc...
• sometimes you might end up in a project where only behaviour tests have been written and not unit tests, for any reason outside of your control (lack of resources, anyone?)
• when developing and testing a REST API, the diving line between unit test and functional test gets much thinner and you do not have selenium acting as a middleman. Sending to the API tested the magical cookie which starts xdebug code coverage and retrieving it for treatment is not so hard

I short, I do not advocate adding it by default in behat, but I vote plus one for documenting how it is possible, or having a behat extension making it easier.

[SvenBrunk]

Exactly. For that I would even like to have it down to line coverage (or more path coverage, so I know what logical paths were taken and which not)

[stof]

@gggeek there is already an extension doing this: https://packagist.org/packages/vipsoft/code-coverage-extension

[gggeek]

@stof thanks. It is not very clear from the docs if that only works for Sf, though, or if its more generic