Group exclusions are ignored on folder level #358

Closed
juriansluiman opened this Issue Sep 28, 2011 · 7 comments

Comments

Projects
None yet
4 participants

In both the phpunit.xml declarations as for the command line arguments, my excluded test are still running. This happens when I try to run a directory of tests, a single file works fine.

So assume I have this phpunit.xml:

<phpunit bootstrap="./Bootstrap.php" colors="true">
    <groups>
        <exclude>
            <group>disable</group>
        </exclude>
    </groups>
</phpunit>

And a testcase which looks like this

/**
 * @group disable
 */
class SomeTest extends \PHPUnit_Framework_TestCase
{    
    public function testMethod()
    {
        $this->assertEquals(true, true);
    }
}

And assume the testcase is loaded fine by the autoloader, then the following occurs:

This works (ie. test does NOT run):

  • $ phpunit path/To/SomeTest.php
  • $ phpunit -c phpunit.xml path/To/SomeTest.php
  • $ phpunit --exclude-group disable path/To/SomeTest.php

This does not work (ie test DOES run):

  • $ phpunit path/To/
  • $ phpunit -c phpunit.xml path/To/
  • $ phpunit --exclude-group disable path/To/

This gave me the following conclusion:

  1. Disabling a testcase is working, because:
  2. Try to run that specific case by providing the path to the file containing the class works
  3. But run (multiple) cases by providing the path to a directory is not working

To complete the info, I use phpunit 3.5.15 which is installed through pear. Though this bug report is a simplified version, this problem occurs with the current zf2 code base (https://github.com/zendframework/zf2). Users with 3.5.14 didn't have this problem, they told me all was working fine.

Collaborator

edorian commented Nov 17, 2011

It seems this has been fixed. My tests for this work fine in 3.6.3 just using phpunit and phpunit tests/otherFolder.

My phpunit.xml looks like the linked one so i think this is solved.

Feel free to reopen if the problem persists for you. Maybe with the complete reproduce code including the folder structure :)

edorian closed this Nov 17, 2011

We faced the same issue. OS: Ubuntu 11.10. PHPUnit installed using apt-get.

$ phpunit --version
PHPUnit 3.6.4 by Sebastian Bergmann.

$ uname -a
Linux shirjaev 2.6.38-13-generic #53-Ubuntu SMP Mon Nov 28 19:33:45 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

I tried PHPUnit installed by PEAR. The problem is still present.

$ phpunit --version
PHPUnit 3.6.10 by Sebastian Bergmann.

Collaborator

edorian commented Apr 3, 2012

Like stated without a reproduce case there is nothing much I can do

We were having the same "problem" here, so we created a repository to try to reproduce the "bug": https://github.com/diogobaeder/phpunit-bug-358

But, actually, /eriksencosta discovered that the tests excluded by "--exclude-group" are not being run, which is the expected behaviour, but the number of tests being shown at each pointed line end is not consistent with the total number of tests which is printed at the execution.

For example, this is the output of the complete test suite of our internal project:

PHPUnit 3.6.10 by Sebastian Bergmann.

............................................................. 61 / 1505 ( 4%)
....................................S.S...................... 122 / 1505 ( 8%)
............................................................. 183 / 1505 ( 12%)
.......................................................I..... 244 / 1505 ( 16%)
............................................................. 305 / 1505 ( 20%)
............................................................. 366 / 1505 ( 24%)
............................................................. 427 / 1505 ( 28%)
............................................................. 488 / 1505 ( 32%)
............................................................. 549 / 1505 ( 36%)
............................................................. 610 / 1505 ( 40%)
.............SS.SS....S...................................... 671 / 1505 ( 44%)
............................................................. 732 / 1505 ( 48%)
............................................................. 793 / 1505 ( 52%)
............................................................. 854 / 1505 ( 56%)
............................................................. 915 / 1505 ( 60%)
............................................................. 976 / 1505 ( 64%)
............................................................. 1037 / 1505 ( 68%)
............................................................. 1098 / 1505 ( 72%)
............................................................. 1159 / 1505 ( 77%)
............................................................. 1220 / 1505 ( 81%)
............................................................. 1281 / 1505 ( 85%)
................................................I............ 1342 / 1505 ( 89%)
............................................................. 1403 / 1505 ( 93%)
.....S....................................................... 1464 / 1505 ( 97%)
........................

Time: 03:16, Memory: 503.75Mb

OK, but incomplete or skipped tests!
Tests: 1488, Assertions: 3444, Incomplete: 2, Skipped: 8.

Therefore I think this is not properly a "bug", but a bad communication about the number of tests, which can be improved if we show the correct number ("1488" instead of "1505", at the case above)

I am not sure what it was, because I did not conclude this purely on statistics. In the Zend Framework test suite some tests grouped under @group disable were throwing exceptions, which remained when I excluded this "disable" group.

However, with above repository I forked I cannot reproduce this anymore. Meanwhile I am on phpunit 3.6.10 so this can stay closed, perhaps it was a bug but solved between 3.5.15 and 3.6.10 :)

Collaborator

edorian commented Apr 10, 2012

The test count numbers can be off. That happens because with the current test runner not all the filters (like exclude group) are applied before the tests are started but they should not be executed.

If it works for you with 3.6.10 thats great :)

Thanks for the reproduce @diogobaeder.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment