phpunit installed via composer leads to "No tests found" warning for PHPUnit Framework internal class(es) when executing a folder containing more test files/classes and filtering (via exclude-group) a whole test class with (a) @group annotation on test class and (b) a data provider throwing an exception

[reinholdfuereder]

Q	A
PHPUnit version	6.5.x
PHP version	7.0.26
Installation Method	Composer

It works fine when using:

• Phar
• PHPUnit 6.4.4 (and older)

It started failing when upgrading from PHPUnit 6.4.3 to 6.5.4; after rolling back it seemed to have broken with 6.5.0 already; I am naively strongly suspecting the changes in non-released 6.4.5 (that presumably are part of 6.5.0) are the culprits...

How to reproduce?

A folder with e.g. a normal test ("unfilteredTest.php") and the bad/special one ("bugTest.php") -- just the bad/special one is not (!) sufficient:

jenkins@***:/***/workspace$ ll unittests/unit/bug/
total 16
drwxrwxr-x 2 jenkins unix-user 4096 Dec 15 07:57 ./
drwxr-xr-x 6 jenkins unix-user 4096 Dec 15 07:16 ../
-rw-r--r-- 1 jenkins unix-user  384 Dec 15 07:46 bugTest.php
-rw-r--r-- 1 jenkins unix-user  186 Dec 15 07:46 unfilteredTest.php

The normal test ("unfilteredTest.php"):

jenkins@***:/***/workspace$ cat unittests/unit/bug/unfilteredTest.php
<?php

class UnfilteredTest extends PHPUnit\Framework\TestCase {

  /**
   * @test
   */
  public function test() {
    \PHPUnit\Framework\Assert::assertEquals(0, 0);
  }

}

And the bad/special test ("bugTest.php"):

jenkins@***:/***/workspace$ cat unittests/unit/bug/bugTest.php
<?php

/**
 * @group integrationtest
 */
class BugTest extends PHPUnit\Framework\TestCase {

  public function provider() {
    throw new Exception('any exception');
    return [
      [0],
      [1]
    ];
  }

  /**
   * @dataProvider provider
   * @test
   */
  public function test($int) {
    \PHPUnit\Framework\Assert::fail("called with {$int}");
  }

}

When using 'phpunit.phar' it works fine: BugTest is correctly excluded

jenkins@***:/***/workspace$ ~/phpunit.phar --debug --exclude-group integrationtest --include-path . ./unittests/unit/bug
PHPUnit 6.5.0 by Sebastian Bergmann and contributors.

Test 'UnfilteredTest::test' started
Test 'UnfilteredTest::test' ended


Time: 118 ms, Memory: 10.00MB

OK (1 test, 1 assertion)

But when using 'phpunit' from composer it fails "funnily":

jenkins@***:/***/workspace$ phpunit --debug --exclude-group integrationtest --include-path . ./unittests/unit/bug
PHPUnit 6.5.0 by Sebastian Bergmann and contributors.

Test 'Warning' started
Test 'Warning' ended
Test 'UnfilteredTest::test' started
Test 'UnfilteredTest::test' ended


Time: 56 ms, Memory: 4.00MB

There was 1 warning:

1) Warning
No tests found in class "PHPUnit\Framework\WarningTestCase".

WARNINGS!
Tests: 2, Assertions: 1, Warnings: 1.

Now some more details explaining the special circumstances and the long issue title: When there is just the bad/special one:

jenkins@***:/***/workspace$ mv unittests/unit/bug/unfilteredTest.php unittests/unit/bug/unfilteredTest_.php

jenkins@***:/***/workspace$ ll unittests/unit/bug/
total 16
drwxrwxr-x 2 jenkins unix-user 4096 Dec 15 08:06 ./
drwxr-xr-x 6 jenkins unix-user 4096 Dec 15 07:16 ../
-rw-r--r-- 1 jenkins unix-user  384 Dec 15 07:46 bugTest.php
-rw-r--r-- 1 jenkins unix-user  186 Dec 15 07:46 unfilteredTest_.php

jenkins@***:/***/workspace$ phpunit --debug --exclude-group integrationtest --include-path . ./unittests/unit/bug
PHPUnit 6.5.0 by Sebastian Bergmann and contributors.



Time: 29 ms, Memory: 4.00MB

No tests executed!

jenkins@***:/***/workspace$ ~/phpunit.phar --debug --exclude-group integrationtest --include-path . ./unittests/unit/bug
PHPUnit 6.5.0 by Sebastian Bergmann and contributors.



Time: 111 ms, Memory: 10.00MB

No tests executed!

When reverting the file renaming it works again:

jenkins@***:/***/workspace$ mv unittests/unit/bug/unfilteredTest_.php unittests/unit/bug/unfilteredTest.php

jenkins@***:/***/workspace$ ll unittests/unit/bug/
total 16
drwxrwxr-x 2 jenkins unix-user 4096 Dec 15 08:06 ./
drwxr-xr-x 6 jenkins unix-user 4096 Dec 15 07:16 ../
-rw-r--r-- 1 jenkins unix-user  384 Dec 15 07:46 bugTest.php
-rw-r--r-- 1 jenkins unix-user  186 Dec 15 07:46 unfilteredTest.php

jenkins@***:/***/workspace$ phpunit --debug --exclude-group integrationtest --include-path . ./unittests/unit/bug
PHPUnit 6.5.0 by Sebastian Bergmann and contributors.

Test 'Warning' started
Test 'Warning' ended
Test 'UnfilteredTest::test' started
Test 'UnfilteredTest::test' ended


Time: 50 ms, Memory: 4.00MB

There was 1 warning:

1) Warning
No tests found in class "PHPUnit\Framework\WarningTestCase".

WARNINGS!
Tests: 2, Assertions: 1, Warnings: 1.

jenkins@***:/***/workspace$ ~/phpunit.phar --debug --exclude-group integrationtest --include-path . ./unittests/unit/bug
PHPUnit 6.5.0 by Sebastian Bergmann and contributors.

Test 'UnfilteredTest::test' started
Test 'UnfilteredTest::test' ended


Time: 114 ms, Memory: 10.00MB

OK (1 test, 1 assertion)

The same "funny" problem for SkippedTestCase

Please note that we also have the corresponding problem with respect to SkippedTestCase, but I have not spent the same enormous amount of time to investigate that problem too, as I claim it must be strongly related with the one above for WarningTestCase.

PHPUnit 6.5.0 by Sebastian Bergmann and contributors.

...............................................................  63 / 489 ( 12%)
............................................................... 126 / 489 ( 25%)
............................................................... 189 / 489 ( 38%)
............................................................... 252 / 489 ( 51%)
............................................................... 315 / 489 ( 64%)
.............W................................................. 378 / 489 ( 77%)
W.............................................................. 441 / 489 ( 90%)
................................................                489 / 489 (100%)

Time: 26.56 seconds, Memory: 82.00MB

There were 2 warnings:

1) Warning
No tests found in class "PHPUnit\Framework\WarningTestCase".

2) Warning
No tests found in class "PHPUnit\Framework\SkippedTestCase".

WARNINGS!
Tests: 489, Assertions: 930, Warnings: 2.

[sebastianbergmann]

Which version of php-file-iterator was installed?

[sebastianbergmann]

There are no code changes in the 6.4 branch that are unreleased.

[reinholdfuereder]

Concerning code changes in 6.4 branch I refer to 6.4.5 from https://github.com/sebastianbergmann/phpunit/blob/6.4/ChangeLog-6.4.md

[reinholdfuereder]

Concerning php-file-iterator version: "1.4.5" (like allegedly also in the Phar)

[sebastianbergmann]

Sorry, I should have done git diff 6.4.4...origin/6.4 instead of git diff 6.4.4...6.4. My checkout of the 6.4 branch on this box was outdated.

[sebastianbergmann]

@kubawerlos Could this be related to #2833?

[kubawerlos]

@sebastianbergmann yes, it is.

The problem occurs only if ALL these conditions are met:

• there is a test class with @group and provider throwing exception in it
• tests are ran with --exclude-group for that group
• there is another class called later (after handing the class from above)
• the name of that another class does not match it's file name

Fix on the way.

[reinholdfuereder]

Thanks a lot for your quick replies and resolution and new release!

Two small (?) questions:

1. Is the 4th necessary condition mentioned by you ("the name of that another class does not match it's file name") just because of the different case ("unfilteredTest.php" vs. "UnfilteredTest")?
2. Will that also solve the presumably similar/related problem concerning "SkippedTestCase" (that I have not investigated anymore)?

[kubawerlos]

@reinholdfuereder

1. Yes, you can check it out by cloning phpunit and running test, previously commenting out this line - if you "fix" the name of this class all will be fine.
2. Yes, the fix will make all PHPUnit\Framework to be excluded.

[reinholdfuereder]

Thanks again, I can confirm that your fix helped as promised/expected and solved both problems 👍