After upgrading from PHPUnit 8 to 9, the number of tests exploded

[validaide-mark-bijl]

Hi!

Well, first of all, thanks for a great library! I got introduced to it after we adopted a no-longer maintained OS library (https://github.com/egeloen/ivory-google-map). As that library is quite outdated, we've been working step by step to upgrade its dependencies. This time we wanted to upgrade PHPUnit and PHPUnit Selenium from 8 to 9. However, after we did so the number of tests went from 1810 to 15722.

BEFORE:

PHPUnit 8.5.31 by Sebastian Bergmann and contributors.

.............................................................   61 / 1810 (  3%)

AFTER:

PHPUnit 9.5.26 by Sebastian Bergmann and contributors.

Warning:       Your XML configuration validates against a deprecated schema.
Suggestion:    Migrate your XML configuration using "--migrate-configuration"!

...........................................................    59 / 15722 (  0%)

When I ran it verbose, I noticed things like these:

33) Ivory\Tests\GoogleMap\Helper\Functional\CompoundFunctionalTest::__construct
The Selenium Server is not active on host selenium at port 4444.

/Users/markbijl/projects/google-map/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:345
/Users/markbijl/projects/google-map/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:384
/Users/markbijl/projects/google-map/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:360
phpvfscomposer:///Users/markbijl/projects/google-map/vendor/phpunit/phpunit/phpunit:97

Notice the test CompoundFunctionalTest::__construct where the class 'CompoundFunctionalTest' extends the Selenium2TestCase. But... it seems like the constructor is identified as a test, same for every other method of the test classes that extend the Selenium2TestCase.

Basically, it seems like every method of every test class is being identified as a test case. Now I admit that my knowledge of this whole library is limited (getting to know it better every day) and I am still trawling through the PHPUnit setup thinking that perhaps there is some kind of magic happening that is causing every class method to be identified as a test, but... I also though to check here to see if perhaps this is something related to the upgrade of PHPUnit Selenium from 8 to 9.

Any tips, hints, feedback... all much appreciated!

Thanks,
Mark

p.s. this is our fork: https://github.com/validaide/google-map

[paprykasz]

+1
After quick research it seems that fix is included in master 1e9ba60 but not released. I was looking for some quick port but library seems abandoned.

[spiegelm]

I know this is an old issue but I came to the same conclusion. It looks like the commits for the latest release 9.0.1 got confused. So the release 9.0.1 on packagist.org references an old commit . In case somebody needs the latest commit that resolves this issue, using composer, adjust composer.json to match:

"require-dev": {
-        "phpunit/phpunit-selenium": "^9.0.1",
+        "phpunit/phpunit-selenium": "dev-master#b40dc292627e5de5bd8d32570627a6d55b2b846c",
}

Then run composer update phpunit/phpunit-selenium.