severe performance regression after upgrading to 4.0.7 #1187
Comments
|
PHPUnit-3.7.32: OK, but incomplete or skipped tests! PHPUnit-4.0.7: OK, but incomplete, skipped, or risky tests! Command: Notice: I have xdebug disabled and don't want to have code coverage in the normal daily work. With 3.7. PHPUnit wasn't even trying to generate a code coverage report. Now it seems, that he always tries to generate it and gives the normal warning that xdebug has to be enabled. "The Xdebug extension is not loaded. No code coverage will be generated." |
|
Thank you, @cobexer and @dreis2211 for bringing this to my attention. I am investigating the issue right now. |
|
I do have xdebug installed and want to collect coverage (which works fine, just slower). |
|
@dreis2211 PHPUnit was not trying to generate a code coverage report. It just erroneously displayed the "The Xdebug extension is not loaded. No code coverage will be generated." message. This was fixed by 4db4ff9 |
|
I am unable to reproduce a severe regression in performance from PHPUnit 3.7 to PHPUnit 4.0. Without access to a test suite that exhibits the performance regression I cannot investigate and fix the issue. The example project I usedThe PHP version I usedPHPUnit 3.7PHPUnit 4.0 |
|
This is awkward: I was unable to reproduce the performance regression because I have apparently fixed it already since PHPUnit 4.0.8. Because if I run the example test suite with that version I get I think we can close this issue now. |
|
Nevermind. The run with PHPUnit 4.0.8 shown in #1187 (comment) was with Xdebug enabled whereas the other ones were with Xdebug disabled. This means that what I wroted in #1187 (comment) is still valid: without access to a test suite that exhibits the performance regression I cannot investigate and fix the issue. |
|
the project that I have the issue with is: https://github.com/cobexer/templateengine2 |
|
Thank you, @cobexer, for a test suite to reproduce the issue with. The example project I usedThe PHP version I usedPHPUnit 3.7PHPUnit 4.0The only apparent difference between the test suite I used earlier and yours is that your uses process isolation. I will investigate the way that code coverage data is passed from the isolated process to the main process as this is the most likely candiate for a cause for the performance regression. BTW: I was not able to reproduce a performance regression using @cobexer's test suite when no code coverage report is requested. |
|
you probably need a slow netbook to reproduce ;) my main computer also executes the testsuite far faster ;) I could check with my netbook without coverage and show you the numbers on my way back home today. @sebastianbergmann could you check the slowdown with a phpunit build that is not inside a phar (or compressed with a lower compression level)? maybe thats the issue |
|
I did not check with a PHAR but with a Git checkout of PHPUnit. |
|
The numbers from @dreis2211 above are also without code coverage. A lot of the tests are integration tests and involve database fixtures, maybe that made the performance difference more obvious. |
|
@sebastianbergmann Some numbers for imbo/imbo: To test with 3.7: To test with 4.0: |
|
Here's the result of some memory profiling I did with PHPUnit 4.0 and @christeredvartsen's test suite: |
|
We are experiencing a similar increase in runtime and memory usage. From ~15 seconds to ~1 minute and a memory exhaustion error. |
|
@lstrojny Any chance of getting profiling information? |
|
I am experiencing similar performance issues with PHPUnit 4. Unfortunately I can only provide some facts from my test runs, because the library I am working on is not publicly available - but hopefully my results will help finding the bugs nevertheless. (I guess there are at least 2 bugs or design decisions causing bad performance! See below.) All my tests are properly isolated, pure unit tests:
It should be mentioned that
In addition to the results listed below, I noticed the following conspicuous behavior patterns:
Here are the average execution times in seconds: Test suite 1
Test suite 2
Test suite 3
|
|
Reverting sebastianbergmann/phpunit-mock-objects@ee8b777 and d1efb51 reduces the memory footprint from 303.25Mb to 301.75Mb on my machine for PHPUnit 4.0 and @christeredvartsen's test suite. |
<?php
class Foo
{
public function bar($a, $b, $c)
{
}
}
class Test extends PHPUnit_Framework_TestCase
{
public function testOne()
{
$this->getMock('Foo');
}
}When I run the test above with Seems like something related to mock objects got slower. |
|
Unfortunately I couldn't get my XHProf running on my Windows machine with PHP 5.5 with the needed cpu and mem usage flags, but I was able to generate at least some reports, which would confirm a problem somewhere in the creation of mock objects. Below listed the top 10 functions with PHPUnit 3.7.28 and 4.0 PHPUnit 3.7.28 PHPUnit 4.0.9 |
|
@dreis2211 What is |
|
@sebastianbergmann This is part of our class loading part in our project. Interesting side-note: This shouldn't be in there. In fact, this happens when he tries to load a class that doesn't exist. In that case he tries to load a test that is using a data provider. Class he tries to load: |
|
@sebastianbergmann I don't have too much time to look into this at the moment, but is it possible that sebastianbergmann/phpunit-mock-objects@fd2951c makes the mock object cache less useful? |
|
@sebastianbergmann I used your example and added the data provider mechanism to it. Here are the results <?php
class Foo
{
public function bar($a, $b, $c)
{
}
}
class Test extends PHPUnit_Framework_TestCase
{
/**
* @dataProvider testData
*/
public function testOne($test)
{
$this->getMock('Foo');
}
public function testData() {
$array = array();
for ($i = 0; $i < 10000; $i++) {
$array[] = array('test');
}
return $array;
}
}PHPUnit 3.7.32 PHPUnit 4.0.9 |
|
@dreis2211 Thanks! |
|
@whatthejeff That would at least make sense. I'll investigate. |
|
@sebastianbergmann I can confirm that making the cache static fixes the data provider issue posted by @dreis2211. |
|
@dreis2211 Can you check whether sebastianbergmann/phpunit-mock-objects@3ba8622 fixes your issue? |
|
@sebastianbergmann Seems to be better. With my example it still uses 2-3 times more memory than with 3.7.32, but in regards of time it seems okay. Also our real project seems to be back to normal values. PHPUnit4.0.9 PHPUnit3.7.32 |
|
@sebastianbergmann That patch seems to fix the issues in my test suite at least: 4.0.9 (with patch) 3.7.32 4.0.9 (with patch) with code coverage 3.7.32 with code coverage |
|
Closing this issue now. If there are other performance regressions that still exist in PHPUnit 4.0.10 please open a new ticket. Thanks! Thank you, @cobexer, @dreis2211, and @christeredvartsen, @HenryPrang, @lstrojny for your extensive feedback that helped track down the memory usage regression. Thank you, @whatthejeff for figuring out the root cause. |
|
No problem :) |
ghost
mentioned this issue
PHPUnit-3.7.28:
real 1m2.351s
user 0m53.526s
sys 0m8.633s
PHPUnit-4.0.7:
real 5m27.942s
user 4m22.707s
sys 1m4.248s
config parameters:
backupGlobals="false"
backupStaticAttributes="false"
processIsolation="true"
verbose="false"
command:
phpunit -c phpunit.xml --tap --coverage-html
normal test execution (without --coverage-html) also got a little slower, but not that much.
The text was updated successfully, but these errors were encountered: