Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

severe performance regression after upgrading to 4.0.7 #1187

Closed
cobexer opened this issue Mar 16, 2014 · 32 comments

Comments

@cobexer
Copy link

commented Mar 16, 2014

PHPUnit-3.7.28:
real 1m2.351s
user 0m53.526s
sys 0m8.633s

PHPUnit-4.0.7:
real 5m27.942s
user 4m22.707s
sys 1m4.248s

config parameters:
backupGlobals="false"
backupStaticAttributes="false"
processIsolation="true"
verbose="false"
command:
phpunit -c phpunit.xml --tap --coverage-html

normal test execution (without --coverage-html) also got a little slower, but not that much.

@dreis2211

This comment has been minimized.

Copy link

commented Mar 17, 2014

PHPUnit-3.7.32:
Time: 6.52 minutes, Memory: 417.00Mb

OK, but incomplete or skipped tests!
Tests: 4601, Assertions: 8661, Incomplete: 95, Skipped: 17.

PHPUnit-4.0.7:
Time: 8.94 minutes, Memory: 738.25Mb

OK, but incomplete, skipped, or risky tests!
Tests: 4601, Assertions: 8661, Incomplete: 95, Skipped: 17.

Command:
phpunit --bootstrap unit_bootstrap.php AllTests.php

Notice: I have xdebug disabled and don't want to have code coverage in the normal daily work. With 3.7. PHPUnit wasn't even trying to generate a code coverage report. Now it seems, that he always tries to generate it and gives the normal warning that xdebug has to be enabled.

"The Xdebug extension is not loaded. No code coverage will be generated."

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 17, 2014

Thank you, @cobexer and @dreis2211 for bringing this to my attention. I am investigating the issue right now.

@cobexer

This comment has been minimized.

Copy link
Author

commented Mar 17, 2014

I do have xdebug installed and want to collect coverage (which works fine, just slower).

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 17, 2014

@dreis2211 PHPUnit was not trying to generate a code coverage report. It just erroneously displayed the "The Xdebug extension is not loaded. No code coverage will be generated." message.

This was fixed by 4db4ff9

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 17, 2014

I am unable to reproduce a severe regression in performance from PHPUnit 3.7 to PHPUnit 4.0. Without access to a test suite that exhibits the performance regression I cannot investigate and fix the issue.

The example project I used

$ git clone https://github.com/sebastianbergmann/money.git
$ cd money

The PHP version I used

$ php -v
PHP 5.5.10 (cli) (built: Mar  6 2014 06:34:18) 
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies

PHPUnit 3.7

phpunit --bootstrap src/autoload.php --repeat 100 tests
PHPUnit 3.7.32-3-g4dbc3ab by Sebastian Bergmann.

.............................................................   61 / 3100 (  1%)
.............................................................  122 / 3100 (  3%)
.............................................................  183 / 3100 (  5%)
.............................................................  244 / 3100 (  7%)
.............................................................  305 / 3100 (  9%)
.............................................................  366 / 3100 ( 11%)
.............................................................  427 / 3100 ( 13%)
.............................................................  488 / 3100 ( 15%)
.............................................................  549 / 3100 ( 17%)
.............................................................  610 / 3100 ( 19%)
.............................................................  671 / 3100 ( 21%)
.............................................................  732 / 3100 ( 23%)
.............................................................  793 / 3100 ( 25%)
.............................................................  854 / 3100 ( 27%)
.............................................................  915 / 3100 ( 29%)
.............................................................  976 / 3100 ( 31%)
............................................................. 1037 / 3100 ( 33%)
............................................................. 1098 / 3100 ( 35%)
............................................................. 1159 / 3100 ( 37%)
............................................................. 1220 / 3100 ( 39%)
............................................................. 1281 / 3100 ( 41%)
............................................................. 1342 / 3100 ( 43%)
............................................................. 1403 / 3100 ( 45%)
............................................................. 1464 / 3100 ( 47%)
............................................................. 1525 / 3100 ( 49%)
............................................................. 1586 / 3100 ( 51%)
............................................................. 1647 / 3100 ( 53%)
............................................................. 1708 / 3100 ( 55%)
............................................................. 1769 / 3100 ( 57%)
............................................................. 1830 / 3100 ( 59%)
............................................................. 1891 / 3100 ( 61%)
............................................................. 1952 / 3100 ( 62%)
............................................................. 2013 / 3100 ( 64%)
............................................................. 2074 / 3100 ( 66%)
............................................................. 2135 / 3100 ( 68%)
............................................................. 2196 / 3100 ( 70%)
............................................................. 2257 / 3100 ( 72%)
............................................................. 2318 / 3100 ( 74%)
............................................................. 2379 / 3100 ( 76%)
............................................................. 2440 / 3100 ( 78%)
............................................................. 2501 / 3100 ( 80%)
............................................................. 2562 / 3100 ( 82%)
............................................................. 2623 / 3100 ( 84%)
............................................................. 2684 / 3100 ( 86%)
............................................................. 2745 / 3100 ( 88%)
............................................................. 2806 / 3100 ( 90%)
............................................................. 2867 / 3100 ( 92%)
............................................................. 2928 / 3100 ( 94%)
............................................................. 2989 / 3100 ( 96%)
............................................................. 3050 / 3100 ( 98%)
..................................................

Time: 2.23 seconds, Memory: 3.50Mb

OK (3100 tests, 5000 assertions)

PHPUnit 4.0

$ phpunit --bootstrap src/autoload.php --repeat 100 tests
PHPUnit 4.0.8-5-g96cda9f by Sebastian Bergmann.

.............................................................   61 / 3100 (  1%)
.............................................................  122 / 3100 (  3%)
.............................................................  183 / 3100 (  5%)
.............................................................  244 / 3100 (  7%)
.............................................................  305 / 3100 (  9%)
.............................................................  366 / 3100 ( 11%)
.............................................................  427 / 3100 ( 13%)
.............................................................  488 / 3100 ( 15%)
.............................................................  549 / 3100 ( 17%)
.............................................................  610 / 3100 ( 19%)
.............................................................  671 / 3100 ( 21%)
.............................................................  732 / 3100 ( 23%)
.............................................................  793 / 3100 ( 25%)
.............................................................  854 / 3100 ( 27%)
.............................................................  915 / 3100 ( 29%)
.............................................................  976 / 3100 ( 31%)
............................................................. 1037 / 3100 ( 33%)
............................................................. 1098 / 3100 ( 35%)
............................................................. 1159 / 3100 ( 37%)
............................................................. 1220 / 3100 ( 39%)
............................................................. 1281 / 3100 ( 41%)
............................................................. 1342 / 3100 ( 43%)
............................................................. 1403 / 3100 ( 45%)
............................................................. 1464 / 3100 ( 47%)
............................................................. 1525 / 3100 ( 49%)
............................................................. 1586 / 3100 ( 51%)
............................................................. 1647 / 3100 ( 53%)
............................................................. 1708 / 3100 ( 55%)
............................................................. 1769 / 3100 ( 57%)
............................................................. 1830 / 3100 ( 59%)
............................................................. 1891 / 3100 ( 61%)
............................................................. 1952 / 3100 ( 62%)
............................................................. 2013 / 3100 ( 64%)
............................................................. 2074 / 3100 ( 66%)
............................................................. 2135 / 3100 ( 68%)
............................................................. 2196 / 3100 ( 70%)
............................................................. 2257 / 3100 ( 72%)
............................................................. 2318 / 3100 ( 74%)
............................................................. 2379 / 3100 ( 76%)
............................................................. 2440 / 3100 ( 78%)
............................................................. 2501 / 3100 ( 80%)
............................................................. 2562 / 3100 ( 82%)
............................................................. 2623 / 3100 ( 84%)
............................................................. 2684 / 3100 ( 86%)
............................................................. 2745 / 3100 ( 88%)
............................................................. 2806 / 3100 ( 90%)
............................................................. 2867 / 3100 ( 92%)
............................................................. 2928 / 3100 ( 94%)
............................................................. 2989 / 3100 ( 96%)
............................................................. 3050 / 3100 ( 98%)
..................................................

Time: 2.01 seconds, Memory: 3.75Mb

OK (3100 tests, 5000 assertions)
@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 17, 2014

This is awkward: I was unable to reproduce the performance regression because I have apparently fixed it already since PHPUnit 4.0.8. Because if I run the example test suite with that version I get

$ phpunit --bootstrap src/autoload.php --repeat 100 tests                           
PHPUnit 4.0.8 by Sebastian Bergmann.

.............................................................   61 / 3100 (  1%)
.............................................................  122 / 3100 (  3%)
.............................................................  183 / 3100 (  5%)
.............................................................  244 / 3100 (  7%)
.............................................................  305 / 3100 (  9%)
.............................................................  366 / 3100 ( 11%)
.............................................................  427 / 3100 ( 13%)
.............................................................  488 / 3100 ( 15%)
.............................................................  549 / 3100 ( 17%)
.............................................................  610 / 3100 ( 19%)
.............................................................  671 / 3100 ( 21%)
.............................................................  732 / 3100 ( 23%)
.............................................................  793 / 3100 ( 25%)
.............................................................  854 / 3100 ( 27%)
.............................................................  915 / 3100 ( 29%)
.............................................................  976 / 3100 ( 31%)
............................................................. 1037 / 3100 ( 33%)
............................................................. 1098 / 3100 ( 35%)
............................................................. 1159 / 3100 ( 37%)
............................................................. 1220 / 3100 ( 39%)
............................................................. 1281 / 3100 ( 41%)
............................................................. 1342 / 3100 ( 43%)
............................................................. 1403 / 3100 ( 45%)
............................................................. 1464 / 3100 ( 47%)
............................................................. 1525 / 3100 ( 49%)
............................................................. 1586 / 3100 ( 51%)
............................................................. 1647 / 3100 ( 53%)
............................................................. 1708 / 3100 ( 55%)
............................................................. 1769 / 3100 ( 57%)
............................................................. 1830 / 3100 ( 59%)
............................................................. 1891 / 3100 ( 61%)
............................................................. 1952 / 3100 ( 62%)
............................................................. 2013 / 3100 ( 64%)
............................................................. 2074 / 3100 ( 66%)
............................................................. 2135 / 3100 ( 68%)
............................................................. 2196 / 3100 ( 70%)
............................................................. 2257 / 3100 ( 72%)
............................................................. 2318 / 3100 ( 74%)
............................................................. 2379 / 3100 ( 76%)
............................................................. 2440 / 3100 ( 78%)
............................................................. 2501 / 3100 ( 80%)
............................................................. 2562 / 3100 ( 82%)
............................................................. 2623 / 3100 ( 84%)
............................................................. 2684 / 3100 ( 86%)
............................................................. 2745 / 3100 ( 88%)
............................................................. 2806 / 3100 ( 90%)
............................................................. 2867 / 3100 ( 92%)
............................................................. 2928 / 3100 ( 94%)
............................................................. 2989 / 3100 ( 96%)
............................................................. 3050 / 3100 ( 98%)
..................................................

Time: 15.46 seconds, Memory: 4.25Mb

OK (3100 tests, 5000 assertions)

I think we can close this issue now.

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 17, 2014

Nevermind. The run with PHPUnit 4.0.8 shown in #1187 (comment) was with Xdebug enabled whereas the other ones were with Xdebug disabled.

This means that what I wroted in #1187 (comment) is still valid: without access to a test suite that exhibits the performance regression I cannot investigate and fix the issue.

@cobexer

This comment has been minimized.

Copy link
Author

commented Mar 17, 2014

the project that I have the issue with is: https://github.com/cobexer/templateengine2
you probably need to comment out line 14 of tests/TemplateEngineTestBase.php.
then simply run phpunit -c phpunit.xml --tap --coverage-html

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 17, 2014

Thank you, @cobexer, for a test suite to reproduce the issue with.

The example project I used

$ git clone git@github.com:cobexer/templateengine2.git
$ cd templateengine2

The PHP version I used

$ php --version
PHP 5.5.10 (cli) (built: Mar  6 2014 06:34:18) 
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies

PHPUnit 3.7

$ time phpunit --coverage-html /tmp/coverage
PHPUnit 3.7.32-3-g4dbc3ab by Sebastian Bergmann.

Configuration read from /home/sb/templateengine2/phpunit.xml

................................................................. 65 / 82 ( 79%)
..F..............

Time: 19 seconds, Memory: 10.50Mb

There was 1 failure:

1) Warning
No tests found in class "TE_LOGLEVEL_Test".


FAILURES!
Tests: 82, Assertions: 202, Failures: 1.

Generating code coverage report in HTML format ... done
/usr/local/src/phpunit/phpunit.php --coverage-html /tmp/coverage  6.81s user 14.17s system 99% cpu 21.025 total

PHPUnit 4.0

$ time phpunit --coverage-html /tmp/coverage
PHPUnit 4.0.8-5-g96cda9f by Sebastian Bergmann.

Configuration read from /home/sb/templateengine2/phpunit.xml

................................................................. 65 / 82 ( 79%)
..F..............

Time: 1.88 minutes, Memory: 10.75Mb

There was 1 failure:

1) Warning
No tests found in class "TE_LOGLEVEL_Test".

FAILURES!
Tests: 82, Assertions: 202, Failures: 1.

Generating code coverage report in HTML format ... done
/usr/local/src/phpunit/phpunit --coverage-html /tmp/coverage  23.08s user 91.52s system 99% cpu 1:54.67 total

The only apparent difference between the test suite I used earlier and yours is that your uses process isolation. I will investigate the way that code coverage data is passed from the isolated process to the main process as this is the most likely candiate for a cause for the performance regression.

BTW: I was not able to reproduce a performance regression using @cobexer's test suite when no code coverage report is requested.

@cobexer

This comment has been minimized.

Copy link
Author

commented Mar 17, 2014

you probably need a slow netbook to reproduce ;) my main computer also executes the testsuite far faster ;)

I could check with my netbook without coverage and show you the numbers on my way back home today.

@sebastianbergmann could you check the slowdown with a phpunit build that is not inside a phar (or compressed with a lower compression level)? maybe thats the issue

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 17, 2014

I did not check with a PHAR but with a Git checkout of PHPUnit.

@icewolfs

This comment has been minimized.

Copy link

commented Mar 17, 2014

The numbers from @dreis2211 above are also without code coverage. A lot of the tests are integration tests and involve database fixtures, maybe that made the performance difference more obvious.

@christeredvartsen

This comment has been minimized.

Copy link
Contributor

commented Mar 17, 2014

@sebastianbergmann Some numbers for imbo/imbo:

4.0, without coverage: Time: 22.49 seconds, Memory: 309.25Mb
3.7, without coverage: Time: 14.78 seconds, Memory: 32.50Mb
4.0, with coverage:    Time: 3.11 minutes, Memory: 314.50Mb
3.7, with coverage:    Time: 1.55 minutes, Memory: 35.75Mb

To test with 3.7:

git clone https://github.com/imbo/imbo -b develop
composer install
./vendor/bin/phpunit -c tests/phpunit

To test with 4.0:

git clone https://github.com/christeredvartsen/imbo -b phpunit-upgrade
composer install
./vendor/bin/phpunit -c tests/phpunit
@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 17, 2014

Here's the result of some memory profiling I did with PHPUnit 4.0 and @christeredvartsen's test suite:

                                                            Inclusive      Own
function                                            #calls  time   memory  time     memory
----------------------------------------------------.----------------------------------------
str_replace                                          70910  3.4011 255222608 3.4011 255222608
PHPUnit_Framework_MockObject_Generator->evalClass     1435  2.8262 249013808 2.2037 239140472
PHPUnit_Framework_Comparator_Scalar->assertEquals     6591  4.5606 113680040 1.5224 112715960
file_get_contents                                    57716 18.5111 47716840  3.6052  47699160
ReflectionFunctionAbstract->getParameters           107012  4.4482 46293680  4.4482  46293680
ReflectionClass->getName                             79831  3.1903 24529192  3.1903  24529192
ReflectionParameter->getName                         83018  3.3631 24330912  3.3631  24330912
ReflectionFunctionAbstract->getName                 107453  4.3072 22790608  4.3072  22790608
org\bovigo\vfs\vfsStreamAbstractContent->appliesTo  472379 66.9510 22748024 34.1073  22748024
ReflectionClass->getMethods                            197  0.0451 19424272  0.0451  19424272
ReflectionClass->getMethod                           55810  2.5354 17805072  2.5354  17805072
sprintf                                              62505  2.3149 16887384  2.3149  16887384
ReflectionMethod->getDeclaringClass                  74322  3.0610 16836560  3.0610  16836560
array_keys                                            7339  0.2795 16730072  0.2795  16730072
substr                                              314744 10.8291 15482032 10.8291  15482032
org\bovigo\vfs\vfsStreamDirectory::getChildName     306746 42.9897 29667928 21.8820  14773016
explode                                              26722  1.0299 12811472  1.0299  12811472
ob_start                                               747  0.0286 12383224  0.0286  12383224
PHPUnit_Util_Fileloader::load                           99  0.2810 13848440  0.2076  11506200
PHPUnit_Util_GlobalState::backupSuperGlobalArray      5229  3.6456 18205808  1.4205  11196256
get_class_methods                                     1356  0.0774 10187152  0.0774  10187152
eval                                                  1418  0.5682  9873336  0.5674   9867752
get_declared_classes                                   196  0.0212  9480592  0.0212   9480592
Composer\Autoload\includeFile                          293  1.3168 11628864  1.0136   8810200
serialize                                            67756  2.4965  8224952  2.4965   8224952
@lstrojny

This comment has been minimized.

Copy link
Contributor

commented Mar 17, 2014

We are experiencing a similar increase in runtime and memory usage. From ~15 seconds to ~1 minute and a memory exhaustion error.

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 17, 2014

@lstrojny Any chance of getting profiling information?

@ghost

This comment has been minimized.

Copy link

commented Mar 17, 2014

I am experiencing similar performance issues with PHPUnit 4.

Unfortunately I can only provide some facts from my test runs, because the library I am working on is not publicly available - but hopefully my results will help finding the bugs nevertheless. (I guess there are at least 2 bugs or design decisions causing bad performance! See below.)

All my tests are properly isolated, pure unit tests:

  • using only scalar values, small value objects or mock objects
  • containing one to three assertions.

It should be mentioned that

  • both PEAR and Composer installations yielded nearly the same results (differences < 3 %)
  • strict mode on/off had no significant impact on execution times (differences < 5 %)
  • I did not constantly pay attention to memory usage, because the increase from PHPUnit 3.7.32 to 4.0.7 was insignificantly small (presumably due to the quite small codebase with few dependencies)
  • all tests were executed without process isolation.

In addition to the results listed below, I noticed the following conspicuous behavior patterns:

  • Tests that (would) get categorized as being risky by PHPUnit 4 were extremely slower than all the others - but only when executed with PHPUnit 4 with code coverage enabled, regardless of strict mode on/off.
  • Tests that create mock objects (regardless whether or not their methods get configured in some way) were significantly slower than all the others when executed with PHPUnit 4.

Here are the average execution times in seconds:

Test suite 1

  • 10 test case classes
  • 714 tests:
    • ~ half of the tests creating mock objects
    • 303 tests categorized as being risky by PHPUnit 4
                                                PHPUnit 3.7.32  PHPUnit 4.0.7

PHP 5.3.3-7+squeeze19, code coverage disabled   0.950           15.62
PHP 5.3.3-7+squeeze19, code coverage enabled    7.71            56.12

PHP 5.3.28-1~dotdeb.0, code coverage disabled   0.977           15.01
PHP 5.3.28-1~dotdeb.0, code coverage enabled    5.93            50.64

PHP 5.4.4-14+deb7u8, code coverage disabled     1.26            16.17
PHP 5.4.4-14+deb7u8, code coverage enabled      6.26            52.37

PHP 5.4.26-1~dotdeb.1, code coverage disabled   1.16            17.16
PHP 5.4.26-1~dotdeb.1, code coverage enabled    5.87            54.16

PHP 5.5.10-1~dotdeb.1, code coverage disabled   1.20            16.85
PHP 5.5.10-1~dotdeb.1, code coverage enabled    6.11            52.94

Test suite 2

  • subset of test suite 1
  • 1 test case class
  • 85 tests:
    • 0 tests creating mock objects
    • 76 tests categorized as being risky by PHPUnit 4
                                                PHPUnit 3.7.32  PHPUnit 4.0.7

PHP 5.3.3-7+squeeze19, code coverage disabled   0.107           0.173
PHP 5.3.3-7+squeeze19, code coverage enabled    0.911           5.41

PHP 5.3.28-1~dotdeb.0, code coverage disabled   0.112           0.166
PHP 5.3.28-1~dotdeb.0, code coverage enabled    0.713           5.18

PHP 5.4.4-14+deb7u8, code coverage disabled     0.427           1.69
PHP 5.4.4-14+deb7u8, code coverage enabled      1.19            6.91

PHP 5.4.26-1~dotdeb.1, code coverage disabled   0.405           1.72
PHP 5.4.26-1~dotdeb.1, code coverage enabled    1.10            6.73

PHP 5.5.10-1~dotdeb.1, code coverage disabled   0.426           1.68
PHP 5.5.10-1~dotdeb.1, code coverage enabled    1.17            6.77

Test suite 3

  • subset of test suite 1
  • 1 test case class
  • 186 tests:
    • 0 tests creating mock objects
    • 0 tests categorized as being risky by PHPUnit 4
                                                PHPUnit 3.7.32  PHPUnit 4.0.7

PHP 5.3.3-7+squeeze19, code coverage disabled   0.267           0.306
PHP 5.3.3-7+squeeze19, code coverage enabled    1.97            7.92

PHP 5.3.28-1~dotdeb.0, code coverage disabled   0.271           0.287
PHP 5.3.28-1~dotdeb.0, code coverage enabled    1.52            7.17

PHP 5.4.4-14+deb7u8, code coverage disabled     0.586           1.81
PHP 5.4.4-14+deb7u8, code coverage enabled      2.03            9.11

PHP 5.4.26-1~dotdeb.1, code coverage disabled   0.575           1.91
PHP 5.4.26-1~dotdeb.1, code coverage enabled    1.90            8.76

PHP 5.5.10-1~dotdeb.1, code coverage disabled   0.556           1.81
PHP 5.5.10-1~dotdeb.1, code coverage enabled    1.93            8.77
@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 18, 2014

Reverting sebastianbergmann/phpunit-mock-objects@ee8b777 and d1efb51 reduces the memory footprint from 303.25Mb to 301.75Mb on my machine for PHPUnit 4.0 and @christeredvartsen's test suite.

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 18, 2014

<?php
class Foo
{
    public function bar($a, $b, $c)
    {
    }
}


class Test extends PHPUnit_Framework_TestCase
{
    public function testOne()
    {
        $this->getMock('Foo');
    }
}

When I run the test above with --repeat 10000 the memory consumption of both PHPUnit 3.7 and 4.0 is 3.00Mb. PHPUnit 3.7 takes 33.24 seconds to run and PHPUnit 4.0 takes 37.88 seconds.

Seems like something related to mock objects got slower.

@dreis2211

This comment has been minimized.

Copy link

commented Mar 18, 2014

Unfortunately I couldn't get my XHProf running on my Windows machine with PHP 5.5 with the needed cpu and mem usage flags, but I was able to generate at least some reports, which would confirm a problem somewhere in the creation of mock objects. Below listed the top 10 functions with PHPUnit 3.7.28 and 4.0

PHPUnit 3.7.28

PHPUnit_Framework_ComparatorFactory::getComparatorFor
PHPUnit_Framework_Comparator_Array::assertEquals
PHPUnit_Util_Printer::write
PHPUnit_Util_Type::export
PHPUnit_Util_Type::recursiveExport
Text_Template::setFile
PHPUnit_Framework_Comparator_Scalar::assertEquals 
PHPUnit_Util_Type::shortenedExport
PHPUnit_Framework_Constraint_IsEqual::evaluate
PHPUnit_Runner_Version::id

Time: 20.32 seconds, Memory: 36.50Mb

OK (1212 tests, 1868 assertions)

PHPUnit 4.0.9

Text_Template::setFile
ClassLoaderParser::_getClassNamesNamespaceAware 
ClassLoaderParser::parseDirectory
phpunit\phpunit-mock-objects\src\Framework\MockObject\Generator.php(300) : eval()'d code    
Text_Template::render
PHPUnit_Framework_MockObject_Generator::getMethodParameters
PHPUnit_Framework_MockObject_Generator::generateMockedMethodDefinition
PHPUnit_Framework_MockObject_Generator::generateMock
PHPUnit_Framework_MockObject_Generator::generateMockedMethodDefinitionFromExisting  
PHPUnit_Framework_ComparatorFactory::getComparatorFor

Time: 1.36 minutes, Memory: 259.50Mb

OK (1212 tests, 1868 assertions)
@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 18, 2014

@dreis2211 What is ClassLoaderParser? It is not part of PHPUnit.

@dreis2211

This comment has been minimized.

Copy link

commented Mar 18, 2014

@sebastianbergmann This is part of our class loading part in our project.

Interesting side-note: This shouldn't be in there. In fact, this happens when he tries to load a class that doesn't exist. In that case he tries to load a test that is using a data provider.

Class he tries to load:

DailyAwardsTest::testCheckIsDailyAward
@whatthejeff

This comment has been minimized.

Copy link
Contributor

commented Mar 18, 2014

@sebastianbergmann I don't have too much time to look into this at the moment, but is it possible that sebastianbergmann/phpunit-mock-objects@fd2951c makes the mock object cache less useful?

@dreis2211

This comment has been minimized.

Copy link

commented Mar 18, 2014

@sebastianbergmann I used your example and added the data provider mechanism to it. Here are the results

<?php
class Foo
{
    public function bar($a, $b, $c)
    {
    }
}


class Test extends PHPUnit_Framework_TestCase
{
    /**
     * @dataProvider testData
     */
    public function testOne($test)
    {
        $this->getMock('Foo');
    }

    public function testData() {
        $array = array();

        for ($i = 0; $i < 10000; $i++) {
            $array[] = array('test');
        }

        return $array;
    }
}

PHPUnit 3.7.32

PHPUnit_Util_Printer::write
PHPUnit_Framework_TestCase::runTest
PHPUnit_Framework_TestCase::runBare 
PHPUnit_Framework_TestResult::run
PHPUnit_Framework_TestResult::endTest 
PHPUnit_Framework_TestSuite::run@2
PHPUnit_TextUI_ResultPrinter::endTest
PHPUnit_Util_Test::getRequirements 
PHPUnit_Framework_TestCase::run
PHPUnit_Framework_TestResult::startTest

Time: 4.88 seconds, Memory: 28.25Mb

OK (10001 tests, 0 assertions)

PHPUnit 4.0.9

PHPUnit_Framework_TestSuite::run
PHPUnit_Framework_TestSuite::run@1
PHPUnit_Framework_TestSuite::run@2
PHPUnit_Framework_TestCase::run
PHPUnit_Framework_TestResult::run 
PHPUnit_Framework_TestCase::runBare
PHPUnit_Framework_TestCase::runTest
Test::testOne
PHPUnit_Framework_TestCase::getMock 
PHPUnit_Framework_MockObject_Generator::getMock

Time: 31.38 seconds, Memory: 216.50Mb

OK (10001 tests, 0 assertions)
@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 18, 2014

@dreis2211 Thanks!

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 18, 2014

@whatthejeff That would at least make sense. I'll investigate.

@whatthejeff

This comment has been minimized.

Copy link
Contributor

commented Mar 18, 2014

@sebastianbergmann I can confirm that making the cache static fixes the data provider issue posted by @dreis2211.

sebastianbergmann added a commit to sebastianbergmann/phpunit-mock-objects that referenced this issue Mar 18, 2014
sebastianbergmann added a commit to sebastianbergmann/phpunit-mock-objects that referenced this issue Mar 18, 2014
@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 18, 2014

@dreis2211 Can you check whether sebastianbergmann/phpunit-mock-objects@3ba8622 fixes your issue?

@dreis2211

This comment has been minimized.

Copy link

commented Mar 18, 2014

@sebastianbergmann Seems to be better. With my example it still uses 2-3 times more memory than with 3.7.32, but in regards of time it seems okay. Also our real project seems to be back to normal values.

PHPUnit4.0.9

Time: 5.8 seconds, Memory: 75.25Mb
OK (10001 tests, 0 assertions)

PHPUnit3.7.32

Time: 4.88 seconds, Memory: 28.25Mb
OK (10001 tests, 0 assertions)
@christeredvartsen

This comment has been minimized.

Copy link
Contributor

commented Mar 18, 2014

@sebastianbergmann That patch seems to fix the issues in my test suite at least:

4.0.9 (with patch)

Time: 34.51 seconds, Memory: 36.00Mb
OK (747 tests, 2404 assertions)

3.7.32

Time: 34.58 seconds, Memory: 32.50Mb
OK (747 tests, 2404 assertions)

4.0.9 (with patch) with code coverage

Time: 2.06 minutes, Memory: 40.00Mb
OK (747 tests, 2404 assertions)

3.7.32 with code coverage

Time: 1.89 minutes, Memory: 35.75Mb
OK (747 tests, 2404 assertions)
@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Mar 18, 2014

Closing this issue now. If there are other performance regressions that still exist in PHPUnit 4.0.10 please open a new ticket. Thanks!

Thank you, @cobexer, @dreis2211, and @christeredvartsen, @HenryPrang, @lstrojny for your extensive feedback that helped track down the memory usage regression. Thank you, @whatthejeff for figuring out the root cause.

@whatthejeff

This comment has been minimized.

Copy link
Contributor

commented Mar 18, 2014

No problem :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.