New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance regression with process isolation and code coverage in 4.2.6 #1456

Closed
jaydiablo opened this Issue Oct 2, 2014 · 6 comments

Comments

5 participants
@jaydiablo

We recently tried upgrading phpunit on a testsuite of ours that uses some process isolation, and noticed that the time to run the tests (with coverage) increased dramatically. This seems similar to what is described in #1187, but the patch there doesn't help.

The particular suite we ran against made use of both Mockery and PHPUnit's mock objects (using latest version of mock objects, with the patch from #1187 included). To test, I tried removing all of phpunit's Mock objects and use just Mockery, but that didn't help. As a test, I wanted to see if Mockery was to blame at all, so I experimented with adding the @runTestsInSeparateProcesses annotation to the top of a suite that doesn't use any mock objects. The same performance issues were there, but only when code coverage was enabled.

Unforunately, the suites I'm running against are private projects, so I can't provide them as examples, but I do have the output of several runs, under different conditions to provide as an example of the performance issue. I'd like to try adding that annotation to any public php test suite to see if the issue is there. I'll follow up with the results of that. The suite we are running does some DB testing with some large php libraries, so quite a few files are covered in each test run, which may have some bearing on this.

PHPUnit 3.7.37

No Process Isolation, No Coverage

Time: 3.36 seconds, Memory: 21.00Mb

Just Coverage

Time: 14.21 seconds, Memory: 23.50Mb

Just Process Isolation

Time: 9.41 seconds, Memory: 8.75Mb

Process Isolation and Coverage

Time: 26.07 seconds, Memory: 18.00Mb

PHPUnit 4.2.6

No Process Isolation, No Coverage

Time: 3.74 seconds, Memory: 22.00Mb

Just Coverage

Time: 15.89 seconds, Memory: 32.00Mb

Just Process Isolation

Time: 8.36 seconds, Memory: 9.75Mb

Process Isolation and Coverage

Time: 7.71 minutes, Memory: 20.25Mb

@jaydiablo

This comment has been minimized.

Show comment
Hide comment
@jaydiablo

jaydiablo Oct 2, 2014

So, my first attempt at testing this against a public testsuite, I used the suite mentioned in #1187:

git clone https://github.com/sebastianbergmann/money.git

But I modified the three test classes there by adding the process isolation annotation to each class:

/**
 * @runTestsInSeparateProcesses
 */

Here's two runs with PHPUnit 3.7.37:

Without coverage:

./vendor/bin/phpunit --bootstrap src/autoload.php tests
PHPUnit 3.7.37 by Sebastian Bergmann.

.......................................

Time: 2.31 seconds, Memory: 4.50Mb

OK (39 tests, 59 assertions)

With coverage:

./vendor/bin/phpunit --coverage-clover=/tmp/coverage.xml --bootstrap src/autoload.php tests
PHPUnit 3.7.37 by Sebastian Bergmann.

.......................................

Time: 4.38 seconds, Memory: 5.00Mb

OK (39 tests, 59 assertions)

Generating code coverage report in Clover XML format ... done

And now with PHPUnit 4.2.6:

Without coverage:

./vendor/bin/phpunit --bootstrap src/autoload.php tests
PHPUnit 4.2.6 by Sebastian Bergmann.

.......................................

Time: 2.91 seconds, Memory: 5.25Mb

OK (39 tests, 59 assertions)

With coverage:

./vendor/bin/phpunit --coverage-clover=/tmp/coverage.xml --bootstrap src/autoload.php tests
PHPUnit 4.2.6 by Sebastian Bergmann.

.......................................

Time: 15.27 seconds, Memory: 5.50Mb

OK (39 tests, 59 assertions)

Generating code coverage report in Clover XML format ... done

I forgot to mention my environment:

php -v
PHP 5.5.17 (cli) (built: Sep 20 2014 17:12:00)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2014, by Zend Technologies
    with Xdebug v2.2.4, Copyright (c) 2002-2014, by Derick Rethans

So even a fairly small project exhibits a substantial slow down in 4.2.6 with process isolation and code coverage. This seems to be amplified quite a bit as more files are added to coverage.

So, my first attempt at testing this against a public testsuite, I used the suite mentioned in #1187:

git clone https://github.com/sebastianbergmann/money.git

But I modified the three test classes there by adding the process isolation annotation to each class:

/**
 * @runTestsInSeparateProcesses
 */

Here's two runs with PHPUnit 3.7.37:

Without coverage:

./vendor/bin/phpunit --bootstrap src/autoload.php tests
PHPUnit 3.7.37 by Sebastian Bergmann.

.......................................

Time: 2.31 seconds, Memory: 4.50Mb

OK (39 tests, 59 assertions)

With coverage:

./vendor/bin/phpunit --coverage-clover=/tmp/coverage.xml --bootstrap src/autoload.php tests
PHPUnit 3.7.37 by Sebastian Bergmann.

.......................................

Time: 4.38 seconds, Memory: 5.00Mb

OK (39 tests, 59 assertions)

Generating code coverage report in Clover XML format ... done

And now with PHPUnit 4.2.6:

Without coverage:

./vendor/bin/phpunit --bootstrap src/autoload.php tests
PHPUnit 4.2.6 by Sebastian Bergmann.

.......................................

Time: 2.91 seconds, Memory: 5.25Mb

OK (39 tests, 59 assertions)

With coverage:

./vendor/bin/phpunit --coverage-clover=/tmp/coverage.xml --bootstrap src/autoload.php tests
PHPUnit 4.2.6 by Sebastian Bergmann.

.......................................

Time: 15.27 seconds, Memory: 5.50Mb

OK (39 tests, 59 assertions)

Generating code coverage report in Clover XML format ... done

I forgot to mention my environment:

php -v
PHP 5.5.17 (cli) (built: Sep 20 2014 17:12:00)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2014, by Zend Technologies
    with Xdebug v2.2.4, Copyright (c) 2002-2014, by Derick Rethans

So even a fairly small project exhibits a substantial slow down in 4.2.6 with process isolation and code coverage. This seems to be amplified quite a bit as more files are added to coverage.

@GrahamCampbell

This comment has been minimized.

Show comment
Hide comment
@GrahamCampbell

GrahamCampbell Oct 2, 2014

Contributor

I'm experiencing this issue too. See my comments about test execution times in the OP in ClassPreloader/ClassPreloader#28.

Contributor

GrahamCampbell commented Oct 2, 2014

I'm experiencing this issue too. See my comments about test execution times in the OP in ClassPreloader/ClassPreloader#28.

@whatthejeff

This comment has been minimized.

Show comment
Hide comment
@whatthejeff

whatthejeff Oct 3, 2014

Contributor

Seems like sebastianbergmann/php-code-coverage#139 is most likely the problem. During process isolation, the same files will be tokenized over and over again.

Unfortunately I don't have much time for open source at the moment. Maybe @sebastianbergmann can look into a fix when he has some time.

Contributor

whatthejeff commented Oct 3, 2014

Seems like sebastianbergmann/php-code-coverage#139 is most likely the problem. During process isolation, the same files will be tokenized over and over again.

Unfortunately I don't have much time for open source at the moment. Maybe @sebastianbergmann can look into a fix when he has some time.

@Exeu

This comment has been minimized.

Show comment
Hide comment
@Exeu

Exeu Dec 4, 2014

Same problem here.

Exeu commented Dec 4, 2014

Same problem here.

@sebastianbergmann

This comment has been minimized.

Show comment
Hide comment
@sebastianbergmann

sebastianbergmann Dec 4, 2014

Owner

Features such as process isolation are not very high on my list of things to look at or work on, sorry.

Owner

sebastianbergmann commented Dec 4, 2014

Features such as process isolation are not very high on my list of things to look at or work on, sorry.

@sebastianbergmann

This comment has been minimized.

Show comment
Hide comment
@sebastianbergmann

sebastianbergmann Oct 2, 2015

Owner

Dear contributor,

let me start by apologizing for not commenting and/or working on the issue you have reported or merging the pull request you have sent sooner.

PHPUnit 5.0 was released today. And today I am closing all open bug reports and pull requests for PHPUnit and its dependencies that I maintain. Please do not interpret the closing of this ticket as an insult or a lack of interest in your problem. I am sorry for any inconvenience this may cause.

If the topic of this ticket is still relevant then please open a new ticket or send a new pull request. If your ticket or pull request is about a defect then please check whether the issue still exists in PHPUnit 4.8 (which will received bug fixes until August 2016). If your ticket or pull request is about a new feature then please port your patch PHPUnit 5.0 before sending a new pull request.

I hope that today's extreme backlog grooming will allow me to respond to bug reports and pull requests in a more timely manner in the future.

Thank you for your understanding,
Sebastian

Owner

sebastianbergmann commented Oct 2, 2015

Dear contributor,

let me start by apologizing for not commenting and/or working on the issue you have reported or merging the pull request you have sent sooner.

PHPUnit 5.0 was released today. And today I am closing all open bug reports and pull requests for PHPUnit and its dependencies that I maintain. Please do not interpret the closing of this ticket as an insult or a lack of interest in your problem. I am sorry for any inconvenience this may cause.

If the topic of this ticket is still relevant then please open a new ticket or send a new pull request. If your ticket or pull request is about a defect then please check whether the issue still exists in PHPUnit 4.8 (which will received bug fixes until August 2016). If your ticket or pull request is about a new feature then please port your patch PHPUnit 5.0 before sending a new pull request.

I hope that today's extreme backlog grooming will allow me to respond to bug reports and pull requests in a more timely manner in the future.

Thank you for your understanding,
Sebastian

Repository owner locked and limited conversation to collaborators Oct 2, 2015

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.