Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance regression from 3.7.x to 4.4.x #1541

Closed
pprkut opened this issue Dec 21, 2014 · 9 comments
Closed

Performance regression from 3.7.x to 4.4.x #1541

pprkut opened this issue Dec 21, 2014 · 9 comments

Comments

@pprkut
Copy link

pprkut commented Dec 21, 2014

I'm trying to migrate a bigger project from phpunit 3.7.x to the latest version and noticed some rather drastic and unfortunate performance regressions. The project in question is https://github.com/pprkut/lunr

Now, running the entire test suite takes about 2 minutes in 3.7 and about 6 minutes in 4.4, memory usage increases from about 95MB to more than 200MB. It's a rather large testsuite and I'm sure part of the problem is caused by #1456. But I singled out a smaller testsuite that doesn't have any process isolation and still exhibits a visible performance regression.

Using phpunit 3.7 I get

$ phpunit -c tests/phpunit.xml --testsuite Vortex
PHPUnit 3.7.34 by Sebastian Bergmann.

Configuration read from /path/to/project/tests/phpunit.xml

...............................................................  63 / 239 ( 26%)
............................................................... 126 / 239 ( 52%)
............................................................... 189 / 239 ( 79%)
..................................................

Time: 4.42 seconds, Memory: 15.25Mb

OK (239 tests, 452 assertions)

and using phpunit 4.4 I get

$ phpunit -c tests/phpunit.xml --testsuite Vortex
PHPUnit 4.4.0 by Sebastian Bergmann.

Configuration read from /path/to/project/tests/phpunit.xml

...............................................................  63 / 239 ( 26%)
............................................................... 126 / 239 ( 52%)
............................................................... 189 / 239 ( 79%)
..................................................

Time: 6.6 seconds, Memory: 36.75Mb

OK (239 tests, 452 assertions)

Since the config file for phpunit needs minor modifications when switching between the two versions I made convenience branches to allow for easier testing. You can find the branch for 3.7 at https://github.com/pprkut/lunr/tree/phpunit_37 and for 4.4 at https://github.com/pprkut/lunr/tree/phpunit_44.

I'm testing this on a Linux system, using php 5.6.0, phpunit 3.7.34 from pear and the phar version of phpunit 4.4.0
@whatthejeff
Copy link
Contributor

I'm curious what impact disabling the circular reference collector is having.

@pprkut
Copy link
Author

pprkut commented Dec 21, 2014

Good point. There is no real impact for 3.7:

$ phpunit -c tests/phpunit.xml --testsuite Vortex
PHPUnit 3.7.34 by Sebastian Bergmann.

Configuration read from /path/to/project/tests/phpunit.xml

...............................................................  63 / 239 ( 26%)
............................................................... 126 / 239 ( 52%)
............................................................... 189 / 239 ( 79%)
..................................................

Time: 4.32 seconds, Memory: 15.25Mb

OK (239 tests, 452 assertions)

phpunit 4.4 shows a slight memory usage decrease

$ phpunit -c tests/phpunit.xml --testsuite Vortex
PHPUnit 4.4.0 by Sebastian Bergmann.

Configuration read from /path/to/project/tests/phpunit.xml

...............................................................  63 / 239 ( 26%)
............................................................... 126 / 239 ( 52%)
............................................................... 189 / 239 ( 79%)
..................................................

Time: 6.4 seconds, Memory: 25.75Mb

OK (239 tests, 452 assertions)

@whatthejeff
Copy link
Contributor

a slight memory usage decrease

~30% reduction in memory consumption is not too bad. Are you disabling xdebug before running these? That might yield a better baseline.

@pprkut
Copy link
Author

pprkut commented Dec 21, 2014

I agree. I ran the entire testsuite to get a better picture of the effect and memory usage dropped back to 125MB (from more than 200) so it's also closer to 50% reduction. There is still a 25% increase though compared to 3.7, but it's not a huge deal anymore. The runtime drop remained though.

Previous numbers were all with xdebug enabled, tested again without xdebug:

$ phpunit -c tests/phpunit.xml --testsuite Vortex
PHPUnit 3.7.34 by Sebastian Bergmann.

The Xdebug extension is not loaded. No code coverage will be generated.


...............................................................  63 / 239 ( 26%)
............................................................... 126 / 239 ( 52%)
............................................................... 189 / 239 ( 79%)
..................................................

Time: 474 ms, Memory: 12.75Mb

OK (239 tests, 452 assertions)

and phpunit 4.4

$ phpunit -c tests/phpunit.xml --testsuite Vortex
PHPUnit 4.4.0 by Sebastian Bergmann.

Configuration read from /path/to/project/tests/phpunit.xml

The Xdebug extension is not loaded. No code coverage will be generated.

...............................................................  63 / 239 ( 26%)
............................................................... 126 / 239 ( 52%)
............................................................... 189 / 239 ( 79%)
..................................................

Time: 534 ms, Memory: 18.00Mb

OK (239 tests, 452 assertions)

The runtime numbers are now really close, so I also had a look what the effect would be on the entire testsuite and it's 13.35s/77.25MB for 3.7 vs. 14.84s/102MB for 4.4. Still a ~40% memory usage increase, but the runtime regression looks like coming from an xdebug related feature. Not sure it's code coverage as I get much less a performance boost when I disable that alone.

@whatthejeff
Copy link
Contributor

Would you mind trying the current alpha release which contains some performance improvements? Also, maybe it's better to compare apples to apples by testing against the 3.7.x phar.

@pprkut
Copy link
Author

pprkut commented Dec 23, 2014

I did some more testing with multiple versions and can confirm that when taking #1546 out of the equation the current alpha release fixes all remaining runtime performance regressions. In certain scenarios it was now even running faster than 3.7. The memory usage regression remains however, still about 30-40%.

There was no difference in performance numbers between the pear installed phpunit and the phar version (as kind of expected).

@whatthejeff
Copy link
Contributor

@pprkut Thanks so much for your continued support on this issue :)

I've noticed small increases in peak memory consumption between versions, but it's usually ~1MB and varies from suite to suite. I ran a number of the smaller suites from your library and found very similar results:

Core Suite (PHPUnit 3.7.34):

$ ./phpunit-3.7.34.phar -c tests/phpunit.xml --testsuite Core
PHPUnit 3.7.34 by Sebastian Bergmann.

The Xdebug extension is not loaded. No code coverage will be generated.


...............................................................  63 / 281 ( 22%)
............................................................... 126 / 281 ( 44%)
............................................................... 189 / 281 ( 67%)
............................................................... 252 / 281 ( 89%)
.............................

Time: 174 ms, Memory: 7.00Mb

OK (281 tests, 377 assertions)

Core Suite (PHPUnit alpha):

$ ./phpunit-alpha.phar -c tests/phpunit.xml --testsuite Core
PHPUnit alpha-2014-12-26 by Sebastian Bergmann and contributors.

Configuration read from /lunr/tests/phpunit.xml

The Xdebug extension is not loaded. No code coverage will be generated.

...............................................................  63 / 281 ( 22%)
............................................................... 126 / 281 ( 44%)
............................................................... 189 / 281 ( 67%)
............................................................... 252 / 281 ( 89%)
.............................

Time: 181 ms, Memory: 7.75Mb

OK (281 tests, 377 assertions)

Feedback Suite (PHPUnit 3.7.34):

$ ./phpunit-3.7.34.phar -c tests/phpunit.xml --testsuite Feedback
PHPUnit 3.7.34 by Sebastian Bergmann.

The Xdebug extension is not loaded. No code coverage will be generated.


.......................................................

Time: 98 ms, Memory: 5.00Mb

OK (55 tests, 84 assertions)

Feedback Suite (PHPUnit alpha):

$ ./phpunit-alpha.phar -c tests/phpunit.xml --testsuite Feedback
PHPUnit alpha-2014-12-26 by Sebastian Bergmann and contributors.

Configuration read from /lunr/tests/phpunit.xml

The Xdebug extension is not loaded. No code coverage will be generated.

.......................................................

Time: 91 ms, Memory: 5.50Mb

OK (55 tests, 84 assertions)

Flare Suite (PHPUnit 3.7.34):

$ ./phpunit-3.7.34.phar -c tests/phpunit.xml --testsuite Flare
PHPUnit 3.7.34 by Sebastian Bergmann.

The Xdebug extension is not loaded. No code coverage will be generated.


...............................................................  63 / 133 ( 47%)
............................................................... 126 / 133 ( 94%)
.......

Time: 87 ms, Memory: 4.75Mb

OK (133 tests, 331 assertions)

Flare Suite (PHPUnit alpha):

$ ./phpunit-alpha.phar -c tests/phpunit.xml --testsuite Flare
PHPUnit alpha-2014-12-26 by Sebastian Bergmann and contributors.

Configuration read from /lunr/tests/phpunit.xml

The Xdebug extension is not loaded. No code coverage will be generated.

...............................................................  63 / 133 ( 47%)
............................................................... 126 / 133 ( 94%)
.......

Time: 93 ms, Memory: 5.25Mb

OK (133 tests, 331 assertions)

Spark Suite (PHPUnit 3.7.34):

$ ./phpunit-3.7.34.phar -c tests/phpunit.xml --testsuite Spark
PHPUnit 3.7.34 by Sebastian Bergmann.

The Xdebug extension is not loaded. No code coverage will be generated.


...............................................................  63 / 591 ( 10%)
............................................................... 126 / 591 ( 21%)
............................................................... 189 / 591 ( 31%)
............................................................... 252 / 591 ( 42%)
............................................................... 315 / 591 ( 53%)
............................................................... 378 / 591 ( 63%)
............................................................... 441 / 591 ( 74%)
............................................................... 504 / 591 ( 85%)
............................................................... 567 / 591 ( 95%)
........................

Time: 370 ms, Memory: 11.25Mb

OK (591 tests, 943 assertions)

Spark Suite (PHPUnit alpha):

$ ./phpunit-alpha.phar -c tests/phpunit.xml --testsuite Spark
PHPUnit alpha-2014-12-26 by Sebastian Bergmann and contributors.

Configuration read from /lunr/tests/phpunit.xml

The Xdebug extension is not loaded. No code coverage will be generated.

...............................................................  63 / 591 ( 10%)
............................................................... 126 / 591 ( 21%)
............................................................... 189 / 591 ( 31%)
............................................................... 252 / 591 ( 42%)
............................................................... 315 / 591 ( 53%)
............................................................... 378 / 591 ( 63%)
............................................................... 441 / 591 ( 74%)
............................................................... 504 / 591 ( 85%)
............................................................... 567 / 591 ( 95%)
........................

Time: 354 ms, Memory: 12.25Mb

OK (591 tests, 938 assertions)

It would require a bit of time for me to setup the proper environment to run all the suites. I was able to narrow the increased memory usage in the Vortex suite down to the APNS tests. If I skip those tests, I get the following results:

Vortex Suite (PHPUnit 3.7.34):

$ ./phpunit-3.7.34.phar -c tests/phpunit.xml --testsuite Vortex
PHPUnit 3.7.34 by Sebastian Bergmann.

The Xdebug extension is not loaded. No code coverage will be generated.


......................SSSSSSSSSSSSSSSSSSSSS....................  63 / 238 ( 26%)
............................................................... 126 / 238 ( 52%)
............................................................... 189 / 238 ( 79%)
.................................................

Time: 253 ms, Memory: 11.50Mb

OK, but incomplete or skipped tests!
Tests: 238, Assertions: 422, Skipped: 21.

Vortex Suite (PHPUnit alpha):

$ ./phpunit-alpha.phar -c tests/phpunit.xml --testsuite Vortex
PHPUnit alpha-2014-12-26 by Sebastian Bergmann and contributors.

Configuration read from /lunr/tests/phpunit.xml

The Xdebug extension is not loaded. No code coverage will be generated.

......................SSSSSSSSSSSSSSSSSSSSS....................  63 / 238 ( 26%)
............................................................... 126 / 238 ( 52%)
............................................................... 189 / 238 ( 79%)
.................................................

Time: 244 ms, Memory: 12.50Mb

OK, but incomplete, skipped, or risky tests!
Tests: 238, Assertions: 422, Skipped: 21.

As much as I'd like to help figure out what's happening with the APNS tests, I don't really feel comfortable installing the apn extension since I've never heard of it before and I don't really have the time to review it.

Anyway, I think I will close this for now since it's currently hard to reproduce without installing a handful of extensions. If you have time to track down the issue in the APNS tests and you can provide a simple reproducible example without requiring a lot of 3rd party code, please let me know and we can reopen this issue.

Thanks again :)

@pprkut
Copy link
Author

pprkut commented Dec 27, 2014

Thank you for taking a closer look! I doubt the apn extension might be the reason for it, but I do use runkit extensively across all testsuites. Maybe that has something to do with it? I would find it weird though that that would cause an increased memory usage in phpunit, without changing anything else in the stack. I'll see what I can find out. If I know anything more, I'll post it here :)

@whatthejeff
Copy link
Contributor

Thank you for taking a closer look!

No problemo :)

I doubt the apn extension might be the reason for it, but I do use runkit extensively across all testsuites. Maybe that has something to do with it? I would find it weird though that that would cause an increased memory usage in phpunit, without changing anything else in the stack.

I don't doubt that some change in PHPUnit over the past couple of years is responsible for the increased memory usage. There's just really only so much time that we can invest into tracking down a ~5MB difference in peak memory usage. I've already invested a couple of hours and installed a number of extensions and I'm still unable to run the full suite or reproduce the issue.

If I know anything more, I'll post it here :)

Please do!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pprkut @whatthejeff