Significant reduction in performance when running phpunit after upgrade from 8.1.9 to 8.1.10 #9993
Comments
|
The diff between the two versions is: php-8.1.10...php-8.1.9 which doesn't seem to indicate much. Are you sure opcache is in the same configuration between both versions? |
|
@Girgias thanks for the reply!
Yes I do, I ran tests with exact the same configuration. |
If that is a general issue with phpunit performance, I wonder why that has not already been reported. Maybe @sebastianbergmann has heard about it? |
|
This is the first time I hear about a PHPUnit performance degredation related to PHP 8.1.9->PHP 8.1.10. |
|
@piszczek, is the test suite you are referring to publicly available, so someone could check it? |
|
Furthermore (and just to rule out obvious mistakes): Are you using the same PHPUnit version? Does one PHP version have Xdebug enabled and the other does not? |
|
@cmb69 unfortunately code isn't public. I noticed this issue on commercial project in my work. @sebastianbergmann here is a configuration of phpunit.xml <?xml version="1.0" encoding="UTF-8"?>
<!-- https://phpunit.de/manual/current/en/appendixes.configuration.html -->
<phpunit backupGlobals="false" colors="true"
bootstrap="config/bootstrap.php"
>
<php>
<ini name="error_reporting" value="-1"/>
<env name="APP_ENV" value="test"/>
<env name="KERNEL_CLASS" value="App\Kernel"/>
<env name="SHELL_VERBOSITY" value="-1"/>
<env name="SYMFONY_DEPRECATIONS_HELPER" value="weak"/>
<env name="SYMFONY_PHPUNIT_VERSION" value="9.5"/>
<!-- ###+ symfony/mailer ### -->
<!-- MAILER_DSN=smtp://localhost -->
<!-- ###- symfony/mailer ### -->
</php>
<testsuites>
<testsuite name="Project Test Suite">
<directory>tests</directory>
</testsuite>
</testsuites>
<listeners>
<listener class="Symfony\Bridge\PhpUnit\SymfonyTestsListener"/>
</listeners>
<extensions>
<extension class="DAMA\DoctrineTestBundle\PHPUnit\PHPUnitExtension"/>
</extensions>
</phpunit>(Extension: DAMA\DoctrineTestBundle\PHPUnit\PHPUnitExtension) Project is Symfony based 6.1 |
github-actions
bot
added
Status: Needs Triage
and removed
Status: Needs Feedback
labels
I'm afraid this ticket is not actionable, if there is no reproducer. |
|
@cmb69 I raised issue with hope that someone had similar problem and maybe solution... I will try to spend some time in December to reproduce that again and do more tests then I will back to you with code samples (that I can share) |
github-actions
bot
added
Status: Needs Triage
and removed
Status: Needs Feedback
labels
|
We are experiencing the exact same issue. Related to the same version upgrade. @piszczek have you been able to make any progress on this or find a workaround? |
|
@gitnik do you have a reproducible? |
|
No. But I will try to bisecting the diff from 8.1.9 to 8.1.10 to see which commit is the culprit |
|
@gitnik unfortunately I didn't find any workaround to this. I did more tests in December - but unfortunately, I didn't have time to create a version for reproduction this issue. What I observe, and this might be a hint ; When I ran one PHPUnit test with one controller test (SymfonyTestClient in the same process) then the timing was the same in 8.1.9 and 8.1.10 but When I ran one test with multiple controllers tests then the performance was significantly worse on PHP 8.1.10. Sample PHP code $start = microtome(true);
$response = $this
->requestBuilder(
$client,
$method = 'GET',
$url = sprintf(Endpoint::GET_PANEL, $panelId)
)
->send()
;
$diff = microtome(true) - $start ; // one $response has the same time between version 8.1.9 and 8.1.10
// another test with multiple requests (request in the same process)
$start = microtome(true);
for ($i = 1; $i <= 100; $i++) {
$response = $this
->requestBuilder(
$client,
$method = 'GET',
$url = sprintf(Endpoint::GET_PANEL, $panelId)
)
->send()
;
}
$diff = microtome(true) - $start ; // much worse timing on 8.1.10 (and also on 8.2 !)and I saw some correlation when the session was created, without the creation of the session differences in timing were minimal. |
|
Can you elaborate on "much worse", in other words: what are the numbers you get? |
|
@sebastianbergmann |
|
There are only 77 commits between PHP 8.1.9 and PHP 8.1.10, with even fewer actually changing the code. It should be relatively easy to find the exact change that causes this using git-bisect. As you are experiencing this in a complex setup (Symfony, PHPUnit), it is unlikely that you can provide a minimal, self-contained test case to debug this issue. What you can do is bisect between PHP 8.1.9 ("good") and PHP 8.1.10 ("bad"), build PHP for each bisect step, and run your tests. Then report the commit between PHP 8.1.9 and PHP 8.1.10 that causes this performance regression for you. |
|
This is the commit that doubles PHPUnit's run times for us: Any hint as to what behaviour we should be looking for in our code (or dependencies) that could potentially cause this increase in run time, given this commit? |
|
I'm not sure if that's expected but here's what fast binary (pre 0709578): array(4) {
["runs"]=>
int(89)
["collected"]=>
int(6020079)
["threshold"]=>
int(230001)
["roots"]=>
int(183697)
}slow binary (at 0709578): array(4) {
["runs"]=>
int(1889)
["collected"]=>
int(6307068)
["threshold"]=>
int(10001)
["roots"]=>
int(6231)
} |
|
But ultimately it also means that the heuristics we are using for growing the GC root buffer are not well suited for his code. |
@bwoebi thats a great point. I did not consider this as we use php-cs fixer to clean this up automatically @gitnik in case you have this problem, the cs fixer rule might help you |
|
In my case |
Unfortunately that didn't help. I also spend all day tracking down cyclical dependencies and looking for missing tearDown actions, but no luck so far. |
|
hi @bwoebi You mentioned that PHP is not well suited for this kind of code, do you think PHP team will work on this? @gitnik did you solve your problem? We did a further investigation of our code - and we found that indirectly our OAS pre-validation is causing those performance issues. Each request is validated against schema declared in our OpenAPI Schema. The OpenAPI object is large (because it contains every endpoint definition) and it contains around 60000 objects - objects are cached as valid php code and are loaded once when the first request in test suite is runned. The problem only appears in tests when they are run in a single process. GC at some point hits the limit and we guess it causes problem. Unfortunatelly we stuck at this and we have below options:
|
|
No we did not. We were able to shave off a few seconds by setting null in some tearDown methods, but never came close to the "original" performance. We opted to simply disable GC and accept the increase in memory. Once PHP 8.3 and the updated |
|
FYI: Starting with PHPUnit 10.1, the telemetry for events emitted by PHPUnit will include information from |
|
I'm experiencing a significant slowdown of our test suite with 8.2. Not sure if it's related. Our tests finish in 19 minutes on PHP 8.1 and it took 3 hours to get to 52% on PHP 8.2. I just couldn't wait anymore and aborted the run. So I'm stuck with PHP 8.1 for now. Also the memory usage went up crazily — like over 20G of RAM. While on 8.1 it was around 1-2G. |
Same PHP configuration and PHPUnit version/configuration? |
Yep. Everything else is the same. We use Vagrant with Ansible, so the VMs are destroyed and configured from scratch when upgrading PHP and other stuff. I just changed Using the |
|
Figured it out. The cause was a crazy number of deprecation warnings. I guess they all get accumulated during a test run to be printed out after the run is finished. There were so many of them that it consumed tens of GBs of RAM. And I guess the slowdown was caused by having to swap RAM because it exceeded my machine's capacity. |
|
@elnur You never mentioned which version of PHPUnit you are using experiencing this with. |
9.6.6. Why? Are deprecation warnings handled differently in 10? |
Yes, see the section titled "Test Outcome versus Test Issues" in the PHPUnit 10 release announcement. |
Good to know. Thanks. I tried upgrading to PHPUnit 10 while dealing with this PHP 8.2 problem, but couldn't because of the BC breaks. Will get to it later. |
|
Found same problem. We have big Symfony 5.2 project with unit tests on PHP Unit 9.6.3. After update from PHP 8.1.9 to PHP 8.1.10 our tests was slow down: please let me know how we can fix this issue. Update to PHPUnit 10 can fix this issue? |
Unlikely. |
|
We found an a possible solution. If we run tests in parallel mode with paratest it give us more fast result on PHP 8.1.10 and 8.2 that we have before with paratest on PHP 8.1.9. So @piszczek and @elnur I suggest try to run tests in parallel mode. Provably it will give you a chance update PHP without significant reduction in performance |
|
hi @stepozer unfortunately paratest won't work for us. We are using doctrine-test-bundle which works on the rollback/transactions - but with a combination with paratests it generates db errors (described on that link). In a previous post, I suggested that our OAS Schema validator causes those performance issues. We spent some time refactoring this library. It helped us when we ran a group of tests (we tested this issue on the specific directory with tests) - and we were very happy that finally, we got it - but when we ran all the test - we stuck again 🐒 - our OAS library was unguilty. At the beginning of the tests, the timing was the same on 8.1.9 and 8.2.0, but later 8.1.9 was faster; after More tests = the differences were bigger. We spent a lot of time investigating that, team are tired ("@piszczek, leave it we can work on 8.1.9 for now") . We decided to leave this topic, and we will back to this after 3 months. We still have to check suggestions with deprecation messages. |
|
@piszczek interesting... We also using Symfony + doctrine-test-bundle with paratest and it is working well for us. Do you create DB for each paratest process? Because it is require setup separate DB for each execution thread. Also we split redis for each thread. |
|
@stepozer no, we ran that test on the same database (MariaDB). 🤔 Thanks, for the suggestion. So you are using, let say 8 test databases? eg. test_1, test_2 ... for each thread? |
|
@piszczek on one DB it will not work. Each thread should be run in separate environment (DB, redis, etc). And yes, we create 8 DB as I remember. I suggest try to use 8 threads. This is documentation: https://github.com/paratestphp/paratest#test-token so paratest generate special ENV variable that can be used for setting DB number. This give us speed up for our tests into 2 or 3 times. I don't remember |
|
@stepozer thank for this; We will test this approach, and I will back with answer . |
|
I've solved my PHP 8.2 problem by updating dependencies that caused many deprecation notices and also refactoring our code to remove the remaining deprecation notices. So it's all good here. But I guess my problem was different from the one this issue is about. |
|
PHPUnit 9.5.28, 139 tests, 411 assertions, 100% pass (a Symfony project ~4400 LoC testsuite bootstraping many kernels & co) Reference run, php 7.4
Run 2, php 8.2 (zend.enable_gc=0)
Run 3, php 8.2 (zend.enable_gc=0) after fixing some deprecations
Run 4, php 8.2 (with enable_gc)
|
Description
Last time we did update from PHP 8.1.7 to 8.1.12 and we noticed that phpunit tests slowed down.
I did a research where is the problem and it looks that some changes between version 8.1.9 and 8.1.10 are causing performance issues.
The phpunit timing between PHP version
Tests: 2981, Assertions: 19876, Incomplete: 1
I couldn't find any specific code which is raising issue; but there is analogy more tests then the differences in time are bigger (longer duration on 8.1.10)
PHP configuration:
PHP Version
PHP 8.1.10
Operating System
Docker Image PHP-FPM 8.1.10
The text was updated successfully, but these errors were encountered: