Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests take 4 times more time since PhpUnit 8 #3745

Closed
VincentLanglet opened this issue Jul 2, 2019 · 23 comments
Closed

Tests take 4 times more time since PhpUnit 8 #3745

VincentLanglet opened this issue Jul 2, 2019 · 23 comments

Comments

@VincentLanglet
Copy link

@VincentLanglet VincentLanglet commented Jul 2, 2019

Q A
PHPUnit version 8.2.x
PHP version 7.2.x
Installation Method Composer

I use this command line to run tests.

symfony/vendor/bin/phpunit --testdox -c symfony/ symfony/tests/Unit

With this phpunit.xml.dist

<?xml version="1.0" encoding="UTF-8"?>

<!-- https://phpunit.de/manual/current/en/appendixes.configuration.html -->
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/6.5/phpunit.xsd"
         backupGlobals="false"
         colors="true"
         bootstrap="config/bootstrap.php"
         failOnWarning="true"
>
    <php>
        <ini name="error_reporting" value="-1" />
        <env name="APP_ENV" value="test" />
        <env name="DATABASE_URL" value="mysql://sagesconseil:pleaseChangeMe@127.0.0.1/asv_test" />
        <env name="SYMFONY_DEPRECATIONS_HELPER" value="weak" />
    </php>

    <filter>
        <whitelist>
            <directory>src</directory>
            <exclude>
                <directory>src/Admin</directory>
                <directory>src/Entity</directory>
                <directory>src/Form/Helper</directory>
                <directory>src/Form/Type</directory>
                <directory>src/Resources</directory>
            </exclude>
        </whitelist>
    </filter>

    <testsuites>
        <testsuite name="All tests">
            <directory>tests</directory>
        </testsuite>
    </testsuites>

    <listeners>
        <listener class="\DAMA\DoctrineTestBundle\PHPUnit\PHPUnitListener" />
    </listeners>
</phpunit>

With phpunit 7, it takes 8 seconds.
image

With phpunit 8, it takes 31 seconds.
image

@epdenouden
Copy link
Collaborator

@epdenouden epdenouden commented Jul 2, 2019

Hello @VincentLanglet!

Thanks for taking the time to post an issue with timing differences between PHPUnit v7 and v8. I tried to search for the project shown in the images, but I think it is private, right?

Question: do you see this 20+ seconds difference when you switch the project back-and-forth between PHPUnit v7 and v8? What changes do you see when running composer install?

It is impossible to tell from the configuration and console output what the reason for the difference is. If I had to debug something like this in my own projects, a few things immediately catch my eye:

  • judging by SYMFONY_DEPRECATIONS_HELPER you are using Symfony-PHPUnit bridge, correct?
  • there's database interaction and a Doctrine <listener>
  • --testdox has had a major refactoring since v7; the printer itself isn't much slower (a bit with the ANSI-color rendering, but insignificant even on slow laptops)

Perhaps @sebastianbergmann has heard about similar changes in runtime from the community. The PHPUnit self-tests are quite extensive and did not show any significant changes in runtime and resource usage between 7 and 8.

Loading

@VincentLanglet
Copy link
Author

@VincentLanglet VincentLanglet commented Jul 2, 2019

@epdenouden Yes it's a private project for the work. Not sure if I can make a reproductible repository.

Yes, I just have to

  • Run my test in phpunit 8.2 in 30+ seconds
  • Downgrading phpunit from 8.2 to 7.5 (Removing sebastian/type, downgrading sebastian/global-statefrom 3.0.0 to 2.0.0, downgrading phpunit/php-code-coverage from 7.0.5 to 6.1.4)
  • Run my test in phpunit 7.5 in 10 seconds

I do use symfony/phpunit-bridge, I do have database interaction, I don't know for the listener.

Loading

@epdenouden
Copy link
Collaborator

@epdenouden epdenouden commented Jul 2, 2019

@VincentLanglet Thanks for the clarification! I do know there was a lot going on surrounding code coverage. Unfortunately I know very little about this part of PHPUnit.

Some questions that come to mind:

  • do you run the tests with code coverage? (I think not, based on the config)
  • is it any faster when you change the config to colors="false"?
    Maybe there is an undiscovered performance 'leak' in the new prettifier.

Loading

@sebastianbergmann
Copy link
Owner

@sebastianbergmann sebastianbergmann commented Jul 2, 2019

I am not aware of any performance degration of the scale you describe. I would not be surprised if this were related to Symfony Bridge or similar extensions that wrap PHPUnit.

Loading

@VincentLanglet
Copy link
Author

@VincentLanglet VincentLanglet commented Jul 2, 2019

I don't run with code coverage. If I do, it's not wanted.
colors=false does nothing.

I am not aware of any performance degration of the scale you describe. I would not be surprised if this were related to Symfony Bridge or similar extensions that wrap PHPUnit.

So the same version of symfony bridge produce a more performant bridge for phpunit 7 than phpunit 8 ?

Loading

@sebastianbergmann
Copy link
Owner

@sebastianbergmann sebastianbergmann commented Jul 2, 2019

That would be a question for the maintainers of the Symfony Bridge. I do not know whether they have different code paths depending on the version of PHPUnit used.

Loading

@VincentLanglet
Copy link
Author

@VincentLanglet VincentLanglet commented Jul 2, 2019

For information, on circle.ci my test were running in 1-2 seconds with phpunit 7 and now run in 3-5 seconds with phpunit 8.

Loading

@snapshotpl
Copy link

@snapshotpl snapshotpl commented Jul 2, 2019

In my framework-less test suite I see 10% performance degration in phpunit 8 against v7

Loading

@epdenouden
Copy link
Collaborator

@epdenouden epdenouden commented Jul 2, 2019

In my framework-less test suite I see 10% performance degration in phpunit 8 against v7

That is significant. I'm busy with on-demand data providers, giving you back some performance. ;-) Will have a look at comparing 7.x and 8.x with xdebug tracing after that, but it will be a while before I get to it. Any suggestions and help welcome of course. 🔮

Loading

@sebastianbergmann
Copy link
Owner

@sebastianbergmann sebastianbergmann commented Jul 2, 2019

@epdenouden You may want to give @beberlei's Tideways profiler a try while investigating this.

Loading

@epdenouden
Copy link
Collaborator

@epdenouden epdenouden commented Jul 2, 2019

@epdenouden You may want to give @beberlei's Tideways profiler a try while investigating this.

Ohhhh toys tools! Thanks Sebastian :)

Loading

@bizley
Copy link

@bizley bizley commented Jul 17, 2019

We noticed the performance hit as well (~10 times slower) after upgrade from 7 (with Symfony 4.2) to 8 (Symfony 4.3) - PHP version has not been changed (7.2). But running it with n flag (no ini configuration) like

php -n vendor/bin/phpunit

makes it fast again.

Loading

@epdenouden
Copy link
Collaborator

@epdenouden epdenouden commented Jul 17, 2019

@bizley thank you for this bit of information! This provides a good starting point for comparison testing.

Loading

@epdenouden
Copy link
Collaborator

@epdenouden epdenouden commented Jul 17, 2019

@bizley Rereading, you are talking about the -n flag of PHP itself. So most likely the difference lies in different module configuration, opcaches, etc. Did you investigate the local configuration?

Loading

@bizley
Copy link

@bizley bizley commented Jul 17, 2019

Yes, it's for PHP itself.

I haven't investigated it since the configuration remain the same. But I'm happy to help (just let me know what you need).

Loading

@bizley
Copy link

@bizley bizley commented Jul 17, 2019

I did a little experiment and run the test from my local env with PHP 7.3 (previous was dockerized with PHP 7.2) and it was slow again. Then I switched off xdebug (2.7.1) - and tests were fast. Looks like xdebug is the reason for this performance drop. I know, it's expected but it was never so huge.

Loading

@sebastianbergmann
Copy link
Owner

@sebastianbergmann sebastianbergmann commented Jul 18, 2019

Stating the obvious here: a performance degradation is only a performance degredation in PHPUnit if the only difference between two test executions is the version of PHPUnit used. You cannot compare "PHPUnit X without Xdebug" with "PHPUnit Y!=X with Xdebug" and conclude that ""PHPUnit Y!=X" is slower.

Loading

@bizley
Copy link

@bizley bizley commented Jul 18, 2019

Of course you are right! Nevertheless PHPUnit is slower no matter with xdebug or not. Here are my results (tests set the same in all cases, PHP 7.2):

PHPUnit -n flag Time Memory
7.5.14 no 2.82 s 32 MB
7.5.14 yes 716 ms 28 MB
8.2.5 no 29.62 s 34 MB
8.2.5 yes 4.09 s 30 MB

Loading

@epdenouden
Copy link
Collaborator

@epdenouden epdenouden commented Jul 19, 2019

@bizley Thank you for those statistics! Obviously Xdebug makes things a lot slower, however I will investigate deeper after the data provider improvements. That will have me cleaning up quite a bit of old stuff. For example, loading the providers can cause annotation data being read twice, instantiating extra reflection objects and whatnot.

My eye is drawn to the difference in slowing down: without php.ini it is 4x but with it is 10x. Statistics/lies, signal/noise, etc.

Loading

@sebastianbergmann
Copy link
Owner

@sebastianbergmann sebastianbergmann commented Sep 6, 2019

@MichelHartmann, @sebastianfeldmann, and @SenseException have used git-bisect to track down this performance degradation to #3673.

Checking the type of values passed to willReturn() and enforcing that they are compatible with the declared return type of a stubbed method is additional work that was previously not performed. This additional work costs time. However, the degree of performance degradation observed here is unexpected.

Loading

@mrtnzagustin
Copy link

@mrtnzagustin mrtnzagustin commented Mar 5, 2020

Hi there, i have a problem when executing ./vendor/bin/simple-phpunit in a docker testing stage container. Each time i make a build, test are slow cause simple-phpunit dowload phpunit. How can i download it with other command to mantain it in cache?

Loading

@sebastianbergmann
Copy link
Owner

@sebastianbergmann sebastianbergmann commented Mar 5, 2020

Neither simple-phpunit nor Docker are on topic in this repository.

Loading

@crisnunez1
Copy link

@crisnunez1 crisnunez1 commented Jun 22, 2020

Hi, I have the same performance problem but with version 8.5.5, even with very simple testing it can take more than 25 seconds.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants