Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[6.0] Add PHPUnit bootstrap file to allow execution of console commands before a test run #5050

Merged
merged 2 commits into from Jul 30, 2019
Merged

[6.0] Add PHPUnit bootstrap file to allow execution of console commands before a test run #5050

merged 2 commits into from Jul 30, 2019

Conversation

@timacdonald
Copy link
Contributor

@timacdonald timacdonald commented Jun 30, 2019

This PR introduces a PHPUnit watcher bootstrap file that caches the config before the test suite runs for the first time, which in my benchmarks reduces the boot time of the Kernel during a test run by ~50% without any additional work by the developer.

This will only really have an impact on a larger test suite, but personally I think it is a worthwhile thing to consider.

The benchmark test suite ended up with the following durations:

  • Without caching: 9.15s
  • With caching: 4.2s

Caching the config before a test run is good for a couple of reasons:

  1. It's faster.
  2. It is closer reflects what you use in production.

If, for example, you cache your routes in production but try deploy a sneaky closure route, your app is gonna blow up when you go to cache the routes during production. Having this in place will help you catch that kind of thing during testing, either locally or during CI. (you could catch this in CI if you are already calling this command before running the tests).

The PHPUnit listener bootstrap file will cache the config for the test environment before a test is run. Because it is in "User land" it is 100% opt-in. If you don't want it, don't use it. One handy thing about running the cache command in a PHPUnit listener bootstrap file (as opposed to say in a composer script) is that it will already have the environment variables from the phpunit.xml file loaded, so we don't need to read those ourselves.

Notes

  • As this doesn't require any changes to the framework, it is totally possible to just package up, which I'm more than happy to do.

  • The watcher could be kept in core with just the phpunit.xml changes made here.

  • it is possible we could see similar results by caching routes and events, but I haven’t tested this yet. I plan to extend the benchmark repo I setup to also have some feature (http) tests so we can see what kind of impact that might have on a test suite as well.

This PR is a simpler approach to my first run at this: laravel/framework#28998

tests/Listener.php Outdated Show resolved Hide resolved
@timacdonald timacdonald changed the title Add PHPUnit listener to cache config (Alternative approach) Add PHPUnit listener to cache config Jun 30, 2019
@timacdonald timacdonald changed the base branch from master to develop Jun 30, 2019
@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jun 30, 2019

@mfn as per your comment on my first PR - would love you to give this version a spin on your suite of 6,000+ tests.

And to answer your question: There is no need to manually do any config cache / busting. This version caches to a dedicated cache.phpunit.php file (or as specified in the phpunit.xml).

@GrahamCampbell GrahamCampbell changed the title Add PHPUnit listener to cache config [5.9] Add PHPUnit listener to cache config Jun 30, 2019
@mfn
Copy link
Contributor

@mfn mfn commented Jun 30, 2019

I tried it and realized I'm on 5.8 (obviously), had to replace Env::get with env()

But unfortunately, as I thought, it's not measureable with my suite. I also can only run it in a timely manner on CI (in parallel) in the cloud and we all know they don't give you the prime resources for doing that.

Sorry for bringing up hopes.

tests/Listener.php Outdated Show resolved Hide resolved
@driesvints
Copy link
Member

@driesvints driesvints commented Jul 1, 2019

Can't we move the listener to the foundation component in Illuminate\Foundation\Testing? It feels weird to include it in the skeleton.

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jul 1, 2019

@driesvints I think that would probably be a good idea yea (and I mentioned this in my initial comment). I'll send the PR to laravel/framework if you are keen to move forward with this as a thing in core - just don't wanna spam you with PRs to manage.

@mfn
Copy link
Contributor

@mfn mfn commented Jul 1, 2019

The only bad thing with the Listener is, exactly as you said, the framework boot problem.

Not too long ago I used a dedicated listener for a Laravel project and it would not have worked together with your because I too had to boot the framework they.

Eventually I found a different solution for the same problem without using a listener.

Do you think it's possible without that?

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jul 1, 2019

@mfn could you shed light on why your listener would have conflicted with this one?

This listener does boot the framework - but the thing that takes the longest during booting is collating all the config. This listener caches the config on first run, meaning that all subsequent boots are ~50% quicker (as per my benchmarking).

I would think that means your listener would actually become faster because when it boots the framework it doesn't have to collate all the config on the fly. Or am I missing something?

@mfn
Copy link
Contributor

@mfn mfn commented Jul 1, 2019

The Listener was, as in your case, a special one which also booted the framework.

But: it just came to my mind, I'm not using it anymore and I think hardly anyone does so I guess you can just disregard my comment ;)

tests/Listener.php Outdated Show resolved Hide resolved
@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jul 1, 2019

@mfn no worries. if you do think of any specific scenarios where this could conflict (with or without another listener) please do mention it. I'd be happy to run some tests and see if we need to make adjustments if any are needed.

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jul 3, 2019

I've just pushed a new commit to this. You can now specify arbitrary commands to run before the test suite, rather than locking it down to config caching only.

It makes it more generic so devs can define the commands to run before the testsuite runs for the first time.

<listeners>
    <listener class="Illuminate\Foundation\Testing\Listener">
        <arguments>
            <!-- Define artisan commands to execute before the test suite runs -->
            <string>config:cache</string>
            <string>route:cache</string>
            <string>event:cache</string>
            <string>some-3rd-party-package:cache-stuff</string>
        </arguments>
    </listener>
</listeners>

@m1guelpf
Copy link
Contributor

@m1guelpf m1guelpf commented Jul 3, 2019

Isn't Listener a really gwneral name? I'd rather have a CacheForTestsListener or another name that describes what this is doing better

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jul 3, 2019

Agreed. I think PreRunListener or SetupListener or something along those lines would be better. I’ve been more focused on the implementation than those details up until now so if anyone has any other suggestions, please do share.

I also figured Taylor, if he wants this, would know what he’d want to call it as well.

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jul 3, 2019

...although if this is the Laravel listener, I dont think it would be the worst thing to call it Listener. Just like the TestCase is called TestCase instead of BootsFrameworkTestCase.

But I don’t have strong opinions on the naming either way TBH

@timacdonald timacdonald changed the title [5.9] Add PHPUnit listener to cache config [5.9] Add PHPUnit listener to execute console commands before a test run Jul 3, 2019
@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jul 4, 2019

Someone just suggested this could be done in a bootstrap file...and I think I was too deep down the rabbit hole to realise that is probably the better way to go compared to a listener.

Just ship a tests/bootstrap.php or bootstrap/tests.php file, reference it in the phpunit.xml bootstrap tag and then you can provide classic Laravel style documentation as guidance as well. Again it is all in "user land" so the dev is in full control if they want to take advantage of it.

pretending these comments were not terrible...it could be something like this...

<?php

use Illuminate\Contracts\Console\Kernel;

require_once __DIR__.'/../vendor/autoload.php';

/*
|--------------------------------------------------------------------------
| Bootstrap the testing environment
|--------------------------------------------------------------------------
|
| You have the option to specify console commands that will execute before your
| test suite is run. Caching config, routes, & events may improve performance
| and bring your testing environment closer to production.
|
*/

$commands = [
    'config:cache',
    'route:cache',
    'event:cache',
];

$app = require __DIR__.'/../bootstrap/app.php';

$console = tap($app->make(Kernel::class))->bootstrap();

foreach ($commands as $command) {
    $console->call($command);
}

@taylorotwell
Copy link
Member

@taylorotwell taylorotwell commented Jul 5, 2019

Yeah I think that makes decent sense probably.

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jul 5, 2019

I've just pushed changes to implement to bootstrap file instead of the Listener.

I've commented out the route:cache command as the routes that come in the skeleton are not cacheable. Commenting it out means that the 2 tests in the skeleton still pass out of the box.

@driesvints
Copy link
Member

@driesvints driesvints commented Jul 5, 2019

My two cents: I think the file would be better lived under tests/bootstrap.php since that's a common location in most PHP apps I've seen so far.

I also believe the current bootstrap/app.php file would be better placed as ./bootstrap.php and the cache directory as a top-level directory or underneath storage but that's a different topic :)

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jul 7, 2019

We've actually found in our initial feature test benchmarks that caching the routes can actually slow down the test suite. Looks like it is due the the performance of unserialize. Our initial benchmarks were against a test app with 1,000 routes. I'm going to run some more tests against different route numbers through the week. Looks like most apps, according to this Twitter poll have far less routes.

Gonna do some more digging into this throughout the week and will report back regarding route caching.

Happy to close this and re-open once I have a better overall understanding of if route caching is a worthwhile thing to include in the bootstrapping process.

@m1guelpf
Copy link
Contributor

@m1guelpf m1guelpf commented Jul 7, 2019

Now that I think about it, this would also break tests which register routes on runtime (I think that's a common approach for testing middleware

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jul 7, 2019

@m1guelpf runtime defined routes continue to work if the routes are cached, e.g. the following test passes with routes cached before the test run:

public function test_runtime_routing()
{
    Route::get('my-test-route', function () {
        return 'expected';
    });

    $response = $this->get('my-test-route');

    $this->assertSame('expected', $response->content());
}

Let me know if this isn't what you are referring to though. I might have misunderstood.

@driesvints
Copy link
Member

@driesvints driesvints commented Jul 8, 2019

@timacdonald can you move the bootstrap file to tests/bootstrap.php?

@GrahamCampbell GrahamCampbell changed the title [5.9] Add PHPUnit bootstrap file to allow execution of console commands before a test run [6.0] Add PHPUnit bootstrap file to allow execution of console commands before a test run Jul 28, 2019
@taylorotwell
Copy link
Member

@taylorotwell taylorotwell commented Jul 29, 2019

@adrianb93 what did you actually do in the file to achieve those results?

@adrianb93
Copy link

@adrianb93 adrianb93 commented Jul 29, 2019

@adrianb93 what did you actually do in the file to achieve those results?

Nothing crazy. They're both apps that work with many databases. It was just moving most of the setup work to the start of the suite.

  • 23-minute suite: It needed to copy a couple of database schemas and do some seeding for some critical third-party connections we need to integrate with that are subject to schema change. This was a slow setup for each test.
    • No, this integration has no API or staging environment.
    • Yes, we still develop our app as if there is in case we can switch to another the third-party or if there are improvements.
    • It's less than ideal integration work, but it is what it is.
  • 3-minute suite: This is a multi-tenant database app. The bootstrap provisions a default testing tenant database before running the suite.
    • If you fake the tenant db handler, the default connection is used.
    • Provisioning time seemed reasonable in this app but there was time to be saved here.
    • It was also valuable to use register_shutdown_function in the bootstrap to delete any databases starting with testing_tenant_* in case a test didn't call deprovision on a tenant. This is cleanup work that was once done manually.

In both cases, each test starts transactions in each test setup and rolls back in the before application destroy.

These aren't the first apps I've worked on that have built up this sort of unavoidable DB cruft. The bootstrap file helped make things better for these apps.

@taylorotwell taylorotwell merged commit 8ca5622 into laravel:develop Jul 30, 2019
1 check passed
@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jul 31, 2019

Really appreciate so many people taking the time, here and on other forums, to give me feedback on this PR ❤️ Hope it comes in handy for you and others.

@patrickbrouwers
Copy link
Contributor

@patrickbrouwers patrickbrouwers commented Jul 31, 2019

@timacdonald awesome job! Working great for me when running from the command line (+-2s faster), however having some issues getting it work when running individual test via PHPStorm interface. Anyone who has managed to get that working?

@cbaconnier
Copy link

@cbaconnier cbaconnier commented Jul 31, 2019

By the way, if anybody wants to run something after the suite finishes, add this to the end of the booststrap file:

// Will run once the entire test suite has finished
register_shutdown_function(function () use ($console) {
    // Figured this is a helpful example for those on a Laravel version below 5.8
    $console->call('config:clear');
});

Thanks for this @adrianb93 I'm on Laravel 5.8 and I was experimenting 419 errors after the tests.

Edit: I just realised that env variables are correctly set when I dump them during tests but this is still config.php, services.php, ... that are in use instead of config.phpunit.php.
I had to remove the variables from phpunit.xml and put them in my .env.testing file. Now it's working without having to call 'config:clear'.


however having some issues getting it work when running individual test via PHPStorm interface. Anyone who has managed to get that working?

@patrickbrouwers I'm on Ubuntu with PHPStorm 192.5728.108 and run them through docker. For me, they still work individually with no extra configuration.

@bastien-phi
Copy link

@bastien-phi bastien-phi commented Jul 31, 2019

@patrickbrouwers Same problem here, I can't run individual tests. I got this error :

In PackageManifest.php line 168:
                                                               
  The bootstrap/cache directory must be present and writable.  

Same for you ? I'm investigating for a solution...

@patrickbrouwers
Copy link
Contributor

@patrickbrouwers patrickbrouwers commented Jul 31, 2019

@bastien-phi yes that's the error I'm getting too. I got it to work when I specify the absolute path in the env settings for the cache files.

@bastien-phi
Copy link

@bastien-phi bastien-phi commented Jul 31, 2019

@patrickbrouwers I found the exact same solution right now !
Not very convenient when sharing the phpunit.xml file with teammates IMO ...

@patrickbrouwers
Copy link
Contributor

@patrickbrouwers patrickbrouwers commented Jul 31, 2019

@bastien-phi I personally always gitignore the phpunit.xml and ship a phpunit.xml.dist

@bastien-phi
Copy link

@bastien-phi bastien-phi commented Jul 31, 2019

@patrickbrouwers I will just update tests/bootstrap.php to update environment variables with absolute path before calling the framework :

$envVars = ['APP_CONFIG_CACHE', 'APP_SERVICES_CACHE', 'APP_PACKAGES_CACHE', 'APP_ROUTES_CACHE', 'APP_EVENTS_CACHE'];
foreach ($envVars as $envVar) {
    $_ENV[$envVar] = __DIR__ . '/../' . $_ENV[$envVar];
}

@timacdonald Any other idea as APP_CONFIG_CACHE (and others) are defined by default as absolute path in Illuminate/Foundation/Application ?

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jul 31, 2019

@cbaconnier as per my comment here: #5050 (comment) have you set the env vars correctly in the phpunit.xml? I found most people were skipping this step and thus having the issue you are where the cached values are overriding the normal config files.

The env values need to be set differently in 5.8. Instead of <server you need to use <env e.g.

- <server name="APP_CONFIG_CACHE" value="bootstrap/cache/config.phpunit.php"/>
+ <env name="APP_CONFIG_CACHE" value="bootstrap/cache/config.phpunit.php"/>

@cbaconnier
Copy link

@cbaconnier cbaconnier commented Jul 31, 2019

@timacdonald Oh... 😅 Thanks! That was it

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Jul 31, 2019

@cbaconnier no worries, that has tripped a few people up.

@bastien-phi not sure. Can you put your setup in here so we can try and work out why it isn't working for you? Are you calling it from inside PHPStorm as well? If so, could you try on the command line as well and see if it is any different? Would be good to work what part of your stack is causing issue for ya.

@bastien-phi
Copy link

@bastien-phi bastien-phi commented Jul 31, 2019

@timacdonald I created a repo from a fresh install of laravel, with the phpunit bootstrap file : laravel-phpunit-bootstrap

I updated the bootstrap file with a "patch" in order to make it work properly.

Before the patch, ./vendor/bin/phpunit --config phpunit.xml works properly but calling phpunit from PHPStorm in a single file does not work.
PHPStorm calls the command /usr/bin/php /path/to/laravel-phpunit-bootstrap/vendor/phpunit/phpunit/phpunit --configuration /path/to/laravel-phpunit-bootstrap/phpunit.xml Tests\Unit\ExampleTest /path/to/laravel-phpunit-bootstrap/tests/Unit/ExampleTest.php --teamcity and I get

In PackageManifest.php line 168:
                                                               
  The bootstrap/cache directory must be present and writable.  

Calling this command from the project root, the tests goes well.

The difference between the two calls is that from inside PHPStorm, current working directory is the directory when the test file belongs (in that case tests/Unit), and from the command line current directory is the project root.

If calling the command from another directory (let's say tests), the command line fails too :

$ cd tests && /usr/bin/php /path/to/laravel-phpunit-bootstrap/vendor/phpunit/phpunit/phpunit --configuration /path/to/laravel-phpunit-bootstrap/phpunit.xml Tests\Unit\ExampleTest /path/to/laravel-phpunit-bootstrap/tests/Unit/ExampleTest.php --teamcity

In PackageManifest.php line 168:
                                                               
  The bootstrap/cache directory must be present and writable. 

As the environment variables are relative paths, the application tries to find the directory bootstrap/cache relatively to current working directory which cause the fails.

The patch updates relative paths to absolute path and everything goes well. Unfortunately, this is far from being a clean fix...

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Aug 1, 2019

@bastien-phi I can confirm it works without the patch in vim-test and sublime-phpunit, but unfortunately I do not have a copy of PHPStorm to help debug this directly. But a google lead me to this: https://www.jetbrains.com/help/phpstorm/run-debug-configuration-phpunit.html#commandLine

If I am understanding that correctly, they are saying the CWD is always set to the project root.

By default, the field is empty and the working directory is the root of the project.

Can you check that option out (and possibly whatever else is in there), confirm you don't have a value in there, and maybe have a play and see if there is anything in there you need to change to get it working?

@bastien-phi
Copy link

@bastien-phi bastien-phi commented Aug 1, 2019

@timacdonald This option is kept empty and the CWD is not set to the project root but to the directory where the test class belongs :

$ /usr/bin/php /path/to/laravel-phpunit-bootstrap/vendor/phpunit/phpunit/phpunit --configuration /path/to/laravel-phpunit-bootstrap/phpunit.xml Tests\Unit\ExampleTest /path/to/laravel-phpunit-bootstrap/tests/Unit/ExampleTest.php --teamcity
current working directory : /path/to/laravel-phpunit-bootstrap/tests/Unit

In PackageManifest.php line 168:
                                                               
  The bootstrap/cache directory must be present and writable. 

It is possible to explicitly set CWD as project root for phpunit in PHPStorm with Run Configuration Templates. Be sure to remove all existing configurations first.
Capture d’écran de 2019-08-01 09-00-05
It works this way.

But IMO, there is still a problem because CWD has to be project root to run the tests. Before this feature, it was possible to launch tests from another location.

The problem does not comes from the bootstrap file itself but from the fact that the files are defined with relative paths in the phpunit.xml. Currently I find only 3 solutions :

  1. remove the path definitions in phpunit.xml
  2. update the path definitions in phpunit.xml with absolute paths
  3. explicitlty update CWD in tests/bootstrap.php to set it to project root.

1 is not a good idea at all !
With 2, there is a need to ship a phpunit.xml.example, gitignore phpunit.xml and manually update the paths.
My favorite solution is 3 but I don't know if there could be side effects.

<?php

use Illuminate\Contracts\Console\Kernel;

chdir(__DIR__.'/../');

require_once __DIR__.'/../vendor/autoload.php';

/*
|--------------------------------------------------------------------------
| Bootstrap the testing environment
|--------------------------------------------------------------------------
|
| You have the option to specify console commands that will execute before your
| test suite is run. Caching config, routes, & events may improve performance
| and bring your testing environment closer to production.
|
*/

$commands = [
    'config:cache',
    'event:cache',
    // 'route:cache',
];

$app = require __DIR__.'/../bootstrap/app.php';

$console = tap($app->make(Kernel::class))->bootstrap();

foreach ($commands as $command) {
    $console->call($command);
}

@stevelacey
Copy link

@stevelacey stevelacey commented Aug 4, 2019

Laravel 5.8 with 171 tests 5-10% improvement 👏️

Before:

Time: 53.27 seconds, Memory: 68.25 MB
Time: 48.13 seconds, Memory: 68.25 MB
Time: 51.73 seconds, Memory: 68.25 MB
Time: 52.96 seconds, Memory: 68.25 MB
Time: 48.15 seconds, Memory: 68.25 MB

After:

Time: 42.16 seconds, Memory: 70.25 MB
Time: 42.26 seconds, Memory: 70.25 MB
Time: 47.57 seconds, Memory: 70.25 MB
Time: 43.92 seconds, Memory: 70.25 MB
Time: 44.53 seconds, Memory: 70.25 MB
OK (171 tests, 1135 assertions)

🙂️👌️

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Aug 4, 2019

nice @stevelacey! Thanks for sharing. Glad to see a little perf bonus there for ya

@foremtehan
Copy link

@foremtehan foremtehan commented Aug 5, 2019

@timacdonald I replaced your codes and it improved ~10% of test suit which is nice, But i dont know why i'm getting General error: 1 no such table: users on the browser after i using your codes , and it gone when i clear cache folder any idea?

@stevelacey
Copy link

@stevelacey stevelacey commented Aug 5, 2019

@foremtehan I just had this same problem, I suspect it’s because 5.8 doesn’t have the separate caches per env logic thats coming in 6.0, and thus Laravel sees cache and ignores the .env? And thus your DB_NAME is wrong, and thus no such table.

@timacdonald is there something else we should patch in?

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Aug 6, 2019

@foremtehan @stevelacey see #5050 (comment)

You will also want to run just once right now php artisan config:clear to make sure you get the test values out of your local config cache.

p.s. @foremtehan ~10% is a nice speed improvement 🎉

@driesvints
Copy link
Member

@driesvints driesvints commented Sep 5, 2019

@timacdonald this PR is causing some issues:

laravel/framework#29862
#5091

We'll need to figure out solutions for these fast or else it might be best to revert this PR.

@bastien-phi
Copy link

@bastien-phi bastien-phi commented Sep 5, 2019

#5071 does the job

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Sep 6, 2019

@driesvints digging into this now. Will respond in the relevant threads to keep discussions together.

@foremtehan
Copy link

@foremtehan foremtehan commented Sep 6, 2019

I had exact problem, I just comment these out in phpunit.xml and everything back to work :

<env name="APP_CONFIG_CACHE" value="bootstrap/cache/config.phpunit.php"/>
<server name="APP_SERVICES_CACHE" value="bootstrap/cache/services.php"/>
<server name="APP_PACKAGES_CACHE" value="bootstrap/cache/packages.php"/>
<server name="APP_ROUTES_CACHE" value="bootstrap/cache/routes.phpunit.php"/>
<server name="APP_EVENTS_CACHE" value="bootstrap/cache/events.phpunit.php"/>

@timacdonald
Copy link
Contributor Author

@timacdonald timacdonald commented Sep 6, 2019

@foremtehan that’ll mean that your testing env is being cached in your normal cache files, so you probably hit some weird bugs when using your app locally. Once laravel/framework#29890 is tagged it should be fixed and you’ll be right to restore everything

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet