[8.x] Memory leak on testing

[DarkGhostHunter]

• Laravel Version: 8.6.3 / Framework 8.64.0
• PHP Version: 8.0.11
• Database Driver & Version: NA

Description:

On extensive PHPUnit testing, the application suddenly stops with the following message:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in P:\[REDACTED]\example-app\routes\web.php on line 1

After further investigation, we couldn't pin-point at time of writing the culprit on application-code. Our tests failed on different places, but this line was the most common. We couldn't end a single test run without a memory leak.

We suspect that something is kept while refreshing the application with $this->createApplication(), as we could replicate the same behaviour using a loop.

Using --parallel gives us no memory leak errors, but some other weird errors exclusive of parallel testing and we won't/can't fix.

Steps To Reproduce:

Just simply make a new Laravel Framework in your PC.

composer new-project laravel/laravel example-app

cd example-app

Edit the included tests/Feature/ExampleTest.php first test with a loop refreshing the application.

<?php

namespace Tests\Feature;

use Illuminate\Foundation\Testing\RefreshDatabase;
use Tests\TestCase;

class ExampleTest extends TestCase
{
    /**
     * A basic test example.
     *
     * @return void
     */
    public function test_example()
    {
        for ($i = 0; $i < 1000; ++$i) {
            $this->createApplication();
        }
    }
}

Once done, call artisan to test (or PHPUnit directly).

php artisan test

You should receive this line after a short while:

PS P:\[REDACTED]\example-app> php artisan test
Warning: TTY mode is not supported on Windows platform.

   PASS  Tests\Unit\ExampleTest
  ✓ example

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in P:\[REDACTED]\example-app\routes\api.php on line 1

You should get the same problem, at least on Windows. We haven't tested on MacOS or Linux because we're poor AF and noob AF. We couldn't decipher the Xdebug profiler snapshots we made under that test because we're dumb AF too.

[DarkGhostHunter]

After some overnight digging, I tried to make some barebone calls by just requiring the bootstrap file manually, instantiating the Console Kernel, and bootstraping the app.

for ($i = 1; $i <= 10000; ++$i) {
    $app = include __DIR__.'/../../bootstrap/app.php';
    $app->make(Kernel::class)->bootstrap();
}

This still makes the test suite crash around 128MB, which is the limit set to PHP globally. But, if I don't use bootstrap() on the Console Kernel, the memory sits around 30MB or so, reaching 10,000 loops in a few seconds.

for ($i = 1; $i <= 10000; ++$i) {
    $app = include __DIR__.'/../../bootstrap/app.php';
    $app->make(Kernel::class);
}

Then, I decided to setup the memory limit 512MB while bootstraping the Console Kernel, and after some rounds, it hit the memory limit, so I can confirm it's a memory leak by just bootstraping the Console Kernel constantly.

While using $app=null or unset($app) do absolutely nothing, probably because GC, using flush() on the container after bootstrapping only delays the inevitable:

for ($i = 1; $i <= 10000; ++$i) {
    $app = include __DIR__.'/../../bootstrap/app.php';
    $app->make(Kernel::class)->bootstrap();
    $app->flush();
}

I'll try later to check if there is any Service Provider that leaks from being booted again and again, even after the container flushes itself.

[DarkGhostHunter]

I can confirm that disabling \Illuminate\Foundation\Bootstrap\BootProviders::class on the bootstrapers array in Illuminate\Foundation\Console\Kernel.php keeps the application memory consumption steady (as long it's flushed accordingly).

    /**
     * The bootstrap classes for the application.
     *
     * @var string[]
     */
    protected $bootstrappers = [
        \Illuminate\Foundation\Bootstrap\LoadEnvironmentVariables::class,
        \Illuminate\Foundation\Bootstrap\LoadConfiguration::class,
        \Illuminate\Foundation\Bootstrap\HandleExceptions::class,
        \Illuminate\Foundation\Bootstrap\RegisterFacades::class,
        \Illuminate\Foundation\Bootstrap\SetRequestForConsole::class,
        \Illuminate\Foundation\Bootstrap\RegisterProviders::class,
//        \Illuminate\Foundation\Bootstrap\BootProviders::class,
    ];

This array is called when bootstraping the Console Kernel to prepare the application before adding the rest of services into the container, which is done by the last two Service Providers.

Pne provider being booted is causing havoc and doesn't "frees itself" appropriately.

[DarkGhostHunter]

Well, after checking manually each Service Provider (tested larger groups until I found the group that contained the culprit, I ended up with the \App\Providers\RouteServiceProvider.

Even when tested in isolation, the RouteServiceProvider would gladly take all memory available. The boot() method of this provider is simple: sets some Rate Limits, and then register the routes of the application.

/**
     * Define your route model bindings, pattern filters, etc.
     *
     * @return void
     */
    public function boot()
    {
        $this->configureRateLimiting();

        $this->routes(function () {
            Route::prefix('api')
                ->middleware('api')
                ->namespace($this->namespace)
                ->group(base_path('routes/api.php'));

            Route::middleware('web')
                ->namespace($this->namespace)
                ->group(base_path('routes/web.php'));
        });
    }

    /**
     * Configure the rate limiters for the application.
     *
     * @return void
     */
    protected function configureRateLimiting()
    {
        RateLimiter::for('api', function (Request $request) {
            return Limit::perMinute(60)->by(optional($request->user())->id ?: $request->ip());
        });
    }

Disabling the configureRateLimiting() did nothing, but it was until I disabled all routes registration when the memory leak was fixed.

Meddling with how the routes are created, the Router group() method accepts both a file or a closure to register "child" routes.

First, I tried to make a bunch of routes using just static closures returning void, something like this:

Route::middleware('web')
    ->namespace($this->namespace)
    ->group(static function(): void {
        Route::get('foo', fn(): void { });
        Route::get('bar', fn(): void { });
        Route::get('baz', fn(): void { });
       // ...
});

This made no difference. It radically changed my approach and decided to make a sea of files which the Route group() would receive, starting with base_path().

Route::middleware('web')
    ->namespace($this->namespace)
    ->group(base_path('routes/test.php')

// routes/test.php
Route::group(__DIR__ . 'test/foo.php');

// routes/test/foo.php
Route::group(__DIR__ . 'bar.php');

// You catch my drift...

Memory leak found

[DarkGhostHunter]

Thinking that caching the routes with php artisan route:cache would fix or help this problem, but it didn't work, as the memory leak presented itself again.

If I remove all routes, I can easily get 5x times more runs, so it seems that something doesn't flush and its kept there. Again, everything goes AWOL once bootstrap() on the Console Kernel is hit. It seems also related to the config itself.

There are also Deferred Providers that are run after the other registered services providers are booted, which may pose another point of contention. I can confirm there is no memory leaks if I comment some Facades, disable loading the Deferred Providers, and only leaving from the Console Kernel bootstrappers:

• \Illuminate\Foundation\Bootstrap\LoadEnvironmentVariables,
• \Illuminate\Foundation\Bootstrap\LoadConfiguration

Once I enable HandleExceptions, I see the memory slowly rising to 50MB at 10,000 iterations. That's kind of acceptable. The RegisterFacades doesn't do much along the SetRequestForConsole. Is after RegisterProviders and BootProviders I see more aggressive memory consumption, growing up each loop, until depletion, and not counting the Deferred Provided which I commended from loading.

So, I'm back to square one. It's not that the Routes are a huge memory leak, but rather, something in PHP is leaking memory each time the application is bootstrapped. Kinda of out ideas.

[driesvints]

<?php

namespace Tests\Feature;

use Illuminate\Foundation\Testing\RefreshDatabase;
use Tests\TestCase;

class ExampleTest extends TestCase
{
    /**
     * A basic test example.
     *
     * @return void
     */
    public function test_example()
    {
        for ($i = 0; $i < 1000; ++$i) {
            $this->createApplication();
        }
    }
}

This isn't a realistic example as it's only expected to run the createApplication once when creating a test.

Look, I appreciate it that you've been writing this down in such length but if there isn't an issue with a real life scenario then I don't think there's an actual problem here. This is also the first report we've ever gotten about this. If you really believe there's an issue or memory leak and you can find a solution then feel free to PR a fix.

I'm moving this to discussions in the meantime.

[christianmarx]

This comment doesn' hit the point: this is just an example to show the problem.
In our case we have over 2600 unit tests and each test starts with a createApplication().
And we run in a memory leak as well.

[pxlrbt]

We also ran into this issue today hitting the 128MB memory limit with 225 tests. With a basic Laravel install the memory increase is quite slow (2MB every 20 tests). Registering event listeners and routes increases the memory consumption faster.

Currently we just increased the memory limit for our CI.

[vpratfr]

We are running into the same issue since the beginning in our project. We have a large test suite, and after around 150 test runs, we run out of memory.

We too found out that the issue was in the application's RouteServiceProvider, more specifically when loading the routes from our files.

All other application service providers are disabled. When enabling all except the RouteServiceProvider, we can run the test without any issue.

There is something not getting released properly when setting up the routes.

This is a REAL LIFE scenario, app is in production. Problem exists at least since 2018 on our side, this is not specific to Laravel 8.

[donnysim]

I'm not sure really how relevant this actually is in real world applications because if we take the loop as a sample and create 1000 actual tests, it just works with memory usage of 16MB, it runs out of memory only on this sample where you bootstrap it multiple times in the same test, if we really want to dig into this then the first culprit is:

framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php

Lines 45 to 47 in a9b3fb2

	set_error_handler([$this, 'handleError']);
	set_exception_handler([$this, 'handleException']);

set_error_handler and set_exception_handler stack the handlers, so each createApplication registers a new handler that is never restored.

I'm really not sure if this is worth going deeper unless real case scenario is presented, which points to the framework being at fault and not some application code that is never cleaned up.

[vpratfr]

What do you call a real case scenario?

We have a test suite with 3000+ tests, 4 year old project. Doesn't that represent a real use case?

Our test suite fails after around 150 tests have run. We cannot complete the whole test suite in a single shot.

I am not even talking about the example given to hunt the culprit, just real world tests testing real code.

Commenting out the RouteServiceProvider allows a much better result.

That does not mean the framework is at fault, but for sure something is not right there.

I have found quite a few reports which seem to point at either leaks from PHP, from laravel or from phpunit. Quite difficult to get an idea of what could be going on.

[donnysim]

I can understand a memory leak because of the increasing 2MB each couple of tests, I might take a deeper look into that, but failing on 150 etc. seems too strange to be a framework problem when it's increasing only so slightly with every nth test so 150 tests should still have plenty of memory left (I'm not discarding the possibility of a framework memory leak) and more of a application code or maybe some used library problem which is why this needs some real case example representing the problem, because from what's given here is not sufficient to get an idea of which part exactly might be impacting it because there are no leads to go on, at this point it can be anything that is not used in the clean installation and we're just chasing wild goose, and memory leaks are problematic to pinpoint because they can stop anywhere.

[donnysim]

Ok, so there are what we could call a memory leak:

framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php

Lines 45 to 49 in a9b3fb2

	set_error_handler([$this, 'handleError']);
	set_exception_handler([$this, 'handleException']);
	register_shutdown_function([$this, 'handleShutdown']);

nowhere in the framework I've found that it cleans them up, no restore_error_handler or restore_exception_handler calls, in addition register_shutdown_function does not have a cleanup function in php so it will always keep whatever was passed as an argument in memory, in this case the HandleExceptions will be created on each test and kept in memory forever, it could've been worked around somewhat by adding

restore_error_handler();
restore_exception_handler();

in tearDown but register_shutdown_function will still prevent it from cleaning up and there is no way to avoid it without rethinking how the framework registers these so that it happens only once. I do think there are more small scattered weird behaviors but it's hard to tell if it's php behavior, phpunit or the framework? e.g. container instance method bumps memory usage where the same written as singleton with callback doesn't, but at the same time some make(some::class) bumps memory but if you do it twice it goes down.

Note that all "tests" were executed with forced garbage collection after each test to make sure they report the same usage every time. My eyes are bleeding already so going to end here.

[OrkhanAlikhanov]

@donnysim I also thought that the leak is coming from not calling restore_x_handler functions, which is actually true, but I also found out that as soon as I disabled facade/ignition in the bootstrap/cache/packages.php, The memory consumption of my tests dropped from ~114 MB to ~40 MB. I switched to spatie/laravel-ignition which is now default in laravel. There is no more that amount of leak.

I was using this snippet in the tests/TestCase.php to monitor memory usage:

protected function tearDown(): void
{
    parent::tearDown();

    gc_collect_cycles();
    gc_collect_cycles();
    dump('Memory in use: (' . ((int) (memory_get_usage() / 1024 / 1024)) . 'M)');
}

[vpratfr]

I got rid of quite a lot of leaking memory by removing some large config files we had in the config/ directory.

I am wondering if the configuration repository also does not keep some stuff in between tests ? Or if that is simply a side effect from some other leak somewhere else.

[donnysim]

@OrkhanAlikhanov ooh, that's nice to know. The exception handler was partially fixed to keep only single Application instance in memory so that saved some memory, but because of possible issues with Octane and Vapor the HandleExceptions is still kept in memory for every test you run, so 1k tests = 1k HandleExceptions instances in memory, fixing that can drop ~20MB of memory for 10k tests, But most likely Octane needs to be updated and I think they didn't want to get into that. I'm not familiar with how Octane does things so can't tell if it actually has a memory leak because of this.

What I found is that PHPUnit is really hard to debug memory with, because it in itself keeps a lot of objects in memory before actually starting any test so the initial memory is jumping depending by the number of tests available - it creates a new test class instance for each test, so UserTest having 1k test declarations/function will result in 1k UserTest class instances before it actually starts processing them, which I'd say is more of a inefficiency from the PHPUnit side, the more tests you have the higher the starting memory will be. I thought of trying to replicate something like a test case without all the PHPUnit bootstrapping, but that's an undertaking in itself to find what where and when needs to be executed to match current behavior which not sure I want to get into, at least for now.

For what else can eat up memory - everything, it's hard to tell what other static variables there are that are not reset between tests or state that lingers and the tooling is not making it easier to discover them. I believe I saw an issue about middleware also keeping it's "skipCallbacks" between tests, but that's relevant only if someone registers them.

[donnysim]

So kind of did mostly extract the logic needed for testing and the result, at least for the OOB experience (L9.1, without executing a request), only HandleExceptions has memory leak without handler restore and the shutdown function. What else I found is PHP is weird and you don't want anonymous functions in a plain file that you include many times 😅

Route::get('/', function () {
    return view('welcome');
});

believe it or not, including this file over and over again for each test actually is a memory leak. Making it static function makes no difference. Also, it has nothing to do with Laravel, if you convert it to (the whole file is just a function)

<?php
function () {};

same behavior persists. It somehow for some reason cannot garbage collect anonymous functions created in a plain file. Convert it to

Route::get('/', [SomeController::class, 'someMethod']);

and the memory issue is solved. Now take into account all the group, channels, console, api routes that contain functions and it adds up. I'm honestly baffled by this.

[stetim94]

that cuts memory usage for my test-suite in half (from 450MB to 200MB), but still there are more issues given parallel tests brings down the memory usages even more (to 50MB)

Thank you for this suggestion :)

[donnysim]

Parallel testing uses less memory because it splits the test cases, ignoring the unsolved error handler instance problem, I mentioned before about the PHPUnit creating instances of the test class for each test method and keeping them all in memory, with parallel testing it's just a smaller number of what it has to keep resulting in lower memory usage and also because it has to run less amount of tests, there are less instances of laravel error handler created that persist between all tests. So all these impact how it is reflected in memory. Unfortunately there's nothing else we can really do about this from laravel side (again, ignoring the unsolved error handler persistence because of possible impact on Octane and Vapor etc.).

[OrkhanAlikhanov]

Good to know that it reduced memory usage 🙂 But do not forget clearing the cache during development, you may update routes and the changes will not be picked up if cached. I'd experienced that.

Can you also try commenting out \Illuminate\Foundation\Bootstrap\HandleExceptions:class from $bootstrappers array in vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php to see if that reduces memory as well?

[donnysim]

The route memory problem could possibly be circumvented by registering them directly in a service provider instead of including a routes file, that way PHP has a reference when to clean it up because it's more attached to the class rather than global scope, but all libraries and such that also include route files the default way and uses functions will still leak some memory.

[stetim94]

I am well aware of the route caching, I cleared the route cache afterwards. But I could use route caching as part of the pipeline.

Awesome, this certainly helped me understand the problem a lot better. This will certainly help me to bring memory down.

Thank you guys 👍

[donnysim]

Seems like there is an old PHP issue https://bugs.php.net/bug.php?id=76982 that was partially fixed in PHP 8.1

[OrkhanAlikhanov]

One of the comments in that thread says:

Seems to be fixed in PHP 8.1, but only when opcache is enabled (which is not by default for CLI).

[ghost]

Had this issue with a large set of scenarios via dataProvider.

    protected function assertPasswordPass($expectedResult, $passwordValue)
    {
        $validationRule = new Password();

        $this->assertEquals($expectedResult, $validationRule->passes('attribute', $passwordValue));
    }

    public function valid_password_scenarios()
    {
        $numbers = str_split('1234567890');
        $specialChars = str_split('@$!%*#?&-_.,!"#$€¢£¥µ%&§°\'()*+,-./:;<=>?@[\]^_`´{|}~½¼¾²³');
        $scenarios = [];
        foreach ($numbers as $number) {
            foreach ($specialChars as $specialChar) {
                $scenarios['number_'.$number.'_char_'.$specialChar] = [$number, $specialChar];
            }
        }

        return $scenarios;
    }

    /**
     * @test
     * @dataProvider valid_password_scenarios
     */
    public function it_pass_valid_passwords($number, $specialChar)
    {
        $this->assertPasswordPass(true, "Pas${number}sw${specialChar}ord");
        $this->assertPasswordPass(true, 'Pa1sswor!d');
        $this->assertPasswordPass(true, 'Pa1sswor!d');
    }

Maybe it is the result itself, that takes large amount of space?

[ghost]

ok, our runner had a low memory setting, but parallel still not working here.

[brysonreece]

Jumping in here to add that we've been experiencing this same issue lately using the latest release at the time of writing (v10.42.0) and PHP 8.2.15. In a fresh application, we can craft a test case that reinstantiates the app and flushes it 1K times, revealing the same gradual 2MB increase mentioned by @pxlrbt. Unfortunately, none of the suggested workarounds mentioned here has any effect.

In our real-world codebase, this presents itself as an eventual out-of-memory error about 2/3 through our test suite of ~1500 files.

[donnysim]

The exception handler leak is seemingly fully fixed in Laravel 11 but won't be ported to 10. For other leaks would need a repo to reproduce as there's too many variables on what could leak. Make note that all test classes are kept in memory by phpunit until all tests complete so you also have to cleanup any class variables setUp during the tests on tearDown to reduce memory consumption. And do note that each test in a class is an instance of new class, so having 1 file with 10 tests = 10 class instances. Also 1 test with data provider that generates 10 entries = 10 class instances that already includes the generated data.

[shehi]

After upgrading to Laravel 11, all my Feature tests stopped working:

With out-of-the-box PHPUnit 10.5, tests just get stuck. CPU usage at 100%, no memory increase in System Monitor.

Upgrading to PHP 11, I am getting Out-of-memory errors right away.

Is anyone experiencing anything similar after upgrading to L11?

[shehi]

After upgrading to Laravel 11, all my Feature tests stopped working:

With out-of-the-box PHPUnit 10.5, tests just get stuck. CPU usage at 100%, no memory increase in System Monitor.

Upgrading to PHP 11, I am getting Out-of-memory errors right away.

Is anyone experiencing anything similar after upgrading to L11?

newrelic exception handler was the culprit. removed.

also don't forget to rename your schema dump: mine became mariadb-schema.sql (was: mysql-schema.sql).

[bartgloudemans]

I've encountered the same issue with New Relic and PHPUnit 11. You do not necessarily need to remove New Relic.
Instead you need to force New Relic to see the multiple tests as multiple transactions. By default it sees the whole long-running process as one transaction with memory exhaustion as a result.

See this issue for the concept of the solution:
https://docs.newrelic.com/docs/apm/agents/php-agent/troubleshooting/performance-issues-long-running-task/

In practice define the following in your phpunit bootstrap file

if (extension_loaded('newrelic')) {
    newrelic_end_transaction();
}

and the following in your custom test classes that extend TestCase

public function setup(): void {
    parent::setup();

    if (extension_loaded('newrelic')) {
        newrelic_start_transaction(ini_get('newrelic.appname'));
    }
}

public function teardown(): void {
    parent::teardown();

    if (extension_loaded('newrelic')) {
        newrelic_end_transaction();
    }
}