Memory leak in PhpFilesAdapter

[strider2038]

Symfony version(s) affected: 4.2.x

Description
After upgrading from Symfony 4.1 to Symfony 4.2 Behat tests started to eat a lot of memory. Before upgrade 5k tests used about ~200 Mb of memory, after upgrading process fails on ~2Gb memory limit.

The problem is that this exception https://github.com/symfony/cache/blob/5ec26f9aacf8a054f437f1f12f6d3c8ec23e8c18/Adapter/PhpFilesAdapter.php#L35 somehow references to Behat\Behat\Context\Environment\InitializedContextEnvironment. So all application classes are stored in memory and each test increases memory usage significantly.

I used meminfo to find leaks

Path from 0x7f675614f9c0
+----------------------------------------------------------------------+
| Id: 0x7f674d12f180                                                   |
| Type: object                                                         |
| Class: Behat\Behat\Context\Environment\InitializedContextEnvironment |
| Object Handle: 56439                                                 |
| Size: 72 B                                                           |
| Is root: No                                                          |
| Children count: 3                                                    |
+----------------------------------------------------------------------+
         ^          
         |          
         0          
         |          
         |          
+--------------------+
| Id: 0x7f674d51a440 |
| Type: array        |
| Size: 72 B         |
| Is root: No        |
| Children count: 5  |
+--------------------+
         ^          
         |          
        args        
         |          
         |          
+--------------------+
| Id: 0x7f674cfd8ea8 |
| Type: array        |
| Size: 72 B         |
| Is root: No        |
| Children count: 6  |
+--------------------+
         ^          
         |          
         21         
         |          
         |          
+--------------------+
| Id: 0x7f674d2f7438 |
| Type: array        |
| Size: 72 B         |
| Is root: No        |
| Children count: 40 |
+--------------------+
         ^          
         |          
       trace        
         |          
         |          
+----------------------+
| Id: 0x7f674d2f73c0   |
| Type: object         |
| Class: Exception     |
| Object Handle: 68053 |
| Size: 72 B           |
| Is root: No          |
| Children count: 7    |
+----------------------+
         ^          
         |          
         e          
         |          
         |          
+--------------------+
| Id: 0x7f674d357fe0 |
| Type: array        |
| Size: 72 B         |
| Is root: No        |
| Children count: 1  |
+--------------------+
         ^          
         |          
       static       
         |          
         |          
+----------------------+
| Id: 0x7f674cc84f80   |
| Type: object         |
| Class: Closure       |
| Object Handle: 68040 |
| Size: 72 B           |
| Is root: No          |
| Children count: 2    |
+----------------------+
         ^          
         |          
   includeHandler   
         |          
         |          
+--------------------------------------------------------+
| Id: 0x7f674ca63340                                     |
| Type: object                                           |
| Class: Symfony\Component\Cache\Adapter\PhpFilesAdapter |
| Object Handle: 67663                                   |
| Size: 72 B                                             |
| Is root: No                                            |
| Children count: 16                                     |
+--------------------------------------------------------+
         ^          
         |          
cache.annotations.recorder_inner
         |          
         |          
+---------------------+
| Id: 0x7f674d550c48  |
| Type: array         |
| Size: 72 B          |
| Is root: No         |
| Children count: 123 |
+---------------------+
         ^          
         |          
      privates      
         |          
         |          
+---------------------------------------------------------+
| Id: 0x7f674d550c00                                      |
| Type: object                                            |
| Class: ContainerAlglvav\srcApp_KernelTestDebugContainer |
| Object Handle: 69160                                    |
| Size: 72 B                                              |
| Is root: No                                             |
| Children count: 20                                      |
+---------------------------------------------------------+
         ^          
         |          
     container      
         |          
         |          
+---------------------+
| Id: 0x7f6755b61400  |
| Type: object        |
| Class: App\Kernel   |
| Object Handle: 1966 |
| Size: 72 B          |
| Is root: No         |
| Children count: 12  |
+---------------------+
         ^          
         |          
       kernel       
         |          
         |          
+------------------------------------------------------------+
| Id: 0x7f6752e85d20                                         |
| Type: object                                               |
| Class: Behat\Symfony2Extension\Suite\SymfonySuiteGenerator |
| Object Handle: 9630                                        |
| Size: 72 B                                                 |
| Is root: No                                                |
| Children count: 3                                          |
+------------------------------------------------------------+
         ^          
         |          
         0          
         |          
         |          
+--------------------+
| Id: 0x7f6755ac6118 |
| Type: array        |
| Size: 72 B         |
| Is root: No        |
| Children count: 2  |
+--------------------+
         ^          
         |          
     generators     
         |          
         |          
+-------------------------------------------+
| Id: 0x7f6755ac60e0                        |
| Type: object                              |
| Class: Behat\Testwork\Suite\SuiteRegistry |
| Object Handle: 5924                       |
| Size: 72 B                                |
| Is root: No                               |
| Children count: 4                         |
+-------------------------------------------+
         ^          
         |          
      registry      
         |          
         |          
+-------------------------------------------------+
| Id: 0x7f67537ee280                              |
| Type: object                                    |
| Class: Behat\Testwork\Suite\Cli\SuiteController |
| Object Handle: 3486                             |
| Size: 72 B                                      |
| Is root: No                                     |
| Children count: 2                               |
+-------------------------------------------------+
         ^          
         |          
         4          
         |          
         |          
+--------------------+
| Id: 0x7f6755194b18 |
| Type: array        |
| Size: 72 B         |
| Is root: No        |
| Children count: 18 |
+--------------------+
         ^          
         |          
    controllers     
         |          
         |          
+-----------------------------------+
| Id: 0x7f6755194a00                |
| Type: object                      |
| Class: Behat\Testwork\Cli\Command |
| Object Handle: 13700              |
| Size: 72 B                        |
| Is root: No                       |
| Children count: 16                |
+-----------------------------------+
         ^          
         |          
       behat        
         |          
         |          
+--------------------+
| Id: 0x7f675614f9e8 |
| Type: array        |
| Size: 72 B         |
| Is root: No        |
| Children count: 5  |
+--------------------+
         ^          
         |          
      commands      
         |          
         |          
+---------------------------------------+
| Id: 0x7f675614f9c0                    |
| Type: object                          |
| Class: Behat\Testwork\Cli\Application |
| Object Handle: 32                     |
| Size: 72 B                            |
| Is root: Yes                          |
| Execution Frame: <GLOBAL>             |
| Symbol Name: application              |
| Children count: 17                    |
+---------------------------------------+

How to reproduce

It is hard to reproduce on Symfony skeleton.

Possible Solution

Create new exception inside closure. It solves the problem,

https://github.com/symfony/cache/blob/master/Adapter/PhpFilesAdapter.php#L35

$this->includeHandler = function () {
        throw new \Exception();
};

[nicolas-grekas]

Can you please confirm #29644 fixes the issue?

[strider2038]

Can you please confirm #29644 fixes the issue?

Yes, it fixes the issue.

Before fix

Running API tests for identity
...................................................................... 70
...................................................................... 140
...................................................................... 210
...................................................................... 280
...................................................................... 350
...................................................................... 420
...................................................................... 490
...................................................................... 560
...................................................................... 630
....................................................

73 scenarios (73 passed)
682 steps (682 passed)
1m25.00s (450.58Mb)

After fix

Running API tests for identity
...................................................................... 70
...................................................................... 140
...................................................................... 210
...................................................................... 280
...................................................................... 350
...................................................................... 420
...................................................................... 490
...................................................................... 560
...................................................................... 630
....................................................

73 scenarios (73 passed)
682 steps (682 passed)
1m2.07s (118.87Mb)