High memory consumption

[ckuran]

Q	A
Version	2.20.0
Bug?	yes
New feature?	no
Question?	no
Documentation?	no
Related tickets	~

Issue

Executing multiple tasks in a repository with thousands of PHP files can lead to Fatal errors due to exceeding allowed memory and the need to manually increase memory_limit just for GrumPHP process. I ended up requiring myself to set the memory_limit=2G because otherwise the process fails and cuts the parallel processing. Extending the number of tasks requires constantly increasing number of memory.

Error

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 8192 bytes) in /app/vendor/phpro/grumphp/src/Collection/FilesCollection.php on line 256

Examples

Repository: ~9900 PHP files only

Example with just one linter:

Running tasks with priority 0!
==============================

Running task 1/1: jsonlint... ✔

Peak memory usage: 192.0 MB

Example with all linters:

Running tasks with priority 0!
==============================

Running task 1/3: jsonlint... ✔
Running task 2/3: xmllint... ✔
Running task 3/3: yamllint... ✔

Peak memory usage: 368.0 MB

Example with all my current suite:

Running tasks with priority 0!
==============================

Running task 1/12: jsonlint... ✔
Running task 2/12: xmllint... ✔
Running task 3/12: yamllint... ✔
Running task 4/12: openapi... ✔
Running task 5/12: rector... ✔
Running task 6/12: behat... ✔
Running task 7/12: phpmd... ✔
Running task 8/12: phpcs... ✔
Running task 9/12: phpstan... ✔
Running task 10/12: psalm... ✔
Running task 11/12: test-unit... ✔
Running task 12/12: test-integration... ✔

Peak memory usage: 1152.0 MB

It's worth noting that setting parallel: false reduces the memory consumption, but also raises the time to complete all tasks to unacceptable level.

Steps to reproduce:

• Checkout some huge repository
• Configure GrumPHP
• Before return in RunCommand.php add the following to check memory usage:

if ($this->io->isVerbose()) {
    $this->io->write([
        PHP_EOL,
        sprintf(
            '<comment>Peak memory usage: %.1f MB</comment>',
            (float) memory_get_peak_usage(true) / 1024.0 / 1024.0
        )
    ]);
}

• Run ./vendor/bin/grumphp run -v