High memory usage on php8.1 and laravel 9.

[mad-briller]

Recently i have been updating all my projects from php7.4 to php8.1, aswell as moving from laravel 6 to laravel 9.

However, after performing these upgrades and running Phan i'm seeing a huge increase in the memory footprint required to analyse a codebase or run a language server.
This huge increase isn't consistent either, it wont always happen;
for example to gather the information for this issue i had to swap php versions using update-alternatives and reinstall composer dependencies for the package & php version, and as a result the memory increase seems to have gone away but will eventually come back.

These versions are causing the issue:

=> php ~/.config/composer/vendor/bin/phan --version
Phan 5.3.2
php-ast version 1.0.16
PHP version used to run Phan: 8.1.2

Comparing running Phan on a few versions of my package which are very similar in terms of my code i see the following results:

PHP7.4 / Phan 5.3.2 / Laravel 6: 2027MB
PHP8.0 / Phan 5.3.2 / Laravel 8: 2006MB
PHP8.1 / Phan 5.3.2 / Laravel 9: Wont complete on my system due to running out memory, gets to about 14gb and 20% before i have to kill the process

Here is an example of the same issue occuring when running as a language server:

One thing i know Laravel added in 9 is a lot of @template annotations, some of which Phan doesn't properly understand as it used some syntax that is not supported by Phan e.g. @template K of array-key, i'm wondering if these misinterpreted templates are leading to type explosion or something?

tbh i'm not really sure how to go about debugging something like this when it happens again, any help or guidance would be appreciated, thank you.

[TysonAndre]

phan --debug will tell you what file it's stuck on analyzing. See phan --extended-help for a list of all debug options.

Building and running https://github.com/adsr/phpspy will let you know what it's stuck on.

Alternately, phan --debug-signal-handler would help.

Do you have a minimal repo to reproduce this?

 --debug-signal-handler
  Set up a signal handler that can handle interrupts, SIGUSR1, and SIGUSR2.
  This requires pcntl, and slows down Phan. When this option is enabled,

  Ctrl-C (kill -INT <pid>) can be used to make Phan stop and print a crash report.
  (This is useful for diagnosing why Phan or a plugin is slow or not responding)
  kill -USR1 <pid> can be used to print a backtrace and continue running.
  kill -USR2 <pid> can be used to print a backtrace, plus values of parameters, and continue running.

[TysonAndre]

composer.json also has an option to pin the php version you're targeting for installing dependencies of the application

    "config": {
        "platform": {
            "php": "8.0.16"
        }
    },

[TysonAndre]

e.g. sudo phpspy --pid=2548891 (optionally --top) could be used to see where phan was spending it's time in a running language server

[TysonAndre]

https://github.com/psliwa/image-optimizer - it could be the callers of that somehow, code that attempts to parse image formats has caused high runtime before due to the complex types it generates

This huge increase isn't consistent either, it wont always happen;

I'm surprised that you'd also see that in batch analysis - files should be sorted, though with --processes N the analyzed file groups may be unpredictable https://github.com/phan/phan/wiki/Different-Issue-Sets-On-Different-Numbers-of-CPUs and vary as files are added or removed from your project. Are you using the regular php-ast library (e.g. are you using the polyfill with caching, though that shouldn't matter)

[mad-briller]

Hey, it's interesting you mention that image-optimizer library as that is included by the package in question, what led you to mention it? past experience or my recent interactions with that repo?
however, that library is also included in the php7.4 and php8.0 cases aswell, the only difference is for php8.1 i'm using a forked version that allows symfony 6.

Also yeah, i'm using php-ast 1.0.16 not the polyfill, i'm also not running with a processes flag passed to phan so it will be using the default.
I've never seen nondeterministic between runs but i have seen different results from language server vs command line.

It's not happening right now since swapping about versions, but when it happens again i will definitely use the debugging approaches mentioned above and provide more information

thanks for your hasty and thorough response Tyson

[TysonAndre]

Hey, it's interesting you mention that image-optimizer library as that is included by the package in question, what led you to mention it? past experience or my recent interactions with that repo?

Just your recent interactions, I was trying to find the repo you were talking about to see if I could reproduce any new performance issues related to 8.1 specifically rather than dependencies. exclude_file_regex, exclude_analysis_directory_list, exclude_file_list may help in those cases, or narrowing down directory_list to be as specific as possible, e.g. phan's own config limits to only direct vendor dependencies

    'directory_list' => [
        'internal/lib',
        'src',
        'tests/Phan',
        'vendor/composer/semver/src',
        'vendor/composer/xdebug-handler/src',
        'vendor/felixfbecker/advanced-json-rpc/lib',
        'vendor/microsoft/tolerant-php-parser/src',
        'vendor/netresearch/jsonmapper/src',
        'vendor/phpunit/phpunit/src',
        'vendor/psr/log',  // subdirectory depends on dependency version
        'vendor/sabre/event/lib',
        'vendor/symfony/console',
        'vendor/symfony/polyfill-php80',
        'vendor/tysonandre/var_representation_polyfill/src',
        '.phan/plugins',
        '.phan/stubs',
    ],

Repeatedly running phan_client -l vendor/nikic/php-parser/lib/PhpParser/Parser/Php7.php surprisingly grows memory usage slowly when run repeatedly (and takes 1.8 seconds to analyze) though that file probably isn't analyzed in your case - Php7.php is automatically generated and not typical code, though.

The latest v5 commit may help if the issue only shows up when running the language server for a long time (and the files are complex enough for the leak to be noticeable)

[mad-briller]

Hey @TysonAndre ,

This finally happened again today and it is caused by the orchestra/testbench-core package that is used to test laravel packages.

To do that, it has a laravel installation inside the package, and sets up a symlink from the "emulated" laravel to the real vendor folder of the package, producing a recursive loop.

Thanks to the help of the --debug flag i was able to see this:

Going to parse 'vendor/orchestra/testbench-core/laravel/vendor/orchestra/testbench-core/laravel/vendor/orchestra/testbench-core/laravel/vendor/orchestra/testbench-core/laravel/vendor/orchestra/testbench-core/laravel/vendor/orchestra/testbench-core/laravel/vendor/orchestra/testbench-core/laravel/vendor/orchestra/testbench-core/laravel/vendor/orchestra/testbench-core/laravel/vendor/orchestra/testbench-core/laravel/vendor/orchestra/testbench-core/laravel/vendor/orchestra/testbench-core/laravel/vendor/laravel/framework/src/Illuminate/Routing/UrlGenerator.php'

which made me see what the issue is.

As a result, i don't think this is a Phan issue, unless you think it should ignore symlinks / prevent recursion in some way ?

Thanks for your help debugging this mate