Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce the memory consumption by forcing PHP to release the unused memory #6230

Merged
merged 2 commits into from Aug 5, 2021
Merged

Reduce the memory consumption by forcing PHP to release the unused memory #6230

merged 2 commits into from Aug 5, 2021

Conversation

esnelubov
Copy link

@esnelubov esnelubov commented Jul 5, 2021

At our company we run our codeception tests in parallel using multiple processes.
When 50 Codeception processes run in parallel the memory consumption of a single Codeception process starts to matter. We noticed that some of our Cests consume 40MB of RAM while others consume 80MB (run on Ubuntu 20.04, PHP 7.4.3 and Centos 7, PHP 7.4.14).

I profiled our tests to find how can I reduce their memory footprint. I found that the source of the problem lies in a single call to token_get_all() in the \Codeception\Lib\Parser::getClassesFromFile() function.

It seems this is a well known issue:

https://bugs.php.net/bug.php?id=71375

On large files the result of token_get_all() can consume dozens of megabytes of RAM, but even after result got out of scope PHP doesn't return the consumed memory to the OS.

I added lines:

$tokens = null; 
gc_mem_caches();

to the end of the \Codeception\Lib\Parser::getClassesFromFile() function to force PHP to return the cached memory. It reduced the memory consumption of our parallel run by 1GB.

Though \Codeception\SuiteManager::loadTests() looks like a better place for the gc_mem_caches() call:

public function loadTests($path = null)
{
    $testLoader = new Loader($this->settings);
    $testLoader->loadTests($path);
    gc_mem_caches();

for some unknown reason a call to gc_mem_caches() in \Codeception\SuiteManager::loadTests() releases less memory for some Cests than the same call at the end of \Codeception\Lib\Parser::getClassesFromFile() (700MB vs 1GB in the case of our parallel run).

src/Codeception/Lib/Parser.php Outdated Show resolved Hide resolved
@Naktibalda
Copy link
Member

Naktibalda commented Jul 31, 2021

@esnelubov I and @eXorus tried to reproduce your issue with memory consumption, but your patch made no difference at all.

How large are your Cest files?

@esnelubov
Copy link
Author

esnelubov commented Aug 2, 2021

@Naktibalda We have around 2000 Cest files: 500-1000 lines each.

To reproduce the problem I wrote a simple test, filled with garbage. Without my patch it consumes 46 MB of memory on my PC and with my patch its memory consumption is reduced to 23 MB.

Of course in our real tests we don't need such a wall of code to reproduce the problem, but I can't predict which combinations of PHP statements produce such memory consuming arrays of tokens, so I just illustrated the problem in a simple way.

@Naktibalda Naktibalda merged commit d69ab79 into Codeception:4.1 Aug 5, 2021
13 checks passed
@Naktibalda
Copy link
Member

Naktibalda commented Aug 5, 2021

Thank you, I am convinced now :)

@esnelubov esnelubov deleted the 4.1-free-memory branch Aug 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants