[Behat]Cannot run full tests suites with symfony 4.4

[michaoj]

Symfony version(s) affected: 4.4.0

Description
When I run my behat tests, I end up getting a Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/project/var/cache/test/pools/AqNbLoad6z/E/7/9TL8Zv8iSmZZttWPui9Q on line 1 after several scenarii

How to reproduce
This occurs every time I try to run the full behat tests suite

Possible Solution
Maybe a container reload issue ?

Additional context

• This worked flawlessly with symfony version 4.3.x
• I use the symfony/http-kernel 4.4.x-dev
• I updated my Kernel with $container->setParameter('container.dumper.inline_factories', true);

[nicolas-grekas]

Likely fixed by #34580

[michaoj]

@nicolas-grekas : I guess you mean #34567 ?

[michaoj]

I checked my file vendor/symfony/http-kernel/Kernel.php

And here's the content for the line 517 has mentionned in #34580 change log

if (file_exists($cachePath) && \is_object($this->container = include $cachePath)
            && (!$this->debug || (self::$freshCache[$cachePath] ?? $cache->isFresh()))
        ) {
            self::$freshCache[$cachePath] = true;
            $this->container->set('kernel', $this);
            error_reporting($errorLevel);

            return;
        }

I still have the memory error

[adrienfr]

Same here since SF 4.4, Behat tests seems to use a lot more memory and full tests suite fails on CircleCI with error Received 'killed' signal at random scenario.

[michaoj]

Same here since SF 4.4, Behat tests seems to use a lot more memory and full tests suite fails on CircleCI with error Received 'killed' signal at random scenario.

Thought I was alone 😄

[nicolas-grekas]

Reproducer and/or Blackfire profile welcome. Otherwise it's going to be yours to debug.

[michaoj]

I profile two calls of a subset of my behat tests
https://blackfire.io/profiles/551c95b2-2f40-405e-a0d8-ab7e2a085c45/graph
And
https://blackfire.io/profiles/09125477-3b93-4b28-8e25-34fa1cba4d22/graph

If I try to combine the two tags, I'm unable to profile with an out of memory error

[xabbuh]

Can you profile the same but with Symfony 4.3 and share the profile comparison?

[nicolas-grekas]

I updated my Kernel with $container->setParameter('container.dumper.inline_factories', true);

I did not understand this part: does setting this to true fix the issue? or setting it to false maybe?

[michaoj]

I updated my Kernel with $container->setParameter('container.dumper.inline_factories', true);

I did not understand this part: does setting this to true fix the issue? or setting it to false maybe?

No setting this to true doesn't fix the problem, and without it it doesn't work either

[michaoj]

Can you profile the same but with Symfony 4.3 and share the profile comparison?

I'm trying : with sf 4.3 the full behat tests suite run

But running the same profiling than yesterday brings me a

[adrienfr]

I tried again with SF 4.4.1 (since #34580 has been merged) and still have the issue on CircleCI, it fails 90% of the time with the Received 'killed' signal error.

SF 4.3.8

265 scenarios (264 passed, 1 failed)
3811 steps (3642 passed, 1 failed, 168 skipped)
27m13.66s (301.51Mb)

SF 4.4.1

265 scenarios (264 passed, 1 failed)
3811 steps (3642 passed, 1 failed, 168 skipped)
26m10.73s (2.44Gb)

As I don't have this issue on my local env but only on CircleCI (not the same memory amount available), I don't have a blackfire profile to share... Is there any other way to help you investigate this (other than setup blackfire on CircleCI) ?

Thanks

[nicolas-grekas]

Providing a small app we could run locally to reproduce the issue would be the best help!

[adrienfr]

@michaoj, do you use Panther (Symfony\Component\Panther\PantherTestCase) or another tool using BrowserKit ?

The issue might come from this, as I don't have any issue with pure PHPUnit using Symfony\Bundle\FrameworkBundle\Test\WebTestCase :

4.3.9

PHPUnit: Time: 6.96 minutes, Memory: 1.11 GB
Behat: 27m13.66s (301.51Mb)

4.4.1

PHPUnit: Time: 6.41 minutes, Memory: 1.16 GB
Behat: 26m10.73s (2.44Gb)

[michaoj]

@michaoj, do you use Panther (Symfony\Component\Panther\PantherTestCase) or another tool using BrowserKit ?

No, my test suites does not rely yet on a browser driver