[Console] Symfony Console memory usage increased after upgrade to Symfony 3.3

[umpirsky]

Q	A
Bug report?	yes
Feature request?	no
BC Break report?	yes
RFC?	no
Symfony version	3.3.5

There is a noticable memory increase in console Application initialization after the upgrade from v3.1.10 to v3.3.5.

Were there any changes in recent version that may be causing this? I saw #22900, but does not look related, I tried removing add() method and got similar results.

[nicolas-grekas]

I guess the full Blackfire comparison profile would help, can you provide it please? (even in private eg on Slack)

[umpirsky]

@nicolas-grekas Sure, I can share it privately, can you provide link to slack org or email address please? Thanks!

[nicolas-grekas]

http://symfony.com/slack-invite

[nicolas-grekas]

I looked at your profile, and the issue comes from
Symfony\Component\HttpKernel\Kernel::initializeContainer, with a cost increase related to 3 items:

• Symfony\Component\Config\ConfigCache::isFresh (+64ms)
• compile::qa/appQaDebugProjectContainer.php (+40ms)
• run::qa/appQaDebugProjectContainer.php (+8ms)

The total (+112ms) holds almost 100% of the wall time delta.
This means that the container is bigger (thus it takes longer to check for freshness + parse + execute).

I can't say if it's legitimate or not without the app. You should compare the dumped container in 3.1 and 3.3 and see if you can suggest any fix/issue.

[umpirsky]

@nicolas-grekas Thanks for taking time to check this. 👍

I checked compiled containers and there is a slight increase in dumped container size (3.1M -> 3.7M). The reason for this is simple increase (3%) of number of services with the new Symfony version.

However, I'm not concerned about wall time increase as much as I am about memory increase in:

That's the problem I am interested in and it does seem to come from the same direction - initializeContainer!
But I'm not sure I can justify such a huge memory consumption increase.

This:

also shows that memory consumption is moved from run to registering commands, which is fine, but still, commands need to be registered (unless lazy iirc) before running the app. Also, it sums up to 6.85M more in total.

[nicolas-grekas]

memory consumption is moved from run to registering commands

this is just the result of architectural change, don't focus on that, as the graph shows that real memory increase comes from the common node: the boot method. That's where you need to focus (click on the magnifying glass to redraw the graph around that node, etc.)

But is 7Mb really an issue?

[umpirsky]

But is 7Mb really an issue?

@nicolas-grekas It is. The way we noticed this is CPU spike on the server after the upgrade is deployed. Our guess is that it comes from resque jobs, it forks to process jobs. Memory size is the biggest factor when it comes to forking taking CPU. It takes CPU to make that copy, and the amount of CPU it takes is based on the total size of the memory.

[nicolas-grekas]

Dunno what we can do here, for now, I'd say nothing... If the container is bigger, it takes more memory...
BTW, the isFresh check means you're running in env=dev (or debug=true), isn't that the real issue?

[umpirsky]

@nicolas-grekas I will investigate this further. yes, this profiles are done in debug mode, but actual issue was present with debug=false. I'll post more findings if I have them, thanks for your help so far, really appreciate it!

[umpirsky]

I profiled again ./app/console --no-debug with v3.1.10 and v3.3.5.

Memory consumed by AppKernel::initializeContainer() increased from 15M to 22M while number of services went from 1761 to 1803.
That's 32% memory increase for 3% number of service increase.
appQaProjectContainer.php size grew from 2933889 to 3560866.

Results are similar with 3.3@dev.

[nicolas-grekas]

@umpirsky can you please test with #23741?

[umpirsky]

@nicolas-grekas Will do in coming days, thanks!

[umpirsky]

Blocked by #23924.

[chalasr]

#23924 has been fixed.

[umpirsky]

Good news everyone!

I profiled with latest 3.4.x-dev 2b4b224 version.

The results are awesome on the first look, great work @nicolas-grekas!

Here is how AppKernel::initializeContainer() compares now from v3.1.10 to 3.4.x-dev 2b4b224:

All we need to do now is wait for 3.4 stable. :)

[umpirsky]

@nicolas-grekas Is there a chance to get this optimizations in Symfony 3.3?

[stof]

@umpirsky we won't backport such big refactoring into a patch release. This is too risky (and it actually relies on new features)

[nicolas-grekas]

Amazing, thanks for providing such a clear proof.
Closing then.