Hitting memory limit of 128MB during staging

[ChristianRiesen]

I run a fairly large application with symfony 2. Now when deploying it fails in multiple colorful ways, though I got most of them licked by having the composer install done before the push and let CF take the entire vendor folder with it.

It still dies on one spot though. If I let the .bp-config/options.json file have this config for composer it fails:

  "COMPOSER_INSTALL_OPTIONS": [
    "--no-dev",
    "--no-interaction",
    "--prefer-dist"
  ]

But with this it works, though the cahce is not build and other things are not executed that are needed to make it run well:

  "COMPOSER_INSTALL_OPTIONS": [
    "--no-dev",
    "--no-interaction",
    "--no-scripts",
    "--prefer-dist"
  ]

It's clear to me that the limit comes from the php.ini file in the defaults folder. So I thought I'd be clever and supply my own. I copied the PHP 5.5 ini file (using PHP 55 latest) to .bp-config/php/php.ini and just up the memory limit.

That "worked" as in the limit was upped, but then weirdly it couldn't find the .so extensions anymore. It tried to load them (so adding the lines worked) but for some reason the files did not seem to be at the path they should be. I noticed that with the supplied php.ini it would not expand the @ marked values for HOME and so on, while it would on the BP internal one. I think there must be more going on beyond that since the extension files were not present either apparently.

I got the runtime php to up the limit by making a small ini in .bp-config/php/conf.d/project.ini with just the memory limit raising, then set the environment variable in the manifest:

  env:
    DOCUMENT_ROOT: web
    SYMFONY_ENV: prod
    PHP_INI_SCAN_DIR: /home/vcap/app/.bp-config/php/conf.d

That works with the droplet running, but still during staging, it doesn't.

So how do I raise the limit during staging (the composer install fails since running out of memory) and still have the extensions working?

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/103668060.

[jtarchie]

Could you clarify, how much memory have you allocated towards your application?

This can be defined two ways, the -m flag with cf CLI, or with manifest.yml with the memory property.

[ChristianRiesen]

I've configured it to use 2G of memory. That is not the issue. The issue is that setting in the php.ini and how to tell the buildpack to use the correct setting during staging.

[jtarchie]

@ChristianRiesen, the app directory structure for staging and running are different.

For staging application code is under /tmp/staged/app.That would explain why PHP_INI_SCAN_DIR is not picking up project.ini. It is not clear to me if you can specify multiple paths.

As for your original .bp-config/php/php.ini not expanding the templated values. We'll investigate that issue, too.

[dmikusa]

I copied the PHP 5.5 ini file (using PHP 55 latest) to .bp-config/php/php.ini and just up the memory limit.

While this is technically possible, it's not a great idea because it can cause problems. If you're going to go this route, you need to copy the default file from the PHP build pack which is here and start from this. That will save you some headaches.

Things to watch out for when doing this:

• the custom php.ini will override the default provided by the build pack completely, so the custom one needs to contain all the config and not just the one or two items you want to override.
• the custom php.ini needs to retain the place holders. The build pack inserts place holders that it overwrites with things relevant to the environment. This includes the extensions that have been installed. If you remove the place holders, that will likely cause problems with extensions loading.

Better ways to override php.ini settings:

1.) In your PHP script, set a request for a higher memory limit i.e. ini_set
2.) Use the -d argument to php. Ex: php -d memory_limit=128M my-large-script.php
3.) Create a .user.ini file and put the setting there. Docs

That "worked" as in the limit was upped, but then weirdly it couldn't find the .so extensions anymore.

I think you removed the place holders. Without these, you'd need to explicitly tell it what extensions to load.

PHP_INI_SCAN_DIR: /home/vcap/app/.bp-config/php/conf.d

This is another option, but paths at runtime are different from staging. So this path will exist during runtime, but not at staging. The staging path is /tmp/staged/app/.... I'd try one of the three options above as those don't need a path specified.

Dan

[ChristianRiesen]

I copied the file and changed the memory_limit line only. So no, I haven't removed the placeholders.

Your suggestions do not work, sorry, because it's in staging.

1. It's composer running, I can't alter composer, it's part of the buildpack.
2. See 1), part of the buildpack. I haven't even found where I could possibly add that in there, could I eve change the buildpack.
3. Tried, and wont run. I added it to the root of the project, bpconfig folder, php folder inside that as well. It works in the running app, but during staging it ignores this file.

There is no ability to tell PHP to change settings during staging, a crucial thing to do if you need extensions or more memory.

[dmikusa]

I copied the file and changed the memory_limit line only. So no, I haven't removed the placeholders.

Can you provide sample app? I've used custom php.ini files recently and it worked just fine.

1. It's composer running, I can't alter composer, it's part of the buildpack.

You haven't included the exact error message so there's no way for me to provide you with specific advise. Those are the three options that generally work around problems like this. If you were to include the full output of cf push, the build pack version you're using and the staging error perhaps we could tell you more.

There is no ability to tell PHP to change settings during staging, a crucial thing to do if you need extensions or more memory.

I've had similar issues and we were able to work around them. I'd suspect it's something specific to what you're doing. If you can attach a sample app that replicates, I'd be happy to investigate more.

[ChristianRiesen]

The specific is that composer runs with a php.ini that only allows for 128MB ram to be allocated during the staging process. That is the core of my problem. How can I change that?

I'm looking for a solution to solve that problem, change the ini for this to run properly. The ini settings also influence other things, like the locale, time settings and more, so memory is just one small part of the potential pitfalls here.

What happens is that there are cache warmers running, which do not run of course with the "--no-scripts" parameter added. So the app get's booted and there is a reason why the instance is configured with 2GB instead of 128MB.

There was one work around, which runs these on the instances when they are booted, but that completely defeats the purpose of running these expensive operations once during staging and being able to scale new instances quickly.

[dmikusa]

I played around with this a bit and it seems like you'll need to use a custom php.ini file here. I tried using a .user.ini file, but PHP doesn't seem to respect this. Using a custom php.ini file seems to work fine for me though. I took a simple composer app and added a post install cmd to dump the memory limit.

Ex:

        "post-install-cmd": [
            "php -r 'print \"Memory: \" . ini_get(\"memory_limit\") . \"\n\";'"
        ]

When I run without a custom php.ini file it prints 128M. When added a custom .bp-options/php/php.ini file with the memory limit set to 256M, it printed that.

...
Loading composer repositories with package information
Installing dependencies from lock file
  - Installing doctrine/lexer (v1.0.1)
    Loading from cache
  - Installing gregwar/cache (v1.0.10)
    Loading from cache
  - Installing twig/twig (v1.22.1)
    Loading from cache
  - Installing symfony/translation (v2.7.4)
    Loading from cache
  - Installing symfony/templating (v2.7.4)
    Loading from cache
  - Installing symfony/stopwatch (v2.7.4)
    Loading from cache
  - Installing symfony/security-core (v2.7.4)
    Loading from cache
  - Installing symfony/security-csrf (v2.7.4)
    Loading from cache
  - Installing symfony/routing (v2.7.4)
    Loading from cache
  - Installing symfony/http-foundation (v2.7.4)
    Loading from cache
  - Installing symfony/event-dispatcher (v2.7.4)
    Loading from cache
  - Installing psr/log (1.0.0)
    Loading from cache
  - Installing symfony/debug (v2.7.4)
    Loading from cache
  - Installing symfony/http-kernel (v2.7.4)
    Loading from cache
  - Installing symfony/filesystem (v2.7.4)
    Loading from cache
  - Installing symfony/dependency-injection (v2.7.4)
    Loading from cache
  - Installing symfony/config (v2.7.4)
    Loading from cache
  - Installing symfony/asset (v2.7.4)
    Loading from cache
  - Installing doctrine/annotations (v1.2.7)
    Loading from cache
  - Installing symfony/framework-bundle (v2.7.4)
    Loading from cache
  - Installing gregwar/image (v2.0.20)
    Loading from cache
  - Installing gregwar/image-bundle (v2.0.21)
    Loading from cache
  - Installing monolog/monolog (1.0.2)
    Loading from cache
Generating autoload files
Memory: 256M
...

Please give that a try. If it doesn't work, point me to a test case that reproduces the problem and I'll look into it more.

[ChristianRiesen]

I'm working on a test case.

Meanwhile, it makes little sense to have the CLI with this restrictive setting. Is there a way to add memory_limit = -1 to the buildpack for the CLI execution? That would simply allow it to use the memory given depending on what settings you have for app you provision.

See also here:
https://getcomposer.org/doc/articles/troubleshooting.md#memory-limit-errors

[ChristianRiesen]

@dmikusa-pivotal Would something speak against a -1 memory limit for composer execution? Otherwise we might be able to produce a PR for that.

[hairmare]

I strongly oppose a -1 memory limit. As soon as the containers limit is reached fpm users will get either the fpm or nginx process killed. All that gets logged would be a out of memory error.

Could it be that you are pushing your app without a composer.lock file? Composer needs heaps more RAM if you don't add one and 256MB is usually enough to do a composer install if a lockfile is given.

[ChristianRiesen]

@hairmare: It would send the same "cannot allocate memory" error that it sends now. This is a localized change to just composer during staging, nothing else. The entire issue is that php.ini changes during staging are a large problem.

The app has a composer.lock and we even include the vendor folder prepped perfectly so it doesn't have to do anything (since it can't access the private repos). The problem it has is due to scripts running with the cache warmers. Those boot the application and there is a good reason to require 2 GB or memory for the application.

Changing the memory is possible with the custom php.ini in the .bp-config/php/php.ini but then it doesn't expand the paths correctly and extensions can't be found. So staging will be fine mainly, but then the app is broken.

[ChristianRiesen]

So, after some more trying out, a solution presented itself. PHP allows for an ENV variable called PHP_INI_SCAN_DIR you can populate with a directory to search for additional ini files. The staging as well as the live app will follow this directive. The trick is to make it relative as a path though. So this manifest works:

---
applications:
- name: my-app
  memory: 2048M
  instances: 1

  env:
    DOCUMENT_ROOT: web
    SYMFONY_ENV: prod
    PHP_INI_SCAN_DIR: .bp-config/php/conf.d

And then just add ini files to .bp-config/php/conf.d that contain only what is needed to overwrite. Ours looks like this:

[PHP]
memory_limit = 2048M

[Date]
date.timezone = Europe/Zurich

We need the date time one in order for everything to run correctly, you might not.

Now this solution will make the ini the same for staging and for the running app, which is good enough for our use case. This way we don't need an entire copy of the php.ini file, just the parts we want to override.

Maybe this could be added to the documentation.

[dmikusa]

Nice! That's a good solution. Yet another item that showcases the flexibility of PHP's configuration. +1 for adding that to the docs. That plus the three other options above, give users a lot of different ways to modify PHP's configuration.