New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory issues on composer require or update #732

Closed
chrisarcherhydrant opened this Issue Feb 28, 2018 · 35 comments

Comments

@chrisarcherhydrant
Copy link

chrisarcherhydrant commented Feb 28, 2018

Hi,

I'm using the latest version of Lando with a Drupal 8 project when ever I try and run lando composer update I keep getting:

`PHP Fatal error: Allowed memory size of 1610612736 bytes exhausted (tried to allocate 72 bytes) in phar:///usr/local/bin/composer/src/Composer/DependencyResolver/RuleWatchGraph.php on line 100

Fatal error: Allowed memory size of 1610612736 bytes exhausted (tried to allocate 72 bytes) in phar:///usr/local/bin/composer/src/Composer/DependencyResolver/RuleWatchGraph.php on line 100

Check https://getcomposer.org/doc/articles/troubleshooting.md#memory-limit-errors for more info on how to handle out of memory errors.zend_mm_heap corrupted`

My .lando.yml file is:

name: custom-drupal8 recipe: drupal8 config: webroot: web php: '5.6'

@nicrodgers

This comment has been minimized.

Copy link

nicrodgers commented Mar 6, 2018

Hi Chris.

You need to increase the size of the PHP memory limit. You can do that by providing a php.ini within your project that sets the override (and any others you might want), then pointing the lando config at it.

For example, create the file php.ini in your app root, and put in it:
memory_limit = -1

Then add this to your .lando.yml:

config:
  conf:
    php: php.ini

Then a lando rebuild should pick up the changes.

@pirog

This comment has been minimized.

Copy link
Member

pirog commented Mar 15, 2018

@dustinleblanc similar situation to #704. Is this something we should consider setting higher so that it "just works" for people?

@akalata

This comment has been minimized.

Copy link

akalata commented Mar 15, 2018

@nicrodgers your instructions didn't seem to work for me. I'm still getting the memory errors from Composer, and the result of lando php -r "echo ini_get('memory_limit').PHP_EOL;" hasn't changed from 512M.

@dustinleblanc

This comment has been minimized.

Copy link
Member

dustinleblanc commented Mar 15, 2018

@pirog so we have people who hit their container's memory limit all the time on the platform, even if they don't exceed the memory limit for a PHP process.

For Lando being a dev solution, I am usually +1 on ripping out all the stops to make development more fun. For Pantheon recipes, we do tend to tout the 'production mirror' thing which makes me want to clamp down on the memory limit.

Can we maybe find a way to set no limit to memory on CLI ops and keep a strict limit for php-fpm while its running a web process? We set some params different for CLI ops on the platform so I imagine this has to be possible...

The other thing is that Docker I think does like 2gb of ram by default for your entire docker VM, so folks may start throwing SIGKILL errors no matter what they do unless they up that limit in docker's config

@pirog

This comment has been minimized.

Copy link
Member

pirog commented Mar 15, 2018

@dustinleblanc yeah im def on board with -1 for CLI ops and i'm guessing we could easily just build our composer command behind the scenes to set that eg something like php -d memory_limit=-1 composer

@pirog pirog self-assigned this Mar 16, 2018

@pirog pirog added the improvement label Mar 16, 2018

@pirog pirog added this to the 3.0.0-beta.37 milestone Mar 16, 2018

pirog added a commit that referenced this issue Mar 16, 2018

@pirog pirog closed this Mar 16, 2018

@pirog pirog reopened this Mar 19, 2018

@pirog

This comment has been minimized.

Copy link
Member

pirog commented Mar 20, 2018

@dustinleblanc so reopening this because we probably want to figure out the best way to set this on the php level so all cli tools eg drush, composer, artisan all have INFINITE MEMORY!

@pirog pirog removed this from the 3.0.0-beta.37 milestone Mar 20, 2018

@pirog pirog removed their assignment Mar 20, 2018

@com2

This comment has been minimized.

Copy link

com2 commented Aug 8, 2018

OK, this is still unfixed in v3.0.0-beta.47, right? So what is the present work around to have drush, drupal (console) and composer run with infinite memory? I tried with .lando.yml:
config
conf: php.ini

But that does not even override the apache php memory limit

@alphex

This comment has been minimized.

Copy link

alphex commented Aug 10, 2018

following

@dustinleblanc

This comment has been minimized.

Copy link
Member

dustinleblanc commented Aug 17, 2018

So I was looking at this today, trying to get some ideas on how to get ALLTHEMEMORY for Composer, interestingly according to https://getcomposer.org/doc/articles/troubleshooting.md#memory-limit-errors Composer already defaults to 1.5G. I did however find that its not super hard to just give us ALLTHEMEMORY for CLI activities a la https://serverfault.com/questions/807763/how-can-i-create-separate-configuration-files-for-php-cli-and-php-fpm-on-an-ar

Testing it out, might be able to get this into our next release

@wturrell

This comment has been minimized.

Copy link

wturrell commented Aug 27, 2018

A couple of people (@akalata, @com2) have said @nicrodgers method of raising PHP's memory_limit isn't working for them. Right now, it does seem to work in both the browser and CLI for me (lando v3.0.0-beta.47, docker 18.06.0-ce-mac70 (26399), macOS Sierra 10.12.6).

However: note I'm yet to run into the original problem with composer running out of memory.

Make sure you have this exact syntax (your webroot, server (via) and php version may of course vary):

config:
  webroot: web
  via: nginx
  php: '7.1'
  conf:
    php: php.ini

i.e. it needs to be nested with the rest of your .lando.yml config and you need a 'php' key.

www-data@f0b3d1f9ac96:/app$ php -i | grep 'memory'
memory_limit => -1 => -1

@hkirsman

This comment has been minimized.

Copy link

hkirsman commented Sep 14, 2018

I fixed it by adding alternative command. I hope it's fixed in Lando someday:

services:
  appserver:
    overrides:
      services:
        volumes:
          - $LANDO_APP_ROOT_BIND/.lando/composer.sh:/helpers/composer.sh

tooling:
  # Fix memory limit issues for composer by using composer2 command instead.
  # I wasn't able to use composer here to override the original command.
  composer2:
    service: appserver
    description: Run composer commands
    cmd: /helpers/composer.sh
@dustinleblanc

This comment has been minimized.

Copy link
Member

dustinleblanc commented Sep 14, 2018

@hkirsman it should be someday, I ran into a little bit of an issue with my fix but it should be possible to fix permanently. The way we load ini files doesn't immediately allow us to load a separate cli ini file, but that is the goal

@anacona16

This comment has been minimized.

Copy link

anacona16 commented Sep 19, 2018

@hkirsman what is the content of helpers/composer.sh?

@tanc

This comment has been minimized.

Copy link
Contributor

tanc commented Sep 20, 2018

Having a separate cli ini would also solve #1140

@com2

This comment has been minimized.

Copy link

com2 commented Sep 22, 2018

I use lando v3.0.0-beta.47 and docker 18.06.1-ce (linux/amd64) and I can confirm that what @wturrell did works here too.

I noticed however that you have to do "lando stop" and "lando start" to make it work. Changes in .lando.yml are picked up with "lando start" only , but for changes in php.ini you need "lando stop". This behaviour might have influenced our evaluation.

@dustinleblanc

This comment has been minimized.

Copy link
Member

dustinleblanc commented Sep 24, 2018

@tanc even more reason to go that route, I think it's a good idea overall, @pirog @serundeputy what do you dudes think?

@serundeputy

This comment has been minimized.

Copy link
Member

serundeputy commented Sep 24, 2018

re: 1140 I can understand that (and in general am for that), but we've also had issues posted in the lando queue asking how to allow xdebug on drush commands, so we don't want to break things for those users (#874)

so like many things it is not necessarily simple to implement this

that doesn't mean we can't or shouldn't think about it; just need to consider multiple use cases.

@tanc

This comment has been minimized.

Copy link
Contributor

tanc commented Sep 24, 2018

At the moment there is no choice, its not possible (as far as I understand) to have a separate php.ini for cli. If that choice was introduced then the question is more around what the default should be. A non-breaking change would be to clone the current php.ini settings to the (new) cli php.ini so xdebug would still work.

But as breaking changes are still allowed in this part of the release cycle is it worth changing the default for php cli so that xdebug is disabled and writing a change note so that uses who need to debug command line apps understand how to enable it?

@dustinleblanc

This comment has been minimized.

Copy link
Member

dustinleblanc commented Sep 24, 2018

I am all for getting separate ini files going as soon as we can and I agree simply cloning the existing file is a quicker win. I started working on this and it just turns out how we handle ini files in the code wasn't what I expected. We do some renaming/appending magic to make overrides work intuitively which somewhat interferes with the often default PHP system in some distros of having the two files. I'd love to get us there, its just more code changes than I anticipated. I am also not sure if it would impact the easy overridability and I definitely don't want to mess that up

@tanc

This comment has been minimized.

Copy link
Contributor

tanc commented Oct 3, 2018

I just hit the composer memory limit issue and can confirm the solution wturrell posted worked for me

@lhridley

This comment has been minimized.

Copy link

lhridley commented Oct 3, 2018

Running Lando version 3.0.0-rc.1, trying to install a project using a Pantheon recipe, with the following .lando.yml file:

name: example-app
recipe: pantheon
config:
  framework: drupal
  site: example-app
  id: 0
  php: '7.2'
  conf:
    php: php.ini
  webroot: web
  via: nginx

php.ini file (which is in my project root where the .lando.yml file is located):

memory_limit = -1

The php.ini file is being ignored, and composer is hitting a memory limit of 1610612736 consistently.

Docker is configured to use 4 CPUs and 8G ram on a MacBook Pro 15" with 16G and a quad core processor running Sierra 10.12.6

This is my first attempt to use Lando, trying to help a client troubleshoot a local dev environment, so far not having any success.

@anacona16

This comment has been minimized.

Copy link

anacona16 commented Oct 4, 2018

I solved using this config:

tooling:
  composer:
    service: appserver
    description: Run composer commands
    cmd:
      - php -dmemory_limit=-1 /usr/local/bin/composer
@pirog

This comment has been minimized.

Copy link
Member

pirog commented Oct 4, 2018

chiming in here @tanc @serundeputy @dustinleblanc

i think the fundamental thing that makes this harder than it should be is that there is no "easy" way, at least that i know of, to define a separate php.ini that loads just for cli ops in the underlying docker images that we are using.

At first i thought we might be able to do something like @anacona16 has done in comment above (and maybe that isnt a bad stop gap measure) but ideally we find something that "just works" for all php cli ops and something that we wouldn't have to add for each and every php tool we define in lando.

Another possible alternative we could explore is thinking about this from the other direction, eg set the default php config to be for the cli and then have apache and fpm set up so the load in needed "server" overrides eg a more restrictive memory limit, xdebug config, etc

@pirog

This comment has been minimized.

Copy link
Member

pirog commented Oct 4, 2018

@lhridley as far as i know the pantheon recipe does not allow you to use your own custom php.ini so i would not expect that to work. the reason for this is to preserve some level of parity with how the pantheon dev environment works.

@lhridley

This comment has been minimized.

Copy link

lhridley commented Oct 4, 2018

@pirog Thanks! that makes sense (I don't use Pantheon on a regular basis so I'm a bit rusty).

For development purposes though, it does appear to me that we need to address the composer memory limitation issue though, whether a project is hosted on Pantheon or not -- managing dependencies with composer and ultimately committing an artifact that Pantheon can successfully deploy should not be a mutually exclusive activity from a project maintenance standpoint.

@pirog

This comment has been minimized.

Copy link
Member

pirog commented Oct 4, 2018

@rjbain

This comment has been minimized.

Copy link

rjbain commented Nov 2, 2018

@pirog @dustinleblanc Is there any workaround while using the pantheon recipe? None of the solutions on this page seem to be working for me. This is what my .lando.yml looks like now:

name: XXXXXX
recipe: pantheon
config:
  webroot: .
  via: nginx
  php: '7.0'
  conf:
    php: conf/php.ini
  framework: drupal
  site: XXXXXX
  id: XXXXXX

tooling:
  composer:
    service: appserver
    description: Run composer commands
    cmd:
      - php -dmemory_limit=-1 /usr/local/bin/composer

This is for a D7 site and using drush for features is causing Error: Allowed memory size of 536870912 bytes exhausted. Thanks!

@rjbain

This comment has been minimized.

Copy link

rjbain commented Nov 2, 2018

Thanks, to @dustinleblanc I was able to get drush working by changing this section:

tooling:
  drush:
    service: appserver
    description: Run commands
    cmd: php -d memory_limit=-1 /usr/local/bin/drush
@pirog

This comment has been minimized.

Copy link
Member

pirog commented Nov 3, 2018

@dustinleblanc @serundeputy i dont think there has been much evolution in thought on this since #732 (comment) so i think it might be worth just hardcoding our php tools for now like the workarounds here eg with php -d memory_limit=-1 /path/to/my/thing until we have a more "complete" solution.

We could either do this directly with each internal php tool or we could provide some sort of internal tooling key like _php_tool: true which would know to prepend the php -d memory_limit=-1 to the command

@pirog pirog added this to the RC1 milestone Nov 28, 2018

@pirog pirog self-assigned this Nov 28, 2018

@gitressa

This comment has been minimized.

Copy link

gitressa commented Dec 3, 2018

I just got hit by this while exporting huge text-files, and @nicrodgers tip followed by lando rebuild worked perfectly. But having a general memory_limit=-1 would be even better.

@bob-hinrichs

This comment has been minimized.

Copy link

bob-hinrichs commented Jan 18, 2019

Tried everything in this thread, thank you. Running lando v3.0.0-rc.1 cli and web server both run out of memory and are always 512. I am attempting D8 migration tasks and am getting nowhere. Thanks for the work on this.

@dustinleblanc

This comment has been minimized.

Copy link
Member

dustinleblanc commented Jan 19, 2019

@pirog I went ahead and tried to implement the cmd: 'php -d memory_limit=-1 COMMAND' pattern and we have an issue where php can't infer the input script by passing the command name. I tried doing php -d memory_limit=-1 $(which COMMAND) which doesn't work either, still investigating how I can pass the fully qualified script path. I also noticed here: http://php.net/manual/en/configuration.file.php That we can load the memory limit into an env variable, so perhaps it would be another method to do like: PHP_MEMORY_LIMIT=-1 drush or something like that...

dustinleblanc added a commit that referenced this issue Jan 19, 2019

dustinleblanc added a commit that referenced this issue Jan 19, 2019

@Mar2zz

This comment has been minimized.

Copy link

Mar2zz commented Jan 21, 2019

Getting this error only on php 5.x containers. php 7 no problems.

@dustinleblanc

This comment has been minimized.

Copy link
Member

dustinleblanc commented Jan 22, 2019

@Mar2zz makes sense, 7.x uses significantly less memory to accomplish the same tasks.

pirog added a commit that referenced this issue Jan 30, 2019

pirog added a commit that referenced this issue Jan 30, 2019

Merge pull request #1357 from lando/732/php_cli_memory
Issue #732: Give PHP unlimited memory for tooling commands.

@pirog pirog closed this Jan 30, 2019

@dcorb

This comment has been minimized.

Copy link

dcorb commented Feb 11, 2019

lando php -d memory_limit=-1 /usr/local/bin/composer require drupal/simplesamlphp_auth worked for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment