New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PHP 5.6 Ubuntu 16.04 test is failing due to out-of-memory error with Composer require #1875

Closed
geerlingguy opened this Issue Dec 7, 2018 · 9 comments

Comments

1 participant
@geerlingguy
Copy link
Owner

geerlingguy commented Dec 7, 2018

Issue Type

  • Bug Report / Support Request

Your Environment

Travis CI

Your OS

  • Linux

Full console output

1/1:\thttp://repo.packagist.org/p/provider-latest$f4412209f17e4f39a0ed33698639293eaf4ea5ccd5d58892188bdbe67a032a6f.json
    Finished: success: 1, skipped: 0, failure: 0, total: 1
Do not run Composer as root/super user! See https://getcomposer.org/root for details
[135.3MB/3.34s] ./composer.json has been updated
[139.1MB/5.28s]     1/1:\thttp://repo.packagist.org/p/provider-latest$d188fda52fa9abef88501d111349789deeb975ff1680976e2f50419767af62d0.json
[139.1MB/5.28s]     Finished: success: 1, skipped: 0, failure: 0, total: 1
[138.7MB/5.51s] > pre-update-cmd: DrupalProject\\composer\\ScriptHandler::checkComposerVersion
[140.6MB/5.52s] Loading composer repositories with package information
[141.1MB/5.96s] Updating dependencies (including require-dev)
PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 32 bytes) in phar:///usr/bin/composer/src/Composer/DependencyResolver/RuleWatchNode.php on line 40

Summary

In the PHP 5.6 test, it seems that Composer runs out of memory in the last step of a composer require drupal/drush:^1.2, when it does Updating dependencies (including require-dev).

This is currently breaking the Drupal VM automated test build—for example: https://travis-ci.org/geerlingguy/drupal-vm/jobs/465006817#L2468

It looks like something in the dependency resolver is just eating up all the RAM when building the dependency tree... but only on PHP 5.6. On 7.1 and 7.2 there is no issue.

@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented Dec 7, 2018

Note that this started happening sometime in the past week:

screen shot 2018-12-07 at 9 59 00 am

geerlingguy added a commit that referenced this issue Dec 7, 2018

@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented Dec 7, 2018

Bumping PHP memory_limit on that 5.6 test to 4 GB. Seems insane, but if it works, I'm willing to leave it at that, especially since 5.6 is not supported in a little over three weeks!

@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented Dec 7, 2018

That worked. Ugh, don't want to spend much more time on this for a dying PHP version, so I'll merge that and leave it alone.

@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented Dec 7, 2018

Just posting here since it worked locally but not with memory_limit = 2G, here are the steps to reproduce:

# Check PHP version.
$ php -v
PHP 7.2.10 (cli) (built: Sep 14 2018 07:07:08) ( NTS )

# Create new drupal project.
$ composer create-project drupal-composer/drupal-project:8.x-dev composer-test --prefer-dist --no-interaction
...
  - ../.editorconfig (https://cgit.drupalcode.org/drupal/plain/.editorconfig?h=8.6.4): Downloading (100%)         
  - ../.gitattributes (https://cgit.drupalcode.org/drupal/plain/.gitattributes?h=8.6.4): Downloading (100%)         
> DrupalProject\composer\ScriptHandler::createRequiredFiles
Create a sites/default/settings.php file with chmod 0666
Create a sites/default/files directory with chmod 0777

# Install Devel module.
$ cd composer-test
$ composer require drupal/devel:^1.2 -vv --profile
[134.3MB/1.16s] ./composer.json has been updated
[136.5MB/1.51s] > pre-update-cmd: DrupalProject\composer\ScriptHandler::checkComposerVersion
[138.0MB/1.52s] Loading composer repositories with package information
[138.4MB/1.99s] Updating dependencies (including require-dev)
[1232.3MB/15.10s] Dependency resolution completed in 0.727 seconds
[1232.7MB/15.18s] Analyzed 18912 packages to resolve dependencies
[1232.7MB/15.18s] Analyzed 1577548 rules to resolve dependencies
[1234.4MB/15.20s] Dependency resolution completed in 0.001 seconds
[1233.1MB/15.20s] Package operations: 1 install, 0 updates, 0 removals
[1233.1MB/15.20s] Installs: drupal/devel:1.2.0
[1233.1MB/15.20s] Patching is disabled. Skipping.
[1233.1MB/15.20s]   - Installing drupal/devel (1.2.0): [1233.1MB/15.21s] Loading from cache[1233.1MB/15.21s] 
[1233.1MB/15.21s]  Extracting archive[1233.2MB/15.48s]     REASON: Required by the root package: Install command rule (install drupal/devel 1.x-dev|install drupal/devel 1.2.0)
[1233.2MB/15.48s] 
[1233.2MB/15.48s] No patches found for drupal/devel.
[459.2MB/16.40s] Writing lock file
[459.2MB/16.40s] Generating autoload files
[459.5MB/16.70s] > post-update-cmd: DrupalProject\composer\ScriptHandler::createRequiredFiles
[459.3MB/16.90s] Memory usage: 459.33MB (peak: 1234.48MB), time: 16.9s
@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented Dec 7, 2018

To test with Composer and PHP 5.x:

# Run PHP 5.6 in a container.
$ docker run --rm -it drupaldocker/php-dev:5.6-cli /bin/bash
# php -v      
PHP 5.6.36 (cli) (built: Jun 20 2018 23:33:51) 

# composer create-project drupal-composer/drupal-project:8.x-dev composer-test --prefer-dist --no-interaction

# Install Devel module.
# cd composer-test
# composer require drupal/devel:^1.2 -vv --profile
Do not run Composer as root/super user! See https://getcomposer.org/root for details
[126.7MB/5.04s] ./composer.json has been updated
[129.6MB/6.08s] > pre-update-cmd: DrupalProject\composer\ScriptHandler::checkComposerVersion
[131.5MB/6.10s] Loading composer repositories with package information
[131.9MB/6.52s] Updating dependencies (including require-dev)
[2054.4MB/58.32s] Dependency resolution completed in 3.713 seconds
[2054.9MB/61.89s] Analyzed 18867 packages to resolve dependencies
[2054.9MB/61.89s] Analyzed 1577311 rules to resolve dependencies
[2056.9MB/62.68s] Dependency resolution completed in 0.002 seconds
[2055.5MB/62.69s] Package operations: 1 install, 0 updates, 0 removals
[2055.5MB/62.69s] Installs: drupal/devel:1.2.0
[2055.5MB/62.70s] Patching is disabled. Skipping.
[2055.5MB/62.80s]   - Installing drupal/devel (1.2.0): [2055.6MB/62.83s] [2055.6MB/63.02s] Downloading (0%)[2055.6MB/63.02s]                   [2[2055.6MB/63.04s] Downloading (5%)[2[2055.6MB/63.06s] Downloading (15%)[[2055.7MB/63.08s] Downloading (30%)[[2055.7MB/63.10s] Downloading (40%)[[2055.8MB/63.12s] Downloading (55%)[[2055.8MB/63.14s] Downloading (65%)[[2055.9MB/63.15s] Downloading (75%)[[2055.9MB/63.15s] Downloading (80%)[[2055.9MB/63.17s] Downloading (90%)[[2056.0MB/63.18s] Downloading (100%)[2055.5MB/63.19s] 
[2055.5MB/63.19s]  Extracting archive[2055.6MB/63.57s]     REASON: Required by the root package: Install command rule (install drupal/devel 1.x-dev|install drupal/devel 1.2.0)
[2055.6MB/63.57s] 
[2055.6MB/63.57s] No patches found for drupal/devel.
[731.5MB/71.30s] Writing lock file
[731.5MB/71.30s] Generating autoload files
[731.8MB/73.01s] > post-update-cmd: DrupalProject\composer\ScriptHandler::createRequiredFiles
[731.6MB/78.82s] Memory usage: 731.61MB (peak: 2057.24MB), time: 78.82s

So PHP 5.6 uses 823 MB more RAM, or 50% more RAM than PHP 7.2. Yowza.

@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented Dec 9, 2018

It was recommended by @zaporylie that I try out https://github.com/zaporylie/composer-drupal-optimizations to see if it saves a bit of RAM/makes for better performance. Testing that again now.

@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented Dec 9, 2018

$ docker run --rm -it drupaldocker/php-dev:5.6-cli /bin/bash
# composer create-project drupal-composer/drupal-project:8.x-dev composer-test --prefer-dist --no-interaction
# cd composer-test
# composer require zaporylie/composer-drupal-optimizations:^1.0 -vv --profile
...
[732.2MB/64.42s] Memory usage: 732.17MB (peak: 2004.29MB), time: 64.42s
# composer require drupal/devel:^1.2 -vv --profile
[127.1MB/3.03s] ./composer.json has been updated
[130.1MB/4.24s] > pre-update-cmd: DrupalProject\composer\ScriptHandler::checkComposerVersion
[132.1MB/4.26s] Loading composer repositories with package information
[132.5MB/4.97s] Updating dependencies (including require-dev)
[537.2MB/15.46s] Dependency resolution completed in 0.228 seconds
[537.7MB/15.54s] Analyzed 11729 packages to resolve dependencies
[537.7MB/15.54s] Analyzed 103511 rules to resolve dependencies
[539.7MB/15.58s] Dependency resolution completed in 0.002 seconds
[538.3MB/15.59s] Package operations: 1 install, 0 updates, 0 removals
[538.3MB/15.59s] Installs: drupal/devel:1.2.0
[538.3MB/15.59s] Patching is disabled. Skipping.
[538.3MB/15.59s]   - Installing drupal/devel (1.2.0): [538.4MB/15.60s] [538.4MB/15.74s] Downloading (0%)[538.4MB/15.74s]                  [[538.4MB/15.76s] Downloading (5%)[[538.4MB/15.78s] Downloading (15%)[538.5MB/15.80s] Downloading (30%)[538.5MB/15.80s] Downloading (40%)[538.6MB/15.81s] Downloading (55%)[538.6MB/15.82s] Downloading (65%[538.7MB/15.83s] Downloading (75%)[538.7MB/15.83s] Downloading (80%)[538.7MB/15.84s] Downloading (90%)[538.8MB/15.85s] Downloading (100%)[538.4MB/15.85s] 
[538.4MB/15.85s]  Extracting archive[538.4MB/15.90s]     REASON: Required by the root package: Install command rule (install drupal/devel 1.x-dev|install drupal/devel 1.2.0)
[538.4MB/15.90s] 
[538.4MB/15.90s] No patches found for drupal/devel.
[434.6MB/16.52s] Writing lock file
[434.6MB/16.52s] Generating autoload files
[434.9MB/16.93s] > post-update-cmd: DrupalProject\composer\ScriptHandler::createRequiredFiles
[434.1MB/17.28s] Memory usage: 434.1MB (peak: 540.02MB), time: 17.28s

So using that little project: 2057.24 MB --> 540.02 MB on PHP 5.6 (saves 1.5 GB of RAM!!!)

Wow.

Note that this doesn't help the use case of the first time composer require is used, though—that first run still took 2+ GB of RAM :(

So it would be best to get this upstreamed into the drupal project composer template.

Going to test on PHP 7.2 now to see what the reduction is there.

@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented Dec 9, 2018

PHP 7.2 - before:

[459.3MB/16.90s] Memory usage: 459.33MB (peak: 1234.48MB), time: 16.9s

PHP 7.1 - after composer require zaporylie/composer-drupal-optimizations:^1.0:

# composer require zaporylie/composer-drupal-optimizations:^1.0 -vv --profile
...
[454.0MB/23.45s] Memory usage: 453.99MB (peak: 1124.52MB), time: 23.45s
# composer require drupal/devel:^1.2 -vv --profile
...
[367.6MB/9.70s] Memory usage: 367.59MB (peak: 426.64MB), time: 9.7s

1234.48 MB --> 426.64 MB on PHP 7.1 (saves 800 MB of RAM!)

@geerlingguy

This comment has been minimized.

Copy link
Owner

geerlingguy commented Dec 31, 2018

I just realized I never did an after test on PHP 7.2...

$ composer require drupal/devel:^1.2 -vv --profile
...
[360.4MB/14.55s] Memory usage: 360.41MB (peak: 423.93MB), time: 14.55s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment