Add support for one-file-per-provider composer repositories #1168

Merged
merged 12 commits into from Oct 21, 2012

Conversation

Projects
None yet
9 participants
@Seldaek
Member

Seldaek commented Oct 1, 2012

This reduces the memory usage dramatically: it only loads stuff that it needs now, so it should not grow along with the packages on packagist, but rather with the amount of dependencies you have.
It also contains some performance improvements in a couple places which should speed things up a bit.

To try it you can download this preview phar: http://getcomposer.org/composer-new.phar

To run a comparison, you can run your existing composer + the new one like this (be sure to add --dry-run to avoid applying any changes):

$ composer update --dry-run --profile 
$ php composer-new.phar update --dry-run --profile

Here are the results I got trying it on composer itself:

First run (clear cache):

OLD: Memory usage: 60.17MB (peak: 67.52MB), time: 19.59s
NEW: Memory usage: 10.86MB (peak: 14.29MB), time: 20.79s

Second run (primed cache):

OLD: Memory usage: 60.17MB (peak: 67.52MB), time: 10.63s
NEW: Memory usage: 10.86MB (peak: 14.29MB), time: 3.97s

As you can see, the first runs are almost equal because they both need to download a bunch of files (larger ones for OLD, more smaller ones for NEW), but then it gets much faster, and especially the memory usage is reduced quite a bit.

I would be interested to get feedback from people with large projects that encountered issue with memory limits of 512MB+, to see how much it's reduced.

@stof

This comment has been minimized.

Show comment Hide comment
@stof

stof Oct 1, 2012

Contributor

Wouldn't this break the priority of repositories ?

Contributor

stof commented Oct 1, 2012

Wouldn't this break the priority of repositories ?

@Seldaek

This comment has been minimized.

Show comment Hide comment
@Seldaek

Seldaek Oct 1, 2012

Member

It has bigger issues right now, but no I don't think so since https://github.com/Seldaek/composer/blob/6a403839329ab361bcac94a2d2e19e2be308f0be/src/Composer/DependencyResolver/Pool.php#L178 is independant of the stuff I changed.

Member

Seldaek commented Oct 1, 2012

It has bigger issues right now, but no I don't think so since https://github.com/Seldaek/composer/blob/6a403839329ab361bcac94a2d2e19e2be308f0be/src/Composer/DependencyResolver/Pool.php#L178 is independant of the stuff I changed.

@Seldaek

This comment has been minimized.

Show comment Hide comment
@Seldaek

Seldaek Oct 15, 2012

Member

Ok I updated the code & the issue description above with instructions how to try it. It would be nice to get feedback.

Member

Seldaek commented Oct 15, 2012

Ok I updated the code & the issue description above with instructions how to try it. It would be nice to get feedback.

@RobLoach

This comment has been minimized.

Show comment Hide comment
@RobLoach

RobLoach Oct 15, 2012

Contributor

Had no problems running through Drupal's composer.json. That dependency tree is less than complicated though. Ran it on a few others without any issue. I'll ask around to see if there was anyone who had the memory issue popping up.

Contributor

RobLoach commented Oct 15, 2012

Had no problems running through Drupal's composer.json. That dependency tree is less than complicated though. Ran it on a few others without any issue. I'll ask around to see if there was anyone who had the memory issue popping up.

@petesiss

This comment has been minimized.

Show comment Hide comment
@petesiss

petesiss Oct 15, 2012

1st run

old: Memory usage: 107.85MB (peak: 189.44MB), time: 24.7s
new: Memory usage: 38.12MB (peak: 94.79MB), time: 28.46s

2nd run

old: Memory usage: 107.85MB (peak: 189.44MB), time: 22.14s
new: Memory usage: 38.12MB (peak: 94.8MB), time: 12.52s

Seems pretty good to me.

1st run

old: Memory usage: 107.85MB (peak: 189.44MB), time: 24.7s
new: Memory usage: 38.12MB (peak: 94.79MB), time: 28.46s

2nd run

old: Memory usage: 107.85MB (peak: 189.44MB), time: 22.14s
new: Memory usage: 38.12MB (peak: 94.8MB), time: 12.52s

Seems pretty good to me.

@baldurrensch

This comment has been minimized.

Show comment Hide comment
@baldurrensch

baldurrensch Oct 15, 2012

Contributor

1st run

old: Memory usage: 109.98MB (peak: 212.64MB), time: 88.25s
new: Memory usage: 39.29MB (peak: 117.38MB), time: 118.28s

2nd run

old: Memory usage: 109.98MB (peak: 212.64MB), time: 102.54s
new: Memory usage: 39.29MB (peak: 117.37MB), time: 38.39s

I like it!

Contributor

baldurrensch commented Oct 15, 2012

1st run

old: Memory usage: 109.98MB (peak: 212.64MB), time: 88.25s
new: Memory usage: 39.29MB (peak: 117.38MB), time: 118.28s

2nd run

old: Memory usage: 109.98MB (peak: 212.64MB), time: 102.54s
new: Memory usage: 39.29MB (peak: 117.37MB), time: 38.39s

I like it!

@simensen

This comment has been minimized.

Show comment Hide comment
@simensen

simensen Oct 15, 2012

Contributor

1st run

OLD: Memory usage: 151.58MB (peak: 211.35MB), time: 32.25s
NEW: Memory usage: 39.81MB (peak: 81.82MB), time: 59.83s

2nd run

OLD: Memory usage: 151.57MB (peak: 211.34MB), time: 13.32s
NEW: Memory usage: 39.81MB (peak: 81.8MB), time: 7.1s

In my case the cold cache startup time of the new build is almost twice as long the old version. Primed cache calls are half as long as the old version.

I have a git repository (type: git) that is being loaded. Is it possible that additional git repositories would take an upfront hit with the new build?

The memory benefits are great so I think that if this meant you either need to deal with long git repositories warming up or putting them all under satis I think that would be a good trade.

Contributor

simensen commented Oct 15, 2012

1st run

OLD: Memory usage: 151.58MB (peak: 211.35MB), time: 32.25s
NEW: Memory usage: 39.81MB (peak: 81.82MB), time: 59.83s

2nd run

OLD: Memory usage: 151.57MB (peak: 211.34MB), time: 13.32s
NEW: Memory usage: 39.81MB (peak: 81.8MB), time: 7.1s

In my case the cold cache startup time of the new build is almost twice as long the old version. Primed cache calls are half as long as the old version.

I have a git repository (type: git) that is being loaded. Is it possible that additional git repositories would take an upfront hit with the new build?

The memory benefits are great so I think that if this meant you either need to deal with long git repositories warming up or putting them all under satis I think that would be a good trade.

@simensen

This comment has been minimized.

Show comment Hide comment
@simensen

simensen Oct 15, 2012

Contributor

I found that --dev was giving me issues similar to those reported in this tweet. I've been able to reproduce this with a standalone composer.json and the composer.json and full instructions can be found here.

The first big error message seems to indicate to me something weird is going on with how extensions are being handled. The smaller error message is hopefully directly related to this. :)

...
Installing dev dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for ext-pdo_sqlite == 1.0.1.0 -> satisfiable by ext-pdo_sqlite 1.0.1.
    - ext-pdo_sqlite 1.0.1 requires doctrine/common master-dev -> no matching package found.
  Problem 2
    - Installation request for ext-sqlite3 == 0.7.0.0-dev -> satisfiable by ext-sqlite3 0.7-dev.
    - ext-sqlite3 0.7-dev requires doctrine/common master-dev -> no matching package found.
  Problem 3
    - Installation request for ext-xml == 0.0.0.0 -> satisfiable by ext-xml 0.
    - Can only install one of: ext-xmlreader 0.1, ext-xml 0.
    - Installation request for ext-xmlreader == 0.1.0.0 -> satisfiable by ext-xmlreader 0.1.

Potential causes:
 - A typo in the package name
 - The package is not available in a stable-enough version according to your minimum-stability setting
   see <https://groups.google.com/d/topic/composer-dev/_g3ASeIFlrc/discussion> for more details.

Read <http://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.
Contributor

simensen commented Oct 15, 2012

I found that --dev was giving me issues similar to those reported in this tweet. I've been able to reproduce this with a standalone composer.json and the composer.json and full instructions can be found here.

The first big error message seems to indicate to me something weird is going on with how extensions are being handled. The smaller error message is hopefully directly related to this. :)

...
Installing dev dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for ext-pdo_sqlite == 1.0.1.0 -> satisfiable by ext-pdo_sqlite 1.0.1.
    - ext-pdo_sqlite 1.0.1 requires doctrine/common master-dev -> no matching package found.
  Problem 2
    - Installation request for ext-sqlite3 == 0.7.0.0-dev -> satisfiable by ext-sqlite3 0.7-dev.
    - ext-sqlite3 0.7-dev requires doctrine/common master-dev -> no matching package found.
  Problem 3
    - Installation request for ext-xml == 0.0.0.0 -> satisfiable by ext-xml 0.
    - Can only install one of: ext-xmlreader 0.1, ext-xml 0.
    - Installation request for ext-xmlreader == 0.1.0.0 -> satisfiable by ext-xmlreader 0.1.

Potential causes:
 - A typo in the package name
 - The package is not available in a stable-enough version according to your minimum-stability setting
   see <https://groups.google.com/d/topic/composer-dev/_g3ASeIFlrc/discussion> for more details.

Read <http://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.
@marcosgdf

This comment has been minimized.

Show comment Hide comment
@marcosgdf

marcosgdf Oct 15, 2012

I'd like to run a benchmark like those but I don't understand how to clear composer's cache. Is there any special command?

I'd like to run a benchmark like those but I don't understand how to clear composer's cache. Is there any special command?

@Seldaek

This comment has been minimized.

Show comment Hide comment
@Seldaek

Seldaek Oct 16, 2012

Member

@simensen nope git repos loading should not be affected by these changes. I'll investigate the ext issue, thanks.

@marcosgdf you can rm -rf ~/.composer/cache/* if you're on *nix, or C:/Users/<user>/AppData/Roaming/Composer/cache/ on windows.

Member

Seldaek commented Oct 16, 2012

@simensen nope git repos loading should not be affected by these changes. I'll investigate the ext issue, thanks.

@marcosgdf you can rm -rf ~/.composer/cache/* if you're on *nix, or C:/Users/<user>/AppData/Roaming/Composer/cache/ on windows.

@marcosgdf

This comment has been minimized.

Show comment Hide comment
@marcosgdf

marcosgdf Oct 16, 2012

1st run (clear cache)

OLD: Memory usage: 223MB (peak: 533.47MB), time: 140.14s
NEW: Memory usage: 82.05MB (peak: 342.33MB), time: 177.8s

2nd run:

OLD: Memory usage: 222.78MB (peak: 533.21MB), time: 110.26s
NEW: Memory usage: 82.05MB (peak: 342.31MB), time: 57.28s

1st run (clear cache)

OLD: Memory usage: 223MB (peak: 533.47MB), time: 140.14s
NEW: Memory usage: 82.05MB (peak: 342.33MB), time: 177.8s

2nd run:

OLD: Memory usage: 222.78MB (peak: 533.21MB), time: 110.26s
NEW: Memory usage: 82.05MB (peak: 342.31MB), time: 57.28s
@davidwindell

This comment has been minimized.

Show comment Hide comment
@davidwindell

davidwindell Oct 17, 2012

👍 This fixes a problem I was experiencing

Fatal error: Out of memory (allocated 71041024) (tried to allocate 82 bytes) in phar:///home/zebreco/app/composer.phar/src/Composer/Json/JsonFile.php on line 278

👍 This fixes a problem I was experiencing

Fatal error: Out of memory (allocated 71041024) (tried to allocate 82 bytes) in phar:///home/zebreco/app/composer.phar/src/Composer/Json/JsonFile.php on line 278

@Seldaek

This comment has been minimized.

Show comment Hide comment
@Seldaek

Seldaek Oct 18, 2012

Member

@simensen @jcarouth I fixed the --dev issue and updated composer-new.phar if you want to try and confirm.

Member

Seldaek commented Oct 18, 2012

@simensen @jcarouth I fixed the --dev issue and updated composer-new.phar if you want to try and confirm.

@jcarouth

This comment has been minimized.

Show comment Hide comment
@jcarouth

jcarouth Oct 18, 2012

Confirmed fixed for me. Thanks @Seldaek

First run (clear cache):

OLD: Memory usage: 111.01MB (peak: 186.32MB), time: 71.09s
NEW: Memory usage: 37.49MB (peak: 87.71MB), time: 110.16s

Second run:

OLD: Memory usage: 111.01MB (peak: 186.32MB), time: 67.67s
NEW: Memory usage: 37.49MB (peak: 87.71MB), time: 27.44s

Confirmed fixed for me. Thanks @Seldaek

First run (clear cache):

OLD: Memory usage: 111.01MB (peak: 186.32MB), time: 71.09s
NEW: Memory usage: 37.49MB (peak: 87.71MB), time: 110.16s

Second run:

OLD: Memory usage: 111.01MB (peak: 186.32MB), time: 67.67s
NEW: Memory usage: 37.49MB (peak: 87.71MB), time: 27.44s
@simensen

This comment has been minimized.

Show comment Hide comment
@simensen

simensen Oct 18, 2012

Contributor

@Seldaek Confirmed that --dev no longer fails for me. :)

Contributor

simensen commented Oct 18, 2012

@Seldaek Confirmed that --dev no longer fails for me. :)

@Seldaek Seldaek referenced this pull request Oct 19, 2012

Closed

Composer just dies. #1183

@Seldaek Seldaek merged commit 5978197 into composer:master Oct 21, 2012

@Seldaek

This comment has been minimized.

Show comment Hide comment
@Seldaek

Seldaek Oct 21, 2012

Member

I merged this. Let's hope things go well for everyone.

Member

Seldaek commented Oct 21, 2012

I merged this. Let's hope things go well for everyone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment