Load only explicitly named package data #3994

naderman · 2015-05-04T15:27:09Z

This will significantly reduce the amount of loaded metadata for the solver. This should improve network time as well as solver complexity.

…eplaced

stof · 2015-05-04T15:46:41Z

src/Composer/Repository/ComposerRepository.php

@@ -96,6 +96,151 @@ public function setRootAliases(array $rootAliases)
        $this->rootAliases = $rootAliases;
    }

+    public function ensureLoaded($pool, array $constrainedNames)


$pool should be typehinted

I don't mean to leave any of this code as-is, this was a POC to see if this is doable, I'll rewrite all of this anyway.

stof · 2015-05-04T15:48:14Z

src/Composer/Repository/ComposerRepository.php

+            $packages = $this->loadName($pool, $packageSpec['name']);
+
+            foreach ($packages as $package) {
+                foreach ($package->getRequires() as $link) {


shouldn't this be handled in the Pool to ensure that dependencies can be provided by other repositories ?

please read my reply :(

and what about dev requirements for the root package ?

@naderman your replies don't appear in the diff of the PR until I reload the diff 😄

stof · 2015-05-04T15:52:09Z

Something looks broken. Tests are not passing

naderman · 2015-05-04T15:53:28Z

Yes because the current list of loaded packages doesn't actually contain everything that is needed.

naderman · 2015-05-04T15:53:53Z

Again this is really just a FYI PR that I was discussing with jordi which I didn't want to just live on my disk, no need to discuss this yet.

naderman · 2015-05-04T16:08:21Z

current composer:

naderman@Montsoreau:~/projects/composer/test/ezpublish-community (master) $ ./composer.phar update -v --profile --dry-run --ignore-platform-reqs --no-dev                                                                                                                      
[8.6MB/0.11s] Loading composer repositories with package information
[8.9MB/0.37s] Updating dependencies
[312.7MB/29.30s] Memory usage: 312.66MB (peak: 899.13MB), time: 29.3s

with this patch which still loads full provider files, and only reduces the data after it is downloaded:

naderman@Montsoreau:~/projects/composer/test/ezpublish-community (master) $ ../../composer/bin/composer update -v --profile --dry-run --ignore-platform-reqs --no-dev
[8.9MB/0.11s] Loading composer repositories with package information
[9.2MB/0.35s] Updating dependencies
[259.6MB/22.85s] Memory usage: 259.6MB (peak: 759.64MB), time: 22.85s

So pretty confident this can make a significant performance impact if implemented properly.

… test

naderman · 2015-05-05T18:36:09Z

Alright I think this is ready to be reviewed as-is @Seldaek @stof - as a next step we could look into changing the json data that is downloaded off of packagist, this is unchanged in the PR for now.

stof · 2015-06-07T09:51:28Z

src/Composer/Repository/ComposerRepository.php

+        return $this->rootAliases;
+    }
+
+    public function loadRecursively(array $packageNames, $acceptableCallback)


please add some phpdoc to document that the second argument is a callable, as we cannot typehint it

naderman · 2015-06-08T17:00:11Z

Fixed everything @stof brought up - want to merge this then @Seldaek ?

Seldaek · 2015-06-18T10:39:01Z

Alright looking good, it reduces memory footprint a little and runtime as well since we process less files so that means less network time too which is nice.

A good side effect though is that it moves all the loading ahead of the solver, so there won't be confusion anymore as to what it's doing between downloading two files, now they should download quick and then solving starts.

ghost · 2015-06-18T13:20:25Z

@Seldaek @naderman After this was merged I've started to get errors when using private (Satis) and public packages together.
Here's a slightly obfuscated error message:

Problem 1
  - Installation request for private/package1 dev-master -> satisfiable by private/package1[dev-master].
  - private/package1 dev-master requires anchu/ftp 0.0.1 -> no matching package found.
Problem 2
   - Installation request for private/package2 dev-master -> satisfiable by private/package2[dev-master].
   - private/package2 dev-master requires squizlabs/php_codesniffer 1.* -> no matching package found.
 Problem 3
   - Installation request for private/package3 1.*@dev -> satisfiable by private/package3[1.0.x-dev].
   - private/package3 1.0.x-dev requires aws/aws-sdk-php 2.6.* -> no matching package found.

Reverting composer to a version right before this change (b025d09) fixes this.

stof · 2015-06-18T13:25:13Z

Do you have capital letters in the name of packages in your Satis repo ?

ghost · 2015-06-18T13:28:13Z

Nope, no capital letters in package names. There are capital letters in some version names though (branch names) - not sure if that matters.

stof · 2015-06-18T13:28:30Z

nope, it should not

naderman · 2015-06-18T13:32:26Z

Well the issue seems to be that for some reason some packages are not being loaded. Since this is multiple repositories I figure there is some issue in that section of the code, that tries to load names it found as deps in one repo in another.

naderman · 2015-06-18T13:32:57Z

Would be great if we had a test for this ^_^ Locally just using a satis repo works fine.

naderman · 2015-06-18T13:38:31Z

Going to revert this, while I debug, as this may take a while.

naderman added 4 commits May 4, 2015 17:25

POC Implementation of loading only explicitly named package data

4d0db5a

Remove debug output

65f69c4

ComposerRepository works differently now, so comment out test until r…

709943a

…eplaced

Remove dead code paths and use exceptions instead of die

9b9ad9d

stof reviewed May 4, 2015
View reviewed changes

naderman changed the title ~~Load only explicitly named package data~~ [WIP] Load only explicitly named package data May 4, 2015

stof reviewed May 4, 2015
View reviewed changes

naderman added 4 commits May 5, 2015 19:08

Completely move loading of packages from composer repo to pool

090711b

Entirely remove whatProvides from Composer repository

b99c03e

Update the composer repository whatprovides test to a loadrecursively…

da02c53

… test

loadDev parameter is nonsense, properly load packages in create command

ec5416f

naderman force-pushed the load-no-providers branch from caf91b4 to ec5416f Compare May 5, 2015 17:58

naderman added 4 commits May 5, 2015 20:24

PHP 5.3 compatible function invokation

aa7d145

Move private function further down in pool

cfd0e44

Add proper docblocks in composer repo

63e96a4

PHP 5.3 this/that compatibility

67f10c1

naderman changed the title ~~[WIP] Load only explicitly named package data~~ Load only explicitly named package data May 5, 2015

stof reviewed Jun 7, 2015
View reviewed changes

naderman added 2 commits June 8, 2015 18:45

Move methods for PR readability and fix phpdoc

65243ca

Optimisation: Composer repo version listings are index by package name

8b28d4b

Fix composer repo test to match what composer repos send

e25b86c

Seldaek merged commit e25b86c into composer:master Jun 18, 2015

naderman mentioned this pull request Jun 18, 2015

[WIP] RepositorySet / Pool Split & Load only explicitly referenced package data #4163

Closed

7 tasks

alcohol mentioned this pull request Jul 6, 2015

Composer needs npm like repo model #4216

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load only explicitly named package data #3994

Load only explicitly named package data #3994

naderman commented May 4, 2015

stof May 4, 2015

naderman May 4, 2015

stof May 4, 2015

naderman May 4, 2015

stof May 4, 2015

stof May 4, 2015

stof commented May 4, 2015

naderman commented May 4, 2015

naderman commented May 4, 2015

naderman commented May 4, 2015

naderman commented May 5, 2015

stof Jun 7, 2015

naderman commented Jun 8, 2015

Seldaek commented Jun 18, 2015

ghost commented Jun 18, 2015

stof commented Jun 18, 2015

ghost commented Jun 18, 2015

stof commented Jun 18, 2015

naderman commented Jun 18, 2015

naderman commented Jun 18, 2015

naderman commented Jun 18, 2015

Load only explicitly named package data #3994

Load only explicitly named package data #3994

Conversation

naderman commented May 4, 2015

stof May 4, 2015

Choose a reason for hiding this comment

naderman May 4, 2015

Choose a reason for hiding this comment

stof May 4, 2015

Choose a reason for hiding this comment

naderman May 4, 2015

Choose a reason for hiding this comment

stof May 4, 2015

Choose a reason for hiding this comment

stof May 4, 2015

Choose a reason for hiding this comment

stof commented May 4, 2015

naderman commented May 4, 2015

naderman commented May 4, 2015

naderman commented May 4, 2015

naderman commented May 5, 2015

stof Jun 7, 2015

Choose a reason for hiding this comment

naderman commented Jun 8, 2015

Seldaek commented Jun 18, 2015

ghost commented Jun 18, 2015

stof commented Jun 18, 2015

ghost commented Jun 18, 2015

stof commented Jun 18, 2015

naderman commented Jun 18, 2015

naderman commented Jun 18, 2015

naderman commented Jun 18, 2015