Limit the import of versions of packages for a VCS Repository

[francoispluchino]

On the install, or especially during the update, an VCS repository containing many versions can become extremely slow.

For example, 2 slow packages:

• jquery/jquery with 7 branchs and 122 tags
• angular/bower-angular with 2 branchs and 1 683 tags (extremely slow)

Solution 1:

The solution would be to add in the configuration of root Composer package, a constraint version for each package. In this way the Solver will search only in packages really imported, and therefore, it would limit the number of import of the informations of each package.

Example of configuration:

{
    "extra": {
        "asset-package-filters": {
            "bower-asset/jquery": ">=2.1@stable"
        }
    }
}

With this example, Composer will import only 2 versions of jquery (2.1.0 and 2.1.1) for the update.

Of course, if your project requires a dependency with a non-imported version, an exception will thrown, but this exception may not indicate that this is caused by the filter of asset package version. It is also for this reason that the configuration is only found in the root Composer package.

Solution 2:

This solution doesn't add any configuration, but uses the dependencies in the root Composer package. It complements the lazy loading system (already implemented).

At the plugin initialization,the plugin retrieves the list of dependencies added to the root Composer package. If the asset packages are defined, then the plugin will add only the branches and tags accepted by the constraint version defined in the root Composer package.

In this way, we will keep the same experience that currently, tout en but also keeping the native behavior of dependency analysis, while dramatically optimizing the number of requests for getting the informations of packages.

Example:

{
    "name": "acme/root-project",
    "minimum-stability": "dev",
    "require": {
        "bower-asset/jquery": "2.1.*",
        "bower-asset/boostrap": "3.2.*"
    }
}

With this example, Composer will import only 2 versions of jquery (2.1.0 and 2.1.1) for the update, even if bower-asset/bootstrap has a dependency for jquery >=1.9.

[ghost]

I made
"asset-package-filters": {
"bower-asset/angular": ">=1.2.25@stable",
"bower-asset/angular-resource": ">=1.2.26@stable"
}

It limited reading for angular, but not for angular-resources.

[francoispluchino]

@walletzilla It's a proposition of implementation, not a feature already implemented! :-)

[cebe]

I'd vote for Solution 2 as this is a natural approach and works in most cases without the user knowing about this. Solution 1 is not good imo, performance should not be configurable.

[francoispluchino]

@cebe we are all agree. I would do the tests tomorrow.

[francoispluchino]

Solution 2 added by c1d81f0.

[francoispluchino]

@cebe The update of doc and the FAQs will be available soon.

[cebe]

I have just tested this with the latest version of composer, running the following command:

composer create-project -vvv --prefer-dist --stability=dev yiisoft/yii2-app-basic basic

I get the following error after all packages are installed:

  [Composer\DependencyResolver\SolverProblemsException]                                                                     
    Problem 1                                                                                                               
      - yiisoft/yii2-bootstrap dev-master requires bower-asset/bootstrap 3.2.* | 3.1.* -> no matching package found.        
      - yiisoft/yii2-bootstrap 2.0.x-dev requires bower-asset/bootstrap 3.2.* | 3.1.* -> no matching package found.         
      - Installation request for yiisoft/yii2-bootstrap * -> satisfiable by yiisoft/yii2-bootstrap[2.0.x-dev, dev-master].  
  Potential causes:                                                                                                         
   - A typo in the package name                                                                                             
   - The package is not available in a stable-enough version according to your minimum-stability setting                    
     see <https://groups.google.com/d/topic/composer-dev/_g3ASeIFlrc/discussion> for more details.                          
  Read <http://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.                                



Exception trace:
 () at phar:///usr/local/bin/composer.phar/src/Composer/DependencyResolver/Solver.php:189
 Composer\DependencyResolver\Solver->solve() at phar:///usr/local/bin/composer.phar/src/Composer/Installer.php:266
 Composer\Installer->run() at phar:///usr/local/bin/composer.phar/src/Composer/Command/CreateProjectCommand.php:168
 Composer\Command\CreateProjectCommand->installProject() at phar:///usr/local/bin/composer.phar/src/Composer/Command/CreateProjectCommand.php:129
 Composer\Command\CreateProjectCommand->execute() at phar:///usr/local/bin/composer.phar/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:252
 Symfony\Component\Console\Command\Command->run() at phar:///usr/local/bin/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:889
 Symfony\Component\Console\Application->doRunCommand() at phar:///usr/local/bin/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:193
 Symfony\Component\Console\Application->doRun() at phar:///usr/local/bin/composer.phar/src/Composer/Console/Application.php:135
 Composer\Console\Application->doRun() at phar:///usr/local/bin/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:124
 Symfony\Component\Console\Application->run() at phar:///usr/local/bin/composer.phar/src/Composer/Console/Application.php:84
 Composer\Console\Application->run() at phar:///usr/local/bin/composer.phar/bin/composer:43
 require() at /usr/local/bin/composer.phar:15

Verbose composer output is here: https://gist.github.com/cebe/b69c1694b03632ff5a8a

Despite the error it is running really fast compared to before :)

[cebe]

We are using version constraints like "bower-asset/bootstrap": "3.2.* | 3.1.*" which are not covered by your tests as far as I see. Maybe the problem is related to that?

[francoispluchino]

@cebe There is no asset dependencies in the root package, I do not understand why there is this error. I look this tomorrow morning.

[francoispluchino]

For example for jquery, the dev-master branch is selected and not the 2.1.1. I think that the problem comes from here.

[cebe]

Thanks!

[francoispluchino]

@cebe Fixed by bc35a60.

It's a simple mistake by renaming the method !!!! I have not changed the return value... When the package is not in the list of dependencies in the root package, it must be added, not the skipped (to originally, the method called 'keepTag').

[francoispluchino]

For example, if you add the dependencies bellow in your root Composer package:

{
    "require": {
        "bower-asset/jquery": ">=2.1@beta",
        "bower-asset/punycode": ">=1.3.1@stable",
        "bower-asset/jquery.inputmask": ">=3.1.27@stable",
        "bower-asset/bootstrap": ">=3.2@stable",
        "bower-asset/typeahead.js": ">=0.10.5@stable"
    }
}

you will get:

$ composer update --prefer-dist -v
Loading composer repositories with package information
Updating dependencies (including require-dev)
Reading bower.json of bower-asset/jquery (2.1.1-RC2)
Importing tag 2.1.1-RC2 (2.1.1.0-RC2)
Reading bower.json of bower-asset/jquery (2.1.1-RC1)
Importing tag 2.1.1-RC1 (2.1.1.0-RC1)
Reading bower.json of bower-asset/jquery (2.1.1-beta1)
Importing tag 2.1.1-beta1 (2.1.1.0-beta1)
Reading bower.json of bower-asset/jquery (2.1.1)
Importing tag 2.1.1 (2.1.1.0)
Reading bower.json of bower-asset/jquery (2.1.0)
Importing tag 2.1.0 (2.1.0.0)
Reading bower.json of bower-asset/jquery.inputmask (3.1.27)
Importing tag 3.1.27 (3.1.27.0)
Reading bower.json of bower-asset/punycode (v1.3.1)
Importing tag v1.3.1 (1.3.1.0)
Reading bower.json of bower-asset/yii2-pjax (v2.0.0)
Importing tag v2.0.0 (2.0.0.0)
Reading bower.json of bower-asset/bootstrap (v3.2.0)
Importing tag v3.2.0 (3.2.0.0)
Reading bower.json of bower-asset/typeahead.js (v0.10.5)
Importing tag v0.10.5 (0.10.5.0)
  - Installing yiisoft/yii2-composer (dev-master 458cae9)
    Loading from cache
    Extracting archive

  - Installing bower-asset/jquery (2.1.1)
    Loading from cache
    Extracting archive

  - Installing bower-asset/yii2-pjax (v2.0.0)
    Loading from cache
    Extracting archive

  - Installing bower-asset/punycode (v1.3.1)
    Loading from cache
    Extracting archive

  - Installing bower-asset/jquery.inputmask (3.1.27)
    Loading from cache
    Extracting archive

  - Installing ezyang/htmlpurifier (v4.6.0)
    Loading from cache
    Extracting archive

  - Installing bower-asset/bootstrap (v3.2.0)
    Loading from cache
    Extracting archive

  - Installing swiftmailer/swiftmailer (dev-master 9359e59)
    Loading from cache
    Extracting archive

  - Installing cebe/markdown (0.9.x-dev 6a3c373)
    Loading from cache
    Extracting archive

  - Installing yiisoft/yii2 (dev-master e16233a)
    Loading from cache
    Extracting archive

  - Installing phpspec/php-diff (dev-master 30e103d)
    Loading from cache
    Extracting archive

  - Installing yiisoft/yii2-swiftmailer (dev-master 4ac514c)
    Loading from cache
    Extracting archive

  - Installing bower-asset/typeahead.js (dev-master 0fd2467)
    Loading from cache
    Extracting archive

  - Installing yiisoft/yii2-codeception (dev-master e47cb29)
    Loading from cache
    Extracting archive

  - Installing yiisoft/yii2-bootstrap (dev-master 362bda6)
    Loading from cache
    Extracting archive

  - Installing yiisoft/yii2-debug (dev-master db03889)
    Loading from cache
    Extracting archive

  - Installing yiisoft/yii2-gii (2.0.0-beta)
    Loading from cache
    Extracting archive

  - Installing fzaninotto/faker (dev-master 1ce76e8)
    Loading from cache
    Extracting archive

  - Installing yiisoft/yii2-faker (dev-master 7c4de16)
    Loading from cache
    Extracting archive

Writing lock file
Generating autoload files

[francoispluchino]

Doc: 80e597d.
FAQs: db83a75.

[cebe]

Great, it is working now. Thanks!