Optimization suggestion for partial updates

[stof]

Currently, a partial update loads every possible package versions for all packages in the composer.json. However, most of them are locked to the version available in the lock file because of the exact match rule added during a partial update.
This could be taken into account when fetching providers for a dependency (similar to what is done based on root package requirements currently). We could even avoid loading the package metadata from Packagist entirely in this case.
I think this could make partial updates much faster than they are currently (where the are equivalent to a full update for the resolution part).

[stof]

@naderman you are probably the best one to deal with this.

[fduch]

👍

[naderman]

Yeah this is actually already on my todo list, Jordi tried loading smaller sets of packages from packagist a while ago and ran into some problems, but I'm going to try that again, as we no longer need all provider versions. So reducing what we need to load based on the lock file would be a next step to that.

[francoispluchino]

@naderman Is that this optimization can load only the package version that is the latest compatible version?

Example:

The package my/library with versions:

• 2.0.x-dev (dev-master)
• 1.1.0
• 1.0.5
• 1.0.4
• 1.0.3
• 1.0.2
• 1.0.1
• 1.0.0

The composer.json:

{
    "require": {
        "my/library": "1.0.*"
    }
}

Is that the 1.0.5 version will only be charged? Or is that the Composer will load all compatible packages (1.0.5, 1.0.4, 1.0.3, 1.0.2, 1.0.1, 1.0.0)?

Currently, this is the behavior for the Install. If this is not the case, it would be really interesting to have this optimization.

Regarding the reduction in the number of loading using the file lock, I have already done this optimization in my plugin, but there is a problem when a dependency change to the down version (ex. 1.0.1 to 1.0.0).

Do you have a technical solution (in the solver?) to avoid this problem?

[stof]

@francoispluchino no, because knowing that 1.0.5 should be selected depends on all other dependencies in the graph too.

This optimization suggestion is only about partial updates. If your lock files is locking the library to 1.0.4 and the package is not whitelisted in the package update, Composer is not allowed to select anything else than 1.0.4. So it does not actually need to look for other versions.

Btw, note that for composer install, Composer only reads the lock file already. So there is nothing to optimize in the dependency resolution process regarding the number of packages in this case (we are already limiting it to the needed packages only)

[francoispluchino]

It is therefore possible to create a package with only the name and version of the package, and load the full content only if necessary (lazy loading). In this manner, the Solver can check the package version and load only the version(s) that may correspond.

This optimization is most useful for VCS repositories, because the Composer must actually load all of all versions every time (even if there is a cache for tags).

My implementation of Lazy Loading is functional, but this implementation could be much more effective, if the Solver was able to take into account the Package with lazy loading. It would take that the Solver load the package dependencies only if it considers that the version can correspond, and not every time.

For the install with lock file:
Yes, Composer read only the intalled packages, and not analysis the dependencies.

For the install without lock file:
Only the corresponding packages to the lowest version required have the methods getRequires and getDevRequires called.

For the update (with or without lock file):
All packages are loaded every time.

My proposal still correspond to the optimization of the partial update, even if it goes further.

[stof]

An install without the lock file is exactly the same than an update (it just calls update internally, because there is nothing else to do here).

And no, your proposal does not correspond to my optimization proposal, as you are suggesting to change the way the solver works (reinjecting extra versions of a given package in the middle of the resolution will impact the resolution itself)

[ghost]

it's so sad that this feature still not implemented...

[Seldaek]

AFAIK this will be fixed by #4163 once it is mergeable (2.0..), just for reference.

[stof]

@Seldaek this could potentially be fixed earlier than 2.0, but making the same kind of optimization than the one done based on root requirements, simply doing it for locked requirements too.

[ruchernchong]

What is the update for this? This issue has been open for almost 2 years+ now.

[stof]

@ruchern nobody worked on implementing this, so this optimization is not yet there.

[douglas-srs]

I really hope someone work on this someday :/

[ruchernchong]

@stof thank you for the update. I really wish to see some optimisation heading towards Composer.

[stof]

@ruchern some optimizations have been done during these 2 years, but on different parts of Composer. This optimization opportunity concerns only partial updates, which might be why it was not the priority.

[jjsty1e]

is there something new for now ? just another year passed.

[stof]

@Jaggle for now, this is planned to be done in Composer 2.x as the refactoring of the solver will make this much easier to implement.
If you want to see this optimization happening in Composer 1.x, I suggest you to send a PR implementing it.

[greg-1-anderson]

I am working on an alternative strategy under Composer 1.x using a Composer plugin. I'm not sure that my experiment will be successful, but I am hopeful, and am going to give it a shot. At a minimum, perhaps my results will help inform the Composer 2 work.

See the README in g1a/reposer for details on the proposed tradeoffs.

[AntonioCS]

So is this still not solved?
I am doing an update on a big project, 100+ packages, and every time I get conflict after doing composer update I go in to composer.json fix the issue and then run again composer update -vvv and every time it downloads a bunch of the same things (including packages.json) even if it downloaded that file 2m ago....

Can we please fix this? I am running inside docker, inside a vm, net speed is no great, every composer update takes 5m+ to tell me it fails...

[stof]

@AntonioCS if the command you run is composer update -vvv, this issue is totally irrelevant for your case, as it is about partial updates.
And composer already has a cache of the Packagist metadata (but the main packages.json gets updated very often by Packagist).

@greg-1-anderson your plugin is not about implementing the optimization of partial updates suggested here. It is about changing the resolution algorithm entirely, moving away from a SAT solver (and expecting to have the root package resolve any conflict when the latest release cannot be used, which is a very bad experience compared to composer capabilities).