Skip resolving dependencies for wp-cli/wp-cli to improve performance #2566

Closed
danielbachhuber opened this Issue Mar 16, 2016 · 3 comments

Projects

None yet

1 participant

@danielbachhuber
Member

If a package requires wp-cli/wp-cli as a dependency, we intelligently skip installing it by using the unintended side effect of a Composer implementation detail: #2546

However, when wp-cli/wp-cli is required as a package dependency, Composer still traverses the dependency graph, causing huge memory usage and a lot of unnecessary cycles. It would be nice if we could skip this entirely, given none of the dependencies will ever be installed.

@danielbachhuber danielbachhuber changed the title from wp package: Skip resolving dependencies for wp-cli/wp-cli to improve performance to Skip resolving dependencies for wp-cli/wp-cli to improve performance Mar 21, 2016
@danielbachhuber
Member

To be more specific, this is what I see when installing a package for the first time:

Installing package runcommand/hook (dev-master)
Updating /Users/danielbachhuber/salty-wordpress/wp-cli/packages/composer.json to require the package...
Using Composer to install the package...
---
Loading composer repositories with package information
Updating dependencies
Resolving dependencies through SAT
Dependency resolution completed in 0.077 seconds
Analyzed 3170 packages to resolve dependencies
Analyzed 72808 rules to resolve dependencies
 - Installing runcommand/hook (dev-master 1263610)
Writing lock file
Generating autoload files
---
Success: Package installed.

I wouldn't expect WP-CLI to have to analyze 3170 packages in order to install a package without a dependency.

@danielbachhuber
Member

schlessera [7:00 AM] That could have multiple issues. One thing is that there's no caching the first time around, so all requirements are parsed, all packages are parsed, and the solver tries to find the best match.
You might also have issues with circular dependencies. So, packageA => wp-cli => packageB => packageA. Such circular dependencies can quickly cause the system to go over pretty much everything, as the dependency tree will expand exponentially.
[7:01] I also see that you're using dev-master. Typically, proper releases are much faster than commits.
schlessera [7:08 AM] What does the composer.json look like for runcommand/hook?
schlessera [7:14 AM] You're probably fetching this dev-master version from a git repository, I assume. The problem is that satis does not have direct access to the composer.json for a git repository. So, basically, it loads every single release of every single branch to look into the composer.json file to find out what the version of the release is. The git tags are meaningless to Composer, they are just arbitrary tags.
[7:18] Just a guess, did you try this:


When GitHub or BitBucket repositories are mirrored on your local satis, the build process will include the location of the downloads these platforms make available. This means that the repository and your setup depend on the availability of these services.

At the same time, this implies that all code which is hosted somewhere else (on another service or for example in Subversion) will not have downloads available and thus installations usually take a lot longer.

To enable your satis installation to create downloads for all (Git, Mercurial and Subversion) your packages, add the following to your satis.json:

{
    "archive": {
        "directory": "dist",
        "format": "tar",
        "prefix-url": "https://amazing.cdn.example.org",
        "skip-dev": true
    }
}

[7:19] Did you look into the packages.json file on the satis side, to check what it pulls in?
danielbachhuber [8:18 AM] oh, huh
[8:18] @schlessera Satis is definitely not creating builds
schlessera [8:19 AM] So it's always cycling through all repos and branches?
danielbachhuber [8:20 AM] I guess so? I wouldn't think Satis would need to if it's only installing one package
schlessera [8:22 AM] The problem is that is has no information about the git repositories. A git repository named wordpress\core with the branch named master and the tag 4.7.0 could in reality be some old Tetris game, according to the composer.json file. The git information is not really semantically relevant for Composer/Satis. That's why all releases of all branches are downloaded, and for each of them, the composer.json gets parsed to find out what that package is actually about.

[8:22] If you have proper release builds, the package information is directly available to Composer/Satis, which makes this much, much faster.

@danielbachhuber
Member

Given this will add a good number of infrastructure requirements, I've added this to the wish list

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment