Parallel download of dist archives

[h4cc]

When using dist files for downloading, it would be very cool to have a parallel download of these archives.

Is there any chance to add such a feature?

[Seldaek]

There is a chance, but it would require quite a few changes in the way things are done, so right now it's not really on the roadmap.

[firstrow]

Really interesting feature!

[jakoch]

I just want to leave a small note here, that i'm working on this and already have a working version.
It was quite hard to find a way to add this functionality. I spend several hours to find a way to implement this. At first, i tried to use downloaders, but they are not only downloaders, but complete handlers and its not possible to intercept or overload their behaviour to parallelize them.
I skipped that approach completely and came up with a simple cache-warming-strategy.

Context: Installer->doInstall().
After the dependencies are resolved, but before installations/updates are run, the "$operations" stack is fetched. The $operations array contains the packages infos with the desired distUrls. The distUrls are placed into a download list, which is written to a temp file and handed over to an external download util. The files are downloaded to the cache folder. Then Composer proceeds with installations/updates and finds the cached files and re-uses them.

The following downloaders are implemented:

• "aria2c"
• "parallel+wget"
• "wget"
• "curl multi"

In other words: composer will use aria2c for parallalized downloading, if installed.

The effect on download speed is huge - especially when downloading with "aria2c".
That's due to server connection re-usage and parted downloads.

I have several questions.
Mostly, they are related to the level of output the external tool should provide and
when to provide ("debug", "dry-run" and "verbosity level") output.

1. The external download util produces "download progress" output.
   Should that be displayed or hidden by default?

2. The external download util has the option, to log to file.
   Only in debug mode? Where to store the log file? To the "home" folder?

3. Because files are downloaded to the cache, i guess, i need to wrap this into a "cache on" check.
   How can i check that, caching is active/enabled.. or is it on by default?

4. aria supports downloading one file from multiple servers (mirrors) in parallel.

• pre-condition: multiple dist urls are needed, referencing Allow set of multiple URLs for file dist type #2057 and [RFC] Multiple dist url per package #1233
• 2a7a954

{
   "type": "package",
   "package": {
   "name": "vendor/package",
   "version": "1.2.3",
   "dist": {
       "url": ["http://host.a/package-1.2.3.zip", "http://host.b/package-1.2.3.zip"],
       "type": "zip"
   }
}

Related: #3015

Regards, Jens

[Koc]

is #7904 closes this issue?

[stof]

@Koc no. That one implemented parallel downloading inside the Composer repository, which is a place managing a bunch of downloads on its own.
Supporting parallel downloading of archives requires refactoring the InstallationManager (as the current API deals with only 1 download at a time). I think this will be done only after the refactoring of the solver architecture is completed. Lower-level APIs of the downloaders have already been prepared for the parallel downloading in #7904 though.

[Seldaek]

Fixed by 006985a