parallelize git clone

[staabm]

in #9527 we are talking about the already parallelized archive extraction.

in a similar use case as described in #9527 when running composer global update when https://github.com/rectorphp/rector is updated, I get a very serial process of slow cloning xxx into cache commands.

would it make sense to parallelize the "clone into cache"-step in a similar way the archive process is already parallelized?

extract of a single composer update run which takes >5 minutes to complete:

// snip
  - Syncing symfony/finder (5.x-dev fd83055) into cache
  - Syncing symfony/filesystem (5.x-dev bb92ba7) into cache
  - Syncing symplify/smart-file-system (dev-master 9e158ca) into cache
  - Syncing symplify/symplify-kernel (dev-master 9934bcd) into cache
  - Syncing symplify/package-builder (dev-master ac3f4c7) into cache
  - Syncing symplify/easy-testing (dev-master c223941) into cache
  - Syncing symfony/deprecation-contracts (dev-main be5a366) into cache
  - Syncing symfony/yaml (5.x-dev 1ea2135) into cache
  - Syncing symfony/polyfill-php80 (dev-main 3a11f3d) into cache
  - Syncing symfony/polyfill-php73 (dev-main 8c0d39c) into cache
  - Syncing symfony/polyfill-mbstring (dev-main 401c9d9) into cache
  - Syncing symfony/http-foundation (5.x-dev e457627) into cache
  - Syncing symfony/http-client-contracts (dev-main 41db680) into cache
  - Syncing psr/event-dispatcher (dev-master 3ef040d) into cache
  - Syncing symfony/event-dispatcher-contracts (dev-main 8e3016b) into cache
  - Syncing symfony/event-dispatcher (5.x-dev aa13a09) into cache
  - Syncing symfony/var-dumper (5.x-dev 173a79c) into cache
  - Syncing psr/log (dev-master dd738d0) into cache
  - Syncing symfony/error-handler (5.x-dev 289008c) into cache
  - Syncing symfony/http-kernel (5.x-dev 3b9e61c) into cache
  - Syncing psr/container (dev-master 381524e) into cache
  - Syncing symfony/service-contracts (dev-main 0aeee2f) into cache
  - Syncing symfony/dependency-injection (5.x-dev d7a2980) into cache
  - Syncing symfony/polyfill-intl-normalizer (dev-main 69609f9) into cache
  - Syncing symfony/polyfill-intl-grapheme (dev-main be09274) into cache
  - Syncing symfony/string (5.x-dev 40e975e) into cache
  - Syncing symfony/console (5.x-dev 3f7fc42) into cache
  - Syncing symfony/config (5.x-dev fa1219e) into cache
  - Syncing nette/neon (dev-master 05e7f53) into cache
  - Syncing symplify/composer-json-manipulator (dev-master c508eb7) into cache
  - Syncing symplify/autowire-array-parameter (dev-master 6b04292) into cache
  - Syncing symplify/skipper (dev-master 45e0b21) into cache
  - Syncing phpstan/phpdoc-parser (dev-master 33d5f73) into cache
  - Syncing symplify/simple-php-doc-parser (dev-master 8b75587) into cache
  - Syncing symplify/set-config-resolver (dev-master e3382b1) into cache
  - Syncing symplify/php-config-printer (dev-master babeccc) into cache
  - Syncing sebastian/diff (dev-master 43f41ff) into cache
  - Syncing symplify/markdown-diff (dev-master 1ae7e14) into cache
  - Syncing nette/robot-loader (dev-master 8e5690e) into cache
  - Syncing symplify/rule-doc-generator (dev-master 816fcba) into cache
  - Syncing symplify/console-color-diff (dev-master 9c6fc82) into cache
  - Syncing symfony/process (5.x-dev 240e741) into cache
  - Syncing symfony/var-exporter (5.x-dev fbc3507) into cache
  - Syncing psr/cache (dev-master b9aa2cd) into cache
  - Syncing symfony/cache-contracts (dev-main b1128c0) into cache
  - Syncing symfony/cache (5.x-dev c15fd2b) into cache
  - Syncing psr/simple-cache (dev-master 5a7b96b) into cache
  - Syncing phpstan/phpstan (dev-master 0cc663d) into cache
  - Syncing phpstan/phpstan-phpunit (dev-master ef674b8) into cache
  - Syncing jean85/pretty-package-versions (dev-master a917488) into cache
  - Syncing doctrine/inflector (2.1.x-dev c6a0da4) into cache
  - Syncing doctrine/annotations (dev-master 5cb4064) into cache
  - Syncing rector/rector (dev-master d962e3f) into cache
// snip

[Seldaek]

It's just not worth the complexity IMO - the GitDownloader is a whole other beast than zip extraction, with retries and various URLs and whatnot, it's probably doable but in most projects you don't clone a whole lot, so I doubt it's worth it.

What may be worth doing is making the download step (Syncing x into cache) work in parallel as this is usually the slower bit due to network, and it is much simpler I believe, but I'd have to check.

Anyway run a rm -rf of your global vendor dir and then composer install --prefer-dist, that'll be faster and parallelized :)

[staabm]

Thx for your feedback.

Note to self: to achieve jordis suggestion one would need to adjust

composer/src/Composer/Downloader/GitDownloader.php

Line 71 in 378a5b7

if ($this->gitUtil->fetchRefOrSyncMirror($url, $cachePath, $ref) && is_dir($cachePath)) {

to get async and return a promise similar to

composer/src/Composer/Downloader/ZipDownloader.php

Line 115 in 378a5b7

$promise = $this->process->executeAsync($command);