Checking packages from custom repositories is very slow

[T2L]

Basically, when a package is not available in the main packagist.org repo Composer does a lot (really a lot) of network requests, which are cacheable and 304, which is OK locally.

But this nuance creates a lot of overhead leading to a very slow Dependabot runs, which in the end just time out

My composer.json:

{
  "require": {
    "drupal/address": "^2.0"
  },
  "repositories": {
    "_drupal": {
      "type": "composer",
      "url": "https://packages.drupal.org/8"
    }
  },
  "minimum-stability": "dev",
  "prefer-stable": true
}

Output of composer diagnose:

Checking composer.json: WARNING
No license specified, it is recommended to do so. For closed-source software you may use "proprietary" as license.
Checking platform settings: OK
Checking git settings: OK git version 2.42.0
Checking http connectivity to packagist: OK
Checking https connectivity to packagist: OK
Checking github.com oauth access: OK
Checking disk free space: OK
Checking pubkeys: 
Tags Public Key Fingerprint: 57815BA2 7E54DC31 7ECC7CC5 573090D0  87719BA6 8F3BB723 4E5D42D0 84A14642
Dev Public Key Fingerprint: 4AC45767 E5EC2265 2F0C1167 CBBB8A2B  0C708369 153E328C AD90147D AFE50952
OK
Checking composer version: You are not running the latest stable version, run `composer self-update` to update (2.5.8 => 2.6.3)
Composer version: 2.5.8
PHP version: 8.1.23
PHP binary path: /usr/local/Cellar/php@8.1/8.1.23/bin/php
OpenSSL version: OpenSSL 3.1.2 1 Aug 2023
cURL version: 8.3.0 libz 1.2.11 ssl (SecureTransport) OpenSSL/3.1.2
zip: extension present, unzip present, 7-Zip not available

When I run this command:

composer show symfony/debug -a -vvv

I get the following output (redacted to leave only network):

Reading /Users/user/Library/Caches/composer/repo/https---packages.drupal.org-8/packages.json from cache
Downloading https://packages.drupal.org/8/packages.json if modified
[304] https://packages.drupal.org/8/packages.json
Reading /Users/user/Library/Caches/composer/repo/https---repo.packagist.org/packages.json from cache
Downloading https://repo.packagist.org/packages.json if modified
[304] https://repo.packagist.org/packages.json
Reading /Users/user/Library/Caches/composer/repo/https---repo.packagist.org/provider-symfony~debug.json from cache
Downloading https://repo.packagist.org/p2/symfony/debug.json if modified
Reading /Users/user/Library/Caches/composer/repo/https---repo.packagist.org/provider-symfony~debug~dev.json from cache
Downloading https://repo.packagist.org/p2/symfony/debug~dev.json if modified
[304] https://repo.packagist.org/p2/symfony/debug.json
[304] https://repo.packagist.org/p2/symfony/debug~dev.json
Reading /Users/user/Library/Caches/composer/repo/https---repo.packagist.org/provider-psr~log.json from cache
Downloading https://repo.packagist.org/p2/psr/log.json if modified
Reading /Users/user/Library/Caches/composer/repo/https---repo.packagist.org/provider-symfony~polyfill-php80.json from cache
Downloading https://repo.packagist.org/p2/symfony/polyfill-php80.json if modified
Reading /Users/user/Library/Caches/composer/repo/https---repo.packagist.org/provider-psr~log~dev.json from cache
Downloading https://repo.packagist.org/p2/psr/log~dev.json if modified
Reading /Users/user/Library/Caches/composer/repo/https---repo.packagist.org/provider-symfony~polyfill-php80~dev.json from cache
Downloading https://repo.packagist.org/p2/symfony/polyfill-php80~dev.json if modified
[304] https://repo.packagist.org/p2/psr/log.json
[304] https://repo.packagist.org/p2/symfony/polyfill-php80.json
[304] https://repo.packagist.org/p2/psr/log~dev.json
[304] https://repo.packagist.org/p2/symfony/polyfill-php80~dev.json

but when I run

composer show drupal/address -a -vvv

I get the following output (redacted to leave only network):

See the attached log file: composer.log

There are literally ~60x more lines (and more network and cache readings)

[fredden]

From a cursory look, it seems this isn't anything to do with the repository set-up, but instead is due to the dependency tree for each package.

app@230ffa95c90a:~/html$ composer show -vvva composer/xdebug-handler 2>&1 | grep -F '] https://repo.packagist.org/'
[304] https://repo.packagist.org/packages.json
[304] https://repo.packagist.org/p2/composer/xdebug-handler.json
[304] https://repo.packagist.org/p2/composer/xdebug-handler~dev.json
[304] https://repo.packagist.org/p2/psr/log.json
[304] https://repo.packagist.org/p2/composer/pcre.json
[304] https://repo.packagist.org/p2/psr/log~dev.json
[304] https://repo.packagist.org/p2/composer/pcre~dev.json
app@230ffa95c90a:~/html$ composer show -vvva psr/log 2>&1 | grep -F '] https://repo.packagist.org/'
[304] https://repo.packagist.org/packages.json
[304] https://repo.packagist.org/p2/psr/log.json
[304] https://repo.packagist.org/p2/psr/log~dev.json
app@230ffa95c90a:~/html$ composer show -vvva composer/pcre 2>&1 | grep -F '] https://repo.packagist.org/'
[304] https://repo.packagist.org/packages.json
[304] https://repo.packagist.org/p2/composer/pcre.json
[304] https://repo.packagist.org/p2/composer/pcre~dev.json
app@230ffa95c90a:~/html$ jq < vendor/composer/xdebug-handler/composer.json .require
{
  "php": "^7.2.5 || ^8.0",
  "psr/log": "^1 || ^2 || ^3",
  "composer/pcre": "^1 || ^2 || ^3"
}
app@230ffa95c90a:~/html$ 

[stof]

The composer metadata loading involves 1 request per package to load the metadata about its releases (and a second one to load the branches in case you have minimum-stability: dev because the metadata about branches gets invalidated much more often than the metadata about releases and most projects use the default setting which does not need them).
This is done on purpose. Loading the metadata about all packages available on packagist in a single request would not be possible at all (this is far too big, and you would not even be able to fit the JSON in memory).

However, it looks weird to me that composer show -a composer/xdebug-handler would need to load the metadata about dependencies.
@Seldaek should the ShowCommand use RepositorySet::findPackages instead of creating a Pool ?

[Seldaek]

Yeah i need to check why we do this. It might be to be able to reference packages by the name they replace/provide too, but I'm not sure off the top of my head. Definitely should be able to optimize the common case here as this does seem wasteful

[T2L]

I've checked this one more time and I don't think it's related to use of the custom packagist repository, but probably to the dependency tree of the certain dependency. Any way to optimize that ?

[T2L]

PS It's obviously out of Composer's scope, but I think this is the code that checks for updates: https://github.com/dependabot/dependabot-core/blob/main/composer/helpers/v2/src/UpdateChecker.php

Any suggestion here for optimization?

[stof]

Well, this logic in dependabot runs the installer. So it is indeed a different topic than this issue which is about composer show. I suggest opening a separate discussion to keep this one on topic.

[Seldaek]

See #11659