composer2 can't find package versions

[garak]

My composer.json:

none

Output of composer diagnose:

Checking platform settings: OK
Checking git settings: OK
Checking http connectivity to packagist: OK
Checking https connectivity to packagist: OK
Checking github.com oauth access: OK
Checking disk free space: OK
Checking pubkeys: 
Tags Public Key Fingerprint: 57815BA2 7E54DC31 7ECC7CC5 573090D0  87719BA6 8F3BB723 4E5D42D0 84A14642
Dev Public Key Fingerprint: 4AC45767 E5EC2265 2F0C1167 CBBB8A2B  0C708369 153E328C AD90147D AFE50952
OK
Checking composer version: OK
Composer version: 2.0.7
PHP version: 7.4.9
PHP binary path: /usr/bin/php7.4
OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
cURL version: 7.68.0 libz 1.2.11 ssl OpenSSL/1.1.1f
zip extension: OK

When I run this command:

composer show -a ergebnis/json-printer

I get the following output:

[...]
versions : 9999999-dev, dev-main, dev-dependabot/composer/ergebnis/php-cs-fixer-config-2.5.3, dev-dependabot/composer/phpunit/phpunit-8.5.9
[...]

And I expected this to happen:

I expect to see all versions (mainly, all stable versions are missing)
Following is output when running the same command with composer 1.10.1

[...]
versions : 3.1.1, 3.1.0, 3.0.2, 3.0.1, 3.0.0, 2.0.1, 2.0.0, 1.1.0, 1.0.0, dev-main, dev-dependabot/composer/phpunit/phpunit-8.5.9, dev-dependabot/composer/ergebnis/php-cs-fixer-config-2.5.3
[...]

[WyriHaximus]

Not sure why this happens but this seems to be the only package where I've ran into this. And it's impacting any package requiring this package such as: ergebnis/composer-normalize#613

[stof]

@Seldaek could it be that one of the dumped file is missing ? seeing only dev versions looks like it could be the cause, as they are in separate files in the v2 metadata

[Seldaek]

Yes that's most likely the cause.. a dumping or mirroring issue somehow affecting only that file.

[Seldaek]

It seems like one mirror had a corrupt ergebnis/json-printer containing metadata from ergonode/comment (dev versions too.. so not only name mismatch but dev stuff in a non-dev file.. I have no idea what the hell happened there. It seems to have happened during the last mirror resync I did on Oct 29th which does a few hundreds of thousands of requests, and so it seems like something went wrong there, bug or a weird bit flip I don't know. Could be a bug on my end or in symfony/http-client or in curl's http2 implementation.. This is loading everything quite a bit so weird stuff happens.

Anyway leaving this open for now until I can at least guard against the issue happening again, because I can't see right now what would be the bug on my end at least.

I made sure the file has a newer timestamp now so your corrupted local caches should get invalidated.

[WyriHaximus]

Can confirm it works now 👍 !

[localheinz]

Thank you, @Seldaek!

[stof]

@Seldaek maybe the mirroring script could implement some safeguards by checking that the package name inside the file matches the filename, to reject incorrect ones.

[Seldaek]

Yup that's the plan

[nicolas-grekas]

Silent data corruption? Not cool. This is reproducible I think with your mirroring script: some hashes don't match, on files that are verified by a hash. I saw it a few times, I should look closer at why... I'd suspect an issue with h2. Dunno if it can be detected by curl.

[jacques]

18:43 $ composer update --lock
Loading composer repositories with package information
Updating dependencies
Unable to find a compatible set of packages based on your non-dev requirements alone.
Your requirements can be resolved successfully when require-dev packages are present.
You may need to move packages from require-dev or some of their dependencies to require.

  Problem 1
    - Root composer.json requires league/csv ^9.6, found league/csv[9.5.0] but it does not match the constraint.

Saw this earlier today while trying to install league/csv:^9.6. Any ideas on what is causing this issue where packages randomly don't resolve? Seen this also yesterday with rector/rector where ^0.8 was not resolving as well.

[stof]

@jacques a composer update --lock will find packages only in your lock file, not on packagist, as you ask to run an update with locking all existing versions of paclages (assuming league/csv is in your lock file)

[Seldaek]

@jacques please report a new issue for this following the issue template strictly so we get enough information.

[nicolas-grekas]

@Seldaek does this content remind you something for https://repo.packagist.org/p/marcelomx/php-facedetection$07a4473494562fb666f69dd0f4d07ef70d48d705a44dfdc9259104c56d2028b7.json?

{"packages":{"marcelomx/php-facedetection":[]},"minified":"composer/2.0"}

[Seldaek]

Yeah that's most likely a package with no release so the v2 metadata for it is empty and it landed in this wrong file too.. what a mess :/

That's the original file at the correct url btw https://repo.packagist.org/p2/marcelomx/php-facedetection.json

[nicolas-grekas]

I'm not sure why I get this content for v2 under this URL then, while when I access the URL from the browser, I get the content for v1. Looks like something that http-client isn't responsible for, or could it be?

[nicolas-grekas]

Note that the script doesn't always choke on this specific URL. The exact URL is usually kinda random.

[nicolas-grekas]

Another failure, which might be a different one:

for https://repo.packagist.org/p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json, I get this (see below).
As you'll see, the json is about another package than the one in the URL 🤯
Again, this doesn't happen on the same URL all the time. The URL looks random.

{"packages":{"irmnet/fiscal":{"0.0.0":{"name":"irmnet/fiscal","description":"Compras module for IRMServices intranet.","keywords":["framework","zf2","doctrine","intranet"],"homepage":"","version":"0.0.0","version_normalized":"0.0.0.0","license":[],"authors":[],"source":{"type":"git","url":"https://github.com/IRMServ/IRMNet-Fiscal.git","reference":"f83c9eb1478ada59e2f51b95ac35c084ce815449"},"dist":{"type":"zip","url":"https://api.github.com/repos/IRMServ/IRMNet-Fiscal/zipball/f83c9eb1478ada59e2f51b95ac35c084ce815449","reference":"f83c9eb1478ada59e2f51b95ac35c084ce815449","shasum":""},"type":"library","time":"2013-03-21T17:28:46+00:00","autoload":{"psr-0":{"Fiscal":"src"},"classmap":["Module.php"]},"require":{"php":">=5.3.3"},"uid":43863},"0.0.1":{"name":"irmnet/fiscal","description":"Compras module for IRMServices intranet.","keywords":["framework","zf2","doctrine","intranet"],"homepage":"","version":"0.0.1","version_normalized":"0.0.1.0","license":[],"authors":[],"source":{"type":"git","url":"https://github.com/IRMServ/IRMNet-Fiscal.git","reference":"8af661a7f066971ab8f7d0b401d0bcb358ba83db"},"dist":{"type":"zip","url":"https://api.github.com/repos/IRMServ/IRMNet-Fiscal/zipball/8af661a7f066971ab8f7d0b401d0bcb358ba83db","reference":"8af661a7f066971ab8f7d0b401d0bcb358ba83db","shasum":""},"type":"library","time":"2013-06-04T15:15:35+00:00","autoload":{"psr-0":{"Fiscal":"src"},"classmap":["Module.php"]},"require":{"php":">=5.3.3"},"uid":59516},"dev-master":{"name":"irmnet/fiscal","description":"Compras module for IRMServices intranet.","keywords":["framework","zf2","doctrine","intranet"],"homepage":"","version":"dev-master","version_normalized":"9999999-dev","license":[],"authors":[],"source":{"type":"git","url":"https://github.com/IRMServ/IRMNet-Fiscal.git","reference":"8af661a7f066971ab8f7d0b401d0bcb358ba83db"},"dist":{"type":"zip","url":"https://api.github.com/repos/IRMServ/IRMNet-Fiscal/zipball/8af661a7f066971ab8f7d0b401d0bcb358ba83db","reference":"8af661a7f066971ab8f7d0b401d0bcb358ba83db","shasum":""},"type":"library","time":"2013-06-04T15:15:35+00:00","autoload":{"psr-0":{"Fiscal":"src"},"classmap":["Module.php"]},"default-branch":true,"require":{"php":">=5.3.3"},"uid":4053220}}}}

[nicolas-grekas]

Last but not least, the log from curl is the following.
The repeated errors (Connection died, retrying a fresh connect) that in the end switch to HTTP/1.1 while all the previous attempts started with HTTP/2 are a common pattern when I see a hash mismatch.
This log is generated by curl. No PHP code is involved here. curl must have an internal retry+fallback mechanism when connections are broken.

This kind of log makes me think the issue may be in curl (or the way we use it). On the other side, the previous issue (composer v2 metadata coming in) looks like a broken mirror somewhere.

TL;DR, I'm a bit lost. If you could check your mirrors to eliminate one possibility (a broken mirror for one of the submitted URL), that'd be cool.

    * Found bundle for host repo.packagist.org: 0x55d33ae22590 [can multiplex]\n
    * Multiplexed connection found!\n
    * Re-using existing connection! (#0) with host repo.packagist.org\n
    * Using Stream ID: 3865 (easy handle 0x55d34522d2b0)\n
    > GET /p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json HTTP/2\r\n
    Host: repo.packagist.org\r\n
    accept: */*\r\n
    user-agent: Symfony HttpClient/Curl\r\n
    accept-encoding: gzip\r\n
    \r\n
    * Connection died, retrying a fresh connect\n
    * Issue another request to this URL: 'https://repo.packagist.org/p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json'\n
    * Found bundle for host repo.packagist.org: 0x55d33ae22590 [can multiplex]\n
    * No more connections allowed to host repo.packagist.org: 6\n
    * No connections available.\n
    * Found bundle for host repo.packagist.org: 0x55d33ae22590 [can multiplex]\n
    * No more connections allowed to host repo.packagist.org: 6\n
    * No connections available.\n
    * Found bundle for host repo.packagist.org: 0x55d33ae22590 [can multiplex]\n
    * Multiplexed connection found!\n
    * Re-using existing connection! (#1) with host repo.packagist.org\n
    * Transfer was pending, now try another\n
    * Using Stream ID: 308b (easy handle 0x55d34522d2b0)\n
    > GET /p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json HTTP/2\r\n
    Host: repo.packagist.org\r\n
    accept: */*\r\n
    user-agent: Symfony HttpClient/Curl\r\n
    accept-encoding: gzip\r\n
    \r\n
    * OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0\n
    * Failed receiving HTTP2 data\n
    * Connection died, retrying a fresh connect\n
    * Issue another request to this URL: 'https://repo.packagist.org/p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json'\n
    * Found bundle for host repo.packagist.org: 0x55d33ae22590 [can multiplex]\n
    * Multiplexed connection found!\n
    * Re-using existing connection! (#2) with host repo.packagist.org\n
    * Transfer was pending, now try another\n
    * Using Stream ID: 291b (easy handle 0x55d34522d2b0)\n
    > GET /p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json HTTP/2\r\n
    Host: repo.packagist.org\r\n
    accept: */*\r\n
    user-agent: Symfony HttpClient/Curl\r\n
    accept-encoding: gzip\r\n
    \r\n
    * OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0\n
    * Failed receiving HTTP2 data\n
    * Connection died, retrying a fresh connect\n
    * Issue another request to this URL: 'https://repo.packagist.org/p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json'\n
    * Found bundle for host repo.packagist.org: 0x55d33ae22590 [can multiplex]\n
    * Multiplexed connection found!\n
    * Re-using existing connection! (#3) with host repo.packagist.org\n
    * Transfer was pending, now try another\n
    * Using Stream ID: 21bb (easy handle 0x55d34522d2b0)\n
    > GET /p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json HTTP/2\r\n
    Host: repo.packagist.org\r\n
    accept: */*\r\n
    user-agent: Symfony HttpClient/Curl\r\n
    accept-encoding: gzip\r\n
    \r\n
    * OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0\n
    * Failed receiving HTTP2 data\n
    * Connection died, retrying a fresh connect\n
    * Issue another request to this URL: 'https://repo.packagist.org/p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json'\n
    * Found bundle for host repo.packagist.org: 0x55d33ae22590 [can multiplex]\n
    * Multiplexed connection found!\n
    * Re-using existing connection! (#4) with host repo.packagist.org\n
    * Transfer was pending, now try another\n
    * Using Stream ID: 1a3f (easy handle 0x55d34522d2b0)\n
    > GET /p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json HTTP/2\r\n
    Host: repo.packagist.org\r\n
    accept: */*\r\n
    user-agent: Symfony HttpClient/Curl\r\n
    accept-encoding: gzip\r\n
    \r\n
    * OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0\n
    * Failed receiving HTTP2 data\n
    * Connection died, retrying a fresh connect\n
    * Issue another request to this URL: 'https://repo.packagist.org/p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json'\n
    * Found bundle for host repo.packagist.org: 0x55d33ae22590 [can multiplex]\n
    * Multiplexed connection found!\n
    * Re-using existing connection! (#5) with host repo.packagist.org\n
    * Transfer was pending, now try another\n
    * Using Stream ID: 12ed (easy handle 0x55d34522d2b0)\n
    > GET /p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json HTTP/2\r\n
    Host: repo.packagist.org\r\n
    accept: */*\r\n
    user-agent: Symfony HttpClient/Curl\r\n
    accept-encoding: gzip\r\n
    \r\n
    * Connection died, retrying a fresh connect\n
    * Issue another request to this URL: 'https://repo.packagist.org/p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json'\n
    * Found bundle for host repo.packagist.org: 0x55d33ae22590 [can multiplex]\n
    * Multiplexed connection found!\n
    * Re-using existing connection! (#6) with host repo.packagist.org\n
    * Transfer was pending, now try another\n
    * Using Stream ID: b25 (easy handle 0x55d34522d2b0)\n
    > GET /p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json HTTP/2\r\n
    Host: repo.packagist.org\r\n
    accept: */*\r\n
    user-agent: Symfony HttpClient/Curl\r\n
    accept-encoding: gzip\r\n
    \r\n
    * Connection died, retrying a fresh connect\n
    * Issue another request to this URL: 'https://repo.packagist.org/p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json'\n
    * Found bundle for host repo.packagist.org: 0x55d33ae22590 [can multiplex]\n
    * Multiplexed connection found!\n
    * Re-using existing connection! (#7) with host repo.packagist.org\n
    * Transfer was pending, now try another\n
    > GET /p/jeka/vlabs-media-bundle$d454d3c809ed3de2580cc7abbe12f29090f314bac46c47b365332870895eaf98.json HTTP/1.1\r\n
    Host: repo.packagist.org\r\n
    Accept: */*\r\n
    User-Agent: Symfony HttpClient/Curl\r\n
    Accept-Encoding: gzip\r\n
    \r\n
    * Mark bundle as not supporting multiuse\n
    < HTTP/1.1 200 OK\r\n
    < Server: nginx\r\n
    < Date: Wed, 18 Nov 2020 23:33:05 GMT\r\n
    < Content-Type: application/json\r\n
    < Content-Length: 547\r\n
    < Last-Modified: Fri, 16 Oct 2020 18:50:05 GMT\r\n
    < Connection: keep-alive\r\n
    < Vary: Accept-Encoding\r\n
    < ETag: "5f89eb5d-223"\r\n
    < Content-Encoding: gzip\r\n
    < \r\n
    """

[Seldaek]

I need to check the mirrors to see if any has the wrong data in the wrong place.. can't do that from my phone will get back to you tomorrow. However the one where v2 metadata ends up in a v1 file is very strange, because afaik those never get downloaded by the same process so it can't be a client issue I'd guess (unless your test script downloads both?). Server is nginx serving static files.. which I can only hope is fairly bug-free.

[jacques]

@Seldaek I manually edited the composer.json file to change the league/csv version after composer did not show the versions and did the composer update --lock.

[stof]

@jacques if you want your update to find a new version satisfying your new requirement, you mist whitelist league/csv in the partial update. composer update --lock is whitelisting nothing.

[Seldaek]

Ok closing this, issues in mirroring script and http-client have been fixed and I verified that all our mirrors now have 100% the same data.