TransportException 404 Not Found for packagist.org urls

[Krinkle]

Filing a separate issue following from #470, given this is not related to SlowTimer.

This still happens on a regular basis in Wikimedia CI pipeline. Is there a way in which we can prevent this issue in a more general way, that hopefully also avoids too much load or storage pressure on your side?

See https://phabricator.wikimedia.org/P7484$1052 for a recent transcript of a build in which this error occurred.

For that build, it seems like the failure was printed only 4 seconds after the composer require command.

01:00:57  DEBUG:quibble.cmd:composer require composer/spdx-licenses=1.4.0 psy/psysh=0.9.6 monolog/monolog=~1.22.1 phpunit/phpunit=4.8.36 \|\| ^6.5 wikimedia/avro=1.8.0 mediawiki/mediawiki-codesniffer=21.0.0 wmde/hamcrest-html-matchers=^0.1.0 seld/jsonlint=1.7.1 wikimedia/testing-access-wrapper=~1.0 jakub-onderka/php-parallel-lint=0.9.2 jetbrains/phpstorm-stubs=dev-master#38ff1a581b297f7901e961b8c923862ea80c3b96 hamcrest/hamcrest-php=^2.0 nmred/kafka-php=0.1.5 justinrainbow/json-schema=~5.2 nikic/php-parser=3.1.3 cache/integration-tests=0.16.0
--
01:01:00
01:01:00    [Composer\Downloader\TransportException]
01:01:00    The "http://packagist.org/p/provider-latest%2489d14de6aab31e05c38c5a490ca286c3895c7713604e767413aeb3ee5f9d0784.json" file could not be downloaded (HTTP/1.1 404 Not Found)
01:01:00
01:01:00  Exception trace:
01:01:00   () at /srv/composer/vendor/composer/composer/src/Composer/Util/RemoteFilesystem.php:430
01:01:00   Composer\Util\RemoteFilesystem->get() at /srv/composer/vendor/composer/composer/src/Composer/Util/RemoteFilesystem.php:101
01:01:00   Composer\Util\RemoteFilesystem->getContents() at 

Also:

• packagist.org causes "SlowTimer" and fails with Composer\Downloader\TransportException  #470
• https://packagist.org/p/provider-active$... file could not be downloaded (404 Not Found) composer#2198
• Unable to install, provider-active$6ae78f236512073ce26e7c01e55b67726a998e665d9b18dcfdcd20d68c226e14.json" file could not be downloaded (HTTP/1.1 404 Not Found) composer#2228
• Composer file could not be downloaded (404 not found) composer#7021
• file could not be downloaded (HTTP/1.1 404 Not Found) composer#7404
• [Composer\Downloader\TransportException] The "http://repo.packagist.org/p/provider-2018-07%245714a0dc840abb2a5da64345abc53df9354b01309b55a8fd83f112e2929fa949.json" file could not be down loaded (HTTP/1.1 404 Not Found)  composer#7599
• Downstream: https://phabricator.wikimedia.org/T182266

[Krinkle]

As of last week this is happening multiple times a day. After having seemingly been almost unseen for months. Looks like a recent regressions perhaps?

I'm also seeing it on Travis CI now, which is a first.

https://travis-ci.org/wikimedia/mediawiki/jobs/422335462

$ composer --version
Composer version 1.7.2 2018-08-16 16:57:12

(154.23s)$ composer install --prefer-source --quiet --no-interaction
                                                                                                                                                                                   
  [Composer\Downloader\TransportException]                                                                                                                                         
  The "http://repo.packagist.org/p/provider-latest%24233d2d1a537144f307feb64eaddbf9a265c5ec874d0ac94f08743a653aec2100.json"
  file could not be downloaded (HTTP/1.1 404 Not Found)

Note that the run-time there is also measured. Travis clocks it at 154s (2 minutes and 34 seconds), which is far less than the previously stated age of "older than 10 minutes" for files that may be removed on the Packagist servers, per #470 (comment).

[denzkiuy]

Hi, I'm also having the same issue. I was trying to perform "composer update" on my laravel project but I encountered the exact same error . I have tried several possible workarounds suggested by different people but nothing worked for me. I hope anyone could help.

After composer update:

[Seldaek]

This should not happen anymore now, it was a race condition between mirrors but I put a fallback in place now so they should go fetch the missing file if a race happens. Please do tell if you see it again.

[cloudways-haider]

Hi, @Seldaek we are still facing the same issue upon running composer require barryvdh/laravel-snappy

[justkristin]

Same issue here.

The "http://packagist.org/p/symfony/process%24ac4ca6b027eeff047d0e37f31208a8d978b98da6c16989614914a639e632ca84.json" file could not be downloaded: failed to open stream: Connection timed out

[belzaaron]

Issue popped up again, overnight. Here's the trace(s) this time:

New Project:

  [Composer\Downloader\TransportException]
  The "https://packagist.org/p/provider-2019-04%244380046a2dfa549bfcf196bfc2c1a5cca9d58c63fb9b8e836367bfe2c01f062b.json" file could not be downloaded (HTTP/1.1 404 Not Found)

Exiting Project:

  [Composer\Downloader\TransportException]                                                                                                                                      
  The "https://packagist.org/p/provider-2018-07%241e5761c3575f5f87c86fa2ca8099d924ad72d7995718b1bfedf2c3fc27241568.json" file could not be downloaded (HTTP/1.1 404 Not Found) 

Happens on any new Laravel project and any call to Composer/Packagist on existing ones. Also may be a different url per-project.

Not a very helpful string of numbers/letters for the url. Also WaybackMachine has no record for this. So I'm stuck here.

@Seldaek letting you know because of #940 (comment)

[phil-davis]

@Seldaek I am also experiencing this a lot today. e.g. https://drone.owncloud.com/owncloud/core/18699 and pick any of the red jobs.
https://drone.owncloud.com/owncloud/core/18699/130

  [Composer\Downloader\TransportException]                                                                                                                                   
The "https://packagist.org/p/provider-2016%24e6f26a3fa2ea1261459b39ffa1f461ca4fb2ae031d46b8ecb204fc2bdb38bd1c.json" file could not be downloaded (HTTP/1.1 404 Not Found)

and the errors have various different provider-yyyy*.json file names.

I am guessing that there is some problem with packagist server(s) somewhere?

Is there a status page or twitter thread or... where real-time status of the packagist servers is posted?
That would be handy - to know if there is already a known problem being worked.

And how are we supposed to report this sort of error? which is probably a real-time packagist mirror/server sysadmin problem not a packagist software bug.

[Seldaek]

Hey, sorry looks like one of the mirror nodes failed to sync properly after I did some updates yesterday.. Should be fixed now.

[phil-davis]

@Seldaek

1. is there somewhere that reports known packagist status? similar to https://www.githubstatus.com/ and https://www.traviscistatus.com/ etc.
2. is there somewhere that we can report when there seems to be an issue?

Because otherwise we are left trying to report via the bug-reporting mechanism on GitHub, and that is not the ideal way to report system-network problems.

[SV-ZeroOne]

Hello, I recently encountered the same following error when trying to composer require seostats/seostats package.
Was first getting this error
[Composer\Downloader\TransportException] The "https://packagist.org/p/provider-2019-04%244380046a2dfa549bfcf196bfc2c1a5cca9d58c63fb9b8e836367bfe2c01f062b.json" file could not be downloaded (HTTP/1.1 404 Not Found)

Then tried a composer clear-cache
Now getting this error
[Composer\Downloader\TransportException] Content-Length mismatch, received 551423 bytes out of the expected 2019004

Can anybody explain why this is happening?

Here is my composer diagnose results
Checking platform settings: OK
Checking git settings: OK
Checking http connectivity to packagist: OK
Checking https connectivity to packagist: OK
Checking github.com rate limit: OK
Checking disk free space: OK
Checking pubkeys:
Checking composer version: OK
Composer version: 1.8.6
PHP version: 7.2.20
PHP binary path: /opt/cpanel/ea-php72/root/usr/bin/php

[Krinkle]

This is happening again, seems to affect Wikimedia CI consistently in each job for the past 20 minutes.

– https://phabricator.wikimedia.org/T233264

[Seldaek]

@Krinkle did it get resolved? I messed something up and we had delays in processing metadata updates last night.

[hashar]

@Seldaek that seems to have disappeared now (our task https://phabricator.wikimedia.org/T233264 ).

I guess we should really look at setting up a packagist mirror for CI purposes.

[Krinkle]

Thanks @Seldaek! All green now.

[Krinkle]

@Seldaek It would appear the issue is back as of a few hours ago. Affects about 1/10 CI builds.

20:10:56   [Composer\Downloader\TransportException]                                                                                                                                   
20:10:56   The "https://packagist.org/p/provider-2017%2462534e44c7215c59b58e5ae6fba06ba61a46d713507c3a58bd130e1940517d96.json" file could not be downloaded (HTTP/1.1 404 Not Found)  

[phil-davis]

I am also seeing this sort of thing:

 [Composer\Downloader\TransportException]                                                                                                                                      
  The "https://packagist.org/p/provider-2019-07%24d56f91aee95bccb2a4018a08708bc0990ba8888f45389df7de412062affd4651.json" file could not be downloaded (HTTP/1.1 404 Not Found)

e.g. https://drone.owncloud.com/owncloud/files_texteditor/452/1/2

I had lots of these errors in overnight jobs.

[mariushoch]

So 51.38.227.34 at least seems to be flaky:

curl -v 'https://packagist.org/p/provider-2017%242361f4e66f731385b3f231a53c107c99231cbeb7496b2194a0eba6aa033fe4df.json' --resolve 'packagist:443:51.38.227.34'

[johny-smith]

Hi, im having the same problem with

composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition .

Downloading https://repo.packagist.org/p/symfony/process%245c5f786d7473c01dfa73e8b7d64e9bf4421871a0b87715596246536ed9fc707c.json

Downloading https://repo.packagist.org/p/symfony/process%245c5f786d7473c01dfa73e8b7d64e9bf4421871a0b87715596246536ed9fc707c.json

                                                                                                        
  [Composer\Downloader\TransportException]                                                              
  The "https://repo.packagist.org/p/symfony/process%245c5f786d7473c01dfa73e8b7d64e9bf4421871a0b8771559  
  6246536ed9fc707c.json" file could not be downloaded (HTTP/1.1 404 Not Found)                          
                                                                                                        

Exception trace:
 () at phar:///usr/bin/composer/src/Composer/Util/RemoteFilesystem.php:440
 Composer\Util\RemoteFilesystem->get() at phar:///usr/bin/composer/src/Composer/Util/RemoteFilesystem.php:104
 Composer\Util\RemoteFilesystem->getContents() at phar:///usr/bin/composer/src/Composer/Repository/ComposerRepository.php:687
 Composer\Repository\ComposerRepository->fetchFile() at phar:///usr/bin/composer/src/Composer/Repository/ComposerRepository.php:358
 Composer\Repository\ComposerRepository->whatProvides() at phar:///usr/bin/composer/src/Composer/DependencyResolver/Pool.php:204
 Composer\DependencyResolver\Pool->computeWhatProvides() at phar:///usr/bin/composer/src/Composer/DependencyResolver/Pool.php:193
 Composer\DependencyResolver\Pool->whatProvides() at phar:///usr/bin/composer/src/Composer/DependencyResolver/RuleSetGenerator.php:164
 Composer\DependencyResolver\RuleSetGenerator->whitelistFromPackage() at phar:///usr/bin/composer/src/Composer/DependencyResolver/RuleSetGenerator.php:304
 Composer\DependencyResolver\RuleSetGenerator->whitelistFromJobs() at phar:///usr/bin/composer/src/Composer/DependencyResolver/RuleSetGenerator.php:355
 Composer\DependencyResolver\RuleSetGenerator->getRulesFor() at phar:///usr/bin/composer/src/Composer/DependencyResolver/Solver.php:217
 Composer\DependencyResolver\Solver->solve() at phar:///usr/bin/composer/src/Composer/Installer.php:475
 Composer\Installer->doInstall() at phar:///usr/bin/composer/src/Composer/Installer.php:229
 Composer\Installer->run() at phar:///usr/bin/composer/src/Composer/Command/CreateProjectCommand.php:195
 Composer\Command\CreateProjectCommand->installProject() at phar:///usr/bin/composer/src/Composer/Command/CreateProjectCommand.php:145
 Composer\Command\CreateProjectCommand->execute() at phar:///usr/bin/composer/vendor/symfony/console/Command/Command.php:245
 Symfony\Component\Console\Command\Command->run() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:835
 Symfony\Component\Console\Application->doRunCommand() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:185
 Symfony\Component\Console\Application->doRun() at phar:///usr/bin/composer/src/Composer/Console/Application.php:267
 Composer\Console\Application->doRun() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:117
 Symfony\Component\Console\Application->run() at phar:///usr/bin/composer/src/Composer/Console/Application.php:106
 Composer\Console\Application->run() at phar:///usr/bin/composer/bin/composer:61
 require() at /usr/bin/composer:24

create-project [-s|--stability STABILITY] [--prefer-source] [--prefer-dist] [--repository REPOSITORY] [--repository-url REPOSITORY-URL] [--dev] [--no-dev] [--no-custom-installers] [--no-scripts] [--no-progress] [--no-secure-http] [--keep-vcs] [--remove-vcs] [--no-install] [--ignore-platform-reqs] [--] [<package>] [<directory>] [<version>]

[rpounder]

I've had this issue for several days now, might be one of the mirrors because it works randomly if I just keep doing it over 30 mins or so;

[Composer\Downloader\TransportException]
The "https://packagist.org/p/provider-2019-07%24925f76d9450d25d63f482102431fca16a8a791a1977d123c0611acb360ff8a59.json" file could not be downloaded (HTTP/1.1 404 Not Found)

[johny-smith]

Is there a way to get around this?

[dzabel]

This mirror works for me: https://packagist.phpcomposer.com

use:
composer config -g repo.packagist composer https://packagist.phpcomposer.com

and/or add to your repository section in composer.json

                { "type": "composer", "url": "https://packagist.phpcomposer.com"},

[johny-smith]

that worked ! thanks a lot!

where did you find that extra mirror?

[bcastagna]

You shouldn't use a mirror you don't know.

[jcheron]

You shouldn't use a mirror you don't know.

@bcastagna It is at least known from packagist: see https://packagist.org/mirrors
And it looks better than a TransportException (404 error).

[slakbal]

This is back today: running with latest version:

  [Composer\Downloader\TransportException]                                                                                                                                                 
  The "https://repo.packagist.org/p/symfony/polyfill-iconv%24b166f62b97f898a535e7affe6ffa17c65e4ba9217812e70d62efa41d9ceb2794.json" file could not be downloaded (HTTP/1.1 404 Not Found)

[checcoux]

[Composer\Downloader\TransportException]
The "http://repo.packagist.org/p/provider-2013%24e141175196c33d87744406e650a7ace8e320d4d8a1033eb1e8ecc93f
70ab190b.json" file could not be downloaded: failed to open stream: HTTP request failed!

using last version

[ThePeterMick]

Same here
[Composer\Downloader\TransportException]
The "https://packagist.org/p/paragonie/random_compat%244a8fbabca55a661e93fb
f2683edec96d140551495786c09184236c31934ed17a.json" file could not be downlo
aded (HTTP/1.1 404 Not Found)

@Seldaek whattup?

[Sidlegionair]

@Seldaek Seemed to be back again today, I also had this last week as @ptrm04 reported. Any idea what is causing this? It's interrupting our services pretty hard (Its functioning now, but we were updating some websites this morning and all 200 of them failed due to this error).

The "https://packagist.org/p/provider-2017%240e110647cae1251cc87614228c5ee5ac1c4d69290931bf25b9633b9cb61b389c.json" file could not be downloaded (HTTP/1.1 404 Not Found)

[phil-davis]

I saw this problem also last night and today. I forgot where to find this closed issue! And I raised composer/composer#9112

Is it possible to have somewhere that we can report packagist "run-time-production" issues that are not code bugs?

[Seldaek]

Can you please do a ping packagist.org or something and let me know which IP it resolves to for you (from the machine experiencing the issue)?

[Sidlegionair]

Hi @Seldaek,

Here you go:

[xxxx@xxxx xxxx]$ ping packagist.org
PING packagist.org (54.37.2.184) 56(84) bytes of data.
64 bytes from ip-54-37-2.eu (54.37.2.184): icmp_seq=1 ttl=49 time=11.1 ms
64 bytes from ip-54-37-2.eu (54.37.2.184): icmp_seq=2 ttl=49 time=11.3 ms

[phil-davis]

It's a bit tricky. The CI systems are a number of drone agents in a data center. They start as they are required, no chance to "login" to them and do ordinary commands.

Locally at my desk I am seeing both 139.99.120.115 and 51.79.162.48 - I did composer clearcache, cleaned my repo and started composer install and it is running OK.

I don't know how to reproduce it, because sometimes today I got success on a whole set of CI pipelines, and other times there were up to a dozen pipelines that failed like this.

[Seldaek]

Ok found the problem I think, hopefully you can confirm it is back to normal now.

[Sidlegionair]

Hi @Seldaek,

I am still experiencing these issues

`Installation failed, reverting ./composer.json to its original content.

[Composer\Downloader\TransportException]
The "https://packagist.org/p/provider-2016%24213e60dbd2e7f903a68123906dc41d08e0edfbf13734b00ce024150030b2be5f.json" file could not be downloaded (HTTP/1.1 404 Not Found)

` (this error was ~2 min ago)

Ping output:
[xxxx@xxxx ~]$ ping packagist.org PING packagist.org (51.77.109.217) 56(84) bytes of data. 64 bytes from 51.77.109.217 (51.77.109.217): icmp_seq=1 ttl=48 time=11.0 ms 64 bytes from 51.77.109.217 (51.77.109.217): icmp_seq=2 ttl=48 time=11.1 ms ^C

[Seldaek]

Ah yeah sorry a couple more nodes were having the same issue, I rejoiced too quickly, checked all of them now..

[Sidlegionair]

@Seldaek Seems resolved, thanks!

[dilshandesh]

@Seldaek

[Seldaek]

@dilshandesh that seems unrelated, please don't pile up random stuff in this old issue, rather open a new one if you have an issue you can reliably reproduce.

[dilshandesh]

@dilshandesh that seems unrelated, please don't pile up random stuff in this old issue, rather open a new one if you have an issue you can reliably reproduce.

This one also the same issue which people have faced up there. and I solved it. Thanks!

[Sidlegionair]

@dilshandesh The issue reported here was HTTP status code 404. Yours is reporting HTTP status code 500 with a different message.

[ThePeterMick]

Wonder how many production workloads this broke / is breaking that are based on the public packagist - private packagist to the rescue? Or are there issues too?

[ThePeterMick]

Still ongoing... @Seldaek can you please help

[Composer\Downloader\TransportException]                                     
  The "https://packagist.org/p/symfony/polyfill-php56%2461f926ceed20b48df6e5c  
  02f19b8874cf3b3bebf6fbf202bac3d52279f33d36d.json" file could not be downloa  
  ded (HTTP/2 404 )

ping packagist.org
PING packagist.org (54.37.131.18): 56 data bytes
64 bytes from 54.37.131.18: icmp_seq=0 ttl=44 time=57.359 ms
64 bytes from 54.37.131.18: icmp_seq=1 ttl=44 time=54.724 ms
64 bytes from 54.37.131.18: icmp_seq=2 ttl=44 time=53.596 ms
64 bytes from 54.37.131.18: icmp_seq=3 ttl=44 time=53.011 ms
64 bytes from 54.37.131.18: icmp_seq=4 ttl=44 time=52.556 ms
64 bytes from 54.37.131.18: icmp_seq=5 ttl=44 time=52.534 ms

[ThePeterMick]

Moved on from 404 to 302 now:

  [Composer\Downloader\TransportException]                                     
  The "http://repo.packagist.org/p/symfony/polyfill-php56%246256af99f55d49470  
  e29994e51758c5b472305e322b30b72e26b6956ecdb7c26.json" file could not be dow  
  nloaded (HTTP/1.1 302 Found)

[Seldaek]

@ptrm04 maybe try using composer 2 preview with composer self-update --preview. It might help with network trouble too.

[Sidlegionair]

@ptrm04 You can also run your own mirror if packagist.org is too unstable, see https://github.com/Webysther/packagist-mirror

[ThePeterMick]

Thanks guys. Also, why on Earth would it want to install "polyfill-php56" - I've searched the entire solution (including vendors) but "polyfill-php56" was not found anywhere. Symfony 5.1 and composer 1.10.10.

Are any of you getting those? I want to work on a project but heck can't get it to run due to all these errors.

[ossycodes]

@dilshandesh that seems unrelated, please don't pile up random stuff in this old issue, rather open a new one if you have an issue you can reliably reproduce.

This one also the same issue which people have faced up there. and I solved it. Thanks!

Please how did you fix?

[benelori]

Hello!

We are starting to experience this 404 issue in our pipelines, when trying to install phpunit 8.5 via simple-phpunit.

After a few retries it works, and I'm wondering if not finding the provider json file is due to it expiring sooner than expected.
We are aware that our MS agents are slow, but we are getting 2 minutes install time, when everything is ok, so I'm not sure.

[lulco]

Hello, today we also see this error in our pipelines. And it also works after few retries.

[cslevy]

I encounter the same issue. This is something related to the packagist servers/load balancers.

For ex if you try to access a package json directly from the browser (ex: https://repo.packagist.org/p/drupal/libraries.json). If you hit a couple of times Ctrl+F5 (or Cmd+F5) force reload sometimes you get the json sometimes you get 404.

[Seldaek]

@cslevy this is an issue with v1 metadata not handling deletions very cleanly. Composer 2 correctly does not list that package anymore.

[bizmate]

To anyone using github, while using PATs and stepping into this problem for instance during a composer require , ensure your PAT has full repo rights so that metadata can be read or you might still experience a 404 instead of a 401

[NiMeDia]

[Composer\Downloader\TransportException]
The "https://packagist.org/p/symfony/polyfill-php72%2483c53d509f08db3f1f98a9cb422a7b0515c169cd5691f54b554c5dac0e96248f.json" file could not be downloaded (HTTP/1.1 404 Not Found)

Currently happening again (indirectly on Satis 1.0.0).