Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why is Composer Install Failing All of the Sudden? #8710

Closed
cgunnels opened this issue Mar 25, 2020 · 34 comments
Closed

Why is Composer Install Failing All of the Sudden? #8710

cgunnels opened this issue Mar 25, 2020 · 34 comments

Comments

@cgunnels
Copy link

@cgunnels cgunnels commented Mar 25, 2020

I'm running composer install in my local env and in production env and they are both failing. This just started today. I've reviewed the code changes and the composer files haven't changed. Can anyone shed some light on this...i have no idea what it could be. I thought github was down or something but I do get some packages installed successfully. I'm getting errors like this:

...

Installing phpunit/php-timer (2.1.2): Downloading (100%)
Installing phpunit/php-text-template (1.2.1): Downloading (100%)
Installing phpunit/php-file-iterator (2.0.2): Downloading (0%) Failed to download phpunit/php-file-iterator from dist: Could not authenticate against github.com Now trying to download from source
Installing phpunit/php-file-iterator (2.0.2): Cloning 050bedf145 from cache
Installing theseer/tokenizer (1.1.3): Downloading (0%) Failed to download theseer/tokenizer from dist: Could not authenticate against github.com Now trying to download from source
Installing theseer/tokenizer (1.1.3): Cloning 11336f6f84 from cache
Installing sebastian/code-unit-reverse-lookup (1.0.1): Downloading (0%) Failed to download sebastian/code-unit-reverse-lookup from dist: Could not authenticate against github.com Now trying to download from source
Installing sebastian/code-unit-reverse-lookup (1.0.1): Cloning 4419fcdb5e from cache
Installing phpunit/php-code-coverage (6.1.4): Downloading (0%) Failed to download phpunit/php-code-coverage from dist: Could not authenticate against github.com Now trying to download from source
Installing phpunit/php-code-coverage (6.1.4): Cloning 807e6013b0 from cache
Installing doctrine/instantiator (1.3.0): Downloading (0%) Failed to download doctrine/instantiator from dist: Could not authenticate against github.com Now trying to download from source
Installing doctrine/instantiator (1.3.0): Cloning ae466f7262 from cache
Installing phpspec/prophecy (v1.10.2): Downloading (0%) Failed to download phpspec/prophecy from dist: Could not authenticate against github.com Now trying to download from source
Installing phpspec/prophecy (v1.10.2): Cloning b4400efc9d from cache
Installing phar-io/version (2.0.1): Downloading (0%) Failed to download phar-io/version from dist: Could not authenticate against github.com Now trying to download from source
Installing phar-io/version (2.0.1): Cloning 45a2ec53a7 from cache
Installing phar-io/manifest (1.0.3): Downloading (0%) Failed to download phar-io/manifest from dist: Could not authenticate against github.com Now trying to download from source
Installing phar-io/manifest (1.0.3): Cloning 7761fcacf0 from cache
Installing myclabs/deep-copy (1.9.5): Downloading (0%) Failed to download myclabs/deep-copy from dist: Could not authenticate against github.com Now trying to download from source
Installing myclabs/deep-copy (1.9.5): Cloning b2c28789e8 from cache
Installing phpunit/phpunit (7.5.20): Downloading (0%) Failed to download phpunit/phpunit from dist: Could not authenticate against github.com Now trying to download from source
Installing phpunit/phpunit (7.5.20): Cloning 9467db479d
[Symfony\Component\Process\Exception\ProcessTimedOutException]
The process "git clone --no-checkout 'https://github.com/sebastianbergmann/phpunit.git' '/var/www/vendor/phpunit/phpunit' && cd '/var/www/vendor/phpunit/phpunit' && git remote add composer 'https://github.com/sebastianbergmann/phpunit.git' && git fetch composer && git remote set-url origin 'https://github.com/sebastianbergmann/phpunit.git' && git remote set-url composer 'https://github.com/sebastianbergmann/phpunit.git'" exceeded the timeout of 300 seconds. > install [--prefer-source] [--prefer-dist] [--dry-run] [--dev] [--no-dev] [--no-custom-installers] [--no-autoloader] [--no-scripts] [--no-progress] [--no-suggest] [-v|vv|vvv|--verbose] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--ignore-platform-reqs] [--] []...

ERROR: Service 'contianer_name' failed to build: The command '/bin/sh -c composer install && rm $(which composer)' returned a non-zero code: 1

@rmilesson

This comment has been minimized.

Copy link

@rmilesson rmilesson commented Mar 25, 2020

We are facing the same issue. Running composer install in Bitbucket Pipelines. Problems started 16 hours ago for us when we pushed a build to staging environment.

Locally it seems to work though. rm -r vendor/ && rm composer.lock && composer install --no-cache

@dmetzner

This comment has been minimized.

Copy link

@dmetzner dmetzner commented Mar 25, 2020

We are having the same problems using GithubActions. ~70% of the builds are failing.

@rmilesson

This comment has been minimized.

Copy link

@rmilesson rmilesson commented Mar 25, 2020

@dmetzner Are you having problems locally as well or is it just in CI env?

@dmetzner

This comment has been minimized.

Copy link

@dmetzner dmetzner commented Mar 25, 2020

@rmilesson
Locally it seems to work though. rm -r vendor/ && rm composer.lock && composer install --no-cache

@ashbaker1

This comment has been minimized.

Copy link

@ashbaker1 ashbaker1 commented Mar 25, 2020

Similar issue. No idea on the cause. It seems to fail on random packages on every run. Occasionally all will install (and I think that it is solved, but on reruns it fails). Only happens on Jenkins - installing locally works fine. Always a 401 error. Tried with composer config --global github-protocols https as the issue seemed to be git related but this only worked once (maybe just was luck on that run...)

@danpowley-hee

This comment has been minimized.

Copy link

@danpowley-hee danpowley-hee commented Mar 25, 2020

We're having the same issue, we're using Azure Pipelines, example error:

- Installing phpunit/php-code-coverage (6.1.4): Downloading    Failed to download phpunit/php-code-coverage from dist: Could not authenticate against github.com

Our current workaround is to configure composer to use an OAuth token for github, I believe this prevents rate limiting by the github api.

We added this to our Dockerfile to achieve this:

# pass OAuth token as a docker build argument
ARG GITHUB_OAUTH
# configure token
RUN composer config -g github-oauth.github.com $GITHUB_OAUTH
# list config to check that its working
RUN composer config --list

As with others, we were also having this happen intermittently, and always a different dependency would be the one that fails. So, I've gone and back and forth twice to check that this workaround is working, its working so far...

I can't offer any further insight, hopefully someone else will be able to resolve/explain this fully.

@stof

This comment has been minimized.

Copy link
Contributor

@stof stof commented Mar 25, 2020

Was there a change in the composer version being used by your pipeline at the time it started failing ?

@rmilesson

This comment has been minimized.

Copy link

@rmilesson rmilesson commented Mar 25, 2020

@stof We're running curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer to install composer. This does not designate a version and I assume it's installing the latest version. Latest release was 12 days ago so I think it's unlikely unless the installer has been compromised.

@stof

This comment has been minimized.

Copy link
Contributor

@stof stof commented Mar 25, 2020

Then maybe there has been a change on the github side.

@danpowley-hee

This comment has been minimized.

Copy link

@danpowley-hee danpowley-hee commented Mar 25, 2020

@stof We use the official composer dockerhub image composer:1.9, this has been updated recently, but we were definitely on 1.9 throughout, only minor version update.

@danpowley-hee

This comment has been minimized.

Copy link

@danpowley-hee danpowley-hee commented Mar 25, 2020

I get this error also running composer install locally now, not just in Azure Pipelines. Configuring github OAuth token is the only workaround I have.

@KajdeMunter

This comment has been minimized.

Copy link

@KajdeMunter KajdeMunter commented Mar 25, 2020

Definitely looks like a rate limit issue on GitHubs side, I can't seem to find any changes on their documentation however.

Ive also gotten the following:

- Installing phpunit/phpunit: API limit (0 calls/hr) is exhausted, could not fetch https://api.github.com/repos/sebastianbergmann/...... Create a GitHub OAuth token to go over the API rate limit. You can also wait until ? for the rate limit to reset.
Head to https://github.com/settings/tokens/new?scopes=repo&description=Composer
to retrieve a token. It will be stored in "/root/.composer/auth.json" for future use by Composer.

Adding the OAuth token seems to work, but this is not exactly desirable.

@aaronbushnell

This comment has been minimized.

Copy link

@aaronbushnell aaronbushnell commented Mar 25, 2020

Having the same issue and I sent a support message to GitHub's team to see if they have any info on what might have changed. I hope this is an oversight on their end and they can correct this so we don't need to add tokens to every project that runs composer install in a CI environment.

We're even seeing this issue when a server runs composer install before pulling down the site files (outside of a Jenkins/Travis/Buddy CI workflow)

Here's the most recent failure we've encountered:

Executing command: composer install
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Package operations: 113 installs, 0 updates, 0 removals
  - Installing yiisoft/yii2-composer (2.0.8): Downloading (connecting...)Downloading (0%)           
                                             
  [Composer\Downloader\TransportException]   
  Could not authenticate against github.com  
                                             

install [--prefer-source] [--prefer-dist] [--dry-run] [--dev] [--no-dev] [--no-custom-installers] [--no-autoloader] [--no-scripts] [--no-progress] [--no-suggest] [-v|vv|vvv|--verbose] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--ignore-platform-reqs] [--] [<packages>]...

Build finished successfully!.
@siebird

This comment has been minimized.

Copy link

@siebird siebird commented Mar 25, 2020

FWIW, this happened to me yesterday on a deployment. I had to add a personal access token for one dependency that was requiring it.

Try running composer install directly on the server to see if it prompts for that? 🤷‍♂️

@sghoweri

This comment has been minimized.

Copy link

@sghoweri sghoweri commented Mar 25, 2020

We're running into this exact same issue as well... 🤦‍♂

@ChexWarrior

This comment has been minimized.

Copy link

@ChexWarrior ChexWarrior commented Mar 25, 2020

Also experiencing this issue in Bitbucket Pipelines...

@mpskovvang

This comment has been minimized.

Copy link

@mpskovvang mpskovvang commented Mar 25, 2020

We're facing the exact same issue.

Using the COMPOSER_AUTH env doesn't solve the issue despite the following message from the output:

Executing command (CWD): git config github.accesstoken

However, it seems like setting the GitHub token directly in the composer.json works:

"config": {
    "github-oauth": {
        "github.com": "XXXXXXXXXXXXXXXXXXXXXX"
    }
},

Using GitHub token authentication

158 packages installed without any timeouts or errors.

Using RUN composer config -g github-oauth.github.com $GITHUB_OAUTH inside Dockerfile also works as suggested by @danpowley-hee #8710 (comment)

@mtnakahara

This comment has been minimized.

Copy link

@mtnakahara mtnakahara commented Mar 25, 2020

Not sure if it helps anyone, in my case I used --prefer-source and some other options and it worked fine. Probably doesn't work for everyone since I am also using --no-dev

php composer.phar install --prefer-source --no-interaction --no-dev -o
@danepowell

This comment has been minimized.

Copy link
Contributor

@danepowell danepowell commented Mar 26, 2020

I recall (could be wrong) that this has happened a couple times in the past. At least once, Github admitted to it being a mistaken/inadvertent configuration change on their part, and they reverted it after being contacted. Unfortunately I can't find any documentation on that incident at the moment.

@JumpIfBelow

This comment has been minimized.

Copy link

@JumpIfBelow JumpIfBelow commented Mar 26, 2020

Hi everyone,

I was struggling yesterday to understand why some of our CI were failing due to the really same issue. I wasn't sure if this was due to a new way of Composer was running the authentication since the 1.10 version, but the fact that I did had the same issue with 1.9 suggests me not.

Moreover, the limit rate was clearly questionable and looked bugeous:

GitHub API limit (0 calls/hr) is exhausted, could not fetch https://api.github.com/repos/Ocramius/PackageVersions/zipball/44af6f3a2e2e04f2af46bcb302ad9600cba41c7d. Create a GitHub OAuth token to go over the API rate limit. You can also wait until ? for the rate limit to reset.

The "workaround" was to use --prefer-source as a option with composer install.
It is really not optimal at all (slow, breaks the behavior of symfony/flex) but at least it was working after rebuilding all the scripts and Docker images.

As for now, it seems like everything is back to normal, it definitely looks like it was an issue with GitHub and not Composer.

@stereoshoots

This comment has been minimized.

Copy link

@stereoshoots stereoshoots commented Mar 26, 2020

Still having problems with it.

Installing 'unzip' may remediate them.
  - Installing ocramius/package-versions (1.4.0): 
Downloading (connecting...) Downloading (0%)
Failed to download ocramius/package-versions from dist: Could not authenticate against github.com

Any new information?

@jozefkoscak

This comment has been minimized.

Copy link

@jozefkoscak jozefkoscak commented Mar 26, 2020

We're having the same issue.
A little explanation brings info from:
https://stackoverflow.com/questions/35496848/composer-could-not-fetch-github

curl -i https://api.github.com/users/octocat

HTTP/1.1 403 rate limit exceeded
Date: Thu, 26 Mar 2020 11:15:05 GMT
Server: Varnish
Content-Type: application/json
X-Ratelimit-Limit: 60
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 1585221449
Content-Length: 247
X-GitHub-Request-Id: DD9A:12279:1CE304B:22372E9:5E7C8EB9

{"message":"API rate limit exceeded for IP. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://developer.github.com/v3/#rate-limiting"}
@dbussink

This comment has been minimized.

Copy link

@dbussink dbussink commented Mar 26, 2020

👋 everyone here. This is @dbussink from GitHub and I'd like to let you all know that this issue is now resolved. In our efforts to continuously protect GitHub.com, we accidentally deployed stricter rate limits to the endpoints used here. We're sorry about that and the behavior has been fixed.

@wimleers

This comment has been minimized.

Copy link

@wimleers wimleers commented Mar 26, 2020

@dbussink Did GitHub take measures to prevent this regression from being reintroduced? 🤞

@rmilesson

This comment has been minimized.

Copy link

@rmilesson rmilesson commented Mar 26, 2020

@dbussink Thanks for your quick response!

@njovin

This comment has been minimized.

Copy link

@njovin njovin commented Mar 26, 2020

@dbussink Thanks for confirming this. When was the issue identified and why isn't this issue listed as an incident on the Github status page?

A lot of teams burned a lot of time over the last few days dealing with this. A simple status entry on the Github status site would have saved mountains of wasted time.

@Seldaek Seldaek closed this Mar 26, 2020
@spencersmith

This comment has been minimized.

Copy link

@spencersmith spencersmith commented Mar 26, 2020

@njovin I agree. My team spent a few hours late last night trying to figure this out. Fortunately I was able to find this thread, but it would have been nice if it was on the GH status page.

@richRemer

This comment has been minimized.

Copy link

@richRemer richRemer commented Mar 26, 2020

@dbussink We're still seeing this with Travis builds, at least intermittently. Was the fix to the rate limits expected to impact Travis builds, too, or are Travis builds expected to be caught up in the new API limits?

@dmetzner

This comment has been minimized.

Copy link

@dmetzner dmetzner commented Mar 27, 2020

@dbussink It still affects GitHub Actions once in a while too.
~1 out of 50 builds fails because of the API limit.

@njovin

This comment has been minimized.

Copy link

@njovin njovin commented Mar 30, 2020

@dbussink Can we get an update on this? Still nothing on the Github status page and I can't get a response from Github support.

@cgunnels

This comment has been minimized.

Copy link
Author

@cgunnels cgunnels commented Mar 30, 2020

@njovin did you try adding your github oauth token to composer.json/composer config? https://stackoverflow.com/a/60844840/584757

@cgunnels

This comment has been minimized.

Copy link
Author

@cgunnels cgunnels commented Mar 30, 2020

I ended up removing composer from the production deployment and added vendor folder to my vcs. Not happy that I had to do this, but it speeds up deployment and removes these types of issues.

@dbussink

This comment has been minimized.

Copy link

@dbussink dbussink commented Mar 31, 2020

@dbussink Thanks for confirming this. When was the issue identified and why isn't this issue listed as an incident on the Github status page?

The issue was identified based on reports to our support team and subsequently resolved within a few hours after we heard about it.

I understand how having this on the status page would have helped here, but today our status systems are not set up for the granularity needed to be able to identify very specific impacts to specific systems like Composer. Thank you for this feedback.

@dbussink Did GitHub take measures to prevent this regression from being reintroduced? 🤞

Yes, we added tests to our integration suites to cover this.

Anyone who is still seeing rate limits are likely hit by the generic rate limit we have in place for anonymous requests to other URLs separate from the zipball or tarball URLs. Those rate limits have not changed and are still in place. That is separate from the issue that we identified and fixed last week.

For anyone who is still having problems, the best suggestion I have is to open an issue with support with all the specific details you can provide. We'll see if there's anything that can be done there. The solution might be though to add authentication to the requests (so you don't get hit by the anonymous public rate limit of 60 requests / hour from a single IP, see https://developer.github.com/v3/#rate-limiting). Another solution as mentioned above is to vendor libraries instead.

@nick-studiolabs

This comment has been minimized.

Copy link

@nick-studiolabs nick-studiolabs commented Mar 31, 2020

@dbussink In my situation my composer install is making over 60 requests per hour, so I understand why I continue to see this error. However, I am wondering why I had never experienced this issue before last Wednesday. I didn't change anything in my composer.json, and it failed, and continues to fail due to authentication (when I remove the token).

UPDATE
I just tried to run another composer install without the token, and now it works. It seems almost random. My composer.lock references more than 60 packages downloaded from Github. So I am wondering why I am not hitting this rate limit now. Even though it was fixed last week, and I still got the error yesterday.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.