GitHub API Limitations when Using Composer on Pagoda Box #1861

Closed
sanderson opened this Issue May 3, 2013 · 5 comments

Comments

Projects
None yet
5 participants
@sanderson

Composer, Pagoda Box, & the GitHub API

Composer recently switched from using GitHub's http interface to using the GitHub API to load dependencies (commit 259a25). When using Composer on Pagoda Box, dependencies are loaded each time an app is deployed. With 100s of Pagoda Box users using Composer and deploying often, calls to the GitHub API add up fast.

By default, Composer's requests to the GitHub API are unauthenticated. GitHub limits unauthenticated API requests to 60 per hour per IP. This means that more people using Composer on Pagoda Box (or any other multi-tenant service provider), the faster the request limit is reached and the more users will be unable to download their dependency libraries. Apps will not deploy on Pagoda Box until dependencies can be successfully downloaded.

Pagoda Box reached out GitHub for possible solutions. Below is GitHub's official response:

We really wish Composer would use Git instead of the GitHub API because it's far more efficient for this use case.

Are you supplying a GitHub API token 1 in your config? Here's a guide 2 that might help. Be sure to create your token WITHOUT ANY SCOPES 3 if it will be visible to anyone outside your organization.

Solution

In order for users of any multi-tentant service provider, like Pagoda Box, to be able to successfully and consistently use Composer, Composer will either need to switch to using Git as opposed to GitHub's API, or Composer users will need to include a GitHub API token in their repos.

@stof

This comment has been minimized.

Show comment
Hide comment
@stof

stof May 4, 2013

Contributor

the issue is that github downloads are part of the API and so are subject to the rate limit (which is a really weird choice IMO).
the workaround is to force the installation from source (--prefer-source), which is slower but works.

Contributor

stof commented May 4, 2013

the issue is that github downloads are part of the API and so are subject to the rate limit (which is a really weird choice IMO).
the workaround is to force the installation from source (--prefer-source), which is slower but works.

@stof

This comment has been minimized.

Show comment
Hide comment
@stof

stof May 4, 2013

Contributor

and the commit you linked is about reading the content of the composer.json file, which is never done when you install from a lock file (which should be the case on pagodabox) and rarely done locally (for all packages hosted on Packagist, it is done on the Packagist server)

Contributor

stof commented May 4, 2013

and the commit you linked is about reading the content of the composer.json file, which is never done when you install from a lock file (which should be the case on pagodabox) and rarely done locally (for all packages hosted on Packagist, it is done on the Packagist server)

@cs278

This comment has been minimized.

Show comment
Hide comment
@cs278

cs278 May 9, 2013

Contributor

Actually the change to use the API for download is here 2249348. By using the API URLs Composer is using documented resource and not relying on something that could change at a whim. I don't think it's unreasonable for GitHub to remove/significantly raise the rate limiting on the download endpoints after all they simply redirect to their download infrastructure.

Contributor

cs278 commented May 9, 2013

Actually the change to use the API for download is here 2249348. By using the API URLs Composer is using documented resource and not relying on something that could change at a whim. I don't think it's unreasonable for GitHub to remove/significantly raise the rate limiting on the download endpoints after all they simply redirect to their download infrastructure.

@mfdj

This comment has been minimized.

Show comment
Hide comment
@mfdj

mfdj May 26, 2013

I'm relieved to find this is a known issue, prior to reading I had assumed this issue was due to poor uptime in Github's servers (partially attributed to a coincidental server outage at one point). This issue does make deploying on Pagoda Box a pain but I'm plussed that --prefer-source may smooth it out. Implementing now.

mfdj commented May 26, 2013

I'm relieved to find this is a known issue, prior to reading I had assumed this issue was due to poor uptime in Github's servers (partially attributed to a coincidental server outage at one point). This issue does make deploying on Pagoda Box a pain but I'm plussed that --prefer-source may smooth it out. Implementing now.

paul-m added a commit to paul-m/VirtualPersistAPI that referenced this issue May 24, 2014

@briankiewel briankiewel referenced this issue in briankiewel/pagodabox-laravel-4 Sep 18, 2014

Closed

Changed to prefer dist #20

@jujugrrr jujugrrr referenced this issue in djoos-cookbooks/composer May 29, 2015

Merged

Added --prefer-source #49

nijel added a commit to phpmyadmin/error-reporting-server that referenced this issue Jul 14, 2015

@Seldaek

This comment has been minimized.

Show comment
Hide comment
@Seldaek

Seldaek Feb 27, 2016

Member

Closing as this is a very old issue and not extremely relevant anymore.

Member

Seldaek commented Feb 27, 2016

Closing as this is a very old issue and not extremely relevant anymore.

@Seldaek Seldaek closed this Feb 27, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment