Travis-CI + Github API limit

[fprochazka]

Is there any suggested way to solve the Github API limit problem? Or will my builds just randomly fail?

https://travis-ci.org/Kdyby/Framework/jobs/3142911

Not sure, if this is problem of Composer or Travis-CI.

[Seldaek]

If you rely on VCS repos for your travis test suite, you'd better run install with --no-interaction, that way it'll fallback to a git clone if the api limit is exhausted. For travis pro I guess you could include an oauth token so that it gets a higher limit, but for public travis repos it's kinda tricky to do that.

[fprochazka]

I already did that. Didn't know it would fallback on clone. Thank you!

[amacneil]

Thanks! This was really helpful. Wish it was documented somewhere.

[amacneil]

Wait, this doesn't work with latest composer. Back to square 1. Any suggestions?

[Seldaek]

@adrianmacneil what do you mean doesn't work? And is "latest composer" after running self-update or using the latest tag? Because the last tag is quite old by now.

[amacneil]

I mean --non-interactive isn't a valid option. See this build failure: https://travis-ci.org/adrianmacneil/omnipay/jobs/4744814

I've worked around the issue now by using --prefer-source for Travis, though this seems like a sub-optimal solution.

Forgive my ignorance.. is there is a reason composer doesn't just use the public github zip URLs, rather than the API URLs which cause rate limit issues on Travis?

[stof]

it is --no-interaction (or -n), not --non-interactive

[Seldaek]

Right, I adjusted my comment above to avoid confusing more people :)

[amacneil]

Awesome, thanks guys! I just tested with -n instead of --prefer-source but it seems I'm back under the Github API rate limit so will report back if I run into any issues.

[amacneil]

It seems -n doesn't fall back to a clone like you suggested: https://travis-ci.org/adrianmacneil/omnipay/jobs/4758560

Don't worry. I think I'll just go back to --prefer-source. Hopefully this will help someone else in future.

[Seldaek]

@adrianmacneil eh sorry what you experienced is another issue. -n helps when the parsing of a custom package repository fails due to the github rate limiting, but for downloads indeed the only ways to fix it right now are using --prefer-source, or providing an oauth API token but that's not really easy to do on travis for an open source project.

[covex-nn]

There is a solution of this problem: Encryption keys

• http://about.travis-ci.org/docs/user/encryption-keys/
• http://about.travis-ci.org/docs/user/build-configuration/#Secure-environment-variables

I followed the instructions and created encrypted variable GITHUB_TOKEN

• travis encrypt -r covex-nn/travis-test GITHUB_TOKEN=qwerty_asdf

Then i created .travis.yml and composer.json and pushed them to repository

And here is stable build on travis-ci.org with configured github-oauth.github.com in composer.json after running all scripts:

• https://travis-ci.org/covex-nn/travis-test/builds/6901276

[stof]

It won't work fully. See this quote from their doc:

Please note that secure env variables are not available for pull requests. This is done due to the security risk of exposing such information in submitted code. Everyone can submit a pull request and if an unencrypted variable is available there, it could be easily displayed.

[aertmann]

Had this issue as well when using create-project. Neither adding --no-interaction nor --prefer-source solved the problem, but using git clone and composer install didn't have the issue. No idea why there would be any difference, but seems there is. Maybe it's handled differently internally and those arguments are only used for the root repository? Just a tip if anyone else comes across the same issue.

[bojanz]

http://blog.wyrihaximus.net/2015/09/github-auth-token-on-travis/ documents most of the solution.

What we did:

1. Created the token. No scopes (read only access to public info only).
2. Added it to Travis as the GITHUB_OAUTH_TOKEN env variable.
3. Added git config --global github.accesstoken $GITHUB_OAUTH_TOKEN to .travis.yml (before-install)
   The article advocates doing composer config github-oauth.github.com instead, but that still failed on --prefer-dist for us. Composer bug?
4. Removed --prefer source, kept -n.

[mykz]

@bojanz really nice solution, Thanks for taking time to post it. 👍

[danepowell]

Hmm... so Github was supposed to have fixed this by removing anonymous rate limits:
#4884 (comment)

But I'm still seeing composer installs failing on Travis with errors like this:
"Downloading: Connecting... Failed to download composer/installers from dist: Could not authenticate against github.com"

We never added an OAuth token to composer.json or .travis.yml, so it shouldn't be authenticating. I think this must be due to rate limiting.

[bojanz]

Yeah, I don't think there's a way to avoid rate limiting in Travis, cause it's simply a case of many builds coming from the same IPs. No matter how high the limit is on GitHub, Travis will reach it.

We need to keep using the OAuth tokens.

[danepowell]

Actually it seems to be only intermittently failing. I wonder if some of the Travis IPs are still blacklisted from the old rate limit and it just takes a while to clear them or something...

[hkdobrev]

Travis CI uses a lot of different IPs and it changes them on every build. That's why it cannot provide a list or a range to whitelist in firewalls. I guess GitHub has rate-limited some IP, but the next built uses another one.

[alexislefebvre]

It may not solve all the problems, but I have successfully tested a workaround which consist in caching the vendor/ directory, an archive is created with tar and stored in Travis CI's cache. I presented this method on Stack Overflow. A composer update --prefer-source takes 30 seconds when this cache is used, and 2 minutes without this cache. I haven't checked if it's quicker with --prefer-dist and if the API limit is not reached.

[spekary]

I recently ran into this problem, grunted and groaned over it for a couple of days, and wanted to post a fix for our specific problem as it might help others. The cause of this for us was that we had "repository" entries in our composer.json file pointing to our github source for some new repositories under development. Even though we had registered these with packagist, we were still seeing the problem. It wasn't until we removed these repository entries from the composer.json file so that packagist could be the pointer to them, did we finally see our travis builds start to work consistently.