Proxy requests to https://api.github.com #3426

Closed
danielbachhuber opened this Issue Sep 26, 2016 · 14 comments

Projects

None yet

3 participants

@danielbachhuber
Member

https://api.github.com has a request rate limit (previously #1612):

local ➜  wp-cli git:(master) http HEAD https://api.github.com
HTTP/1.1 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 59
X-RateLimit-Reset: 1474920559

If we proxied requests to https://api.github.com through something like https://github-api.wp-cli.org, then we could:

  1. Have CI run the updater tests on each run.
  2. Open ourselves to a future where the updater isn't dependent on the Github API.

What's the simplest possible implementation of this proxy?

@danielbachhuber danielbachhuber added this to the 0.25.0 milestone Sep 26, 2016
@danielbachhuber danielbachhuber self-assigned this Sep 26, 2016
@afragen
afragen commented Sep 26, 2016

If you add an access token to the API request the limit increases to 5000 per hour.

@danielbachhuber
Member

If you add an access token to the API request the limit increases to 5000 per hour.

Good to know, thanks.

@afragen
afragen commented Sep 26, 2016

It's a simple query arg in the format ?access_token=xxxxxxxxxxxxxxxxxxxxxxxxx

You would have to set it; but simple enough to define as a constant in wp-config.php. Also, the access token needs to only have read public repo privileges. Unless you're dealing with private repos but I assume you would have come across this before. 😉

@danielbachhuber
Member

You would have to set it; but simple enough to define as a constant in wp-config.php.

Sure. I don't think it's going to work in this case though.

@afragen
afragen commented Sep 26, 2016

Hmm, well if you can figure out how to add it, it should solve the timeout issue.

@danielbachhuber danielbachhuber removed this from the 0.25.0 milestone Oct 4, 2016
@danielbachhuber danielbachhuber removed their assignment Oct 4, 2016
@danielbachhuber
Member

I spent several hours last week trying to get this infrastructure to work. Ultimately, I wasn't successful because something in the network stack intermittently returned a 403 error.

For now, I've reverted use of https://github-api.wp-cli.org. I think we could do this in the future, but we need a better plan in place for all of this infrastructure.

@afragen
afragen commented Oct 4, 2016

@danielbachhuber can you point where in the code this is, it like to take a look at it. Not sure I can help, but never know.

@andreiglingeanu
andreiglingeanu commented Oct 4, 2016 edited

Maybe try to use this?

Unyson is happy with that.

http://github-api-cache.unyson.io/

And you query api like normally, while having caching done by this 3rd party package. Just switch the hosts out.

@danielbachhuber
Member

Maybe try to use this?

Oh, that's pretty cool. Thanks for the recommendation, @andreiglingeanu !

I think the 403 was actually coming from CloudFlare, because WebFaction didn't have a 403 in their logs. However, to remove CloudFlare from the DNS stack, I'd need to set up wp-cli.org on WebFaction in order to serve it over SSL. It's a bit more involved of a project than I have time for right now.

@andreiglingeanu

@danielbachhuber Why would you even keep CloudFlare in the game, in this case?

@danielbachhuber
Member

Why would you even keep CloudFlare in the game, in this case?

Free CDN, so I don't have to worry about bandwidth overages.

@andreiglingeanu
andreiglingeanu commented Oct 4, 2016 edited

I think they have some sort of excluding mechanisms, no?

What about forcing CloudFlare to skip caching for api subdomain at all and make it point to a WebFaction server directly that will serve the GitHub API cache?

Seems legit to me.

@danielbachhuber
Member

I think they have some sort of excluding mechanisms, no?

They do, but I was still running into issues with it.

@danielbachhuber
Member

Adding this to the website wish list because it'll need to happen when I figure out a long-term plan for infrastructure maintenance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment