brew search rapidly decreases remaining rate limit on Github API

[awendt]

I'm aware of #21023, but I'd still like to bring your attention to this:

$ curl https://api.github.com/rate_limit
{
  "rate": {
    "limit": 60,
    "remaining": 40,
    "reset": 1376556079
  }
}
$ brew search git
bagit      git-cola       git-flow-avh   git-now      git-url-sub        magit
easy-git     git-crypt        git-flow-clone   git-sh       giter8         stgit
geogit       git-encrypt        git-ftp    git-ssh      gitslave         topgit
git      git-extras       git-gerrit   git-tf       legit
git-archive-all    git-flow       git-multipush  git-tracker      libgit2
$ curl https://api.github.com/rate_limit
{
  "rate": {
    "limit": 60,
    "remaining": 30,
    "reset": 1376556079
  }
}
$

This means that each brew search git decreases the number of remaining API hits by 10 (!). Even though I can set HOMEBREW_GITHUB_API_TOKEN, I still find it unacceptable that homebrew, by default:

1. uses 10 API hits for this search
2. doesn't use the search API which has a custom rate limit of 5rpm instead of 60rph.
3. doesn't use HTTP caching as suggested by Github

I'd like to improve on this, but I don't know where to start. E.g., where can I find the search code?

[samueljohn]

Valid points.

You find the relevant code https://github.com/mxcl/homebrew/blob/master/Library/Homebrew/cmd/search.rb#L72-L96 and https://github.com/mxcl/homebrew/blob/master/Library/Homebrew/utils.rb#L241-L297.

We are happy to get a pull request improving this. I am not sure if the search API has the needed features for our search. At least I can remember that back then there was something missing, so we had to use the old API. But please have a try, perhaps we can get a better search now.

[awendt]

Thanks for the pointers, I'll definitely have a try.

[jakob-stoeck]

As far as I can see the current implementation is the only possibility with the Github API v3.

We could however cache the etag or last modified date of each tap locally in a file and use conditional requests which don’t count in the rate limit, so subsequent searches should be a little faster and w/o limits.

Any thoughts on this?

[jacknagel]

I think that would be a step in the right direction.

[jakob-stoeck]

Should this be in an external dependency or do you want it directly inside the GitHub class? I would have put it in a dependency but it seems that the homebrew code is pretty much self-contained.

[jacknagel]

It needs to be part of the existing code; the core of Homebrew can't have any external dependencies.

[jacknagel]

The situation has improved since this issue was originally opened (we are using the updated search API). Further improvements are definitely welcome, but I'm going to close this.