surf-client hangs up because surf-server is taking too long

[cjb]

In one terminal:

 λ DEBUG=* surf-server keybase/client

In another:

 λ DEBUG=* surf-client -s http://localhost:3000 -r https://github.com/keybase/client -- surf-build
[... after a few minutes ...]
Failed to fetch from http://localhost:3000/info/keybase/client: Error: socket hang up

The surf-server logs show:

  surf:github-api Fetching GitHub URL: https://api.github.com/repos/keybase/client/git/refs?per_page=100 +3ms
...
  surf:github-api Fetching GitHub URL: https://api.github.com/repositories/29099181/git/refs?per_page=100&page=18 +191ms

followed by what looks like hundreds of:

  surf:github-api Fetching GitHub URL: https://api.github.com/repos/keybase/client/git/commits/754a5ed580e9ca877e080a5b55c7c4aef118773b +380ms

at a rate of a few every second.

Does Surf have a problem scaling with large repos? I wonder why it's pulling down so many commits.

[anaisbetts]

Does Surf have a problem scaling with large repos?

Yeah, we have to pull every ref right now, which can take forever long on the first run. After that though, surf-server should be pretty fast. I really wish GitHub would allow you to request commit info more than a single SHA at a time

[cjb]

Unfortunately the client times out and then surf-server starts all over again on the next connection, so it never gets there. :) Gonna delete some refs.

[anaisbetts]

@cjb Our goal is to monitor changes to every ref, but I might have to fall back to monitoring changes to open PRs :(

[cjb]

I actually exceeded the API rate-limit too. Wonder if surf-server should have some inbuilt protection against that.

[cjb]

Oh weird, deleting branches from now-merged pull requests doesn't seem to remove those PRs from the API refs endpoint. So I'm not sure how I could cut down on the number of refs in this project to avoid timing out the client.

[anaisbetts]

We probably need to drop the "Give me all refs" code and replace it with "Give me all open PRs" code and concede trying to build branches that aren't PRs and aren't the default branch (i.e. master)

[cjb]

Alternatively, perhaps you could do this spidering when surf-server starts up, rather than when surf-client first connects to surf-server? The problem is trying to get it all done while surf-client's sitting there waiting for a response, I think?

[anaisbetts]

Alternatively, perhaps you could do this spidering when surf-server starts up, rather than when surf-client first connects to surf-server? The problem is trying to get it all done while surf-client's sitting there waiting for a response, I think?

Yeah, this is a good short-term fix though it won't solve the "I literally have so many refs that I can't read them all without exhausting a token" problem

[anaisbetts]

@cjb I've done some work to make us only poll open PRs, the perf issues on startup should be for the most part completely solved