Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/maintner: poll Gerrit more often than 15 minutes #21192

Open
bradfitz opened this issue Jul 27, 2017 · 7 comments

Comments

@bradfitz
Copy link
Member

commented Jul 27, 2017

If Gerrit emails are delayed (there's no SLA?), maintner (our Gerrit+GitHub mirror) doesn't get updated until the next Gerrit poll every 15 minutes.

We should probably do a cheap poll more often than 15 minutes.

I just saw an email delay of 3-5 minutes.

@andybons, what's a cheap (for Gerrit, and for our quota) way to ask Gerrit what's changed recently, including comments & votes?

A full git ls-remote on each project is pretty expensive, both in network & since we need to do it for each project.

A query over all changes/projects with a Limit of 1 seems cheap network-wise, but not sure what the quota is for that. Can we do it every minute?

@gopherbot gopherbot added this to the Unreleased milestone Jul 27, 2017

@gopherbot gopherbot added the Builders label Jul 27, 2017

@andybons andybons self-assigned this Aug 21, 2017

@andybons

This comment has been minimized.

Copy link
Member

commented Aug 21, 2017

The query over all changes/projects with a Limit of 1 is the best way forward, but unauthenticated requests to Gerrit are capped at 0.2 qps over a day, while authenticated requests are at 1 qps over a day. So, if unauthenticated, we could have a max of 12 clients hitting Gerrit at 1 query per minute. I think that would be fine for now, but it’d get bad as more things are moved/built using maintner.

Might as well look into authenticated requests, then we can reduce the poll interval.

@dborowitz

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2017

As I told Andy privately, I strongly recommend setting up authentication. If you run over your quota, we can increase it, but only on a per-user basis. So you may find life easier if you're not scrambling to go from anonymous to authenticated while in the middle of an outage.

I'm slightly curious if you reduce the polling interval to 1 minute, how many intervals you will actually observe no updates to any changes. It might just be worth doing a "full" poll every 1 minute, not a "quick" poll to see if anything's changed, followed by a 100% chance of a full poll.

@andybons

This comment has been minimized.

Copy link
Member

commented Aug 21, 2017

We can do this rather easily with our instances using https://gerrit.googlesource.com/gcompute-tools/. Will start by tackling that.

@gopherbot

This comment has been minimized.

Copy link

commented Aug 21, 2017

Change https://golang.org/cl/57710 mentions this issue: maintner: decrease poll interval from 15 to 5 min

gopherbot pushed a commit to golang/build that referenced this issue Aug 21, 2017
maintner: decrease poll interval from 15 to 5 min
When Gerrit emails are not sent, maintner polls Gerrit every
15 minutes. We can safely reduce this to every 5 min given the
relatively small number of bots using maintner and the current
quota of 0.2 qps per day that Gerrit permits for unauthenticated
requests.

Update golang/go#21192

Change-Id: I233bc50998927daedb34d81436e182579d2b4031
Reviewed-on: https://go-review.googlesource.com/57710
Reviewed-by: Sarah Adams <shadams@google.com>
@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Aug 21, 2017

I would not use gcompute-tools. Many of our Docker containers only have a single process and don't even contain a filesystem. Just use the GCE metadata service like we do already. In fact, Gerrit auth credentials are already in there and used by some Gerrit client usages. We just need to use it in more places.

@andybons

This comment has been minimized.

Copy link
Member

commented Aug 22, 2017

I don’t see Gerrit auth credentials in the instance metadata. It is named something not obvious?

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Aug 22, 2017

Not on corp so I can't check. But gopherbot posts to Gerrit, so gopherbot is doing getting Gerrit auth somehow.

One of these three: https://github.com/golang/build/blob/158d74e/cmd/gopherbot/gopherbot.go#L76

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.