Join GitHub today
x/build: gitmirror or maintnerd consumed all our Gerrit quota #23853
Maintner is blocked:
So now all bots are down.
GerritBot consumed all of our Gerrit quota.
@andybons, please stop GerritBot and/or increase our Gerrit quota.
changed the title
x/build: GerritBot consumed all our Gerrit quota and starved maintner
Feb 15, 2018
Looking at logs in the Cloud Logging console (which captures all the GKE pod output), started at "2018-02-15T13:27:16.527663505Z" , or 5:27am Pacific: "2018-02-15 05:27:14.000 PST"
I'll look at logs just before that to see if it's obvious what went crazy.
From another bug:
We recently upgraded our Kubernetes cluster and our 10 random container jobs re-laid themselves out onto our 4 physical nodes (i.e. 4 egress IP addresses)
We got lucky before and our 3 gerrit-hitting jobs were using different nodes (different IPs).
But after this latest upgrade, all 3 are on the same node, so we burn through that IP's quota by the end of the day.
That's the current theory.
I'm adding auth now.