Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: builders and github push not working for new CLs #21849

Closed
martisch opened this issue Sep 12, 2017 · 13 comments

Comments

Projects
None yet
6 participants
@martisch
Copy link
Member

commented Sep 12, 2017

http://build.golang.org does not show builder runs for the last few CLs that were submitted (~24 hours)
and the github repo does not seem to have catched up on the last few commits that were submitted.

@martisch martisch changed the title builders and github push not updating builders and github push not working for new CLs Sep 12, 2017

@kevinburke

This comment has been minimized.

Copy link
Contributor

commented Sep 12, 2017

Last time this happened there was a stack trace in the maintner logs (which I think is powering a lot of the data sources here) but I don't have access to the production logs for that service. @adams-sarah ?

@adams-sarah

This comment has been minimized.

Copy link
Contributor

commented Sep 12, 2017

+andy

Probably pubsubhelper:

2017/09/12 18:26:17 Client error: read error: EOF
2017/09/12 18:26:18 Client: "AUTH LOGIN\r\n", verhb: "AUTH"
2017/09/12 18:26:18 Client error: read error: EOF
2017/09/12 18:26:19 Client: "AUTH LOGIN\r\n", verhb: "AUTH"
2017/09/12 18:26:19 Client error: read error: EOF
2017/09/12 18:26:19 Client: "AUTH LOGIN\r\n", verhb: "AUTH"
2017/09/12 18:26:19 Client error: read error: EOF
2017/09/12 18:26:20 Client: "AUTH LOGIN\r\n", verhb: "AUTH"
2017/09/12 18:26:20 Client error: read error: EOF
2017/09/12 18:28:15 Client: "AUTH LOGIN\r\n", verhb: "AUTH"

@andybons

This comment has been minimized.

Copy link
Member

commented Sep 12, 2017

Hm. https://pubsubhelper.golang.org/recent is indicating that new GitHub events are being processed, but not new Gerrit events...

@bradfitz

This comment has been minimized.

Copy link
Member

commented Sep 12, 2017

Is the "AUTH LOGIN" coming from Google's MTA or just random spammers from the Internet finding a port 25 open? If the latter, it's a red herring.

But if Google has changed its delivery to now attempt AUTH, that's weird.

FWIW, our SMTP greeting is:

$ telnet pubsubhelper.golang.org 25
Trying 35.184.237.80...
Connected to pubsubhelper.golang.org.
Escape character is '^]'.
220  ESMTP gosmtpd
^]q

Maybe it needs to say something more or less to make clients not try to authenticate?

Or maybe we should just reply to any AUTH request with a success message.

But first I'd like to see if that's actually Google authenticating.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Sep 12, 2017

(Btw, I'm not debugging this. I still only have few minute chunks of time sporadically.)

gopherbot pushed a commit to golang/build that referenced this issue Sep 12, 2017

cmd/pubsubhelper: log new smtp connections with addr
Update golang/go#21849
Change-Id: I0c41aee4cd2f477a8b046dff75f358543b67d4c2
Reviewed-on: https://go-review.googlesource.com/63350
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Sep 12, 2017

Change https://golang.org/cl/63350 mentions this issue: cmd/pubsubhelper: log new smtp connections with addr

@rakyll rakyll changed the title builders and github push not working for new CLs x/build: builders and github push not working for new CLs Sep 12, 2017

@gopherbot gopherbot added this to the Unreleased milestone Sep 12, 2017

@adams-sarah

This comment has been minimized.

Copy link
Contributor

commented Sep 12, 2017

subrepo commits are appearing in datastore cache, but not go main repo commits since 2d69e9e yesterday

datastore

@andybons

This comment has been minimized.

Copy link
Member

commented Sep 12, 2017

Found the issue. The bot that is set to receive the emails had all of its watches deleted from its Gerrit account. Re-added a watch on All-Projects and will follow up with the Gerrit team. Let’s see if this fixes the underlying problem.

@adams-sarah

This comment has been minimized.

Copy link
Contributor

commented Sep 12, 2017

🎉 🎉 thanks andy

@bradfitz

This comment has been minimized.

Copy link
Member

commented Sep 12, 2017

Pubsubhelper can't be the only problem. It's only a wake-up for maintner. Maintner polls Gerrit regularly as a fallback without any pubsubhelper.

So what's the deeper problem?

@adams-sarah

This comment has been minimized.

Copy link
Contributor

commented Sep 12, 2017

maintner is seeing the new commits over and over...
eg.

EDIT: pasted wrong log entry. will update in ~hr

could it be that something is off with processGerritMutation() (maintner/gerrit.go:483)?

have to run to a mtg but will keep looking at this after.

@adams-sarah

This comment has been minimized.

Copy link
Contributor

commented Sep 12, 2017

ok they're back up.
i still have no idea what happened. but this seemed related to #21555 so tried restarting gitmirror b/c i have to leave work now.
that worked...

... will look at this more tomorrow.

@adams-sarah

This comment has been minimized.

Copy link
Contributor

commented Sep 26, 2017

Copy-pasting my update from #21555 :
[The root cause of this issue is that] the builders are set up such that if a commit is pushed to the dashboard, and the dashboard does not recognize the new commit's parent commit SHA, it will reject the new commit. This puts us in a place where, if we lose one commit, we can never recover (until we restart gitmirror).

@bradfitz bradfitz closed this Feb 14, 2018

@golang golang locked and limited conversation to collaborators Feb 14, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.