Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/tip: add _tipstatus error monitoring to farmer.golang.org#health #15266

Closed
adg opened this issue Apr 13, 2016 · 6 comments

Comments

Projects
None yet
4 participants
@adg
Copy link
Contributor

commented Apr 13, 2016

A recent change to cmd/go (5b3e576) broke cmd/tip, but we didn't realize it was broken because the active instances just kept serving the last working version. Eventually all four instances were restarted and they all failed to start godoc, being unable to find it. By that stage it was hard to understand what was going on.

It would be better if cmd/tip could somehow report that it has been wedged for a long time. Ideally before it stops serving properly.

An additional, more radical idea is for cmd/tip to—on startup—try earlier versions of go and tools to see if they work. That seems like a lot of fiddly nonsense though.

@adg adg self-assigned this Apr 13, 2016

@adg

This comment has been minimized.

Copy link
Contributor Author

commented Apr 13, 2016

@dmitshur

This comment has been minimized.

Copy link
Member

commented Jan 22, 2019

/cc @cnoellekb @bradfitz @andybons This enhancement is something we might want to take into account as part of moving the website to the x/website repository (issue #29206).

A similar issue happened in #29874, where tip.golang.org was out of date for 13 days before someone noticed.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jan 22, 2019

Well, we could add monitoring for tip's /_status and send emails when it's angry.

@FiloSottile FiloSottile changed the title x/tools/cmd/tip: better diagnostics when a build fails x/build/cmd/tip: better diagnostics when a build fails Feb 1, 2019

@gopherbot gopherbot added the Builders label Feb 1, 2019

@dmitshur

This comment has been minimized.

Copy link
Member

commented Jul 5, 2019

We have https://farmer.golang.org/#health now, which is a great place to add a check for this.

However, we'll likely need to resolve #32949 first.

@dmitshur dmitshur added the NeedsFix label Jul 5, 2019

@dmitshur dmitshur changed the title x/build/cmd/tip: better diagnostics when a build fails x/build/cmd/tip: add _tipstatus error monitoring to https://farmer.golang.org/#health Jul 5, 2019

@dmitshur dmitshur changed the title x/build/cmd/tip: add _tipstatus error monitoring to https://farmer.golang.org/#health x/build/cmd/tip: add _tipstatus error monitoring to farmer.golang.org#health Jul 5, 2019

@gopherbot

This comment has been minimized.

Copy link

commented Jul 5, 2019

Change https://golang.org/cl/185139 mentions this issue: cmd/tip: clear error when process starts successfully

gopherbot pushed a commit to golang/build that referenced this issue Jul 9, 2019

cmd/tip: clear error when process starts successfully
We want the error to represent an ongoing problem. It's less useful
to constantly show the last error; we have logs for that.

This will help make the _tipstatus output more readable for humans,
and more friendly to being monitored by farmer.golang.org#health.

Updates golang/go#15266
Fixes golang/go#32949

Change-Id: I20e5f180209c54b31e81e29bd45af0e2d205d3cc
Reviewed-on: https://go-review.googlesource.com/c/build/+/185139
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>

@dmitshur dmitshur self-assigned this Jul 12, 2019

@gopherbot

This comment has been minimized.

Copy link

commented Jul 12, 2019

Change https://golang.org/cl/185981 mentions this issue: cmd/coordinator: add health item for tip.golang.org

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.