Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/tip: add _tipstatus error monitoring to farmer.golang.org#health #15266

Closed
adg opened this issue Apr 13, 2016 · 6 comments
Closed

x/build/cmd/tip: add _tipstatus error monitoring to farmer.golang.org#health #15266

adg opened this issue Apr 13, 2016 · 6 comments

Comments

@adg
Copy link
Contributor

@adg adg commented Apr 13, 2016

A recent change to cmd/go (5b3e576) broke cmd/tip, but we didn't realize it was broken because the active instances just kept serving the last working version. Eventually all four instances were restarted and they all failed to start godoc, being unable to find it. By that stage it was hard to understand what was going on.

It would be better if cmd/tip could somehow report that it has been wedged for a long time. Ideally before it stops serving properly.

An additional, more radical idea is for cmd/tip to—on startup—try earlier versions of go and tools to see if they work. That seems like a lot of fiddly nonsense though.

@adg adg self-assigned this Apr 13, 2016
@adg
Copy link
Contributor Author

@adg adg commented Apr 13, 2016

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Jan 22, 2019

/cc @cnoellekb @bradfitz @andybons This enhancement is something we might want to take into account as part of moving the website to the x/website repository (issue #29206).

A similar issue happened in #29874, where tip.golang.org was out of date for 13 days before someone noticed.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jan 22, 2019

Well, we could add monitoring for tip's /_status and send emails when it's angry.

@FiloSottile FiloSottile changed the title x/tools/cmd/tip: better diagnostics when a build fails x/build/cmd/tip: better diagnostics when a build fails Feb 1, 2019
@gopherbot gopherbot added the Builders label Feb 1, 2019
@dmitshur
Copy link
Member

@dmitshur dmitshur commented Jul 5, 2019

We have https://farmer.golang.org/#health now, which is a great place to add a check for this.

However, we'll likely need to resolve #32949 first.

@dmitshur dmitshur added the NeedsFix label Jul 5, 2019
@dmitshur dmitshur changed the title x/build/cmd/tip: better diagnostics when a build fails x/build/cmd/tip: add _tipstatus error monitoring to https://farmer.golang.org/#health Jul 5, 2019
@dmitshur dmitshur changed the title x/build/cmd/tip: add _tipstatus error monitoring to https://farmer.golang.org/#health x/build/cmd/tip: add _tipstatus error monitoring to farmer.golang.org#health Jul 5, 2019
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 5, 2019

Change https://golang.org/cl/185139 mentions this issue: cmd/tip: clear error when process starts successfully

gopherbot pushed a commit to golang/build that referenced this issue Jul 9, 2019
We want the error to represent an ongoing problem. It's less useful
to constantly show the last error; we have logs for that.

This will help make the _tipstatus output more readable for humans,
and more friendly to being monitored by farmer.golang.org#health.

Updates golang/go#15266
Fixes golang/go#32949

Change-Id: I20e5f180209c54b31e81e29bd45af0e2d205d3cc
Reviewed-on: https://go-review.googlesource.com/c/build/+/185139
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
@dmitshur dmitshur self-assigned this Jul 12, 2019
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 12, 2019

Change https://golang.org/cl/185981 mentions this issue: cmd/coordinator: add health item for tip.golang.org

codebien added a commit to codebien/build that referenced this issue Nov 13, 2019
We want the error to represent an ongoing problem. It's less useful
to constantly show the last error; we have logs for that.

This will help make the _tipstatus output more readable for humans,
and more friendly to being monitored by farmer.golang.org#health.

Updates golang/go#15266
Fixes golang/go#32949

Change-Id: I20e5f180209c54b31e81e29bd45af0e2d205d3cc
Reviewed-on: https://go-review.googlesource.com/c/build/+/185139
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
codebien added a commit to codebien/build that referenced this issue Nov 13, 2019
This change adds health monitoring for the tip.golang.org website
to farmer.golang.org#health, so we can know when there's an issue
with it without having to manually check it separately from every
other item already monitored at farmer.golang.org#health.

This is possible now that golang/go#32949 is resolved and
the _tipstatus page reports currently ongoing errors only.

Tested by running coordinator locally in dev mode.

Fixes golang/go#15266
Updates golang/go#32949

Change-Id: I3680a916e4b45336df837c72ac7bc555d37f3a9b
Reviewed-on: https://go-review.googlesource.com/c/build/+/185981
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@golang golang locked and limited conversation to collaborators Jul 14, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.