x/build: monitor all the parts of the build system #15760
Comments
The watcher has caught up again, so this issue is now about watching the watcher. |
This sounds like the fd leak I noticed recently. Do you remember if you noticed a large number of fds when you saw this in May? |
I didn't notice, sorry. |
The FD leak that caused most of these problems was fixed with golang/build@44e74d5 Can we close this issue for now and reopen if we have different problems with the watcher again? |
This bug is about setting up monitoring, which we still don't have. |
CL https://golang.org/cl/37457 mentions this issue. |
Will be used for dynamic creation/destruction of Mac VMs in subsequent CL. Updates golang/go#9495 (Mac virtualization) Updates golang/go#15760 (monitoring) Change-Id: I48b17589b258d5d742bad5a3ddae18de98778149 Reviewed-on: https://go-review.googlesource.com/37457 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Currently all builds start and think they're running, but most are just fighting over a mutex to grab a builder. That will be fixed, but in the meantime it's nice to see what's actually working vs what's waiting on e.g. arm5 hardware which won't be available for hours. This is a baby step towards more monitoring. Currently this is just HTML output, but the same data could be exported via JSON or something else later for graphing. Updates golang/go#19178 (add a buildlet scheduler) Updates golang/go#15760 (monitor everything) Change-Id: I36e16ea0919afe8023fe7fedd981f2e857f0d6df Reviewed-on: https://go-review.googlesource.com/40397 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
CL https://golang.org/cl/40397 mentions this issue. |
CL https://golang.org/cl/47490 mentions this issue. |
CL https://golang.org/cl/47934 mentions this issue. |
Carried over from https://golang.org/cl/47490. Updates golang/go#15760 Change-Id: I8b4cc007dea8e32a23cac4cb13bb313d9ec5d4ac Reviewed-on: https://go-review.googlesource.com/47934 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Change https://golang.org/cl/178798 mentions this issue: |
Updates golang/go#15760 Change-Id: I046cc02118daa79e62b408bda591670a8ff70b17 Reviewed-on: https://go-review.googlesource.com/c/build/+/178798 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Change https://golang.org/cl/181217 mentions this issue: |
…oordinator This adds information on warnings & errors to makemac's JSON status handler that is then parsed by the coordinator's health checking code, which already polls this JSON endpoint. Updates golang/go#32449 Updates golang/go#15760 Change-Id: I69bea7b07c184d1f62a358bc317376aa97018230 Reviewed-on: https://go-review.googlesource.com/c/build/+/181217 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The watcher is not pushing new commits to build.golang.org.
I see:
And http://farmer.golang.org/debug/watcher at the end says only:
Server time is:
So it's just the go repo that's hung?
But the git subprocess for the go repo keeps coming & going. It's not hung there.
We should have a debugging endpoint to get the watcher's goroutines too.
I'll kick it for now.
This also needs to be monitored.
The text was updated successfully, but these errors were encountered: