x/build: monitor all the parts of the build system

[bradfitz]

The watcher is not pushing new commits to build.golang.org.

I see:

  949 ?        Ssl    8:51 docker daemon --host=fd:// --selinux-enabled
17215 ?        Ssl   15:55  \_ /usr/local/bin/watcher -role=watcher -watcher.repo=https://go.googlesource.com/go -watcher.dash=https://build.golang.org/ -watcher.poll=10s -watcher.http=127.0.0.1:21536 -watcher.mirror=htt
 5570 ?        Z      0:00      \_ [git-remote-http] <defunct>
 5608 ?        Z      0:00      \_ [git-remote-http] <defunct>
 5618 ?        Z      0:00      \_ [git-remote-http] <defunct>
16843 ?        S      1:34      \_ git push -f --mirror dest
18371 ?        S      0:00      |   \_ git-remote-https dest https://gopherbot:[redacted]@github.com/golang/go
18178 ?        S      0:00      \_ git push -f --mirror dest
18389 ?        S      0:00      |   \_ git-remote-https dest https://gopherbot:[redacted]@github.com/golang/mobile
18335 ?        S      0:00      \_ git push -f --mirror dest
18387 ?        R      0:00          \_ git-remote-https dest https://gopherbot:[redacted]@github.com/golang/oauth2

And http://farmer.golang.org/debug/watcher at the end says only:

2016/05/19 18:43:40 go: sending commit to dashboard: ---40hexomitted---[master]("build: unset GOBIN during build")
2016/05/19 19:29:24 exp: found new commit ---40hexomitted---("shiny/widget/flex: add HTML printing of tests")
2016/05/19 19:29:24 exp: updated branch head: "master"(Head: ---40hexomitted---[master]("shiny/widget/flex: add HTML printing of tests") LastSeen: ---40hexomitted---[master]("shiny/driver/x11driver: have textureImpl.draw use an opaque mask."))
2016/05/19 19:29:24 exp: sending commit to dashboard: ---40hexomitted---[master]("shiny/widget/flex: add HTML printing of tests")
2016/05/19 19:36:46 exp: found new commit ---40hexomitted---("shiny/widget/flex: basics of flex algorithm")
2016/05/19 19:36:46 exp: updated branch head: "master"(Head: ---40hexomitted---[master]("shiny/widget/flex: basics of flex algorithm") LastSeen: ---40hexomitted---[master]("shiny/widget/flex: add HTML printing of tests"))
2016/05/19 19:36:46 exp: sending commit to dashboard: ---40hexomitted---[master]("shiny/widget/flex: basics of flex algorithm")
2016/05/19 22:45:58 exp: found new commit ---40hexomitted---("shiny/driver/x11driver: tighten the textureImpl.draw dst rectangle.")
2016/05/19 22:45:58 exp: updated branch head: "master"(Head: ---40hexomitted---[master]("shiny/driver/x11driver: tighten the textureImpl.draw dst rectangle.") LastSeen: ---40hexomitted---[master]("shiny/widget/flex: basics of flex algorithm"))
2016/05/19 22:45:58 exp: sending commit to dashboard: ---40hexomitted---[master]("shiny/driver/x11driver: tighten the textureImpl.draw dst rectangle.")

Server time is:

$ date
Thu May 19 23:06:28 UTC 2016

So it's just the go repo that's hung?

But the git subprocess for the go repo keeps coming & going. It's not hung there.

We should have a debugging endpoint to get the watcher's goroutines too.

I'll kick it for now.

This also needs to be monitored.

[adg]

The watcher has caught up again, so this issue is now about watching the watcher.

[quentinmit]

This sounds like the fd leak I noticed recently. Do you remember if you noticed a large number of fds when you saw this in May?

[adg]

I didn't notice, sorry.

[quentinmit]

The FD leak that caused most of these problems was fixed with golang/build@44e74d5

Can we close this issue for now and reopen if we have different problems with the watcher again?

[bradfitz]

This bug is about setting up monitoring, which we still don't have.

[gopherbot]

CL https://golang.org/cl/37457 mentions this issue.

[gopherbot]

CL https://golang.org/cl/40397 mentions this issue.

[gopherbot]

CL https://golang.org/cl/47490 mentions this issue.

[gopherbot]

CL https://golang.org/cl/47934 mentions this issue.

[gopherbot]

Change https://golang.org/cl/178798 mentions this issue: cmd/coordinator: add some health status handlers & output on front page

[gopherbot]

Change https://golang.org/cl/181217 mentions this issue: cmd/makemac: export warnings and error state in status JSON endpoint