Monitor stop working after some time

[athieriot]

Hi !

I was using cachet-monitor for about a day or so and noticed that is wasn't properly reporting incident to Cachet.

When I looked at the logs, here what I found:

goroutine 5785 [chan receive]:
net/http.func·016()
    /usr/src/go/src/net/http/transport.go:507 +0x65
created by net/http.func·017
    /usr/src/go/src/net/http/transport.go:513 +0xba

goroutine 5857 [chan receive]:
net/http.func·016()
    /usr/src/go/src/net/http/transport.go:507 +0x65
created by net/http.func·017
    /usr/src/go/src/net/http/transport.go:513 +0xba

goroutine 5712 [chan receive]:
net/http.func·016()
    /usr/src/go/src/net/http/transport.go:507 +0x65
created by net/http.func·017
    /usr/src/go/src/net/http/transport.go:513 +0xba

goroutine 5940 [IO wait]:
net.(*pollDesc).Wait(0xc208289aa0, 0x72, 0x0, 0x0)
    /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208289aa0, 0x0, 0x0)
    /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208289a40, 0xc20823a000, 0x1000, 0x1000, 0x0, 0x7fc61e2dab70, 0xc208232ab0)
    /usr/src/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc208034498, 0xc20823a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/src/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7fc61e2dc060, 0xc208034498, 0xc2082b5bd8, 0xc20823a000, 0x1000, 0x1000, 0x7fc61e2c9010, 0x0, 0x0)
    /usr/src/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20822e0c0, 0xc20823a000, 0x1000, 0x1000, 0xc207ffcbb5, 0x0, 0x0)
    <autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc2080ef500)
    /usr/src/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc2080ef500, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/src/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc2082b5b80)
    /usr/src/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
    /usr/src/go/src/net/http/transport.go:660 +0xc9f

goroutine 5735 [IO wait]:
net.(*pollDesc).Wait(0xc2081e5170, 0x77, 0x0, 0x0)
    /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitWrite(0xc2081e5170, 0x0, 0x0)
    /usr/src/go/src/net/fd_poll_runtime.go:93 +0x43
net.(*netFD).connect(0xc2081e5110, 0x0, 0x0, 0x7fc61e2db078, 0xc2081daea0, 0xecccaab9d, 0x12dde8a5, 0x83ce20, 0x0, 0x0)
    /usr/src/go/src/net/fd_unix.go:114 +0x24b
net.(*netFD).dial(0xc2081e5110, 0x7fc61e2db028, 0x0, 0x7fc61e2db028, 0xc208196a50, 0xecccaab9d, 0x12dde8a5, 0x83ce20, 0x0, 0x0)
    /usr/src/go/src/net/sock_posix.go:139 +0x37a
net.socket(0x6de250, 0x3, 0x2, 0x1, 0x0, 0xc208196a00, 0x7fc61e2db028, 0x0, 0x7fc61e2db028, 0xc208196a50, ...)
    /usr/src/go/src/net/sock_posix.go:91 +0x422
net.internetSocket(0x6de250, 0x3, 0x7fc61e2db028, 0x0, 0x7fc61e2db028, 0xc208196a50, 0xecccaab9d, 0x12dde8a5, 0x83ce20, 0x1, ...)
    /usr/src/go/src/net/ipsock_posix.go:137 +0x148
net.dialTCP(0x6de250, 0x3, 0x0, 0xc208196a50, 0xecccaab9d, 0x12dde8a5, 0x83ce20, 0xc2081f5a30, 0x0, 0x0)
    /usr/src/go/src/net/tcpsock_posix.go:156 +0x125
net.dialSingle(0x6de250, 0x3, 0xc2081f5850, 0x10, 0x0, 0x0, 0x7fc61e2daf98, 0xc208196a50, 0xecccaab9d, 0x12dde8a5, ...)
    /usr/src/go/src/net/dial.go:235 +0x3f5
net.func·016(0xecccaab9d, 0xe12dde8a5, 0x83ce20, 0x0, 0x0, 0x0, 0x0)
    /usr/src/go/src/net/dial.go:163 +0x12c
net.dial(0x6de250, 0x3, 0x7fc61e2daf98, 0xc208196a50, 0xc2080e7b08, 0xecccaab9d, 0xe12dde8a5, 0x83ce20, 0x0, 0x0, ...)
    /usr/src/go/src/net/fd_unix.go:40 +0x6f
net.(*Dialer).Dial(0xc20802e100, 0x6de250, 0x3, 0xc2081f5850, 0x10, 0x0, 0x0, 0x0, 0x0)
    /usr/src/go/src/net/dial.go:170 +0x4be
net.*Dialer.Dial·fm(0x6de250, 0x3, 0xc2081f5850, 0x10, 0x0, 0x0, 0x0, 0x0)
    /usr/src/go/src/net/http/transport.go:38 +0x79
net/http.(*Transport).dial(0xc208042000, 0x6de250, 0x3, 0xc2081f5850, 0x10, 0x0, 0x0, 0x0, 0x0)
    /usr/src/go/src/net/http/transport.go:479 +0x84
net/http.(*Transport).dialConn(0xc208042000, 0x0, 0xc20803aba0, 0x4, 0xc2081f5850, 0x10, 0x14ce5811154, 0x0, 0x0)
    /usr/src/go/src/net/http/transport.go:564 +0x1678
net/http.func·019()
    /usr/src/go/src/net/http/transport.go:520 +0x42
created by net/http.(*Transport).getConn
    /usr/src/go/src/net/http/transport.go:522 +0x335

Does it mean anything to you?

My configuration contains around 7 endpoints to check so maybe that doesn't help (Maybe I should have more than one instance of the monitor I mean)

Thanks,

Aurélien

[matejkramny]

Hi, is that the full stack trace?

It doesn't have the error which would help a lot to debug the issue.

Thanks

[athieriot]

Hi,

This error keeps repeating over and over so I thought it was the full stack trace.

I will try to go back further and see if there is an error on top of the logs.

[athieriot]

Ah you were right ! There is a top level error:

2015/04/23 08:59:48 /go/src/github.com/castawaylabs/cachet-monitor/cachet/incident.go:42: Cannot parse incidents: invalid character '<' looking for beginning of value
2015/04/23 08:59:48 /go/src/github.com/castawaylabs/cachet-monitor/cachet/incident.go:42: Cannot parse incidents: invalid character '<' looking for beginning of value
2015/04/23 08:59:48 /go/src/github.com/castawaylabs/cachet-monitor/cachet/incident.go:69: 401 <html>
<head><title>401 Authorization Required</title></head>
<body bgcolor="white">
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx</center>
</body>
</html>

2015/04/23 08:59:48 /go/src/github.com/castawaylabs/cachet-monitor/cachet/incident.go:74: Cannot parse incident body.
panic: invalid character '<' looking for beginning of value

goroutine 5872 [running]:
github.com/castawaylabs/cachet-monitor/cachet.(*Incident).Send(0xc2080ee8a0)
    /go/src/github.com/castawaylabs/cachet-monitor/cachet/incident.go:75 +0x767
github.com/castawaylabs/cachet-monitor/cachet.(*Monitor).AnalyseData(0xc2080101c0)
    /go/src/github.com/castawaylabs/cachet-monitor/cachet/monitor.go:96 +0x8dc
github.com/castawaylabs/cachet-monitor/cachet.(*Monitor).Run(0xc2080101c0)
    /go/src/github.com/castawaylabs/cachet-monitor/cachet/monitor.go:34 +0x1e0
created by main.main
    /go/src/github.com/castawaylabs/cachet-monitor/main.go:26 +0x671

goroutine 1 [chan receive]:
main.main()
    /go/src/github.com/castawaylabs/cachet-monitor/main.go:24 +0x601

So it seems one of my service returns HTML content as a 401 error.

Would you see it should be fixed upstream of this should be handled by cachet-monitor?

[craftbyte]

Seems like you don’t use the API key or have set the wrong one

[matejkramny]

As @craftbyte pointed out, its an authentication failure for the Cachet API.

cachet-monitor is either configured incorrectly, or the API key has changed and stopped working.

I think this should be caught by cachet-monitor and should quit gracefully or user friendly.. We'll keep this in mind when a new version.

[athieriot]

I will take a look at this API key but what worries me is that it worked fine for a while

[matejkramny]

It did because it didn’t contact the API until an incident happened.

This won’t happen once we release the email notifications for cachet-monitor.

Best,
Matej Kramny

[athieriot]

It successfully reported some incidents before.
I'll try to regenerate the token key but I understand better now.

Thanks for your help, I'll try to keep this issue up to date if I manage to make it work.

[athieriot]

Ok my bad...
There was an htpasswd configuration there for some reason.

Sorry for the troubles for nothing !

Thanks a lot again