Monitor errors with runOnSourceError

[dorinclisu]

It would be useful to monitor problems with sources.

Currently I only see this in the logs: 2020/12/17 19:11:14 [0/0/0] [path 29b6ee50] rtsp source ERR: dial tcp 77.141.112.100:554: connect: connection refused

Which is ok for manual monitoring of logs but doesn't look suitable for automatic parsing and generation of alerts when something wrong happens with a specific source.
I think it could be done with a runOnSourceError field, and setting a REASON environment variable that could be:

• timeout
• 404_not_found
• 403_forbidden
• 503_unavailable
• ...

[aler9]

mmm

From my experience with monitoring systems (Grafana, Prometheus, Loki, etc), alarms are often generated from logs, it's enough to setup a modern logger (Loki, Logstash, InfluxDB), extract metadata from log lines through regular expressions, and the configuring alarms - it's way more versatile and efficient than scripts, so my advice is to first try these solutions, then, if they're not enough, we can discuss enhancements.

[dorinclisu]

I already implemented such parsing but it's not so versatile. For example, the logging format changed between version 0.12 and 0.13 so it kind of breaks parsing made for 0.12.

Basically, the rtsp server should be free to log as it wishes, while something like runOnSourceError would be a contract that defines the most common encountered RTSP errors in a less fragile format.

[dorinclisu]

@aler9 Would you consider this as an enhacement? In my opinion (use-case) it's not suitable to generate alerts from a log monitoring dashboard because the end users who edit the connection strings do not (cannot, and should not) access such a dashboard which is only available internally for devops.

So in my application I would of course not generate alerts from the script itself, it's a complex pipeline including NoSQL database but the first step is for the controlling application to get the errors in a reliable format, and the only way I see that is with a runOnSourceError script.

I started digging through the codebase, hoping I could implement myself and PR, but I could not figure it out. For example, I discovered that the error "rtsp source ERR" does not even originate from this code, but from gortsplib code..

[aler9]

errors of sourcertsp can be intercepted here
https://github.com/aler9/rtsp-simple-server/blob/97305af2724810d673907c33e024acd604ec3be2/internal/sourcertsp/source.go#L145

and here
https://github.com/aler9/rtsp-simple-server/blob/97305af2724810d673907c33e024acd604ec3be2/internal/sourcertsp/source.go#L168