Log fatal errors

[lars-t-hansen]

See #51. We should detect unrecoverable error situations that prevent monitoring from working, and we should arguably log them to some standard syslog or other medium where they will be seen.

[lars-t-hansen]

Probably there are two aspects of this, one is propagating the errors properly within sonar - add_gpu_info in ps.rs drops the error on the floor, for example - and the other is logging in a standard location in a standard way.

[lars-t-hansen]

It could look like it might be sufficient to use the syslog crate, https://crates.io/crates/syslog, and that whatever plumbing is necessary to adapt to the local system conventions is hidden behind the syslog service.

[lars-t-hansen]

An alternative view is that sonar will always be run by cron and that the logging and output handling performed by cron - mailing the output to the owner of the job - is sufficient. I think there's no rush to implement anything here, we need to examine the entire pipeline first.

[bast]

We can then follow this: https://rust-lang-nursery.github.io/rust-cookbook/development_tools/debugging/log.html

[lars-t-hansen]

Related to logging, I see intermittent clusters of sonar errors reported by cron of this form:

SONAR ERROR: "CPU process listing failed"

(This morning there was a cluster of six of these on ML7. It would have been useful to see more information here, to better diagnose the problem. Over the summer I had a similar cluster on another of the nodes.)

[lars-t-hansen]

PR #71 addresses all of this.