Problem reporting system

[kriszyp]

First, the problem statement:
Logging.

Logging sucks for most of what we use it for. To be clear, we have a good logging system, but most of the time the goal of logs and log monitoring is to determine when there is a problem that needs to be addressed. Logging involves sending string messages to a file or stdout/err, capturing that, sending that to 3rd party servers and then spending large amount of money to analyze those logs, finding problems and reporting them.

Instead of: just reporting the problem.

Proposal

The proposal is to pervasively track the status of the various parts of Harper rather than just logging things that happen. As it turns out, we are already have a great foundation/mechanism for this with our component status system. We just haven't really used it much.

Goal

The goal is to have a hierarchical set of statuses returned from get_status that comprehensively reports any problems in the system that need to be addressed. There are some important aspects of this status:

• It should be easy to scan/filter get_status to determine if there are any problems that need to be addressed. We can proactively see this, or a monitoring system can potentially alert admins.
• An error occurring needs to be recorded as a problem to be addressed. An error occurring a million times does not need to be reported a million times. The frequency is just a data point that helps to assess the problem and severity.
• "need to be addressed" is something that involves user/external feedback. "This error has occurred" may be followed by external input that that is ok. The status may still include the error, but no longer is in a state of needing to be addressed. An error that is no longer occurring, based on some established time periods should also be "retired" as a problem that needs to be addressed.

Integration

Current the component status system basically only monitors the component loading cycle, reporting on the success or failure of that. I don't believe we have exposed any way for components themselves to report more details on their status yet. Clearly we could expose these APIs, and make an effort to more pervasively report status information from other components.

Logging Integration

One possible way that we could very quickly move towards expansive reporting is to actually integrate logging with our status reporting. With this approach, log messages would be directly reported to the component status. An error log immediately puts the corresponding component status in an error state. The component status would show that an error has occurred. Of course the error could be marked as irrelevant, or admins could act on it.
This is approach is actually much more viable with how our logging system has been integrated with components. Most logging is (or can be) associated with a specific component, allowing for clear attribution of a log message to a component within the component status hierarchy. Additionally, "tags" can also help define an identity and hierarchy of statuses as well.
There are some compelling aspects of using log integration. First, we don't need to necessarily change code to use this reporting. Our code and user code can just use the logger as we have, and the status will automatically pick up on issues. Also, an integration can provides a clear connection between any status change and associated logs. The record timestamp (or if there is another identity we want to use), allows us to match a status to initial log entry when investigation through logs is warranted.
Integration with logging also has existing/intrinsic severity levels that can inform the component status.
This system could also be a dynamic feedback mechanism. If we have an interactive component status system where users could mark problems as important or as irrelevant, that information could feed back into altering log levels or even omitting them.
This may reemphasize the need to carefully audit/assess our logging. There is a natural tendency to start to ignore noisy logs. If noisy logs carries over to noisy statuses, then we will have status system that will probably be ignored as well. In general, most challenges with monitoring are not due to a lack of information, but too much information to effectively act on.
Another thing that may be missing from a log-based system is a clear way for the code to indicate that a problem has been resolved. For example, if replication can't connect to other nodes, that is probably a problem. But once a connection is established, that problem is/should be resolved. We probably need a clear connection between reporting an error and resolving it.

There are certainly other ways we can build a system for reporting problems, and there could be advantages to distinct APIs for this. For now, I haven't explored this deeply.