Detect excessive "Health changed to" messages and warn about them

[magiconair]

The issue of having lots of log messages about changed health comes up on a regular basis.

2017/09/06 14:45:07 [INFO] consul: Health changed to #10372843

Users have generally asked on how to suppress them by looking to change the log level. I've refused to implement that with the reason that a) fabio does not log excessively if the system is stable and b) seeing lots of these messages indicates that the system is either not stable (lots of flapping health checks), is getting lots of writes usually from having volatile output like timestamps, pids, ... in the check output. This causes additional writes to the consul master and the raft log replication and should be avoided. The fix is to remove the volatile data from the health checks and not to change the log level.

However, for other reasons I've now added leveled logging in #366 and that means that users will suppress this message by changing the log level instead of acting upon it. Since the message is not explanatory in itself and one has to understand the connection to consul, raft and how fabio uses them it isn't clear what to do.

Therefore, fabio should detect excessive changes in the health checks and log those as WARN. The simplest way would be to determine the rate of change in changes/min or changes/sec and provide a configurable threshold after which fabio starts logging.

See #345 and #205

[manos]

@magiconair is this possible to add to 1.5.3, if it's not too late? :)

[magiconair]

@manos I don't have code for that yet but for now you can set the -discard-check-output option in Consul.

[manos]

oh, right. Thanks :)