[zookeeper] bytes_received and bytes_sent metrics do not represent bytes

[eliaslevy]

The zookeeper.bytes_received and zookeeper.bytes_sent metrics are misnamed.

The data for these metrics is collected from the ZK stat command, which prints Received and Sent fields. But these fields represent packets received and sent, not bytes. This in turn means that the DD ZK dashboard is wrong when it displays the data in a widget named Bytes sent/received/outstanding per second.

In addition, both metrics are being submitted as gauge types. In reality they are monotonic counts, except they are not truly monotonic as they can be reset using the ZK srst command and they will reset if the ZK server is restarted.

It appears that the developers of the ZK dashboard where aware of the wrong metric type applied to zookeeper.bytes_received and zookeeper.bytes_sent, as the dashboard uses the per_second function to convert the "gauges" into rates.

As an aside, the ZK dashboard mistakenly displays the zookeeper.bytes_outstanding metric in the same widget as the zookeeper.bytes_received and zookeeper.bytes_sent metrics. Even if those two metrics did refer to received and sent bytes displaying zookeeper.bytes_outstanding with them would be an error. As the comment in the collector notes, that metric refers to outstanding requests, not outstanding bytes.