Logging performance and status measurements

[ion1]

To have built-in logging of measurements in order to generate graphs such as the ones in #1750, we could simply have an on-disk circular array of e.g. 8640 floating-point samples for 24 hours of data at 1 sample/10 seconds along with a pointer to the last update, its timestamp and the number of samples in the array. If appending a new sample would overwrite the oldest sample, first free the oldest sample by decrementing the number of samples.

If we want to be fancier and store measurements from a longer time period but without growing the size of the log linearly, we could instead (upon an append to a full array) free the two oldest samples and append their average to another array. The second array would have 8640 samples for 48 hours of data at 1 sample/20 seconds. Given a chain of such arrays, this would result in a log size of about O(log₂ time).

A chain of arrays as described above would be maintained for every value to be measured.

This log could be plotted and analyzed using any tool such as RRDtool, Gnuplot, Flot, IPFS GUIs etc.

If we implemented even fancier things such as having multiple downsampled array chains for different downsampling functions (average, min, max, …), we would basically have reimplemented RRDtool’s logging part and at that point could just depend on it. But if we’re happy with a simple circular array or a repeatedly downsampled chain of circular arrays, we could implement it with very little code and no dependencies.

[ghost]

We're exposing a Prometheus scraping endpoint at :5001/debug/metrics/prometheus and run an internal Grafana dashboard.

The only IPFS-specific metric so far is ipfs_p2p_peers_total: 8b164f9 && 9c30b85

[jbenet]

Yeah, we should add more metrics to that \o/

@ion1 want to try adding the metrics you need for your nifty dup graphs? (we could turn certain metrics on with certain config options.)

[ion1]

I would certainly like to contribute code but I’m afraid I can’t say if and when I’ll have the energy to study the go language and the go-ipfs codebase.

[whyrusleeping]

metrics! cc @Kubuxu

[eingenito]

This issue has been superseded by #5783, #5604 and ongoing work on gateway performance monitoring.