NaN values from Prometheus GoCollector breaking the Status API

[tsandall]

The Status API is broken in v0.37.0 because of an update to the Prometheus client library (v1.12.0 => v1.12.1). With v1.12.1 the GoCollector contains metrics that have NaN values that cannot be JSON serialized. As a result, the status messages that embed these metrics cannot be serialized.

I've filed an issue with the Prometheus client library to see what can be done. prometheus/client_golang#981. In the meantime we'll revert to the old version of the library.

Before close this issue we should:

• Find out whether the metrics coming out of Prometheus are expected to always be JSON serializable (or if this is expected behaviour)
• If some metrics cannot be JSON serialized, we need to exclude them from the Status API or make the Status API tolerant of them being set to NaN
• Add an integration test that enables the Status API with the GoCollector metrics included. Our existing test suite didn't catch this because the GoCollector metrics are only included via the runtime package which doesn't contain integration tests for all of the plugins.

[tsandall]

@srenatus what's happening with this? Do we need to move to a different JSON serialization library? If so, it would be good to update this issue with the details.

[srenatus]

#4324 should do the trick, I believe.

[srenatus]

Fixed by #4324