Implement metrics reader

[albertteoh]

Signed-off-by: Albert Teoh albert.teoh@logz.io

Which problem is this PR solving?

• Relates to "Monitor" tab for service health metrics #2954

Short description of the changes

• Implements GetLatencies, GetCallRates and GetErrorRates of the Prometheus MetricsReader.

[codecov]

Codecov Report

Merging #3004 (9e80fa6) into master (23bbd63) will increase coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3004      +/-   ##
==========================================
+ Coverage   95.94%   95.99%   +0.04%     
==========================================
  Files         224      225       +1     
  Lines        9773     9891     +118     
==========================================
+ Hits         9377     9495     +118     
  Misses        326      326              
  Partials       70       70              

Impacted Files	Coverage Δ
...trics/prometheus/metricsstore/dbmodel/to_domain.go	100.00% <100.00%> (ø)
plugin/metrics/prometheus/metricsstore/reader.go	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 23bbd63...9e80fa6. Read the comment docs.

[albertteoh]

Friendly nudge on this, if anyone could take a look please; I'd like to move forward on this feature.

I'm cognisant that this PR is on the larger side as I wanted to provide more context for the reviewer; however, if the preference is to make this smaller, I could break it up into smaller PRs too.

The failing unit test is caused by a known flaky test issue, which I'm looking at now: #3025

[jpkrohling]

I had a question about the usage of sum(rate()), but other than that, looks good to me.

[albertteoh]

Thank you for the review, @jpkrohling!