Add TLS support for Prometheus Reader

[albertteoh]

Signed-off-by: albertteoh albert.teoh@logz.io

Which problem is this PR solving?

• Relates to "Monitor" tab for service health metrics #2954.

Short description of the changes

• Adds TLS encryption to Prometheus server.

Testing

• As documented, Prometheus does not natively support TLS encryption for connections to its HTTP API, however, it does suggest an alternative means of enforcing TLS encryption at the proxy layer with something like nginx.
• This PR was tested using nginx as a reverse proxy to Prometheus, as suggested above.

# Queries succeed: Directly connect to Prometheus, bypassing nginx
$ METRICS_STORAGE_TYPE=prometheus ./cmd/query/query-darwin-amd64 \
  --prometheus.server-url http://localhost:9090

# Queries succeed: Enable TLS and provide public key
$ METRICS_STORAGE_TYPE=prometheus ./cmd/query/query-darwin-amd64 \
  --prometheus.server-url https://example.com/prometheus \
  --prometheus.tls.ca=example.com.crt \
  --prometheus.tls.enabled

# Queries fail: TLS not enabled
$ METRICS_STORAGE_TYPE=prometheus ./cmd/query/query-darwin-amd64 \
  --prometheus.server-url https://example.com/prometheus  \
  --prometheus.tls.ca=example.com.crt 
...failed executing metrics query: Post "https://example.com/prometheus/api/v1/query_range": x509: certificate signed by unknown authority...

# Queries fail: Missing public key
$ METRICS_STORAGE_TYPE=prometheus ./cmd/query/query-darwin-amd64 \
  --prometheus.server-url https://example.com/prometheus  \
  --prometheus.tls.enabled
...failed executing metrics query: Post "https://example.com/prometheus/api/v1/query_range": x509: certificate signed by unknown authority...

[albertteoh]

Motivation for renaming:

• To use TLS with Prometheus requires the protocol scheme https, as with how ES's server URLs are defined. Hence, I felt it made more sense that users are required to provide the appropriate protocol scheme of either http or https.
• Consistency with ES options server-urls, and made more sense than host-port if requiring the protocol scheme.

[codecov]

Codecov Report

Merging #3055 (e9d6712) into master (0b59004) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3055      +/-   ##
==========================================
- Coverage   96.03%   96.01%   -0.03%     
==========================================
  Files         229      229              
  Lines        9941     9956      +15     
==========================================
+ Hits         9547     9559      +12     
- Misses        325      328       +3     
  Partials       69       69              

Impacted Files	Coverage Δ
plugin/metrics/prometheus/factory.go	100.00% <100.00%> (ø)
plugin/metrics/prometheus/metricsstore/reader.go	100.00% <100.00%> (ø)
plugin/metrics/prometheus/options.go	100.00% <100.00%> (ø)
...lugin/sampling/strategystore/adaptive/processor.go	99.07% <0.00%> (-0.93%)	⬇️
plugin/storage/integration/integration.go	77.34% <0.00%> (-0.56%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b59004...e9d6712. Read the comment docs.

[yurishkuro]

validate:"nonzero" mapstructure:"server"

Where are these being used? I thought they were introduced when @pavolloffay tried to use OTEL's config mechanism to populate Jaeger configs, but we stopped doing that.

[yurishkuro]

If not used, I would remove the annotations

[albertteoh]

The annotations are based on existing example config structs, and I didn't know what they were used for as well tbh; removed.

[yurishkuro]

that means you never want to support non-TLS connections?

[albertteoh]

It means to use the DefaultRoundTripper supplied by Prometheus client (the client will do this on our behalf), but you make a good point that we lose our ConnectTimeout configuration with this approach. I've written a test to fail this code and fixed the implementation.

[albertteoh]

Thanks for the review @yurishkuro!