Prometheus: reading text format failed

[adamcarter81]

I am using redis exporter on AKS, deployed using the kubernetes sample file to load redis exporter as a sidecar and expose the redis stats. This is working when Prometheus scrapes the data and I can query and see the data in Prometheus for all Redis instances, and I could graph it all in Grafana - awesome!

I am now trying to expose the redis stats to Azure Log Analytics, and this is failing. AKS provides a facility to to scrape Prometheus stats and store them in Azure using it's own OMS Agent. This is working for other stats (e.g. istio), but it fails for redis.

Checking the logs for the oms-agent container I see 2 errors when scraping Redis:

1. With the environment variable REDIS_EXPORTER_REDIS_ONLY_METRICS set to true:

Error in plugin: error reading metrics for http://15.0.0.45:15020/stats/prometheus: reading text format failed: text format parsing error in line 703: expected float as value, got ""

I forwarded the port and did a curl to see what is being returned:

envoy_server_initialization_time_ms_sum{} 285
envoy_server_initialization_time_ms_count{} 1

# HELP redis_active_defrag_running active_defrag_running metric
# TYPE redis_active_defrag_running gauge
redis_active_defrag_running 0

Line 699 is: # HELP redis_active_defrag_running active_defrag_running metric

2. With the environment variable REDIS_EXPORTER_REDIS_ONLY_METRICS set to false:
   E! [inputs.prometheus] Error in plugin: error reading metrics for http://15.0.0.30:15020/stats/prometheus: reading text format failed: text format parsing error in line 699: invalid metric name

envoy_server_initialization_time_ms_bucket{le="+Inf"} 1
envoy_server_initialization_time_ms_sum{} 275
envoy_server_initialization_time_ms_count{} 1

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.18e-05
go_gc_duration_seconds{quantile="0.25"} 7.2499e-05
go_gc_duration_seconds{quantile="0.5"} 9.8699e-05

Line 703 is:# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.

What version of redis_exporter are you running?
Redis Metrics Exporter v1.29.0 build date: 2021-10-20-13:29:25

Running the exporter
Running as a sidecar in Kubernetes

Expected behavior
That the redis statistics are read and processed by the OMS agent, which scrapes the prometheus stats. I checked the format of the returned stats from a curl, and I can't see anything obvious:

5.txt

It may be that I need to open the issue on Microsoft's OMS Agent, but I thought I would get a check here first

Thanks

[oliver006]

Agreed, this doesn't look like an issue with the exporter (it's using the official Prometheus libraries to generate the metrics output) but with Microsoft's OMS Agent.

[adamcarter81]

I've been doing some digging and I think it is related to this issue in Telegraf (used by Microsoft's OMS agent):

influxdata/telegraf#8366

I'll look to see if I can update the OMS agent / put a request in for MS to update Telegraf in the OMS agent image

Thanks