PrometheusConnect.check_prometheus_connection() does not work

[FutureSharks]

Describe the bug

I can get metrics but PrometheusConnect.check_prometheus_connection() always returns false.

To Reproduce

>>> from prometheus_api_client.utils import parse_datetime
>>> from prometheus_api_client import PrometheusConnect
>>> from datetime import timedelta
>>>
>>> prom = PrometheusConnect(url="http://localhost:8481/select/1/prometheus", disable_ssl=True)
>>> start_time = parse_datetime("10min")
>>> end_time = parse_datetime("now")
>>> chunk_size = timedelta(seconds=60)
>>>
>>> metric_data = prom.get_metric_range_data(
...     'nginx_ingress_controller_requests{service="as01-prd01-oma"}',
...     start_time=start_time,
...     end_time=end_time,
...     chunk_size=chunk_size,
... )
>>> len(metric_data)
2940
>>> prom.check_prometheus_connection()
False
>>>

Expected behavior

PrometheusConnect.check_prometheus_connection() to return True

[FutureSharks]

Perhaps because we are using VictoriaMetrics as storage for prometheus.

$ curl -v http://localhost:8481/select/1/prometheus
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8481 (#0)
> GET /select/1/prometheus HTTP/1.1
> Host: localhost:8481
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Thu, 21 Jan 2021 14:47:47 GMT
< Content-Length: 82

But getting metrics works:

curl 'http://localhost:8481/select/1/prometheus/api/v1/query_range?query=sum(rate(nginx_ingress_controller_requests%7B....%20(status)%20*%2060&start=1611235110&end=1611236910&step=15'

Is that the reason?

[4n4nd]

It looks like your proxy doesn't have the endpoint .../prometheus open, but .../prometheus/api/ is open.

Is that the reason?

I think that is the reason

[FutureSharks]

OK. I guess VM is not common enough to warrant an adaptation in this project so I'll close this and implement my own solution.

[4n4nd]

@FutureSharks do you have any suggestions for a solution?

[FutureSharks]

I did this:

prom = PrometheusConnect(url=prom_url, disable_ssl=disable_ssl)

test_data = prom.get_metric_range_data(
    "prometheus_build_info",
    start_time=parse_datetime("1min"),
    end_time=parse_datetime("now"),
)

if len(test_data) > 0:
    return prom
else:
    raise Exception(f"Prometheus connection not working: {prom_url}")

But it's not perfect. I will see if there's a /health endpoint we can call when using VM....

[4n4nd]

I think you can use

prom.get_current_metric_value(metric_name='up')

[max-rocket-internet]

@4n4nd I think that query is quite heavy, for example it returns over 4K items for me. prometheus_build_info is 1 item per prom server I think 🙂

[4n4nd]

oh my bad, i meant to say that you can use get_current_metric_value instead of get_metric_range_data to make it look cleaner

[max-rocket-internet]

OK I see what you mean, yes get_current_metric_value is cleaner. Thanks!