Add Victoria Metrics support

[AndrewChubatiuk]

Is your feature request related to a problem? Please describe.
We're using Victoria Metrics for monitoring, which is provides same API as Prometheus but has a different architecture.

In a multicluster monitoring configuration instead of having Prometheus instance on each cluster Victoria Metrics provides component called VMAgent, which has no /query or /query_range endpoints. It scrapes metrics from K8s cluster and pushes them to VMCluster. VMCluster stores all metrics and has /query and /query_range endpoints to query them, but VMCluster has no /status/config endpoint (which is used by OpenCost to get scrape configuration). Opencost has ability to configure only one Prometheus endpoint for both /query and /status/config endpoints, which is not applicable for Victoria Metrics setup.

Another problem is that Opencost relies on a local Prometheus setup, that's why it doesn't filter current cluster metrics, which is not applicable for VMCluster as well.

Describe the solution you'd like
To support Victoria Metrics it's needed to:

• add current cluster filter for all PromQL queries
• add an optional client for scrape endpoint when it's defined

[mattray]

This sounds like a worthwhile enhancement we'd take PRs for.

@AjayTripathy do you have recommendations for how to proceed?

[AndrewChubatiuk]

@mattray #1430 and #1861 should solve this issue

[mattray]

#1430 and #1861 were merged. https://www.opencost.io/blog/victoria-metrics

[AndrewChubatiuk]

@mattray #1861 was then reverted, this issue is not resolved

[joelhess]

@mattray Is there a current status on this? It looks like the MRs's required referenced, were merged, one was reverted, and then put back in and re-merged.

[AndrewChubatiuk]

@joelhess you can partially follow a guide https://victoriametrics.com/blog/monitoring-kubernetes-costs-with-opencost-and-victoriametrics/. Unfortunately image of version 2.2.0, which contains functionality, described in a guide was removed from registry. You need to build your own image

[mattray]

The 2.2.0 was an inadvertent Kubecost-versioned build that we've removed. OpenCost 1.110.0 should contain this and ship soon.

[joelhess]

Is there a target date for v1.110? We're trying to plan out some work and this would be helpful to know if we should just use Prom for now. (We'd much prefer Victoria as that's what we're using for everything else)

[mattray]

We're hoping to get the release out soon, but we're blocking on getting a UI for the Carbon Costs and Datadog integration merged.

[PabloSebastian]

Is the VictoriaMetrics support included in the latest release, v1.110?

[AndrewChubatiuk]

@PabloSebastian yes, it was released in 1.110. you can find information in a blog post
https://new.victoriametrics.com/blog/monitoring-kubernetes-costs-with-opencost-and-victoriametrics/

[PabloSebastian]

@AndrewChubatiuk Thanks for the quick response!
I followed that documentation. I'm working with a VictoriaMetrics vmsingle, but I got the following error:

2024-04-30T19:40:00.011925792Z ??? Log level set to info
2024-04-30T19:40:00.012051994Z INF Starting cost-model version 1.110 (a1dfd1c)
2024-04-30T19:40:00.012064494Z INF Kubernetes enabled: true
2024-04-30T19:40:00.012139095Z INF Prometheus/Thanos Client Max Concurrency set to 5
2024-04-30T19:40:00.026886311Z ERR Failed to query prometheus at http://vmsingle-victoria-metrics-k8s-stack.victoria-metrics.svc:8429/prometheus. Error: no running jobs on Prometheus at /prometheus/api/v1/query . Troubleshooting help available at: https://www.opencost.io/docs/integrations/prometheus
2024-04-30T19:40:00.028577735Z INF Retrieved a prometheus config file from: http://vmsingle-victoria-metrics-k8s-stack.victoria-metrics.svc:8429/prometheus
2024-04-30T19:40:00.028617836Z INF Using scrape interval of 60.000000
2024-04-30T19:40:00.029258045Z INF NAMESPACE: opencost
2024-04-30T19:40:00.230367689Z INF Done waiting

Do you know if it is a common issue?

[AndrewChubatiuk]

You've copied VictoriaMetrics address from blogpost, is it available at http://vmsingle-victoria-metrics-k8s-stack.victoria-metrics.svc:8429/prometheus in your case? also have you deployed vmstack helm chart it in the same cluster in victoria-metrics namespace? What is a service name for vmsingle?

[PabloSebastian]

Yes, I have deployed the VictoriaMetrics stack in the same cluster in the victoria-metrics namespace. Here is the logs after making a kubectl get svc

$ kubectl get svc -n victoria-metrics
NAME                                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)            AGE
victoria-metrics-k8s-stack-cen-kube-state-metrics          ClusterIP   10.255.26.132   <none>        8080/TCP           24d
victoria-metrics-k8s-stack-cen-prometheus-node-exporter    ClusterIP   10.255.28.53    <none>        9100/TCP           24d
victoria-metrics-k8s-stack-cen-victoria-metrics-operator   ClusterIP   10.255.11.140   <none>        8080/TCP,443/TCP   24d
vmagent-victoria-metrics-k8s-stack                         ClusterIP   10.255.14.234   <none>        8429/TCP           22d
vmsingle-victoria-metrics-k8s-stack                        ClusterIP   10.255.9.7      <none>        8429/TCP           17h

[AndrewChubatiuk]

Sorry, haven't read your error correctly. this error is expected, as Opencost didn't want to support multiple prometheus endpoints (one was required to retrieve scrape config from VMAgent, another one - to scrape metrics from VMSingle). That's why scrape interval should be set manually for opencost as it's mentioned in a blog

[mattray]

@AndrewChubatiuk can we close this issue?

[joelhess]

We're trying to follow up on this. There's another issue tracking on the helm chart repo, but we don't really think it's related there.. opencost/opencost-helm-chart#201