Add remote_write support to victoria metrics (single)

[zbindenren]

Is your feature request related to a problem? Please describe.
Currently if someone wants to replace prometheus (with remote_write configured), he must install and configure two services:

• vmagent -> scraping and remote write
• victoria-metrics -> metrics server

vmagent has to be configured to remote write to localhost (victoria-metrics) and to the instance configured in prometheus.yaml.

This makes it harder for someone to switch from prometheus to victoria-metrics or just to try it out.

Describe the solution you'd like

Pefered Solution:
Since victoria-metrics (single) already supports:

• scraping
• alerting

via prometheus.yaml it would be great if it would also pick up the configured remote write entries in prometheus.yaml and starts sending metrics to the configured server.

With that change it would be easier to switch from prometheus to victoria-metrics (for people that do not need a cluster yet).

Other Solution:
For me personally it would already be ok to configure the remote write stuff with the already known flags from vmagent (-remoteWrite.url etc).

Describe alternatives you've considered
Use vmagent with two remote urls as described above.

Additional context
This issue was created because it was suggested so in slack channel.

[hagen1778]

Hi @zbindenren! For people who already own a Prometheus server and wants to try out VictoriaMetrics - why don't just add VictoriaMetrics to remote_write config?
If scraping feature needs to be tested then user can feed prometheus.yaml config directly to VictoriaMetrics.

[zbindenren]

Hi @zbindenren! For people who already own a Prometheus server and wants to try out VictoriaMetrics - why don't just add VictoriaMetrics to remote_write config?
If scraping feature needs to be tested then user can feed prometheus.yaml config directly to VictoriaMetrics.

Yes that is true. I just explain our situation.

Currently we are using some prometheus instances to scrape and remote write some data. Each prometheus server scrapes all other prometheus server. We have alert rules in place when one server is down. We have promxy configured to to query all prometheus servers and there are always two prometheus server scraping the same instance.

To accomplish the same with victoria metrics I have to configure 3 services (victoria-metrics, vmagent and vmalert). So I have to monitor on each node 3 services. It would be much simpler if I could do that with one service. In my opinion it is more complex to configure and operate above scenario with victoria metrics.

I see now, that I would still need two services, even if remote write is supported. As I see vmalert is not included too. So if this feature has any chance to get implemented, I would also like vmauth included.

[slavitch]

In my use case I'd like to have a victoria-metrics behind a NAT remote-write to grafana could,

[Haleygo]

In my case, I would like to scrape and recording in worker cluster, then remote write data to global cluster to do query job.
To accomplish that, I need single victoria-metrics, vmagent and vmalert in worker cluster now, it would be great if that can reduce to two services or even one :)

[ptimofee]

Wow, I was really surprised and disappointed when I found out vmsingle can not remote_write like prometheus does.
In this case vmsingle CAN NOT be a prometheus replacement as you can't build the same chain of regional -> global cluster remote_write like prometheus can do.

[hagen1778]

@ptimofee it is expected that VictoriaMetrics is a final destination in a chain.
For traffic distribution please use vmagent - it provides a wide list of options for traffic routing.

[jensentanlo]

I think that this feature is useful in the common use case where we want to use recording rules in individual clusters to aggregate metrics and join labels.

Ideally we’d want to use VM here because it deals with high cardinality situations better, which is exactly where using recording rules to aggregate would be useful.

[hagen1778]

@jensentanlo if I understand it right, you plan to have something like following:

1. vm-single-node + vmalert installed per k8s cluster
2. A centralized VM cluster installation to accumulate data from many k8s clusters
3. vm-single-node will collect metrics from its local k8s cluster and forwards to centralized cluster only limited number of time series. Other are dropped via relabeling rules.
4. vmalert performs aggregation and forwards to centralized cluster results of recording rules

If my understanding is right, you can still do this but with adding a vmagent. vmagent will be installed per k8s cluster and will be responsible for metrics collection and forwarding to two destinations:

1. centralized VM cluster will receive only limited number of time series, as in p3.
2. local vm-single-node will receive only metrics required for aggregation
3. vmalert will do the same thing as in p4

So you pay some extra for running additional vmagent. But it doesn't look too expensive, since you can store in local vm-single-node not all series, but those are needed for aggregation.

A better solution for this would be using stream aggregation, which potentially could eliminate the need in vmalert and vm-single-node.

[jensentanlo]

@hagen1778 Thank you for the detailed write-up, I didn’t realize that vmalert would handle recording rules as well. Your second suggestion looks like it’ll work; thanks a bunch!

[Giovanniuum]

Hi, another use case for me to add in the balance: we have lots of hosts (around 150 or so) that run a Prometheus agent localy, that scrapes localhost for given metrics, and other exporters for hosts monitoring. All this is already setup, and have working alerting using Mimir alertmanager (with a Terraform toolchain etc., so that it's easy to create our alerting rules).

Because we have a company-deployed Mimir remote storage, we remote_write from those agents to those Mimir. But the mimir remote_write from our company has inconveniants for us, so we want to have our own VictoriaMetrics storage in single-node mode (managed by our team, on around 15 nodes).
To monitor those 15 hosts running VictoriaMetrics, we use node_exporter mainly, that is scraped from prometheus.yaml and exporter to VM server.
But to include the monitoring of these hosts alongside our other 150 hosts monitored through Mimir, we want those node_exporter metrics to be written on Mimir too to let Mimir alertmanager handle all this. But that would mean having VM do remote_write to Mimir tenants, which is currently not possible unless running vmagent on the 15 hosts (or deploying a new alerting chain with vmalert but it would complicate things...).