Implement remote_write method

[iglov]

So that we can build the scheme

[valyala]

Thanks for the feature request! It looks useful. I'll think about the best implementation.

Currently the following workarounds exist that allow writing metric subsets into distinct VictoriaMetrics instances:

1. Prometheus can be configured with multiple remote_write sections. Each section can contain distinct configs for write_relabel_configs and distinct url values pointing to distinct VictoriaMetrics instances. This way Prometheus will write data simultaneously to all the configured remote_write destinations after applying write_relabel_configs to them.

2. VictoriaMetrics supports /federate endpoint, so it is possible to configure additional Prometheus instances that would scrape the required subset of metrics from VictoriaMetrics via /federate endpoint and write them to another VictoriaMetrics instances.

[valyala]

@iglov , the outlined scheme with metrics relabeling will be possible soon with the upcoming vmagent tool. It can accept data via remote_write API, relabel it and then write it to other remote storage systems via remote_write API. Read more information about it here.

[valyala]

FYI, vmagent has been released - see https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md

[iglov]

Hey, Sashka! :)
Well, seems like it isn't exactly what i want, but ok. Is this possible with vmagent to do something like this:

[valyala]

Currently -remoteWrite.relabelConfig is shared among all the configured -remoteWrite.url values. I think it would be nice to add ability to set per-remote storage relabel configs in vmagent. In this case your schema should work.

In the mean time you can add yet another vmagent in front of each victoriametrics, which will delete unneeded metrics according to its relabel configs. Note that vmagent instances may be chained, since they support data ingestion via Prometheus remote_write API at http://vmagent:8429/api/v1/write endpoint.

[valyala]

@iglov , the per-remote storage relabeling has been added to vmagent in the commit 02b55c7 . It allows setting -remoteWrite.urlRelabelConfig for each configured -remoteWrite.url:

/path/to/vmagent \
  -remoteWrite.url=http://vm1:8428/api/v1/write -remoteWrite.urlRelabelConfig=relabel1.yml \
  -remoteWrite.url=http://vm2:8428/api/v1/write -remoteWrite.urlRelabelConfig=relabel2.yml

Try building vmagent from sources according to these instructions and verifying whether it works for your case.

[valyala]

The per-remote storage relabeling via -remoteWrite.urlRelabelConfig has been added in v1.34.2.

Closing the feature request as done. @iglov , feel free re-opening it or creating new feature request if you feel you need additional functionality related to this feature request.

[iglov]

O-o-ok, i'll see and if i'll have any question, i'll ask on telegram, thx Sashka ;)