Add support for Prometheus Remote Write

[simskij]

Add support for Prometheus remote write. This would allow for watchtower metrics to be used with, for instance, Grafana Cloud without having to put agents in between and/or expose data sources publicly.

[piksel]

Whats with the interest in Grafana all of a sudden?!

[simskij]

You can use it for remote writing to whatever prometheus server you want! Stahp it! 😅 🤓

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[simskij]

Not stale.

[tammert]

@simskij how would this look like? In my head it sounds like you want to have Watchtower directly push to a remote Prometheus, but I think you need a Pushgateway for that and Grafana Cloud doesn't support that for example: https://community.grafana.com/t/how-to-push-prometheus-metrics-in-grafana-cloud/47297/5. They recommend having a local Prometheus scraping, which is configured with remote_write to the storage of Grafana cloud, but in that regard I don't see what we'd need to change in Watchtower?

[simskij]

@simskij how would this look like? In my head it sounds like you want to have Watchtower directly push to a remote Prometheus, but I think you need a Pushgateway for that and Grafana Cloud doesn't support that for example: https://community.grafana.com/t/how-to-push-prometheus-metrics-in-grafana-cloud/47297/5. They recommend having a local Prometheus scraping, which is configured with remote_write to the storage of Grafana cloud, but in that regard I don't see what we'd need to change in Watchtower?

The idea is to not have to spin up this additional scraping container but rather use the promclient to push the metrics directly to a remote_write endpoint. I did a POC for this in k6 which is available (and continued by someone else) over at https://github.com/grafana/xk6-output-prometheus-remote. Likely, we'd be able to follow the same approach here.

[simskij]

I think we'll settle with putting a grafana agent next to watchtower for now and have that do the remote write. If this setup becomes widely popular with the watchtower community, we can reconsider this and add native support for remote write.