Allow Prometheus to start up with a remote storage as primary fanout storage

[vjsamuel]

Proposal

We the monitoring team at eBay are interested in seeing if it is possible to use the Prometheus server in a mode where a remote store is the primary storage so that we can leverage the power of PromQL. In its current state the prometheus server bootstrap doesn't allow that. Would it be acceptable in the eyes of the community to be able to spin up the Prometheus server in a more flexible manner that allows that? We would be glad to refactor cmd/prometheus/main.go to support that.

[brian-brazil]

This has come up before, and it's not something we plan on supporting as you should always have a local database.

For this sort of use case you're usually best using Prometheus as a library, rather than using the official binaries.

[vjsamuel]

@brian-brazil, thanks for the comment. The idea is to break down cmd/main.go and introduce a cmd/server package that has server.Server, server.Config, server.NewPrometheusServer(config *Config). The main.go would bootstrap Prometheus exactly how community intents but it will also enable rest of the community to use cmd/server as a library to run it with custom requirements as ours where we want remote storage as the default. Would that be acceptable?

[brian-brazil]

That'd end up being a new API we have to support, and in general we don't permit code to exist that the project itself doesn't need, nor containing custom code - as that ends up being a maintenance problem. This is something you'd have to maintain yourself in your own repo. Be aware we offer no stability guarantees on internal libraries.

[henridf]

where a remote store is the primary storage so that we can leverage the power of PromQL.

@vjsamuel it sounds like the Prometheus server in this case is being used read only, is that correct? If so you can just run it without any scraping jobs configured and your local TSDB will stay empty. (You'll still need to provide a volume for it, even if no samples are written to it)

[tomwilkie]

As of 2.8 this is going to get harder, as the new remote_write code is tied to the WAL code for TSDB. It might be possible for you to have a WAL-only TSDB mind you.

[valyala]

We the monitoring team at eBay are interested in seeing if it is possible to use the Prometheus server in a mode where a remote store is the primary storage so that we can leverage the power of PromQL.

You can configure Prometheus to store data in systems with native PromQL support such as Thanos, Uber M3, Cortex or VictoriaMetrics. Then the data may be queried directly from these systems using PromQL without resorting to Prometheus.