feature request: Add agent support for beyond WAL persistent storage and metric export

[bwplotka]

Just an idea.

Currently, we are creating an implementation that has a scrape and WAL only https://github.com/prometheus/prometheus/pull/8785/files

I believe we can go further and offer persistent storage beyond WAL, so if agent is disconnected for longer than 2h it would simply compact block and upload to object storage. This is would an amazing solution for "Too old data" ingestion problem pointed in cortexproject/cortex#2366

Would work well for both Cortex and Thanos, might be more tricky for other vendors since this escapes just Remote Write.

Something to consider 🤗

cc @tomwilkie @rfratto @gouthamve

[rfratto]

Does this require direct access to the object stores used by Cortex/Thanos? (ie no support for a hosted Prometheus backed by either?)

[bwplotka]

Either that or Thanos/Cortex could provide API for upload - whatever works 🤷🏽

[roidelapluie]

Creating blocks, why not, but uploading to bucket storage changes the scope of the project and should be discussed on the dev mailing list or during dev summits.

[codesome]

I believe we can go further and offer persistent storage beyond WAL, so if agent is disconnected for longer than 2h it would simply compact block and upload to object storage.

This idea seems applicable to current Prometheus too where the compaction is already done, just short of shipping the block. It will be easiest (and fastest) to try it out there first if we ever decide to allow it.

[bwplotka]

Totally to be discussed on next Dev Summit 👍🏽 Happy to shepherd this discussion.

[dginhoux]

Hi,

That's a great idea.
So, it scrape targets and forward to a tsdb backend like victoriametrics... very simple and very usefull mode.

How, in agent mode, will work alerting with altermanager ?

I've see the log "field alerting is not allowed in agent mode"... i made a mistake or i don't understand something.

[roidelapluie]

In Agent mode, the alerting has currently to be done by the remote write target (in your case, victoriametrics).

[valyala]

@dginhoux , if you forward data from Prometheus agent to VictoriaMetrics, then alerting rules can be evaluated at VictoriaMetrics side with vmalert.