Persist Retroactive Rule Reevaluations

[juliusv]

We can already reevaluate queries on old data, but we should be able to persist that for a certain window from [Oldest, Now).

[matttproud]

@juliusv, we know that we can support retroactive rule evaluation now. Is that sufficient to close this issue, or should it be repurposed to handle the case of "archiving samples after retroactive rule evaluation"?

[juliusv]

It's not our top priority, but persisting timeseries resulting from rule evaluations against historical data is still a nice-to-have feature, so we can keep this around.

[matttproud]

Great. I'll re-title this issue. :)

2013/3/28 juliusv notifications@github.com

It's not our top priority, but persisting timeseries resulting from rule
evaluations against historical data is still a nice-to-have feature, so we
can keep this around.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/11#issuecomment-15575551
.

[brian-brazil]

My thoughts on this are to permit backfilling via the API, and then we can have a separate tool that does this.

[bwplotka]

Hi, I would like to start a new discussion on this.
Basically, there are some clear benefits of having some back-in-time rule evaluation option for Prometheus.

Currently, there are certain UX issues while using rules. Let's have some basic example from a user perspective:

• I created some cool, new metric.
• I created some fancy dashboard.
• Everybody loves my dashboard and it's quite popular, so I want to create a rule for the query used there to take out some load from Prometheus.
• I created a rule.

Problems:

1. How to test my rule? I need to wait some time to have it filled to check if I got it right for a longer period (that could include interesting cases). That requires lot's of context switches in my work and frustration (multiple frustrations by 100x when one is a newbie in PromQL)
2. I cannot immediately put my rule on the dashboard because it's not yet filled

I totally agree that this is not critical and quite hard to achieve when we are talking about persisting new time series back in time when that time range is already packed in blocks and maybe even compacted or backed up/downsampled in case of thanos or remote-written.

But what about some simple solution (A), for example:

• User created new rule
• If flag XXX is set (or certain RPC is invoked on API), reevaluate the rule against all samples you have in memory (WAL)

The only unknown here is remote write, but when we make this feature optional - you can control if you want to have back-in-time rule evaluation or not.

Or maybe another option (B):

• User created new rule
• Backfill the rule in ephemeral space only (mem only) up to flag-rule-reeval-time or API RPC argument to make testing and using this rule on dashboards easier. "Past" samples will be ephemeral. Only new samples will be written normally.

Any thoughts?

CC @juliusv @brian-brazil @fabxc @devnev

[brian-brazil]

The requirement here is basically for bulk import, as I don't think this functionality belongs in Prometheus itself (though may be tied together via promtool).

[bwplotka]

bulk import

Sorry, what do you mean by that?

[brian-brazil]

#535