Arithmetic on the argument of the @ (at) modifier

[aamkye]

What did you do?

I was trying to retrieve specific metrics at yesterdays midnight (23:59:59) by using something like this:

some_custom_metric @ <timestamp>

while

(time() - (hour(vector(time())) * 60 * 60 + minute(vector(time())) * 60) - 60)
and
0+(time() - (hour(vector(time())) * 60 * 60 + minute(vector(time())) * 60) - 60)

returns an OK timestamp, but while trying to use it like

some_custom_metric @ (time() - (hour(vector(time())) * 60 * 60 + minute(vector(time())) * 60) - 60)
or
some_custom_metric @ 0+(time() - (hour(vector(time())) * 60 * 60 + minute(vector(time())) * 60) - 60)

I'm receiving these kinds of errors

Error executing query: 1:10: parse error: unexpected "(" in @, expected timestamp
or
Error executing query: 1:1: parse error: @ modifier must be preceded by an instant vector selector or range vector selector or a subquery

What did you expect to see?

Metric at a specific time.

What did you see instead? Under which circumstances?

Errors:

Error executing query: 1:10: parse error: unexpected "(" in @, expected timestamp
or
Error executing query: 1:1: parse error: @ modifier must be preceded by an instant vector selector or range vector selector or a subquery

System information

Prometheus in K8s via Helm (2.43.0)

Prometheus version

Version	2.43.0
Revision	edfc3bcd025dd6fe296c167a14a216cab1e552ee
Branch	HEAD
BuildUser	root@8a0ee342e522
BuildDate	20230321-12:56:07
GoVersion	go1.19.7

Prometheus configuration file

-

Alertmanager version

-

Alertmanager configuration file

-

Logs

-

[beorn7]

There have been thoughts around this for a long time, see this exploratory document, some parts of which are already implemented, some are being worked on, and some are somewhat out in the distance, among them arithmetic on durations and on the argument of the @ modifier.

[NeerajGartia21]

Is it possible to currently work on this issue, or is it still somewhat in the distance? If possible I can have a look into it.

[beorn7]

I would say we shouldn't look at arithmetic for the @ modifier in isolation, but for all durations.

The challenge here is that we need in a way a new concept of "static" arithmetic, i.e. evaluations that do not touch the TSDB. Otherwise, it would be possible to create expressions that require multiple subsequent TSDB lookups (which is something that we might want to support in the future, but not in this first step).

So for example, it should be possible to write
rate(foo_total[1m + 2m] offset (1d + 1h)) but not rate(foo_total[interval_set_in_recording_rule])

This might be "not as easy as it looks", but if you would like to give it a spin, it would certainly be very welcome.

[beorn7]

It will touch the parser, for sure, but it might also require quite deep changes in the PromQL engine (which I don't know well enough to tell).

[NeerajGartia21]

I would say we shouldn't look at arithmetic for the @ modifier in isolation, but for all durations.

So to achieve this we need to replace all the Duration component with DurationArithmetic component right? And also I think It will be better and easy to start with this one after #13412 gets merged.

This might be "not as easy as it looks", but if you would like to give it a spin, it would certainly be very welcome.

Yeah, it looks complex but I'll try to look into it in my free time. I need to deep dive into the promql engine for this.

It will touch the parser, for sure, but it might also require quite deep changes in the PromQL engine

Yeah I think it will touch both parser as well as the engine and needs a deep understanding of promql engine I guess.

[schettgen]

I need exactly the same which is mentioned above.

In my example I need to get the first of the current month.
I would substract that with the current counter value and it would return the usage for this month.

Then I could create an alert on it to see if we run out out budget on some third party API