Last value in graph is wrong (query_range/maxStepForPointsAdjustment)

[mxsiegle]

Hey,

I noticed that all of our graphs in Grafana have a drop in their value for the last datapoint. It looks like this:

This only affects range queries (/api/query_range); instant values are calculated correctly. The actual metric values are correct as well, i.e. if you wait a moment until new data arrives, the datapoint which had issues suddenly reports the correct value while the new one is wrong. You can repeat that indefinitely, it's always the latest datapoint that is wrong.

While troubleshooting, I made the following observations:

Sometimes the values are reported correctly, other times they are not. This is how it looks like when I refresh the table a couple of times. Note the last entry on the bottom right:

When I run VMSelect with -search.disableCache the wrong value appears 100% of the time. So somehow the cache affects the value and sometimes reports it correctly.

The issue only appears when a time range is selected with a step value higher than the max. adjusted time set by -search.maxStepForPointsAdjustment.
The default value is 1m0s. When I selected 'Last 6 hours' my step value in Grafana becomes 20s and the correct value is reported:

When I change it to 'Last 7 days' it has 5m steps and then the issue is there:

If I set -search.maxStepForPointsAdjustment to e.g. 6m, the issue will disappear for 'Last 7 days' but it will re-appear of course as soon as the step value is bigger than that (e.g. 'Last 30 days' -> 30m steps).
Can anyone tell me why this is happening and how I can fix it?

Setup: Cluster with 4x VMStorage, 4x VMSelect and 3x VMInsert.
Version: v1.62.0
Retention: 90d
Scrape interval: 20s (VMAgent)

It does not matter which metric I choose or which target, all of them have that issue. Weirdly, recordings are totally fine. I assume it's because they are using the (correct) instant values but for some reason their last values are reported correctly in the graphs/tables.

I only noticed this behavior recently. Before, we had updated to v1.62.0 so I thought maybe it's that, but a rollback to v1.59.0 and v1.55.1 didn't make any difference (those were the versions we were running before).

Thanks in advance.

[valyala]

This may be related to #1526

[valyala]

FYI, VictoriaMetrics and vmagent gained support for Prometheus staleness markers starting from the release v1.64.0.

[hagen1778]

Closing the issue as inactive. Feel free to reopen if the problem still exists.