avg_over_time for multiple time series fails at durations over 3 hours

[dandv]

Describe the bug

avg_over_time seems to call internally sum_over_time, then divide that by count_over_time. This works, but apparently not if the duration is longer than 2 hours, even though the series selected have one data point each, inserted within the last minute.

To Reproduce

rm -rf vicmetest && ./victoria-metrics-prod-1.96.0 -storageDataPath ./vicmetest -search.latencyOffset=1s

curl http://localhost:8428/api/v1/import/prometheus -d 'series1{label="foo"} 10
series2{label="foo"} 20'

# wait a second then confirm 2 series are returned
curl http://localhost:8428/api/v1/query -d 'query=count_over_time({__name__=~"series.*"}) keep_metric_names' | jq

# sum and count work, so avg should work, right?
curl http://localhost:8428/api/v1/query -d 'query=sum_over_time({__name__=~"series.*"}) keep_metric_names' | jq

# right!
curl http://localhost:8428/api/v1/query -d 'query=avg_over_time({__name__=~"series.*"}) keep_metric_names' | jq

# still works
curl http://localhost:8428/api/v1/query -d 'query=avg_over_time({__name__=~"series.*"}[2h]) keep_metric_names' | jq

# 💥 , 3h too much
curl http://localhost:8428/api/v1/query -d 'query=avg_over_time({__name__=~"series.*"}[3h]) keep_metric_names' | jq

If the duration is 3+ hours, the following error is returned:

{
  "status": "error",
  "errorType": "422",
  "error": "cannot evaluate \"(sum_over_time({__name__=~\\\"series.*\\\"}[3h]) keep_metric_names / (count_over_time({__name__=~\\\"series.*\\\"}[3h]) keep_metric_names)) keep_metric_names\": duplicate time series on the left side of / (): {label=\"foo\"} and {label=\"foo\"}"
}

Version

victoria-metrics-20231212-224437-tags-v1.96.0-0-g304fe0565

Logs

No warnings or errors.

Used command-line flags

-search.latencyOffset=1s but the bug is reproduced without it as well.

[hagen1778]

Thanks for reporting @dandv !

Looks like this problem is caused by rollup cache enhancement for instant queries intorduced in 1.95. As a workaround, disable cache for queries by passing &nocache=1 GET param to your queries.

@zekker6 may I ask you to take a look at this issue? It seems to be like relevant to another issue you was involved is about preserving metric names for left and right parts of binary operators.

[valyala]

This looks like a duplicate for #5659 and #5730 . These issues were addressed in v1.97.0 release. @dandv , could you verify whether the issue is fixed in v1.97.0?

[valyala]

@dandv , could you build VictoriaMetrics from the commit ec0ca8e according to these docs and verify whether it fixes the issue?

[valyala]

The bugfix for this issue has been included in v1.97.1 release. Closing the issue as fixed then.

@dandv , please re-open this issue if it still persists in v1.97.1.