Query timeout with larger offsets

[jordanmat]

Describe the bug
When running query with count_over_time range [24h] , using offset higher than 48h my queries timeout.

To Reproduce

1. Run query

sum by(container) (count_over_time({cluster="$Env", level="ERROR"} [24h] offset 72h))

this returns

Status: 504. Message: Get (http://...) context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Expected behavior
The same query with an offsets up to 48h works fine, implying it's the amount of data that causes the timeout. Since my daily logs are roughly the same, querying each 24h period should not timeout. I assumed that using offset modifier will look at data from now() - offset and older, but seems like it's querying all the data?

Environment:

• Infrastructure: Kubernetes
• Loki: v2.9.4
• Grafana: v10.0.0 (81d85ce802)