Store gateway spends most of the time waiting to load series and chun…

[duricanikolic]

What this PR does

Store gateway spends most of the time waiting to load series and chunks. During that waiting time, a context.Canceled error might occur. The latter must not be handled as a consistency check, but should be propagated as context.Canceled.

Which issue(s) this PR fixes or relates to

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[CLAassistant]

All committers have signed the CLA.

[colega]

Awesome, thanks! (I'd recommend reviewing with whitespace changes ignored).

[dimitarvdimitrov]

LGTM, nicely done!

[dimitarvdimitrov]

LGTM, thanks for addressing my comments

[callumj]

@duricanikolic I've noticed that this consistency check can also show up when a Ruler runs an expensive rule against the SGW that results in an response from the SGW with rpc error: code = DeadlineExceeded desc = context deadline exceeded - should that also be considered as something that doesn't surface as a consistency check failure?

Conversation: https://grafana.slack.com/archives/C039863E8P7/p1675357321799079

[duricanikolic]

@duricanikolic I've noticed that this consistency check can also show up when a Ruler runs an expensive rule against the SGW that results in an response from the SGW with rpc error: code = DeadlineExceeded desc = context deadline exceeded - should that also be considered as something that doesn't surface as a consistency check failure?

Conversation: https://grafana.slack.com/archives/C039863E8P7/p1675357321799079

@pracucci, @56quarters What do you think about this?

[pracucci]

It should have been fixed in #4007. @callumj does it happen in Mimir 2.6 too?