Do not fail as consistency check for canceled requests

[colega]

What this PR does

Stops ignoring the context.Canceled error from store gateway interaction: this made the entire request fail as "consistency check" because that store gateway didn't have the block, isntead of actually failing as context.Canceled which isn't a server's fault (usually).

Which issue(s) this PR fixes or relates to

Fixes internal.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pstibrany]

Looks good. Thanks!

[pracucci]

Good job!

[dimitarvdimitrov]

Since this is a stream, would it also make sense to handle the cancellations when receiving from the stream as well?

mimir/pkg/querier/blocks_store_queryable.go

Line 741 in ddf6d54

resp, err := stream.Recv()

[pracucci]

Should be covered by the line above:

mimir/pkg/querier/blocks_store_queryable.go

Lines 735 to 739 in ddf6d54

	// Ensure the context hasn't been canceled in the meanwhile (eg. an error occurred
	// in another goroutine).
	if gCtx.Err() != nil {
	return gCtx.Err()
	}

[dimitarvdimitrov]

thinking about the store-gateway streaming implementation - most of the time in the sotre-gateway is spent waiting to load series and chunks. And much less sending to the querier. To me this means that most of the time in the querier will be spent waiting for a store-gateway response. So the context cancellation is more likely to occur while the querier is waiting on a response from the store-gateway.

unless i'm missing something i can open a PR for this

[pracucci]

So the context cancellation is more likely to occur while the querier is waiting on a response from the store-gateway.

True, but you will still catch it right after the next message is received from gRPC.

[dimitarvdimitrov]

i don't understand. I thought srv.Recv() will return a contect.Cancelled error here and we will just swallow it

mimir/pkg/querier/blocks_store_queryable.go

Lines 741 to 748 in 6fca816

	resp, err := stream.Recv()
	if errors.Is(err, io.EOF) {
	break
	}
	if err != nil {
	level.Warn(spanLog).Log("msg", "failed to receive series", "remote", c.RemoteAddress(), "err", err)
	return nil
	}

[pracucci]

Now I see what you mean and you're right. I misunderstood your previous message. We need to fix it and happy if you can work on it, as you proposed above.

[colega]

Yes, you're definitely right @dimitarvdimitrov, good catch.