HTTPoGRPC masking errors

[tomwilkie]

As long as all the IO works, all HTTP responses will be returned as success. We should ensure we return HTTP 500s as error.

[bboreham]

Do you remember why 500s were singled out? Why should a 400 be a success?

[tomwilkie]

For alerting/SLO purposes, 400's aren't normally considered errors (there problems with the request, not the serve). So the idea behind this change was that we could alert on gRPC-level errors if 500s are considered errors, but not if 400s are considered errors. Does that make sense?

[bboreham]

OK, thanks for the explanation.

In #40 code was added to extract the underlying status code and report that in the metric, which makes it possible to filter the errors at the time SLIs are computed. Filtering must already be necessary for http (not-over-grpc) errors which are not masked.

I find it confusing to see (say) Mimir Distributor reporting errors and not its caller; it makes it look like the problem is in the Distributor.