Fix issue where applying httpgrpc tracing middleware results in all HTTP requests having duplicate trace spans

[charleskorn]

What this PR does:

This PR fixes the issue where applications that use the httpgrpc tracing middleware (through middleware.InitHTTPGRPCMiddleware) would have traces for all incoming HTTP calls duplicated.

For example, note the duplicate HTTP POST - api_v1_push spans in this trace from a Mimir distributor:

The fix changes the HTTP middleware to a gRPC interceptor that is then able to accurately filter incoming requests and apply the additional information only to httpgrpc requests.

One downside of this is that we'll convert the request to a http.Request twice now: once in the interceptor, and again in the httpgrpc handler, but this seems like the neatest solution.

Related: #353

Which issue(s) this PR fixes:

(none)

Checklist

• Tests updated
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[francoposa]

LGTM!
If you were feeling particularly enterprising, you could figure out how to use the an OTEL tracer in the tests instead of this jaeger one.
The jaeger tracer is the one dskit used to use for opentracing before it was converted to OTEL. It stuck around in this test and maybe others but isn't actually used in the application anymore

[charleskorn]

If you were feeling particularly enterprising, you could figure out how to use the an OTEL tracer in the tests instead of this jaeger one.
The jaeger tracer is the one dskit used to use for opentracing before it was converted to OTEL. It stuck around in this test and maybe others but isn't actually used in the application anymore

Sounds good, I'll take a look at this in a separate PR.