Additional Metrics for System.Net.Http

[antonfirsov]

Opening a separate issue for the HTTP part of #79371, also including requests from #79459. We should eventually implement the Metrics counterpart of all System.Net.Http EventCounters.

Implemented as part of #84978

• requests-failed, current-requests
• http(11|20|30)-connections-current-total

Note that Metrics variants of requests-failed, current-requests will be semantically different from EventCounters since we need to comply with the OTel specification which wants us to record the counters until "headers receieved" and exclude response buffering in case HttpCompletionOption.ResponseContentRead is passed. See the discussion in open-telemetry/opentelemetry-specification#3520.

Event counters not discussed in #84978

• requests-started
• requests-started-rate
• requests-failed-rate
• http(11|20|30)-requests-queue-duration

Additional requests from #79459; we don't have existing counters for these

• number of idle connections
• average idle connection time
• number of connections evicted from pool due to idle timeout
• number of connection timeouts
• average time to connect
• queue length
• number of http 2 keep alive pings which didn't receive a response
• average response time of http 2 keep alive pings

Regarding tags, there is a wish in #79459:

Ideally these metrics would be available per FQDN:Port combination, broken down by type (i.e. HTTP1.1/HTTP2) and there would be some way to track which http client each pool belongs to.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Opening a separate issue for the HTTP part of #79371, also including requests from #79459. We should eventually implement the Metrics counterpart of all System.Net.Http EventCounters.

The following counters already have their Metrics counterparts specified and tracked by #84978

• requests-failed, current-requests
• http11-connections-current-total, http20-connections-current-total, http30-connections-current-total
• http11-requests-queue-duration, http20-requests-queue-duration, http30-requests-queue-duration

Event counters not discussed in #84978

• requests-started
• requests-started-rate
• requests-failed-rate

Additional requests from #79459, not listed above

• number of idle connections
• average idle connection time
• number of connections evicted from pool due to idle timeout
• number of connection timeouts
• average time to connect
• queue length
• number of http 2 keep alive pings which didn't receive a response
• average response time of http 2 keep alive pings

Regarding tags, there is a wish in #79459:

Ideally these metrics would be available per FQDN:Port combination, broken down by type (i.e. HTTP1.1/HTTP2) and there would be some way to track which http client each pool belongs to.

Author:	antonfirsov
Assignees:	-
Labels:	area-System.Net.Http
Milestone:	-

[antonfirsov]

@davidfowl all connection instruments from #84978 have been added now. I mistakenly though that http(11|20|30)-requests-queue-duration is also tracked there which is not the case, and it fits the definition of the connection pooling data that exists inside of SocketHttpHandler's guts from #48885 (comment). Do you find this critical for 8.0?

[davidfowl]

Yes, that's very important to add. One of the most important counters when looking at slowness of requests.

[MihaZupan]

requests-queue-duration has been added in #88981