[release/6.0] Add additional trace to HttpConnectionPool

[CarnaViire]

Backport of #66605 to release/6.0

Customer Impact

.NET 6 introduced changes to HttpConnectionPool that decouples HTTP connection creation from a request that initiated it. This made issues related to connection creation harder to troubleshoot.
We had recently 2 investigations which took unnecessarily long to root cause due to suboptimal logging. As a result, we want to add additional logging to make such investigations in future easier for us (Networking team) and for the customers (self-diagnostics).

For example on of the investigations was about hanged "pending" HTTP/1.1 connection issues (i.e. the reasons why new connections were not created) in HttpConnectionPool. The issue manifests as growing number of timeouted requests. It is not easy to link a timeouted request (esp. with a small timeout) to a connection that was created 3 minutes ago but still is not connected (Linux default connection timeout is more than 3 minutes). The trace of connection being initiated may even not be captured, if tracing was turned on when the visible trouble occur. There were no traces of the state of HttpConnectionPool, e.g. number of pending connections, so it is unclear from existing traces why the pool decided not to create a connection.

The additional trace should help troubleshoot why new connection was not created and why the pending connection failed.

Testing

Presence of new traces verified by customer deploying private bits and by a test app.

Risk

Low. This is private-telemetry-only (Private.InternalDiagnostics.System.Net.Http) change.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Backport of #66605 to release/6.0

Customer Impact

.NET 6 introduced changes to HttpConnectionPool that decouples HTTP connection creation from a request that initiated it. This made the issues related to connection creation harder to troubleshoot.

For example, hanged "pending" HTTP/1.1 connection issues (i.e. the reasons why new connections were not created) in HttpConnectionPool. The issue manifests as growing number of timeouted requests. It is not easy to link a timeouted request (esp. with a small timeout) to a connection that was created 3 minutes ago but still is not connected (Linux default connection timeout is more than 3 minutes). The trace of connection being initiated may even not be captured, if tracing was turned on when the visible trouble occur. There were no traces of the state of HttpConnectionPool, e.g. number of pending connections, so it is unclear from existing traces why the pool decided not to create a connection.

The additional trace should help troubleshoot why new connection was not created and why the pending connection failed.

Testing

Presence of new traces verified by customer deploying private bits and by a test app.

Risk

Low. This is private-telemetry-only (Private.InternalDiagnostics.System.Net.Http) change.

Author:	CarnaViire
Assignees:	CarnaViire
Labels:	area-System.Net.Http
Milestone:	-

[CarnaViire]

All CI failures are not related to the change.

• runtime-libraries enterprise-linux failure is "runtime-libraries enterprise-linux" jobs failing due to network issue #65621
• runtime (Build windows x64 Release SingleFile), runtime (Libraries Test Run release coreclr windows x64 Debug), runtime (Libraries Test Run release coreclr windows x86 Release) failures are due to Windows.10.Amd64.Server19H1.ES.Open reaching EOL Backport Move to Windows.10.Amd64.Server2022.ES.Open to 6.0 #67048
• runtime-staging failures are due to runtime-staging (Build Browser wasm release Browser_wasm_Windows) timing out due to an infra issue Infra issue (deadletter) in CI runtime-staging (Build Browser wasm release Browser_wasm_Windows) in 6.0 #67058