HttpClient contention due to shared CTS registrations

[MihaZupan]

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/httpclient.benchmarks.yml --scenario httpclient-kestrel-get --profile aspnet-gold-lin --server.framework net9.0 --client.framework net9.0 --variable useHttpMessageInvoker=true --variable concurrencyPerHttpClient=1 --variable numberOfHttpClients=256 --json 256x1.json
crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/httpclient.benchmarks.yml --scenario httpclient-kestrel-get --profile aspnet-gold-lin --server.framework net9.0 --client.framework net9.0 --variable useHttpMessageInvoker=true --variable concurrencyPerHttpClient=256 --variable numberOfHttpClients=1 --json 1x256.json

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/httpclient.benchmarks.yml --scenario httpclient-kestrel-get --profile aspnet-gold-lin --server.framework net9.0 --client.framework net9.0 --variable concurrencyPerHttpClient=1 --variable numberOfHttpClients=256 --json 256x1-client.json
crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/httpclient.benchmarks.yml --scenario httpclient-kestrel-get --profile aspnet-gold-lin --server.framework net9.0 --client.framework net9.0 --variable concurrencyPerHttpClient=256 --variable numberOfHttpClients=1 --json 1x256-client.json

At high RPS, the contention behind using a single CancellationTokenSource to handle CancelPendingRequests shows up:

client	256 clients	1 client
Invoker	1,389,074	1,363,790
HttpClient	1,326,386	754,139

A quick test of splitting requests across ProcessorCount CTS instances based on CurrentManagedThreadId recovers the whole perf difference.

When we get around to improving this, we could also consider avoiding the linked CTS allocation we incur per request.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.