Http2 observability and understanding connection performance in hyper

[brent-statsig]

We've been using hyper for a custom proxy, and are starting to run into some bottlenecks. We've been trying to tune the available connection parameters for things like window size, stream concurrency, etc, but it feels like we are shooting in the dark a bit trying to understand where the bottleneck is occuring. Average instance is serving ~5-10k QPS, and ~100-500 MBps. We are using tokio as our async runtime.

• We have latency metrics that measures the time elapsed from the start to the end of our service_fn - but that doesn't actually give us insight into the time taken to serialize + flush bytes through the connection. At P99 - we are seeing about a 10x difference between our service level metrics and load balancer metrics. Is there a hook or non-dirty way to get the true observed server e2e time for a request?
• Are there any metrics, or an easy way to instrument our connections, to understand possible congestion in streams/connections? E.g. bytes pending write, number of concurrent streams, number of connections.

[0x676e67]

The bottleneck is here. You need to create an http2 instance for each core runtime of tokio to fully utilize the multi-core performance. Just like actix-rt

hyperium/h2#531