http2 performance issue (2x slower than http1.1 and vs h2load)

[huntharo]

Overview

I was testing the download speed characteristics from Japan (AWS CloudShell in ap-northeast-1) to Fastly's LAX PoP using both HTTP1.1 and HTTP2 using oha (this is about a 100 ms RTT).

I observed that the steady state response time for a 2.5 MB file (non-compressible) stabilized as follows:

Metric	HTTP/2	HTTP/1.1
Average Time per Download	~250 ms	~120 ms
Total Time for 100 Downloads	~25 seconds	~13 seconds

I reported this issue to Fastly and they asked that I retest with h2load from nghttp2's reference implementation.

Steps to reproduce are below.

At first I thought maybe h2load had a reporting issue as it is more difficult to tell the steady state response times (the initial response takes 1.2 seconds due to their slow start and long RTT) given that they only report mean time and not median. But running the commands with time showed the overall process for oha taking 2x longer as well, which matched up.

Commands for Testing

# oha - http1.1
time ./oha-linux-amd64 -c 1 -n 100 https://media2.giphy.com/media/pE4zqXvpwpaUh67tkw/giphy.gif

# oha - http2
time ./oha-linux-amd64 -c 1 -p 1 -n 100 --http2 https://media2.giphy.com/media/pE4zqXvpwpaUh67tkw/giphy.gif

# h2load - http1.1
time h2load -c 1 -n 100 --h1 https://media2.giphy.com/media/pE4zqXvpwpaUh67tkw/giphy.gif

# h2load - http2
time h2load -c 1 -n 100 -m 1 https://media2.giphy.com/media/pE4zqXvpwpaUh67tkw/giphy.gif

Results

✅ oha - http/1.1

❌ oha - http2

✅ h2load - http/1.1

✅ h2load - http2

Building h2load on AWS CloudShell

# Install deps
sudo yum install -y gcc g++ make zlib-devel openssl-devel c-ares-devel libev-devel

# Download the latest stable release
wget https://github.com/nghttp2/nghttp2/releases/download/v1.55.1/nghttp2-1.55.1.tar.gz

# Extract the tarball
tar xvf nghttp2-1.55.1.tar.gz

# Change to the extracted directory
cd nghttp2-1.55.1

# Configure, compile, and install nghttp2
./configure
make
sudo make install

# Testing - Should report an error
h2load

[reneleonhardt]

Wow, awesome analysis, thank you! 🚀

[hatoo]

Thank you for reporting it.

I've been able to reproduce it.
tested at AWS CloudShell in ap-northeast-1

$ time h2load -c 1 -n 100 -m 1 https://media2.giphy.com/media/pE4zqXvpwpaUh67tkw/giphy.gif
starting benchmark...
spawning thread #0: 1 total client(s). 100 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES128-GCM-SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 15.74s, 6.35 req/s, 15.59MB/s
requests: 100 total, 100 started, 100 done, 100 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 245.37MB (257291904) total, 7.97KB (8159) headers (space savings 82.86%), 245.23MB (257140500) data
                     min         max         mean         sd        +/- sd
time for request:   138.92ms    900.98ms    153.99ms     76.99ms    98.00%
time for connect:   337.22ms    337.22ms    337.22ms         0us   100.00%
time to 1st byte:   451.57ms    451.57ms    451.57ms         0us   100.00%
req/s           :       6.35        6.35        6.35        0.00   100.00%

real    0m15.755s
user    0m0.318s
sys     0m0.198s

$ time ./oha-linux-amd64 -c 1 -p 1 -n 100 --http2 https://media2.giphy.com/media/pE4zqXvpwpaUh67tkw/giphy.gif                                                                                                                                                                                                                                               
Summary:
  Success rate: 100.00%
  Total:        25.3859 secs
  Slowest:      0.8901 secs
  Fastest:      0.2342 secs
  Average:      0.2505 secs
  Requests/sec: 3.9392

  Total data:   245.23 MiB
  Size/request: 2.45 MiB
  Size/sec:     9.66 MiB

Response time histogram:
  0.234 [1]  |
  0.300 [98] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.365 [0]  |
  0.431 [0]  |
  0.497 [0]  |
  0.562 [0]  |
  0.628 [0]  |
  0.693 [0]  |
  0.759 [0]  |
  0.825 [0]  |
  0.890 [1]  |

Response time distribution:
  10.00% in 0.2351 secs
  25.00% in 0.2396 secs
  50.00% in 0.2456 secs
  75.00% in 0.2461 secs
  90.00% in 0.2485 secs
  95.00% in 0.2543 secs
  99.00% in 0.8901 secs
  99.90% in 0.8901 secs
  99.99% in 0.8901 secs


Details (average, fastest, slowest):
  DNS+dialup:   0.0000 secs, 0.0000 secs, 0.0000 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0000 secs

Status code distribution:
  [200] 100 responses

real    0m25.459s
user    0m1.256s
sys     0m0.425s

And I've created a simple load generator to find whether the difference is came from oha or hyper which is backend of oha.
https://github.com/hatoo/oha-538

$ time ./oha-538 https://media2.giphy.com/media/pE4zqXvpwpaUh67tkw/giphy.gif
....................................................................................................
real    0m24.733s
user    0m0.664s
sys     0m0.642s

I guess the difference came from hyper.
I'll investigate further.

[hatoo]

#540 solves on my side.

Could you check on your side?

[huntharo]

Oh cool will try

[huntharo]

Hmm... it's kind of tricky to do given that I'm on a Mac and CloudShell doesn't seem to have enough space to setup the Rust toolchain. I could do it a thousand other way but it would be so nice if the linux binary with that fix was available?

[hatoo]

Ok
Binary here

cargo build --release --target x86_64-unknown-linux-musl

oha.zip

[huntharo]

@hatoo On main builds (and even on branches) - What if we saved the built binaries as artifacts that we could download them when we need to test a specific version or pre-release. Thoughts?

[huntharo]

#540 solves on my side.

Could you check on your side?

@hatoo did you check the performance vs h2load with the fix? What I see is:

• oha before fix: 25 seconds for 100 copies of file
• oha with fix: 18 seconds
• h2load: 14.5 seconds

The fix has closed 64% of the gap with h2load, which is GREAT!

But there is something else it seems? I wonder if Rustls is the remaining slowness or if we have to change any other parameters on hyper?

[hatoo]

In my observation, #540 constantly did 100 downloads by around 15.5s which is about the same as h2load
But adaptive_window may work vary on runtime (although I don't know what adaptive_window means).

I've found that h2loads's default window size setting is larger than hyper's default
I've added on #541. It takes time the same as h2load in my tests.

Could you check it? Binary is attached in the PR.

@hatoo On main builds (and even on branches) - What if we saved the built binaries as artifacts that we could download them when we need to test a specific version or pre-release. Thoughts?

It looks good but I don't know how to do it without spamming https://github.com/hatoo/oha/releases page