HTTP/3 file uploads is significantly slower comparing to HTTP/2 or an in house HTTP/3 client

[alexsn]

I did this

I'm settings to a http3 transfer by using CURLOPT_HTTP_VERSION and capturing request stats, as can be seen below the H3 request send takes a lot longer to complete.

I expected the following

I expect the file upload using H3 to be at least as fast as H2, currently it's about 3 - 4 times slower:

Request stats

// Request stats using H2, notice the delta between last_byte_flushed_ms and first_byte_flushed_ms which is 314ms
{
  "dns_resolution_time_ms": 4,
  "request_queue_time_ms": 0,
  "request_header_size": 700,
  "time_to_last_byte_ms": 2718,
  "time_to_first_byte_ms": 2718,
  "total_request_time_ms": 2718,
  "status_code": 200,
  "request_body_size": 1022505,
  "last_byte_flushed_ms": 497,
  "first_byte_flushed_ms": 183,
  "connect_time_ms": 58,
  "response_body_size": 159,
  "http_method": "POST",
  "new_connection": true,
  "server_address": "2a03:2880:f042:112:face:b00c:0:2",
  "handshake_time_ms": 120,
  "response_header_size": 145,
  "http_version": "2.0"
}

// Same request using H3, notice the time it takes to send the request is 1028ms
{
  "dns_resolution_time_ms": 4,
  "request_queue_time_ms": 0,
  "request_header_size": 697,
  "time_to_last_byte_ms": 3102,
  "time_to_first_byte_ms": 3027,
  "total_request_time_ms": 3102,
  "status_code": 200,
  "request_body_size": 1022505,
  "last_byte_flushed_ms": 1112,
  "first_byte_flushed_ms": 84,
  "connect_time_ms": 17,
  "response_body_size": 159,
  "host_name": "rupload.facebook.com",
  "http_method": "POST",
  "new_connection": true,
  "server_address": "2a03:2880:f042:112:face:b00c:0:2",
  "handshake_time_ms": 60,
  "response_header_size": 162,
  "http_version": "3.0"
}

Analysis:

I've taken packet capture from libcurl transfer and it looks like libcurl sends 64k of data then blocks until it all frames are acked which suspends the upload for about 40ms-50ms on every 64k chunk

When comparing this to a in house HTTP/3 client there are no hangs and the upload continues smoothly throughout the transfer

Checking curl_ngtcp2.c it looks like there's a buffering issue where the obvious suspect is sendbuf_len_in_flight which is where the 64k part comes from and there are multiple locations waiting for this value to drop to zero (like here and here) before resuming.

I've tried setting CURLOPT_UPLOAD_BUFFERSIZE to a larger value (512kb) which does help (makes it about 2x faster) though it's still about 2x slower then H2.

curl/libcurl version

libcurl 8.8.0 with wolfssl / ngtcp2 / nghttp3 and nghttp2.

operating system

Linux 5.12.0-0 #1 SMP Mon Jun 26 17:52:47 PDT 2023 x86_64 x86_64 x86_64 GNU/Linux

[dfandrich]

Stefan did a talk about performance in curl at curl://up 2024 that you might find interesting: https://www.youtube.com/watch?v=4eBAzi7D_WM

[icing]

@alexsn nice find. You are probably right with your analysis that the flow window is only restarted once everything is acked. I'll have a look at how we can make this more "flowing".

[icing]

Added scorecard upload tests in #14208. These show the abysmal performance on HTTP/3 PUTs.

[alexsn]

Added scorecard upload tests in #14208. These show the abysmal performance on HTTP/3 PUTs.

Can you test with #14209 applied?