curl: (55) Failed sending HTTP request when trying to send a request with large header size. >8.1.2

[dimaboyko]

Hi, I am not certain this is a bug, perhaps this was mentioned somewhere, however, I did not discover it, looking at the changelog.
After updating curl from 8.0.1 I have started seeing requests failing to be sent. Further investigation showed, that this is only happening for requests with large header size (10K characters)

I did this

Send a request with large header size. Example:
curl -X GET -L -H "MyHeader: $(printf '%*s' 10000 | tr ' ' 'A')" https://postman-echo.com/get
This is working ok on 8.0.1, but not on 8.1.2.
8.1.2 is returning curl: (55) Failed sending HTTP request

I expected the following

Response from the server.

curl/libcurl version

Error happening with:

curl 8.1.2 (aarch64-alpine-linux-musl) libcurl/8.1.2 OpenSSL/1.1.1u zlib/1.2.12 brotli/1.0.9 nghttp2/1.47.0
Release-Date: 2023-05-30
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe TLS-SRP UnixSockets

Request is sent successfully with:

curl 8.0.1 (aarch64-alpine-linux-musl) libcurl/8.0.1 OpenSSL/1.1.1u zlib/1.2.12 brotli/1.0.9 nghttp2/1.43.0
Release-Date: 2023-03-20
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe TLS-SRP UnixSockets

operating system

Alpine Linux 3.16 Docker image.
Linux 5.15.49-linuxkit

Screenshot

[dfandrich]

Could this be related to #10720 which changed internal header handling in 8.0.2? Note that the -X option in that command should be removed.

[icing]

That seems likely. I assume you see different behaviour when adding --http1.1?

[dimaboyko]

@icing yes, adding that is fixing the issue.
As visible on the screenshot, nghttp2 is different in those versions. So it's only happening with HTTP2.

[icing]

We introduced a limit when we refactored the http2 code. Most servers I know have a limit on header length to protect themselves from malicious input.

Is this an experiment you are running or do you have a use case for such headers? If yes, what is the limit you are looking for?

[dimaboyko]

I have discovered this, as libcurl is a downstream dependency of a library that I use in (ethon)
It is a real life usecase, had to adjust what is being sent with the header. The limit I keep in mind always is one from aws alb - https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-limits.html
That would be a little over 15k characters.
I understand this might be quite the edge case, just wanted to bring this to attention, also in case somebody starts getting the error after updating from 8.0.1.

[dfandrich]

A number of popular web servers support up to 16KB headers (a few support even longer), so 16KB is probably a reasonable limit (if libcurl needs to limit them at all).

If it helps, below is result with Go net/http. I would say that the given server is a poor example, as it doesnt even handle the request as given:

https://developer.mozilla.org/docs/Web/HTTP/Status/431

package main

import (
   "net/http"
   "strings"
   "time"
)

func main() {
   req, err := http.NewRequest("", "https://postman-echo.com/get", nil)
   if err != nil {
      panic(err)
   }
   i := 7989
   for {
      req.Header.Set("MyHeader", strings.Repeat("A", i))
      res, err := new(http.Transport).RoundTrip(req)
      if err != nil {
         panic(err)
      }
      res.Body.Close()
      println(res.Status, i)
      time.Sleep(time.Second)
      i++
   }
}

result:

200 OK 7989
502 Bad Gateway 7990
502 Bad Gateway 7991
502 Bad Gateway 7992
502 Bad Gateway 7993
502 Bad Gateway 7994
502 Bad Gateway 7995
502 Bad Gateway 7996
502 Bad Gateway 7997
502 Bad Gateway 7998
502 Bad Gateway 7999
502 Bad Gateway 8000
502 Bad Gateway 8001
502 Bad Gateway 8002
502 Bad Gateway 8003
502 Bad Gateway 8004
502 Bad Gateway 8005
502 Bad Gateway 8006
502 Bad Gateway 8007
502 Bad Gateway 8008
502 Bad Gateway 8009
502 Bad Gateway 8010
502 Bad Gateway 8011
502 Bad Gateway 8012
431 Request Header Fields Too Large 8013

[icing]

Thanks. I made #11407 just now which overcomes these limitations.

Curl internally has a general buffer limited to 1MB for assembling a HTTP request. This limit is now also in effect for HTTP/2 and HTTP/3 request processing with this PR.

[dimaboyko]

Thank you!