hyper: Missing/incorrect pseudo-headers in HTTP/2 requests

[divergentdave]

I did this

I built curl with hyper enabled, and tried to make a request to a Google web server (i.e. https://www.google.com/). The request failed with curl: (56) Hyper: [1] error reading a body from connection: protocol error: unspecific protocol error detected.

I expected the following

I expected the request to be accepted by the server.

After some experimentation (see here hyperium/h2#548 (comment)) I determined that the server rejects HTTP/2 requests unless :scheme is correctly set to https, :authority is set, and no Host header is present.

Currently, when curl creates a request using hyper, it passes in just the path as the URI, (i.e., /) and passes a Host header in with the list of headers. Inside hyper, this pseudo-URI gets parsed, a default scheme of http is assumed, and the eventual request on the wire has :scheme: http, :path: /, host: www.google.com, and no :authority pseudo-header.

It seems many servers and reverse proxies are permissive about Host headers in HTTP/2, but Google's GFE seems to insist not only on the presence of :authority, but the absence of Host.

To fix this, c-hyper.c will need to add special handling for the Host header, similar to how the nghttp code works now. I have proposed adding a new FFI API to hyper that will make addressing this easier by passing the URI's scheme, authority and path/query as separate C strings, see here hyperium/hyper#2581.

curl/libcurl version

curl 7.79.0-DEV (x86_64-pc-linux-gnu) libcurl/7.79.0-DEV OpenSSL/1.1.1d zlib/1.2.11 zstd/1.3.8 libidn2/2.0.5 nghttp2/1.36.0 Hyper/0.14.11 libgsasl/1.8.0 OpenLDAP/2.4.47
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS Debug gsasl HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP TrackMemory UnixSockets zstd

operating system

Linux desktop 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64 GNU/Linux

[bagder]

I haven't done more than very basic h2 testing with hyper so far. Clearly more work to do there.

[bagder]

The fact that we don't even know what pseudo-headers are sent is an issue. I'd like to be able to find out so that we can show them - at least when built debug-enabled like we do with nghttp2.