HTTP/2 (h2c) upgrade in clear-text problem #7633

tk-xiong · 2021-08-26T03:04:11Z

I did this

I use curl and Golang HttpClient to call my service.
Golang's HttpClient is working fine.

curl:
This does not work if --http2 is used.
This works if use --http2-prior-knowledge.

I expected the following

both use --http2 and --http2-prior-knowledge working fine.

curl/libcurl version

Windows:
curl 7.78.0 (x86_64-pc-win32) libcurl/7.78.0 OpenSSL/1.1.1l (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.5.0 libidn2/2.3.2 libssh2/1.9.0 nghttp2/1.44.0 libgsasl/1.10.0
Release-Date: 2021-07-21
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli gsasl HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP UnixSockets zstd

Linux:
curl 7.78.0 (x86_64-pc-linux-gnu) libcurl/7.78.0 zlib/1.2.7 nghttp2/1.44.0
Release-Date: 2021-07-21
Protocols: dict file ftp gopher http imap mqtt pop3 rtsp smtp telnet tftp
Features: alt-svc AsynchDNS HTTP2 IPv6 Largefile libz UnixSockets

[curl -V output]

*   Trying 192.168.0.11:10001...
* Connected to 192.168.0.11 (192.168.0.11) port 10001 (#0)
> Get /hello/ HTTP/1.1
> Host: 192.168.0.11:10001
> User-Agent: curl/7.78.0
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 101 Switching Protocols
HTTP/1.1 101 Switching Protocols
< Connection: Upgrade
Connection: Upgrade
< Upgrade: h2c
Upgrade: h2c
* Received 101
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

operating system

Linux 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

tk-xiong · 2021-08-26T04:15:37Z

Upon receiving the 101 response, the client MUST send a connection
preface (Section 3.5), which includes a SETTINGS frame.

tk-xiong · 2021-08-26T04:19:47Z

Use Wireshark，i can't find connection preface

tk-xiong · 2021-08-26T04:39:03Z

and read golang src, golang.org/x/net/http2/h2c/h2c.go line 192.

	// A conforming client will now send an H2 client preface which need to drain
	// since we already sent this.
	if err := drainClientPreface(rw); err != nil {
		return nil, err
	}

bagder · 2021-08-26T05:45:41Z

$ curl --http2 localhost -v
*   Trying ::1:80...
* Connected to localhost (::1) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.74.0
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 101 Switching Protocols
< Upgrade: h2c
< Connection: Upgrade
* Received 101
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200 
< date: Thu, 01 Jan 1970 00:00:00 GMT
< server: Apache/2.4.48 (Debian)
< last-modified: Sat, 31 May 2014 22:05:33 GMT
< etag: W/"d-4fab95c704b1c"
< accept-ranges: bytes
< content-length: 13
< content-type: text/html
< 
inken tinken
* Connection #0 to host localhost left intact

tk-xiong · 2021-08-26T06:49:27Z

Can you use some tools to capture the package into file and show it to me.

i use wireshark on windows and not find client send this -> " PRI *HTTP/2.0\r\n\r\nSM\r\n\r\n"

maybe nghttp2 is not Compatible with golang server.

tk-xiong · 2021-08-26T06:57:23Z

Please teach me whether the client should send a Connection preface message after receiving the Upgrade message.

i think client need, and read golang server, it also drainClientPreface.

but i use wireshark, i can't find package Client Preface(this -> " PRI *HTTP/2.0\r\n\r\nSM\r\n\r\n").

tk-xiong · 2021-08-26T07:22:08Z

Yes! I Found!
If annotation this code， it run：

golang.org/x/net/http2/h2c/h2c.go line 192 to line 196.

// A conforming client will now send an H2 client preface which need to drain
// since we already sent this.
if err := drainClientPreface(rw); err != nil {
return nil, err
}

tk-xiong · 2021-08-26T07:33:33Z

But I don't think it was Golang's fault.

In RFC7540 (Section 3.2):
Upon receiving the 101 response, the client MUST send a connection
preface (Section 3.5), which includes a SETTINGS frame.

I think Curl or NGHTTP2 does not follow the RFC specification.

bagder · 2021-08-26T07:49:29Z

I've failed to reproduce this problem myself, so can I ask you to provide us with more details on how to go ahead and repeat this. Preferably with a command line or a stand-alone program we can run from our ends against a public URL to trigger the problem?

tk-xiong · 2021-08-26T08:02:16Z

I Send e-mail to you. Please do not disclose this IP address, thank you.

bagder · 2021-08-26T08:17:04Z

You then assume that nobody else than me wants and can debug or help out here?

tk-xiong · 2021-08-26T08:25:05Z

I'm Sorry，My server is not secure, so I dare not expose the public network.
please try this url:
curl -v --http2 http://121.37.5.232:10000/hello/

bagder · 2021-08-26T14:46:43Z

Reverting 455a63c fixes this problem for me. Presumably it would however reopen #7036...

@starrify, any ideas?

tk-xiong · 2021-08-27T08:29:54Z

I try revert 455a63c
Maybe I think too little, I can't understand #7036

$ ./curl -vI --http2 http://192.168.0.11:10001/hello/
*   Trying 192.168.0.11:10001...
* Connected to 192.168.0.11 (192.168.0.11) port 10001 (#0)
> HEAD /hello/ HTTP/1.1
> Host: 192.168.0.11:10001
> User-Agent: curl/7.78.0
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 101 Switching Protocols
HTTP/1.1 101 Switching Protocols
< Connection: Upgrade
Connection: Upgrade
< Upgrade: h2c
Upgrade: h2c
* Received 101
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* stopped the pause stream!
* Connection #0 to host 192.168.0.11 left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

bagder · 2021-08-30T10:57:32Z

I can't repro that error. With the reverted commit I get this (debug build):

$ ./src/curl -v --http2 http://121.37.5.232:10000/hello/
* STATE: INIT => CONNECT handle 0x55e7b84e20a8; line 1780 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* family0 == v4, family1 == v6
*   Trying 121.37.5.232:10000...
* STATE: CONNECT => CONNECTING handle 0x55e7b84e20a8; line 1841 (connection #0)
* Connected to 121.37.5.232 (121.37.5.232) port 10000 (#0)
* STATE: CONNECTING => PROTOCONNECT handle 0x55e7b84e20a8; line 1971 (connection #0)
* STATE: PROTOCONNECT => DO handle 0x55e7b84e20a8; line 1994 (connection #0)
> GET /hello/ HTTP/1.1
> Host: 121.37.5.232:10000
> User-Agent: curl/7.79.0-DEV
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA
> 
* STATE: DO => DID handle 0x55e7b84e20a8; line 2068 (connection #0)
* STATE: DID => PERFORMING handle 0x55e7b84e20a8; line 2187 (connection #0)
* Mark bundle as not supporting multiuse
* HTTP/2 found, allow multiplexing
< HTTP/1.1 101 Switching Protocols
< Connection: Upgrade
< Upgrade: h2c
* Received 101
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* multi changed, check CONNECT_PEND queue!
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* multi changed, check CONNECT_PEND queue!
* HTTP/2 found, allow multiplexing
< HTTP/2 200 
< content-type: text/plain; charset=utf-8
< content-length: 6
< date: Mon, 30 Aug 2021 10:54:30 GMT
< 
* nread == 0, stream closed, bailing
* STATE: PERFORMING => DONE handle 0x55e7b84e20a8; line 2386 (connection #0)
* multi_done
* Connection #0 to host 121.37.5.232 left intact
* Expire cleared (transfer 0x55e7b84e20a8)

…witch" This reverts commit 455a63c. Reported-by: Tk Xiong Fixes #7633

tk-xiong · 2021-08-30T12:12:02Z

I revert 455a63c
It returns the correct results, but I still think there are some problems.

$ ./curl -v --http2 -g "121.37.5.232:10000/hello/"

Trying 121.37.5.232:10000...
Connected to 121.37.5.232 (121.37.5.232) port 10000 (#0)
GET /hello/ HTTP/1.1
Host: 121.37.5.232:10000
User-Agent: curl/7.78.0
Accept: /
Connection: Upgrade, HTTP2-Settings
Upgrade: h2c
HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA

Mark bundle as not supporting multiuse
< HTTP/1.1 101 Switching Protocols
< Connection: Upgrade
< Upgrade: h2c
Received 101
Using HTTP2, server supports multiplexing
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< content-type: text/plain; charset=utf-8
< content-length: 6
< date: Mon, 30 Aug 2021 12:04:46 GMT
<
Connection #0 to host 121.37.5.232 left intact
Hello.

$ ./curl -vI --http2 -g "121.37.5.232:10000/hello/"

Trying 121.37.5.232:10000...
Connected to 121.37.5.232 (121.37.5.232) port 10000 (#0)
HEAD /hello/ HTTP/1.1
Host: 121.37.5.232:10000
User-Agent: curl/7.78.0
Accept: /
Connection: Upgrade, HTTP2-Settings
Upgrade: h2c
HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA

Mark bundle as not supporting multiuse
< HTTP/1.1 101 Switching Protocols
HTTP/1.1 101 Switching Protocols
< Connection: Upgrade
Connection: Upgrade
< Upgrade: h2c
Upgrade: h2c

Received 101

Using HTTP2, server supports multiplexing

Connection state changed (HTTP/2 confirmed)

Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

Connection state changed (MAX_CONCURRENT_STREAMS == 250)!

HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

stopped the pause stream!

Connection #0 to host 121.37.5.232 left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
Does this need to be fixed?

bagder · 2021-08-30T12:12:43Z

Does this need to be fixed?

Probably, but as I've said: I cannot reproduce that.

tk-xiong · 2021-08-30T13:02:20Z

I Use git clone master branch and revert code.

git clone https://github.com/curl/curl.git
 
autoreconf -fi
 
# change configure file:
want_h2_path="$withval/lib/pkgconfig" to want_h2_path="$withval/lib64/pkgconfig"
 
# revert code.
... ... 
 
# build
./configure --prefix=/data/tkxiong/curl/release \
    --with-nghttp2=/data/tkxiong/nghttp2/release \
    --disable-shared \
    --without-ssl
 
make && make install

$ ./curl -vI -g --http2 "http://121.37.5.232:10000/hello/"
*   Trying 121.37.5.232:10000...
* Connected to 121.37.5.232 (121.37.5.232) port 10000 (#0)
> HEAD /hello/ HTTP/1.1
> Host: 121.37.5.232:10000
> User-Agent: curl/7.79.0-DEV
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 101 Switching Protocols
HTTP/1.1 101 Switching Protocols
< Connection: Upgrade
Connection: Upgrade
< Upgrade: h2c
Upgrade: h2c
* Received 101
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* stopped the pause stream!
* Connection #0 to host 121.37.5.232 left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

$ ./curl -v -g --http2 "http://121.37.5.232:10000/hello/"
*   Trying 121.37.5.232:10000...
* Connected to 121.37.5.232 (121.37.5.232) port 10000 (#0)
> GET /hello/ HTTP/1.1
> Host: 121.37.5.232:10000
> User-Agent: curl/7.79.0-DEV
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 101 Switching Protocols
< Connection: Upgrade
< Upgrade: h2c
* Received 101
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< content-type: text/plain; charset=utf-8
< content-length: 6
< date: Mon, 30 Aug 2021 13:17:41 GMT
< 
* Connection #0 to host 121.37.5.232 left intact
Hello.

tk-xiong · 2021-08-30T13:16:15Z

$ ./curl --version
curl 7.79.0-DEV (x86_64-unknown-linux-gnu) libcurl/7.79.0-DEV zlib/1.2.7 nghttp2/1.44.0
Release-Date: [unreleased]
Protocols: dict file ftp gopher http imap mqtt pop3 rtsp smtp telnet tftp 
Features: alt-svc AsynchDNS HTTP2 IPv6 Largefile libz UnixSockets

bagder · 2021-08-30T14:45:26Z

I believe the HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR error is unrelated and a different thing.

Your server sends a RST_STREAM with this error when curl sends a HEAD request, but not when a GET is sent.

tk-xiong · 2021-08-30T15:41:39Z

Ok, thank you very much. I'll try to learn the difference between HEAD and GET requests.
close the issue.

bagder · 2021-08-30T15:57:11Z

The revert is still needed so this issue is still real and thus you closed it prematurely.

…witch" This reverts commit 455a63c. Reported-by: Tk Xiong Fixes #7633 Closes #7648

tk-xiong · 2021-08-31T05:50:44Z

Sorry, I will reopen it, so when should I close this issue?

bagder · 2021-08-31T06:58:53Z

When we merged the fix, which was done already.

bagder added the HTTP/2 label Aug 26, 2021

bagder added a commit that referenced this issue Aug 30, 2021

Revert "http2: skip immediate parsing of payload following protocol s…

dc97050

…witch" This reverts commit 455a63c. Reported-by: Tk Xiong Fixes #7633

bagder mentioned this issue Aug 30, 2021

Revert "http2: skip immediate parsing of payload following protocol s… #7648

Closed

bagder changed the title ~~curl --http2 Not Work~~ HTTP/2 (h2c) upgrade in clear-text problem Aug 30, 2021

tk-xiong closed this as completed Aug 30, 2021

bagder added a commit that referenced this issue Aug 30, 2021

Revert "http2: skip immediate parsing of payload following protocol s…

c364f5f

…witch" This reverts commit 455a63c. Reported-by: Tk Xiong Fixes #7633 Closes #7648

tk-xiong reopened this Aug 31, 2021

bagder closed this as completed Aug 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTP/2 (h2c) upgrade in clear-text problem #7633

HTTP/2 (h2c) upgrade in clear-text problem #7633

tk-xiong commented Aug 26, 2021 •

edited by bagder

Loading

tk-xiong commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

tk-xiong commented Aug 26, 2021 •

edited

Loading

bagder commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

bagder commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

bagder commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

bagder commented Aug 26, 2021

tk-xiong commented Aug 27, 2021 •

edited

Loading

bagder commented Aug 30, 2021

tk-xiong commented Aug 30, 2021

bagder commented Aug 30, 2021

tk-xiong commented Aug 30, 2021 •

edited

Loading

tk-xiong commented Aug 30, 2021 •

edited

Loading

bagder commented Aug 30, 2021 •

edited

Loading

tk-xiong commented Aug 30, 2021

bagder commented Aug 30, 2021

tk-xiong commented Aug 31, 2021

bagder commented Aug 31, 2021

HTTP/2 (h2c) upgrade in clear-text problem #7633

HTTP/2 (h2c) upgrade in clear-text problem #7633

Comments

tk-xiong commented Aug 26, 2021 • edited by bagder Loading

I did this

I expected the following

curl/libcurl version

operating system

tk-xiong commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

tk-xiong commented Aug 26, 2021 • edited Loading

bagder commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

bagder commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

bagder commented Aug 26, 2021

tk-xiong commented Aug 26, 2021

bagder commented Aug 26, 2021

tk-xiong commented Aug 27, 2021 • edited Loading

bagder commented Aug 30, 2021

tk-xiong commented Aug 30, 2021

bagder commented Aug 30, 2021

tk-xiong commented Aug 30, 2021 • edited Loading

tk-xiong commented Aug 30, 2021 • edited Loading

bagder commented Aug 30, 2021 • edited Loading

tk-xiong commented Aug 30, 2021

bagder commented Aug 30, 2021

tk-xiong commented Aug 31, 2021

bagder commented Aug 31, 2021

tk-xiong commented Aug 26, 2021 •

edited by bagder

Loading

tk-xiong commented Aug 26, 2021 •

edited

Loading

tk-xiong commented Aug 27, 2021 •

edited

Loading

tk-xiong commented Aug 30, 2021 •

edited

Loading

tk-xiong commented Aug 30, 2021 •

edited

Loading

bagder commented Aug 30, 2021 •

edited

Loading