Support HTTP/2 communication with clients

[normal-cock]

I've tried to forward http2.0 request to a http2.0 server using h2 , it seems that proxy.py doesn't work normally and I can't get response from the http2.0 server. I also read the source code of proxy.py, but didn't find any code related to http2.0. However, according to Readme.md, it is said that http2 is supported . I'm not sure if I miss something?

[abhinavsingh]

@normal-cock How did you try sending h2 request via proxy.py?

Using browser

If you configure your system/browser to use proxy.py, you'll see h2 protocol requests being proxied successfully.

Using curl with --http2

$ curl -v --http2 -x localhost:8899 http://httpbin.org/get
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8899 (#0)
> GET http://httpbin.org/get HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.54.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
> 
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Content-Type: application/json
< Date: Sun, 22 Sep 2019 15:57:26 GMT
< Referrer-Policy: no-referrer-when-downgrade
< Server: nginx
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-XSS-Protection: 1; mode=block
< Content-Length: 202
< Connection: keep-alive
< 
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.54.0"
  }, 
  "origin": "73.202.54.219, 73.202.54.219", 
  "url": "https://httpbin.org/get"
}
* Connection #0 to host localhost left intact

[abhinavsingh]

Note that curl --http2 flag is an indication for curl to upgrade protocol from HTTP 1.x to HTTP 2.x. This works fine.

However, if you use --http2-prior-knowledge which assumes immediate peer (in this case proxy.py) implements the h2 framing protocol, then it won't currently work with proxy.py. Example:

Curl with --http2-prior-knowledge hangs for HTTP requests

$ curl -v --http2-prior-knowledge -x localhost:8899 http://httpbin.org/get
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8899 (#0)
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7ff651006600)
> GET http://httpbin.org/get HTTP/2
> Host: httpbin.org
> User-Agent: curl/7.54.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 

Curl with --http2-prior-knowledge WORKS for HTTPS requests

$ curl -v --http2-prior-knowledge -x localhost:8899 https://httpbin.org/get
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8899 (#0)
* Establish HTTP proxy tunnel to httpbin.org:443
> CONNECT httpbin.org:443 HTTP/1.1
> Host: httpbin.org:443
> User-Agent: curl/7.54.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied OK to CONNECT request
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=httpbin.org
*  start date: Feb 17 00:00:00 2019 GMT
*  expire date: Mar 17 12:00:00 2020 GMT
*  subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fcc62803c00)
> GET /get HTTP/2
> Host: httpbin.org
> User-Agent: curl/7.54.0
> Accept: */*
> 
* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame.  Perhaps, peer does not support HTTP/2 properly.
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (16) Error in the HTTP2 framing layer

In the latter case request succeeds but curl reports that immediate peer does not support h2 properly. Which is true because proxy.py doesn't communicate with client using h2, but it helps client proxy h2 protocol to upstream servers.

[normal-cock]

@abhinavsingh
I send request by the following code using hyper:

from hyper import HTTP20Connection
# proxy.py is listening on 8899 
# h2 server is listening on 8443 using http instead of https
conn = HTTP20Connection('127.0.0.1:8443', secure=False, proxy_host='127.0.0.1', proxy_port='8890')
conn.request('GET', '/')

The execution was block after conn.request('GET', '/').
But thanks for your response and I'll try your method of sending h2-request later.

[abhinavsingh]

@normal-cock Thanks for clarifying. Unfortunately, hyper won't work straight out of box with proxy.py. I haven't used hyper myself but IIUC hyper uses HTTP/2 protocol.

While proxy.py do support proxying HTTP/2 protocol, it doesn't support communication with client using HTTP/2. Sometime in future proxy.py will support this though :)

[abhinavsingh]

@normal-cock Also I realized httpbin.org probably doesn't support HTTP2 protocol in first place, hence my curl examples in #79 (comment) are invalid.

See example below for my personal website which does support HTTP2:

Curl with --http2 for http:// resources

$ curl -v --http2 -x localhost:8899 http://abhinavsingh.com
* Rebuilt URL to: http://abhinavsingh.com/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8899 (#0)
> GET http://abhinavsingh.com/ HTTP/1.1
> Host: abhinavsingh.com
> User-Agent: curl/7.54.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
> 
< HTTP/1.1 301 Moved Permanently
< Date: Mon, 23 Sep 2019 05:20:46 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: max-age=3600
< Expires: Mon, 23 Sep 2019 06:20:46 GMT
< Location: https://abhinavsingh.com/
< X-Content-Type-Options: nosniff
< Server: cloudflare
< CF-RAY: 51aa123f9ffc6bd2-SJC
< 
* Connection #0 to host localhost left intact

Curl with --http2 for https:// resources

$ curl -v --http2 -x localhost:8899 https://abhinavsingh.com
* Rebuilt URL to: https://abhinavsingh.com/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8899 (#0)
* Establish HTTP proxy tunnel to abhinavsingh.com:443
> CONNECT abhinavsingh.com:443 HTTP/1.1
> Host: abhinavsingh.com:443
> User-Agent: curl/7.54.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied OK to CONNECT request
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL Multi-Domain; CN=sni53413.cloudflaressl.com
*  start date: Sep  6 00:00:00 2019 GMT
*  expire date: Mar 14 23:59:59 2020 GMT
*  subjectAltName: host "abhinavsingh.com" matched cert's "abhinavsingh.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO ECC Domain Validation Secure Server CA 2
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f834c006600)
> GET / HTTP/2
> Host: abhinavsingh.com
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
< date: Mon, 23 Sep 2019 05:23:12 GMT
< content-type: text/html
< server: cloudflare
< cf-ray: 51aa15d37dcc9629-SJC
< 
<!DOCTYPE html> ... [ redacted ] ... </html>


* Connection #0 to host localhost left intact

So both cases works just fine.

[abhinavsingh]

@abhinavsingh
I send request by the following code using hyper:

from hyper import HTTP20Connection
# proxy.py is listening on 8899 
# h2 server is listening on 8443 using http instead of https
conn = HTTP20Connection('127.0.0.1:8443', secure=False, proxy_host='127.0.0.1', proxy_port='8890')
conn.request('GET', '/')

The execution was block after conn.request('GET', '/').
But thanks for your response and I'll try your method of sending h2-request later.

For above errors there can be two reasons:

• Hyper is trying to establish a HTTP2 connection with proxy.py. This ain't gonna work for now. I have already re-purposed this issue to add support for HTTP2 communication with clients.

• Or, your upstream server doesn't support HTTP2 protocol properly. You can verify the same first using curl on command line.

[normal-cock]

I figure out the reason of different phenomena.

• Using --http2-prior-knowledge instead of --http2
  The command "curl -v --http2 http://abhinavsingh.com" actually using http 1.1 according to the protocol information in response headers.

  < HTTP/1.1 301 Moved Permanently
  < Date: Mon, 23 Sep 2019 05:20:46 GMT

• Most browser use http 1.1 version CONNECT when asking proxy to build tunnel
  This is true no matter whether the client connect to the destination service using http 2. For example when we visit www.google.com in browser, the CONNECT request sent to proxy is http 1.1 even though we are communicating with www.google.com using http 2.
  We can get this conclusion by capturing package in Wireshark, the result of which is text data not binary data.
  
  

Actual result

So the actual result is that proxy.py doesn't understand http 2 package, but we can still use it when connecting to http 2 server because many browsers or --http2 option of curl communicate with http proxy using http 1.1.

[abhinavsingh]

@normal-cock Awesome. Thanks for verification via wireshark :) This is the expected behavior for proxy.py, we currently don't support HTTP2 communication with clients but do allow proxying HTTP2 requests.