Filter connection-related headers when forwarding HTTP/1.1 responses over H/2

[thomaswmanion]

Summary and context

We are unable to make HTTPS requests in all browsers.

Here is the request we're doing:

curl -X POST
https://endpoint
-H 'Content-Type: application/json'
-d '{"username":"zdsfasdf","password":"dsfdafsd"}'

Here are the results we are seeing:

1. Chrome does not work – Fails with:
   POST https://endpoint net::ERR_SPDY_PROTOCOL_ERROR
2. Safari does not work – Fails with:
   [Error] Failed to load resource: The operation couldn’t be completed. Protocol error
3. Curl 7.54.0 does not work – Fails with:
   curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
4. Curl 7.54.0 works when adding the flag --http 1.1
5. Curl 7.64.1 works
6. Postman works (after disabling SSL certificate verification)
7. Firefox works

How to reproduce

Run the RIG on 2.1.1 with this configuration:

1: Here’s our port configuration:
- name: INBOUND_PORT
value: "3010"
- name: INBOUND_HTTPS_PORT
value: "3011"
- name: API_HTTP_PORT
value: "4010"
- name: API_HTTPS_PORT
value: "4011"
2: Here’s our https certs config:
- name: HTTPS_CERTFILE
value: "/gateway-certs/selfsigned.pem"
- name: HTTPS_KEYFILE
value: "/gateway-certs/selfsigned_key.pem"
3: The above certs are the same ones as in the RIG repo

Make a POST request to this server in any of the failing clients above.

Describe how your clients are set up, whether they connect via SSE or WS, etc.

This is happening in the REST request to authorize, prior to SSE or WS connection.

A clear and concise description of what you expected to happen.

Versions (please complete the following information):**

• Host OS: FROM elixir:1.6-alpine as build
• Frontend Chrome, Safari, Firefox
• RIG 2.1.1

[kevinbader]

I understand the request that fails is supposed to be forwarded to a backend service.

• Does the backend service offer HTTP/1.1 or HTTP/2 (or both)?
• In the cases that fail, does the request reach the backend service?
• Does running Curl with the verbose flag reveal more details on the protocol error?

[chadmott]

-it is a HAPI server with http/1.1
-it does reach the backend service, and the service appears to reply the same way to both HTTP and HTTPS calls
-curl verbose:

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying IPADDRESS...
* TCP_NODELAY set
* Connected to reactive-gateway-service (IPADDRESS) port 3011 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Phoenix Framework; CN=Self-signed test certificate
*  start date: Dec 13 00:00:00 2018 GMT
*  expire date: Dec 13 00:00:00 2019 GMT
*  issuer: O=Phoenix Framework; CN=Self-signed test certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55847c4405e0)
> POST /is/auth HTTP/2
> Host: reactive-gateway-service:3011
> User-Agent: curl/7.61.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 45
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* We are completely uploaded and fine
* http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [connection], value: [keep-alive]
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* stopped the pause stream!
* Connection #0 to host reactive-gateway-service left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

[chadmott]

could this be related? https://www.rhymewithgravy.com/2016/08/23/Beware-the-Connection-Header-in-HTT-P.html

Is there a way to tell RIG to NOT use HTTP/2?

[chadmott]

the more i dig the more it looks like that HTTP2 proxying is not something the phoenix project supports. the Underlying services use http/1.1, and upgrading them is not trivial.

https://elixirforum.com/t/the-hard-3-day-lesson-i-learned-trying-to-use-http2-with-phoenix-cowboy-and-nginx/17629

[kevinbader]

ah yes, the connection-related headers, as I've suspected (thanks for providing the curl output). RIG currently forwards the response headers from the backend as-is - if they include HTTP/1.1 headers that are forbidden in a HTTP/2 response, the client bails out. (The blog post you've linked is unrelated.)

I think I'll have RIG strip any illegal (= connection related) response headers from the backend response before forwarding it, that should do the trick.

[kevinbader]

I'm working on a larger PR related to this, but I'll cut a smaller one later today for allow you to test the fix.