Envoy always got remote_reset on HTTP/2 with Microsoft-IIS/10.0

[mrnonz]

Title: Envoy always got remote reset on HTTP/2 with Microsoft-IIS/10.0

Description:

I try to use envoy upstream to partner Microsoft-IIS/10.0 and got remote reset error and every thing work fine when I disabled http2 on alpn.

Please help me read trace log and find this problem.

Envoy: v1.29.2
TLSv1.2

My traffic like curl -(HTTP http/1.1)-> envoy -(TLS http/2) -> Microsoft-IIS will fail.
but for curl -(TLS http/2) -> Microsoft-IIS will success.

Here my config on envoy

         {
            "dns_lookup_family": "V4_ONLY",
            "load_assignment": {
               "cluster_name": "upstream",
               "endpoints": [
                  {
                     "lb_endpoints": [
                        {
                           "endpoint": {
                              "address": {
                                 "socket_address": {
                                    "address": "partner.com",
                                    "port_value": 443
                                 }
                              }
                           }
                        }
                     ]
                  }
               ]
            },
            "name": "partner.com",
            "respect_dns_ttl": true,
            "transport_socket": {
               "name": "envoy.transport_sockets.tls",
               "typed_config": {
                  "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
                  "common_tls_context": {
                     "alpn_protocols": "h2", // After change this to http/1.1 request will success
                     "tls_params": {
                        "tls_maximum_protocol_version": "TLSv1_3",
                        "tls_minimum_protocol_version": "TLSv1_2"
                     },
                     "validation_context": {
                        "match_typed_subject_alt_names": [
                           {
                              "matcher": {
                                 "exact": "partner.com"
                              },
                              "san_type": "DNS"
                           }
                        ],
                        "trusted_ca": {
                           "filename": "/etc/ssl/certs/ca-certificates.crt"
                        }
                     }
                  },
                  "sni": "partner.com"
               }
            },
            "type": "LOGICAL_DNS",
            "typed_extension_protocol_options": {
               "envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
                  "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
                  "auto_config": { },
                  "common_http_protocol_options": {
                     "idle_timeout": {
                        "seconds": 10
                     },
                     "max_connection_duration": {
                        "seconds": 3600
                     }
                  }
               }
            }
         },

and I attached success log of http/1.1 and fail log http/2 as well.
success-http1.1.log
fail-h2.log

[nezdolik]

cc @alyssawilk @RyanTheOptimist

[alyssawilk]

if Envoy is getting a remote reset, IIS is resetting and you need to look at logs there. The most common failure mode here AFIK is having scheme not match crypto (e.g. https scheme over cleartext.

[mrnonz]

But when I try to curl to IIS everything look good, even when open on Chrome, Safari browser good as well.

I don't know why that happen and I request partner to turn HTTP2 off.

[wtzhang23]

Tangentially related but I'm currently having issues where when Envoy upgrades plaintext connections to https. Envoy seems to use the downstream :scheme/xff/downstream connection to infer the scheme. This can cause problems with HTTP2 servers that validate whether the scheme pseudo-header matches the transport layer protocol (example).

Stuff I haven't verified but looked into while digging (still personally looking for ways to address this in the VHDS service in a per-route basis):

• If your operation applies globally to the HttpConnectionManager, the scheme header transformation configuration might work?
• Otherwise, header mutation rules enabling allow_all_routing might work? Seems like this is only for the ExternalProcessor filter

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.