Envoy proxy v1.29.x does not forward TE header for gRPC (HTTP2) requests

[straffalli]

Title: Envoy proxy v1.29.x does not forward TE header for gRPC (HTTP2) requests

Description:
We are using Envoy as proxy and loadbalancer for gRPC requests, we had a working setup but after an upgrade from v1.28.1 to 1.29.x our client is reporting a missing TE header error.

Repro steps:
Deploy a basic python gRPC client/server from helloworld example.
Deploy Envoy proxy v1.29.2 (latest stable version) and configure it to forward gRPC requests from client to upstream server.
Configure client to send request to proxy address and port.

Config:

admin:
  address:
    socket_address:
      protocol: TCP
      address: 0.0.0.0
      port_value: 9901
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 8080
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
          codec_type: AUTO
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: service-grpc
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: service-grpc
    connect_timeout: 0.5s
    type: STRICT_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    http2_protocol_options: {}
    load_assignment:
      cluster_name: service-grpc
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: <grpc-server-address>
                port_value: 50051

Logs:
I turned logging level to TRACE on the Envoy proxy.

Using Envoy proxy v1.28.1
Here is the request headers forwarding part in envoy logs, the TE header is present in the request from the client and it is correctly forwarded to uptreams servers by envoy:

[2024-03-19 14:52:49.464][17][debug][http] [source/common/http/conn_manager_impl.cc:1194] [Tags: "ConnectionId":"1","StreamId":"8062068683630422289"] request headers complete (end_stream=false):
':path', '/Greeter/SayHello'
':authority', '<ip_address_redacted>:8080'
':method', 'POST'
':scheme', 'http'
'content-type', 'application/grpc'
'te', 'trailers'
'grpc-accept-encoding', 'identity, deflate, gzip'
'user-agent', 'grpc-python/1.62.0 grpc-c/39.0.0 (linux; chttp2)'

[2024-03-19 14:52:49.464][17][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"1"] current connecting state: false
[2024-03-19 14:52:49.464][17][debug][router] [source/common/router/router.cc:520] [Tags: "ConnectionId":"1","StreamId":"8062068683630422289"] cluster 'service-grpc' match for URL '/Greeter/SayHello'
[2024-03-19 14:52:49.464][17][debug][router] [source/common/router/router.cc:732] [Tags: "ConnectionId":"1","StreamId":"8062068683630422289"] router decoding headers:
':path', '/Greeter/SayHello'
':authority', '<ip_address_redacted>:8080'
':method', 'POST'
':scheme', 'http'
'content-type', 'application/grpc'
'te', 'trailers'
'grpc-accept-encoding', 'identity, deflate, gzip'
'user-agent', 'grpc-python/1.62.0 grpc-c/39.0.0 (linux; chttp2)'
'x-forwarded-proto', 'http'
'x-request-id', '6f59b376-2d09-4df0-892b-477fd3dd03fc'
'x-envoy-expected-rq-timeout-ms', '15000'

On the client side, the response is received as expected:

$ python client.py 
Will try to greet world ...
Greeter client received: Hello, you!

Using Envoy proxy v1.29.2 (with same envoy config and same grpc client and server)

We can see in the envoy logs, that TE header appears in the request received but this header is not forwarded to upstreams:

[2024-03-19 14:42:17.201][18][debug][http] [source/common/http/conn_manager_impl.cc:1192] [Tags: "ConnectionId":"1","StreamId":"15151355021444603627"] request headers complete (end_stream=false):
':path', '/Greeter/SayHello'
':authority', '<ip_address_redacted>:8080'
':method', 'POST'
':scheme', 'http'
'content-type', 'application/grpc'
'te', 'trailers'
'grpc-accept-encoding', 'identity, deflate, gzip'
'user-agent', 'grpc-python/1.62.0 grpc-c/39.0.0 (linux; chttp2)'

[2024-03-19 14:42:17.201][18][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"1"] current connecting state: false
[2024-03-19 14:42:17.201][18][debug][router] [source/common/router/router.cc:514] [Tags: "ConnectionId":"1","StreamId":"15151355021444603627"] cluster 'service-grpc' match for URL '/Greeter/SayHello'
[2024-03-19 14:42:17.201][18][debug][router] [source/common/router/router.cc:731] [Tags: "ConnectionId":"1","StreamId":"15151355021444603627"] router decoding headers:
':path', '/Greeter/SayHello'
':authority', '<ip_address_redacted>:8080'
':method', 'POST'
':scheme', 'http'
'content-type', 'application/grpc'
'grpc-accept-encoding', 'identity, deflate, gzip'
'user-agent', 'grpc-python/1.62.0 grpc-c/39.0.0 (linux; chttp2)'
'x-forwarded-proto', 'http'
'x-request-id', 'b132c956-c787-43ec-a17c-f7759e543f14'
'x-envoy-expected-rq-timeout-ms', '15000'

It results in the following error on client side:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Missing :te header"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Missing :te header", grpc_status:2, created_time:"2024-03-19T14:42:17.334807621+00:00"}"

I also did the test with v1.29.0 and v1.29.1 and the issue is the same, it's only working with Envoy versions prior to 1.29.0.

When searching for similar issues I found an old one from 2019 talking about clearing TE header: #5541
It was mentioned that TE header is mandatory and used by all standard gRPC clients. Because gRPC is using HTTP2, maybe this header should not be removed because it will result in breaking gRPC forwarding.

In addition I also noticed in the release changes for Envoy v1.29.0:

HTTP: clear hop by hop Transfer-Encoding header.

If this has been applied to HTTP1.1 and HTTP2 it could be a possible issue and breaks gRPC requests.

[htuch]

This just merged into the 1.29 branch as as backport, #33018. @phlax