cluster_header router & golang cluster_specifier plugin ignore headers added by envoy's http filters

[willemveerman]

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
envoy-security@googlegroups.com where the issue will be triaged appropriately.

Title: Headers added by Envoy's HTTP filters are ignored by the router filter

Description:
Envoy's cluster_header config option, as detailed here, will only route requests based on headers which are in the downstream request before it is processed by envoy's HTTP filters. If you add the specified header in the http_filters config section, this is disregarded by cluster_header.

I would expect cluster_header to act upon headers which are added in the http_filters section. My use-case is that I have written a golang plugin which adds a header to each request, and I want the request to be routed to a cluster based on the value of the header.

Moreover, HTTP filters are processed in order, and the router filter must be placed last, therefore it's surprising that headers being added before route_config are being ignored by cluster_header

The golang cluster_specifier plugin behaves in the same way; it will ignore headers which are added within envoy in the http_filters section.

Repro steps:
image: envoyproxy/envoy:contrib-v1.30.1

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          http_filters:
          - name: extensions.filters.http.header_mutation.v3.HeaderMutation
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.header_mutation.v3.HeaderMutation
              mutations:
                request_mutations: 
                - append:
                    header: 
                      key: simple_test_header
                      value: magenta
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster_header: simple_test_header

  clusters:
  - name: magenta
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: magenta
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: magenta
                port_value: 8081

1. this curl request to listener_0 returns a 503 - curl localhost:10000 -v
2. this curl request routes to cluster magenta - curl localhost:10000 -H "simple_test_header: magenta" -v

I would expect 1. to also route to magenta, because the simple_test_header has been added to the request.

The issue also appears if I try to dynamically add headers using the golang http plugin.

This code snippet within the golang plugin:

func (f *filter) DecodeHeaders(header api.RequestHeaderMap, endStream bool) api.StatusType {

	header.Set("simple_test_header", "blue")

	return api.Continue
}

with similar envoy config:

    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          http_filters:
          - name: envoy.filters.http.golang
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.golang.v3alpha.Config
              library_id: filter
              library_path: "/etc/filter/filter.so"
              plugin_name: filter
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster_header: simple_test_header

produces the same behaviour: the header added by the golang plugin is ignored by cluster_header

Logs:
logs from scenario 1. above

 [2024-04-30 22:43:52.708][29][debug][conn_handler] [source/common/listener_manager/active_tcp_listener.cc:160] [Tags: "ConnectionId":"0"] new connection from 172.19.0.1:56404
 [2024-04-30 22:43:52.716][29][debug][http] [source/common/http/conn_manager_impl.cc:398] [Tags: "ConnectionId":"0"] new stream
 [2024-04-30 22:43:52.728][29][debug][http] [source/common/http/conn_manager_impl.cc:1147] [Tags: "ConnectionId":"0","StreamId":"5870889276111733605"] request headers complete (end_stream=true):
 ':authority', 'localhost:10000'
 ':path', '/'
 ':method', 'GET'
 'user-agent', 'curl/8.4.0'
 'accept', '*/*'

 [2024-04-30 22:43:52.729][29][debug][http] [source/common/http/conn_manager_impl.cc:1130] [Tags: "ConnectionId":"0","StreamId":"5870889276111733605"] request end stream
 [2024-04-30 22:43:52.733][29][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"0"] current connecting state: false
 [2024-04-30 22:43:52.739][29][debug][router] [source/common/router/router.cc:498] [Tags: "ConnectionId":"0","StreamId":"5870889276111733605"] unknown cluster ''
 [2024-04-30 22:43:52.740][29][debug][http] [source/common/http/filter_manager.cc:1027] [Tags: "ConnectionId":"0","StreamId":"5870889276111733605"] Preparing local reply with details cluster_not_found
 [2024-04-30 22:43:52.743][29][debug][http] [source/common/http/filter_manager.cc:1069] [Tags: "ConnectionId":"0","StreamId":"5870889276111733605"] Executing sending local reply.
 [2024-04-30 22:43:52.745][29][debug][http] [source/common/http/conn_manager_impl.cc:1838] [Tags: "ConnectionId":"0","StreamId":"5870889276111733605"] encoding headers via codec (end_stream=true):
 ':status', '503'
 'date', 'Tue, 30 Apr 2024 22:43:52 GMT'
 'server', 'envoy'

 [2024-04-30 22:43:52.747][29][debug][http] [source/common/http/conn_manager_impl.cc:1950] [Tags: "ConnectionId":"0","StreamId":"5870889276111733605"] Codec completed encoding stream.

logs from scenario 2.

 [2024-04-30 22:49:03.738][24][debug][conn_handler] [source/common/listener_manager/active_tcp_listener.cc:160] [Tags: "ConnectionId":"4"] new connection from 172.19.0.1:59582
 [2024-04-30 22:49:03.739][24][debug][http] [source/common/http/conn_manager_impl.cc:398] [Tags: "ConnectionId":"4"] new stream
 [2024-04-30 22:49:03.740][24][debug][http] [source/common/http/conn_manager_impl.cc:1147] [Tags: "ConnectionId":"4","StreamId":"2228885928813311211"] request headers complete (end_stream=true):
 ':authority', 'localhost:10000'
 ':path', '/'
 ':method', 'GET'
 'user-agent', 'curl/8.4.0'
 'accept', '*/*'
 'simple_test_header', 'magenta'

 [2024-04-30 22:49:03.741][24][debug][http] [source/common/http/conn_manager_impl.cc:1130] [Tags: "ConnectionId":"4","StreamId":"2228885928813311211"] request end stream
 [2024-04-30 22:49:03.742][24][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"4"] current connecting state: false
 [2024-04-30 22:49:03.742][24][debug][router] [source/common/router/router.cc:515] [Tags: "ConnectionId":"4","StreamId":"2228885928813311211"] cluster 'magenta' match for URL '/'
 [2024-04-30 22:49:03.744][24][debug][router] [source/common/router/router.cc:738] [Tags: "ConnectionId":"4","StreamId":"2228885928813311211"] router decoding headers:
 ':authority', 'localhost:10000'
 ':path', '/'
 ':method', 'GET'
 ':scheme', 'http'
 'user-agent', 'curl/8.4.0'
 'accept', '*/*'
 'simple_test_header', 'magenta'
 'x-forwarded-proto', 'http'
 'x-request-id', '9984eb58-5a2e-4fad-8da1-9cf5c4598eda'
 'simple_test_header', 'magenta'
 'x-envoy-expected-rq-timeout-ms', '15000'

 [2024-04-30 22:49:03.744][24][debug][pool] [source/common/conn_pool/conn_pool_base.cc:265] [Tags: "ConnectionId":"3"] using existing fully connected connection
 [2024-04-30 22:49:03.745][24][debug][pool] [source/common/conn_pool/conn_pool_base.cc:182] [Tags: "ConnectionId":"3"] creating stream
 [2024-04-30 22:49:03.745][24][debug][router] [source/common/router/upstream_request.cc:581] [Tags: "ConnectionId":"4","StreamId":"2228885928813311211"] pool ready
 [2024-04-30 22:49:03.745][24][debug][client] [source/common/http/codec_client.cc:142] [Tags: "ConnectionId":"3"] encode complete
 [2024-04-30 22:49:03.749][24][debug][router] [source/common/router/router.cc:1528] [Tags: "ConnectionId":"4","StreamId":"2228885928813311211"] upstream headers complete: end_stream=false
 [2024-04-30 22:49:03.750][24][debug][http] [source/common/http/conn_manager_impl.cc:1838] [Tags: "ConnectionId":"4","StreamId":"2228885928813311211"] encoding headers via codec (end_stream=false):
 ':status', '200'
 'content-type', 'text/plain; charset=utf-8'
 'content-length', '7'
 'date', 'Tue, 30 Apr 2024 22:49:03 GMT'
 'server', 'envoy'
 'x-envoy-upstream-service-time', '3'

 [2024-04-30 22:49:03.750][24][debug][client] [source/common/http/codec_client.cc:129] [Tags: "ConnectionId":"3"] response complete

[spacewander]

The cluster_xxx features in the route action are invoked during the route match. So the headers added in the HTTP filter, which is chosen after the route match, won't affect the behavior of cluster selection.

[spacewander]

I don't try them myself but there may be some ways to satisfy your requirement:

1. break down the adding header logic and cluster selection, so the cluster will be dynamically set via cluster-specific plugin without setting the header.
2. find some way to clear route cache and update the cluster: https://www.envoyproxy.io/docs/envoy/latest/intro/life_of_a_request#http-filter-chain-processing. There may be some side-effect.

[willemveerman]

break down the adding header logic and cluster selection, so the cluster will be dynamically set via cluster-specific plugin without setting the header.

Yes I think this is the best way. If the ability to read all existing headers is added to the golang cluster_specifier plugin then I can read the headers, decide on a cluster and then route to it - using only a single filter and within the route action section. This will remove the need to clear the route cache on every request.

I see now from the doc that you shared that a route is chosen before the HTTP filters are run, as stated here:

When decodeHeaders() is invoked on the router filter, the route selection is finalized and a cluster is picked. The HCM selects a route from its RouteConfiguration at the start of HTTP filter chain execution. This is referred to as the cached route. Filters may modify headers and cause a new route to be selected, by asking HCM to clear the route cache and requesting HCM to reevaluate the route selection.

But it's a bit confusing because in other places in the Envoy docs they strongly imply that the router filter will run after the HTTP filter chain, as per point 6 in the request flow section:

For each HTTP stream, an Downstream HTTP filter chain is created and runs. The request first passes through CustomFilter which may read and modify the request. The most important HTTP filter is the router filter which sits at the end of the HTTP filter chain. When decodeHeaders is invoked on the router filter, the route is selected and a cluster is picked.

[doujiang24]

6e3d574
HeaderMap.Set won't clear route cache by default, it's first introduced in 1.30.0.
You need to clear route cache by using ClearRouteCache instead.

[willemveerman]

Yep, api.FilterCallbackHandler.ClearRouteCache enables the header change to take effect

Thank you for the information.

Once the capability to read the entire HeaderMap is added to the cluster_specifier it will remove the need to clear the route cache