Enable proxy protocol for Ingress Envoy

[b1tamara]

Is your feature request related to a problem? Please describe.
Some Load Balancers like AWS ELB replace the client IP by their own IP. In that case the feature "X-Forwarded-For" cannot be used because the X-Forwarded-For-Header contains the IP of the Load Balancer. By using Proxy Protocol between LB and Ingress Envoy the client IP will be set in the X-Forwarded-For-Header by Envoy.

Describe the solution you'd like
Apply EnvoyFilter for enabling the proxy protocol on the Envoy side. Make it configurable via values.

#@ load("@ytt:data", "data")
#@ if data.values.enable_proxy_protocol:
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: proxy-protocol
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: LISTENER
    patch:
      operation: MERGE
      value:
        listener_filters:
        - name: envoy.listener.proxy_protocol
        - name: envoy.listener.tls_inspector
#@ end

Additional context
The enabling of the proxy protocol of the Load Balancer side is not part of this feature request because it is a Load Balancer specific configuration.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175595206

The labels on this github issue will be updated when the story is started.

[mike1808]

Hi. Can you say why X-Forwarded-For header is not a solution? AFAIK Envoy will just add another IP address to that header so it will contain 1. ip of the user 2. ip of the AWS ELB

[mike1808]

Also, have you tried configuring gatewayTopology config in Istio? https://istio.io/latest/docs/ops/configuration/traffic-management/network-topologies/#configuring-x-forwarded-for-headers

[b1tamara]

Hello mike1808,
thank you for your suggested solution. Unfortunately it will not work for LoadBalancers which are not transparent (Transparent indicates whether client IP addresses are visible to the backend servers).
I enabled the proxy protocol on AWS LB that the Load Balancer preserves the client IP. Then I applied the IstioOperator using the annotation for Gateway istio-ingressgateway:

"proxy.istio.io/config": "{\"gatewayTopology\":{\"numTrustedProxies\":1}}",

After that I tried to curl the app which returns the "x-Forwarded-For-Header". I got:

curl https://x-forwarded-for.<domain>

curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to x-forwarded-for.<domain>

The problem is that the LoadBalancer tries to speak with EnvoyProxy the proxy protocol which Envoy does not understand as it is disabled per default. If the proxy protocol is enabled on both side, then a single additional tcp packet -containing the real client IP- that is sent during the initial connection set up.

If I apply the EnvoyFilter in order to enable the proxy protocol on the Envoy side, I get the real client IP:

curl https://x-forwarded-for.<domain>
<h1>X-Forwarded-For: <myClientIP></h1>  

We would like to provide the PR in order to enable this feature.

Best regards,
Tamara

[shalako]

I heard about this PR and wanted to thank the SAP team for contributing!

With regard to user experience for enabling this capability in cf-for-k8s, what is the risk of having it enabled all the time, rather than offering a field in cf-values.yml to turn it on and off?

[stefanlay]

Hi @shalako, thanks for the feedback.

The proxy protocol needs to be enabled on both sides of the communication. The proxy_protocol filter in Envoy expects that the proxy protocol is used, otherwise the tcp connection cannot be established.

We also tested it with AWS and GCP loadbalancers. In both cases we got the error:

wget: error getting response: Connection reset by peer
Connecting to <URL> (<IP>:443)
ssl_client: <URL>: handshake failed: Connection reset by peer
wget: error getting response: Connection reset by peer

Therefore we think that it is necessary to make it configurable.

See also [1]: "Note: if the filter is enabled, the Proxy Protocol must be present on the connection (either version 1 or version 2), the standard does not allow parsing to determine if it is present or not."

[1] https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/listener_filters/proxy_protocol#proxy-protocol

[shalako]

Thank you for the explanation. That sounds reasonable.

[jamespollard8]

#564 was closed to due changes in SAP's priorities.

If this issue becomes a priority for anyone, we'd encourage them to re-open this issue and/or the associated PR #564

cc my pair @Birdrock