Source IP not preserved in HTTP headers

[bogdansucila]

Bug description

Brief overview: In our current implementation of Istio, the ingress gateway sits behind one of our partner's reverse proxies, i.e. the client owns the domain name, which gets resolved to their reverse proxy, which in turn forwards traffic to our ingress gateway IP. In turn, the ingress gateway acts as reverse proxy itself, routing traffic to either an Azure Web App (the web frontend) or an API service (running on Kubernetes).

While this system works fine, we've found that since migrating to this new implementation (before requests would go directly from the partner's reverse proxy to Azure or Kubernetes), we lost track of the client source IP in the HTTP headers. These were recorded in the X-Forwarded-For header, which should normally contain the client IP and the partner gateway's IP appended to the right, e.g.:

X-Forwarded-For: (client IP),(partner gateway IP)

Now, only the rightmost IP (that of the partner gateway) gets preserved. This is an issue for us as our IP blocking system uses the data it extracts from XFF as a filtering logic, and with all requests seemingly originating from the same IP, it is rendered pretty useless.

After consulting a few similar issues (namely #7607 and #10875) I have made the following changes to my deployment:

1. Changed the externalTrafficPolicy setting to Local in the ingressgateway config
2. Tried setting the useRemoteAddress setting of the Envoy config to false by using an EnvoyFilter, however HTTP requests begantiming out after applying the filter
3. Finally managed to push the setting by modifying the source code and recompiling a custom version of Pilot's discovery container image, which I then deployed from my DockerHub to my Pilot deployment in Kubernetes.

So far, all regular HTTP requests yield the partner's gateway IP in their X-Forwarded-For headers. Requests sent with custom XFF headers (e.g. via Reqbin) retain their custom data, however this is not the case for the majority of our clients.

Any idea how I could go about enforcing XFF header preservation in Istio beyond these two settings? Also, is there an easier way to push the useRemoteAddress setting in Envoy besides building a custom Pilot image? In a different thread (#15236) someone mentioned injecting a custom environment variable via helm - is this possible?

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior

X-Forwarded-For headers containing the source IP of the client that made the request.

Steps to reproduce the bug

Publish a web application via Istio; for better accuracy, use an intermediate proxy to reach the web application (Fiddler can provide this functionality, also any transparent web proxy that you can find online).

Set the externalTrafficPolicy to Local in the ingress-gateway section of install/kubernetes/helm/istio/charts/gateways/values.yaml

The useRemoteAddress setting should also be switched to false - if you need a custom Pilot image with this setting you can use bogdansucila/istio_test:v1 (in v2 and v2.1 I played around with the xff_num_trusted_hops setting, v1 should only have this one modification to the defaults); I hear the setting can also be injected via an environment variable with helm, but I haven't tested this yet.

Version (include the output of istioctl version --remote and kubectl version)

Istio: 1.1.0
Kubernetes: 1.12

How was Istio installed?

Using helm

Environment where bug was observed (cloud vendor, OS, etc)

Azure Kubernetes Service (AKS)

Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.

[howardjohn]

someone mentioned injecting a custom environment variable via helm - is this possible?

You can pass this into helm with --set pilot.env.PILOT_SIDECAR_USE_REMOTE_ADDRESS

This is only in 1.2.2 by the way

[howardjohn]

I believe this is fixed with the above. If not feel free to reopen

[tuwid]

I've set this to true and I'm facing the same problem after deleting the pods. Is there any additional steps I'm missing @howardjohn ?

Host="test.domain.com" Client-IP="127.0.0.1" Masked-Header="custom-app-header custom_id=XXX-XX-45e0-80c2-XXXX" URI="/v2/XXXX/:id/XXXXXX" Status="201" Request-Time="0.272" User-Agent="XXX.mpos.android/2.47.1 (client: f0cc4651ec56570b, mode: TEST, os: Android, version: 7.1.1)" Server-Response-Time="0.272"

[Arpeet-gupta]

https://blog.opstree.com/2020/04/07/preserve-source-ip-in-aws-classic-load-balancer-and-istios-envoy-using-proxy-protocol/