Incorrect X-FORWARDED-PROTO, X-FORWARDED-PORT for external TLS termination and proxy protocol

[peterkappelt]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

I think I've discovered an edge case (or myself not completely understanding the docs) that relates to setting the X-FORWARDED-PROTO and -PORT when proxy protocol is enabled.

I was able to reproduce the bug with this minimal setup:

• my browser @ 80.1.2.3
  • connects to the public ip of the external load balancer
• an external load balancer service, does TLS termination, 10.11.0.200
  • wireshark shows that it correctly sets X-Forwarded-Proto https, X-Forwarded-Port 443
  • proxies to Traefik on port 8181, proxy protocol enabled
• Traefik
  • --entrypoints.web.address=:8181/tcp --entrypoints.web.forwardedHeaders.trustedIPs=10.11.0.200 --entrypoints.web.proxyProtocol.trustedIPs=10.11.0.200
  • service that connects to the sample app
• Sample App, simply dumps all request headers in the response

I'd expect the sample app to receive those headers:

x-forwarded-for 80.1.2.3, 10.11.0.200
x-forwarded-host hostname.example
x-forwarded-port 443
x-forwarded-proto https
x-forwarded-server traefik-node-name
x-real-ip 80.1.2.3

What did you see instead?

Traefik forwards the headers, except for X-FORWARDED-PROTO and -PORT. The sample app receives:

x-forwarded-for 80.1.2.3 
x-forwarded-host hostname.example
x-forwarded-port 80
x-forwarded-proto http
x-forwarded-server traefik-node-name
x-real-ip 80.1.2.3

I can get the expected x-forwarded-proto and x-forwarded-port by:

• disabling proxy protocol on both Traefik and the external load balancer
• setting forwardedHeaders.trustedIPs=0.0.0.0/0 (or forwardedHeaders.insecure=true)
• setting forwardedHeaders.trustedIPs=80.1.2.3 (my client IP)

None of this is really suitable for production.

As it seems, traefik uses the client ip as the source ip when determining whether it is a trusted ip for header forwarding, when proxy protocol is enabled. I'd still expect traefik to use the actual source ip (load balancer).

What version of Traefik are you using?

Version: 2.9.6
Codename: banon
Go version: go1.19.4
Built: 2022-12-07T14:17:58Z
OS/Arch: linux/amd64

What is your environment & configuration?

See description and cases above

If applicable, please paste the log output in DEBUG level

No response

[mpl]

Hello @peterkappelt ,

I am not done fully analysing the situation, so I do not know whether I agree with you on the final expected output yet, but one thing already strikes me as odd:

If the external LB is trusted by traefik (which should be the intent of --entrypoints.web.forwardedHeaders.trustedIPs=10.11.0.200 , right?), then the final x-forwarded-for should contain both the client IP, and the external LB IP, so it should be:
X-Forwarded-For 80.1.2.3, 10.11.0.200.

However, you said that you would expect x-forwarded-for 80.1.2.3 (and that you did receive that on the backend). Was that a typo? If not, I think that in itself is already problematic, isn't it?

Btw, you say that your browser is 8.1.2.3, but then everywhere else I see 80.1.2.3. I assume that's a typo, and these two are in fact the same IP?

[peterkappelt]

Hi @mpl,

sorry for the confusion - I wrote this issue after a long day of debugging symptoms that eventually resulted in this specific problem and had way too many snippets in my clipboard.

Btw, you say that your browser is 8.1.2.3, but then everywhere else I see 80.1.2.3. I assume that's a typo, and these two are in fact the same IP?

Yes, it was a typo, I changed it in the initial comment

However, you said that you would expect x-forwarded-for 80.1.2.3 (and that you did receive that on the backend). Was that a typo? If not, I think that in itself is already problematic, isn't it?

I didn't really think about this (yet), for now I just cared about proto and port. It doesn't really matter for my setup if the external load balancer (10.11.0.200) is included in X-Forwarded-For since it's part of a trusted environment. However, I do agree that it probably should be part of X-Forwarded-For in general.

I checked X-Forwarded-For once again:

• with proxy protocol disabled: X-Forwarded-For: 80.1.2.3, 10.11.0.200 (as expected)
• with proxy protocol enabled and forwardedHeaders.insecure=true: X-Forwarded-For: 80.1.2.3, 80.1.2.3 (duplicate client ip)
• with proxy protocol enabled and forwardedHeaders.trustedIPs=10.11.0.200: X-Forwarded-For 80.1.2.3 (along with port 80 and http protocol)

Cheers!

[mpl]

yeah ok, I think there's a bug.

What I believe is happening is that Proxy Protocol changes the result/value of the RemoteAddr() call that is used at some point. In your case, it will return 80.1.2.3, instead of 10.11.0.200.

So then, when we get in forwardedHeaders, and we ask the question "hey, is the last hop I'm talking to a trusted IP?" we use RemoteAddr() (maybe we shouldn't, or maybe the result of RemoteAddr() itself should be fixed), which tells us that the last hop is 80.1.2.3 (which of course factually isn't, it should be 10.11.0.200). And since traefik is configured to trust only 10.11.0.200 (and not 80.1.2.3), it goes into untrusted mode, erases all the X-Forwarded headers (as it should in these circumstances), and writes to them whatever it thinks the last hop was. It explains the behaviour regarding X-Forwarded-For, and probably for the others as well (I haven't looked into them in detail, but I would be surprised if it's not the same root cause).

Hopefully we'll look further into a fix soon.

[yongzhang]

expericing the same issue, spent a couple of days struggling with this ...

I set forwardedHeaders.trustedIPs=0.0.0.0/0, still got

X-Forwarded-Port: 80
X-Forwarded-Proto: http

[tinchoram]

👋 Hi, same issue here.

I can't see the client IP into mi app k8s when use NLB.
I try use https://github.com/jramsgz/traefik-real-ip/tree/main but not work 🫤

i hope resolution

[nmengin]

Hello,

We would love community support to address it.

If you or another community member would like to build it, let us know and we will work with you to make sure you have all the information needed so that it can be merged.

We prefer to work with our community members at the beginning of the design process so that we can make sure that we are aligned and can move quickly with the review and merge process. Let us know here or create a PR before you start and we will work with you there.

Don’t forget to check out the contributor docs and to link the PR to this issue.
If not, we will leave the issue open to any interested community member.

[andsarr]

I would like to help with this issue, but I am having problems to reproduce it locally.

Is it possible to use two instances of Traefik in proxy mode?

[rtribotte]

@andsarr Sure, thank you!

Is it possible to use two instances of Traefik in proxy mode?

Do you mean using two Traefik instances, one proxying the other, and using proxy protocol on both, for producing the proxy protocol headers on one side and reading them on the other? If yes, I think so.

[andsarr]

This is how I created the test environment (I hope I didn't over-complicated it). I created three Docker containers:

1. external_elb: This is suppose to be the external load balancer. I choose HAproxy for this.
2. traefik: Traefik container running version 2.11. It only has one entry-point with Proxy Protocol enabled. To do this I set the proxyProtocol.truestedIPs configuration.
3. whoami: The service for testing.

The following figure illustrate the setup:

Notice that the elb has two network interfaces. I used mkcert to set HTTPS support in the elb.

So, when I execute curl this is what I get:

Hostname: whoami
IP: 127.0.0.1
IP: 10.150.100.15
RemoteAddr: 10.150.100.5:40536
GET / HTTP/1.1
Host: whoami.example.com
User-Agent: curl/8.6.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.18.100.1
X-Forwarded-Host: whoami.example.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik
X-Real-Ip: 172.18.100.1

I also double check that the PROXY Header was being sent using tcpdump inside the elb container. This is what I get:

If I understood correctly, the issue is that I should get this answer instead:

Hostname: whoami
IP: 127.0.0.1
IP: 10.150.100.15
RemoteAddr: 10.150.100.5:40536
GET / HTTP/1.1
Host: whoami.example.com
User-Agent: curl/8.6.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.18.100.1, 10.150.30.10
X-Forwarded-Host: whoami.example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik
X-Real-Ip: 172.18.100.1

Could you confirm @peterkappelt, @mpl, @rtribotte ?

To reproduce my steps, here the files I am using:

issue-9757.zip

I didn't include the SSL certificate, but one can be easily generated using mkcert.

[polarathene]

I am having problems to reproduce it locally

I am a bit late, but you can inject your own PROXY protocol header via plain HTTP requests:

# Use `printf` command to format a raw HTTP request with plaintext PROXY header prepended,
# then pipe into netcat to send to `localhost:80`:
printf "PROXY TCP4 10.20.30.40 172.16.13.37 12345 44380\r\nGET / HTTP/1.1\r\nHost: http-without-redirect.test\r\n\r\n" | nc -w 1 localhost 80

Within a Docker Compose config, you can quite easily setup containers all within the same network for better IP assignments and minimizing gotchas.

For example since you sent the curl request to localhost:4443, Docker routed that through the subnet gateway IP 172.18.100.1 (although I guess this is irrelevant due to using PROXY protocol 😅 ).

---

I wrote a little proxy CLI program to do similar to the printf above, but for any connection over TCP. Not sure if it'd be useful to anyone but I could share it if useful.

If no trusted IP is configured, raw TCP is forwarded, so untrusted clients can sneak a header through but I think that's intentional/expected to blindly forward and not make assumptions (since there's a small possibility of the data randomly being a PROXY header but legitimate traffic routed to a destination that doesn't expect a PROXY header, thus discarding by default may be bad?).

The docs don't mention this though and the two config options available imply:

• "trust everyone" (insecure=true, consumes the PROXY header)
• "trust only these IP" (trustedIPs=..., consume the PROXY header or if not trusted discard)

Which I think requires adding an IP that shouldn't be a client (but not the subnet gateway IP either of course), if you want to discard any incoming PROXY headers for TCP routers?

[andsarr]

Thanks for the tip @polarathene, it is helping me a lot!