Double proxy X-Real-Ip incorrect

[ju5t]

Do you want to request a feature or report a bug?

A (potential) bug.

What did you do?

Our setup:

• Docker Swarm
• HaProxy as the main entry for our public IP's
• Traefik is running on the inside of the cluster

HaProxy is running inside Docker Swarm.

This means we are proxying traffic twice. The X-Forwarded-For header is passed on according to the tests we do with emilevauge/whoami.

What did you expect to see?

X-Forwarded-For: 123.123.123.123, 172.18.0.1
X-Forwarded-Host: whoami.bellmann.nl
X-Forwarded-Port: 80
X-Forwarded-Proto: https
X-Forwarded-Server: 749a78b15337
X-Real-Ip: 123.123.123.123

What did you see instead?

X-Forwarded-For: 123.123.123.123, 172.18.0.1
X-Forwarded-Host: whoami.bellmann.nl
X-Forwarded-Port: 80
X-Forwarded-Proto: https
X-Forwarded-Server: 749a78b15337
X-Real-Ip: 172.18.0.1

X-Forwarded-For headers are generally appended, which means the original IP is on the left, as is the case here. Traefik seems to use 172.18.0.1 and as a result our whitelisting doesn't work.

But as soon as we move HaProxy out of Docker Swarm it seems like Traefik changes the XFF order where the IP on the right is the IP address of the client and it sets X-Real-Ip to 172.18.0.1. I think that it's Traefik because HaProxy has no knowledge of other proxies and only sets XFF to the original IP address.

Output of traefik version: (What version of Traefik are you using?)

Version:      v1.4.5
Codename:     roquefort
Go version:   go1.9.2
Built:        2017-12-06_10:16:48AM
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

--docker --docker.swarmmode --docker.domain=domain.com --docker.watch --entrypoints=Name:http ProxyProtocol.TrustedIPs:123.123.123.0/24 ProxyProtocol.Insecure:false Address::80 --defaultentrypoints=http

We don't use toml.

[ju5t]

I would love to go live with our test setup but this turns out to be a blocking issue for us.

Unfortunately I'm no good with Go to debug this but if there is anything I can do to gather more information that can help to resolve the issue; let me know. I'm more than happy to help.

[juliens]

• X-Real-Ip always contains the real ip that call your Traefik instance (for you, your previous proxy)
• X-Forwarded-For contains : client, proxy1 in this order (if you have 3 proxy you will have client, proxy1, proxy2)

But I think an explanation is missing in the documentation

[ju5t]

@juliens thanks for your feedback.

Unfortunately there's no RFC covering either of them. I don't believe X-Real-Ip was meant to be used for the previous proxy though. It doesn't add up as you would have headers for that already. It's use is arbitrary but I would expect X-Real-Ip to be the IP of the client.

Either way, there's still something out of the ordinary here. We're using ProxyProtocol.TrustedIPs and that doesn't work for us. That's where the issue originates from.

Case 1: client, proxy1 (external), traefik, container
Case 2: client, proxy1 (internal), traefik, container

By external I mean a proxy that runs outside Docker Swarm. Internal means the proxy runs within the same cluster as Traefik.

Case 1 works. X-Real-Ip gives back the IP address of the client and ProxyProtocol.TrustedIPs allows us to get to container.

Case 2 fails. X-Real-Ip gives back the IP address of proxy and ProxyProtocol.TrustedIPs returns Forbidden.

Is this a misconfiguration on our end or a bug in Traefik?

[ldez]

@ju5t you have put the same case, could you edit?

[ju5t]

@ldez I'm sorry, I don't understand what you mean there. It was also described in the original post though with a little less text. What do you need to me to clear up?

[aantono]

I think there might be a bug in headerRewrite.Rewrite method (https://github.com/containous/traefik/blob/master/server/header_rewriter.go#L40)
The secureRewriter is being used in cases when there are parsing or otherwise errors and, ironically, when if h.insecure || authorized condition evaluates to true

Inside Oxy's forward.HeaderRewriter, however, when the boolean TrustForwardHeader is set to true (which is what secureRewriter does, the various X-* headers are augmented, but when the TrustForwardHeader is set to false, they are all removed. So it makes me wonder if the logic somehow got accidentally reversed, so when you have a matching IP, it actually doesn't do the right thing. (am I missing something and being confused?)

[aantono]

@ju5t Looking at go-proxyproto library, I don't actually see any usage of X-Real-IP header, everything is being driven from conn.RemoteAddr() getter, which would either retrieve it from the PROXY... line of the HTTP request - should be (PROXY <type> <src addr> <dst addr> <src port> <dst port>), if ProxyProtocol is being used, or by returning the address from the socket where the connection originated from. So while in your tests the X-Real-IP happens to correspond to the last IP of the proxy chain, which is the same as the remote IP, as returned from the socket, the value of the X-Real-IP header is actually not driving the logic behind ProxyProtocol.TrustedIPs (or so it seems based on my digging around of the code)

Based on your explanation, where you state that as soon as you move HAProxy out of the docker container and onto a real VM/host, things start working, I would actually advise to inspect the HAProxy settings, as it is what sets the PROXY marker of the HTTP request, and it is probably putting its internal container src addr instead of the correct external one.

[ju5t]

@aantono thanks for the thorough digging and explanation.

In our test case we kept HaProxy inside a Docker container. Instead of running it within the same Docker Swarm setup as Traefik it ran from my local machine without any changes to its configuration.

I just found a setting called send-proxy and send-proxy-v2 which can be appended to the server definition in HaProxy. For example:

server lb1 10.1.1.1:8000 check send-proxy-v2

It may be that HaProxy isn't sending the PROXY-header at all -- or as you said incorrectly. I will test this today. Thanks again for the explanation.

[ju5t]

It turns out send-proxy-v2 is not supported. Traefik returns a 400 Bad Request, probably because v2 doesn't use plain text but binary. Unfortunately send-proxy didn't make a difference. I'm not 100% sure what to expect in the output from the whoami container but there is no PROXY-header (my knowledge of the HTTP protocol isn't great though..).

If you believe this is not a bug in Traefik but a configuration error on our end feel free to close the issue and let me know. In that case I'll take the discussion to Slack.

[aantono]

@ju5t Can you do a TCP packet capture to see the raw request going out? The whoami will not show you the PROXY section, because it is not an HTTP Header, but an initiating protocol line, so it would not come from the Headers collection. You can find the spec details at https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt.

[ju5t]

@aantono sure.

PROXY TCP4 172.17.0.2 PUBLIC_IP_SERVER 49760 8000

172.17.0.2 is the IP address of the container running HaProxy. PUBLIC_IP_SERVER is the public address to which we're sending traffic. Apparently HaProxy gets it wrong. I don't always trust Docker's iptables-magic so for now I put the blame on that.

[ju5t]

A basic installation of HaProxy works. It sends the correct PROXY-header and Traefik handles everything as expected now. I'm pretty sure this is due to Docker's iptables rules but we can work with HaProxy running on a host too.

Thanks so much for all input. This turns out to be an external configuration issue and has nothing to do with Traefik. I will close the issue.