New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to retrieve user's IP address in docker swarm mode #25526
Comments
/cc @aluzzardi @mrjana ptal |
@PanJ can you please share some details on how debugging-simple-server determines the |
@PanJ you still use your workaround or found some better solution? |
@PanJ When I run your app as a standalone container..
and access the published port from another host I get this
192.168.33.11 is the IP of the host in which I am running curl. Is this the expected behavior ? |
@sanimej Yes, it is the expected behavior that should be on swarm mode as well. |
@marech I am still using the standalone container as a workaround, which works fine. In my case, there are 2 nginx intances, standalone and swarm instances. SSL termination and reverse proxy is done on standalone nginx. Swarm instance is used to route to other services based on request host. |
@PanJ The way the published port of a container is accessed is different in swarm mode. In the swarm mode a service can be reached from any node in the cluster. To facilitate this we route through an |
@sanimej I kinda saw how it works when I dug into the issue. But the use case (ability to retrieve user's IP) is quite common. I have limited knowledge on how the fix should be implemented. Maybe a special type of network that does not alter source IP address? Rancher is similar to Docker swarm mode and it seems to have expected behavior. Maybe it is a good place to start. |
@marech standalone container listens to port
If you have to do SSL termination, add another server block that listens to port Swarm mode's nginx publishes
|
In our case, our API RateLimit and other functions is depend on the user's ip address. Is there any way to skip the problem in swarm mode? |
I've also run into the issue when trying to run logstash in swarm mode (for collecting syslog messages from various hosts). The logstash "host" field always appears as 10.255.0.x, instead of the actual IP of the connecting host. This makes it totally unusable, as you can't tell which host the log messages are coming from. Is there some way we can avoid translating the source IP? |
+1 for a solution for this issue. Without the ability to retrieve user's IP prevents us from using monitoring solutions like Prometheus. |
Perhaps the linux kernel IPVS capabilities would be of some use here. I'm guessing that the IP change is taking place because the connections are being proxied in user space. IPVS, on the other hand, can redirect and load balance requests in kernel space without changing the source IP address. IPVS could also be good down the road for building in more advanced functionality, such as different load balancing algorithms, floating IP addresses, and direct routing. |
For me, it would be enough if I could somehow find out the relation between the virtual IP and the IP of the server the endpoint belongs to. That way, when Prometheus send an alert related to some virtual IP, I could find out what is the affected server. It would not be a good solution but it would be better than nothing. |
@vfarcic I don't think that's possible with the way it works now. All client connections come from the same IP, so you can't translate it back. The only way that would work is if whatever is doing the proxy/nat of the connections saved a connection log with timestamp, source ip, and source port. Even then, it wouldn't be much help in most use cases where the source IP is needed. |
I probably did not explain well the use case. I use Prometheus that is configured to scrap exporters that are running as Swarm global services. It uses tasks.<SERVICE_NAME> to get the IPs of all replicas. So, it's not using the service but replica endpoints (no load balancing). What I'd need is to somehow figure out the IP of the node where each of those replica IPs come from. |
I just realized the "docker network inspect <NETWORK_NAME>" provides information about containers and IPv4 addresses of a single node. Can this be extended so that there is a cluster-wide information of a network together with nodes? Something like:
Note the addition of the "Node". If such information would be available for the whole cluster, not only a single node with the addition of a |
I agree with @dack , given the ingress network is using IPVS, we should solve this issue using IPVS so that the source IP is preserved and presented to the service correctly and transparently. The solution need to work at the IP level so that any service that are not based on HTTP can still work properly as well (Can't rely on http headers...). And I cant stress out how important this is, without it, there are many services that simply cant operate at all in swarm mode. |
That's how HaProxy is solving this issue: http://blog.haproxy.com/2012/06/05/preserve-source-ip-address-despite-reverse-proxies/ |
@kobolog might be able to shed some light on this matter given his talk on IPVS at DockerCon. |
How about setting up traefik reverse proxy on top of your docker swarm and then change it's port mode to host: What it does is what you are looking for: change both X-Forwarded-For and X-Real-Ip headers to correct values and you keep all swarm capabilities. |
@dcniko using traefik in front of my application containers was how I tackled this in the past and it worked fine for the needs we had at the time :) |
We've now released v3.1.0 of https://github.com/newsnowlabs/docker-ingress-routing-daemon, which modifies docker's ingress mesh routing to expose true client IPs to service containers:
As far as I know, the docker-ingress-routing-daemon is the most lightweight way to access client IPs from within containers launched by docker services. Summary of features:
Please check it out and raise any issues you find. |
@struanb I have tried your script, but, when installed access to all containers stop. Any idea? Tried on 2 different installation. |
@zimbres If you can raise an issue at https://github.com/newsnowlabs/docker-ingress-routing-daemon/issues outlining your setup, DIND version, and so on, and I'll be pleased to respond there. N.B. v3.3.0 has just been released, which is necessary to upgrade to for UDP-based services like DNS. |
I'm currently fighting this issue. Specifically my X-Forwarded-For header is having all except my external load balancer's floating ip stripped off of it. On my non-Docker systems that header has both the LB and the client ip stored in it. Does Docker Swarm mess with that header? Traefik is also involved, so I've posted on their forums as well. Full details of my set up are there. I am trying @struanb 's latest docker-ingress-routing-daemon, but the only change I saw was that I went from only seeing Docker ip's in my logs to also seeing my LB's floating ip. Anyway, if any Docker devs are still watching, judging from the many many posts all over the internet I found while searching, getting the real client ip from inside a Docker container is both wanted and needed. And apparently really hard to make happen. It would be awesome if you could figure out something that would make it simple. |
Hi @jerrac. Thank you for trying Docker Ingress Routing Daemon (DIND). Please raise an issue describing the symptoms you're experiencing and we'll be happy to help. DIND is used in production, so we know it does work (at least in many standard configurations). |
FWIW, I've used Traefik in host mode to avoid this issue - details at https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik/ |
@funkypenguin Huh, I thought that since I just tried adding |
IIRC it was necessary to use the "long form" of the ports definition, like so:
|
@funkypenguin Yep, that's what I'm using. ports:
- target: 80
published: 80
protocol: tcp
mode: host
- target: 443
published: 443
protocol: tcp
mode: host |
Cross reference: https://github.com/b-data/docker-swarm-ipv6-nftables/blob/main/NOTES.md#træfik-proxy, i.e.
https://whoami.b-data.ch is traefik/whoami deployed as discussed above. |
@jerrac what version of Docker are you using? Mine doesn't complain about not supporting network mode upon deploy...
|
@jerrac What |
@funkypenguin I'm on Docker 20.10.17. Here is my full docker-compose file: https://pastebin.com/wcCw5i7C I've been applying it via |
Re: traefik - don't you also have to deploy it as |
Update (posting this in all 3 places I asked for help...): A while later, after a meeting, I randomly decided to try terminating https at my external load balancer. After re-configuring Traefik to not redirect to 443, and configuring my service to use the 80 entrypoint, I can see my client ip in my container logs. Does this make sense? |
@jerrac - as also explained here: newsnowlabs/docker-ingress-routing-daemon#24 (comment) :- To be clear, DIND exists to transform Docker's ingress routing mesh to use policy routing instead of SNAT, to redirect client traffic to service nodes. It will only work to preserve the client IP if incoming requests directly reach a load-balancer node on a port published for a service via the ingress routing mesh. DIND is a network-layer tool (IPv4) and cannot inspect or modify HTTP headers. I understand Traefik has often been used as a reverse proxy to work around the same limitation as DIND. In this model, incoming requests much directly reach the reverse proxy, which presumably must not be using the ingress routing mesh, but instead have its ports published using host mode, and be launched using DIND therefore exists to solve a similar problem as a Traefik reverse proxy service placed in front of an internal application service, but without the need for the extra Traefik service (or for proxying, or for introduction/modification of XFF headers) and therefore without modification of the application service (if it doesn't natively support XFF headers). Combining DIND with Traefik should allow Traefik itself to be deployed using the ingress routing mesh, which could be useful if Traefik is providing additional benefits in one's setup. However, I'm not sure I can see a use-case for combining DIND with an internal application service published via the ingress routing mesh, and still fronted by a Traefik reverse proxy. Since the reverse proxy node is the client for the internal application service request, doing this will just expose the Docker network IP of that node, instead of the ingress network IP, to the internal application service. Hope this makes sense. |
Yes, that makes sense. Thanks for the clear explanation. :) Traefik also provides routing based on hostname. So I think I'll likely stick to using it. Thanks again, your help was appreciated! |
Output of
docker version
:Output of
docker info
:Additional environment details (AWS, VirtualBox, physical, etc.):
Steps to reproduce the issue:
http://<public-ip>/
.Describe the results you received:
Neither
ip
norheader.x-forwarded-for
is the correct user's IP address.Describe the results you expected:
ip
orheader.x-forwarded-for
should be user's IP address. The expected result can be archieved using standalone docker containerdocker run -d -p 80:3000 panj/debugging-simple-server
. You can see both of the results via following links,http://swarm.issue-25526.docker.takemetour.com:81/
http://container.issue-25526.docker.takemetour.com:82/
Additional information you deem important (e.g. issue happens only occasionally):
This happens on both
global
mode andreplicated
mode.I am not sure if I missed anything that should solve this issue easily.
In the meantime, I think I have to do a workaround which is running a proxy container outside of swarm mode and let it forward to published port in swarm mode (SSL termination should be done on this container too), which breaks the purpose of swarm mode for self-healing and orchestration.
The text was updated successfully, but these errors were encountered: