Unable to retrieve user's IP address in docker swarm mode

Output of docker version:

Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:00:36 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:00:36 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 155
 Running: 65
 Paused: 0
 Stopped: 90
Images: 57
Server Version: 1.12.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 868
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host overlay null bridge
Swarm: active
 NodeID: 0ddz27v59pwh2g5rr1k32d9bv
 Is Manager: true
 ClusterID: 32c5sn0lgxoq9gsl1er0aucsr
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot interval: 10000
  Heartbeat tick: 1
  Election tick: 3
 Dispatcher:
  Heartbeat period: 5 seconds
 CA configuration:
  Expiry duration: 3 months
 Node Address: 172.31.24.209
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-92-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.42 GiB
Name: ip-172-31-24-209
ID: 4LDN:RTAI:5KG5:KHR2:RD4D:MV5P:DEXQ:G5RE:AZBQ:OPQJ:N4DK:WCQQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: panj
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

Steps to reproduce the issue:

run following service which publishes port 80

docker service create \
--name debugging-simple-server \
--publish 80:3000 \
panj/debugging-simple-server

Try connecting with http://<public-ip>/.

Describe the results you received:
Neither ip nor header.x-forwarded-for is the correct user's IP address.

Describe the results you expected:
ip or header.x-forwarded-for should be user's IP address. The expected result can be archieved using standalone docker container docker run -d -p 80:3000 panj/debugging-simple-server. You can see both of the results via following links,
http://swarm.issue-25526.docker.takemetour.com:81/
http://container.issue-25526.docker.takemetour.com:82/

Additional information you deem important (e.g. issue happens only occasionally):
This happens on both global mode and replicated mode.

I am not sure if I missed anything that should solve this issue easily.

In the meantime, I think I have to do a workaround which is running a proxy container outside of swarm mode and let it forward to published port in swarm mode (SSL termination should be done on this container too), which breaks the purpose of swarm mode for self-healing and orchestration.

@aluzzardi

/cc @aluzzardi @mrjana ptal

@PanJ

@PanJ can you please share some details on how debugging-simple-server determines the ip ? Also what is the expectation if a service is scaled to more than 1 replica across multiple hosts (or global mode) ?

@mavenugo

@mavenugo it's koa's request object which uses node's remoteAddress from net module. The result should be the same for any other libraries that can retrieve remote address.

The expectation is that ip field should always be remote address regardless of any configuration.

@PanJ

@PanJ you still use your workaround or found some better solution?

@PanJ

@PanJ When I run your app as a standalone container..

docker run -it --rm -p 80:3000 --name test panj/debugging-simple-server

and access the published port from another host I get this

vagrant@net-1:~$ curl 192.168.33.12
{"method":"GET","url":"/","header":{"user-agent":"curl/7.38.0","host":"192.168.33.12","accept":"*/*"},"ip":"::ffff:192.168.33.11","ips":[]}
vagrant@net-1:~$

192.168.33.11 is the IP of the host in which I am running curl. Is this the expected behavior ?

@sanimej

@sanimej Yes, it is the expected behavior that should be on swarm mode as well.

@marech

@marech I am still using the standalone container as a workaround, which works fine.

In my case, there are 2 nginx intances, standalone and swarm instances. SSL termination and reverse proxy is done on standalone nginx. Swarm instance is used to route to other services based on request host.

@PanJ

@PanJ The way the published port of a container is accessed is different in swarm mode. In the swarm mode a service can be reached from any node in the cluster. To facilitate this we route through an ingress network. 10.255.0.x is the address of the ingress network interface on the host in the cluster from which you try to reach the published port.

@sanimej

@sanimej I kinda saw how it works when I dug into the issue. But the use case (ability to retrieve user's IP) is quite common.

I have limited knowledge on how the fix should be implemented. Maybe a special type of network that does not alter source IP address?

Rancher is similar to Docker swarm mode and it seems to have expected behavior. Maybe it is a good place to start.

@sanimej

@sanimej good idea could be add all IPs to X-Forwarded-For header if its possible then we can see all chain.

@PanJ hmm, and how your nignx standalone container communicate to swarm instance, via service name or ip? Maybe can share nginx config part where you pass it to swarm instance.

@marech

@marech standalone container listens to port 80 and then proxies to localhost:8181

server {
  listen 80 default_server;
  location / {
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
    proxy_pass          http://localhost:8181;
    proxy_read_timeout  90;
  }
}

If you have to do SSL termination, add another server block that listens to port 443, then do the SSL termination and proxies to localhost:8181 as well

Swarm mode's nginx publishes 8181:80 and routes to another service based on request host.

server {
  listen 80;
  server_name your.domain.com;
  location / {
    proxy_pass          http://your-service:80;
    proxy_set_header Host $host;
    proxy_read_timeout  90;
  }
}

server {
  listen 80;
  server_name another.domain.com;
  location / {
    proxy_pass          http://another-service:80;
    proxy_set_header Host $host;
    proxy_read_timeout  90;
  }
}

@struanb

@struanb

Yes, that makes sense. Thanks for the clear explanation. :)

Traefik also provides routing based on hostname. So I think I'll likely stick to using it.

Thanks again, your help was appreciated!

@jerrac

@jerrac - as also explained here: newsnowlabs/docker-ingress-routing-daemon#24 (comment) :-

To be clear, DIND exists to transform Docker's ingress routing mesh to use policy routing instead of SNAT, to redirect client traffic to service nodes. It will only work to preserve the client IP if incoming requests directly reach a load-balancer node on a port published for a service via the ingress routing mesh. DIND is a network-layer tool (IPv4) and cannot inspect or modify HTTP headers.

I understand Traefik has often been used as a reverse proxy to work around the same limitation as DIND. In this model, incoming requests much directly reach the reverse proxy, which presumably must not be using the ingress routing mesh, but instead have its ports published using host mode, and be launched using --mode global. The Traefik reverse proxy will see the client IP of requests and can add these to the XFF header before reverse proxying them to an internal application service.

DIND therefore exists to solve a similar problem as a Traefik reverse proxy service placed in front of an internal application service, but without the need for the extra Traefik service (or for proxying, or for introduction/modification of XFF headers) and therefore without modification of the application service (if it doesn't natively support XFF headers).

Combining DIND with Traefik should allow Traefik itself to be deployed using the ingress routing mesh, which could be useful if Traefik is providing additional benefits in one's setup.

However, I'm not sure I can see a use-case for combining DIND with an internal application service published via the ingress routing mesh, and still fronted by a Traefik reverse proxy. Since the reverse proxy node is the client for the internal application service request, doing this will just expose the Docker network IP of that node, instead of the ingress network IP, to the internal application service.

Hope this makes sense.

hi! thanks for posting this. any idea if this comes with any unexpected limitations (as compared to how k3s ingress lbs do it)?
I'd really like to keep my docker composes and run swarm for now, and anything helping me see the originating ip (which is an obvious expectation) is crucial

Are there any recommended solutions yet?

This bug eliminates the whole class of applications - basicaly, every application that needs to get the IP address of a user. So, no dockerized firewalls, no anti-spam/anti-fraud detection software, no IP address whitelists. Awful.

@xucian

@xucian @bohdan-shulha Have you tried Docker Ingress Routing Daemon (DIRD, formerly DIND), or are there particulars of your server/infrastructure setup that prevent its use?

If so, I'd be keen to hear what those are. Several years on, we continue to run DIRD stably in production at NewsNow.

I know it's not an officially recommended solution but, as explained elsewhere, once DIRD is installed across the nodes of a Docker swarm, there's really nothing else you have to do for swarm services to see the originating client IP. This would therefore allow a range of software, like anti-spam, to be deployed using docker service create.

No, not yet tried, sorry.

Had to ask just because the thread is soooo long already (8-year long discussion puts it's fingerprints) and I was curious whether there is an official solition to the problem.

Thanks once more for the tips. <3

@struanb

Nice, seems to be a working solution.

Thanks, @struanb !

Docker Ingress Routing Daemon

finally some light, thanks for sharing!

Is there any risk to installing DIRD on an existing swarm? will it break the normal overlay routing or any other downside?

@scyto

@scyto There is always risk, modifying a live service, but it should work if done carefully. My recommendations for calling DIRD safely are:

work out the <ips> argument for --ingress-gateway-ips <ips> carefully in advance - getting this wrong will easily break things;
- a command to run on a manager to generate <ips> is docker node ls --format "{{.Hostname}}" | pdsh -N -R ssh -w - "docker network inspect ingress --format '{{index (split (index .Containers \"ingress-sbox\").IPv4Address \"/\") 0}}' 2>/dev/null" | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -k 4,4' (pdsh and ssh needed) though N.B. this <ips> list is overkill if you only use a subset of nodes as load balancers;
- be sure to embed <ips> in double quotes if it contains spaces e.g. --ingress-gateway-ips "10.0.0.2 10.0.0.3"; otherwise separate IPs with commas e.g. --ingress-gateway-ips 10.0.0.2,10.0.0.3;
use the --preexisting option so that when DIRD is launched it applies policy routing rules to each matching preexisting container;
use --services <services>, --tcp-ports <tcp-ports> and if you need it --udp-ports <udp-ports> to whitelist DIRD behaviour for only the specific swarm services and ports that you need to;
use --iptables-wait or --iptables-wait-seconds <n> to avoid possible errors resulting from contention with other firewall apps for the iptables lock;
using [OPTIONS] derived accordingly, prepare a dird.service systemd unit that launches docker-ingress-routing-daemon --install [OPTIONS] and deploy it (to /etc/systemd/system, or /usr/local/lib/systemd/system according to your distribution) to all swarm nodes (whether load balancer and/or service container nodes) BUT without running it;
enable the systemd unit across all nodes using e.g. docker node ls --format "{{.Hostname}}" | pdsh -R ssh -w - 'systemctl daemon-reload; systemctl enable dird'
launch across all nodes as quickly and as simultaneously as possible, using e.g. docker node ls --format "{{.Hostname}}" | pdsh -R ssh -w - systemctl start dird

Even using the pdsh launch method, a few (milli)seconds of downtime might be unavoidable. I'll look into whether there is any way to avoid even that, and if I find a way I'll update this comment.

@scyto

@scyto From my tests I've found that during the migration to DIRD, if you can restrict incoming public requests to a subset of your nodes (i.e. nodes you nominate as "load balancers"), a recipe that works very smoothly is as follows:

First, launch DIRD, fully configured, on all nominated non-load-balancer nodes; this will not break connection handling, as long as the incoming connections are terminated by a nominated load-balancer node not yet running DIRD.
Second, bring up DIRD simultaneously on all remaining nodes, i.e. on the nominated load-balancer nodes.

Put another way, before launching DIRD on some node(s), make sure you have previously removed their public load-balancer IPs from the pool of load balancer endpoints your public requests are reaching. Repeat the process as many times as needed until you are left with a minimal set of load balancer node(s). Finally launch DIRD on the remaining load balancer node(s).

This works because:

Nodes already running DIRD will be able to load-balance only to containers on nodes already running DIRD.
Nodes not yet running DIRD will be able to load-balance to containers, both on nodes already running and not yet running DIRD.

Hope this helps.

Any updates on progress for this?

is not possible add a special header with the remote ip? like cloudflare do with CF-Connecting-IP

is not possible add a special header with the remote ip? like cloudflare do with CF-Connecting-IP

This is IPv6 and not application level HTTP...

infra->docker: Preserve client IPs by bypassing Swarm ingress for nginx

I'd like to explore an opt-in source-IP-preserving mode for published Swarm ingress ports.

Our use case is two HAProxy replicas behind the same published ports, with a single public endpoint for failover. Client IPs are needed for access rules and audit logging. The initial target is Docker Engine on Linux; Docker Desktop's VM port-forwarding layer would need separate validation.

In the inspected Moby revision, configureFWMark() installs the ingress SNAT rule. That rule also ensures replies return through the correct ingress gateway. Preserving the source IP therefore needs an explicit return-routing mechanism. An HTTP header added after ingress cannot reconstruct that IP.

Before implementing this, I'd appreciate guidance on:

Whether an optional ingress mode, retaining the current NAT mode by default, is a direction the networking maintainers would support.
Whether the existing DSR code is intended to support published ingress ports, or connection-aware policy routing would be the better starting point.
What initial scope and regression coverage would be required for a PR.

Source analysis is prepared; there is no implemented or runtime-validated Engine patch yet. Any proposed routing changes would need to cover external clients, different published and target ports, task replacement and remote backends on a second Swarm node.

Navigation Menu

Repository navigation

Unable to retrieve user's IP address in docker swarm mode #25526

350 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Unable to retrieve user's IP address in docker swarm mode #25526

Description

Activity

thaJeztah commented on Aug 9, 2016

mavenugo commented on Aug 9, 2016

PanJ commented on Aug 9, 2016

marech commented on Sep 19, 2016

sanimej commented on Sep 19, 2016

PanJ commented on Sep 19, 2016

PanJ commented on Sep 19, 2016

sanimej commented on Sep 19, 2016

PanJ commented on Sep 19, 2016

marech commented on Sep 20, 2016

PanJ commented on Sep 20, 2016

350 remaining items

jerrac commented on Jun 16, 2022

xucian commented on Feb 25, 2024

bohdan-shulha commented on Aug 25, 2024

struanb commented on Aug 25, 2024

bohdan-shulha commented on Aug 25, 2024

bohdan-shulha commented on Aug 25, 2024

xucian commented on Aug 29, 2024

scyto commented on Nov 28, 2024

struanb commented on Nov 28, 2024

struanb commented on Nov 30, 2024

bodhi-one commented on Sep 27, 2025

devMls commented on Feb 24, 2026

kolbma commented on Feb 28, 2026

rpx99 commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions