[Traefik] Rootless Docker with slirp4netns doesn't show user IP

[aureliensoria]

Description

When using rootless docker with slip4netns an internal IP is shown instead of the real one.
Example:

Hostname: 44220974d172
IP: 127.0.0.1
IP: 172.20.0.18
RemoteAddr: 172.20.0.7:33402
GET / HTTP/1.1
Host: XXX
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US;q=0.7,en;q=0.3
Dnt: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Te: trailers
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 172.20.0.1
X-Forwarded-Host: XXX
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 7770b7ffda5a
X-Real-Ip: 172.20.0.1

Reproduce

Add
`[Service]
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=slirp4netns"

in ~/.config/systemd/user/docker.service.d/override.conf `
Reload the docker daemon and restart it.

Run two containers : traefik and traefik/whoami instance.

Expected behavior

X-Real-Ip: Should contain the real user IP instead of the IP of the docker network gateway.

docker version

Client: Docker Engine - Community
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        a5ee5b1
 Built:             Thu Feb  9 19:46:54 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       bc3805a
  Built:            Thu Feb  9 19:46:54 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.18
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.1.0
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       slirp4netns
  StateDir:         /tmp/rootlesskit3603596150
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.16.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 29
  Running: 29
  Paused: 0
  Stopped: 0
 Images: 29
 Server Version: 23.0.1
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.1.0-5-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 15.63GiB
 Name: XXX
 ID: X2LR:NAD7:AJMU:YOIV:TZY3:EPGN:VRCK:ZU3A:VVGV:M23Y:PFX2:B5PH
 Docker Root Dir: /home/virt/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional Info

No response

[aureliensoria]

In addition here is the slirp4netns launch options

ps aux | grep slirp
virt     2675625  0.0  0.1 1311380 20948 ?       Ssl  18:25   0:00 rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=slirp4netns --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
virt     2675636  0.0  0.1 1163324 19712 ?       Sl   18:25   0:00 /proc/self/exe --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=slirp4netns --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
virt     2675654  0.3  0.0  14216  4516 ?        S    18:25   0:02 slirp4netns --mtu 65520 -r 3 --disable-host-loopback --api-socket /tmp/rootlesskit3603596150/.s4nn.sock --enable-sandbox --enable-seccomp 2675636 tap0

Thank you !

[danishprakash]

Mind If I take a look at this?

[AkihiroSuda]

Run two containers : traefik and traefik/whoami instance.

Please provide a full reproducer.
Also wondering if this is reproducible with a simple docker run -p 8080:80 nginx. (Not reproducible for me)

Mind If I take a look at this?

You don't need to ask for a permission 🙂

[aureliensoria]

When using docker run -p 8080:80 nginx, I can't reproduce my issue. I will provide you my full docker-compose ASAP.

[aureliensoria]

Here is a dump of my traefik configuration. It was working well when using standard docker instead of rootless docker.

version: '3.4'

services:

  # External dependencies
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: always
    ports:
      - "SERVER_IP:80:80"
      - "SERVER_IP:443:443"
      - "SERVER_IP:9080:8080"
    volumes:
      - "/data/cloud/traefik/:/etc/traefik/"
      - "/run/user/1001/docker.sock:/var/run/docker.sock:ro"
    labels:
      - "traefik.http.routers.traefik.rule=Host(`traefik.DOMAIN_NAME`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.tls=true"
      - "traefik.http.routers.traefik.tls.certresolver=myresolver"
      - "traefik.http.routers.traefik.middlewares=sso@docker,https_headers@file"
    networks:
      - default
      - traefik

  whoami:
    image: traefik/whoami
    restart: always
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.who.rule=Host(`who.DOMAIN_NAME`) || Host(`who.DOMAIN_NAME2`)"
      - "traefik.http.routers.who.service=svc_who"
      - "traefik.http.routers.who.tls=true"
      - "traefik.http.routers.who.tls.certresolver=myresolver"
      - "traefik.http.routers.who.middlewares=admin@file,https_headers@file"
      - "traefik.http.services.svc_who.loadbalancer.server.port=80"
    networks:
      - default
      - traefik

Traefik configuration

[global]
  sendAnonymousUsage = false
  checkNewVersion = true

[log]
  level = "ERROR"
  filePath = "/etc/traefik/traefik.log"

[accessLog]
  filePath = "/etc/traefik/access.log"
    [accessLog.fields.headers]
      defaultMode = "keep"
      [accessLog.fields.headers.names]
        "ClientAddr" = "keep"

[entryPoints]
  [entryPoints.web]
      address = ":80"
      [entryPoints.web.http.redirections]
        [entryPoints.web.http.redirections.entryPoint]
          to = "websecure"

  [entryPoints.websecure]
      address = ":443"

[serversTransport]
  insecureSkipVerify = true

[providers]
  docker = true
  [providers.file]
    directory = "/etc/traefik/"
    watch = true

[api]
  dashboard = true

[tls.options]
  [tls.options.default]
    minVersion = "VersionTLS12"
  [tls.options.mintls13]
    minVersion = "VersionTLS13"

[certificatesResolvers.myresolver.acme]
  email = "admin@DOMAIN_NAME"
  storage = "/etc/traefik/acme.json"
  [certificatesResolvers.myresolver.acme.httpChallenge]
    # used during the challenge
    entryPoint = "web"

[sam-thibault]

Given that docker run works as expected, this seems to be a traefik specific issue. I will move to a discussion.

[Georgekonsec]

Actually this does not seem to be a traefik specific issue. The problem exists if you run haproxy, httpd or nginx applications and it is related how you run the docker container.

It works if you run the following docker container:

docker run --rm -d --name without_ip -p 7000:80 httpd
docker logs -f without_ip

# Then curl the server to that port
curl <hostname>:7000

but it returns docker internal IP, if you run the same docker container with a Server IP:

docker run --rm -d --name with_ip -p <server_ip>:7001:80 httpd
docker logs -f with_ip

# Then curl the server to that port
curl <hostname>:7001

We would like to be able to expose an application to a specific Server ip and not to all of them (0.0.0.0).

After checking slirp4netns there is an option (--outbound-addr) to define preferred outbound ipv4/ipv6 addresses or interface names.
Could we provide an environment variable for dockerd-rootless.sh script like DOCKERD_ROOTLESS_ROOTLESSKIT_OUTBOUND_ADDR so to be able to define an ip address/network interface ?

Could we use/check pasta network driver because it handles differently the network interfaces and this probably fixes the exact problem with the Docker Internal IP instead of the client IP.