IPv6 is no longer proxied by default anymore

[terencehonles]

Description

IPv6 is no longer proxied by default as of 20.10.2 (#41805). This is expanded in greater detail in moby/libnetwork#2607, but downgrading to 20.10.1 will restore the previous behavior. This change should be either documented or fixed to not surprise users.

Steps to reproduce the issue:

1. Bind NGINX with published ports --publish 443:443 and get "IPv6 support" (proxy is just accepting any address and proxying to the container. This may be abusing the proxy, but it works.)
2. Upgrade to Docker 20.10.2 and now IPv6 is broken and curl returns "Connection refused"

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:17:43 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       f001486
  Built:            Tue Dec 15 04:32:52 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 22
  Running: 22
  Paused: 0
  Stopped: 0
 Images: 14
 Server Version: 20.10.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-1034-aws
 Operating System: Ubuntu 20.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.786GiB
 Name: web1.dev.internal.nexleaf.org
 ID: 7G37:MVZI:VTVY:Q5UJ:CDA6:UKRI:7PF7:LLM6:YFAR:LTZR:6AQ4:Y7ZF
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No blkio weight support
WARNING: No blkio weight_device support

[fbezdeka]

In my case I'm unable to get an IPv6 only port forwarding working.
2a03:dead:beef::1 is one of the hosts IPv6 addresses, so it's not a "no ipv6 proxying by default anymore" in my case.

sudo docker run \
    --hostname gitlab \
    --env 'GITLAB_PORT=443' \
    --env 'GITLAB_HTTPS=true' \
    --publish [2a03:dead:beef::1]:2222:22 \
    --publish [2a03:dead:beef::1]:8080:80 \
    --publish [2a03:dead:beef::1]:8443:443 \
    --name gitlab \
    --restart always \
    --volume /srv/gitlab/config:/etc/gitlab:Z \
    --volume /srv/gitlab/logs:/var/log/gitlab:Z \
    --volume /srv/gitlab/data:/var/opt/gitlab:Z \
    gitlab/gitlab-ce:latest

That worked with 20.10.1 but does no longer work with 20.10.2. That's a very unexpected behaviour. I guess there are many IPv6 only setups out there that are now broken.

[anuragbhatia]

Upgrade to Docker 20.10.2 and now IPv6 is broken and curl returns "Connection refused"

Having exactly the same issue. In version before 20.10.2 IPv6 port binding would "stuck" even when the container was taken down via docker-compose. And it would be released only when docker-engine was restarted.

Now in 20.10.2 IPv6 binding is not working at all anymore.

anurag@server8:~$ docker --version
Docker version 20.10.2, build 2291f61
anurag@server8:~$ docker container inspect npm_app_1 | jq .[].HostConfig.PortBindings
{
  "443/tcp": [
    {
      "HostIp": "78.47.110.149",
      "HostPort": "443"
    },
    {
      "HostIp": "2a01:4f8:c0c:681b::1",
      "HostPort": "443"
    }
  ],
  "80/tcp": [
    {
      "HostIp": "78.47.110.149",
      "HostPort": "80"
    },
    {
      "HostIp": "2a01:4f8:c0c:681b::1",
      "HostPort": "80"
    }
  ]
}
anurag@server8:~$

This shows the port is bind in IPv4 and IPv6 but host only shows IPv4 ports binded.

anurag@server8:~$ sudo netstat -ltnp  | grep 80
tcp        0      0 78.47.110.149:80        0.0.0.0:*               LISTEN      30376/docker-proxy
anurag@server8:~$ sudo netstat -ltnp  | grep 443
tcp        0      0 78.47.110.149:443       0.0.0.0:*               LISTEN      30364/docker-proxy
anurag@server8:~$

[cpuguy83]

This should be fixed by moby/libnetwork#2608

[seblu]

I give it a try testing a new Arch package release but I still have the same issue:

with version 20.10.1

# ss -lup|grep docker
UNCONN 0      0                                     *:25826                    *:*    users:(("docker-proxy",pid=1364157,fd=4))    

with a patched version 20.10.2:

# ss -lup|grep docker
UNCONN        0             0                                             0.0.0.0:25826                             0.0.0.0:*            users:(("docker-proxy",pid=738727,fd=4))             

[janpio]

20.10.2 recently hit the Ubuntu18 images we are using on Github Actions CI, and caused our Prisma MySQL and SQL Server database clients to fail in unexpected ways - which turned out to be bugs on our side: prisma/prisma#5499 If our investigation is right, this was caused by the behavior described above (and in linked issues).

[Trolldemorted]

I have recently migrated to a new server and noticed that all my containers don't respond to ipv6 any more (docker from docker's apt repos, same daemon.json, same docker-compose files).

This is the ss -tulpen output from the old server (docker version 19.03.11)

Netid  State    Recv-Q   Send-Q      Local Address:Port     Peer Address:Port
tcp    LISTEN   0        128                     *:80                  *:*       users:(("docker-proxy",pid=2313,fd=4)) ino:37291 sk:4 v6only:0 <->
tcp    LISTEN   0        128                     *:443                 *:*       users:(("docker-proxy",pid=2300,fd=4)) ino:40102 sk:7 v6only:0 <->

This is the ss -tulpen output from the new server (docker version 20.10.3)

Netid   State    Recv-Q   Send-Q     Local Address:Port      Peer Address:Port  Process
tcp     LISTEN   0        4096             0.0.0.0:80             0.0.0.0:*      users:(("docker-proxy",pid=1069,fd=4)) ino:46593 sk:4 <->
tcp     LISTEN   0        4096             0.0.0.0:443            0.0.0.0:*      users:(("docker-proxy",pid=1058,fd=4)) ino:46543 sk:9 <->                    

As you can see, docker-proxy used to bind on ipv6 with v6only:0, but now does bind on ipv4 only, so I think we are being hit by this bug.

When can we expect a release where this is fixed?

[Trolldemorted]

Any idea when you will fix this? This bug has been breaking prod environments for months.

Is this caused by anything that would justify the long wait? My intuition was you'd just have to bind to everything again.

[haohetao]

This issue has been bothering me for a long time and I hope to release a fix version soon