Bind for ip:port failed: port is already allocated

[hernandanielg]

Description

Container fails to create or start due to ip:port is already allocated, but no container is exposing the ip:port address

Container is in Created status as it failed to start when creating

root@some-host:~# docker ps -af name=some-container
CONTAINER ID        IMAGE                                                                 COMMAND                 CREATED             STATUS              PORTS               NAMES
b521a2a39d9f        some-image:some-tag   "/home/entrypoint.sh"   30 seconds ago      Created                                 some-container

Container fails to start with error: port is already allocated

root@some-host:~# docker start some-container
Error response from daemon: driver failed programming external connectivity on endpoint some-container (f0728f12adf3656f4b6772526993112a3819c93082b4619dd2947ee1f2164b6f): Bind for 10.76.16.126:21425 failed: port is already allocated
Error: failed to start containers: some-container

Port is allocated by docker daemon

root@some-host:~# netstat -plunt | grep 10.76.16.126:21425
tcp        0      0 10.76.16.126:21425      0.0.0.0:*               LISTEN      513034/dockerd

root@some-host:~# cat /var/run/docker.pid
513034

there's a DNAT iptables rule for the port

root@some-host:~# iptables -vnL -t nat | grep :21425
11287  677K DNAT       tcp  --  *      *       0.0.0.0/0            10.76.16.126         tcp dpt:21425 to:172.17.0.173:25

Noted that docker internal IP is in use by another container, but it is not exposing ports

root@some-host:~# docker network inspect bridge 
...
            "2a90f67a9d246603fb55f3e0bf9f5eb2fb49fbc077b98b2d765a4cd47f383d45": {
                "Name": "some-other-container",
                "EndpointID": "5879a83280f69a082698462cd1d8890b7852b6b0bb76a8ff06631b19b0833e3c",
                "MacAddress": "02:42:ac:11:00:ad",
                "IPv4Address": "172.17.0.173/16",
                "IPv6Address": ""
            },

root@some-host:~# docker ps -f name=some-other-container
CONTAINER ID        IMAGE                                                                    COMMAND                 CREATED             STATUS              PORTS               NAMES
2a90f67a9d24        some-other-images:some-other-tag   "/home/entrypoint.sh"   14 hours ago        Up 14 hours                             some-other-container

root@some-host:~# docker port some-other-container
root@some-host:~#

Socket is in LISTEN state

root@some-host:~# ss -plunt | grep :21425                          
tcp    LISTEN     0      128    10.76.16.126:21425                 *:*                   users:(("dockerd",pid=513034,fd=1791))

root@some-host:~# ls -l /proc/513034/fd/1791
lrwx------ 1 root root 64 Mar  8 05:00 /proc/513034/fd/1791 -> socket:[5026616]

root@some-host:~# grep 5026616 /proc/net/tcp
 585: 7E104C0A:53B1 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 5026616 1 ffff9f88fb3ec000 100 0 0 10 0

root@some-host:~# lsof -n -i :21425
COMMAND    PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
dockerd 513034 root 1791u  IPv4 5026616      0t0  TCP 10.76.16.126:21425 (LISTEN)

root@some-host:~# ss -4n state all | grep :21425
tcp    LISTEN     0      128    10.76.16.126:21425                 *:*

Note: The only workaround for this until now is to restart docker daemon but for us this represents some downtime and we want to avoid to do this.

Read somewhere that maybe deleting /var/lib/docker/network/files/local-kv.db would solve the situation but we are not sure if this is safe to do it.

Steps to reproduce the issue:
We don't have steps to reproduce this behavior but basically:

1. Container is exposing port through host IP
2. Stop and Remove container
3. Try to recreate the container

Describe the results you received:
Container fails to create or start due to ip:port is already allocated, but no container is exposing the ip:port address

root@some-host:~# docker start some-container
Error response from daemon: driver failed programming external connectivity on endpoint some-container (f0728f12adf3656f4b6772526993112a3819c93082b4619dd2947ee1f2164b6f): Bind for 10.76.16.126:21425 failed: port is already allocated
Error: failed to start containers: some-container

the following command returns empty

docker ps --format {{.Names}} | while read c ; do docker port $c | grep 21425 ; done

Describe the results you expected:
Docker daemon releases the socket after container is removed and this way it won't fail to start

root@some-host:~# docker start some-container
some-container

Additional information you deem important (e.g. issue happens only occasionally):

• Issue happens only occasionally

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   723d5b934b
 Built:        Sat Nov 18 13:54:44 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   723d5b934b
 Built:        Sat Nov 18 13:53:45 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 371
 Running: 369
 Paused: 0
 Stopped: 2
Images: 173
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.14.0-041400-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 128
Total Memory: 1.876TiB
Name: some-host
ID: VXXM:RN5G:WFOM:ZRTU:Q72T:OMDG:Z4LK:2TJ7:5BWZ:DLLQ:7RLL:RPFX
Docker Root Dir: /srv/docker_root
Debug Mode (client): false
Debug Mode (server): false
Username: vkaynakchiev
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true

Additional environment details (AWS, VirtualBox, physical, etc.):

• AWS EC2 instance x1.32xlarge

[hernandanielg]

ping @thaJeztah do you know anyone who can help on this? Thanks in advance :) 👍

[dongdongking008]

Have same problem running docker (v17.09.1-ce) via swarm service. HELP !