Too many open files. Swarm mode port 7946.

[Xustyx]

Hello,
I am getting this error after a few days the swarm is running. It also reapears after a few days if I restart the docker service

docker[1341]: time="2016-11-15T12:26:33.959356093Z" level=warning msg="2016/11/15 12:26:33 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files\n"

I have tried incrementing the open file limits, setting it at:

/etc/security/limits.conf
* soft nofile 64000
* hard nofile 64000

Steps to reproduce the issue:

1. Create a cluster with swarm mode consisted of 2 nodes
2. The swarm is a testing env. It runs only an ngnix images (2 instances)
3. Wait few days (3-4) and the docker command stops responding, journalctl shows:

docker[1341]: time="2016-11-15T12:26:33.959356093Z" level=warning msg="2016/11/15 12:26:33 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files\n"

Output of docker info:

Containers: 21
 Running: 8
 Paused: 0
 Stopped: 13
Images: 76
Server Version: 1.12.3
Storage Driver: devicemapper
 Pool Name: docker-thinpool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 7.232 GB
 Data Space Total: 153 GB
 Data Space Available: 145.8 GB
 Metadata Space Used: 3.346 MB
 Metadata Space Total: 1.606 GB
 Metadata Space Available: 1.603 GB
 Thin Pool Minimum Free Space: 15.3 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2015-10-14)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay host bridge null
Swarm: active
 NodeID: 151lib7nor0gqx0078h4chm0j
 Is Manager: true
 ClusterID: 5hc7iux4evo194nz76ciw9wcz
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: *.*.1.93
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.5 GiB
Name: *****1210
ID: 7BPZ:4L37:BLMN:ZGCY:7MZJ:QGGX:T2ZL:4BCU:KIPF:4ZAI:7XUA:M4K5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://*****:8080
No Proxy: localhost,127.0.0.1,****.registry
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Labels:
 provider=generic
Insecure Registries:
 127.0.0.0/8

Output of docker version:

Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:
 OS/Arch:      linux/amd64

Additional info`:

cat /lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

[cpuguy83]

ping @mrjana

[marclr]

I'm having similar issues after a few days

[mavenugo]

@sanimej It seems like a networkdb issue on TCP sync ?

[thaJeztah]

Let me close this ticket for now, as it looks like it went stale.