Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upOpen files after docker containers removed #8693
Comments
and others
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
tnachen
Oct 22, 2014
Just a bit more context, we're calling docker logs --follow on the container, and after the container is removed also do a kill -TERM on the docker logs --follow process since there is an existing bug where docker logs --follow hangs forever if the container doesn't exist.
Not sure if it's related though.
tnachen
commented
Oct 22, 2014
|
Just a bit more context, we're calling docker logs --follow on the container, and after the container is removed also do a kill -TERM on the docker logs --follow process since there is an existing bug where docker logs --follow hangs forever if the container doesn't exist. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jessfraz
Feb 26, 2015
Contributor
are you able to reproduce on the latest version of docker and if so can you please give exact steps to do so
|
are you able to reproduce on the latest version of docker and if so can you please give exact steps to do so |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jessfraz
Mar 2, 2015
Contributor
closing as stale please ping me with exact steps to reproduce on the latest version and I will reopen
|
closing as stale please ping me with exact steps to reproduce on the latest version and I will reopen |
jessfraz
closed this
Mar 2, 2015
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
timothysc
Mar 13, 2015
Not so quick on the draw @jfrazelle, here is your repo https://bugzilla.redhat.com/show_bug.cgi?id=1189028 , happens on 1.5 too.
timothysc
commented
Mar 13, 2015
|
Not so quick on the draw @jfrazelle, here is your repo https://bugzilla.redhat.com/show_bug.cgi?id=1189028 , happens on 1.5 too. |
jessfraz
reopened this
Mar 13, 2015
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jeremyeder
Mar 13, 2015
- Comparing static binary with dynbinary...try latest docker (non-dynbinary):
# date
Fri Mar 13 15:28:44 EDT 2015
# wget https://get.docker.com/builds/Linux/x86_64/docker-latest -O docker
# DEBUG=1 ./docker -d --storage-opt dm.metadatadev=/dev/docker/metadata --storage-opt dm.datadev=/dev/docker/data --selinux-enabled
# DEBUG=1 ./docker info | grep -i fd
Fds: 11
# ./docker run -t -i --rm busybox /bin/true
# DEBUG=1 ./docker info | grep -i fd
Fds: 12
# ./docker run -t -i --rm busybox /bin/true
# DEBUG=1 ./docker info | grep -i fd
Fds: 12
# ./docker run -t -i --rm busybox /bin/true
# DEBUG=1 ./docker info | grep -i fd
Fds: 12
# ls -al /proc/`pidof docker`/fd
total 0
dr-x------. 2 root root 0 Mar 13 15:21 .
dr-xr-xr-x. 8 root root 0 Mar 13 15:21 ..
lr-x------. 1 root root 64 Mar 13 15:21 0 -> /dev/null
lrwx------. 1 root root 64 Mar 13 15:21 1 -> socket:[157480]
lrwx------. 1 root root 64 Mar 13 15:21 14 -> socket:[174651]
lrwx------. 1 root root 64 Mar 13 15:21 18 -> socket:[177423]
lrwx------. 1 root root 64 Mar 13 15:21 2 -> socket:[157480]
lr-x------. 1 root root 64 Mar 13 15:21 3 -> /dev/urandom
lrwx------. 1 root root 64 Mar 13 15:21 4 -> /dev/mapper/control
lrwx------. 1 root root 64 Mar 13 15:21 5 -> socket:[183012]
lrwx------. 1 root root 64 Mar 13 15:21 6 -> /var/lib/docker/linkgraph.db
lrwx------. 1 root root 64 Mar 13 15:21 7 -> anon_inode:[eventpoll]
lr-x------. 1 root root 64 Mar 13 15:21 8 -> anon_inode:inotify
- After creating/destroying a few containers the FD count remains constant.
- Now with the dynbinary:
# systemctl start docker
# ls -al /proc/`pidof docker`/fd
total 0
dr-x------. 2 root root 0 Mar 13 15:23 .
dr-xr-xr-x. 8 root root 0 Mar 13 15:23 ..
lr-x------. 1 root root 64 Mar 13 15:23 0 -> /dev/null
lrwx------. 1 root root 64 Mar 13 15:23 1 -> socket:[211719]
lrwx------. 1 root root 64 Mar 13 15:23 2 -> socket:[211719]
lr-x------. 1 root root 64 Mar 13 15:23 3 -> /dev/urandom
lrwx------. 1 root root 64 Mar 13 15:23 4 -> /dev/mapper/control
lrwx------. 1 root root 64 Mar 13 15:23 5 -> /var/lib/docker/linkgraph.db
lr-x------. 1 root root 64 Mar 13 15:23 6 -> anon_inode:inotify
lrwx------. 1 root root 64 Mar 13 15:23 7 -> socket:[191437]
lrwx------. 1 root root 64 Mar 13 15:23 8 -> anon_inode:[eventpoll]
lrwx------. 1 root root 64 Mar 13 15:23 9 -> socket:[191439]
# for i in $(seq 10); do ./docker run -t -i --rm busybox /bin/true ; done
# ls -al /proc/`pidof docker`/fd
total 0
dr-x------. 2 root root 0 Mar 13 15:23 .
dr-xr-xr-x. 8 root root 0 Mar 13 15:23 ..
lr-x------. 1 root root 64 Mar 13 15:23 0 -> /dev/null
lrwx------. 1 root root 64 Mar 13 15:23 1 -> socket:[211719]
lrwx------. 1 root root 64 Mar 13 15:23 14 -> socket:[202348]
lrwx------. 1 root root 64 Mar 13 15:23 15 -> socket:[216107]
lrwx------. 1 root root 64 Mar 13 15:23 17 -> socket:[220181]
lrwx------. 1 root root 64 Mar 13 15:23 19 -> socket:[216153]
lrwx------. 1 root root 64 Mar 13 15:23 2 -> socket:[211719]
lrwx------. 1 root root 64 Mar 13 15:23 20 -> socket:[197957]
lrwx------. 1 root root 64 Mar 13 15:23 21 -> socket:[210670]
lrwx------. 1 root root 64 Mar 13 15:23 22 -> socket:[216488]
lrwx------. 1 root root 64 Mar 13 15:23 24 -> socket:[212262]
lrwx------. 1 root root 64 Mar 13 15:23 25 -> socket:[222317]
lrwx------. 1 root root 64 Mar 13 15:23 26 -> socket:[201685]
lrwx------. 1 root root 64 Mar 13 15:23 27 -> socket:[225364]
lr-x------. 1 root root 64 Mar 13 15:23 3 -> /dev/urandom
lrwx------. 1 root root 64 Mar 13 15:23 4 -> /dev/mapper/control
lrwx------. 1 root root 64 Mar 13 15:23 5 -> /var/lib/docker/linkgraph.db
lr-x------. 1 root root 64 Mar 13 15:23 6 -> anon_inode:inotify
lrwx------. 1 root root 64 Mar 13 15:23 7 -> socket:[191437]
lrwx------. 1 root root 64 Mar 13 15:23 8 -> anon_inode:[eventpoll]
- All these extra sockets don't seem to ever close, and they keep growing as containers come and go...only seems to happen with dynbinary.
# for i in $(seq 100); do ./docker run -t -i --rm busybox /bin/true ; done
# DEBUG=1 docker info | grep Fd
Fds: 122
# ls -al /proc/`pidof docker`/fd | wc -l
123
jeremyeder
commented
Mar 13, 2015
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
tianon
Mar 13, 2015
Member
|
Do we open a socket to talk to dockerinit, or does that happen through some
other means?
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rhvgoyal
Mar 13, 2015
Contributor
I can't seem to reproduce this using your test case. Once for loop exits, number of Fds open still remain the same. I am using latest docker dynamically build binary on lvm thin pool as graphdriver.
|
I can't seem to reproduce this using your test case. Once for loop exits, number of Fds open still remain the same. I am using latest docker dynamically build binary on lvm thin pool as graphdriver. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jeremyeder
Mar 13, 2015
@rhvgoyal not sure if there are other parts to the puzzle, seemed pretty cut and dry to me. We have several other reports noted in https://bugzilla.redhat.com/show_bug.cgi?id=1189028
jeremyeder
commented
Mar 13, 2015
|
@rhvgoyal not sure if there are other parts to the puzzle, seemed pretty cut and dry to me. We have several other reports noted in https://bugzilla.redhat.com/show_bug.cgi?id=1189028 |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rhvgoyal
Mar 13, 2015
Contributor
That report also says that "It doesn't happen on RHEL7 + docker 1.4.1"
So looks like it does not happen on all combinations. I guess it is a good thing as it should help us pin point the problem. BTW, I am using upstream kernel 4.0.-rc2 and latest docker. Following is how my docker info looks like.
Containers: 0
Images: 130
Storage Driver: devicemapper
Pool Name: docker-docker--pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file:
Metadata file:
Data Space Used: 4.518 GB
Data Space Total: 489.6 GB
Data Space Available: 485.1 GB
Metadata Space Used: 11.5 MB
Metadata Space Total: 503.3 MB
Metadata Space Available: 491.8 MB
Udev Sync Supported: true
Library Version: 1.02.93 (2015-01-30)
Execution Driver: native-0.2
Kernel Version: 4.0.0-rc2+
Operating System: Fedora 21 (Twenty One)
CPUs: 24
Total Memory: 31.1 GiB
|
That report also says that "It doesn't happen on RHEL7 + docker 1.4.1" So looks like it does not happen on all combinations. I guess it is a good thing as it should help us pin point the problem. BTW, I am using upstream kernel 4.0.-rc2 and latest docker. Following is how my docker info looks like. Containers: 0 |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
42wim
Mar 19, 2015
Same issue here, using latest docker running on RHEL6.6.
Doesn't matter if using static binary from docker or dynbinary from EPEL
# for i in $(seq 100); do docker run -t -i --rm busybox /bin/true ; done
# /proc/`pidof docker`/fd 21:42 # ls -al|wc
247 2816 26061
Containing this stuff:
lr-x------ 1 root root 64 Mar 19 21:41 224 -> /cgroup/memory/docker/4702d400101a5c598181d37711049343263910e2876f161b46241c5e9dbdcff2/memory.oom_control (deleted)
lr-x------ 1 root root 64 Mar 19 21:41 226 -> /cgroup/memory/docker/952f932435c2b6bf9c11181a9a1e9fc0a4534134a1a789f05a87a12c22da4e7c/memory.oom_control (deleted)
lrwx------ 1 root root 64 Mar 19 21:41 227 -> [eventfd]
lr-x------ 1 root root 64 Mar 19 21:41 228 -> /cgroup/memory/docker/12f831d86f76ba5a02bee444625feb5779f6d76508f0788d3436e3c57ef36c89/memory.oom_control (deleted)
# ./docker --version
Docker version 1.5.0, build a8a31ef
# ./docker info
Containers: 150
Images: 436
Storage Driver: devicemapper
Pool Name: docker-253:5-73734-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 36.89 GB
Data Space Total: 107.4 GB
Metadata Space Used: 42.81 MB
Metadata Space Total: 2.147 GB
Udev Sync Supported: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.8.1.el6.x86_64
Operating System: <unknown>
CPUs: 8
Total Memory: 7.684 GiB
42wim
commented
Mar 19, 2015
|
Same issue here, using latest docker running on RHEL6.6. Containing this stuff: |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
iangkent
Apr 24, 2015
We are experiencing the same issue with docker 1.4.1 on RHEL 6.5 (2.6.32-504.8.1.el6.x86_64).
Our current workaround is to restart docker daemon.
Is there a better recommend workaround?
iangkent
commented
Apr 24, 2015
|
We are experiencing the same issue with docker 1.4.1 on RHEL 6.5 (2.6.32-504.8.1.el6.x86_64). Is there a better recommend workaround? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
amaltson
May 21, 2015
I'm experiencing the exact same problem with Docker 1.6 and RHEL 6.6. We ran into this when the docker user reached it's limit on open files. I took a look at lsof and found a lot of the (deleted) file descriptors pointing to removed Docker containers.
# lsof -nP | grep docker
docker 2725 root cwd DIR 253,0 4096 2 /
docker 2725 root rtd DIR 253,0 4096 2 /
docker 2725 root txt REG 253,0 15443675 149299 /usr/bin/docker
docker 2725 root 1w REG 253,0 14221171 815016 /var/log/docker
docker 2725 root 2w REG 253,0 14221171 815016 /var/log/docker
docker 2725 root 3r CHR 1,9 0t0 4170 /dev/urandom
docker 2725 root 4u CHR 10,58 0t0 6084 /dev/mapper/control
docker 2725 root 5r REG 0,28 0 19076234 /cgroup/memory/docker/8b155e930840e09e36ad7ddb65598e7412dbddf2a64add39fe0775ab7ef02e37/memory.oom_control (deleted)
docker 2725 root 6u unix 0xffff880236faeb00 0t0 16234 /var/run/docker.sock
docker 2725 root 7u REG 0,9 0 4163 [eventpoll]
docker 2725 root 8u REG 253,2 26624 29633572 /transient/opt/docker/linkgraph.db
docker 2725 root 9u REG 253,2 26624 29633572 /transient/opt/docker/linkgraph.db
docker 2725 root 10r DIR 0,10 0 1 inotify
docker 2725 root 11r REG 0,28 0 32400713 /cgroup/memory/docker/0ad1fc937d3ed540a2ab1656059d6205e739fa079d539a6740d4d6d542cfd3d1/memory.oom_control (deleted)
docker 2725 root 12r REG 0,28 0 17925027 /cgroup/memory/docker/33dcb46c0d288198318c7e9f901b8e0ba7b6a5c390828529d7c9bfcb385b9d2f/memory.oom_control (deleted)
docker 2725 root 13u REG 0,9 0 4163 [eventfd]
....
< over a 1000 more here>
Here's the docker info
# docker info
Containers: 2
Images: 114
Storage Driver: devicemapper
Pool Name: docker-253:2-29633562-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 9.234 GB
Data Space Total: 107.4 GB
Data Space Available: 98.14 GB
Metadata Space Used: 10.08 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.137 GB
Udev Sync Supported: false
Data loop file: /transient/opt/docker/devicemapper/devicemapper/data
Metadata loop file: /transient/opt/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.16.2.el6.x86_64
Operating System: <unknown>
CPUs: 4
Total Memory: 7.681 GiB
Name: devbuild1
ID: BURZ:PUHG:ECHK:KRO2:7N3L:HD7O:TJZE:VQP2:LFQL:3N3R:STCS:LBSE
amaltson
commented
May 21, 2015
|
I'm experiencing the exact same problem with Docker 1.6 and RHEL 6.6. We ran into this when the docker user reached it's limit on open files. I took a look at Here's the |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
JohnMorales
Jun 5, 2015
@iangkent: Another work around is to attach to the docker process and tell it to close the file descriptors.
I'm doing something like this:
gdb --batch-silent -p $(cat /var/run/docker.pid) <<< "$( ls /proc/$(cat /var/run/docker.pid)/fd -l --time-style=+'%s' | grep -E 'deleted' | awk '{ printf("p close(%s)\n", $7)}')"
Note: what I'm doing here is attaching to the docker daemon using gdb, and telling it to close file descriptors that have 'deleted' in the file listing..
It's ugly but better than restarting the containers that I want to keep running.
UPDATE: I noticed that killing eventfd descriptors can cause a tainted panic. Updated command to not remove those eventfd
JohnMorales
commented
Jun 5, 2015
|
@iangkent: Another work around is to attach to the docker process and tell it to close the file descriptors. I'm doing something like this: Note: what I'm doing here is attaching to the docker daemon using gdb, and telling it to close file descriptors that have 'deleted' in the file listing.. It's ugly but better than restarting the containers that I want to keep running. UPDATE: I noticed that killing eventfd descriptors can cause a tainted panic. Updated command to not remove those eventfd |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
LK4D4
Jun 5, 2015
Contributor
It's pretty weird that there is fds from cgroups :/ I'll try to reproduce.
|
It's pretty weird that there is fds from cgroups :/ I'll try to reproduce. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
JohnMorales
commented
Jun 5, 2015
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
JohnMorales
Jun 5, 2015
@LK4D4: it always seems to be the 'memory.oom_control' file that is left open for each cgroup instance.
JohnMorales
commented
Jun 5, 2015
|
@LK4D4: it always seems to be the 'memory.oom_control' file that is left open for each cgroup instance. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
LK4D4
Jun 5, 2015
Contributor
Hmm, I really can't reproduce on master. Even if I thought that I saw this before.
|
Hmm, I really can't reproduce on master. Even if I thought that I saw this before. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
induane
Jun 8, 2015
It's present at least as recently as docker 1.6 - If I look at the file descriptors I end up with:
sudo ls /proc/$(cat /var/run/docker.pid)/fd -l --time-style=+'%s'
total 0
lr-x------. 1 root root 64 1433797124 0 -> pipe:[4511692]
l-wx------. 1 root root 64 1433797124 1 -> /var/log/docker
lrwx------. 1 root root 64 1433797315 10 -> socket:[4543140]
lrwx------. 1 root root 64 1433797364 100 -> /var/lib/docker/containers/ef02c72cf1190048f693042e34e1a4117c506246cebe5e0067c92842e177c15d/ef02c72cf1190048f693042e34e1a4117c506246cebe5e0067c92842e177c15d-json.log
lrwx------. 1 root root 64 1433797348 101 -> [eventfd]
lrwx------. 1 root root 64 1433797364 102 -> /var/lib/docker/containers/b8152fdb4ef387bd59a61c5b311dc89082c368673cdc113dc48ad81fcd1dd81a/b8152fdb4ef387bd59a61c5b311dc89082c368673cdc113dc48ad81fcd1dd81a-json.log
lrwx------. 1 root root 64 1433797364 103 -> /var/lib/docker/containers/b8152fdb4ef387bd59a61c5b311dc89082c368673cdc113dc48ad81fcd1dd81a/b8152fdb4ef387bd59a61c5b311dc89082c368673cdc113dc48ad81fcd1dd81a-json.log
lrwx------. 1 root root 64 1433797364 104 -> socket:[4542056]
lr-x------. 1 root root 64 1433797364 105 -> pipe:[4542057]
lr-x------. 1 root root 64 1433797364 106 -> /cgroup/memory/docker/ef02c72cf1190048f693042e34e1a4117c506246cebe5e0067c92842e177c15d/memory.oom_control
lr-x------. 1 root root 64 1433797364 107 -> pipe:[4542058]
lrwx------. 1 root root 64 1433797378 108 -> /var/lib/docker/containers/a05678f737dde4a62fd9e94289ae850f766efbf71b5fb60ec73a70f6fe16a302/a05678f737dde4a62fd9e94289ae850f766efbf71b5fb60ec73a70f6fe16a302-json.log
lrwx------. 1 root root 64 1433797364 109 -> [eventfd]
lrwx------. 1 root root 64 1433797315 11 -> /var/lib/docker/containers/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf-json.log
lrwx------. 1 root root 64 1433797364 110 -> socket:[4542319]
lr-x------. 1 root root 64 1433797364 111 -> pipe:[4542320]
lr-x------. 1 root root 64 1433797364 112 -> /cgroup/memory/docker/b8152fdb4ef387bd59a61c5b311dc89082c368673cdc113dc48ad81fcd1dd81a/memory.oom_control
lr-x------. 1 root root 64 1433797364 113 -> pipe:[4542321]
lrwx------. 1 root root 64 1433797364 114 -> [eventfd]
lr-x------. 1 root root 64 1433797378 115 -> /cgroup/memory/docker/a05678f737dde4a62fd9e94289ae850f766efbf71b5fb60ec73a70f6fe16a302/memory.oom_control
lrwx------. 1 root root 64 1433797378 117 -> [eventfd]
lrwx------. 1 root root 64 1433797378 118 -> socket:[4544837]
lr-x------. 1 root root 64 1433797378 119 -> pipe:[4544838]
lr-x------. 1 root root 64 1433797315 12 -> /cgroup/memory/docker/a5d7c8b0389cb1984e8b89b985dbf70cbbdb470f50f82b4d1e52fe87009b724a/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797378 121 -> pipe:[4544839]
lr-x------. 1 root root 64 1433797315 13 -> /cgroup/memory/docker/dde9f099822f9e3449d9ba6331e70bcb57f9da4dbd7e99d920db7e1c0b6668ea/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 14 -> [eventfd]
lrwx------. 1 root root 64 1433797315 15 -> /var/lib/docker/containers/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf-json.log
lrwx------. 1 root root 64 1433797315 16 -> /cgroup/memory/docker/eb2d0297059e41d424020de288a829d7fdecf57a6a1997f4f1d01cbd3147f8ba/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 17 -> /cgroup/memory/docker/f7bb76da79cc96994de160a0d5ab5d1e143fe963bcda79516c4f022a0e77e33f/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 18 -> /cgroup/memory/docker/01750e1ed1558bb71d9e2b083e8b280c3c5969c728f95c0806079d73894257ad/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 19 -> /cgroup/memory/docker/f700e9c8443b8091d8c361ac99e674a333820c94c9e444286d24be69d368c061/memory.oom_control (deleted)
l-wx------. 1 root root 64 1433797124 2 -> /var/log/docker
lr-x------. 1 root root 64 1433797315 20 -> /cgroup/memory/docker/2085c48435f324c1e2b75c4cc2096d73ad07ae7157f96e35c4295effd8c7aa27/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 21 -> /cgroup/memory/docker/8e14c68bb68b4fd8fb1bf4c6962942446b94fe959932315df6e357fc92a7de82/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 22 -> [eventfd]
lr-x------. 1 root root 64 1433797315 23 -> /cgroup/memory/docker/97a87edb8ba32c33e0088441999ef0f20b1562d07974a026f105b559bb5b9acf/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 24 -> /cgroup/memory/docker/1def4fa783293557a444181c311a26dd74a1a56ffbfc22505114ff0c3c08bc7a/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 25 -> /cgroup/memory/docker/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf/memory.oom_control
lrwx------. 1 root root 64 1433797315 26 -> [eventfd]
lrwx------. 1 root root 64 1433797315 27 -> [eventfd]
lr-x------. 1 root root 64 1433797315 28 -> /cgroup/memory/docker/7b11f512da9cf761afb33cd770893897723bdf556fd74da01cd534df311bf2cb/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 29 -> [eventfd]
lr-x------. 1 root root 64 1433797124 3 -> /dev/urandom
lrwx------. 1 root root 64 1433797315 30 -> [eventfd]
lrwx------. 1 root root 64 1433797315 31 -> [eventfd]
lr-x------. 1 root root 64 1433797315 32 -> /cgroup/memory/docker/10f1cdc52debf0975428e3b571e27b104c4f3a1c3b08ebba40f1cc9cb8e28a4c/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 33 -> [eventfd]
lrwx------. 1 root root 64 1433797315 34 -> /cgroup/memory/docker/27da402da614bf566987eba81bc03d1cbad5e507e4f89a051784c66a5a55d317/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 35 -> /cgroup/memory/docker/6c74e6e98bfe10fce8705cae1e7b7232f64e0963dab8d34b13d73d80186f3a67/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 36 -> /cgroup/memory/docker/21efccc9fe40fb8134bdc57e6f3dc81dec1b1b3676814ce157d8a0eda542aeaf/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 37 -> [eventfd]
lrwx------. 1 root root 64 1433797315 38 -> [eventfd]
lrwx------. 1 root root 64 1433797315 39 -> [eventfd]
lrwx------. 1 root root 64 1433797124 4 -> /dev/mapper/control
lrwx------. 1 root root 64 1433797315 40 -> [eventfd]
lr-x------. 1 root root 64 1433797315 41 -> /cgroup/memory/docker/4015cc18c4f53eb98ff572e3c9d0a7f1f8f692ffd44c8e9f0bd5c276ec0dad0d/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 42 -> /cgroup/memory/docker/74499a76d96b3a80df8f6be635932b82c26ab48088785e3007275bd213a74636/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 43 -> [eventfd]
lrwx------. 1 root root 64 1433797315 44 -> [eventfd]
lrwx------. 1 root root 64 1433797315 45 -> [eventfd]
lrwx------. 1 root root 64 1433797315 46 -> socket:[4544000]
lrwx------. 1 root root 64 1433797315 47 -> [eventfd]
lrwx------. 1 root root 64 1433797315 48 -> /cgroup/memory/docker/d11ce6756e62c9f1f1fe76df459bb086b528c713e45aacd884907fffdfe4cf88/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 49 -> /cgroup/memory/docker/9ac94647fd1728bd00568814a67e79a18e43ee0595e7f11c3a4e5c13c54338e8/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797206 5 -> /var/lib/docker/linkgraph.db
lrwx------. 1 root root 64 1433797315 50 -> [eventfd]
lrwx------. 1 root root 64 1433797315 51 -> [eventfd]
lr-x------. 1 root root 64 1433797315 52 -> /cgroup/memory/docker/9b2fcd32ad5d1c25f53bd22a3236e68f64fa9b0ab92943c6630f7c93b2d09fa6/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 53 -> /cgroup/memory/docker/d5954b25a28d23153911e90e4863ebc6b6d9bf140bfd3e7177bc2c78224670e0/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 54 -> [eventfd]
lrwx------. 1 root root 64 1433797315 55 -> [eventfd]
lrwx------. 1 root root 64 1433797315 56 -> [eventfd]
lrwx------. 1 root root 64 1433797315 57 -> /cgroup/memory/docker/e27d34618f7283039d7e33231b473bcc0523cc977d70390566badcd97b15a594/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 58 -> socket:[4543636]
lrwx------. 1 root root 64 1433797315 59 -> [eventfd]
lr-x------. 1 root root 64 1433797206 6 -> inotify
lr-x------. 1 root root 64 1433797315 60 -> /cgroup/memory/docker/67b9c79858af4815819aef2b88482e3d8526eac5e3e830bdaea72a580bfb430e/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 61 -> [eventfd]
lrwx------. 1 root root 64 1433797315 62 -> [eventfd]
lr-x------. 1 root root 64 1433797315 63 -> pipe:[4543637]
lr-x------. 1 root root 64 1433797328 64 -> /dev/dm-5
lrwx------. 1 root root 64 1433797315 65 -> pipe:[4543638]
lr-x------. 1 root root 64 1433797315 67 -> /cgroup/memory/docker/c9f5255689a4cce3dd075898ee80d9a014e57d0a832b74bb74bcdc131855cdd0/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 68 -> [eventfd]
lr-x------. 1 root root 64 1433797315 69 -> /cgroup/memory/docker/1ec545d3b37ba926d9898eef2d1f7c188ae2e1eea4c6ff7b3501dddc0a410b04/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 7 -> socket:[4543602]
lrwx------. 1 root root 64 1433797329 71 -> /cgroup/memory/docker/b000c864a5c834af40a0b741ee296e66afb1264dab2605a544adc7aaba8d0a91/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797331 72 -> /cgroup/memory/docker/74dfcf658f19c0f52fc4b2d9faa5d744d251b8e3bd54f5f24c0a88db860c2091/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 73 -> /cgroup/memory/docker/e0f7b680cba4e2af80a0956f5c626e393736600c203e16ae5d24a3f4b438840a/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797331 74 -> [eventfd]
lrwx------. 1 root root 64 1433797331 75 -> [eventfd]
lr-x------. 1 root root 64 1433797315 76 -> /cgroup/memory/docker/ec8262f4e17b74923bd0849e428c0cb12fd32c2fb9a8385e8de459e614802551/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 77 -> [eventfd]
lrwx------. 1 root root 64 1433797315 78 -> [eventfd]
lrwx------. 1 root root 64 1433797315 79 -> [eventfd]
lrwx------. 1 root root 64 1433797124 8 -> socket:[4511739]
lrwx------. 1 root root 64 1433797315 80 -> socket:[4544718]
lr-x------. 1 root root 64 1433797315 81 -> /cgroup/memory/docker/1f75b718831acf4836bca27c41556e07a9ebad6f5fadb46160f26cf95f66f9f1/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 84 -> /cgroup/memory/docker/1c452d966ac6bd2050d7893a374804fb7070d81bb0d6e3eee92ed429d425b64b/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 85 -> socket:[4541761]
lr-x------. 1 root root 64 1433797315 86 -> /cgroup/memory/docker/c15bed7c330bb387a34c80ce7acb8ef0e95ac6d288a8c5af3f3427f8e946ea74/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 87 -> [eventfd]
lr-x------. 1 root root 64 1433797331 88 -> socket:[4544804]
lrwx------. 1 root root 64 1433797331 89 -> [eventfd]
lrwx------. 1 root root 64 1433797124 9 -> [eventpoll]
lrwx------. 1 root root 64 1433797315 91 -> [eventfd]
lr-x------. 1 root root 64 1433797315 92 -> socket:[4541996]
lrwx------. 1 root root 64 1433797348 93 -> /var/lib/docker/containers/a05678f737dde4a62fd9e94289ae850f766efbf71b5fb60ec73a70f6fe16a302/a05678f737dde4a62fd9e94289ae850f766efbf71b5fb60ec73a70f6fe16a302-json.log
lr-x------. 1 root root 64 1433797315 94 -> socket:[4541765]
lr-x------. 1 root root 64 1433797348 95 -> /cgroup/memory/docker/487e2b8d46322a9013be67c0cd67ac89b33e788754dce59a294ea558493fbde2/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797348 96 -> socket:[4542016]
lr-x------. 1 root root 64 1433797348 97 -> /cgroup/memory/docker/b54d79a4016f0de3978b0d8b859217953d080843791705c5996686c2130f935c/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797348 98 -> [eventfd]
lrwx------. 1 root root 64 1433797364 99 -> /var/lib/docker/containers/ef02c72cf1190048f693042e34e1a4117c506246cebe5e0067c92842e177c15d/ef02c72cf1190048f693042e34e1a4117c506246cebe5e0067c92842e177c15d-json.log
[propylon@rnd-stage ~]$ sudo gdb --batch-silent -p $(sudo cat /var/run/docker.pid) <<< "$( sudo ls /proc/$(cat /var/run/docker.pid)/fd -l --time-style=+'%s' | grep -E 'deleted' | awk '{ printf("p close(%s)\n", $7)}')"
[propylon@rnd-stage ~]$ sudo ls /proc/$(cat /var/run/docker.pid)/fd -l --time-style=+'%s'total 0
lr-x------. 1 root root 64 1433797124 0 -> pipe:[4511692]
l-wx------. 1 root root 64 1433797124 1 -> /var/log/docker
lrwx------. 1 root root 64 1433797315 10 -> socket:[4543140]
lrwx------. 1 root root 64 1433797348 101 -> [eventfd]
lr-x------. 1 root root 64 1433797364 106 -> /cgroup/memory/docker/ef02c72cf1190048f693042e34e1a4117c506246cebe5e0067c92842e177c15d/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797364 109 -> [eventfd]
lrwx------. 1 root root 64 1433797315 11 -> /var/lib/docker/containers/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf-json.log
lr-x------. 1 root root 64 1433797364 112 -> /cgroup/memory/docker/b8152fdb4ef387bd59a61c5b311dc89082c368673cdc113dc48ad81fcd1dd81a/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797364 114 -> [eventfd]
lr-x------. 1 root root 64 1433797378 115 -> /cgroup/memory/docker/a05678f737dde4a62fd9e94289ae850f766efbf71b5fb60ec73a70f6fe16a302/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797378 117 -> [eventfd]
lr-x------. 1 root root 64 1433797315 12 -> /cgroup/memory/docker/a5d7c8b0389cb1984e8b89b985dbf70cbbdb470f50f82b4d1e52fe87009b724a/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 13 -> /cgroup/memory/docker/dde9f099822f9e3449d9ba6331e70bcb57f9da4dbd7e99d920db7e1c0b6668ea/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 14 -> [eventfd]
lrwx------. 1 root root 64 1433797315 15 -> /var/lib/docker/containers/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf-json.log
lrwx------. 1 root root 64 1433797315 16 -> /cgroup/memory/docker/eb2d0297059e41d424020de288a829d7fdecf57a6a1997f4f1d01cbd3147f8ba/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 17 -> /cgroup/memory/docker/f7bb76da79cc96994de160a0d5ab5d1e143fe963bcda79516c4f022a0e77e33f/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 18 -> /cgroup/memory/docker/01750e1ed1558bb71d9e2b083e8b280c3c5969c728f95c0806079d73894257ad/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 19 -> /cgroup/memory/docker/f700e9c8443b8091d8c361ac99e674a333820c94c9e444286d24be69d368c061/memory.oom_control (deleted)
l-wx------. 1 root root 64 1433797124 2 -> /var/log/docker
lr-x------. 1 root root 64 1433797315 20 -> /cgroup/memory/docker/2085c48435f324c1e2b75c4cc2096d73ad07ae7157f96e35c4295effd8c7aa27/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 21 -> /cgroup/memory/docker/8e14c68bb68b4fd8fb1bf4c6962942446b94fe959932315df6e357fc92a7de82/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 22 -> [eventfd]
lr-x------. 1 root root 64 1433797315 23 -> /cgroup/memory/docker/97a87edb8ba32c33e0088441999ef0f20b1562d07974a026f105b559bb5b9acf/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 24 -> /cgroup/memory/docker/1def4fa783293557a444181c311a26dd74a1a56ffbfc22505114ff0c3c08bc7a/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 25 -> /cgroup/memory/docker/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf/memory.oom_control
lrwx------. 1 root root 64 1433797315 26 -> [eventfd]
lrwx------. 1 root root 64 1433797315 27 -> [eventfd]
lr-x------. 1 root root 64 1433797315 28 -> /cgroup/memory/docker/7b11f512da9cf761afb33cd770893897723bdf556fd74da01cd534df311bf2cb/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 29 -> [eventfd]
lr-x------. 1 root root 64 1433797124 3 -> /dev/urandom
lrwx------. 1 root root 64 1433797315 30 -> [eventfd]
lrwx------. 1 root root 64 1433797315 31 -> [eventfd]
lr-x------. 1 root root 64 1433797315 32 -> /cgroup/memory/docker/10f1cdc52debf0975428e3b571e27b104c4f3a1c3b08ebba40f1cc9cb8e28a4c/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 33 -> [eventfd]
lrwx------. 1 root root 64 1433797315 34 -> /cgroup/memory/docker/27da402da614bf566987eba81bc03d1cbad5e507e4f89a051784c66a5a55d317/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 35 -> /cgroup/memory/docker/6c74e6e98bfe10fce8705cae1e7b7232f64e0963dab8d34b13d73d80186f3a67/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 36 -> /cgroup/memory/docker/21efccc9fe40fb8134bdc57e6f3dc81dec1b1b3676814ce157d8a0eda542aeaf/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 37 -> [eventfd]
lrwx------. 1 root root 64 1433797315 38 -> [eventfd]
lrwx------. 1 root root 64 1433797315 39 -> [eventfd]
lrwx------. 1 root root 64 1433797124 4 -> /dev/mapper/control
lrwx------. 1 root root 64 1433797315 40 -> [eventfd]
lr-x------. 1 root root 64 1433797315 41 -> /cgroup/memory/docker/4015cc18c4f53eb98ff572e3c9d0a7f1f8f692ffd44c8e9f0bd5c276ec0dad0d/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 42 -> /cgroup/memory/docker/74499a76d96b3a80df8f6be635932b82c26ab48088785e3007275bd213a74636/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 43 -> [eventfd]
lrwx------. 1 root root 64 1433797315 44 -> [eventfd]
lrwx------. 1 root root 64 1433797315 45 -> [eventfd]
lrwx------. 1 root root 64 1433797315 46 -> socket:[4545833]
lrwx------. 1 root root 64 1433797315 47 -> [eventfd]
lrwx------. 1 root root 64 1433797315 48 -> /cgroup/memory/docker/d11ce6756e62c9f1f1fe76df459bb086b528c713e45aacd884907fffdfe4cf88/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 49 -> /cgroup/memory/docker/9ac94647fd1728bd00568814a67e79a18e43ee0595e7f11c3a4e5c13c54338e8/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797206 5 -> /var/lib/docker/linkgraph.db
lrwx------. 1 root root 64 1433797315 50 -> [eventfd]
lrwx------. 1 root root 64 1433797315 51 -> [eventfd]
lr-x------. 1 root root 64 1433797315 52 -> /cgroup/memory/docker/9b2fcd32ad5d1c25f53bd22a3236e68f64fa9b0ab92943c6630f7c93b2d09fa6/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 53 -> /cgroup/memory/docker/d5954b25a28d23153911e90e4863ebc6b6d9bf140bfd3e7177bc2c78224670e0/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 54 -> [eventfd]
lrwx------. 1 root root 64 1433797315 55 -> [eventfd]
lrwx------. 1 root root 64 1433797315 56 -> [eventfd]
lrwx------. 1 root root 64 1433797315 57 -> /cgroup/memory/docker/e27d34618f7283039d7e33231b473bcc0523cc977d70390566badcd97b15a594/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 58 -> socket:[4543636]
lrwx------. 1 root root 64 1433797315 59 -> [eventfd]
lr-x------. 1 root root 64 1433797206 6 -> inotify
lr-x------. 1 root root 64 1433797315 60 -> /cgroup/memory/docker/67b9c79858af4815819aef2b88482e3d8526eac5e3e830bdaea72a580bfb430e/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 61 -> [eventfd]
lrwx------. 1 root root 64 1433797315 62 -> [eventfd]
lr-x------. 1 root root 64 1433797315 63 -> pipe:[4543637]
lr-x------. 1 root root 64 1433797328 64 -> /var/lib/docker/containers/9865e7f70a88af4538a891ce9e013f349d3c9feb97be553906ffdadc1510f0d3/9865e7f70a88af4538a891ce9e013f349d3c9feb97be553906ffdadc1510f0d3-json.log
lrwx------. 1 root root 64 1433797315 65 -> pipe:[4543638]
lrwx------. 1 root root 64 1433797315 66 -> socket:[4545441]
lr-x------. 1 root root 64 1433797315 67 -> /cgroup/memory/docker/c9f5255689a4cce3dd075898ee80d9a014e57d0a832b74bb74bcdc131855cdd0/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 68 -> [eventfd]
lr-x------. 1 root root 64 1433797315 69 -> /cgroup/memory/docker/1ec545d3b37ba926d9898eef2d1f7c188ae2e1eea4c6ff7b3501dddc0a410b04/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 7 -> socket:[4543602]
lrwx------. 1 root root 64 1433797331 70 -> /var/lib/docker/containers/9865e7f70a88af4538a891ce9e013f349d3c9feb97be553906ffdadc1510f0d3/9865e7f70a88af4538a891ce9e013f349d3c9feb97be553906ffdadc1510f0d3-json.log
lrwx------. 1 root root 64 1433797329 71 -> /cgroup/memory/docker/b000c864a5c834af40a0b741ee296e66afb1264dab2605a544adc7aaba8d0a91/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797331 72 -> /cgroup/memory/docker/74dfcf658f19c0f52fc4b2d9faa5d744d251b8e3bd54f5f24c0a88db860c2091/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 73 -> /cgroup/memory/docker/e0f7b680cba4e2af80a0956f5c626e393736600c203e16ae5d24a3f4b438840a/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797331 74 -> [eventfd]
lrwx------. 1 root root 64 1433797331 75 -> [eventfd]
lr-x------. 1 root root 64 1433797315 76 -> /cgroup/memory/docker/ec8262f4e17b74923bd0849e428c0cb12fd32c2fb9a8385e8de459e614802551/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 77 -> [eventfd]
lrwx------. 1 root root 64 1433797315 78 -> [eventfd]
lrwx------. 1 root root 64 1433797315 79 -> [eventfd]
lrwx------. 1 root root 64 1433797124 8 -> socket:[4511739]
lrwx------. 1 root root 64 1433797315 80 -> socket:[4544718]
lr-x------. 1 root root 64 1433797315 81 -> /cgroup/memory/docker/1f75b718831acf4836bca27c41556e07a9ebad6f5fadb46160f26cf95f66f9f1/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 82 -> /cgroup/memory/docker/9865e7f70a88af4538a891ce9e013f349d3c9feb97be553906ffdadc1510f0d3/memory.oom_control
lr-x------. 1 root root 64 1433797315 84 -> /cgroup/memory/docker/1c452d966ac6bd2050d7893a374804fb7070d81bb0d6e3eee92ed429d425b64b/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 85 -> [eventfd]
lr-x------. 1 root root 64 1433797315 86 -> /cgroup/memory/docker/c15bed7c330bb387a34c80ce7acb8ef0e95ac6d288a8c5af3f3427f8e946ea74/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 87 -> [eventfd]
lr-x------. 1 root root 64 1433797331 88 -> socket:[4545889]
lrwx------. 1 root root 64 1433797331 89 -> [eventfd]
lrwx------. 1 root root 64 1433797124 9 -> [eventpoll]
lr-x------. 1 root root 64 1433797331 90 -> pipe:[4545890]
lrwx------. 1 root root 64 1433797315 91 -> [eventfd]
lr-x------. 1 root root 64 1433797315 92 -> /dev/dm-6
lrwx------. 1 root root 64 1433797348 93 -> pipe:[4545891]
lr-x------. 1 root root 64 1433797315 94 -> socket:[4541765]
lr-x------. 1 root root 64 1433797348 95 -> /cgroup/memory/docker/487e2b8d46322a9013be67c0cd67ac89b33e788754dce59a294ea558493fbde2/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797348 96 -> socket:[4542016]
lr-x------. 1 root root 64 1433797348 97 -> /cgroup/memory/docker/b54d79a4016f0de3978b0d8b859217953d080843791705c5996686c2130f935c/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797348 98 -> [eventfd]
lrwx------. 1 root root 64 1433797364 99 -> socket:[4546418]
induane
commented
Jun 8, 2015
|
It's present at least as recently as docker 1.6 - If I look at the file descriptors I end up with: sudo ls /proc/$(cat /var/run/docker.pid)/fd -l --time-style=+'%s' |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
tombee
Jun 9, 2015
Contributor
Seeing this on RHEL 6.6 2.6.32-504.el6.x86_64 too.
docker 7243 root 68r REG 0,21 0 719523 /cgroup/memory/docker/19fabdddc55310941288ddeeacc4ac8e5d7f01afe85a206c1cf3973b22a55b1e/memory.oom_control (deleted)
docker 7243 root 70r REG 0,21 0 720401 /cgroup/memory/docker/a9828f0187b801104c5803d084412f91f53e4e1c55fdca469d26fca80f926e4d/memory.oom_control (deleted)
docker 7243 root 72r REG 0,21 0 722053 /cgroup/memory/docker/34f931808cb158383a25ee0e54fea570210806e482e32ee8fa888cc8905b78c2/memory.oom_control (deleted)
docker 7243 root 74r REG 0,21 0 721475 /cgroup/memory/docker/562809ae20f2fcf877df0f99694ab3b598640ec928ecff380d0bbc777fc96806/memory.oom_control (deleted)
|
Seeing this on RHEL 6.6 2.6.32-504.el6.x86_64 too. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
amaltson
commented
Jun 19, 2015
|
I'm guessing this hasn't been fixed in Docker 1.7? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
induane
Jun 19, 2015
I've had to downgrade to 1.3 as the issue doesn't exist there. I haven't bothered doing a git bisect to try to triage the exact commit that introduced it yet as I simply haven't had the time. It may be simply that the bug won't get a lot of attention since most people running docker are doing so on RHEL7.
induane
commented
Jun 19, 2015
|
I've had to downgrade to 1.3 as the issue doesn't exist there. I haven't bothered doing a git bisect to try to triage the exact commit that introduced it yet as I simply haven't had the time. It may be simply that the bug won't get a lot of attention since most people running docker are doing so on RHEL7. |
jessfraz
added
the
kind/bug
label
Jul 10, 2015
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jsipprell
Jul 22, 2015
@amaltson it is not fixed as of current master (1.7.1 -> 1.8-in-devel) on EL6.6.
jsipprell
commented
Jul 22, 2015
|
@amaltson it is not fixed as of current master (1.7.1 -> 1.8-in-devel) on EL6.6. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jsipprell
Jul 22, 2015
@rhvgoyal This issue is caused by a spurious read on the eventfd in libcontainer/notify_linux.go:notifyOnOOM, although the underlying cause of the spurious read remains undetermined (apparently only on EL6/Centos?).
Because the read occurs before the cgroup's vfs entries have been unlinked, there is a race condition wherein the channel blocking send is reached before the container is cleaned up, the container destroyed and the channel read, and finally the for loop restarts reading the eventfd -- no additional events occur after this first "spurious" one and thus the goroutine/fd leaks. Adding an Lstat test after ch <- struct{}{} allows correct detection of a destroyed container with no leaked fds once synchronization via the channel has occurred.
I can post my patch if it would illustrate this more clearly, although I suspect someone will want to do a bit more thorough job refactoring the loop.
jsipprell
commented
Jul 22, 2015
|
@rhvgoyal This issue is caused by a spurious read on the eventfd in Because the read occurs before the cgroup's vfs entries have been unlinked, there is a race condition wherein the channel blocking send is reached before the container is cleaned up, the container destroyed and the channel read, and finally the for loop restarts reading the eventfd -- no additional events occur after this first "spurious" one and thus the goroutine/fd leaks. Adding an Lstat test after I can post my patch if it would illustrate this more clearly, although I suspect someone will want to do a bit more thorough job refactoring the loop. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
unixist
commented
Aug 25, 2015
|
Bump. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jsipprell
Aug 26, 2015
@unixist (& others interested):
I suggest you test under EL/Centos 6.7. I am unable to replicate the problem on 2.6.32-573.3.1.el6 , and the kernel-2.6.32-573.3.1.el6 rpm has an official RH bugfix that looks suspiciously related.
[root@hiro ~]# uname -a
Linux hiro 2.6.32-504.23.4.el6.x86_64 #1 SMP Tue Jun 9 20:57:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@hiro ~]# cgcreate -g memory:/foo
[root@hiro ~]# ( sleep 1 && rmdir /cgroup/memory/foo )& ./cgroupoom foo
[1] 11757
2015/08/25 23:03:03 monitoring cgroup "foo"
2015/08/25 23:03:04 cgroup "foo": OOM!
meanwhile …
[root@yt ~]# uname -a
Linux yt 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@yt ~]# cgcreate -g memory:/foo
[root@yt ~]# ( sleep 1 && rmdir /cgroup/memory/foo )& ./cgroupoom foo
[1] 21861
2015/08/25 23:05:14 monitoring cgroup "foo"
2015/08/25 23:05:15 cgroup "foo" destroyed
(cgroupoom is just a simple utility that uses the same notification code as docker 1.7's libcontainer)
jsipprell
commented
Aug 26, 2015
|
@unixist (& others interested): I suggest you test under EL/Centos 6.7. I am unable to replicate the problem on 2.6.32-573.3.1.el6 , and the kernel-2.6.32-573.3.1.el6 rpm has an official RH bugfix that looks suspiciously related. meanwhile … (cgroupoom is just a simple utility that uses the same notification code as docker 1.7's libcontainer) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
@unixist can you try if that newer kernel resolves it for you? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
unixist
Sep 8, 2015
@thaJeztah I hope to be able to test soon. I don't yet have -573 in a position to test.
Can anybody else comment?
unixist
commented
Sep 8, 2015
|
@thaJeztah I hope to be able to test soon. I don't yet have -573 in a position to test. Can anybody else comment? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
csmithATsquiz
Sep 9, 2015
Still happening.
Fresh install from the docker repo.
# uname -a
Linux sl6 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu Aug 13 12:55:33 CDT 2015 x86_64 x86_64 x86_64 GNU/Linux
# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
Pool Name: docker-8:1-131846-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 305.7 MB
Data Space Total: 107.4 GB
Data Space Available: 38.78 GB
Metadata Space Used: 729.1 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.95-RHEL6 (2015-04-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 2.6.32-573.3.1.el6.x86_64
# lsof -c docker | wc -l
21
# for i in $(seq 100); do docker run -t -i --rm busybox /bin/true ; done
Unable to find image 'busybox:latest' locally
latest: Pulling from busybox
cf2616975b4a: Pull complete
6ce2e90b0bc7: Pull complete
8c2e06607696: Already exists
busybox:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:38a203e1986cf79639cfb9b2e1d6e773de84002feea2d4eb006b52004ee8502d
Status: Downloaded newer image for busybox:latest
# lsof -c docker | wc -l
34
# lsof -c docker | grep 'deleted'
docker 1203 root 13r REG 0,21 0 37957 /cgroup/memory/docker/e8f3a7e2d4e43b48d46ce501500056ec97b8f45a37d3feb30c2fe0961c9f80fb/memory.oom_control (deleted)
docker 1203 root 17r REG 0,21 0 37061 /cgroup/memory/docker/988f3e4baf253a3d298347d56501c4e097b5880b82753fa4f5cb0d82a233963d/memory.oom_control (deleted)
docker 1203 root 20r REG 0,21 0 48456 /cgroup/memory/docker/5f7a5c381089b055f8d279ffc7b093e4bbb8c5c170191976d7c0cd52830e6d8b/memory.oom_control (deleted)
docker 1203 root 21r REG 0,21 0 42769 /cgroup/memory/docker/65f7b36c8e9925f1754c969df6d639fbfbc246f60a510bcd94731ca2e8a8ddab/memory.oom_control (deleted)
docker 1203 root 24r REG 0,21 0 38814 /cgroup/memory/docker/15d143b9645c0cac744dd68d6e353c02f1d463ecc8d7f978199e65885d50217d/memory.oom_control (deleted)
csmithATsquiz
commented
Sep 9, 2015
|
Still happening. Fresh install from the docker repo. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jsipprell
Sep 20, 2015
I believe you might be hitting a separate, but very potentially related, issue. There is a race condition in the kernel where a write to a memory cgroup's cgroup.event_control file that occurs while the cgroup is in the process of being deleted will cause the kernel dentry cache to repopulate. This then shows up a very short while later as an erroneously successful os.Lstat call. I was able to simulate this approximately 50% of the time on 2.6.32-573.3.1.el6 by reducing the initial pre-delete delay from one second to 25ms. e.g.
( usleep 25000; rmdir /cgroup/memory/foo )& ./cgroupoom foo
If this is what is happening to you, then a simple sleep 1 inside each busybox container run should fix your lingering open files.
To aid those wishing to reproduce the tests I have been running, I will post my test code here (again, it's the same algorithm found in 1.7's libcontainer wrapped in some debug output).
jsipprell
commented
Sep 20, 2015
|
I believe you might be hitting a separate, but very potentially related, issue. There is a race condition in the kernel where a write to a memory cgroup's
If this is what is happening to you, then a simple To aid those wishing to reproduce the tests I have been running, I will post my test code here (again, it's the same algorithm found in 1.7's libcontainer wrapped in some debug output). |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
pmoust
Oct 15, 2015
Contributor
Experiencing the same in CoreOS 766.4.0. Huge container log file remaining open after stoping a container started with --rm
ip-10-30-1-145 pmoust # uname -a
Linux ip-10-30-1-145.ec2.internal 4.1.7-coreos #2 SMP Wed Sep 16 22:54:37 UTC 2015 x86_64 Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz GenuineIntel GNU/Linux
ip-10-30-1-145 pmoust # docker info
Containers: 5
Images: 91
Storage Driver: overlay
Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.7-coreos
Operating System: CoreOS 766.4.0
CPUs: 2
Total Memory: 7.8 GiB
Name: ip-10-30-1-145.ec2.internal
ID: DOID:2HLN:6EWF:SPDE:G24T:YPWG:QLSH:5VFU:FP4X:QAFP:6VKF:IHGI
ip-10-30-1-145 pmoust # docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): df2f73d-dirty
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): df2f73d-dirty
OS/Arch (server): linux/amd64
|
Experiencing the same in CoreOS 766.4.0. Huge container log file remaining open after stoping a container started with --rm |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
elfchief
Oct 23, 2015
@pmoust -- different problem. The problem being discussed here applies specifically to docker keeping file handles open to the memory.oom_control cgroup control file. Anything else (docker logfiles, container logfiles, etc) is unrelated to this bug.
elfchief
commented
Oct 23, 2015
|
@pmoust -- different problem. The problem being discussed here applies specifically to docker keeping file handles open to the memory.oom_control cgroup control file. Anything else (docker logfiles, container logfiles, etc) is unrelated to this bug. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
pmoust
Oct 23, 2015
Contributor
@elfchief not really, I experience the same behavior as the original issue reporter.
That is, container removed but lsof reporting open files (specifically container log files)
|
@elfchief not really, I experience the same behavior as the original issue reporter. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
sebdah
Feb 25, 2016
We are getting hit by the same issue. The file locks are not released as expected when the logs are rotated. The locks are (naturally) released when restarting the docker service.
Sample output from lsof:
docker 1005 15874 root 127r REG 202,1 50000036 410677 /var/lib/docker/containers/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a-json.log.3 (deleted)
docker 1005 15874 root 128r REG 202,1 50000183 410637 /var/lib/docker/containers/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a-json.log.3 (deleted)
docker 1005 15874 root 129r REG 202,1 50000082 410638 /var/lib/docker/containers/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a-json.log.3 (deleted)
docker 1005 15874 root 130r REG 202,1 50000279 410658 /var/lib/docker/containers/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a-json.log.3 (deleted)
docker 1005 15874 root 131r REG 202,1 50000393 410663 /var/lib/docker/containers/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a-json.log.3 (deleted)
Output from docker info:
Containers: 1
Images: 53
Server Version: 1.9.1
Storage Driver: devicemapper
Pool Name: docker-202:1-276622-pool
Pool Blocksize: 65.54 kB
Base Device Size: 107.4 GB
Backing Filesystem:
Data file: /dev/vg-docker/data
Metadata file: /dev/vg-docker/metadata
Data Space Used: 426 MB
Data Space Total: 16.11 GB
Data Space Available: 15.68 GB
Metadata Space Used: 1.577 MB
Metadata Space Total: 4.295 GB
Metadata Space Available: 4.293 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-49-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 2
Total Memory: 3.676 GiB
Name: ip-10-120-0-9
ID: KYSA:QTRE:4HFS:FRR7:M2CZ:HZ2W:YITS:H3UP:AOS5:LMRX:OPMS:OZRX
WARNING: No swap limit support
sebdah
commented
Feb 25, 2016
|
We are getting hit by the same issue. The file locks are not released as expected when the logs are rotated. The locks are (naturally) released when restarting the docker service. Sample output from Output from |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
JohnMorales
Feb 25, 2016
@sebdah (edit: I just noticed that you're on Ubuntu so disregard my notes below on rhel 6 :)
I don't think this issue is going to get any traction as there are other problems running docker on rhel 6, so the current docker docs recommends to run docker only on rhel 7. I've read somewhere if rhel 6 is needed it can be safely used it as the docker image base for the containers running on a rhel7 host.
That said my workaround cronjob has kept the machine from crashing and it's been up for months... even though the open files isn't a problem, I'm noticing inconsistent network stack issues in the containers that moving to rhel 7 is on the horizon for me.
Here's the gist of the cronjob that runs every night on my rhel6 host, not a solution, but a stop gap:
JohnMorales
commented
Feb 25, 2016
|
@sebdah (edit: I just noticed that you're on Ubuntu so disregard my notes below on rhel 6 :) I don't think this issue is going to get any traction as there are other problems running docker on rhel 6, so the current docker docs recommends to run docker only on rhel 7. I've read somewhere if rhel 6 is needed it can be safely used it as the docker image base for the containers running on a rhel7 host. That said my workaround cronjob has kept the machine from crashing and it's been up for months... even though the open files isn't a problem, I'm noticing inconsistent network stack issues in the containers that moving to rhel 7 is on the horizon for me. Here's the gist of the cronjob that runs every night on my rhel6 host, not a solution, but a stop gap: |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
sebdah
commented
Feb 25, 2016
|
@JohnMorales Thanks! We are on Ubuntu 14.04.02 |
icecrime
added
the
version/1.1
label
May 12, 2016
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
exowaucka
Jun 3, 2016
Is this issue supposed to be resolved on RHEL 7? I'm running CentOS 7 with docker 1.9.1, and I'm having the problem reported by the original submitter (large number of deleted json.log files being held open).
$ sudo docker info
Containers: 123
Images: 1899
Server Version: 1.9.1
Storage Driver: devicemapper
Pool Name: docker--vg-docker--pool
Pool Blocksize: 524.3 kB
Base Device Size: 107.4 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 72.94 GB
Data Space Total: 127.6 GB
Data Space Available: 54.62 GB
Metadata Space Used: 42.77 MB
Metadata Space Total: 88.08 MB
Metadata Space Available: 45.31 MB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.107-RHEL7 (2015-12-01)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-327.13.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 7.389 GiB
Name: REDACTED
ID: JSZR:26PG:HBTY:DJ7B:VUV3:IBBK:3ZI3:XJQE:FEHF:WIMC:ZV4U:BBRE
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
The xfs-formatted root disk is 30 GB. There are no other disks on the system.
exowaucka
commented
Jun 3, 2016
|
Is this issue supposed to be resolved on RHEL 7? I'm running CentOS 7 with docker 1.9.1, and I'm having the problem reported by the original submitter (large number of deleted json.log files being held open). The xfs-formatted root disk is 30 GB. There are no other disks on the system. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
pashpashkin
Jun 28, 2016
I have the same problem on Ubuntu 14.04.4.
$ uname -a
Linux Ubuntu-1404-trusty-64-minimal 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ sudo docker info
Containers: 13
Running: 0
Paused: 0
Stopped: 13
Images: 17
Server Version: 1.11.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 208
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.2.0-27-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.797 GiB
Name: Ubuntu-1404-trusty-64-minimal
ID: LPHO:OLCZ:MR6J:IZJN:GJMJ:7HRB:LO4V:XR23:NRE6:6AHJ:EXPM:S2DO
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
$ sudo lsof -p `pidof docker` | grep sock
docker 9731 root 9u unix 0xffff8800bbb3d2c0 0t0 42211226 /var/run/docker.sock
docker 9731 root 10u unix 0xffff8801b7b81680 0t0 42223315 /var/run/docker.sock
docker 9731 root 11u unix 0xffff8801238570c0 0t0 45495496 /var/run/docker.sock
docker 9731 root 12u unix 0xffff8801c4bd92c0 0t0 42578391 /var/run/docker.sock
docker 9731 root 13u unix 0xffff8802338ef0c0 0t0 42578392 /var/run/docker.sock
docker 9731 root 14u unix 0xffff88023529d680 0t0 48283437 /var/run/docker.sock
docker 9731 root 15u unix 0xffff8801b78b3480 0t0 43881975 /var/run/docker.sock
docker 9731 root 16u unix 0xffff8801178e1e00 0t0 43902442 /var/run/docker.sock
docker 9731 root 17u unix 0xffff8801b78b12c0 0t0 43881976 /var/run/docker.sock
docker 9731 root 18u unix 0xffff880073472580 0t0 44313665 /var/run/docker.sock
docker 9731 root 19u unix 0xffff88023327e940 0t0 43902443 /var/run/docker.sock
docker 9731 root 20u unix 0xffff8800bb3483c0 0t0 44337968 /var/run/docker.sock
docker 9731 root 21u unix 0xffff88023335a580 0t0 48435649 /var/run/docker.sock
docker 9731 root 22u unix 0xffff8801c4bd8780 0t0 42595059 /var/run/docker.sock
docker 9731 root 23u unix 0xffff8801c4bda1c0 0t0 42606347 /var/run/docker.sock
docker 9731 root 24u unix 0xffff8802338ee940 0t0 42609880 /var/run/docker.sock
docker 9731 root 25u unix 0xffff880073473480 0t0 43792846 /var/run/docker.sock
docker 9731 root 26u unix 0xffff8801c4bda580 0t0 43918516 /var/run/docker.sock
docker 9731 root 27u unix 0xffff8801b7b81a40 0t0 42212892 /var/run/docker.sock
docker 9731 root 28u unix 0xffff8801b7b803c0 0t0 42225767 /var/run/docker.sock
docker 9731 root 29u unix 0xffff88023324e940 0t0 42178926 /var/run/docker.sock
...
$ sudo lsof -c docker | grep 'deleted'
docker-co 9742 root 11u FIFO 0,18 0t0 16303 /run/containerd/48a85b96b05a9a60608dc6d5a7f7d2298c9641b2bd10fe18f7e6b596f4bdcd6d/init/control (deleted)
docker-co 9742 root 16u FIFO 0,18 0t0 16372 /run/containerd/c6478ca12a87110b6216e44c28d8ed1ebad825e88cbb6c7ac9cb0deb7b990b57/init/control (deleted)
docker-co 9742 root 20u FIFO 0,18 0t0 16451 /run/containerd/063a38e8055c8376e28da5fd69910573526d04f6eb135641dd44f50ca1d43c15/init/control (deleted)
docker-co 9742 root 21u FIFO 0,18 0t0 16455 /run/containerd/b08f4faab33c513226900df521c11669993a79ad5b637d0cda92588d9006d81c/init/control (deleted)
docker-co 9742 root 22u FIFO 0,18 0t0 16466 /run/containerd/e507b67161e900559f05650bd870bf555d6fc9d400a2bc2a9f9d316874028c5d/init/control (deleted)
docker-co 9742 root 23u FIFO 0,18 0t0 16432 /run/containerd/49b504aa707d674497bc976c555de67e799e6a1137432a1ceaeeaf7b08b4836e/init/control (deleted)
docker-co 9742 root 24u FIFO 0,18 0t0 16437 /run/containerd/f9dd9d978ab8f050b343af33ab25d171a50cd4e78867bc24a6aac919ba5cf074/init/control (deleted)
docker-co 9742 root 27u FIFO 0,18 0t0 16464 /run/containerd/1e11c6dd928fe06e54ce9a1f8da02350f9f1204f299fd3769d3d996cd7baae26/init/control (deleted)
docker-co 9742 root 30u FIFO 0,18 0t0 16449 /run/containerd/eb1cbfeb63db2f4090d986b6277f6a196d59420701ed16094c1f930317ae9602/init/control (deleted)
docker-co 9742 root 31u FIFO 0,18 0t0 16446 /run/containerd/fce75b25583f4315f13d7ec0b9875d365d1c95a81e3eb94d7191d895f8ab7f20/init/control (deleted)
pashpashkin
commented
Jun 28, 2016
|
I have the same problem on Ubuntu 14.04.4. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Majkl578
Jul 11, 2016
Same issue here with 1.11.2. This was causing our long-running service to fail periodically, because it had no free file descriptors left...
This bug is almost 2 years open, sad.
Majkl578
commented
Jul 11, 2016
|
Same issue here with 1.11.2. This was causing our long-running service to fail periodically, because it had no free file descriptors left... |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
thaJeztah
Jul 11, 2016
Member
@Majkl578 what storage driver are you using? If you're using overlay, that may be a different issue
|
@Majkl578 what storage driver are you using? If you're using overlay, that may be a different issue |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
cpuguy83
Jul 11, 2016
Contributor
Let's close this, because it's likely been different issues over time, I do not see a reason to keep it open. @thaJeztah WDYT?
|
Let's close this, because it's likely been different issues over time, I do not see a reason to keep it open. @thaJeztah WDYT? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
thaJeztah
Jul 11, 2016
Member
@cpuguy83 agreed.
For those following this; we're closing this issue because it's been collecting various unrelated issues; if you're still having this issue, please open a new issue, and detailed information about your issue.
|
@cpuguy83 agreed. For those following this; we're closing this issue because it's been collecting various unrelated issues; if you're still having this issue, please open a new issue, and detailed information about your issue. |
tnachen commentedOct 22, 2014
docker version:
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070
I ran out of file descriptors running docker run, and I realized docker daemon is holding on to a lot of file handles (11110 of them), while I only have 6 containers with docker ps -a
lots of them looks like this:
6c665c78a63396955dc76873bd8636bb642b38b43ec0e6-json.log (deleted)
docker 6937 root 329u unix 0xffff880166560e00 0t0 353408770 /var/run/docker.sock
docker 6937 root 330r REG 9,2 0 1627517 /var/lib/docker/containers/0734b3b41a4b039cf983cb690b1ad2814a8434289752f8082fd312e837f04672/0734b3b41a4b039cf9
83cb690b1ad2814a8434289752f8082fd312e837f04672-json.log (deleted)
docker 6937 root 331u unix 0xffff880166560a80 0t0 353427790 /var/run/docker.sock
docker 6937 root 332r REG 9,2 219 1627529 /var/lib/docker/containers/933d4e44aa370529284c40b5a1fae6ecf5e887e285d9860ba44ce61439c692f0/933d4e44aa37052928
4c40b5a1fae6ecf5e887e285d9860ba44ce61439c692f0-json.log (deleted)
docker 6937 root 333u unix 0xffff881024179c00 0t0 353428260 /var/run/docker.sock
docker 6937 root 334r REG 9,2 220 1627499 /var/lib/docker/containers/70607073f3aa18cfcd05d8a3ef52b15bc59d4dbafaacd2f45f39890c7eee19e3/70607073f3aa18cfcd
05d8a3ef52b15bc59d4dbafaacd2f45f39890c7eee19e3-json.log (deleted)
docker 6937 root 335u unix 0xffff8809fd121f80 0t0 353441825 /var/run/docker.sock
docker 6937 root 336u unix 0xffff880166562300 0t0 353426334 /var/run/docker.sock
docker 6937 root 337r REG 9,2 75 1627531 /var/lib/docker/containers/6695d56daf8b312e34b613c24e995d3c71d14fba0ee1189db74a189572f01a11/6695d56daf8b312e34
b613c24e995d3c71d14fba0ee1189db74a189572f01a11-json.log (deleted)
docker 6937 root 338r REG 9,2 107 1627524 /var/lib/docker/containers/e9305143bc65c98b116a38aa56d80813e74923d213a1d0c8c2096293fce6814b/e9305143bc65c98b11
6a38aa56d80813e74923d213a1d0c8c2096293fce6814b-json.log (deleted)
docker 6937 root 339u unix 0xffff8809fd124d00 0t0 353442787 /var/run/docker.sock
docker 6937 root 340r REG 9,2 74 1627530 /var/lib/docker/containers/f6ba7a20569957ccee8ea51fb3ea181e1107572217312645e003472933b47fce/f6ba7a20569957ccee
8ea51fb3ea181e1107572217312645e003472933b47fce-json.log (deleted)
docker 6937 root 341u unix 0xffff881024179180 0t0 353455659 /var/run/docker.sock
docker 6937 root 342r REG 9,2 107 1627520 /var/lib/docker/containers/7b692bbda1e77df58b75c78394f7cee05ab476bf665c9d5ea8e6c454be1253e7/7b692bbda1e77df58b
75c78394f7cee05ab476bf665c9d5ea8e6c454be1253e7-json.log (deleted)
docker 6937 root 343u unix 0xffff8809fd123800 0t0 353473830 /var/run/docker.sock
docker 6937 root 344u unix 0xffff880a2eda3100 0t0 353485041 /var/run/docker.sock
docker 6937 root 345r REG 9,2 107 1627532 /var/lib/docker/containers/7f9706c3eb50d0047f0a254fedd8cfcf3c24d5ce6beb30ac17e488c54e600b11/7f9706c3eb50d0047f
0a254fedd8cfcf3c24d5ce6beb30ac17e488c54e600b11-json.log (deleted)
docker 6937 root 346r REG 9,2 219 1627536 /var/lib/docker/containers/93e745632e8ced642b9f87a2a6991aace88e4e36172d430e09715d3d093ab0bd/93e745632e8ced642b
9f87a2a6991aace88e4e36172d430e09715d3d093ab0bd-json.log (deleted)
docker 6937 root 347u unix 0xffff88015c303100 0t0 353485124 /var/run/docker.sock
docker 6937 root 348u unix 0xffff8801b046c600 0t0 353487224 /var/run/docker.sock
docker 6937 root 349r REG 9,2 75 1627538 /var/lib/docker/containers/5951bc184d7678055eb63f0b561657e12831b04d40c1c8b8548ca29d32d8ea85/5951bc184d7678055e
b63f0b561657e12831b04d40c1c8b8548ca29d32d8ea85-json.log (deleted)
docker 6937 root 350u unix 0xffff880166567380 0t0 353476902 /var/run/docker.sock
docker 6937 root 351r REG 9,2 107 1627534 /var/lib/docker/containers/4e3be7bcc0f2e3cc59f68191ed005ef2152821845cca2a16f94c1ea84d4cfb2f/4e3be7bcc0f2e3cc59
f68191ed005ef2152821845cca2a16f94c1ea84d4cfb2f-json.log (deleted)
docker 6937 root 352r REG 9,2 75 1627526 /var/lib/docker/containers/c1f591f680628cc90f0a7bde12d9defc17911cbaf2c8b4a01414cdeff2f30d90/c1f591f680628cc90f
0a7bde12d9defc17911cbaf2c8b4a01414cdeff2f30d90-json.log (deleted)