Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open files after docker containers removed #8693

Closed
tnachen opened this issue Oct 22, 2014 · 41 comments
Closed

Open files after docker containers removed #8693

tnachen opened this issue Oct 22, 2014 · 41 comments
Labels
kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. version/1.1

Comments

@tnachen
Copy link

tnachen commented Oct 22, 2014

docker version:

Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070

I ran out of file descriptors running docker run, and I realized docker daemon is holding on to a lot of file handles (11110 of them), while I only have 6 containers with docker ps -a

lots of them looks like this:

6c665c78a63396955dc76873bd8636bb642b38b43ec0e6-json.log (deleted)
docker 6937 root 329u unix 0xffff880166560e00 0t0 353408770 /var/run/docker.sock
docker 6937 root 330r REG 9,2 0 1627517 /var/lib/docker/containers/0734b3b41a4b039cf983cb690b1ad2814a8434289752f8082fd312e837f04672/0734b3b41a4b039cf9
83cb690b1ad2814a8434289752f8082fd312e837f04672-json.log (deleted)
docker 6937 root 331u unix 0xffff880166560a80 0t0 353427790 /var/run/docker.sock
docker 6937 root 332r REG 9,2 219 1627529 /var/lib/docker/containers/933d4e44aa370529284c40b5a1fae6ecf5e887e285d9860ba44ce61439c692f0/933d4e44aa37052928
4c40b5a1fae6ecf5e887e285d9860ba44ce61439c692f0-json.log (deleted)
docker 6937 root 333u unix 0xffff881024179c00 0t0 353428260 /var/run/docker.sock
docker 6937 root 334r REG 9,2 220 1627499 /var/lib/docker/containers/70607073f3aa18cfcd05d8a3ef52b15bc59d4dbafaacd2f45f39890c7eee19e3/70607073f3aa18cfcd
05d8a3ef52b15bc59d4dbafaacd2f45f39890c7eee19e3-json.log (deleted)
docker 6937 root 335u unix 0xffff8809fd121f80 0t0 353441825 /var/run/docker.sock
docker 6937 root 336u unix 0xffff880166562300 0t0 353426334 /var/run/docker.sock
docker 6937 root 337r REG 9,2 75 1627531 /var/lib/docker/containers/6695d56daf8b312e34b613c24e995d3c71d14fba0ee1189db74a189572f01a11/6695d56daf8b312e34
b613c24e995d3c71d14fba0ee1189db74a189572f01a11-json.log (deleted)
docker 6937 root 338r REG 9,2 107 1627524 /var/lib/docker/containers/e9305143bc65c98b116a38aa56d80813e74923d213a1d0c8c2096293fce6814b/e9305143bc65c98b11
6a38aa56d80813e74923d213a1d0c8c2096293fce6814b-json.log (deleted)
docker 6937 root 339u unix 0xffff8809fd124d00 0t0 353442787 /var/run/docker.sock
docker 6937 root 340r REG 9,2 74 1627530 /var/lib/docker/containers/f6ba7a20569957ccee8ea51fb3ea181e1107572217312645e003472933b47fce/f6ba7a20569957ccee
8ea51fb3ea181e1107572217312645e003472933b47fce-json.log (deleted)
docker 6937 root 341u unix 0xffff881024179180 0t0 353455659 /var/run/docker.sock
docker 6937 root 342r REG 9,2 107 1627520 /var/lib/docker/containers/7b692bbda1e77df58b75c78394f7cee05ab476bf665c9d5ea8e6c454be1253e7/7b692bbda1e77df58b
75c78394f7cee05ab476bf665c9d5ea8e6c454be1253e7-json.log (deleted)
docker 6937 root 343u unix 0xffff8809fd123800 0t0 353473830 /var/run/docker.sock
docker 6937 root 344u unix 0xffff880a2eda3100 0t0 353485041 /var/run/docker.sock
docker 6937 root 345r REG 9,2 107 1627532 /var/lib/docker/containers/7f9706c3eb50d0047f0a254fedd8cfcf3c24d5ce6beb30ac17e488c54e600b11/7f9706c3eb50d0047f
0a254fedd8cfcf3c24d5ce6beb30ac17e488c54e600b11-json.log (deleted)
docker 6937 root 346r REG 9,2 219 1627536 /var/lib/docker/containers/93e745632e8ced642b9f87a2a6991aace88e4e36172d430e09715d3d093ab0bd/93e745632e8ced642b
9f87a2a6991aace88e4e36172d430e09715d3d093ab0bd-json.log (deleted)
docker 6937 root 347u unix 0xffff88015c303100 0t0 353485124 /var/run/docker.sock
docker 6937 root 348u unix 0xffff8801b046c600 0t0 353487224 /var/run/docker.sock
docker 6937 root 349r REG 9,2 75 1627538 /var/lib/docker/containers/5951bc184d7678055eb63f0b561657e12831b04d40c1c8b8548ca29d32d8ea85/5951bc184d7678055e
b63f0b561657e12831b04d40c1c8b8548ca29d32d8ea85-json.log (deleted)
docker 6937 root 350u unix 0xffff880166567380 0t0 353476902 /var/run/docker.sock
docker 6937 root 351r REG 9,2 107 1627534 /var/lib/docker/containers/4e3be7bcc0f2e3cc59f68191ed005ef2152821845cca2a16f94c1ea84d4cfb2f/4e3be7bcc0f2e3cc59
f68191ed005ef2152821845cca2a16f94c1ea84d4cfb2f-json.log (deleted)
docker 6937 root 352r REG 9,2 75 1627526 /var/lib/docker/containers/c1f591f680628cc90f0a7bde12d9defc17911cbaf2c8b4a01414cdeff2f30d90/c1f591f680628cc90f
0a7bde12d9defc17911cbaf2c8b4a01414cdeff2f30d90-json.log (deleted)
@tnachen
Copy link
Author

tnachen commented Oct 22, 2014

Just a bit more context, we're calling docker logs --follow on the container, and after the container is removed also do a kill -TERM on the docker logs --follow process since there is an existing bug where docker logs --follow hangs forever if the container doesn't exist.
Not sure if it's related though.

@jessfraz
Copy link
Contributor

are you able to reproduce on the latest version of docker and if so can you please give exact steps to do so

@jessfraz
Copy link
Contributor

jessfraz commented Mar 2, 2015

closing as stale please ping me with exact steps to reproduce on the latest version and I will reopen

@jessfraz jessfraz closed this as completed Mar 2, 2015
@timothysc
Copy link

Not so quick on the draw @jfrazelle, here is your repo https://bugzilla.redhat.com/show_bug.cgi?id=1189028 , happens on 1.5 too.

@jessfraz jessfraz reopened this Mar 13, 2015
@jeremyeder
Copy link

  • Comparing static binary with dynbinary...try latest docker (non-dynbinary):
# date
Fri Mar 13 15:28:44 EDT 2015
# wget https://get.docker.com/builds/Linux/x86_64/docker-latest -O docker

# DEBUG=1 ./docker -d --storage-opt dm.metadatadev=/dev/docker/metadata --storage-opt dm.datadev=/dev/docker/data --selinux-enabled

# DEBUG=1 ./docker info | grep -i fd
Fds: 11
# ./docker run -t -i --rm busybox /bin/true
# DEBUG=1 ./docker info | grep -i fd
Fds: 12
# ./docker run -t -i --rm busybox /bin/true
# DEBUG=1 ./docker info | grep -i fd
Fds: 12
# ./docker run -t -i --rm busybox /bin/true
# DEBUG=1 ./docker info | grep -i fd
Fds: 12

# ls -al /proc/`pidof docker`/fd
total 0
dr-x------. 2 root root  0 Mar 13 15:21 .
dr-xr-xr-x. 8 root root  0 Mar 13 15:21 ..
lr-x------. 1 root root 64 Mar 13 15:21 0 -> /dev/null
lrwx------. 1 root root 64 Mar 13 15:21 1 -> socket:[157480]
lrwx------. 1 root root 64 Mar 13 15:21 14 -> socket:[174651]
lrwx------. 1 root root 64 Mar 13 15:21 18 -> socket:[177423]
lrwx------. 1 root root 64 Mar 13 15:21 2 -> socket:[157480]
lr-x------. 1 root root 64 Mar 13 15:21 3 -> /dev/urandom
lrwx------. 1 root root 64 Mar 13 15:21 4 -> /dev/mapper/control
lrwx------. 1 root root 64 Mar 13 15:21 5 -> socket:[183012]
lrwx------. 1 root root 64 Mar 13 15:21 6 -> /var/lib/docker/linkgraph.db
lrwx------. 1 root root 64 Mar 13 15:21 7 -> anon_inode:[eventpoll]
lr-x------. 1 root root 64 Mar 13 15:21 8 -> anon_inode:inotify
  • After creating/destroying a few containers the FD count remains constant.
  • Now with the dynbinary:
# systemctl start docker
# ls -al /proc/`pidof docker`/fd
total 0
dr-x------. 2 root root  0 Mar 13 15:23 .
dr-xr-xr-x. 8 root root  0 Mar 13 15:23 ..
lr-x------. 1 root root 64 Mar 13 15:23 0 -> /dev/null
lrwx------. 1 root root 64 Mar 13 15:23 1 -> socket:[211719]
lrwx------. 1 root root 64 Mar 13 15:23 2 -> socket:[211719]
lr-x------. 1 root root 64 Mar 13 15:23 3 -> /dev/urandom
lrwx------. 1 root root 64 Mar 13 15:23 4 -> /dev/mapper/control
lrwx------. 1 root root 64 Mar 13 15:23 5 -> /var/lib/docker/linkgraph.db
lr-x------. 1 root root 64 Mar 13 15:23 6 -> anon_inode:inotify
lrwx------. 1 root root 64 Mar 13 15:23 7 -> socket:[191437]
lrwx------. 1 root root 64 Mar 13 15:23 8 -> anon_inode:[eventpoll]
lrwx------. 1 root root 64 Mar 13 15:23 9 -> socket:[191439]

# for i in $(seq 10); do ./docker run -t -i --rm busybox /bin/true ; done

# ls -al /proc/`pidof docker`/fd
total 0
dr-x------. 2 root root  0 Mar 13 15:23 .
dr-xr-xr-x. 8 root root  0 Mar 13 15:23 ..
lr-x------. 1 root root 64 Mar 13 15:23 0 -> /dev/null
lrwx------. 1 root root 64 Mar 13 15:23 1 -> socket:[211719]
lrwx------. 1 root root 64 Mar 13 15:23 14 -> socket:[202348]
lrwx------. 1 root root 64 Mar 13 15:23 15 -> socket:[216107]
lrwx------. 1 root root 64 Mar 13 15:23 17 -> socket:[220181]
lrwx------. 1 root root 64 Mar 13 15:23 19 -> socket:[216153]
lrwx------. 1 root root 64 Mar 13 15:23 2 -> socket:[211719]
lrwx------. 1 root root 64 Mar 13 15:23 20 -> socket:[197957]
lrwx------. 1 root root 64 Mar 13 15:23 21 -> socket:[210670]
lrwx------. 1 root root 64 Mar 13 15:23 22 -> socket:[216488]
lrwx------. 1 root root 64 Mar 13 15:23 24 -> socket:[212262]
lrwx------. 1 root root 64 Mar 13 15:23 25 -> socket:[222317]
lrwx------. 1 root root 64 Mar 13 15:23 26 -> socket:[201685]
lrwx------. 1 root root 64 Mar 13 15:23 27 -> socket:[225364]
lr-x------. 1 root root 64 Mar 13 15:23 3 -> /dev/urandom
lrwx------. 1 root root 64 Mar 13 15:23 4 -> /dev/mapper/control
lrwx------. 1 root root 64 Mar 13 15:23 5 -> /var/lib/docker/linkgraph.db
lr-x------. 1 root root 64 Mar 13 15:23 6 -> anon_inode:inotify
lrwx------. 1 root root 64 Mar 13 15:23 7 -> socket:[191437]
lrwx------. 1 root root 64 Mar 13 15:23 8 -> anon_inode:[eventpoll]
  • All these extra sockets don't seem to ever close, and they keep growing as containers come and go...only seems to happen with dynbinary.
# for i in $(seq 100); do ./docker run -t -i --rm busybox /bin/true ; done
# DEBUG=1 docker info | grep Fd
Fds: 122
# ls -al /proc/`pidof docker`/fd | wc -l
123

@tianon
Copy link
Member

tianon commented Mar 13, 2015 via email

@rhvgoyal
Copy link
Contributor

@jeremyeder

I can't seem to reproduce this using your test case. Once for loop exits, number of Fds open still remain the same. I am using latest docker dynamically build binary on lvm thin pool as graphdriver.

@jeremyeder
Copy link

@rhvgoyal not sure if there are other parts to the puzzle, seemed pretty cut and dry to me. We have several other reports noted in https://bugzilla.redhat.com/show_bug.cgi?id=1189028

@rhvgoyal
Copy link
Contributor

@jeremyeder

That report also says that "It doesn't happen on RHEL7 + docker 1.4.1"

So looks like it does not happen on all combinations. I guess it is a good thing as it should help us pin point the problem. BTW, I am using upstream kernel 4.0.-rc2 and latest docker. Following is how my docker info looks like.

Containers: 0
Images: 130
Storage Driver: devicemapper
Pool Name: docker-docker--pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file:
Metadata file:
Data Space Used: 4.518 GB
Data Space Total: 489.6 GB
Data Space Available: 485.1 GB
Metadata Space Used: 11.5 MB
Metadata Space Total: 503.3 MB
Metadata Space Available: 491.8 MB
Udev Sync Supported: true
Library Version: 1.02.93 (2015-01-30)
Execution Driver: native-0.2
Kernel Version: 4.0.0-rc2+
Operating System: Fedora 21 (Twenty One)
CPUs: 24
Total Memory: 31.1 GiB

@42wim
Copy link

42wim commented Mar 19, 2015

Same issue here, using latest docker running on RHEL6.6.
Doesn't matter if using static binary from docker or dynbinary from EPEL

# for i in $(seq 100); do docker run -t -i --rm busybox /bin/true ; done
# /proc/`pidof docker`/fd 21:42 # ls -al|wc
    247    2816   26061

Containing this stuff:

lr-x------ 1 root root 64 Mar 19 21:41 224 -> /cgroup/memory/docker/4702d400101a5c598181d37711049343263910e2876f161b46241c5e9dbdcff2/memory.oom_control (deleted)
lr-x------ 1 root root 64 Mar 19 21:41 226 -> /cgroup/memory/docker/952f932435c2b6bf9c11181a9a1e9fc0a4534134a1a789f05a87a12c22da4e7c/memory.oom_control (deleted)
lrwx------ 1 root root 64 Mar 19 21:41 227 -> [eventfd]
lr-x------ 1 root root 64 Mar 19 21:41 228 -> /cgroup/memory/docker/12f831d86f76ba5a02bee444625feb5779f6d76508f0788d3436e3c57ef36c89/memory.oom_control (deleted)

# ./docker --version
Docker version 1.5.0, build a8a31ef

# ./docker info
Containers: 150
Images: 436
Storage Driver: devicemapper
 Pool Name: docker-253:5-73734-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 36.89 GB
 Data Space Total: 107.4 GB
 Metadata Space Used: 42.81 MB
 Metadata Space Total: 2.147 GB
 Udev Sync Supported: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.8.1.el6.x86_64
Operating System: <unknown>
CPUs: 8
Total Memory: 7.684 GiB

@iangkent
Copy link

We are experiencing the same issue with docker 1.4.1 on RHEL 6.5 (2.6.32-504.8.1.el6.x86_64).
Our current workaround is to restart docker daemon.

Is there a better recommend workaround?

@amaltson
Copy link

I'm experiencing the exact same problem with Docker 1.6 and RHEL 6.6. We ran into this when the docker user reached it's limit on open files. I took a look at lsof and found a lot of the (deleted) file descriptors pointing to removed Docker containers.

# lsof -nP | grep docker
docker     2725      root  cwd       DIR              253,0       4096          2 /
docker     2725      root  rtd       DIR              253,0       4096          2 /
docker     2725      root  txt       REG              253,0   15443675     149299 /usr/bin/docker
docker     2725      root    1w      REG              253,0   14221171     815016 /var/log/docker
docker     2725      root    2w      REG              253,0   14221171     815016 /var/log/docker
docker     2725      root    3r      CHR                1,9        0t0       4170 /dev/urandom
docker     2725      root    4u      CHR              10,58        0t0       6084 /dev/mapper/control
docker     2725      root    5r      REG               0,28          0   19076234 /cgroup/memory/docker/8b155e930840e09e36ad7ddb65598e7412dbddf2a64add39fe0775ab7ef02e37/memory.oom_control (deleted)
docker     2725      root    6u     unix 0xffff880236faeb00        0t0      16234 /var/run/docker.sock
docker     2725      root    7u      REG                0,9          0       4163 [eventpoll]
docker     2725      root    8u      REG              253,2      26624   29633572 /transient/opt/docker/linkgraph.db
docker     2725      root    9u      REG              253,2      26624   29633572 /transient/opt/docker/linkgraph.db
docker     2725      root   10r      DIR               0,10          0          1 inotify
docker     2725      root   11r      REG               0,28          0   32400713 /cgroup/memory/docker/0ad1fc937d3ed540a2ab1656059d6205e739fa079d539a6740d4d6d542cfd3d1/memory.oom_control (deleted)
docker     2725      root   12r      REG               0,28          0   17925027 /cgroup/memory/docker/33dcb46c0d288198318c7e9f901b8e0ba7b6a5c390828529d7c9bfcb385b9d2f/memory.oom_control (deleted)
docker     2725      root   13u      REG                0,9          0       4163 [eventfd]
....
< over a 1000 more here>

Here's the docker info

# docker info
Containers: 2
Images: 114
Storage Driver: devicemapper
 Pool Name: docker-253:2-29633562-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 9.234 GB
 Data Space Total: 107.4 GB
 Data Space Available: 98.14 GB
 Metadata Space Used: 10.08 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.137 GB
 Udev Sync Supported: false
 Data loop file: /transient/opt/docker/devicemapper/devicemapper/data
 Metadata loop file: /transient/opt/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.16.2.el6.x86_64
Operating System: <unknown>
CPUs: 4
Total Memory: 7.681 GiB
Name: devbuild1
ID: BURZ:PUHG:ECHK:KRO2:7N3L:HD7O:TJZE:VQP2:LFQL:3N3R:STCS:LBSE

@JohnMorales
Copy link

@iangkent: Another work around is to attach to the docker process and tell it to close the file descriptors.

I'm doing something like this:

gdb --batch-silent -p $(cat /var/run/docker.pid) <<< "$( ls /proc/$(cat /var/run/docker.pid)/fd -l --time-style=+'%s' | grep -E 'deleted' | awk '{ printf("p close(%s)\n", $7)}')"

Note: what I'm doing here is attaching to the docker daemon using gdb, and telling it to close file descriptors that have 'deleted' in the file listing..

It's ugly but better than restarting the containers that I want to keep running.

UPDATE: I noticed that killing eventfd descriptors can cause a tainted panic. Updated command to not remove those eventfd

@LK4D4
Copy link
Contributor

LK4D4 commented Jun 5, 2015

It's pretty weird that there is fds from cgroups :/ I'll try to reproduce.

@JohnMorales
Copy link

@LK4D4 sweet, if there is any info you could need please let me know, this is an annoying problem.

As @42wim said it's as simple as using the latest docker in RHEL6.6

@JohnMorales
Copy link

@LK4D4: it always seems to be the 'memory.oom_control' file that is left open for each cgroup instance.

@LK4D4
Copy link
Contributor

LK4D4 commented Jun 5, 2015

Hmm, I really can't reproduce on master. Even if I thought that I saw this before.

@induane
Copy link

induane commented Jun 8, 2015

It's present at least as recently as docker 1.6 - If I look at the file descriptors I end up with:

sudo ls /proc/$(cat /var/run/docker.pid)/fd -l --time-style=+'%s'
total 0
lr-x------. 1 root root 64 1433797124 0 -> pipe:[4511692]
l-wx------. 1 root root 64 1433797124 1 -> /var/log/docker
lrwx------. 1 root root 64 1433797315 10 -> socket:[4543140]
lrwx------. 1 root root 64 1433797364 100 -> /var/lib/docker/containers/ef02c72cf1190048f693042e34e1a4117c506246cebe5e0067c92842e177c15d/ef02c72cf1190048f693042e34e1a4117c506246cebe5e0067c92842e177c15d-json.log
lrwx------. 1 root root 64 1433797348 101 -> [eventfd]
lrwx------. 1 root root 64 1433797364 102 -> /var/lib/docker/containers/b8152fdb4ef387bd59a61c5b311dc89082c368673cdc113dc48ad81fcd1dd81a/b8152fdb4ef387bd59a61c5b311dc89082c368673cdc113dc48ad81fcd1dd81a-json.log
lrwx------. 1 root root 64 1433797364 103 -> /var/lib/docker/containers/b8152fdb4ef387bd59a61c5b311dc89082c368673cdc113dc48ad81fcd1dd81a/b8152fdb4ef387bd59a61c5b311dc89082c368673cdc113dc48ad81fcd1dd81a-json.log
lrwx------. 1 root root 64 1433797364 104 -> socket:[4542056]
lr-x------. 1 root root 64 1433797364 105 -> pipe:[4542057]
lr-x------. 1 root root 64 1433797364 106 -> /cgroup/memory/docker/ef02c72cf1190048f693042e34e1a4117c506246cebe5e0067c92842e177c15d/memory.oom_control
lr-x------. 1 root root 64 1433797364 107 -> pipe:[4542058]
lrwx------. 1 root root 64 1433797378 108 -> /var/lib/docker/containers/a05678f737dde4a62fd9e94289ae850f766efbf71b5fb60ec73a70f6fe16a302/a05678f737dde4a62fd9e94289ae850f766efbf71b5fb60ec73a70f6fe16a302-json.log
lrwx------. 1 root root 64 1433797364 109 -> [eventfd]
lrwx------. 1 root root 64 1433797315 11 -> /var/lib/docker/containers/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf-json.log
lrwx------. 1 root root 64 1433797364 110 -> socket:[4542319]
lr-x------. 1 root root 64 1433797364 111 -> pipe:[4542320]
lr-x------. 1 root root 64 1433797364 112 -> /cgroup/memory/docker/b8152fdb4ef387bd59a61c5b311dc89082c368673cdc113dc48ad81fcd1dd81a/memory.oom_control
lr-x------. 1 root root 64 1433797364 113 -> pipe:[4542321]
lrwx------. 1 root root 64 1433797364 114 -> [eventfd]
lr-x------. 1 root root 64 1433797378 115 -> /cgroup/memory/docker/a05678f737dde4a62fd9e94289ae850f766efbf71b5fb60ec73a70f6fe16a302/memory.oom_control
lrwx------. 1 root root 64 1433797378 117 -> [eventfd]
lrwx------. 1 root root 64 1433797378 118 -> socket:[4544837]
lr-x------. 1 root root 64 1433797378 119 -> pipe:[4544838]
lr-x------. 1 root root 64 1433797315 12 -> /cgroup/memory/docker/a5d7c8b0389cb1984e8b89b985dbf70cbbdb470f50f82b4d1e52fe87009b724a/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797378 121 -> pipe:[4544839]
lr-x------. 1 root root 64 1433797315 13 -> /cgroup/memory/docker/dde9f099822f9e3449d9ba6331e70bcb57f9da4dbd7e99d920db7e1c0b6668ea/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 14 -> [eventfd]
lrwx------. 1 root root 64 1433797315 15 -> /var/lib/docker/containers/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf-json.log
lrwx------. 1 root root 64 1433797315 16 -> /cgroup/memory/docker/eb2d0297059e41d424020de288a829d7fdecf57a6a1997f4f1d01cbd3147f8ba/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 17 -> /cgroup/memory/docker/f7bb76da79cc96994de160a0d5ab5d1e143fe963bcda79516c4f022a0e77e33f/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 18 -> /cgroup/memory/docker/01750e1ed1558bb71d9e2b083e8b280c3c5969c728f95c0806079d73894257ad/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 19 -> /cgroup/memory/docker/f700e9c8443b8091d8c361ac99e674a333820c94c9e444286d24be69d368c061/memory.oom_control (deleted)
l-wx------. 1 root root 64 1433797124 2 -> /var/log/docker
lr-x------. 1 root root 64 1433797315 20 -> /cgroup/memory/docker/2085c48435f324c1e2b75c4cc2096d73ad07ae7157f96e35c4295effd8c7aa27/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 21 -> /cgroup/memory/docker/8e14c68bb68b4fd8fb1bf4c6962942446b94fe959932315df6e357fc92a7de82/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 22 -> [eventfd]
lr-x------. 1 root root 64 1433797315 23 -> /cgroup/memory/docker/97a87edb8ba32c33e0088441999ef0f20b1562d07974a026f105b559bb5b9acf/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 24 -> /cgroup/memory/docker/1def4fa783293557a444181c311a26dd74a1a56ffbfc22505114ff0c3c08bc7a/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 25 -> /cgroup/memory/docker/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf/memory.oom_control
lrwx------. 1 root root 64 1433797315 26 -> [eventfd]
lrwx------. 1 root root 64 1433797315 27 -> [eventfd]
lr-x------. 1 root root 64 1433797315 28 -> /cgroup/memory/docker/7b11f512da9cf761afb33cd770893897723bdf556fd74da01cd534df311bf2cb/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 29 -> [eventfd]
lr-x------. 1 root root 64 1433797124 3 -> /dev/urandom
lrwx------. 1 root root 64 1433797315 30 -> [eventfd]
lrwx------. 1 root root 64 1433797315 31 -> [eventfd]
lr-x------. 1 root root 64 1433797315 32 -> /cgroup/memory/docker/10f1cdc52debf0975428e3b571e27b104c4f3a1c3b08ebba40f1cc9cb8e28a4c/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 33 -> [eventfd]
lrwx------. 1 root root 64 1433797315 34 -> /cgroup/memory/docker/27da402da614bf566987eba81bc03d1cbad5e507e4f89a051784c66a5a55d317/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 35 -> /cgroup/memory/docker/6c74e6e98bfe10fce8705cae1e7b7232f64e0963dab8d34b13d73d80186f3a67/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 36 -> /cgroup/memory/docker/21efccc9fe40fb8134bdc57e6f3dc81dec1b1b3676814ce157d8a0eda542aeaf/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 37 -> [eventfd]
lrwx------. 1 root root 64 1433797315 38 -> [eventfd]
lrwx------. 1 root root 64 1433797315 39 -> [eventfd]
lrwx------. 1 root root 64 1433797124 4 -> /dev/mapper/control
lrwx------. 1 root root 64 1433797315 40 -> [eventfd]
lr-x------. 1 root root 64 1433797315 41 -> /cgroup/memory/docker/4015cc18c4f53eb98ff572e3c9d0a7f1f8f692ffd44c8e9f0bd5c276ec0dad0d/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 42 -> /cgroup/memory/docker/74499a76d96b3a80df8f6be635932b82c26ab48088785e3007275bd213a74636/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 43 -> [eventfd]
lrwx------. 1 root root 64 1433797315 44 -> [eventfd]
lrwx------. 1 root root 64 1433797315 45 -> [eventfd]
lrwx------. 1 root root 64 1433797315 46 -> socket:[4544000]
lrwx------. 1 root root 64 1433797315 47 -> [eventfd]
lrwx------. 1 root root 64 1433797315 48 -> /cgroup/memory/docker/d11ce6756e62c9f1f1fe76df459bb086b528c713e45aacd884907fffdfe4cf88/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 49 -> /cgroup/memory/docker/9ac94647fd1728bd00568814a67e79a18e43ee0595e7f11c3a4e5c13c54338e8/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797206 5 -> /var/lib/docker/linkgraph.db
lrwx------. 1 root root 64 1433797315 50 -> [eventfd]
lrwx------. 1 root root 64 1433797315 51 -> [eventfd]
lr-x------. 1 root root 64 1433797315 52 -> /cgroup/memory/docker/9b2fcd32ad5d1c25f53bd22a3236e68f64fa9b0ab92943c6630f7c93b2d09fa6/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 53 -> /cgroup/memory/docker/d5954b25a28d23153911e90e4863ebc6b6d9bf140bfd3e7177bc2c78224670e0/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 54 -> [eventfd]
lrwx------. 1 root root 64 1433797315 55 -> [eventfd]
lrwx------. 1 root root 64 1433797315 56 -> [eventfd]
lrwx------. 1 root root 64 1433797315 57 -> /cgroup/memory/docker/e27d34618f7283039d7e33231b473bcc0523cc977d70390566badcd97b15a594/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 58 -> socket:[4543636]
lrwx------. 1 root root 64 1433797315 59 -> [eventfd]
lr-x------. 1 root root 64 1433797206 6 -> inotify
lr-x------. 1 root root 64 1433797315 60 -> /cgroup/memory/docker/67b9c79858af4815819aef2b88482e3d8526eac5e3e830bdaea72a580bfb430e/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 61 -> [eventfd]
lrwx------. 1 root root 64 1433797315 62 -> [eventfd]
lr-x------. 1 root root 64 1433797315 63 -> pipe:[4543637]
lr-x------. 1 root root 64 1433797328 64 -> /dev/dm-5
lrwx------. 1 root root 64 1433797315 65 -> pipe:[4543638]
lr-x------. 1 root root 64 1433797315 67 -> /cgroup/memory/docker/c9f5255689a4cce3dd075898ee80d9a014e57d0a832b74bb74bcdc131855cdd0/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 68 -> [eventfd]
lr-x------. 1 root root 64 1433797315 69 -> /cgroup/memory/docker/1ec545d3b37ba926d9898eef2d1f7c188ae2e1eea4c6ff7b3501dddc0a410b04/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 7 -> socket:[4543602]
lrwx------. 1 root root 64 1433797329 71 -> /cgroup/memory/docker/b000c864a5c834af40a0b741ee296e66afb1264dab2605a544adc7aaba8d0a91/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797331 72 -> /cgroup/memory/docker/74dfcf658f19c0f52fc4b2d9faa5d744d251b8e3bd54f5f24c0a88db860c2091/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 73 -> /cgroup/memory/docker/e0f7b680cba4e2af80a0956f5c626e393736600c203e16ae5d24a3f4b438840a/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797331 74 -> [eventfd]
lrwx------. 1 root root 64 1433797331 75 -> [eventfd]
lr-x------. 1 root root 64 1433797315 76 -> /cgroup/memory/docker/ec8262f4e17b74923bd0849e428c0cb12fd32c2fb9a8385e8de459e614802551/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 77 -> [eventfd]
lrwx------. 1 root root 64 1433797315 78 -> [eventfd]
lrwx------. 1 root root 64 1433797315 79 -> [eventfd]
lrwx------. 1 root root 64 1433797124 8 -> socket:[4511739]
lrwx------. 1 root root 64 1433797315 80 -> socket:[4544718]
lr-x------. 1 root root 64 1433797315 81 -> /cgroup/memory/docker/1f75b718831acf4836bca27c41556e07a9ebad6f5fadb46160f26cf95f66f9f1/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 84 -> /cgroup/memory/docker/1c452d966ac6bd2050d7893a374804fb7070d81bb0d6e3eee92ed429d425b64b/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 85 -> socket:[4541761]
lr-x------. 1 root root 64 1433797315 86 -> /cgroup/memory/docker/c15bed7c330bb387a34c80ce7acb8ef0e95ac6d288a8c5af3f3427f8e946ea74/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 87 -> [eventfd]
lr-x------. 1 root root 64 1433797331 88 -> socket:[4544804]
lrwx------. 1 root root 64 1433797331 89 -> [eventfd]
lrwx------. 1 root root 64 1433797124 9 -> [eventpoll]
lrwx------. 1 root root 64 1433797315 91 -> [eventfd]
lr-x------. 1 root root 64 1433797315 92 -> socket:[4541996]
lrwx------. 1 root root 64 1433797348 93 -> /var/lib/docker/containers/a05678f737dde4a62fd9e94289ae850f766efbf71b5fb60ec73a70f6fe16a302/a05678f737dde4a62fd9e94289ae850f766efbf71b5fb60ec73a70f6fe16a302-json.log
lr-x------. 1 root root 64 1433797315 94 -> socket:[4541765]
lr-x------. 1 root root 64 1433797348 95 -> /cgroup/memory/docker/487e2b8d46322a9013be67c0cd67ac89b33e788754dce59a294ea558493fbde2/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797348 96 -> socket:[4542016]
lr-x------. 1 root root 64 1433797348 97 -> /cgroup/memory/docker/b54d79a4016f0de3978b0d8b859217953d080843791705c5996686c2130f935c/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797348 98 -> [eventfd]
lrwx------. 1 root root 64 1433797364 99 -> /var/lib/docker/containers/ef02c72cf1190048f693042e34e1a4117c506246cebe5e0067c92842e177c15d/ef02c72cf1190048f693042e34e1a4117c506246cebe5e0067c92842e177c15d-json.log
[propylon@rnd-stage ~]$ sudo gdb --batch-silent -p $(sudo cat /var/run/docker.pid) &lt;&lt;&lt; "$( sudo ls /proc/$(cat /var/run/docker.pid)/fd -l --time-style=+'%s' | grep -E 'deleted' | awk '{ printf("p close(%s)\n", $7)}')"
[propylon@rnd-stage ~]$ sudo ls /proc/$(cat /var/run/docker.pid)/fd -l --time-style=+'%s'total 0
lr-x------. 1 root root 64 1433797124 0 -> pipe:[4511692]
l-wx------. 1 root root 64 1433797124 1 -> /var/log/docker
lrwx------. 1 root root 64 1433797315 10 -> socket:[4543140]
lrwx------. 1 root root 64 1433797348 101 -> [eventfd]
lr-x------. 1 root root 64 1433797364 106 -> /cgroup/memory/docker/ef02c72cf1190048f693042e34e1a4117c506246cebe5e0067c92842e177c15d/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797364 109 -> [eventfd]
lrwx------. 1 root root 64 1433797315 11 -> /var/lib/docker/containers/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf-json.log
lr-x------. 1 root root 64 1433797364 112 -> /cgroup/memory/docker/b8152fdb4ef387bd59a61c5b311dc89082c368673cdc113dc48ad81fcd1dd81a/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797364 114 -> [eventfd]
lr-x------. 1 root root 64 1433797378 115 -> /cgroup/memory/docker/a05678f737dde4a62fd9e94289ae850f766efbf71b5fb60ec73a70f6fe16a302/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797378 117 -> [eventfd]
lr-x------. 1 root root 64 1433797315 12 -> /cgroup/memory/docker/a5d7c8b0389cb1984e8b89b985dbf70cbbdb470f50f82b4d1e52fe87009b724a/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 13 -> /cgroup/memory/docker/dde9f099822f9e3449d9ba6331e70bcb57f9da4dbd7e99d920db7e1c0b6668ea/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 14 -> [eventfd]
lrwx------. 1 root root 64 1433797315 15 -> /var/lib/docker/containers/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf-json.log
lrwx------. 1 root root 64 1433797315 16 -> /cgroup/memory/docker/eb2d0297059e41d424020de288a829d7fdecf57a6a1997f4f1d01cbd3147f8ba/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 17 -> /cgroup/memory/docker/f7bb76da79cc96994de160a0d5ab5d1e143fe963bcda79516c4f022a0e77e33f/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 18 -> /cgroup/memory/docker/01750e1ed1558bb71d9e2b083e8b280c3c5969c728f95c0806079d73894257ad/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 19 -> /cgroup/memory/docker/f700e9c8443b8091d8c361ac99e674a333820c94c9e444286d24be69d368c061/memory.oom_control (deleted)
l-wx------. 1 root root 64 1433797124 2 -> /var/log/docker
lr-x------. 1 root root 64 1433797315 20 -> /cgroup/memory/docker/2085c48435f324c1e2b75c4cc2096d73ad07ae7157f96e35c4295effd8c7aa27/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 21 -> /cgroup/memory/docker/8e14c68bb68b4fd8fb1bf4c6962942446b94fe959932315df6e357fc92a7de82/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 22 -> [eventfd]
lr-x------. 1 root root 64 1433797315 23 -> /cgroup/memory/docker/97a87edb8ba32c33e0088441999ef0f20b1562d07974a026f105b559bb5b9acf/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 24 -> /cgroup/memory/docker/1def4fa783293557a444181c311a26dd74a1a56ffbfc22505114ff0c3c08bc7a/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 25 -> /cgroup/memory/docker/7019a02269cae5583b003134f3e47c46baac4cf06f310430c668e3c5b8ae8fdf/memory.oom_control
lrwx------. 1 root root 64 1433797315 26 -> [eventfd]
lrwx------. 1 root root 64 1433797315 27 -> [eventfd]
lr-x------. 1 root root 64 1433797315 28 -> /cgroup/memory/docker/7b11f512da9cf761afb33cd770893897723bdf556fd74da01cd534df311bf2cb/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 29 -> [eventfd]
lr-x------. 1 root root 64 1433797124 3 -> /dev/urandom
lrwx------. 1 root root 64 1433797315 30 -> [eventfd]
lrwx------. 1 root root 64 1433797315 31 -> [eventfd]
lr-x------. 1 root root 64 1433797315 32 -> /cgroup/memory/docker/10f1cdc52debf0975428e3b571e27b104c4f3a1c3b08ebba40f1cc9cb8e28a4c/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 33 -> [eventfd]
lrwx------. 1 root root 64 1433797315 34 -> /cgroup/memory/docker/27da402da614bf566987eba81bc03d1cbad5e507e4f89a051784c66a5a55d317/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 35 -> /cgroup/memory/docker/6c74e6e98bfe10fce8705cae1e7b7232f64e0963dab8d34b13d73d80186f3a67/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 36 -> /cgroup/memory/docker/21efccc9fe40fb8134bdc57e6f3dc81dec1b1b3676814ce157d8a0eda542aeaf/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 37 -> [eventfd]
lrwx------. 1 root root 64 1433797315 38 -> [eventfd]
lrwx------. 1 root root 64 1433797315 39 -> [eventfd]
lrwx------. 1 root root 64 1433797124 4 -> /dev/mapper/control
lrwx------. 1 root root 64 1433797315 40 -> [eventfd]
lr-x------. 1 root root 64 1433797315 41 -> /cgroup/memory/docker/4015cc18c4f53eb98ff572e3c9d0a7f1f8f692ffd44c8e9f0bd5c276ec0dad0d/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 42 -> /cgroup/memory/docker/74499a76d96b3a80df8f6be635932b82c26ab48088785e3007275bd213a74636/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 43 -> [eventfd]
lrwx------. 1 root root 64 1433797315 44 -> [eventfd]
lrwx------. 1 root root 64 1433797315 45 -> [eventfd]
lrwx------. 1 root root 64 1433797315 46 -> socket:[4545833]
lrwx------. 1 root root 64 1433797315 47 -> [eventfd]
lrwx------. 1 root root 64 1433797315 48 -> /cgroup/memory/docker/d11ce6756e62c9f1f1fe76df459bb086b528c713e45aacd884907fffdfe4cf88/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 49 -> /cgroup/memory/docker/9ac94647fd1728bd00568814a67e79a18e43ee0595e7f11c3a4e5c13c54338e8/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797206 5 -> /var/lib/docker/linkgraph.db
lrwx------. 1 root root 64 1433797315 50 -> [eventfd]
lrwx------. 1 root root 64 1433797315 51 -> [eventfd]
lr-x------. 1 root root 64 1433797315 52 -> /cgroup/memory/docker/9b2fcd32ad5d1c25f53bd22a3236e68f64fa9b0ab92943c6630f7c93b2d09fa6/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 53 -> /cgroup/memory/docker/d5954b25a28d23153911e90e4863ebc6b6d9bf140bfd3e7177bc2c78224670e0/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 54 -> [eventfd]
lrwx------. 1 root root 64 1433797315 55 -> [eventfd]
lrwx------. 1 root root 64 1433797315 56 -> [eventfd]
lrwx------. 1 root root 64 1433797315 57 -> /cgroup/memory/docker/e27d34618f7283039d7e33231b473bcc0523cc977d70390566badcd97b15a594/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 58 -> socket:[4543636]
lrwx------. 1 root root 64 1433797315 59 -> [eventfd]
lr-x------. 1 root root 64 1433797206 6 -> inotify
lr-x------. 1 root root 64 1433797315 60 -> /cgroup/memory/docker/67b9c79858af4815819aef2b88482e3d8526eac5e3e830bdaea72a580bfb430e/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 61 -> [eventfd]
lrwx------. 1 root root 64 1433797315 62 -> [eventfd]
lr-x------. 1 root root 64 1433797315 63 -> pipe:[4543637]
lr-x------. 1 root root 64 1433797328 64 -> /var/lib/docker/containers/9865e7f70a88af4538a891ce9e013f349d3c9feb97be553906ffdadc1510f0d3/9865e7f70a88af4538a891ce9e013f349d3c9feb97be553906ffdadc1510f0d3-json.log
lrwx------. 1 root root 64 1433797315 65 -> pipe:[4543638]
lrwx------. 1 root root 64 1433797315 66 -> socket:[4545441]
lr-x------. 1 root root 64 1433797315 67 -> /cgroup/memory/docker/c9f5255689a4cce3dd075898ee80d9a014e57d0a832b74bb74bcdc131855cdd0/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 68 -> [eventfd]
lr-x------. 1 root root 64 1433797315 69 -> /cgroup/memory/docker/1ec545d3b37ba926d9898eef2d1f7c188ae2e1eea4c6ff7b3501dddc0a410b04/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 7 -> socket:[4543602]
lrwx------. 1 root root 64 1433797331 70 -> /var/lib/docker/containers/9865e7f70a88af4538a891ce9e013f349d3c9feb97be553906ffdadc1510f0d3/9865e7f70a88af4538a891ce9e013f349d3c9feb97be553906ffdadc1510f0d3-json.log
lrwx------. 1 root root 64 1433797329 71 -> /cgroup/memory/docker/b000c864a5c834af40a0b741ee296e66afb1264dab2605a544adc7aaba8d0a91/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797331 72 -> /cgroup/memory/docker/74dfcf658f19c0f52fc4b2d9faa5d744d251b8e3bd54f5f24c0a88db860c2091/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797315 73 -> /cgroup/memory/docker/e0f7b680cba4e2af80a0956f5c626e393736600c203e16ae5d24a3f4b438840a/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797331 74 -> [eventfd]
lrwx------. 1 root root 64 1433797331 75 -> [eventfd]
lr-x------. 1 root root 64 1433797315 76 -> /cgroup/memory/docker/ec8262f4e17b74923bd0849e428c0cb12fd32c2fb9a8385e8de459e614802551/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 77 -> [eventfd]
lrwx------. 1 root root 64 1433797315 78 -> [eventfd]
lrwx------. 1 root root 64 1433797315 79 -> [eventfd]
lrwx------. 1 root root 64 1433797124 8 -> socket:[4511739]
lrwx------. 1 root root 64 1433797315 80 -> socket:[4544718]
lr-x------. 1 root root 64 1433797315 81 -> /cgroup/memory/docker/1f75b718831acf4836bca27c41556e07a9ebad6f5fadb46160f26cf95f66f9f1/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 82 -> /cgroup/memory/docker/9865e7f70a88af4538a891ce9e013f349d3c9feb97be553906ffdadc1510f0d3/memory.oom_control
lr-x------. 1 root root 64 1433797315 84 -> /cgroup/memory/docker/1c452d966ac6bd2050d7893a374804fb7070d81bb0d6e3eee92ed429d425b64b/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 85 -> [eventfd]
lr-x------. 1 root root 64 1433797315 86 -> /cgroup/memory/docker/c15bed7c330bb387a34c80ce7acb8ef0e95ac6d288a8c5af3f3427f8e946ea74/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797315 87 -> [eventfd]
lr-x------. 1 root root 64 1433797331 88 -> socket:[4545889]
lrwx------. 1 root root 64 1433797331 89 -> [eventfd]
lrwx------. 1 root root 64 1433797124 9 -> [eventpoll]
lr-x------. 1 root root 64 1433797331 90 -> pipe:[4545890]
lrwx------. 1 root root 64 1433797315 91 -> [eventfd]
lr-x------. 1 root root 64 1433797315 92 -> /dev/dm-6
lrwx------. 1 root root 64 1433797348 93 -> pipe:[4545891]
lr-x------. 1 root root 64 1433797315 94 -> socket:[4541765]
lr-x------. 1 root root 64 1433797348 95 -> /cgroup/memory/docker/487e2b8d46322a9013be67c0cd67ac89b33e788754dce59a294ea558493fbde2/memory.oom_control (deleted)
lr-x------. 1 root root 64 1433797348 96 -> socket:[4542016]
lr-x------. 1 root root 64 1433797348 97 -> /cgroup/memory/docker/b54d79a4016f0de3978b0d8b859217953d080843791705c5996686c2130f935c/memory.oom_control (deleted)
lrwx------. 1 root root 64 1433797348 98 -> [eventfd]
lrwx------. 1 root root 64 1433797364 99 -> socket:[4546418]

@tombee
Copy link
Contributor

tombee commented Jun 9, 2015

Seeing this on RHEL 6.6 2.6.32-504.el6.x86_64 too.

docker  7243 root   68r   REG               0,21         0  719523 /cgroup/memory/docker/19fabdddc55310941288ddeeacc4ac8e5d7f01afe85a206c1cf3973b22a55b1e/memory.oom_control (deleted)
docker  7243 root   70r   REG               0,21         0  720401 /cgroup/memory/docker/a9828f0187b801104c5803d084412f91f53e4e1c55fdca469d26fca80f926e4d/memory.oom_control (deleted)
docker  7243 root   72r   REG               0,21         0  722053 /cgroup/memory/docker/34f931808cb158383a25ee0e54fea570210806e482e32ee8fa888cc8905b78c2/memory.oom_control (deleted)
docker  7243 root   74r   REG               0,21         0  721475 /cgroup/memory/docker/562809ae20f2fcf877df0f99694ab3b598640ec928ecff380d0bbc777fc96806/memory.oom_control (deleted)

@amaltson
Copy link

I'm guessing this hasn't been fixed in Docker 1.7? 😢

@induane
Copy link

induane commented Jun 19, 2015

I've had to downgrade to 1.3 as the issue doesn't exist there. I haven't bothered doing a git bisect to try to triage the exact commit that introduced it yet as I simply haven't had the time. It may be simply that the bug won't get a lot of attention since most people running docker are doing so on RHEL7.

@jessfraz jessfraz added the kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. label Jul 10, 2015
@jsipprell
Copy link

@amaltson it is not fixed as of current master (1.7.1 -> 1.8-in-devel) on EL6.6.

@jsipprell
Copy link

@rhvgoyal This issue is caused by a spurious read on the eventfd in libcontainer/notify_linux.go:notifyOnOOM, although the underlying cause of the spurious read remains undetermined (apparently only on EL6/Centos?).

Because the read occurs before the cgroup's vfs entries have been unlinked, there is a race condition wherein the channel blocking send is reached before the container is cleaned up, the container destroyed and the channel read, and finally the for loop restarts reading the eventfd -- no additional events occur after this first "spurious" one and thus the goroutine/fd leaks. Adding an Lstat test after ch <- struct{}{} allows correct detection of a destroyed container with no leaked fds once synchronization via the channel has occurred.

I can post my patch if it would illustrate this more clearly, although I suspect someone will want to do a bit more thorough job refactoring the loop.

@unixist
Copy link

unixist commented Aug 25, 2015

Bump.

@jsipprell
Copy link

@unixist (& others interested):

I suggest you test under EL/Centos 6.7. I am unable to replicate the problem on 2.6.32-573.3.1.el6 , and the kernel-2.6.32-573.3.1.el6 rpm has an official RH bugfix that looks suspiciously related.

[root@hiro ~]# uname -a
Linux hiro 2.6.32-504.23.4.el6.x86_64 #1 SMP Tue Jun 9 20:57:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@hiro ~]# cgcreate -g memory:/foo
[root@hiro ~]# ( sleep 1 && rmdir /cgroup/memory/foo )& ./cgroupoom foo
[1] 11757
2015/08/25 23:03:03 monitoring cgroup "foo"
2015/08/25 23:03:04 cgroup "foo": OOM!

meanwhile …

[root@yt ~]# uname -a
Linux yt 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@yt ~]# cgcreate -g memory:/foo
[root@yt ~]# ( sleep 1 && rmdir /cgroup/memory/foo )& ./cgroupoom foo
[1] 21861
2015/08/25 23:05:14 monitoring cgroup "foo"
2015/08/25 23:05:15 cgroup "foo" destroyed

(cgroupoom is just a simple utility that uses the same notification code as docker 1.7's libcontainer)

@thaJeztah
Copy link
Member

@unixist can you try if that newer kernel resolves it for you?

@unixist
Copy link

unixist commented Sep 8, 2015

@thaJeztah I hope to be able to test soon. I don't yet have -573 in a position to test.

Can anybody else comment?

@csmithATsquiz
Copy link

Still happening.

Fresh install from the docker repo.

# uname -a
Linux sl6 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu Aug 13 12:55:33 CDT 2015 x86_64 x86_64 x86_64 GNU/Linux

# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-8:1-131846-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 305.7 MB
 Data Space Total: 107.4 GB
 Data Space Available: 38.78 GB
 Metadata Space Used: 729.1 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.95-RHEL6 (2015-04-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 2.6.32-573.3.1.el6.x86_64

# lsof -c docker | wc -l
21

# for i in $(seq 100); do docker run -t -i --rm busybox /bin/true ; done
Unable to find image 'busybox:latest' locally
latest: Pulling from busybox
cf2616975b4a: Pull complete 
6ce2e90b0bc7: Pull complete 
8c2e06607696: Already exists 
busybox:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:38a203e1986cf79639cfb9b2e1d6e773de84002feea2d4eb006b52004ee8502d
Status: Downloaded newer image for busybox:latest

# lsof -c docker | wc -l
34

# lsof -c docker | grep 'deleted'
docker  1203 root   13r   REG               0,21        0   37957 /cgroup/memory/docker/e8f3a7e2d4e43b48d46ce501500056ec97b8f45a37d3feb30c2fe0961c9f80fb/memory.oom_control (deleted)
docker  1203 root   17r   REG               0,21        0   37061 /cgroup/memory/docker/988f3e4baf253a3d298347d56501c4e097b5880b82753fa4f5cb0d82a233963d/memory.oom_control (deleted)
docker  1203 root   20r   REG               0,21        0   48456 /cgroup/memory/docker/5f7a5c381089b055f8d279ffc7b093e4bbb8c5c170191976d7c0cd52830e6d8b/memory.oom_control (deleted)
docker  1203 root   21r   REG               0,21        0   42769 /cgroup/memory/docker/65f7b36c8e9925f1754c969df6d639fbfbc246f60a510bcd94731ca2e8a8ddab/memory.oom_control (deleted)
docker  1203 root   24r   REG               0,21        0   38814 /cgroup/memory/docker/15d143b9645c0cac744dd68d6e353c02f1d463ecc8d7f978199e65885d50217d/memory.oom_control (deleted)

@jsipprell
Copy link

@csmithATsquiz

I believe you might be hitting a separate, but very potentially related, issue. There is a race condition in the kernel where a write to a memory cgroup's cgroup.event_control file that occurs while the cgroup is in the process of being deleted will cause the kernel dentry cache to repopulate. This then shows up a very short while later as an erroneously successful os.Lstat call. I was able to simulate this approximately 50% of the time on 2.6.32-573.3.1.el6 by reducing the initial pre-delete delay from one second to 25ms. e.g.

( usleep 25000; rmdir /cgroup/memory/foo )& ./cgroupoom foo

If this is what is happening to you, then a simple sleep 1 inside each busybox container run should fix your lingering open files.

To aid those wishing to reproduce the tests I have been running, I will post my test code here (again, it's the same algorithm found in 1.7's libcontainer wrapped in some debug output).

cgroupoom.go

@pmoust
Copy link
Contributor

pmoust commented Oct 15, 2015

Experiencing the same in CoreOS 766.4.0. Huge container log file remaining open after stoping a container started with --rm

ip-10-30-1-145 pmoust # uname -a
Linux ip-10-30-1-145.ec2.internal 4.1.7-coreos #2 SMP Wed Sep 16 22:54:37 UTC 2015 x86_64 Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz GenuineIntel GNU/Linux
ip-10-30-1-145 pmoust # docker info
Containers: 5
Images: 91
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.7-coreos
Operating System: CoreOS 766.4.0
CPUs: 2
Total Memory: 7.8 GiB
Name: ip-10-30-1-145.ec2.internal
ID: DOID:2HLN:6EWF:SPDE:G24T:YPWG:QLSH:5VFU:FP4X:QAFP:6VKF:IHGI
ip-10-30-1-145 pmoust # docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): df2f73d-dirty
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): df2f73d-dirty
OS/Arch (server): linux/amd64

@elfchief
Copy link

@pmoust -- different problem. The problem being discussed here applies specifically to docker keeping file handles open to the memory.oom_control cgroup control file. Anything else (docker logfiles, container logfiles, etc) is unrelated to this bug.

@pmoust
Copy link
Contributor

pmoust commented Oct 23, 2015

@elfchief not really, I experience the same behavior as the original issue reporter.
That is, container removed but lsof reporting open files (specifically container log files)

@sebdah
Copy link

sebdah commented Feb 25, 2016

We are getting hit by the same issue. The file locks are not released as expected when the logs are rotated. The locks are (naturally) released when restarting the docker service.

Sample output from lsof:

docker     1005 15874       root  127r      REG              202,1 50000036     410677 /var/lib/docker/containers/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a-json.log.3 (deleted)
docker     1005 15874       root  128r      REG              202,1 50000183     410637 /var/lib/docker/containers/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a-json.log.3 (deleted)
docker     1005 15874       root  129r      REG              202,1 50000082     410638 /var/lib/docker/containers/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a-json.log.3 (deleted)
docker     1005 15874       root  130r      REG              202,1 50000279     410658 /var/lib/docker/containers/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a-json.log.3 (deleted)
docker     1005 15874       root  131r      REG              202,1 50000393     410663 /var/lib/docker/containers/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a/6817a4ba0519875754479401b65f7b75127b97ac9b59937cc642863c8c6fc20a-json.log.3 (deleted)

Output from docker info:

Containers: 1
Images: 53
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-202:1-276622-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem:
 Data file: /dev/vg-docker/data
 Metadata file: /dev/vg-docker/metadata
 Data Space Used: 426 MB
 Data Space Total: 16.11 GB
 Data Space Available: 15.68 GB
 Metadata Space Used: 1.577 MB
 Metadata Space Total: 4.295 GB
 Metadata Space Available: 4.293 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-49-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 2
Total Memory: 3.676 GiB
Name: ip-10-120-0-9
ID: KYSA:QTRE:4HFS:FRR7:M2CZ:HZ2W:YITS:H3UP:AOS5:LMRX:OPMS:OZRX
WARNING: No swap limit support

@JohnMorales
Copy link

@sebdah (edit: I just noticed that you're on Ubuntu so disregard my notes below on rhel 6 :)

I don't think this issue is going to get any traction as there are other problems running docker on rhel 6, so the current docker docs recommends to run docker only on rhel 7. I've read somewhere if rhel 6 is needed it can be safely used it as the docker image base for the containers running on a rhel7 host.

That said my workaround cronjob has kept the machine from crashing and it's been up for months... even though the open files isn't a problem, I'm noticing inconsistent network stack issues in the containers that moving to rhel 7 is on the horizon for me.

Here's the gist of the cronjob that runs every night on my rhel6 host, not a solution, but a stop gap:

https://gist.github.com/JohnMorales/4a4277510190d187bac5

@sebdah
Copy link

sebdah commented Feb 25, 2016

@JohnMorales Thanks! We are on Ubuntu 14.04.02

@exowaucka
Copy link

Is this issue supposed to be resolved on RHEL 7? I'm running CentOS 7 with docker 1.9.1, and I'm having the problem reported by the original submitter (large number of deleted json.log files being held open).

$ sudo docker info
Containers: 123
Images: 1899
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker--vg-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 72.94 GB
 Data Space Total: 127.6 GB
 Data Space Available: 54.62 GB
 Metadata Space Used: 42.77 MB
 Metadata Space Total: 88.08 MB
 Metadata Space Available: 45.31 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2015-12-01)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-327.13.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 7.389 GiB
Name: REDACTED
ID: JSZR:26PG:HBTY:DJ7B:VUV3:IBBK:3ZI3:XJQE:FEHF:WIMC:ZV4U:BBRE
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

The xfs-formatted root disk is 30 GB. There are no other disks on the system.

@pashpashkin
Copy link

I have the same problem on Ubuntu 14.04.4.

$ uname -a
Linux Ubuntu-1404-trusty-64-minimal 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ sudo docker info
Containers: 13
 Running: 0
 Paused: 0
 Stopped: 13
Images: 17
Server Version: 1.11.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 208
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 4.2.0-27-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.797 GiB
Name: Ubuntu-1404-trusty-64-minimal
ID: LPHO:OLCZ:MR6J:IZJN:GJMJ:7HRB:LO4V:XR23:NRE6:6AHJ:EXPM:S2DO
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/

$ sudo lsof -p `pidof docker` | grep sock
docker  9731 root    9u  unix 0xffff8800bbb3d2c0      0t0   42211226 /var/run/docker.sock
docker  9731 root   10u  unix 0xffff8801b7b81680      0t0   42223315 /var/run/docker.sock
docker  9731 root   11u  unix 0xffff8801238570c0      0t0   45495496 /var/run/docker.sock
docker  9731 root   12u  unix 0xffff8801c4bd92c0      0t0   42578391 /var/run/docker.sock
docker  9731 root   13u  unix 0xffff8802338ef0c0      0t0   42578392 /var/run/docker.sock
docker  9731 root   14u  unix 0xffff88023529d680      0t0   48283437 /var/run/docker.sock
docker  9731 root   15u  unix 0xffff8801b78b3480      0t0   43881975 /var/run/docker.sock
docker  9731 root   16u  unix 0xffff8801178e1e00      0t0   43902442 /var/run/docker.sock
docker  9731 root   17u  unix 0xffff8801b78b12c0      0t0   43881976 /var/run/docker.sock
docker  9731 root   18u  unix 0xffff880073472580      0t0   44313665 /var/run/docker.sock
docker  9731 root   19u  unix 0xffff88023327e940      0t0   43902443 /var/run/docker.sock
docker  9731 root   20u  unix 0xffff8800bb3483c0      0t0   44337968 /var/run/docker.sock
docker  9731 root   21u  unix 0xffff88023335a580      0t0   48435649 /var/run/docker.sock
docker  9731 root   22u  unix 0xffff8801c4bd8780      0t0   42595059 /var/run/docker.sock
docker  9731 root   23u  unix 0xffff8801c4bda1c0      0t0   42606347 /var/run/docker.sock
docker  9731 root   24u  unix 0xffff8802338ee940      0t0   42609880 /var/run/docker.sock
docker  9731 root   25u  unix 0xffff880073473480      0t0   43792846 /var/run/docker.sock
docker  9731 root   26u  unix 0xffff8801c4bda580      0t0   43918516 /var/run/docker.sock
docker  9731 root   27u  unix 0xffff8801b7b81a40      0t0   42212892 /var/run/docker.sock
docker  9731 root   28u  unix 0xffff8801b7b803c0      0t0   42225767 /var/run/docker.sock
docker  9731 root   29u  unix 0xffff88023324e940      0t0   42178926 /var/run/docker.sock
...

$ sudo lsof -c docker | grep 'deleted'
docker-co 9742 root   11u  FIFO               0,18      0t0      16303 /run/containerd/48a85b96b05a9a60608dc6d5a7f7d2298c9641b2bd10fe18f7e6b596f4bdcd6d/init/control (deleted)
docker-co 9742 root   16u  FIFO               0,18      0t0      16372 /run/containerd/c6478ca12a87110b6216e44c28d8ed1ebad825e88cbb6c7ac9cb0deb7b990b57/init/control (deleted)
docker-co 9742 root   20u  FIFO               0,18      0t0      16451 /run/containerd/063a38e8055c8376e28da5fd69910573526d04f6eb135641dd44f50ca1d43c15/init/control (deleted)
docker-co 9742 root   21u  FIFO               0,18      0t0      16455 /run/containerd/b08f4faab33c513226900df521c11669993a79ad5b637d0cda92588d9006d81c/init/control (deleted)
docker-co 9742 root   22u  FIFO               0,18      0t0      16466 /run/containerd/e507b67161e900559f05650bd870bf555d6fc9d400a2bc2a9f9d316874028c5d/init/control (deleted)
docker-co 9742 root   23u  FIFO               0,18      0t0      16432 /run/containerd/49b504aa707d674497bc976c555de67e799e6a1137432a1ceaeeaf7b08b4836e/init/control (deleted)
docker-co 9742 root   24u  FIFO               0,18      0t0      16437 /run/containerd/f9dd9d978ab8f050b343af33ab25d171a50cd4e78867bc24a6aac919ba5cf074/init/control (deleted)
docker-co 9742 root   27u  FIFO               0,18      0t0      16464 /run/containerd/1e11c6dd928fe06e54ce9a1f8da02350f9f1204f299fd3769d3d996cd7baae26/init/control (deleted)
docker-co 9742 root   30u  FIFO               0,18      0t0      16449 /run/containerd/eb1cbfeb63db2f4090d986b6277f6a196d59420701ed16094c1f930317ae9602/init/control (deleted)
docker-co 9742 root   31u  FIFO               0,18      0t0      16446 /run/containerd/fce75b25583f4315f13d7ec0b9875d365d1c95a81e3eb94d7191d895f8ab7f20/init/control (deleted)

@Majkl578
Copy link

Same issue here with 1.11.2. This was causing our long-running service to fail periodically, because it had no free file descriptors left...
This bug is almost 2 years open, sad. 😢

@thaJeztah
Copy link
Member

@Majkl578 what storage driver are you using? If you're using overlay, that may be a different issue

@cpuguy83
Copy link
Member

Let's close this, because it's likely been different issues over time, I do not see a reason to keep it open. @thaJeztah WDYT?

@thaJeztah
Copy link
Member

@cpuguy83 agreed.

For those following this; we're closing this issue because it's been collecting various unrelated issues; if you're still having this issue, please open a new issue, and detailed information about your issue.

@manuwela manuwela mentioned this issue Oct 4, 2016
Closed
@suside suside mentioned this issue Oct 13, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. version/1.1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests