Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker out of file descriptors/processes #19124

Closed
njdart opened this issue Jan 6, 2016 · 6 comments
Closed

Docker out of file descriptors/processes #19124

njdart opened this issue Jan 6, 2016 · 6 comments

Comments

@njdart
Copy link

njdart commented Jan 6, 2016

Docker (or postgresql, or i'm not really sure...) seems to be running out of resources,

psql: could not fork new process for connection: Resource temporarily unavailable
bash: fork: retry: No child processes

docker exec -it <id> bash is also track tracing (on any container)

[nic@ARCHY(pts/9)] ~/Downloads/docker/usr/local/bin % docker exec -it 8d9fcbb8a591 bash
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f18b83365f8 m=0

goroutine 0 [idle]:

goroutine 1 [running, locked to thread]:
runtime.systemstack_switch()
  /usr/lib/go/src/runtime/asm_amd64.s:216 fp=0xc82004be90 sp=0xc82004be88
runtime.newproc(0x0, 0x1711a00)
  /usr/lib/go/src/runtime/proc1.go:2213 +0x62 fp=0xc82004bed8 sp=0xc82004be90
runtime.init.4()
  /usr/lib/go/src/runtime/proc.go:141 +0x2b fp=0xc82004bef0 sp=0xc82004bed8
runtime.init()
  /usr/lib/go/src/runtime/zversion.go:9 +0x378 fp=0xc82004bf50 sp=0xc82004bef0
runtime.main()
  /usr/lib/go/src/runtime/proc.go:63 +0x103 fp=0xc82004bfa0 sp=0xc82004bf50
runtime.goexit()
  /usr/lib/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82004bfa8 sp=0xc82004bfa0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
  /usr/lib/go/src/runtime/asm_amd64.s:1696 +0x1

These errors happen both within the container and in the host, dispute seemingly plenty of memory to spare...

At first i thought this was an issue with postgresql, however frequently docker exec -it <id> bash will simply return (without an error as far as i know) and docker inspect <id> will return 404's, and then all my containers seem to be killed

I'm not too familiar with docker, but the limits don't seem to match up with what I would expect (the container having lower limits than the host)...

container:

[root@20ec09af20ab ansible]# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63712
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1048576
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

host:

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-m: resident set size (kbytes)      unlimited
-u: processes                       63712
-n: file descriptors                1024
-l: locked-in-memory size (kbytes)  unlimited
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 63712
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 99
-N 15:                              unlimited

docker info:

Containers: 5
Images: 80
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-8:2-1443374-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: 
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 9.005 GB
 Data Space Total: 107.4 GB
 Data Space Available: 16.17 GB
 Metadata Space Used: 8.634 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.139 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.113 (2015-12-05)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.3.3-2-ARCH
Operating System: Arch Linux (containerized)
CPUs: 8
Total Memory: 15.56 GiB
Name: ARCHY
ID: YWOL:R6JL:TTCO:W4HO:WLCT:76KR:NDPD:6OPP:52EE:VRBP:ZMG3:OJD6

docker version:

Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.5.1
 Git commit:   a34a1d5-dirty
 Built:        Sun Nov 22 00:15:15 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.5.1
 Git commit:   a34a1d5-dirty
 Built:        Sun Nov 22 00:15:15 UTC 2015
 OS/Arch:      linux/amd64

docker inspect on the postgresql container

[
{
    "Id": "20ec09af20ab99363ae0788209780813f584346182bfa776f5c50b83e9a5aa39",
    "Created": "2016-01-06T13:28:50.413698402Z",
    "Path": "/postgres-entrypoint.sh",
    "Args": [
        "postgres"
    ],
    "State": {
        "Status": "running",
        "Running": true,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 26820,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "2016-01-06T13:28:50.761304114Z",
        "FinishedAt": "0001-01-01T00:00:00Z"
    },
    "Image": "70d6748b6e1b582c1533724a83454c0395d33729206676474d5ee8823db7f241",
    "ResolvConfPath": "/var/lib/docker/containers/20ec09af20ab99363ae0788209780813f584346182bfa776f5c50b83e9a5aa39/resolv.conf",
    "HostnamePath": "/var/lib/docker/containers/20ec09af20ab99363ae0788209780813f584346182bfa776f5c50b83e9a5aa39/hostname",
    "HostsPath": "/var/lib/docker/containers/20ec09af20ab99363ae0788209780813f584346182bfa776f5c50b83e9a5aa39/hosts",
    "LogPath": "/var/lib/docker/containers/20ec09af20ab99363ae0788209780813f584346182bfa776f5c50b83e9a5aa39/20ec09af20ab99363ae0788209780813f584346182bfa776f5c50b83e9a5aa39-json.log",
    "Name": "/db_postgres",
    "RestartCount": 0,
    "Driver": "devicemapper",
    "ExecDriver": "native-0.2",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "ExecIDs": [
        "c162328eb19f7e5e989bb1674034023fc0293ca393e76c2373f3325c41357c73"
    ],
    "HostConfig": {
        "Binds": null,
        "ContainerIDFile": "",
        "LxcConf": [],
        "Memory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "KernelMemory": 0,
        "CpuShares": 0,
        "CpuPeriod": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "CpuQuota": 0,
        "BlkioWeight": 0,
        "OomKillDisable": false,
        "MemorySwappiness": -1,
        "Privileged": false,
        "PortBindings": {
            "5432/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "5432"
                }
            ]
        },
        "Links": null,
        "PublishAllPorts": false,
        "Dns": [],
        "DnsOptions": [],
        "DnsSearch": [],
        "ExtraHosts": null,
        "VolumesFrom": null,
        "Devices": [],
        "NetworkMode": "default",
        "IpcMode": "",
        "PidMode": "",
        "UTSMode": "",
        "CapAdd": null,
        "CapDrop": null,
        "GroupAdd": null,
        "RestartPolicy": {
            "Name": "no",
            "MaximumRetryCount": 0
        },
        "SecurityOpt": null,
        "ReadonlyRootfs": false,
        "Ulimits": null,
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "CgroupParent": "",
        "ConsoleSize": [
            0,
            0
        ],
        "VolumeDriver": ""
    },
    "GraphDriver": {
        "Name": "devicemapper",
        "Data": {
            "DeviceId": "1884",
            "DeviceName": "docker-8:2-1443374-20ec09af20ab99363ae0788209780813f584346182bfa776f5c50b83e9a5aa39",
            "DeviceSize": "107374182400"
        }
    },
    "Mounts": [
        {
            "Name": "0e981730b580637439459e4c3635b75f17e90948868ff136ef2dba3edcfd06df",
            "Source": "/var/lib/docker/volumes/0e981730b580637439459e4c3635b75f17e90948868ff136ef2dba3edcfd06df/_data",
            "Destination": "/var/lib/pgsql/data",
            "Driver": "local",
            "Mode": "",
            "RW": true
        }
    ],
    "Config": {
        "Hostname": "20ec09af20ab",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "ExposedPorts": {
            "5432/tcp": {}
        },
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/tmp/ansible/bin:/sbin:/usr/sbin:/usr/bin",
            "ANSIBLE_LIBRARY=/tmp/ansible/library",
            "PYTHONPATH=/tmp/ansible/lib:",
            "BASE_VERSION=0.1",
            "LC_ALL=en_GB.UTF-8",
            "FLO_CONFIG=singleflo"
        ],
        "Cmd": [
            "postgres"
        ],
        "Image": "cradle/postgres_singleflo",
        "Volumes": {
            "/var/lib/pgsql/data": {}
        },
        "WorkingDir": "/tmp/ansible",
        "Entrypoint": [
            "/postgres-entrypoint.sh"
        ],
        "OnBuild": null,
        "Labels": {
            "license": "GPLv2",
            "name": "CentOS Base Image",
            "vendor": "CentOS"
        },
        "StopSignal": "SIGTERM"
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "8e156fae10c92667a88db43bc921e5856a63307413a7c7817bcf737f763b75e7",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {
            "5432/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "5432"
                }
            ]
        },
        "SandboxKey": "/var/run/docker/netns/8e156fae10c9",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "525c4bbae6af11b04fcfdb731d83ff9fa00af803d7c0067e527ea403fc2e7fda",
        "Gateway": "172.17.0.1",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "172.17.0.2",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "MacAddress": "02:42:ac:11:00:02",
        "Networks": {
            "bridge": {
                "EndpointID": "525c4bbae6af11b04fcfdb731d83ff9fa00af803d7c0067e527ea403fc2e7fda",
                "Gateway": "172.17.0.1",
                "IPAddress": "172.17.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:42:ac:11:00:02"
            }
        }
    }
}
]

I'm running Arch Linux Linux ARCHY 4.3.3-2-ARCH #1 SMP PREEMPT Wed Dec 23 20:09:18 CET 2015 x86_64 GNU/Linux and pre-4.3.3 kernel update things were working fine in docker. I've rebuild all the containers since then. These were the packaged updated between it working, and now

[2016-01-04 10:18] [ALPM] upgraded archlinux-keyring (20151206-1 -> 20151220-1)
[2016-01-04 10:18] [ALPM] upgraded gcc-libs (5.3.0-2 -> 5.3.0-3)
[2016-01-04 10:18] [ALPM] upgraded boost-libs (1.59.0-5 -> 1.60.0-1)
[2016-01-04 10:18] [ALPM] upgraded perl (5.22.0-1 -> 5.22.1-1)
[2016-01-04 10:18] [ALPM] upgraded p11-kit (0.23.1-3 -> 0.23.2-1)
[2016-01-04 10:18] [ALPM] upgraded gmp (6.1.0-2 -> 6.1.0-3)
[2016-01-04 10:18] [ALPM] upgraded ca-certificates-mozilla (3.20.1-1 -> 3.21-1)
[2016-01-04 10:18] [ALPM] upgraded libpng (1.6.19-1 -> 1.6.20-1)
[2016-01-04 10:18] [ALPM] upgraded cantarell-fonts (0.0.18.1-1 -> 0.0.19-1)
[2016-01-04 10:18] [ALPM] upgraded libdrm (2.4.65-1 -> 2.4.66-1)
[2016-01-04 10:18] [ALPM] upgraded libdbus (1.10.4-1 -> 1.10.6-1)
[2016-01-04 10:18] [ALPM] upgraded dbus (1.10.4-1 -> 1.10.6-1)
[2016-01-04 10:18] [ALPM] upgraded libldap (2.4.42-2 -> 2.4.43-1)
[2016-01-04 10:18] [ALPM] upgraded mesa (11.0.7-1 -> 11.1.0-2)
[2016-01-04 10:18] [ALPM] upgraded mesa-libgl (11.0.7-1 -> 11.1.0-2)
[2016-01-04 10:18] [ALPM] upgraded poppler (0.38.0-3 -> 0.39.0-1)
[2016-01-04 10:18] [ALPM] upgraded ocl-icd (2.2.7-1 -> 2.2.8-1)
[2016-01-04 10:18] [ALPM] upgraded imagemagick (6.9.2.6-3 -> 6.9.2.10-1)
[2016-01-04 10:18] [ALPM] upgraded liblouis (2.6.3-2 -> 2.6.5-1)
[2016-01-04 10:18] [ALPM] upgraded cups-filters (1.4.0-1 -> 1.5.0-2)
[2016-01-04 10:18] [ALPM] upgraded docbook-xsl (1.78.1-1 -> 1.79.1-1)
[2016-01-04 10:18] [ALPM] upgraded erlang-nox (18.1-1 -> 18.2.1-1)
[2016-01-04 10:18] [ALPM] upgraded libva (1.6.1-1 -> 1.6.2-1)
[2016-01-04 10:18] [ALPM] upgraded libwebp (0.4.4-1 -> 0.5.0-1)
[2016-01-04 10:18] [ALPM] upgraded libxi (1.7.5-1 -> 1.7.6-1)
[2016-01-04 10:18] [ALPM] upgraded ffmpeg (1:2.8.3-1 -> 1:2.8.4-2)
[2016-01-04 10:18] [ALPM] upgraded nspr (4.10.10-1 -> 4.11-1)
[2016-01-04 10:18] [ALPM] upgraded nss (3.20.1-1 -> 3.21-1)
[2016-01-04 10:18] [ALPM] upgraded libgusb (0.2.7-1 -> 0.2.8-1)
[2016-01-04 10:18] [ALPM] upgraded firefox (43.0-2 -> 43.0.1-2)
[2016-01-04 10:18] [ALPM] upgraded mpfr (3.1.3.p4-1 -> 3.1.3.p5-1)
[2016-01-04 10:18] [ALPM] upgraded gcc (5.3.0-2 -> 5.3.0-3)
[2016-01-04 10:18] [ALPM] upgraded gegl (0.3.4-2 -> 0.3.4-3)
[2016-01-04 10:18] [ALPM] upgraded gnupg (2.1.10-1 -> 2.1.10-3)
[2016-01-04 10:18] [ALPM] upgraded grub (1:2.02.beta2-5 -> 1:2.02.beta2-6)
[2016-01-04 10:18] [ALPM] upgraded hspell (1.2-2 -> 1.3-1)
[2016-01-04 10:18] [ALPM] upgraded pciutils (3.3.1-1 -> 3.4.0-1)
[2016-01-04 10:18] [ALPM] upgraded hwloc (1.11.1-1 -> 1.11.2-2)
[2016-01-04 10:18] [ALPM] upgraded poppler-glib (0.38.0-3 -> 0.39.0-1)
[2016-01-04 10:18] [ALPM] upgraded libsigc++ (2.6.1-2 -> 2.6.2-1)
[2016-01-04 10:18] [ALPM] upgraded inkscape (0.91-14 -> 0.91-15)
[2016-01-04 10:18] [ALPM] upgraded lib32-gcc-libs (5.3.0-2 -> 5.3.0-3)
[2016-01-04 10:18] [ALPM] upgraded lib32-libdbus (1.10.4-1 -> 1.10.6-1)
[2016-01-04 10:18] [ALPM] upgraded lib32-libdrm (2.4.65-1 -> 2.4.66-1)
[2016-01-04 10:18] [ALPM] upgraded lib32-libpng (1.6.19-1 -> 1.6.20-1)
[2016-01-04 10:18] [ALPM] upgraded lib32-libxi (1.7.5-1 -> 1.7.6-1)
[2016-01-04 10:18] [ALPM] upgraded lib32-mesa (11.0.7-1 -> 11.1.0-1)
[2016-01-04 10:18] [ALPM] upgraded librevenge (0.0.2-2 -> 0.0.3-1)
[2016-01-04 10:18] [ALPM] upgraded libwpd (0.10.0-2 -> 0.10.1-1)
[2016-01-04 10:18] [ALPM] upgraded libcdr (0.1.1-4 -> 0.1.2-1)
[2016-01-04 10:18] [ALPM] upgraded libinput (1.1.3-1 -> 1.1.4-1)
[2016-01-04 10:18] [ALPM] upgraded libmariadbclient (10.1.9-4 -> 10.1.10-1)
[2016-01-04 10:18] [ALPM] upgraded libnautilus-extension (3.18.3-1 -> 3.18.4-1)
[2016-01-04 10:18] [ALPM] upgraded libnotify (0.7.6-1 -> 0.7.6-2)
[2016-01-04 10:18] [ALPM] upgraded libodfgen (0.1.4-2 -> 0.1.6-1)
[2016-01-04 10:18] [ALPM] upgraded libwpg (0.3.0-2 -> 0.3.1-1)
[2016-01-04 10:18] [ALPM] upgraded libvisio (0.1.3-3 -> 0.1.5-1)
[2016-01-04 10:18] [ALPM] upgraded libreoffice-fresh (5.0.4-1 -> 5.0.4-3)
[2016-01-04 10:18] [ALPM] upgraded libsodium (1.0.7-1 -> 1.0.8-1)
[2016-01-04 10:18] [ALPM] upgraded libva-intel-driver (1.6.1-1 -> 1.6.2-1)
[2016-01-04 10:18] [ALPM] upgraded lighttpd (1.4.38-1 -> 1.4.39-1)
[2016-01-04 10:18] [ALPM] upgraded linux-firmware (20150904.6ebf5d5-1 -> 20151207.bbe4917-1)
[2016-01-04 10:18] [ALPM] upgraded linux (4.2.5-1 -> 4.3.3-2)
[2016-01-04 10:18] [ALPM] upgraded mariadb-clients (10.1.9-4 -> 10.1.10-1)
[2016-01-04 10:18] [ALPM] upgraded mariadb (10.1.9-4 -> 10.1.10-1)
[2016-01-04 10:18] [ALPM] upgraded qt4 (4.8.7-5 -> 4.8.7-6)
[2016-01-04 10:18] [ALPM] upgraded mumble (1.2.11-2 -> 1.2.12-1)
[2016-01-04 10:18] [ALPM] upgraded nautilus (3.18.3-1 -> 3.18.4-1)
[2016-01-04 10:18] [ALPM] upgraded ncmpcpp (0.7-1 -> 0.7-2)
[2016-01-04 10:18] [ALPM] upgraded openvpn (2.3.8-2 -> 2.3.9-1)
[2016-01-04 10:18] [ALPM] upgraded pavucontrol (3.0-2 -> 3.0-3)
[2016-01-04 10:18] [ALPM] upgraded perl-libwww (6.08-1 -> 6.15-1)
[2016-01-04 10:18] [ALPM] upgraded php (5.6.16-3 -> 7.0.1-5)
[2016-01-04 10:18] [ALPM] upgraded php-cgi (5.6.16-3 -> 7.0.1-5)
[2016-01-04 10:18] [ALPM] upgraded python-setuptools (1:19.1.1-1 -> 1:19.2-1)
[2016-01-04 10:18] [ALPM] upgraded python2-cffi (1.4.1-1 -> 1.4.2-1)
[2016-01-04 10:18] [ALPM] upgraded python2-enum34 (1.0.4-1 -> 1.1.1-1)
[2016-01-04 10:18] [ALPM] upgraded python2-ipaddress (1.0.15-1 -> 1.0.16-1)
[2016-01-04 10:18] [ALPM] upgraded python2-urllib3 (1.13-1 -> 1.14-1)
[2016-01-04 10:18] [ALPM] upgraded python2-requests (2.9.0-1 -> 2.9.1-1)
[2016-01-04 10:18] [ALPM] upgraded python2-setuptools (1:19.1.1-1 -> 1:19.2-1)
[2016-01-04 10:18] [ALPM] upgraded qt5-base (5.5.1-6 -> 5.5.1-7)
[2016-01-04 10:18] [ALPM] upgraded qt5-xmlpatterns (5.5.1-6 -> 5.5.1-7)

Thanks,
Nic
@GordonTheTurtle
Copy link

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

For more information about reporting issues, see CONTRIBUTING.md.

You don't have to include this information if this is a feature request

(This is an automated, informational response)

BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

@cpuguy83
Copy link
Member

cpuguy83 commented Jan 6, 2016

ulimits in your shell are not the same as ulimits set by the init system.
Check your init script, check lsof.
By default containers will inherit ulimits from the daemon process(which you can also override at both the container level and the daemon level).

@BurntSushi
Copy link

I was running into this problem recently as well. Whether it's related or not, this bug report's workaround worked for me: https://bugs.archlinux.org/task/47303 That is, add TasksMax=infinity to your docker.service file.

@njdart
Copy link
Author

njdart commented Jan 7, 2016

@BurntSushi That fixed the problem, thanks. This looks like it's a problem (or change?) with Linux containers, so probably of no further relevance here.

@njdart njdart closed this as completed Jan 7, 2016
@joeferner
Copy link

Thanks @BurntSushi wow that was hard to trackdown. Well done.

@micahthomas micahthomas mentioned this issue Jan 15, 2016
Closed
@cdauth cdauth mentioned this issue Jan 16, 2016
Merged
@thaJeztah
Copy link
Member

Note that we're updating the docker.service for docker 1.10 to override the default value; #19391

@icetan icetan mentioned this issue Feb 29, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@BurntSushi @cpuguy83 @joeferner @njdart @thaJeztah @GordonTheTurtle