daemon crash

[runcom]

been spawning 10k containers in a bash loop (command was just top), at around 2.5k I got:

2015/12/18 10:39:39 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport: write unix /run/containerd/containerd.sock->@: write: bad file descriptor"

containerd crashed, containers list is empty after the crash, don't know if it's related to the code or to my system

[crosbymichael]

Could be something weird happening on the system side. Most of my tests and bench marks are running 1-2k containers. I'm not sure how to debug this because that is a weird error, did that come from the daemon or client?

[runcom]

@crosbymichael I've seen that log appearing in both daemon and client :/

the loop was just:

$ cat bench.sh 

#!/bin/bash

for i in {1..10000}
do
    ./bin/ctr containers start $i $(pwd)
done

[LK4D4]

I think it's grpc error, probably due to ulimit hit, but daemon shouldn't crash of course.

[runcom]

testing again

[runcom]

@LK4D4 daemon crashed:

[containerd] 2015/12/18 19:34:46   15-min rate:         1.65
[containerd] 2015/12/18 19:34:46   mean rate:           6.97
ERRO[0253] get process pid                               error=[6] Unknown error: invalid process
ERRO[0257] get process pid                               error=[6] Unknown error: invalid process
ERRO[0259] get process pid                               error=[6] Unknown error: invalid process
ERRO[0271] get process pid                               error=[6] Unknown error: invalid process
[containerd] 2015/12/18 19:35:46 counter containers
[containerd] 2015/12/18 19:35:46   count:            1959
[containerd] 2015/12/18 19:35:46 counter events
[containerd] 2015/12/18 19:35:46   count:            2424
[containerd] 2015/12/18 19:35:46 counter events-subscribers
[containerd] 2015/12/18 19:35:46   count:            2419
[containerd] 2015/12/18 19:35:46 timer container-start-time
[containerd] 2015/12/18 19:35:46   count:            1959
[containerd] 2015/12/18 19:35:46   min:          42991034.00ns
[containerd] 2015/12/18 19:35:46   max:         457644847.00ns
[containerd] 2015/12/18 19:35:46   mean:        138091592.04ns
[containerd] 2015/12/18 19:35:46   stddev:       61120164.91ns
[containerd] 2015/12/18 19:35:46   median:      126740613.00ns
[containerd] 2015/12/18 19:35:46   75%:         168180653.25ns
[containerd] 2015/12/18 19:35:46   95%:         253614820.75ns
[containerd] 2015/12/18 19:35:46   99%:         339428078.60ns
[containerd] 2015/12/18 19:35:46   99.9%:       457092969.49ns
[containerd] 2015/12/18 19:35:46   1-min rate:          5.48
[containerd] 2015/12/18 19:35:46   5-min rate:          4.11
[containerd] 2015/12/18 19:35:46   15-min rate:         1.85
[containerd] 2015/12/18 19:35:46   mean rate:           6.53
[containerd] 2015/12/18 19:35:46 gauge goroutines
[containerd] 2015/12/18 19:35:46   value:            6828
[containerd] 2015/12/18 19:35:46 gauge fds
[containerd] 2015/12/18 19:35:46   value:            3677
ERRO[0304] get process pid                               error=[6] Unknown error: invalid process
2015/12/18 19:36:05 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport: write unix /run/containerd/containerd.sock->@: write: bad file descriptor"
2015/12/18 19:36:05 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport: write unix /run/containerd/containerd.sock->@: write: bad file descriptor"
2015/12/18 19:36:05 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport: write unix /run/containerd/containerd.sock->@: write: bad file descriptor"
2015/12/18 19:36:05 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport: write unix /run/containerd/containerd.sock->@: use of closed network connection"
2015/12/18 19:36:05 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport: write unix /run/containerd/containerd.sock->@: write: bad file descriptor"
2015/12/18 19:36:05 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport: write unix /run/containerd/containerd.sock->@: write: bad file descriptor"
2015/12/18 19:36:05 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport: write unix /run/containerd/containerd.sock->@: write: bad file descriptor"
2015/12/18 19:36:05 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport: write unix /run/containerd/containerd.sock->@: write: bad file descriptor"
2015/12/18 19:36:05 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport: write unix /run/containerd/containerd.sock->@: write: bad file descriptor"
2015/12/18 19:36:05 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport: write unix /run/containerd/containerd.sock->@: write: bad file descriptor"
FATA[0318] accept unix /run/containerd/containerd.sock: bad file descriptor

and this is the cause in client I think:

[ctr] rpc error: code = 2 desc = "[9] System error: fork/exec /proc/self/exe: too many open files"

[LK4D4]

I'm pretty sure that it's error from l.Accept on unix-socket. So, it's Go error as I understand. Probably you want to up ulimit or write your client(it's super easy with grpc and should be the primary use case of containerd) which will reuse a connection.

[crosbymichael]

Maybe left open fds in the client because he is running it in a loop