socket: too many open files

[igagis]

Please check this build:
https://github.com/cppfw/myci/runs/3187046714?check_suite_focus=true#step:1:10

it has failed with some FIXME error

[igagis]

logs don't seem to show anything interesting...

Jul 29 02:44:43 schultz runner[4279]: Running Job 'linux - i386/debian:buster'
Jul 29 02:44:43 schultz runner[4279]: Failed to upload timeline with Status 404
Jul 29 02:44:44 schultz runner[4279]: Finished Job 'linux - i386/debian:buster' with result: Failed

[igagis]

restarting the runner fixes the problem

[ChristopherHX]

On my machine it looks like that the unix docker socket is never released
Open file decriptors

ubuntu@ubuntu:~$ ls -l /proc/2796193/fd
total 0
lr-x------ 1 ubuntu ubuntu 64 Jul 29 09:13 0 -> /dev/null
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 1 -> 'socket:[35451282]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 10 -> 'socket:[35449606]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 11 -> 'socket:[35450636]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 12 -> 'socket:[35450637]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 13 -> 'socket:[35451572]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 14 -> 'socket:[35452257]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 15 -> 'socket:[35449842]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 16 -> 'socket:[35451597]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 17 -> 'socket:[36204890]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 18 -> 'socket:[36204026]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 19 -> 'socket:[36204891]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 2 -> 'socket:[35451282]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 20 -> 'socket:[36205857]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 21 -> 'socket:[36205858]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 3 -> 'socket:[35449748]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 4 -> 'anon_inode:[eventpoll]'
lr-x------ 1 ubuntu ubuntu 64 Jul 29 09:13 5 -> 'pipe:[35451284]'
l-wx------ 1 ubuntu ubuntu 64 Jul 29 09:13 6 -> 'pipe:[35451284]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 7 -> 'socket:[40475239]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 8 -> 'socket:[35451354]'
lrwx------ 1 ubuntu ubuntu 64 Jul 29 09:13 9 -> 'socket:[35450610]'

Netstat Connections:

ubuntu@ubuntu:~$ netstat -a -p | grep 2796193
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 ubuntu.fritz.box:46142  13.107.42.16:https      ESTABLISHED 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     36205857 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     36205858 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     35452257 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     35451597 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     36204890 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     36204891 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     35451354 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     35450637 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     35451282 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     36204026 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     35450636 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     35450610 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     35449606 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     35449842 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     35451572 2796193/github-acti
unix  3      [ ]         STREAM     CONNECTED     35449748 2796193/github-acti

On each docker run, it opens a new unix socket until dial unix /var/run/docker.sock: socket: too many open files.
Maybe this is a bug of act or how I used act in the worker. We need to enshure to close the docker connection on worker completion.
It would be interesting if this is the only file descriptor leak.

[igagis]

So far this happened only once to one of my runners, and it was amd64 architecture runner on ubuntu 20.04.
I don't know what caused the runner to enter that state, I didn't check open descriptors. I'll let you know if I observe this again.

[ChristopherHX]

So far this happened only once

It takes some time to eat all filedescriptors one per executed container job and docker action. The limit is about 1024 open filedescriptors per application.
You installed updates on your other runners so this limit wasn't hit. (if you restart your runner, the OS frees them for you)

I don't know how many jobs your runner has done before this error, but I will fix one file descriptor leak in the next update.

[igagis]

I don't know how many jobs your runner has done before this error

well, for sure, less than 1024, so perhaps there are more leaks somewhere...

[ChristopherHX]

so perhaps there are more leaks somewhere...

The docker leaks are insane 3 (single step) container jobs are leaking more than 10 file descriptors (reduced the limit to 16 sudo prlimit --pid <pid> --nofile=16:16)

Host runner zero leaks, running 20 jobs without leaving any leaked file descriptors.

I need to review any code interacting with the docker interface inside of nectos/act 🤦‍♂️

Found the leak
No code of act cared about closing a single docker client connection