Too many open files error

[dpordomingo]

running a hand-builded version of gitbase from 86955f1 (current master) it crashes when quering over a ~7000 repos backed in siva files

$ mysql --port=3306 --user=root --host=127.0.0.1 --execute "SELECT count(*) from repositories"
ERROR 1105 (HY000) at line 1: unknown error: open ~/repos/35/351674ec3a175bb56438972325746e4ce8e2e7d8.siva: too many open files

it causes gitbase to crash:

$ ./gitbase server --port=3306 --directories ~/repos --index=/tmp/gitbase
INFO[0000] squash tables rule is enabled                
INFO[0000] server started and listening on localhost:3306 
INFO[0004] NewConnection: client 1                      
ERRO[0004] Error parsing auth server config: unexpected end of JSON input 
INFO[0004] audit trail                                   action=authentication address="127.0.0.1:60932" success=true system=audit user=root
INFO[0004] audit trail                                   action=authorization address="127.0.0.1:60932" connection_id=1 permission=read pid=1 query="select @@version_comment limit 1" success=true system=audit user=root
INFO[0004] audit trail                                   action=query address="127.0.0.1:60932" connection_id=1 duration=1.75461ms pid=1 query="select @@version_comment limit 1" success=true system=audit user=root
INFO[0004] audit trail                                   action=authorization address="127.0.0.1:60932" connection_id=1 permission=read pid=2 query="SELECT count(*) from repositories" success=true system=audit user=root
INFO[0005] audit trail                                   action=query address="127.0.0.1:60932" connection_id=1 duration=323.986104ms err="open ~/repos/35/351674ec3a175bb56438972325746e4ce8e2e7d8.siva: too many open files" pid=2 query="SELECT count(*) from repositories" success=false system=audit user=root
INFO[0005] ConnectionClosed: client 1 

then if I run other queries, gitbase is no longer available:

$ mysql --port=3306 --user=root --host=127.0.0.1 --execute "SELECT count(*) from remotes"
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0

If I run first this second query, it success, but whenever I try with the first one, it kills gitbase.

[kuba--]

First of all I would check if there is any limit for number of opened file descriptors, e.g.:

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1418
virtual memory          (kbytes, -v) unlimited

The second thing is to check if gitbases closes repo files.

[kuba--]

The problem here is that gitbase doesn't close opened siva files:

gitbase 7046 kuba  cwd      DIR                1,4     2496  4561697 github.com/src-d/gitbase
gitbase 7046 kuba  txt      REG                1,4 48839612 24845651 github.com/src-d/gitbase/gitbase
gitbase 7046 kuba  txt      REG                1,4   522700 18763162 /usr/local/Cellar/oniguruma/6.9.1/lib/libonig.5.dylib
gitbase 7046 kuba  txt      REG                1,4    21024 24799632 /Library/Preferences/Logging/.plist-cache.eCTh4K6b
gitbase 7046 kuba  txt      REG                1,4   973824 24486327 /usr/lib/dyld
gitbase 7046 kuba    0u     CHR               16,0 0t550536      641 /dev/ttys000
gitbase 7046 kuba    1u     CHR               16,0 0t550536      641 /dev/ttys000
gitbase 7046 kuba    2u     CHR               16,0 0t550536      641 /dev/ttys000
gitbase 7046 kuba    3u   systm 0xeb7a8a1932474011      0t0          [ctl com.apple.netsrc id 8 unit 27]
gitbase 7046 kuba    4u  KQUEUE                                      count=0, state=0xa
gitbase 7046 kuba    5u    unix 0xeb7a8a1939924179      0t0          ->0xeb7a8a19399235c1
gitbase 7046 kuba    6u    IPv4 0xeb7a8a19519f8191      0t0      TCP localhost:mysql (LISTEN)
gitbase 7046 kuba    7u    IPv4 0xeb7a8a193c416811      0t0      TCP localhost:mysql->localhost:51011 (ESTABLISHED)
gitbase 7046 kuba    8r     CHR               14,1   0t4096      587 /dev/urandom
gitbase 7046 kuba    9u    IPv4 0xeb7a8a193ea68191      0t0      TCP localhost:mysql->localhost:51012 (ESTABLISHED)
gitbase 7046 kuba   10u     REG                1,4  1159809 24803492 /private/tmp/repos/siva/latest/00/00e71912d4ec215337d5d959a34ab0ce44fdd3b4.siva
gitbase 7046 kuba   11u     REG                1,4   171693 24803496 /private/tmp/repos/siva/latest/28/284bc24eaafca75786c7e86f1da66ad61adacdb7.siva
gitbase 7046 kuba   12u     REG                1,4   221250 24803515 /private/tmp/repos/siva/latest/36/368ffcdb3c45e9e6538ef60c1d15aed4555c04b2.siva
gitbase 7046 kuba   13u     REG                1,4    70503 24803500 /private/tmp/repos/siva/latest/4a/4a14cc02da0a9280538cd3f3242365601d72f241.siva
gitbase 7046 kuba   14u     REG                1,4    33232 24803494 /private/tmp/repos/siva/latest/53/531c8effde374c3178ca7f04b0e36993f3ebe02a.siva
gitbase 7046 kuba   15u     REG                1,4   966632 24803513 /private/tmp/repos/siva/latest/63/63d374207b3322bcb3399f7c3a342b32ef306892.siva
gitbase 7046 kuba   16u     REG                1,4 16409544 24803510 /private/tmp/repos/siva/latest/7c/7cf89280a2f047913d9a5afbb58d8db466b93ace.siva
gitbase 7046 kuba   17u     REG                1,4  1181360 24803509 /private/tmp/repos/siva/latest/7f/7f1bc3e622dcaac4c05635f859862866c31e69ef.siva
gitbase 7046 kuba   18u     REG                1,4  2391933 24803498 /private/tmp/repos/siva/latest/db/db600b30cdd2398d577eb827ce4fd3c1efa09bc1.siva
gitbase 7046 kuba   19u     REG                1,4   803341 24803511 /private/tmp/repos/siva/latest/ec/ecbe802ef1fc35f8541a1dd1765fece5ddd6a415.siva

[dpordomingo]

Just in case it helps you to debug:
I didn't find that problem with current latest v0.19.0 (26 behind current master)

Let me know if I can help you with any other info.

[kuba--]

It was just coincidence, Every query select count(*) from repositories opens a new files up to some limit and some files will be closed. The same behaviour I also noticed in v0.19.0