Filer: Too many open files after update to 2.38

[divanikus]

Describe the bug
After upgrading from 2.35 to 2.38 we experience full service degradation after several hours of work. I see lots of ESTABLISHED connections to filer and "too many open files" errors in filer's log.

tcp6   64937      0 192.168.65.42:8888      192.168.65.42:44366     ESTABLISHED 9150/weed
tcp6   45708      0 192.168.65.42:8888      192.168.65.42:41398     ESTABLISHED 9150/weed
tcp6   95248      0 192.168.65.42:8888      192.168.65.42:38762     ESTABLISHED 9150/weed
tcp6   50516      0 192.168.65.42:8888      192.168.65.42:36320     ESTABLISHED 9150/weed
tcp6    9399      0 192.168.65.42:8888      192.168.65.42:56408     ESTABLISHED 9150/weed
tcp6   63103      0 192.168.65.42:8888      192.168.65.42:38124     ESTABLISHED 9150/weed
tcp6    1927      0 192.168.65.42:8888      192.168.65.42:37842     ESTABLISHED 9150/weed
tcp6    6561      0 192.168.65.42:8888      192.168.65.42:41580     ESTABLISHED 9150/weed
tcp6   10021      0 192.168.65.42:8888      192.168.65.42:43408     ESTABLISHED 9150/weed
tcp6   22988      0 192.168.65.42:8888      192.168.65.42:58074     ESTABLISHED 9150/weed
tcp6   13524      0 192.168.65.42:8888      192.168.65.42:54530     ESTABLISHED 9150/weed
tcp6   50901      0 192.168.65.42:8888      192.168.65.42:47330     ESTABLISHED 9150/weed
tcp6   60016      0 192.168.65.42:8888      192.168.65.42:49604     ESTABLISHED 9150/weed
tcp6   53926      0 192.168.65.42:8888      192.168.65.42:44792     ESTABLISHED 9150/weed
tcp6   50842      0 192.168.65.42:8888      192.168.65.42:34476     ESTABLISHED 9150/weed
tcp6   32611      0 192.168.65.42:8888      192.168.65.42:39410     ESTABLISHED 9150/weed

LimitNOFILE is set to 65535. I'm seeing literally thousands of open and non-closing sockets. I'm using filer as s3 gateway.

System Setup

• List the command line to start "weed master", "weed volume", "weed filer", "weed s3", "weed mount":

weed filer -s3 -master=192.168.65.42:9333 -defaultReplicaPlacement=000 -s3.config=/etc/seaweedfs/config.json -s3.allowEmptyFolder -s3.port=9000

• OS version: Debian 10.9
• output of weed version: version 30GB 2.38 2327c07 linux amd64
• if using filer, show the content of filer.toml:

[leveldb3]
# similar to leveldb2.
# each bucket has its own meta store.
enabled = true
dir = "/var/lib/weed/filer"                             # directory to store level db files

[chrislusf]

If possible, can you send a "kill -3" to the weed filer process, and collect the screen output? This will kill the process, btw.

[divanikus]

Sure. Here it is

[divanikus]

I have reverted just filer to 2.35 and my cluster work A LOT faster.

[chrislusf]

Sure. Here it is

Is that all the output or is it truncated?

[divanikus]

Oh, sorry, yes, it was truncated, because it 4MB long. Full one is here
out.zip

[chrislusf]

thanks! added a fix.