service not responding: Too many open files

[flyingmrwang]

Description
A clear and concise description of what the bug is.
The service become not responding and log says too many open files.
logs after un-responding:
[ERROR] evhtp.c:3101 ODDITY, resuming when not paused?!? :: (errno: Connection reset by peer)
[ERROR] evhtp.c:3101 ODDITY, resuming when not paused?!? :: (errno: Connection reset by peer)
[ERROR] evhtp.c:3101 ODDITY, resuming when not paused?!? :: (errno: Connection reset by peer)
[ERROR] evhtp.c:3101 ODDITY, resuming when not paused?!? :: (errno: Connection reset by peer)
E0311 14:05:31.892633692 95 tcp_server_posix.cc:213] Failed accept4: Too many open files
E0311 14:05:33.652132566 93 tcp_server_posix.cc:213] Failed accept4: Too many open files
E0311 14:05:33.653766932 94 tcp_server_posix.cc:213] Failed accept4: Too many open files

BTW, I have already increased max socket number with "--ulimit nofile=5000:5000". But seems this could not solve the key issue.

Triton Information
What version of Triton are you using?
20.11

Are you using the Triton container or did you build it yourself?
docker

To Reproduce
Steps to reproduce the behavior.
start server and run: lsof | grep "can't identify protocol" to find unclosed sockets

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).
all my hosting models face the same problem. So it should not be a matter about model.

Expected behavior
A clear and concise description of what you expected to happen.
Stable service with neither too much "can't identify protocol" nor log of "Too many open files" which cause un-responding.

[deadeyegoodwin]

Closing for inactivity, reopen if new information.

[yushcs]

I met the same problem, did you find out how to solve it?

[GuanLuo]

Can you share more information so we may reproduce the issue?

[orkenstein]

I deploy triton in k8s and after 6 or less days of activity it just stops responding

[TannedCung]

Same thing happened to me this morning, everything works well for months and suddenly got signal 11

[gerasim13]

i have the same thing. my server stops after a few hours of work and this is 100% reproducible.

[debasishaimonk]

has anyone got the reason why this happens?

[TranThanh96]

any update on this please?

[handoku]

same problem

[Tushar-ml]

Same problem

[kadmor]

same problem

[Jiang-Jia-Jun]

same problem, any updates here?