descriptor too large for set size

[cvuillemez]

Hello,
I'm experiencing an important issue when using tcp.lua with lots of client connections.
The plugin crashes and report this information:

1509130702213999114 [info] input.tcp detaching received: 1 msg: process_message() /usr/share/luasandbox/sandboxes/heka/input/tcp.lua:118: bad argument #1 to 'select' (descriptor too large for set size)

It appears this plugin is not directly in question, but the lib LuaSocket is.
When Hindsight holds at least FD_SETSIZE open file descriptors, socket.select() returns an error. FD_SETSIZE is defined to 1024 by a macro in Glibc (doc).

socket.select() leverages on this macro, so when I start few hundreds of TCP clients, total of open file descriptor goes quickly above 1024 (with my settings, when no clients are connected, Hindsight still has ~250 fd, so the 1024 value is very limiting, especially for infrastructure which need to connect thousands of clients).

Maybe I could start more (Hindsight) instances to spread connections, but it's not a good choice, I can't be sure they will always be well distributed, and any program could flood input and makes plugin to crash.

This LuaSocket limitation is mentionned in this issue, where @diegonehab suggest to use libevent .

Regards.

[cvuillemez]

Any idea about this issue ? I think it can be so problematic ... 1024 open files is easilly reached.

[diegonehab]

The simplest solution is to implement a poll() function in C and use it instead. I'd be happy to help you design an API that works well with LuaSocket and help you implement it. Once you have something working, send a pull request and we can add it to LuaSocket.

…

On Wed, Jan 24, 2018 at 11:46 AM, cvuillemez ***@***.***> wrote: Any idea about this issue ? I think it can be so problematic ... 1024 open files is easilly reached. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#176 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABj05T8lB2USGVVRp1NGVdUoPLBsOzQJks5tNzRBgaJpZM4QKQz0> .

[cvuillemez]

@trink ?

[trink]

This isn't work we can take on at the moment.

[cvuillemez]

@diegonehab : the todo work is completely inside LUA socket project so I suppose there is for now no contributor to implement this feature ? It sounds strange for me .. it means all applications using LUA in the world don't need to manage large sets of socket/file descriptors , or maybe they use another LUA lib (?)

[diegonehab]

There is no LuaSocket project. There is a dispersed set of power users that occasionally need something and send patches. These rarely come with tests, documentation, or with any guarantee that they will work on any system other than what the user needs for his/her current project. Right now, I don't have cycles to work on LuaSocket. I am a professor and researcher in Computer Graphics. When I started the project in 1999, I was an undergraduate. I have never received a cent for my work on LuaSocket and never asked for anything in return. There are no regular contributors to LuaSocket willing to work on improving the library for the library's sake. I have tried to encourage other developers to contribute code and documentation but with limited success.

…

On Tue, Feb 6, 2018 at 12:07 PM, cvuillemez ***@***.***> wrote: @diegonehab <https://github.com/diegonehab> : the todo work is completely inside LUA socket project so I suppose there is for now no contributor to implement this feature ? It sounds strange for me .. it means all applications using LUA in the world don't need to manage large sets of socket/file descriptors , or maybe they use another LUA lib (?) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#176 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABj05R-0JwG4TN2stHaXAgRiUJxuXQcDks5tSFy6gaJpZM4QKQz0> .

[cvuillemez]

OK I have a better undestanding of the context. But now I don't understand why a so important module is not developped / maintained by the LUA team.
Now It appears this bug will certainly not be solved soon ... but thanks to you for your (free) work !