Running even moderately sophisticated systems frequently results in ezmsg opening too many file descriptors. Whether it's open files, open ports, shared memory handles, etc... We very quickly surpass 256 which is the default ulimit for many linux distributions. Some ezmsg (anti) patterns very quickly result in over a thousand open file descriptors which is not great, but also nowhere near the per-process limit for modern operating systems.
The default number of file descriptors varies by operating system but is typically quite low, 256, 1024, etc.. This was apparently done to protect the operating system from malfunctioning software and the previous implementation of these file descriptors was quite costly at some point. These days, we can theoretically use a lot more of these, and ezmsg certainly does.
It's simple to change the current ulimit for file descriptors using resource.setrlimit and other software that uses a lot of file descriptors have considered doing this: python-trio/trio#2016.
I don't want to do it recklessly -- only where ezmsg has execution entrypoints: things like the GraphServer process and anytime we call the GraphRunner entrypoints. I think we do it by getting the current limit at the start of execution, setting it to a higher value controlled by an overrideable-via-environment-variable module-scoped global, then reverting it to its original value after we're done. Care should be taken to determine how this carries over to subprocesses (both via spawn and fork and how this is implemented across platforms (win/mac/linux). We should also consider what value we default to. ~1000 is probably too low, ~1,000,000 is probably too large. Maybe ~100,000?
Running even moderately sophisticated systems frequently results in ezmsg opening too many file descriptors. Whether it's open files, open ports, shared memory handles, etc... We very quickly surpass 256 which is the default ulimit for many linux distributions. Some ezmsg (anti) patterns very quickly result in over a thousand open file descriptors which is not great, but also nowhere near the per-process limit for modern operating systems.
The default number of file descriptors varies by operating system but is typically quite low, 256, 1024, etc.. This was apparently done to protect the operating system from malfunctioning software and the previous implementation of these file descriptors was quite costly at some point. These days, we can theoretically use a lot more of these, and ezmsg certainly does.
It's simple to change the current ulimit for file descriptors using
resource.setrlimitand other software that uses a lot of file descriptors have considered doing this: python-trio/trio#2016.I don't want to do it recklessly -- only where ezmsg has execution entrypoints: things like the GraphServer process and anytime we call the
GraphRunnerentrypoints. I think we do it by getting the current limit at the start of execution, setting it to a higher value controlled by an overrideable-via-environment-variable module-scoped global, then reverting it to its original value after we're done. Care should be taken to determine how this carries over to subprocesses (both viaspawnandforkand how this is implemented across platforms (win/mac/linux). We should also consider what value we default to. ~1000 is probably too low, ~1,000,000 is probably too large. Maybe ~100,000?