New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Opening concurrent file handles for all instancefiles risks breaching OS limit. #352
Comments
Thanks for the report/request! It's certainly possible to do -- it is how earlier versions worked. I switched to the built-in click behavior when adding support for stdin, as it seemed like a nice simplification. Before I pursue this, a quick question: are you running into this issue today, do you foresee running into it (soon), or is the issue purely theoretical? I'll want to look into click's capabilities to see if it can open the files lazily before pursuing a change here. |
Thanks for the quick response. The issue already occurs for me on a project where I'm globbing around 2,000 files. However, it's simple to work around as users can normally raise the ulimit value to anything they like (eg |
I checked and click does have a lazy mode for file objects, but it rigs them up to close as part of the command exiting. I don't think that using it naively will solve this. I'll have this near the front of my queue when I'm next able to devote some time to this project (likely in a little over a week). |
Minor update: I've been tinkering on this but it's not working quite as smoothly as I wanted. The click lazy file object does an eager open/close to catch some errors (e.g. permissions) early, which messes with some of the fifos in the testsuite. It works in a live context, when I use a shell to write a fifo, so I must be doing something slightly wrong in the tests. I might also file a click bug report, since I think the open/close behavior is slightly unsafe for special file types. I'm going to try to figure out how to fix the fifo tests, but I do have some options for handling this all in the worst case. |
Tested and working perfectly. Thanks very much! |
Wow, thanks for testing before I even cut the release! I was meaning to push this out yesterday but got a bit delayed. I've just dropped v0.27.3 with this included -- it's the primary update in there. Let me know if you see any issues with the released version. 😄 |
The use of the
BinaryIO
type forinstancefiles
causes file handles to be opened concurrently by Click for all instance files supplied at the command line.A user may, for example, run a glob expansion on a number of targets that exceeds the limit of open files they are permitted by their operating system. Currently
check-jsonschema
fails in this case withOSError: [Errno 24] Too many open files
. It would be nice if files were opened sequentially so that the user could supply a far greater number of arguments to a single command without error.A minimal case for simulating this on Linux is as follows:
Could it be possible to type the
instancefiles
argument as astr
orPath
and only open the file handles one by one as needed?The text was updated successfully, but these errors were encountered: