-
Notifications
You must be signed in to change notification settings - Fork 966
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SHM size exceeded Error #2272
Comments
For anyone finding this same issue, the short term fix is to increase the shared memory size of the container or in your OS if your /dev/shm is too small. Typical modern OS's use 50% of your available memory. In my 64G amazon ec2 instance it I set the docker container to use:
|
Agreed. Sorry about that.
Please try dev 1.10.5 and open a new issue if it's still a problem. |
Guys, I'm the one who reported zUMIS/19. I've tried your dev 1.10.5 and it works perfectly. Great work guys! |
Feature request, check the size of /dev/shm to see if it matches the mmap size. In my case I was using R inside a docker container and the default size for /dev/shm is 64M. When the code does a mmap on a piped stream (such as with hadoop fs -cat /myfile.csv) it will only read the shm size bytes from the pipe into mmap. It does not report an error via the C api which I suspect is normal. However, debugging why fread complained about the file format resulted in a deep dive into the R and C code of data.table to discover it uses this mechanism. The error reported (random message based on where my pipe happened to be cut off):
This can be reproduced by doing the following:
In the code: https://github.com/Rdatatable/data.table/blob/master/src/fread.c
Around line 788 perhaps it should check the size of /dev/shm to see if it matches the file it just read into memory. In my case in docker here is the verbose output of the failed test condition:
Expected results:
In addition, when it fails it knows internally the line # and other useful information when it failed. I had to zero in on the value by hand before I discovered the verbose flag. Would be nice if normal error messages indicated the row #. vebose=T shows the location when it calculates the # of delimiters such as for this test case would have be useful output on error (since I knew the file had 20M records):
The text was updated successfully, but these errors were encountered: