-
Notifications
You must be signed in to change notification settings - Fork 968
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Request] With fread, do not throw 32 bit large file support error if nrows is specified #949
Comments
I'd also like this to be implemented. It should be handy for 'nrows=0' situations or when one ins only interested to read in small portions via nrows. |
Please retry with latest dev 1.10.5. From NEWS :
There's a chance it will work now. If not, please file a new issue. For Windows you can download a binary .zip for dev 1.10.5 by following instructions here. |
On a 32-bit system it's not possible to memory-map a file bigger than 4GB in its entirety. It's possible to memory-map only part of the file though. Trying to map the whole file will raise an error
Of course we can map only the first 4GB of the file, and then hope that that would be sufficient to read the amount of rows requested by the user. But what if not? It is possible to work around this by carefully managing the window of the file which is currently mapped; however it would increase the code complexity significantly. In addition, it is unclear how to test this functionality because finding a 32-bit machine is challenging nowadays. |
I no longer have access to a 32-bit machine, so I can not test it anymore. |
For the fread function I wonder if it's possible to not show the 32-bit large file support error if I actually specified the nrows? I am only interested in the first 5000 rows, say, and I need not read in the whole file on a 32bit Windows machine.
The error message:
The text was updated successfully, but these errors were encountered: