on_demand for xlsx and fixed MemoryError for mmaped big xls file #368
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…oryError for big files
…o work with xlsx Add unittest case: xls on_demand mmap: memory error Add unittest case: xlsx on_demand: loads sheet faster, unload sheet does work note: xls test file size reduced to 99.9MB to be able to push to github.
…e acceptable upper limit time. Make sure all variables are used.
|
Please see the note under https://github.com/python-excel/xlrd#xlrd. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
xlsx:
on_demandnow also works withxlsxfile, significantly reduce the time of loading workbook if you only need to access a certain sheet's data. Unittest is also included.xls:
on_demandwithmmaped used to have no effect onxlsfile, whose has non-contiguous sectors. Last line of_locate_streamcaused it to load the whole data frommmaped file into the memory, time consuming andMemoryErrorfor bigxlsfiles (>100MB). Now themmappart work as intended, only needed range of memory is loaded up onsheetaccess. Thus, reduce the access time and memory in case you only need a certain sheet's data. Unittest is also included.