Significant speedups with minimal changes #62
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When unpacking the list of coordinate points, each point pair is unpacked one at a time, making it very slow for long lists of points. So a fairly simple change to read all points at once, which btw is in line with the existing convention elsewhere, leads to 5-9x speedup when reading shapes, with bigger gains for files with more complex shapes (more points).
Implemented basic version from #52 , but without numpy and memoryview to keep it simple. Not sure if any practical/noticable speedup, since reading the offsets is only a one-time issue and very little data involved. More out of principle.
Since each row/record has the same format every time, instead of a regular unpack, precompile the format using struct.Struct to unpack faster. Leads to roughly x1.33 speedup. In addition, when users arent worried about memory (ie the "records" method), we can exploit this by reading all records to memory at once. Leads to 15-20x speedup.
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Up to 8-20x reading speedups in a series of changes based on the principle of unpacking bytes in batch instead of multiple times, while maintaining the general convention and style of the original pyshp code. See commits for more detail.
Changing just one line of code, i improved a weakness in shape reading leading to significant speedup of 5-8x. For instance itershaping through level1 of the global administrative units database is reduced to 10 secs from 80 secs. Greatest speedups for complex shapes, least or nonexistent for singlepoint geometries.
Also implemented a minimal version of the shapeindex speedup suggested in Performance improvement for
Reader.__shapeIndex#52 .Finally a general 1.3x speedup for reading records by "precompiling" the byte format, and more importantly a 15-20x speedup when reading all records at once.