filter: Improve speed of checking duplicates #1466
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pandas's isin(set) converts the set into a list, making each value check scale linearly with the size of the set.¹ In the usage here, it's unfortunate because the size of the set grows with each metadata chunk processed. That means the entire operation scales with (size of chunk * size of metadata). Do the equivalent but much faster by first making a set from the current chunk's index then intersecting that with the set of previously seen IDs. This is faster because both of these steps scale linearly² with the size of the chunk, not the size of the metadata, meaning the entire operation scales with (2 * size of chunk). ¹ <pandas-dev/pandas#25507> ² <https://wiki.python.org/moin/TimeComplexity#set>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #1466 +/- ##
==========================================
+ Coverage 68.82% 68.85% +0.03%
==========================================
Files 69 69
Lines 7595 7607 +12
Branches 1860 1861 +1
==========================================
+ Hits 5227 5238 +11
Misses 2086 2086
- Partials 282 283 +1 ☔ View full report in Codecov by Sentry. |
genehack
approved these changes
May 17, 2024
|
Nice one, @victorlin! |
|
Very clever improvement. Thanks @victorlin! |
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of proposed changes
pandas's
isin(set)converts the set into a list, making each value check scale linearly with the size of the set. In the usage here, it's unfortunate because the size of the set grows with each metadata chunk processed. That means the entire operation scales with(size of chunk * size of metadata).Do the equivalent but much faster by first making a set from the current chunk's index then intersecting that with the set of previously seen IDs. This is faster because both of these steps scale linearly with the size of the chunk, not the size of the metadata, meaning the entire operation scales with
(2 * size of chunk).Related issue(s)
parseandfilter, handleAugurErrorglobally #918Checklist